A teacher-teacher framework for clinical language representation learning
Authors: Feiqing Huang
Shenghan Zhang
Sara Morini Sweet
Tianxi Cai
Venue: NeurIPS 2024 poster (2024)
Keywords: clinical language models
teacher-teacher framework
knowledge alignment
Primary area: natural_language_processing
TL;DR: This paper introduces a teacher-teacher paradigm where two pretrained LLMs achieve knowledge exchange and alignment through a two-step, few-epoch training of the LINE module with a well-designed alignment objective.

Abstract: In recent years, there has been a proliferation of ready-to-use large language models (LLMs) designed for various applications, both general-purpose and domain-specific. Instead of advocating for the development of a new model or continuous pretraining of an existing one, this paper introduces a pragmatic teacher-teacher framework to facilitate mutual learning between two pre-existing models.
By leveraging two teacher models possessing complementary knowledge, we introduce a LIghtweight kNowledge alignmEnt (LINE) module aimed at harmonizing their knowledge within a unified representation space. This framework is particularly valuable in clinical settings, where stringent regulations and privacy considerations dictate the handling of detailed clinical notes. Our trained LINE module excels in capturing critical information from clinical notes, leveraging highly de-identified data. Validation and downstream tasks further demonstrate the effectiveness of the proposed framework.

A teacher-teacher framework for clinical language representation learning
Feiqing Huang"
Shenghan Zhang"
Harvard T.H. Chan School of Public Health fqhuang@hsph. harvard. edu
Harvard Medical School
shenghan
bhenghan_zhang@hms zhang@hms hhwvaard........................
Sara Morini Sweet Harvard Medical School
Tiani Cai
Harvard T.H. Chan School of Public Health
sara_morind @hms hwwvard.d.... edu
Harvard Medical School
tcai @hsph. harvard. edu
Abstract
In recent years, there has been proliferation of ready-to-use large language mod- els (LLMs) designed for various applications, both general-purpose and domain- specific. Instead of advocating for the development of new model continuous pretraining of an existing one, this paper introduces a pragmatic teacher-teacher framework to facilitate mutual learning between two pre-existing models. By lever- aging two teacher models possessing complementary knowledge, we introduce a Lightweight kNowledge alignmEnt (LINE) module aimed harmonizing their knowledge within a unified representation space. This framework is particularly valuable clinical settings, where stringent regulations and privacy considerations dictate the handling of detailed clinical notes. Our trained LINE module excels in capturing critical information from clinical notes, leveraging highly de-identified data. Validation and downstream tasks further demonstrate the effectiveness of the proposed framework.
Introduction
Clinical data frequently appears in various forms, with each format capturing overlapping but complementary aspects of patient information. For instance, narrative details in clinical note may be processed into structured list of clinical concepts for research purposes. Similarly, a CT scan report provides descriptive findings that are directly linked scan. Ideally, both forms of data would be jointly accessible; however, practical challenges, such as privacy restrictions, often limit us to only one form. This limitation raises an important question:
visual data the corresponding
Could we use the accessible form of data to approximate or represent the unobserved one?
To explore this question, we can leverage the capabilities of large pretrained models. Recent years have seen the emergence of large language models (LLMs), which have demonstrated impressive performance across range of tasks, including prediction, generation, and representation learning
27 34 In the clinical domain, many pretrained LLMs such as CODER [38] UmisBERT [5]. Clinical BioBERT (2) SapBERT [33. Pub [8), and BioBERT
are based on BERT architectures with approximately 300 million parameters. These models are typically initialized with BERT weights and then undergo further pretraining on biomedical texts, process often referred to
"Equal contribution.
'Coreepondence: tcai@hsph. harvard. edu
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

as continual pretraining. While this approach allows the model to integrate both general and domain- specific knowledge, is also tmm-consuming and resource-intensing The recent proliferation of off-the-shelff clinical LLMs now prompts a pragmatic idea:
Can the pretrained models be made directly exchange knowledge with one another?
Inspired by the two questions posed above, we propose a teacher-teacher framework for clinical representation learning. The core idea is as follows: given paired training data in two distinct forms and two pretrained models with complementary knowledge bases, the framework processes each form of data separately through one of the models. It then processes the resulting representations through newly proposed module and further aligns them within a unified representation space. This alignment entails mutual teaching process, where both models act as teachers, exchanging their knowledge. The complementary knowledge arises from two sources: first, the complementary information inherent to each form of data, and second, the distinct knowledge embedded in each pretrained model due to differences their original training corpora. We summarize our key innovations as follows:
LINE module for teacher-teacher learning: We propose lightweight knowledge alignment (LINE) module facilitate effective teacher-teacher learning. This module takes representations from two pretrained teacher models and projects them into unified representation space, guided by an alignment loss and a relational loss.
Efficient training with residual information recovery: We adopt a two-stage, few-epoch training process focused solely the LINE module, keeping the pretrained model weights frozen. In the first stage, we train the LINE module for a few epochs to achieve initial alignment and to capture residuals (complementary information across forms of data). In the second stage, these residuals are used to refine the training data for further few-epoch training, offering insights into the complementary information between data forms.
Generating cross-form representation: Once trained, the LINE module enables cross-fort representation for downstream tasks. When only one data form is available, the corresponding teacher model and LINE projection can generate a proxy for the missing data, enabling its use in the downstream tasks.
We evaluate our teacher-teacher framework on two fronts: (1) whether knowledge alignment can improve the performance of one or both teacher models across different tasks, and (2) whether the available data form can be used to generate better proxies for the other, missing data form.
1.1 Clinical use case
While using one form data to represent another is relevant question across various fields, demonstrate the utility of our proposed method through a real-world clinical use case. This example particularly relevant to clinical data privacy concerns, highlighting its practical importance. Specir- ically, we focus on two data forms: clinical notes and their corresponding structured lists of clinical concepts. Clinical notes play crucial role in numerous applications, such disease diagnosis, disease progression prediction, patient classification, and risk assessment [25] 22 [4] However, working with real clinical notes is subject strict regulations due to privacy concerns. Sharing clinical notes between institutions necessitates approval from an institutional review board (IRB), adding layers of regulatory complexity and substantially extending project timelines.
These challenges are a major barrier to scaling up clinical datasets for pre- meetrriining As an alternative, efficient software [c.g. 36 23 can directly extract structured clinical concepts from clinical notes. This approach offers two advantages: First, because clinical concepts are drawn from a predefined dictionary specifically, the Unified Medical Language System (UMLS) [ it reduces the risk of including sensitive information, making data sharing across research groups more feasible. Second, known relationships among many clinical concepts be further incorporated to form a structured graph, capturing both the textual content and interrelationships among concepts.
In this use case, we treat the clinical note the missing data form and reframe our question accordingly: Can we leverage the list of clinical concepts to effectively summarize and provide proxy for the content in the clinical note?

---

1.2 Related works
Diffferntt modes of learning Existing knowledge within pretrained LLMs varies based on their learning modes and domains of their pretrained data. The first mode, learning by generalization, involves training models on extensive datasets to generalize knowledge through example-based learning 1 [27] These models encode knowledge directly into their embeddings, which capture syntactic and semantic relationships in texts. Numerous studies have demonstrated the efficacy of LLMs in encoding syntactic heuristics and retaining factual knowledge to varying extents 77 [9
28 18 21. Recent advancements, exemplified by models such as Sentence-TS 17. E5 31, BGE 34 and GPT-4 []. have focused generating high-quality textual representation for downstream tasks. However, in specialized clinical domains, vocabulary diverges significantly from general corpora, requiring that these general- pureroe models undergo additional training on biomedical data to ensure relevance and accuracy. Models like BioBERT [1]] and PubmedBERT [8), pretrained on biomedical texts, have demonstrated superior performance compared to BERT in biomedical-specini tasks. Clinical BioBERT [2] further augment their training data with MIMIC datasets to incorporate medical knowledge derived from real-world medical practices. Nevertheless, scaling up training data in the clinical domain poses challenges due to privacy regulations, as previously noted.
The second mode, learning by integrating pre-existing knowledge, focuses on encoding established factual associations between concepts into the model's 's representation space. In the clinical domain, knowledge repositories such as UMLS [5] offer comprehensive biomedical ontologies and relatedness information between concepts, which can be treated as existing knowledge graph and harnessed during the pre- pre-training phase. For example, SapBERT [3] leverages synonymous relationships extracted from UMLS during sulf--uper pretraining, strengthening the model's understanding of semantic similarity. UmlsBERT [1] incorporates semantic type information from UMLS, enabling deeper semantic understanding of clinical texts. Furthermore, CODER (38] encodes more granular relationships, such as "may prevent or treat" and "may cause", into their knowledge networks, enhancing its ability to capture medical associations. By integrating such pre-existing domain-specific knowledge graph during training, these models show improved ability to navigate the intricacies of specialized domains, leading to better performance in domain -epaai--peccfic tasks 13 38).
Distinct from these two modes, the proposed teacher-teacher framework adopts the third mode, learning by alignment. This mode of learning involves creating a joint representation space between two models using paired data, such transcript, or multilingual translated texts. This approach focuses on teaching models to distinguish between paired and unpaired data, making it especially effective for multimodal applications. For example, the seminal CLIP framework (20) uses different encoders for text and images to embed each mode of information separately. It then employs contrastive loss to align these multimodal embeddings by end-to-end training on both encoders, ensuring that related text and image pairs are closely aligned in the representation space. Similarly, Li al. [12] utilize a text encoder and a text-referred audio encoder with inter-modality and intra-modality training, enhancing the model's s ability to integrate complementary features from different views. Wang et al. (33) propose an efficient global-local alignment framework for co-training language with videos, and its improved performance further highlights the advantages of learning the homogeneity as well as heterogeneity across language and visual modalities. For multilingual translation tasks, alignment learning has also demonstrated remarkable efficacy. For example, Reimers and Gurevych [2] introduce a teacher-student network model in which a fixed teacher network, well-versed in a target language, guides a student network, initially proficient a different language, to emulate the teacher's 's knowledge. This method not only enhances translation accuracy but also creates joint representation space that offers potentially interesting insights into lins-iing differences.
image with its textual description, audio/video with its
Other related works In a broader context, our approach is remotely related to generative domain adaptation; however, unlike generative domain adaptation, our method requires paired data during training. Traditional methods in this field often utilize generative models to synthesize samples across domains, helping the model generalize effectively in new contexts. Some example frameworks include generative adversarial networks 40), variational autoencoders [32], diffusion models [] or generative pretrained transformers 35 However, our framework diverges from these in that we specifically deploy two pretrained teacher models, each with distinct expertise and capabilities. This design allows the models not only to align but also to exchange complementary knowledge during the learning process, potentially enhancing the performance of one both teacher models.
3

---

Training Data
Teachers
LINE Module
Clinical Notes
Teacher General Purgose
Projection Layer 1
There focal on
plonal effusion proumathanax Blaaterl mosular opacites that moyyrrallyep repole
AlignmentLoss
normall
Clinical Concepts
Residual Refinement
Unifed Space
focal consokttation
embeddlings
Residual
no pluu ril effidion no penumathoro
align
Teacher LLM with Domain
Besidul concapts
Dictionary of concopts
Knowledge
caciumecianiral
Projection Layer
Hisple
mit Arld
Multheed
Knowiedge
The proposed teacher-teacher framework is illustrated for specific case. The two forms of training data are unstructured clinical notes and structured clinical concepts represented in a relational graph. Two pretrained teacher LLMs independently embed these forms of data, with the LINE module facilitating mutual learning and alignment. Through this process, the embeddings of the two data forms converge toward shared representation space, enabling knowledge exchange and cross-form representation between the two teacher models.
Figure 1:
2
The teacher-teacher framework
2.1 Training data and alignment objective
This section provides a detailed illustration of our LINE framework for specific clinical use case, aligning clinical concepts with corresponding clinical texts. The algorithm can be extended to support a broad range other alignment tasks, as discussed in Section[4] Specifically, our paired training data take the form (CLINICAL TEXT, LIST OF CLINICAL CONCEPTS), where each list of clinical concepts derived from the clinical texts. We further augment the clinical concepts with known relationships from UMLS to form structured graph. The primary objective is to create "summarized" embedding for the list of concepts and align with the projected text embedding, as illustrated in Figure
2.2 Teacher models
A key design principle in our framework is selecting two teacher models with complementary knowledge bases. To this end, we choose (1) a general-purpose LLM pretrained on large-scale, general-domain text, and (2) Nomain-specific LLM with established knowledge of biomedical concept relationships. In this setup, the general-purpose purpose model (Teacher 1) learns factual relationships between clinical concepts from the domain-specific model (Teacher 2), while Teacher 2 enhances its grasp of the contextual connections linking these concepts. the paper, we explore two combinations of teacher models: BGE+CODER and GPT-4+CODER.
Teacher 1: General purpose LLM BGE is general-purpor embedding model trained on large. diverse language datasets. Initially trained on 100 million text pairs from open web corpora, BGE further underwent multitask learning on various retrieval and reranking tasks, including the CMedQA- v2 medical question and answer dataset [39]. It ranks highly on the Massive Text Embedding Benchmark (MTEB) dataset [16], which covers tasks like classification, reranking, retrieval, STS, and paired classification tasks. GPT-4 is large-scale, multimodal model considered as one of the sate-offthe-ae tools for language modeling and generation. Although model architecture and trained weights are not publicly available, GPT-4 can generate embeddings for textual data, accessible through the OpenAT API. For our experiments, we used the "text-embedding- 3-siiinnn----lll--- model, which produces embeddings with dimension of 1536.

---

Teacher 2: LLM with domain knowledge CODER is initialized by PubMedBERT 8 and trained on the UMLS 2020AA release relation triplets, encompassing 4.27M concepts. 15.48M terms, and 87.89M relations across 127 semantic types, 14 relation types, and 923 relationship attributes. It employs contrastive learning with relation triplets, i.e. (head entity, relation, tail entity). CODER demonstrates superior performance in term normalization and disease classification tasks compared to SapBERT, Clinical BERT, PubMedBERT, and BioBERT.
In our two combinations of teacher models, the general purpose LLMs (BGE and GPT-4) with their extensive and diverse training, generally outperform CODER. This imbalance might suggest that they cannot learn from CODER. However, our experiments suggest that not only can CODER benefit from the knowledge of the general-purpose purpose LLM, but the performance of BGE or GPT-4 also improves on certain tasks with alignment training. We hypothesize that the alignment process regularizes the embedding space of the general purppse I.L.M by incorporating CODER's association graph, resulting in more diversified and robust representation space.
2.3 Lightweight kNowledge alignmEnt module (LINE)
Our framework eliminates the need for any further training of the original teacher models. Instead, we introduce a trainable Lightweight kNowledge alignmEnt module (LINE), which explicitly models projection operations to align the embeddings from the general-purpose LLM and the domain-specific LLM into a unified representation space.
First, we derive clinical text embeddings and concept embeddings from the ceneral-purposs LLM and domain-specific LLM using pretrained checkpoints available through Huggingface or the OpenAI API, denoted as t (0) E Rd and c(0) E Rd: respectively. These initial embeddings retain the original knowledge encoded within each teacher model. We then incorporate trainable projection layers specific to each LLM to refine and align these embeddings.
Projection layer for general purpose LLM: Fully-connected layer The projection layer for the output embeddings (0) from BGE or GPT-4 is implemented as a simple fully-connected linear layer:
F(t (0) (C).
This approach aligns with prior work 3 activation functions.
20
where linear projections used without non-linear
Projection layer for domain-specife LLM: Multihead graph attention This layer is designed to reinforce the domain-specific knowledge embedded CODER from the UMLS ontology. We focus on modeling seven broad categories of UMLS relations: parent-child hierarchy, synonyms, related and possibly synonymous, broad relationship, narrow relationship, non noo-syyonyynos//nrroowwaaa relationship, and quantifiable relationships. Each relationship type is represented by a distinct adjacency matrix, and a multi-head graph attention (MHGA) layer assigns each type to a separate attention head. This design enables the layer to explicitly account for the unique structure of each relation. The outputs from all attention heads are aggregated and projected into the joint representation space. Formally, for any concept with an initial CODER embedding (.)) we have:
C c: cc)) = averagewert GraphAttn, (W.c(") adjacency matrix,))
where W. E Rd, xd. is the projection matrix that maps the initial CODER embeddings to the same dimension as the BGE or GPT-4 embeddings. Additional details on the handling of negatively mentioned concepts are provided in the Appendix.
2.4 Training strategy
To enable the two teacher models to enhance each other's knowledge in a time-efficient and resource- efficient way, we adopt two-stage, few-epoch training strategy exclusively for the LINE module. This approach effectively aligns the embeddings from the general-purpose and domain-specific LLMs without any additional training of the original pretrained models. It also allows for the use of proprietary models, such as GPT-4, where access to the underlying pretrained model is restricted.
Unified representation via alignment The first stage involves an initial round of few-epoch training aimed at defining residuals, i.e. the differences between the learned representations of the two data forms. This stage is critical for capturing complementary information between the two models.
5

---

Specifically, we aim to align the embedding for clinical texts with its list of associated concept embeddings in the joint representation space. This alignment is quantified using monotonic score function 8, which measures the degree of alignment between the elements of each data pair. Formally, let P. = (fc..) Je
J. t.) denote positive pair, where t; is the clinical text embedding and [C.s)se[m.] are the corresponding concept embeddings with a total of m, concepts, for all i E [N]. The negative pair is defined as Da,"' ((cu)) JE[m.]= (p) for i' + i [M] The score
function is defined as s() s( [j(m)) t) - mean (ejm)))))) Our goal is to ensure
sp,,) 2 s(n,,.). ), indicating better alignment of positive pairs over negative pairs.
To account for complementary information in the positive pairs, we introduce data-dependent alignment weight Ps E (o, 1] to allow varying degrees of information overlap. For example, in extracting clinical concepts from clinical texts, misspellings can result in failure detect key concepts; see Table AA in the Appendix for concrete examples. A smaller P: value indicates more tolerance for misalignment, reflecting cases with less overlap between the text and concept embeddings. The alignment loss is defined as:
ΣE P'f
(spp)),
where f is a contrastive loss function, and we used the triplet loss [24] for our experiments. In our specific context, the weight Pi is computed as the percentage of words in the text captured in the concept list, i.e. P.
number words accounted for in text
total word count in text i
In addition. to preserve the integrity of existing concept relations, we incorporate a relational contrastive loss during training. Here, positive pairs represent known relationships, while negative pairs indicate absent or unknown relationships. This additional loss term ensures that information from the known association graph is retained throughout the alignment process.
Refinement through residual recovery Once the first training stage stabilizes, we move to the second stage, focusing on recovering residual information to further refine the alignment. The residual e, for each data pair is computed as:
e, t, mean((C.)))
E [N].
To further consolidate information from residuals, we make use of projected concept dictionary Cg. consisting of all clinical concepts from UMLS, projected using the trained G defined in ( For each residual e,, we select concepts from Cg that have cosine similarity of 0.9 or higher with e, The most similar concept from the selected subset, denoted as +1. is then added to the concept embeddings set, refining the training pair to (fct. Jiemm.+e)) t,) for further few-epoch training.
3 Training and validation
3.1 Data preprocessing and training
Our model, referred to as LINE, is trained with two configurations of teacher models: BGE+CODER and GPT-4+CODER. For our training dataset, we utilized 332K discharge notes and 2.2M radiology reports from 146K patients available in the MIMIC-IV database [10]. Given the structured nature of both types of clinical notes, we began by segmenting the notes into their respective sections. Radiology reports were divided into seven sections: examination, indication, technique, comparison, finding, procedures, and impression. Similarly, discharge notes were segmented into fourteen sections: chief complaint, major surgical or invasive procedure, history of present illness, past medical history, social history, family history, physical exam, pertinent results, brief hospital course, medications on admission, discharge medications, discharge disposition, discharge diagnosis, discharge condition, discharge instructions, and follow-up instructions.
To extract clinical concepts from these notes, we employed the Narrative Information Linear Ex- traction (NILE) software [6). NILE outputs a list of clinical concepts present in each segmented text, indicating whether each concept is mentioned positively or negatively. Figure provides an illustrative example. Next, we combined the original text with its extracted clinical concepts to create paired training data. This preprocessing resulted in a dataset of 7.2M text-concept pairs. During this process, we located the extracted concepts within the text and counted the number of words accounted for by the list of concepts. The alignment weight for each text was then calculated as number of words accounted for To construct the relational adjacency matrices for the CODER projection count

---

Table 1:
For each positive concept-text pair, we randomly selected 100 other texts to replace the original text, forming negative pairs. We then calculated the cosine similarity between the mean of the clinical concept list and the text, with embeddings generated by various models. By ranking these cosine similarities from highest lowest, we identified the rank of each positive pair among its negative pairs and computed the mean rank, mean reverse rank, and Top-10 1p-100a accuracy (Topl0@ Acc). Improvements over the corresponding teacher models are indicated by "V" for mean rank and "1" for mean reverse rank and Top Acc.
Models Metric
Mean Rank +
Mean Reverse Rank
Top10@ Acc
BioBERT
35.83
0.127
0.258
Clinical BioBERT
34.51
0.143
0.287
SapBERT
15.49
0,346
0.588
PubMedBERT
39.55
0.117
0.210
CODER
43.79
0.092
0,181
BGE
5.526
0,534
0.845
GPT-4
1.778
0.820
0.988
LINE
BGE-CODER
1.437]
0.8731
0.9971
GPT-+CODER
1.477]
0.8721
0.9951
layer, we leveraged the 2022AB release of the UMLS knowledge graph, which encompasses 9M concepts, defined by 4M unique identifiers (CUIs), and includes 25M relationships
We adopted the Adam optimizer with a learning rate of 10-3 3 and a batch size 128. The training process was divided into two stages as detailed in Section 2.4 We first trained the model for three epochs initially, during which we identified residual concepts not adequately captured. Then, using the model checkpoint from the first phase, recovered the residual concepts and refined the training pairs. The model was then trained for an additional two epochs. The alignment loss and the relational loss both converged rapidly. All experiments were conducted using an NVIDIA RTX 8000 GPU with 48GB of VRAM. The entire training process required less than 10 hours on a single GPU.
3.2 Validation on alignment objective
This section validates the effectiveness of our trained LINE model in aligning the representation spaces of the two teachers. To achieve this, we utilized holdout subset of patients whose clinical notes were excluded from the training set. This subset comprised 100K radiology reports from 92K patients. We applied the same data preprocessing steps, resulting in 275K text-concept pairs. Our evaluation method involved the following steps. For each positive pair, we randomly sampled 100 negative pairs by substituting the clinical text with different one. We then computed the cosine similarity between the mean embedding of the concept list and the text embedding for pairs, in line with our alignment objective. Finally, we ranked these pairs based on their cosine similarity scores, from highest to lowest.
To assess performance, we calculated the mean rank, mean reverse rank, and Top- 10 accuracy for all positive pairs. These metrics are presented in Table [ We compared our LINE model against several models: GPT-4, BGE, CODER, SapBERT, Clinical BioBERT, PubMedBERT, and BioBERT. As LINE with BGE+CODER achieves the highest alignment between concept and
shown in Table I text representations. outperforming the other models. Both LINE models show improved alignment compared to their respective teacher models, validating the consistency of our training objectives.
3.3 Validation downstream tasks
In this section, the performance of our trained LINE models on various downstream tasks is evaluated based on two main aspects: (1) determine whether the teacher models can benefit from mutual learning, and (2) to assess whether the LINE module can generate more effective proxies for clinical text using the list of clinical concepts. Specifically, we use the LINE-projected CODER embeddings for tasks involving clinical concepts and the LINE-prooected BGE or GPT-4 embeddings for tasks involving clinical text. Given that two model configurations are used, we refer to each configuration "LINE" in our tables, with "Teacher 1+Tacher 2" specified for clarity. Three types of downstream
7

---

Table 2: AUCs for detecting related pairs versus randomly selected pairs under various models. The classes of clinical concepts include parent-child hierarchy, siblings hierarchy, may treat or may prevent, classifies as, differential diagnosis, method of, and causative of. The last row lists the detailed number of positive pairs in each class. Improvements over the corresponding teacher models are indicated by "T".
Models Relation
Parent
Sibling
May Treat/Prevent
Classifies
Method Causative
BioBERT
0.785
0.754
0..42
0.842
0.665
0.722
0.708
Clinical BioBHRT
0.837
0.814
0.386
0.885
0.733
0.825
0.753
SapBERT
0,940
0.874
0,672
0.960
0,874
0.871
0,914
PudmedBERT
0.637
0,651
0.639
0.640
0.634
0.535
0.604
CODER
0.943
0.886
0.441
0.969
0.844
0.774
0,899
BGE
0.968
0,940
0.840
0,984
0,927
0,906
0,944
GPT-4
0.974
0,940
0.825
0,991
0,939
0.934
0,935
LINE
BGE+CODER
0.978
0.939
0.926 1
0.989 1
0.959 f
0,971
0,953 1
GPTE+CODER
0.977
0.932
0.931 f
0.988
0.938
0.965
0.947 f
relation pairs
125152
48252
14686
14073
9777
6588
3711
Table The mean the Fl scores for different models over five random initializations on i2b2
datasets. The results for BioBERT, Clinical BioBERT and UmlsBERT were directly copied from 15 for comparison. Here, the LINE projection applied to the token-level BGE embeddings. The best result under each metric is highlighted in bold font.
Dataset
i2b2 2006
i2b2 2014
Test. F1
Val. Fl
Test. F1
Val. Fl
BioBERT
94.6
Clinical BioBERT
93.0
UmlsBERT
94.4
94.9
94.3
CODER
98.0
96.9
97.7
97.8
BGE
97.2
96.1
97.1
97.1
LINE
98.1
97.2
98.1
98.1
tasks are considered: (1) a zero-shot clinical concept similarity evaluation task; (2) two standard i2b2 clinical NLP tasks; and (3) renal cancer recurrence detection task.
3.3.1 Clinical concept similarity
We evaluated the quality of the LINE-projected embeddings based on their ability to detect known relationships among 222K pairs of 159K concepts. These pairs were curated from several major UMLS relation classes, including "may treat or may prevent". "classifies" "differential diagnosis", "method of", and "causative". Detailed counts of positive pairs for each class are provided in the last row of Table 2 To ensure fair comparison with the other models, we removed the edges corresponding these evaluation pairs from the relational adjacency matrices used in the LINE model. We then assessed the model's ss ability to capture different types relationships by calculating cosine similarities between the embedding vectors of related pairs and comparing these to randomly selected pairs with similar semantic characteristics. For example, when evaluating the "may treat or may prevent" relationship, used related discase-drug pairs positive examples and compared them against negative pairs also composed disease-drug pairs.
We calculated the Area Under the Curve (AUC) of the cosine similarities to assess the models' ability distinguish known pairs from random ones. As shown Table GPT-4 and BGE outperform CODER and all other baseline clinical LLMs, serving as strong benchmarks. Notably. the LINE- projected embeddings achieve AUCs that not only exceed those of the original CODER model but also outperform the BGE benchmark in six of the seven relation classes and the GPT-4 benchmark in four of the seven. the remaining classes, LINE's AUCs are comparable to their respective BGE and GPT-4 benchmarks, with deviation less than 0.008.

---

Table 4: Performance metrics of sentence embeddings and their proxy embeddings generated from concept lists for various models, averaged over five-fold cross- voos-valiiation. Improvements over the corresponding teacher models are indicated by "1".
Concept
Sentence
Model
Precision
Recall
F1
Accuracy
Precision
Recall
F1
Accuracy
BioBERT
0.701
0.686
0.693
0.701
0.681
0.669
0.675
0.680
Clinical BioBERT
0.722
0.723
0.722
0.726
0.726
0.726
0.711
0.729
SapBERT
0.726
0.711
0.718
0.722
0.741
0.723
0.732
0.722
PuRMedBERT
0.715
0.697
0.706
0.708
0.725
0.711
0.718
0.708
CODER
0.711
0.691
0.701
0.708
0.707
0.706
0.706
0.698
BGE
0.728
0.695
0.711
0.711
0.778
0.768
0.773
0.764
GPT-4
0.718
0.687
0.694
0.711
0.805
0.787
0.791
0.781
LINE
BGERCODER
0.741
0.728
0.734 1
0.722 f
0.778
0.765
0.771
0.771 1
GPT-+CODDER
0.731
0.715
0.714
0.722 f
0.806 1
0.786
0.789
0.781
Table 5: Difference performance metrics between sentence embeddings and their proxy embeddings generated from concept lists, calculated using results in Table Reductions in difference, which indicate improved alignment, are marked by "L".
Model
Precision
Recall
F1
Accuracy
BGE
0.050 0.087
0.073
0.062
0.053
GPT-4
0.100
0.097
0,070
LINE
BGEACODER
0.037 | 0.075
0.037
0.037 0.075 $
0.049
GPT-4+CODER
0.071
0.059 |
3.3.2 Clinical NLP benchmarks
We evaluated our model on two standard biomedical named entity recognition (NER) benchmark tasks: the i2b2 2006 de- de-identification challenge 30 and the i2b2 2014 de-identification challenge [26] We followed the train/validation/eest splits specified in the original challenges, as detailed in Table 1 of (2). The datasets from 2006 and 2014 contain 317 and 43 label classes, respectively. The NER tasks involve tokenizing sentences and then classifying each token within the sentence. For this task, we used token-level embeddings from each model and the LINE projection was applied to token-level BGE embeddings. GPT-4 was not included due to the limited access to its token-level embedding. Our fine-tuning process followed the setting in [15] Specifically, single linear layer was added on top of each model and trained for 20 epochs. We adopted Adam optimizer and the learning rates for CODER, BGE and LINE were set to 2 10-5,2 x 10 and 5 x 5 x 10 respectively. The mean of the F1 scores calculated over five random initializations are reported in Table 3 and can be observed that LINE achieves the best overall performance.
3.3.3 Renal cancer recurrence detection
The dataset used for this downstream task comes from Mass General Brigham and was previously curated, fully de-identified, and annotated for another ongoing project. Here, we briefly describe the curation and annotation process. First, phenotyping algorithm (37] was applied to identify pool of renal cell carcinoma (RCC) patients. From this pool, 300 RCC patients were randomly selected, and clinician reviewed all notes to extract one Mingnostic-relevann sentence per patient. Each sentence was annotated into one of three classes: uninformative RCC recurrence, informative of past or current recurrence, and informative of recurrence. We then used NILE [36] to extract clinical concepts from each sentence, excluding those where NILE failed to extract any concept. This resulted in final set of 288 labeled sentences, each paired with a list of corresponding clinical concepts. The dataset comprises 136 uninformative sentences, 64 informative of recurrence, and 88 informative of no recurrence.
This classification task is designed to test how effectively proxy sentences, generated from concept lists, detect recurrence information. Specifically, we aim to understand how well the proxy sen-

---

tences perform in classifying uninformative, recurrence intrencceiffrmmtivee.. and non renureurreene--e recurrence-informativy cases, simulating scenarios where only concept lists, not full sentences, are available. To this end, we additionally include classification results using raw sentences as a benchmark for expected optimal performance. We used I.INE- CODER to embed the concept lists and L.NEE-rroecced GPT-4 or BGE to embed the raw sentences. A single linear layer was added on top of both models, and we fine-tuned it for 2000 epochs, with early stopping when the change in loss was less than 10-5, We used Adam optimizer with learning rate of 10 The same setup was applied to all models for fair comparison. On average, the fine-tuning process took approximately one or two minutes per model. The average performance metrics from five-fold cross-validation are reported in Table 4 As shown, when raw sentences are available, GPT-4 and BGE achieve significantly better classifcation results than the other benchmarks. LINE-Prope erojeeceee embeddings for GPT-4 and BGE manage to maintain a comparable level of classification performance. However, using only concept lists for GPT-4 and BGE results in substantial drop in classification performance. Notably, adding the LINE projection reduces this performance gap, as shown in Table supporting the effectiveness of the LINE module in better approximating sentence embeddings using concept lists.
4 Conclusion and Discussion
This paper introduces a teacher-teacher paradigm in which two pretrained LLMs align different forms of data to enable knowledge exchange and cross-form representation through a two-stage, few-epoch training of the LINE module, guided by a well-defined alignment objective. Our downstream analysis demonstrates that (1) the proposed LINE module effectively generates cross-form representations, and (2) alignment learning enhances performance for both models, even when one teacher model is more advanced than the other.
Although our primary focus is on specific clinical use case, this teacher-teacher framework has broad potential applications. One promising application is enhancing the indexing and searchability of non-textual data. For example, medical images (e.g., CT or MRI scans) are often accompanied by clinical notes from which clinical concepts can be extracted. By segmenting images into regions and associating them with clinical concepts, we create concept- based index of image content. The macher-teacher framework aligns each image segment with its concept list, allowing the concept list to serve as proxy representation for image content. This enables concept- or keyword- searches for specific image segments via embeddings, making non-textual data more searchable, The framework also supports alignment between modalities without direct pairing. For example, CT and MRT scans are seldom captured together for the same patient, but clinical notes serve as potential intermediary. In this scenario, clinical concepts extracted from notes are embedded by pretrained teacher model. Separate teacher models then embed CT and MRI images. The alignment objective splits into two tasks: aligning clinical concepts with CT images and aligning clinical concepts with MRI images. This dual alignment establishes an indirect connection between CT and MRI images, creating unified representation without direct pairing. These examples illustrate the framework's 's broad applicability in aligning disparate modalities, supporting cross-form searches and enhancing the utility of diverse data sources.
Our clinical use case further demonstrates the practical benefits of this framework in improving patient care and data security. By reducing the need for direct access to clinical notes, the framework minimizes privacy concerns and strengthens data protection. Additionally, facilitates the integration of up-to-date LLMs, improving performance in tasks essential to patient care and potentially leading to better outcomes. The framework's 's use of external, factual knowledge also reduces misinformation risks, fostering trust in Al-assisted clinical tools.
Despite its strengths, the framework has two notable limitations. First, its reliance on paired data may limit its applicability in situations where aligned datasets are scarce. Although intermediaries can help bridge certain gaps, the framework's effectiveness remains constrained cases where no suitable intermediaries are available. Second, the framework requires access to a comprehensive knowledge graph for effective training. cases where such resources are unavailable or incomplete, model performance and generalizability may be impacted.
10

---

Acknowledgment
We are deeply grateful to the area chair and the anonymous reviewers for their valuable comments, which greatly improved the quality of this paper.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidr, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[2] Emily Alsentzer, John R Murphy, Willie Boag, Wei-Hung Weng, Di Jin, Tristan Naumann, and Matthew McDermott. Publicly available clinical bert embeddings. arXiv preprint arXiv: 093333333,, 2019.
[3] Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. Advances neural information processing systems, 32, 2019.
[4] Yasser Benigmim, Subhankar Roy, Slim Essid, Vicky Kalogeiton, and Stéphane Lathuilière. One-shot unsupervised domain adaptation with personalized diffusion models. Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 698-708, 2023.
[5] Olivier Bodenreider. The unified medical language system (umls): integrating biomedical terminology. Nucleic acids research, 12/suppp 2004.
[6] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pages 1597-1607. PMLR, 2020.
[7] Yoav Goldberg. Assessing bert's syntactic abilities. arXiv preprint
2019.
[8] Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. Domain-speciff language model pretraining for biomedical natural language processing. ACM Transactions Computing for Healthcare (HEALTH), 3(1):1-23, 2021.
[9] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv: 106.06855. 2021.
[10] Alistair EW Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom Pollard, Sicheng Hao, Benjamin Moody, Brian Gow, al. Mimic-iv, freely accessible electronic health record dataset. Scientific data, 10(1):1, 2023.
[11] Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4): 1234-1240, 2020.
[12] Hang Li, Yu Kang, Tianqiao Liu, Wenbiao Ding, and Zitao Liu. Ctal: Pre-training cross-modal transformer for mudio-ande and-aandddda language representations. arXiv preprint arXiv:2109.00/8// 2021.
[13] Fangyu Liu, Fhsan Shareghi, Zaiqiao Meng, Marco Basaldella, and Nigel Collier. Self-alignment pretraining for biomedical entity representations. arXiv preprint axXiv:20/0.1178.. 2020.
[14] Ben J Marafino, Miran Park, Jason M Davies, Robert Thombley. Harold S Luft, David C Sing. Dhruv S Kazi, Colette DeJong, W John Boscardin, Mitzi L Dean, et al. Validation of prediction models for critical care outcomes using natural language processing of electronic health record data. JAMA network open, 1(8) 80088-80987888 15097, 2018.
e10977
[15] George Michalopoulos, Yuanxin Wang, Hussam Kaka, Helen Chen, and Alexander Wong. Umlsbert: Clinical domain knowledge augmentation of contextual embeddings using the unified medical language system metathesaurus. arXiv preprint arXiv:2010.1039, 2020.
11

---

[16] Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. Mteb: Massive text embedding benchmark. arXiv preprint arXiv:2210.07316. 2022.
[17] Jianmo Ni, Gustavo Hernandez Abrego, Noah Constant, Ji Ma, Keith B Hall, Daniel Cer, and
Yinfei Yang. Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models. arXiv preprini 1rXiv:2108008890 2021.
[18]
Matthew E Peters, Mark Neumann, Luke Zettlemoyer, and Wen-tau Yih. Dissecting contextual word embeddings: Architecture and representation. arXiv preprint 1808 08949, 2018.
[19] Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. Language models knowledge bases? arXiv preprint 19Xiv:999.00066 2019.
[20] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et Learning transferable visual models from natural language supervision. In International conference machine learning, pages 9748-8763. PMLR, 2021.
[21] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
[22] Nils Reimers and Tryna Gurevych. Making monolingual sentence embeddings multilingual using knowledge distillation. arXiv preprint arXiv:2004.09%5 2020.
[23] Guergana K Savova, James J Masanz, Philip V Ogren, Jiaping Zheng, Sunghwan Sohn, Karin C Kipper-Schuler, and Christopher G Chute. Mayo clinical text analysis and knowledge extraction system (ctakes): architecture, component evaluation and applications. Journal of the American Medical Informatics Association, 17(5):507-513, 2010.
[24] Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 815-823, 2015.
[25] Seyedmostafa Sheikhalishahi, Riccardo Miotto, Joel T Dudley, Alberto Lavelli, Fabio Ri- naldi, Venet Osmani. et al. Natural language processing of clinical notes on chronic diseases: systematic review. JMIR medical informatics, 1(2)xe12239, 2019.
[26] Amber Stubbs, Christopher Kotfila, and Özlem Uzuner. Automated systems for the de- identification of longitudinal clinical narratives: Overview of 2014 i2b2/uthealth shared task track I. Journal of biomedical informatics, 58:$11-S19, 2015.
[27] Gemini Team, Rohan Anil. Sebastian Borgeaud, Yonghui Wu, Jean Caan-Bapistte Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.1188
2023.
[28] Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang. Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R Bowman, Dipanjan Das, et al. What do you learn from context? probing for sentence structure in contextualized word representations. arXiv preprint arXiv: 063 16, 2019.
[29] Fuchiang R Tsui, Lingyun Shi, Victor Ruiz, Neal D Ryan, Candice Biernesser, Satish Iyengar, Colin G Walsh, and David A Brent. Natural language processing and machine learning of electronic health records for prediction of first-time suicide attempts. JAMIA open, 4(1 1ooab001 2021.
[30] Özlem Uzuner, Yuan Luo, and Peter Szolovits. Evaluating the state-of-the-art in automatic de-identification. Journal of the American Medical Informatics Association, 14(5):550-563, 2007.
[31] Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint 2:21::2212.03533, 2022.
12

---

[32] Qian Wang, Fanlin Meng, and Toby P Breckon. Data augmentation with norm-vae for unsuper- vised domain adaptation. arXiv preprint arXiv:2012.00845 2020.
[33] Xiaohan Wang, Linchao Zhu, and Yang. T2vlad: global-local sequence alignment for text- video retrieval. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 5079-5088, 2021.
[34] Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighof. C-pack: Packaged resources to advance general chinese embedding. arXiv preprint urXiv:2309.07597, 2023.
[35] Gokul Yenduri, M Ramalingam, G Chemmalar Selvi, Supriya, Gautam Srivastava, Praveen Kumar Reddy Maddikunta, G Deepti Raj. Rutvij H Jhaveri, B Prabadevi, Weizheng Wang, et al. Gpt (generative pre-trained transformer)-a comprehensive review on enabling technologies, potential applications, emerging challenges, and future directions. IEEE Access, 2024.
[36] Sheng Yu, Tianrun Cai, and Tianxi Cai. Nile: fast natural language processing for electronic health records. arXiv preprint arXiv: 131 1,666, 2013.
[37] Sheng Yu, Yumeng Ma, Jessica Gronsbell, Tianrun Cai, Ashwin N Ananthakrishnan, Vivian Gainer, Susanne E Churchill, Peter Szolovits, Shawn N Murphy, Isaac S Kohane, et al. Enabling phenotypic big data with phenorm. Journal of the American Medical Informatics Association, 25(1):54 60, 2018.
[38] Zheng Yuan, Zhengyun Zhao, Haixia Sun, Jiao Li, Fei Wang, and Sheng Yu. Coder: Knowledge- infused cross-lingual medical term embedding for term normalization. Journal of biomedical informatics, 126: 103980,, 2022.
[39] Sheng Zhang, Xin Zhang, Hui Wang, Lixiang Guo, and Shanshan Liu. Multi-scale attentive interaction networks for chinese medical question answer selection. IEEE Access, 6:74061- 74071, 2018.
[40] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pages 2223-2232, 2017.
13

---

A Appendix
A.1 Handling negation in clinical concepts
Here, we provide additional details on how we handle concepts that are negatively mentioned. For example, consider the sentence, "There is no focal consolidation, pleural effusion, or pneumothorux" The terms "focal consolidation", "pleural effusion", and "pneumothorax" are stated as negative, indicating their absence. This negation significantly alters the sentence's meaning compared to a positive assertion, such as "There focal consolidation, pleural effuusin,, or pneumothorax: Therefore, these negative concepts must be treated differently. To address this, propose creating dictionary negative concepts derived from the original positive concepts, treating them as distinct entities. For example, the positive concept "pneumothorax" would have corresponding negative "concept pneumothorax unobserved". Since the Unified Medical Language System (UMLS) only includes relations for positive concepts, update the negative concepts by first using the multihead graph attention module, as described Section 2.3 to update their corresponding positive concepts. The negative concept embeddings then pass through projection layer. To ensure differentiation, introduce a loss function to maintain a cosine similarity hetween the updated negative concept and its corresponding positive concept below predefined threshold, 6. For each positive-negative concept pair, denoted as (Cp, C,), the corresponding loss is calculated as follows:
log
1 6 c0000((p....)
with < 0.5. This loss function implemented throughout our two-stage training process.
A.2 Information recovered from residuals
In AT we present some examples of crucial clinical concepts that remain undetected due to misspellings but can be subsequently recovered in the first stage of our training process.
Table Al: Examples of crucial clinical concepts that remain undetected due to misspellings but can be subsequently recovered through our two-stage training process.
Original Sentence
Undetected Concept
Recovered Concept
Nasal
fracture Epistaxis NSTEM
Fracture
fracture AaatureepistaaxisNSTe
year old man with traumatic hemothorax s/p pigtail drainageand removal// please assess for interval change s/p pigtail Lemoval>*ss4Peassperrre 3:30PX****
drainage
External drainage
Hypodense appearance of the anterior rightlobe of the liver near the falciform ligament likely perfusional.
right lobe
Right
Pulmonary markings are likely accentuated by lower lung volumes.
accentuated
Somewhat Worse
Globes
lenses
Cornea
intact.
14

---

Table A2: Further results for Table
Models Metric
Mcan Rank
Mcan Reverse Rank 1
Top 10@ Acc
CODER-BGGT
2.509
0.732
0,968
BGE~<CODER
3.281
0.648
0.942
LINE
BGE+CODER
1,437
0.873
0.997
Table A3: Further results for Table
Concept
Sentence
Model
Precision
Recall
Fl
Accuracy
Precision
Recall
F1
Accuracy
CODER CODER
0.690
0.667
0.666
0.670
0.670
0.667
0.656
0.663
0.702
0.687
0.685
0.695
0.783
0.766
0.769
0.767
LINE
BGERCODER
0.741
0.728
0.734
0.722
0.778
0.765
0.771
0.771
A.3
Comparison with related literature
In this subsection, we compare the proposed teacher-teacher framework with Retrieval-Augmented Generation (RAG), method that combines large language models with knowledge databases to improve generation quality. A key distinguishing feature of our framework is that, rather than directly incorporating knowledge database into Teacher enhance generation, we introduce second generative model, Teacher 2, which is inherently endowed with relational knowledge. Teacher 2 then transfers this knowledge to Teacher through alignment learning. This structural difference also supports an additional objective of our framework: enabling Teacher 2 to generate purely concept-based embedding that can serve as proxy for the clinical text, i.e. cross-form generation.
A.4 Further experiment results
We include two additional benchmarks to our validation task on the alignment objective (Section 3.2) and the renal cancer recurrence detection task (Section 3.3.3), both tasks utilize two teacher models. Specifically, we projected the CODER embeddings onto the BGE embedding space (denoted as BRODER-BGER and also performed the reverse projection from BGE to CODER (denoted as "BGE "BGEECCDDER)) using projection matrix. Compared to the two benchmarks, the proposed LINE model achieves better alignment and reduces the performance gap between using sentences and only concepts to detect recurrence information.
Additionally, we report the standard deviation of the Fl scores over five random initializations on two i2b2 datasets in Table A4
A.5
Computational efficiency
To demonstrate the computational efficiency of the proposed framework, using our training data, compare the increase in training time over several training epochs for the following three approaches: (1) directly fine-tuning BGE CODER, (2) fine-tuning with low-rank adaptation (LoRA (9), and
(3) training the LINE module while keeping the pretrained LLMs frozen. The computation time was estimated using the tqdm function
single NVIDIA RTX 8000 GPU with 48GB of VRAM. The rough estimates, shown in Figure [I] indicate that even with LoRA, training large models like BGE takes several days per epoch on single GPU, with increasing computational overhead as training progresses. In contrast, our proposed LINE module requires only 2 hours per epoch on the same hardware.
15

---

Table A4: The mean of the Fl scores for different models over five random initializations on two i2b2 datasets. The results for BioBERT, Clinical BioBERT and UmlsBERT were directly copied from [5 for comparison. Here, the LINE projection is applied to the token-level BGE embeddings. The best result under each metric is highlighted bold font.
Dataset
12b2 2006
i2b2 2014
Test. F!
Val. FI
Test. F1
Val. Fl
BioBERT
93.3 1: 1.300
93.8 0,300
94.6 1 0.200
93.9 1: 0,500
Clinical BioBERT
93.1 1: 1.300
93.4 1 0.200
94.3 0.200
93.0 1: 0.300
UmlsBERT
3.50000500
94.4 + 0.200
=
94.3 0.500
CODER
98.0 + 0.020
96.9 0.030
97.7
97.8 0.010
BGE
97.2 + 0.010
96.1 + 0.040
97.1 0.004
97.1 = 0.030
LINE
98.1 + 0.004
97.2 0.007
98.1
98.1 0.010
4000
BGE
BGE (LoRA)
CODER
3500
CONFR (I aRA)
LINE (RGE=CODER)
(GPTA+CODER)
Computational Time (hrs)
2300
1800
1000
500
9
Epoch
Figure A1: Computational Time Comparison between LINE and Baselines
16

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer:
[Yes]
Justification:
We summarized our main contributions and scope in both the abstract and
introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed limitations in the discussion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification:
The paper does not contain any theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We include the information needed reproduce the main results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer: [No]
Justification: The code is not current available, but we will try to work on that in the near future. For the downstream tasks, use both publicly available data as well as dataset which not publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
7 //ipps.ccppblic//uu.dde sssp:///uuiieeeottotusuiiiaaaaaa ssinisosicyyyyyc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We include all the experiment details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: Standard errors are reported in some tables.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
19

---

It should be clear whether the enror bar is the standard error or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We include information about the computing resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http:: http:p/pw..aaar....co//////////a
Answer: [Yes]
Justification:
We conform with the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss our societal impact in the clinical domain in the introduction and discussion sections.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We have not yet release either our model or the non-public dataset.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have made appropriate citations in places needed.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
21

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper has not released any new assets the moment.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
The only non-public dataset used was previously curated, fully de-identified and annotated for another ongoing project with an existing IRB approval. The dataset used in the paper has already been fully de-identified and annotated. Hence, no human subjects are involved.
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
The only non non-public dataset used was previously curated, fully de-identified and annotated for another ongoing project with an existing IRB approval. The dataset used in the paper has already been fully de-identified and annotated.
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
22

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "jVB6lZzQob", "reviewer": "Reviewer_Yayo", "summary": "The paper proposes a mutual learning framework, called LINE, between two pre-existing LLMs in the healthcare domains. By harmonizing the knowledge of two distinct LLMs into a unified representation space, the model achieves better performance on intrinsic and extrinsic downstream evaluations of clinical tasks.", "questions": "1. Why was NILE selected? Have any other extractors been compared? Does the selection of extractors have a significant impact on results?\n\n2. Line 222, which contrastive loss function was used eventually?", "limitations": "See weakness", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Clear motivation. Overall well written.\n\nThe methodology was reasonably designed to map representations from two distinct LLMs into a unified representation space.\n\nThe method achieves better performance on downstream clinical tasks.", "weaknesses": "1. Only two LLMs (BGE and CODER) were aligned by LINE. It is unclear if LINE will work on combinations of other LLMs. \n\n2. LINE make downstream predictions based on clinical concepts only, rather than the full context. The concepts themselves can be negated, historical and hypothetical in context, but the proposed method does not seem to consider this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments! Below are our responses to your questions, addressed point-by-point:\n\n> It is unclear if LINE will work on combinations of other LLMs.\n\nThank you for raising this important question. To address your concern, we have extended our experiments to include additional combinations of LLMs. Specifically, we tested the alignment of LINE with the combination of GPT4 and CODER. The results, presented in the tables in the rebuttal PDF, indicate that LINE (GPT4+CODER) generally achieves better performance than GPT4 alone. \n\n> The concepts themselves can be negated, historical and hypothetical in context, but the proposed method does not seem to consider this.\n\n Thank you for your insightful comment! We agree that concepts can vary significantly depending on their context, and our approach does take some parts of this into account. Specifically, we have implemented mechanisms to handle negated concepts and will further include a discussion on the possibility of modeling for concepts in historical or hypothetical contexts.\n\n##### Handling Negated Concepts:\n\nIn our approach, we account for negated concepts by creating a separate dictionary of negative concepts derived from the original positive concepts. For instance, the positive concept \"pneumothorax\" is associated with a negated concept represented as \"concept pneumothorax unobserved\".\n\nTo ensure that negated concepts are accurately represented, we use a multihead graph attention module (as detailed in Section 2.2) to update the embeddings of positive concepts, which are then adapted for their corresponding negative concepts through a projection layer. We introduced a loss function specifically designed to maintain a distinct representation for negated concepts. This loss function ensures that the cosine similarity between the updated negative $\\mathbf{c}_p$ concept and its positive counterpart $\\mathbf{c}_p$ remains below a predefined threshold, $\\delta$, calculated as:\n$$\n-\\log\\frac{e^{\\delta - \\text{cos}(\\mathbf{c}_p, \\mathbf{c}_n)}}{1 + e^{\\delta - \\text{cos}(\\mathbf{c}_p, \\mathbf{c}_n)}}, \n$$\nwhere $0 < \\delta \\leq 0.5$. This mechanism is detailed in the Appendix and has been implemented in our two-step training process.\n\n##### Historical and hypothetical contexts:\n\nAdditionally, we recognize the importance of considering whether concepts appear in historical or hypothetical contexts. Using NILE, we can effectively identify and extract contextually relevant keywords such as \"previous\", \"since\", or \"in the near future\". This capability enables us to incorporate context-specific modeling based on the temporal or hypothetical relevance of concepts. However, due to the complexity of real-world contexts, designing appropriate loss functions to account for these nuances requires further exploration and discussion.\n\n> Why was NILE selected? Have any other extractors been compared? Does the selection of extractors have a significant impact on results?\n\nWe chose NILE primarily for its speed and convenience. Compared to cTAKES and MedTagger—both of which are popular extractors—NILE is about 2000 times faster than cTAKES and 400 times faster than MedTagger, while delivering comparable performance. \n\nAdditionally, our framework includes a residual refinement step to recover important clinical concepts that might be missed due to issues like misspellings. This means that while the extractors need to perform reasonably well, they do not need to achieve extremely high accuracy. Consequently, the choice of extractor has a minimal impact on the overall results, as long as it effectively extracts crucial concepts.\n\n> Line 222, which contrastive loss function was used eventually?\n\nThanks for allowing us to clarify! We use the triplet loss.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iEqWNqUn87", "reviewer": "Reviewer_Yopj", "summary": "This paper introduce a teacher-teacher framework for clinical language representation learning. The framework uses a lightweight knowledge alignment module to harmonize the knowledge of both models within a unified space, which including two steps: The first step involves initial training to define residuals and capture complementary information. The second step focuses on refining the alignment by recovering residual information. The framework was validated using the MIMIC-IV database, where the LINE model outperformed baseline models in aligning concept and text representations.", "questions": "1.From Figure 1, if Teacher 2 only serves to provide domain-specific knowledge, why not implement a RAG framework, which is training-free and potentially more reliable?\n\n2. Have you addressed potential hallucination issues? Could one teacher potentially mislead the other during the knowledge exchange process?\n\n3. What are the potential computational and scalability challenges of implementing the teacher-teacher framework in real-world clinical settings? How do you propose to mitigate these challenges?\n\n4. How can regulatory mechanisms be incorporated into the framework for safety?", "limitations": "The paper has limited discussion on the broader implications of implementing the teacher-teacher framework in clinical settings. Consider to add the assessment of how the framework could impact patient care, data security, and trust in AI systems in healthcare.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The main contribution of the work is proposed teacher-teacher framework, and training strategy.\n\n- Originality: The teacher-teacher framework is very interesting as it enables mutual enhancement between two pre-existing LLMs, a unique departure from traditional approaches that typically involve training a new model or continual pre-training of existing models. This innovative method opens new avenues for leveraging existing resources to achieve superior performance.\n\n- Quality: The paper demonstrates high quality through its validation using the MIMIC-IV database, a well-known and respected dataset in the clinical domain, adding significant credibility. Additionally, the LINE model's performance is compared against several strong baseline models, showing clear improvements across various downstream tasks, thus underscoring the robustness and reliability of the proposed framework.\n\n- Clarity: The paper is well-written and clearly structured, making it accessible to both domain experts and those new to the field. The introduction provides a comprehensive background and motivation for the proposed framework, while the methodology section offers detailed descriptions of the teacher models and the LINE module.\n\n- Significance: The practical applications and potential impact on the clinical domain shown the significance of this work. The teacher-teacher idea has substantial implications for more advancing NLP applications in other filed.", "weaknesses": "1. Figure 1 is somewhat confusing. From my understanding, Teacher 1 should be a strong LLM, while Teacher 2 should be an LLM with existing domain-specific knowledge. However, Figure 1 gives the impression that Teacher 2 serves merely as a database, making the framework resemble a RAG framework.\n\n2. Although the paper compares the LINE model against several strong baseline models, it lacks a detailed comparison with the latest strong general LLMs, such as GPT-4, which should be considered a strong baseline. Consider adding a small comparative analysis or stating the advantages of the framework over simply using GPT-4.\n\n3. The paper underscore the practical value of the framework, but it does not sufficiently address potential practical implementation challenges, such as computational requirements and scalability when applied in real-world clinical settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments! In the following, we will address them point-by-point.\n\n> Figure 1 is somewhat confusing. From my understanding, Teacher 1 should be a strong LLM, while Teacher 2 should be an LLM with existing domain-specific knowledge. However, Figure 1 gives the impression that Teacher 2 serves merely as a database. \n\nThank you so much for pointing this out! We have modified Figure 1 into Figure 1(b) in the rebuttal PDF for better clarity. In summary, both Teacher 1 and Teacher 2 can generate embeddings when feeding in new data which are then aligned by LINE. When the data are paired, e.g. they come from the same hospital visit of a patient, their LINE-aligned embeddings will be close to each other.\n\n> From Figure 1, if Teacher 2 only serves to provide domain-specific knowledge, why not implement a RAG framework, which is training-free and potentially more reliable?\n\nThe main role of Teacher 2 is to learn an embedding using a list of concepts to summarize the clinical notes, effectively matching the embedding generated directly from the raw notes. Since raw clinical notes are frequently inaccessible, summarizing and indexing notes using a list of concepts enables de-identification and facilitates sharing among researchers. Additionally, by allowing two teachers to align knowledge with each other via LINE, we hope to also improve the quality of the embeddings .\n\nWhile the RAG framework also leverages pre-existing knowledge networks to enhance learning, it is not a generative model and cannot be used to generate new embeddings. Therefore, it does not fulfill the same role as our proposed method, which not only aligns domain-specific knowledge but also generates new, robust embeddings.\n\nWe will include RAG in our discussion of related work in the paper.\n\n> Although the paper compares the LINE model against several strong baseline models, it lacks a detailed comparison with the latest strong general LLMs, such as GPT4, which should be considered a strong baseline. Consider adding a small comparative analysis or stating the advantages of the framework over simply using GPT4.\n\nThank you for your advice! During the rebuttal phase, we incorporated GPT4 as a strong Teacher 1 in our LINE framework, resulting in LINE (GPT4+CODER). We then compared the performance of this new model with the GPT4 baseline. The results, shown in the tables in the rebuttal PDF, indicate that LINE (GPT4+CODER) generally achieves better performance than GPT4 alone.\n\n> Have you addressed potential hallucination issues? Could one teacher potentially mislead the other during the knowledge exchange process?\n\nThank you for the insightful question! To mitigate potential hallucination issues, we integrate factual knowledge from external reputable sources (e.g., UMLS) into our framework. Specifically, in our framework, external factual knowledge is incorporated into the training of Teacher 2 via the multihead graph attention module and the relational contrastive loss. This knowledge is then further propagated to Teacher 1 through the alignment loss.\n\nIn this regard, Teacher 2 acts as a factually rigorous component that regularizes potentially misleading information from Teacher 1. We conducted an experiment to assess the quality of the LINE-aligned concept embeddings. The results, shown in Table 3 of the paper and the second table in rebuttal PDF, indicate that the generated concept embeddings faithfully preserve factual relationships and even slightly improve upon capturing these relationships compared to the baseline.\n\n> What are the potential computational and scalability challenges of implementing the teacher-teacher framework in real-world clinical settings? How do you propose to mitigate these challenges?\n\nThank you for the question! One of the notable advantages of our proposed framework is its computational and memory efficiency, making it feasible to deploy with limited computational resources. To illustrate this, please refer to Figure1(a) in the rebuttal PDF, where we compare the training time of our model to both direct fine-tuning of LLMs and low-rank approximated fine-tuning of LLMs. The comparison shows that our model can be trained within two day on a RTX8000 48GB card.\n\n> How can regulatory mechanisms be incorporated into the framework for safety? Consider to add the assessment of how the framework could impact patient care, data security, and trust in AI systems in healthcare.\n\nThank you for this important question! Below, we assess how the framework impacts patient care, data security, and trust in AI systems, and discuss the incorporation of regulatory mechanisms for safety. This discussion will be included in the paper.\n\n- **Data Security**: We utilize the publicly accessible MIMIC-IV dataset, which aligns with real-world clinical notes without privacy concerns. During inference, real medical notes aren't required; instead, a concept extractor tool like NILE can be used to extract key concepts for Teacher 2, preventing the use of sensitive patient information. Regulatory mechanisms can monitor and update the medical concepts in the knowledge graph, ensuring they reflect the most up-to-date factual knowledge. For instance, newly defined concepts like COVID-19 can be added, outdated concepts depreciated, and misleading concepts corrected.\n- **Patient Care**: The framework's computational efficiency allows for the effective use of state-of-the-art LLMs, improving downstream tasks such as disease diagnosis and lab results analysis, thereby enhancing patient care.\n- **Trust in AI Systems**: LINE can be retrained on local servers with various combinations of up-to-date LLMs, necessitating continual quality control. For example, our clinical concept similarity task (Section 3.3.1) can initially assess the fidelity of learned concept embeddings. During deployment, regular user feedback should be collected to further improve the system.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ttuoph39t1", "reviewer": "Reviewer_TNnd", "summary": "The authors look to address the question representational alignment between language models trained on different textual domains to improve performance of potentially both models on their out-of-domain text. The authors propose to specifically investigate this in the context of EHR text, and choose as their models for this CODER and BGE. They propose a contrastive loss, and additionally propose to train an alignment module/project layer rather than end-to-end training of the teacher models.", "questions": "Would it be possible to further test the LINE model on other, more varied, benchmarks to see how well those newly aligned representations perform? \n\nIt could also be exciting to explore this with generative models.\n\nWere alternative frameworks considered for the concept alignment? Why not align directly in embedding space without the grounding concepts? This would be an interesting ablation to perform to assess the significance of the extracted concepts on the underlying learned representation. Conversely, could you just fine-tune the generalist model on the extracted concepts as a means of medically aligning it? How well does that perform? \n\nWhy not also compare the BGE-->CODER projection (inverse direction of the BGE-->CODER projection)?\n\nIf it isn't technically feasible to do in an end-to-end fashion, perhaps this could be approximated by tuning LoRA on the base models?", "limitations": "No discussion of limitations. Without additional experiments at the minimum a stated limitation should be the highly restricted domain of application (purely encoder models on medical topics).", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The concept is solid and well implemented and motivated. I wonder if it would be possible to further generalize it beyond medical text - which it is restricted too due to the reliance on alignment with extracted medical concepts by NILE. The discussion mentions this possibility, but it would be exciting to see it in action.\n\nThe clinical NLP benchmarks are particularly appropriate for the task.", "weaknesses": "Some of the benchmark tasks are older, and the comparisons could be more robust. Some ablations are missing.\n\nThe project's scope is incredibly narrow: encoder models on extractive medical tasks. While the authors claim that the technique is broadly generalizable, it would be nice to see proof-of-concept. \n\nThe work seems to me to fit more into the realm of domain adaptation rather than learning by alignment. We aren't learning novel models here via alignment (like CLIP), but rather, pushing the learned representations of two different models into a common space. I'd strongly consider citing and discussing DA literature for this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed comments. In the following, we will address your questions point-by-point.\n\n> The project's scope is incredibly narrow. \n\nThank you for raising this important question. While our motivating example comes from the medical domain, our LINE framework is broadly applicable to various scenarios. The core problem we address involves pairs of unlabeled data points $(\\mathbf{x}_1, \\mathbf{x}_2)$ from different domains, where $\\mathbf{x}_2$ consists of items with known relationships derived from factual knowledge graphs. Our goal is to learn an embedding $\\mathbf{z}_2$ for $\\mathbf{x}_2$ that closely matches the embedding $\\mathbf{z}_1$ for $\\mathbf{x}_1$. This framework can be applied to multiple critical applications, such as:\n\n1. **Text Summarization**: In healthcare, clinical notes ($\\mathbf{x}_1$) often need to be inferred from concepts ($\\mathbf{x}_2$) due to regulatory constraints. Using a list of concepts to summarize notes enables de-identification and facilitates sharing among researchers. These concept lists can replace full clinical notes for downstream analysis.\n2. **Text Summarization for Image and Audio**: The LINE framework can align audio or image data ($\\mathbf{x}_1$) with textual information ($\\mathbf{x}_2$). Medical images are harder to anonymize than text due to metadata and visual information. LINE can use a pre-trained vision transformer to embed images and a language model to embed conceptual entities from clinical text, providing embedding surrogates for images like CT scans. These text summaries can index the images and be used for downstream analyses. Additionally, LINE can identify \"residual\" information in the images not captured by paired data and project it into the text domain for insights.\n\nThis discussion will be included into the paper. While including proof-of-concept on other settings would be beneficial, we are unable to do so due to time constraints.\n\n> I'd strongly consider citing and discussing DA literature for this paper.\n\nThank you for your suggestion. We agree that our work is related to domain adaptation (DA) and will cite relevant literature, though our approach differs in key ways:\n\n- **Label Requirements**: Unlike DA, our method does not require labels during training. DA typically needs task-related labels for the source domain and may use \"pseudo\" labels for the target domain.\n- **Information Retention**: Our goal is to align latent embeddings to preserve both overlapping and complementary information, using a residual alignment training step to refine the alignment with additional concepts. DA focuses only on retaining overlapping information.\n- **Embedding Richness**: We aim to create semantic-rich embeddings by leveraging a graph-attention module, preserving relational information and mitigating rank degeneracy. DA often produces domain-agnostic embeddings, whose information richness depends on the specific task.\n\nWe will further include this discussion in the paper.\n\n> Would it be possible to further test the LINE model on other, more varied, benchmarks?\n\nThank you for your suggestion! In response, we are currently running an additional benchmark to evaluate the performance of the LINE models. We anticipate being able to present these results in the early stages of the discussion phase.\n\n>  It could also be exciting to explore this with generative models.\n\nThe LINE framework generates in the embedding space, not the original data space. Given an unlabeled data pair $(\\mathbf{x}_1, \\mathbf{x}_2)$ from spaces $\\mathcal{X}_1$ and $\\mathcal{X}_2$, respectively, LINE aligns their latent embeddings $\\mathbf{z}_1 \\approx \\mathbf{z}_2$. For a new observation $\\mathbf{x}_2^{\\text{new}}$, we can map it to the latent space to generate $\\mathbf{z}_1^{\\text{new}} \\approx \\mathbf{z}_2^{\\text{new}} = \\text{LINE}(f({\\text{teacher 2}}(\\mathbf{x}_2))$ for the missing $\\mathbf{x}_1^{\\text{new}}$. To extend this, we could add a decoder to map embeddings back to the clinical note space using a simple reconstructive loss. We will discuss this potential extension in future work.\n\n> Why not align directly in embedding space without the grounding concepts? \n\nThank you for your thoughtful question. Grounding concepts in our framework both serve to integrate external factual knowledge to enhance model performance and safety, and also provide an independent data source for generating clinical note summaries, making them integral to our approach.\n\n> Why not also compare the BGE-->CODER projection? \n\nThank you for the suggestion! Following your advice, we have included the results for the BGE-->CODER projection across our tasks in the rebuttal PDF. Overall, the projection between BGE and CODER yields worse results compared to the proposed LINE model.\n\n> Conversely, could you just fine-tune the generalist model on the extracted concepts as a means of medically aligning it? \n\nDuring the rebuttal phase, we attempted to fine-tune generalist models directly and with LoRA, but found it infeasible due to extremely long training times and high computational resource requirements.\n\nThis highlights the computational efficiency of our proposed method. As shown in Figure 1(a) of the rebuttal PDF, even with LoRA, training the generalist BGE model takes at least 11 days per epoch, with rapidly increasing computational overhead. In contrast, our proposed model requires only about 2 hours per epoch using the same resources.\n\n> Without additional experiments at the minimum a stated limitation should be the highly restricted domain of application\n\nWe acknowledge this concern and will include a discussion of the limitation that our current framework has only been tested on medical topics. We recognize the importance of evaluating our framework in other domains and will address this as a key area for future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dcaSJleU0M", "reviewer": "Reviewer_Hfn8", "summary": "This paper presents an interesting topic on LLM but the importance of this problem is not convincing and the methods here is not novel.", "questions": "None.", "limitations": "See details in weakness.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The teacher-teacher concept is novel to some extent.", "weaknesses": "1. The problem's importance is not significant.\n2. There lacks the inclusion of SOTA models like llama, gpt, etc.\n3. The results improvement is limited as shown in Tab. 4,5.\n4. The Fig1 lacks details of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comment.We will address your comments point-by-point.\n\n>  The problem's importance is not significant.\n\nThank you for raising this important question. While our motivating example originated from the medical domain, our LINE framework is broadly applicable to a wide range of scenarios. The core problem we address involves pairs of unlabeled data points $(\\mathbf{x}_1, \\mathbf{x}_2)$ from different domains, where $\\mathbf{x}_2$ consists of items with known relationships derived from pre-existing factual knowledge graphs. Our objective is to learn an embedding $\\mathbf{z}_2$ for $\\mathbf{x}_2$ that closely matches the embedding $\\mathbf{z}_1$ for $\\mathbf{x}_1$. This general framework can be applied to various critical applications, including the alignment of text from structured EHR codes paired with clinical notes, images/audio paired with text. Below, we detail a few specific applications:\n\n1. **Text Summarization**: In healthcare systems, clinical notes ($\\mathbf{x}_1$) often need to be inferred from corresponding concepts ($\\mathbf{x}_2$) due to regulatory constraints. Since raw clinical notes are frequently inaccessible, using a list of concepts to summarize and index notes enables de-identification and facilitates sharing among researchers. The concept list can be utilized instead of full clinical notes for downstream analysis.\n2. **Text Summarization for Image and Audio**: In a cross-modality setting, the LINE framework can align audio/image data ($\\mathbf{x}_1$) with corresponding textual information ($\\mathbf{x}_2$). For instance, medical images are more challenging to anonymize than text due to embedded metadata and visual information that might indirectly reveal patient identity. In this context, LINE can employ a pre-trained vision transformer to embed images and a pre-trained language model to embed conceptual entities from clinical text. Consequently, LINE can provide embedding surrogates for images, such as CT scans, by leveraging only the concepts extracted from corresponding clinical notes. The text summaries can be used to both index the images and for direct downstream analyses. Additionally, our LINE algorithm has the potential to identify \"residual\" information in the images that is not captured by the paired data, which can then be projected into the text domain to provide meaningful insights.\n\nWe will incorporate this into the motivation and discussion sections of the paper.\n\n> There lacks the inclusion of SOTA models like llama, gpt, etc.\n\nThanks for your suggestion! During the rebuttal phase, we are able to use GPT4 as the strong teacher 1 to come up with LINE (GPT4+CODER). And then we further compared the new LINE results with the GPT4 baseline. The results are shown in the tables in the rebuttal pdf. We also provide a summary of the key results below for your easy reference. The following tables are additional results to be added to Tables 2, 3 and 5 of the paper. The better results are highlighted in bold.\n\n| Model / Metrics   | LINE (CODER+GPT4) | GPT4  |\n| ----------------- | ----------------- | ----- |\n| Mean Rank         | **1.477**         | 1.778 |\n| Mean Reverse Rank | **0.872**         | 0.820 |\n| Top10@Acc         | **0.995**         | 0.988 |\n\n| Models / Relation | Parent    | Sibling   | May Treat/Prevent | Classifies | DDX   | Method of  | Causative  |\n| ----------------- | --------- | --------- | ----------------- | ---------- | ----- | ---------- | ---------- |\n| GPT4              | 0.974     | **0.940** | 0.825             | **0.991**  | 0.939 | 0.934      | 0.935      |\n| LINE (GPT4+CODER) | **0.977** | 0.932     | **0.931**↑        | 0.988      | 0.938 | **0.965**↑ | **0.947**↑ |\n\n|                   | Concept   |           |           |           | Sentence  |           |           |           |\n| ----------------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- |\n|                   | Precision | Recall    | F1        | Accuracy  | Precision | Recall    | F1        | Accuracy  |\n| GPT4              | 0.718     | 0.687     | 0.694     | 0.711     | 0.805     | **0.787** | **0.791** | **0.781** |\n| LINE (GPT4+CODER) | **0.731** | **0.715** | **0.714** | **0.722** | **0.806** | 0.786     | 0.789     | **0.781** |\n\nFrom the tables above, it can be observed that in most of the setting, LINE model performs better than directly using GPT4.\n\n> The results improvement is limited as shown in Tab. 4,5.\n\nAs discussed in our response to your first question, our evaluation of the proposed framework is not solely focused on outperforming the benchmark model. It also emphasizes the potential of using a list of concepts to summarize notes, which enables de-identification and facilitates sharing among researchers. In Table 2, we evaluate whether the summarization from the list of concepts can be effectively aligned with the clinical text through a rank-based retrieval task. In Table 5, we assess whether the performance using only concepts can be comparable to the performance using raw text. The results from both the tables in the paper and the additional results presented above consistently demonstrate that the LINE model is more effective in summarizing text.\n\n> Figure 1 lacks details of the proposed method.\n\nThank you for your comment. We have revised Figure 1 to include more details, which can be seen in Figure 1(b) of the rebuttal PDF. Specifically, Teacher 2 utilizes the trainable LINE module to first summarize the set $\\mathbf{x}_2$, guided by pre-existing knowledge (i.e., the graph attention module and relational contrastive loss). Then, Teacher 2 generates an embedding $\\mathbf{z}_2$ for $\\mathbf{\\tilde{x}}_2$ that closely resembles the embedding $\\mathbf{z}_1$ for $\\mathbf{x}_1$ through the alignment loss. Simultaneously, Teacher 1, a stronger LLM, uses a fully connected layer to learn the relational knowledge provided by Teacher 2 via the alignment loss.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7hQTCoTndx", "reviewer": "Reviewer_j7j8", "summary": "The paper introduces a novel teacher-teacher framework named LIghtweight kNowledge alignmEnt (LINE), which facilitates knowledge exchange between two pre-existing large language models (LLMs) to enhance clinical language representation. By leveraging complementary knowledge from general-purpose and domain-specific models, LINE aims to harmonize their knowledge within a unified representation space. The framework is validated through downstream tasks showing that the LINE model outperforms individual pre-existing models in understanding and processing clinical language. This approach allows for more efficient sharing of clinical pretrianed models.", "questions": "1. See weakness, under such situation is it possible to apply your method?", "limitations": "No.\nThe authors discuss the potential improvement instead of the limitation of the current work. Bring more information and try other situations cannot be counted as an adequate discussion of limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Clarity and Structure**: The paper is well-written and structured, offering a clear motivation for the study. This makes it accessible and engaging for readers, facilitating a deeper understanding of the proposed framework.\n\n2. **Novelty and Utility**: The proposed teacher-teacher framework, LIghtweight kNowledge alignmEnt (LINE), is innovative, providing a pragmatic approach to integrating the strengths of different pre-trained models. This methodology is particularly notable for its potential to enhance clinical language representations without the need for developing new models from scratch.\n\n3. **Usability and Efficiency**: The framework is user-friendly and does not require retraining of the original models, which significantly reduces computational overhead and simplifies its adoption in real-world applications.\n\n4. **Empirical Validation**: The experimental results demonstrate stable and significant improvements over existing methods, substantiating the efficacy and value of the proposed framework in practical settings.", "weaknesses": "**Data Requirements and Availability**: A notable limitation of the proposed LINE framework is its dependency on well-aligned and specific types of data sources, which may not be readily available or commonly found in practical settings. For example, integrating data from disparate modalities like CT and MRI requires the availability of cases that include both types of data, which may not always be feasible. This requirement could limit the framework's applicability across different clinical or real-world scenarios where such aligned data sets are scarce.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Data Requirements and Availability**: A notable limitation of the proposed LINE framework is its dependency on well-aligned and specific types of data sources, which may not be readily available or commonly found in practical settings. For example, integrating data from disparate modalities like CT and MRI requires the availability of cases that include both types of data, which may not always be feasible. This requirement could limit the framework's applicability across different clinical or real-world scenarios where such aligned data sets are scarce. See weakness, under such situation is it possible to apply your method?\n\nThis is an extremely insightful question, and we appreciate you bringing this up! Yes, our method can handle such situations by introducing an intermediary to bridge the gap between different data sources.\n\nFor instance, if we want to infer $\\mathbf{x}_1$ and $\\mathbf{x}_3$ about each other but lack paired data $(\\mathbf{x}_1, \\mathbf{x}_3)$, the LINE framework can use an intermediary $\\mathbf{x}_2$. This forms pairs $(\\mathbf{x}_1, \\mathbf{x}_2)$ and $(\\mathbf{x}_2, \\mathbf{x}_3)$, enabling the use of three teachers to jointly learn the alignment between all data pairs.\n\nIn a concrete example involving CT and MRI scans, it is often rare for patients to have both scans simultaneously ($\\mathbf{x}_1$ for CT and $\\mathbf{x}_3$ for MRI). However, each type of scan typically comes with corresponding clinical notes ($\\mathbf{x}_2$). These clinical notes can serve as intermediaries. By breaking down the clinical notes into concepts corresponding to specific scan regions, we can leverage Teacher 2, a pre-trained LLM, to incorporate factual relationships about these concepts. Teachers 1 and 3, pre-trained large vision transformers, can then embed the CT and MRI scans, respectively. The three teachers can train jointly, allowing the framework to effectively handle the alignment of disparate data modalities.\n\n>  The authors discuss the potential improvement instead of the limitation of the current work. Bring more information and try other situations cannot be counted as an adequate discussion of limitations.\n\nThank you for your constructive comment. We recognize two main limitations of our framework:\n\nFirst, our approach requires data to come in pairs. While we can introduce intermediaries to create artificial pairs, this remains a limitation. The reliance on paired data may restrict the framework's applicability in scenarios where such pairs are not readily available.\n\nSecond, effective training with a list of concepts necessitates a pre-existing factual knowledge graph. Access to such a knowledge graph is not always guaranteed, which can hinder the framework's performance and generalizability.\n\nWe appreciate your feedback and will incorporate a more detailed discussion of these limitations in the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thank you all for your comments and questions! Based on your suggestions, we have made the following major changes during the rebuttal phase:

### Additional Experiment

1. **New Teacher Model**: We have adopted the OpenAI text embedding model "text-embedding-v3-small" as Teacher 1. Since it was released alongside GPT4 and demonstrates strong performance, we refer to it as "GPT4" for brevity. The subsequently trained model is referred to as "LINE (GPT4+CODER)". As suggested by the reviewers, we have included "CODER$\to$BGE" and "BGE$\to$CODER" as comparison benchmarks. "CODER$\to$BGE" projects the concept embeddings from CODER into the BGE embedding space using a projection matrix, while "BGE$\to$CODER" performs the inverse operation.

   The results are presented in the tables in the rebuttal PDF. Note that, since token-level embeddings from GPT4 are unavailable, we cannot perform i2b2 tasks using GPT4 or LINE (GPT4+CODER).

2. **Computational Efficiency**: To assess the computational efficiency of the proposed framework, we have benchmarked the estimated training time of our model against both direct fine-tuning of LLMs and low-rank approximated fine-tuning of LLMs [1] on a single RTX8000 48GB card. These results are shown in Figure 1(a) of the rebuttal PDF.

### Modified Figure 1 of the Paper

Following the reviewers' advice, we have modified Figure 1 in the paper for better clarity. Please see Figure 1(b) in the rebuttal PDF.

### Proposed Changes to the Paper

1. **Expanded Scope of Applicability**: We will add concrete examples to illustrate the broad applicability of the LINE framework.
2. **Additional Related Works**: We will include related literature on domain adaptation and retrieval-augmented generation to discuss how the proposed framework differs from these existing lines of research.
3. **Discussions on Safety**: We will add discussions on how the proposed framework addresses the issue of hallucination, manages concepts in different contexts, and assess its impact on patient care, data security, and trust in AI systems in healthcare.
4. **Discussion on Limitations**: We will add a discussion of the limitations of the proposed framework.

We appreciate your feedback and will make these changes to enhance the clarity and comprehensiveness of our paper.

[1] Hu, Edward J., et al. "Lora: Low-rank adaptation of large language models." *arXiv preprint arXiv:2106.09685* (2021).

Meta-review: This paper introduces LINE framework, a novel teacher-teacher approach for clinical language representation learning. The paper presents an innovative method for harmonizing knowledge from two pre-existing large language models (LLMs) in the healthcare domain, resulting in improved performance on downstream clinical tasks.

The reviewers have highlighted several strengths of this work. The clear motivation and well-structured presentation make the paper accessible to both experts and newcomers in the field. The methodology is sound, demonstrating a unique approach to leveraging existing LLMs rather than training new models or continually pre-training existing ones. The use of the MIMIC-IV database for validation adds credibility to the results, and the performance improvements over strong baseline models underscore the robustness of the proposed framework.

During the rebuttal phase, the authors have addressed key concerns raised by reviewers. They expanded their experiments to include GPT4 as a teacher model, demonstrating that LINE (GPT4+CODER) generally outperforms GPT4 alone on various tasks. This addition significantly strengthens the paper by comparing against a state-of-the-art general LLM. The authors have also clarified their approach to handling negated concepts and provided insights into their choice of the NILE extractor.

The paper's weaknesses, such as the initial confusion in Figure 1 and the need for more detailed comparisons with recent LLMs, have been addressed in the rebuttal. The authors have committed to improving the figure and including additional experiments and discussions in the camera-ready version.

Given the innovative nature of the work, its potential impact on clinical NLP applications, and the authors' thorough responses to reviewer concerns, I recommend accepting this paper. The LINE framework presents a significant contribution to the field by introducing a novel approach to aligning and enhancing existing LLMs for clinical applications. The authors have demonstrated a clear understanding of the limitations and potential challenges of their approach and have shown commitment to addressing these in the final version.

In conclusion, this paper offers a technically sound and impactful contribution to the field of clinical NLP, with potential applications beyond healthcare. The authors' responsiveness to reviewer feedback and willingness to enhance the paper further strengthen its case for acceptance.
Decision: Accept (poster)