A teacher-teacher framework for clinical language representation learning

Authors: feiqing  huang
harvard t h chan
shenghan  zhang
sara morini sweet
tianxi  cai
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 19:47:50

Abstract: In recent years, there has been a proliferation of ready-to-use large language models (LLMs) designed for various applications, both general-purpose and domain-specific. Instead of advocating for the development of a new model or continuous pretraining of an existing one, this paper introduces a pragmatic teacher-teacher framework to facilitate mutual learning between two pre-existing models.
By leveraging two teacher models possessing complementary knowledge, we introduce a LIghtweight kNowledge alignmEnt (LINE) module aimed at harmonizing their knowledge within a unified representation space. This framework is particularly valuable in clinical settings, where stringent regulations and privacy considerations dictate the handling of detailed clinical notes. Our trained LINE module excels in capturing critical information from clinical notes, leveraging highly de-identified data. Validation and downstream tasks further demonstrate the effectiveness of the proposed framework.

Main paper: 
 introduction: 
 Clinical data frequently appears in various forms, with each format capturing overlapping but complementary aspects of patient information. For instance, narrative details in a clinical note may be processed into a structured list of clinical concepts for research purposes. Similarly, a CT scan report provides descriptive findings that are directly linked to the visual data in the corresponding scan. Ideally, both forms of data would be jointly accessible; however, practical challenges, such as privacy restrictions, often limit us to only one form. This limitation raises an important question:Could we use the accessible form of data to approximate or represent the unobserved one?To explore this question, we can leverage the capabilities of large pretrained models. Recent years have seen the emergence of large language models (LLMs), which have demonstrated impressive performance across a range of tasks, including prediction, generation, and representation learning [1, 28, 35] . In the clinical domain, many pretrained LLMs-such as CODER [39] , UmlsBERT [16] , Clinical BioBERT [3] , SapBERT [14] , PubmedBERT [9] , and BioBERT [12] -are based on BERT architectures with approximately 300 million parameters. These models are typically initialized with BERT weights and then undergo further pretraining on biomedical texts, a process often referred to as continual pretraining. While this approach allows the model to integrate both general and domainspecific knowledge, it is also time-consuming and resource-intensive. The recent proliferation of off-the-shelf clinical LLMs now prompts a pragmatic idea:Can the pretrained models be made to directly exchange knowledge with one another?Inspired by the two questions posed above, we propose a teacher-teacher framework for clinical representation learning. The core idea is as follows: given paired training data in two distinct forms and two pretrained models with complementary knowledge bases, the framework processes each form of data separately through one of the models. It then processes the resulting representations through a newly proposed module and further aligns them within a unified representation space. This alignment entails a mutual teaching process, where both models act as teachers, exchanging their knowledge. The complementary knowledge arises from two sources: first, the complementary information inherent to each form of data, and second, the distinct knowledge embedded in each pretrained model due to differences in their original training corpora. We summarize our key innovations as follows:• LINE module for teacher-teacher learning: We propose a lightweight knowledge alignment (LINE) module to facilitate effective teacher-teacher learning. This module takes representations from two pretrained teacher models and projects them into a unified representation space, guided by an alignment loss and a relational loss.• Efficient training with residual information recovery: We adopt a two-stage, few-epoch training process focused solely on the LINE module, keeping the pretrained model weights frozen. In the first stage, we train the LINE module for a few epochs to achieve initial alignment and to capture residuals (complementary information across two forms of data). In the second stage, these residuals are used to refine the training data for further few-epoch training, offering insights into the complementary information between data forms.• Generating cross-form representation: Once trained, the LINE module enables cross-form representation for downstream tasks. When only one data form is available, the corresponding teacher model and LINE projection can generate a proxy for the missing data, enabling its use in the downstream tasks.We evaluate our teacher-teacher framework on two fronts: (1) whether knowledge alignment can improve the performance of one or both teacher models across different tasks, and (2) whether the available data form can be used to generate better proxies for the other, missing data form.While using one form data to represent another is a relevant question across various fields, we demonstrate the utility of our proposed method through a real-world clinical use case. This example is particularly relevant to clinical data privacy concerns, highlighting its practical importance. Specirically, we focus on two data forms: clinical notes and their corresponding structured lists of clinical concepts. Clinical notes play a crucial role in numerous applications, such as disease diagnosis, disease progression prediction, patient classification, and risk assessment [26, 30, 15] .However, working with real clinical notes is subject to strict regulations due to privacy concerns. Sharing clinical notes between institutions necessitates approval from an institutional review board (IRB), adding layers of regulatory complexity and substantially extending project timelines.These challenges are a major barrier to scaling up clinical datasets for pre-training. As an alternative, efficient software [e.g. 37, 24] can directly extract structured clinical concepts from clinical notes. This approach offers two advantages: First, because clinical concepts are drawn from a predefined dictionary -specifically, the Unified Medical Language System (UMLS) [6] -it reduces the risk of including sensitive information, making data sharing across research groups more feasible. Second, known relationships among many clinical concepts can be further incorporated to form a structured graph, capturing both the textual content and interrelationships among concepts.In this use case, we treat the clinical note as the missing data form and reframe our question accordingly: Can we leverage the list of clinical concepts to effectively summarize and provide a proxy for the content in the clinical note?Different modes of learning Existing knowledge within pretrained LLMs varies based on their learning modes and domains of their pretrained data. The first mode, learning by generalization, involves training models on extensive datasets to generalize knowledge through example-based learning [1, 28] . These models encode knowledge directly into their embeddings, which capture syntactic and semantic relationships in texts. Numerous studies have demonstrated the efficacy of LLMs in encoding syntactic heuristics and retaining factual knowledge to varying extents [8, 20, 29, 19, 22] . Recent advancements, exemplified by models such as Sentence-T5 [18] , E5 [32] , BGE [35] , and GPT-4 [2] , have focused on generating high-quality textual representation for downstream tasks. However, in specialized clinical domains, vocabulary diverges significantly from general corpora, requiring that these general-purpose models undergo additional training on biomedical data to ensure relevance and accuracy. Models like BioBERT [12] and PubmedBERT [9] , pretrained on biomedical texts, have demonstrated superior performance compared to BERT in biomedical-specific tasks. Clinical BioBERT [3] further augment their training data with MIMIC datasets to incorporate medical knowledge derived from real-world medical practices. Nevertheless, scaling up training data in the clinical domain poses challenges due to privacy regulations, as previously noted.The second mode, learning by integrating pre-existing knowledge, focuses on encoding established factual associations between concepts into the model's representation space. In the clinical domain, knowledge repositories such as UMLS [6] offer comprehensive biomedical ontologies and relatedness information between concepts, which can be treated as existing knowledge graph and harnessed during the pre-training phase. For example, SapBERT [14] leverages synonymous relationships extracted from UMLS during self-supervised pretraining, strengthening the model's understanding of semantic similarity. UmlsBERT [16] incorporates semantic type information from UMLS, enabling a deeper semantic understanding of clinical texts. Furthermore, CODER [39] encodes more granular relationships, such as "may prevent or treat" and "may cause", into their knowledge networks, enhancing its ability to capture medical associations. By integrating such pre-existing domain-specific knowledge graph during training, these models show improved ability to navigate the intricacies of specialized domains, leading to better performance in domain-specific tasks [14, 16, 39] .Distinct from these two modes, the proposed teacher-teacher framework adopts the third mode, learning by alignment. This mode of learning involves creating a joint representation space between two models using paired data, such as an image with its textual description, audio/video with its transcript, or multilingual translated texts. This approach focuses on teaching models to distinguish between paired and unpaired data, making it especially effective for multimodal applications. For example, the seminal CLIP framework [21] uses different encoders for text and images to embed each mode of information separately. It then employs a contrastive loss to align these multimodal embeddings by end-to-end training on both encoders, ensuring that related text and image pairs are closely aligned in the representation space. Similarly, Li et al. [13] utilize a text encoder and a text-referred audio encoder with inter-modality and intra-modality training, enhancing the model's ability to integrate complementary features from different views. Wang et al. [34] propose an efficient global-local alignment framework for co-training language with videos, and its improved performance further highlights the advantages of learning the homogeneity as well as heterogeneity across language and visual modalities. For multilingual translation tasks, alignment learning has also demonstrated remarkable efficacy. For example, Reimers and Gurevych [23] introduce a teacher-student network model in which a fixed teacher network, well-versed in a target language, guides a student network, initially proficient in a different language, to emulate the teacher's knowledge. This method not only enhances translation accuracy but also creates a joint representation space that offers potentially interesting insights into cross-linguistic differences.Other related works In a broader context, if we view the accessible form of data as the source domain and the inaccessible one as the target domain, our approach is weakly related to generative domain adaptation; however, unlike generative domain adaptation, our method requires paired data during training. Traditional methods in this field often utilize generative models to synthesize samples across domains, helping the model generalize effectively in new contexts. Some example frameworks include generative adversarial networks [41] , variational autoencoders [33] , diffusion models [5] , or generative pretrained transformers [36] . However, our framework diverges from these in that we specifically deploy two pretrained teacher models, each with distinct expertise and capabilities. This design allows the models not only to align but also to exchange complementary knowledge during the learning process, potentially enhancing the performance of one or both teacher models. 2 The teacher-teacher framework 
training data and alignment objective: 
 This section provides a detailed illustration of our LINE framework for a specific clinical use case, aligning clinical concepts with corresponding clinical texts. The algorithm can be extended to support a broad range of other alignment tasks, as discussed in Section 4. Specifically, our paired training data take the form (CLINICAL TEXT, LIST OF CLINICAL CONCEPTS), where each list of clinical concepts is derived from the clinical texts. We further augment the clinical concepts with known relationships from UMLS to form a structured graph. The primary objective is to create a "summarized" embedding for the list of concepts and align it with the projected text embedding, as illustrated in Figure 1 .A key design principle in our framework is selecting two teacher models with complementary knowledge bases. To this end, we choose (1) a general-purpose LLM pretrained on large-scale, general-domain text, and (2) a domain-specific LLM with established knowledge of biomedical concept relationships. In this setup, the general-purpose model (Teacher 1) learns factual relationships between clinical concepts from the domain-specific model (Teacher 2), while Teacher 2 enhances its grasp of the contextual connections linking these concepts. In the paper, we explore two combinations of teacher models: BGE+CODER and GPT-4+CODER.Teacher 1: General purpose LLM BGE is a general-purpose embedding model trained on large, diverse language datasets. Initially trained on 100 million text pairs from open web corpora, BGE further underwent multitask learning on various retrieval and reranking tasks, including the CMedQA-v2 medical question and answer dataset [40] . It ranks highly on the Massive Text Embedding Benchmark (MTEB) dataset [17] , which covers tasks like classification, reranking, retrieval, STS, and paired classification tasks. GPT-4 is a large-scale, multimodal model considered as one of the state-of-the-art tools for language modeling and generation. Although its model architecture and trained weights are not publicly available, GPT-4 can generate embeddings for textual data, accessible through the OpenAI API. For our experiments, we used the "text-embedding-3-small" model, which produces embeddings with a dimension of 1536. In our two combinations of teacher models, the general purpose LLMs (BGE and GPT-4) with their extensive and diverse training, generally outperform CODER. This imbalance might suggest that they cannot learn from CODER. However, our experiments suggest that not only can CODER benefit from the knowledge of the general-purpose LLM, but the performance of BGE or GPT-4 also improves on certain tasks with alignment training. We hypothesize that the alignment process regularizes the embedding space of the general-purpose LLM by incorporating CODER's association graph, resulting in a more diversified and robust representation space.Our framework eliminates the need for any further training of the original teacher models. Instead, we introduce a trainable LIghtweight kNowledge alignmEnt module (LINE), which explicitly models projection operations to align the embeddings from the general-purpose LLM and the domain-specific LLM into a unified representation space.First, we derive clinical text embeddings and concept embeddings from the general-purpose LLM and domain-specific LLM using pretrained checkpoints available through Huggingface or the OpenAI API, denoted as t (0) ∈ R dt and c (0) ∈ R dc , respectively. These initial embeddings retain the original knowledge encoded within each teacher model. We then incorporate trainable projection layers specific to each LLM to refine and align these embeddings.Projection layer for general purpose LLM: Fully-connected layer The projection layer for the output embeddings t (0) from BGE or GPT-4 is implemented as a simple fully-connected linear layer:t := F(t (0) ) = FC(t (0) ).This approach aligns with prior work [4, 7, 21] , where linear projections are used without non-linear activation functions.Projection layer for domain-specific LLM: Multihead graph attention This layer is designed to reinforce the domain-specific knowledge embedded in CODER from the UMLS ontology. We focus on modeling seven broad categories of UMLS relations: parent-child hierarchy, synonyms, related and possibly synonymous, broad relationship, narrow relationship, non-synonymous/narrow/broad relationship, and quantifiable relationships. Each relationship type is represented by a distinct adjacency matrix, and a multi-head graph attention (MHGA) layer assigns each type to a separate attention head. This design enables the layer to explicitly account for the unique structure of each relation. The outputs from all attention heads are aggregated and projected into the joint representation space. Formally, for any concept with an initial CODER embedding c (0) , we have:EQUATIONwhere W c ∈ R dt×dc is the projection matrix that maps the initial CODER embeddings to the same dimension as the BGE or GPT-4 embeddings. Additional details on the handling of negatively mentioned concepts are provided in the Appendix.To enable the two teacher models to enhance each other's knowledge in a time-efficient and resourceefficient way, we adopt a two-stage, few-epoch training strategy exclusively for the LINE module. This approach effectively aligns the embeddings from the general-purpose and domain-specific LLMs without any additional training of the original pretrained models. It also allows for the use of proprietary models, such as GPT-4, where access to the underlying pretrained model is restricted.The first stage involves an initial round of few-epoch training aimed at defining residuals, i.e. the differences between the learned representations of the two data forms. This stage is critical for capturing complementary information between the two models.Specifically, we aim to align the embedding for clinical texts with its list of associated concept embeddings in the joint representation space. This alignment is quantified using a monotonic score function s, which measures the degree of alignment between the elements of each data pair. Formally, let p i = ({c i,j } j∈[mi] , t i ) denote a positive pair, where t i is the clinical text embedding and {c i,j } j∈[mi] are the corresponding concept embeddings with a total of m i concepts, for all i ∈ [N ]. The negative pair is defined asn i,i ′ = ({c i,j } j∈[mi] , t i ′ ) for i ′ ̸ = i ∈ [N ]. The score function s is defined as s(•) = s({c j } j∈[m] , t) = -∥t -mean({c j } j∈[m] )∥ 2 .Our goal is to ensure s(p i ) ≥ s(n i,i ′ ), indicating better alignment of positive pairs over negative pairs.To account for complementary information in the positive pairs, we introduce a data-dependent alignment weight ρ i ∈ (0, 1] to allow varying degrees of information overlap. For example, in extracting clinical concepts from clinical texts, misspellings can result in failure to detect key concepts; see Table A1 in the Appendix for concrete examples. A smaller ρ i value indicates more tolerance for misalignment, reflecting cases with less overlap between the text and concept embeddings. The alignment loss is defined as:i i ′ ̸ =i ρ i f (s(p i ), s(n i,i ′ )),where f is a contrastive loss function, and we used the triplet loss [25] for our experiments. In our specific context, the weight ρ i is computed as the percentage of words in the text captured in the concept list, i.e. ρ i = number of words accounted for in text i total word count in text i .In addition, to preserve the integrity of existing concept relations, we incorporate a relational contrastive loss during training. Here, positive pairs represent known relationships, while negative pairs indicate absent or unknown relationships. This additional loss term ensures that information from the known association graph is retained throughout the alignment process.Refinement through residual recovery Once the first training stage stabilizes, we move to the second stage, focusing on recovering residual information to further refine the alignment. The residual e i for each data pair is computed as:e i = t i -mean({c i,j } j∈[mi] ), i ∈ [N ].To further consolidate information from residuals, we make use of a projected concept dictionary C G , consisting of all clinical concepts from UMLS, projected using the trained G defined in (1). For each residual e i , we select concepts from C G that have a cosine similarity of 0.9 or higher with e i . The most similar concept from the selected subset, denoted as c i,mi+1 , is then added to the concept embeddings set, refining the training pair to ({c i,j } j∈[mi+1] , t i ) for further few-epoch training.3 Training and validationQuestion: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We have discussed our limitations in the discussion section. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
data preprocessing and training: 
 Our model, referred to as LINE, is trained with two configurations of teacher models: BGE+CODER and GPT-4+CODER. For our training dataset, we utilized 332K discharge notes and 2.2M radiology reports from 146K patients available in the MIMIC-IV database [11] . Given the structured nature of both types of clinical notes, we began by segmenting the notes into their respective sections. Radiology reports were divided into seven sections: examination, indication, technique, comparison, finding, procedures, and impression. Similarly, discharge notes were segmented into fourteen sections: chief complaint, major surgical or invasive procedure, history of present illness, past medical history, social history, family history, physical exam, pertinent results, brief hospital course, medications on admission, discharge medications, discharge disposition, discharge diagnosis, discharge condition, discharge instructions, and follow-up instructions.To extract clinical concepts from these notes, we employed the Narrative Information Linear Extraction (NILE) software [37] . NILE outputs a list of clinical concepts present in each segmented text, indicating whether each concept is mentioned positively or negatively. Figure 1 provides an illustrative example. Next, we combined the original text with its extracted clinical concepts to create paired training data. This preprocessing resulted in a dataset of 7.2M text-concept pairs. During this process, we located the extracted concepts within the text and counted the number of words accounted for by the list of concepts. The alignment weight for each text was then calculated as ρ = number of words accounted for total word count . To construct the relational adjacency matrices for the CODER projection Table 1 : For each positive concept-text pair, we randomly selected 100 other texts to replace the original text, forming negative pairs. We then calculated the cosine similarity between the mean of the clinical concept list and the text, with embeddings generated by various models. By ranking these cosine similarities from highest to lowest, we identified the rank of each positive pair among its negative pairs and computed the mean rank, mean reverse rank, and Top-10 accuracy (Top10@Acc). Improvements over the corresponding teacher models are indicated by "↓" for mean rank and "↑" for mean reverse rank and Top10@Acc. We adopted the Adam optimizer with a learning rate of 10 -3 and a batch size of 128. The training process was divided into two stages as detailed in Section 2.4: We first trained the model for three epochs initially, during which we identified residual concepts not adequately captured. Then, using the model checkpoint from the first phase, we recovered the residual concepts and refined the training pairs. The model was then trained for an additional two epochs. The alignment loss and the relational loss both converged rapidly. All experiments were conducted using an NVIDIA RTX 8000 GPU with 48GB of VRAM. The entire training process required less than 10 hours on a single GPU.This section validates the effectiveness of our trained LINE model in aligning the representation spaces of the two teachers. To achieve this, we utilized a holdout subset of patients whose clinical notes were excluded from the training set. This subset comprised 100K radiology reports from 92K patients. We applied the same data preprocessing steps, resulting in 275K text-concept pairs. Our evaluation method involved the following steps. For each positive pair, we randomly sampled 100 negative pairs by substituting the clinical text with a different one. We then computed the cosine similarity between the mean embedding of the concept list and the text embedding for all pairs, in line with our alignment objective. Finally, we ranked these pairs based on their cosine similarity scores, from highest to lowest.To assess performance, we calculated the mean rank, mean reverse rank, and Top-10 accuracy for all positive pairs. These metrics are presented in Table 1 . We compared our LINE model against several models: GPT-4, BGE, CODER, SapBERT, Clinical BioBERT, PubMedBERT, and BioBERT. As shown in Table 1 , LINE with BGE+CODER achieves the highest alignment between concept and text representations, outperforming the other models. Both LINE models show improved alignment compared to their respective teacher models, validating the consistency of our training objectives.In this section, the performance of our trained LINE models on various downstream tasks is evaluated based on two main aspects: (1) to determine whether the teacher models can benefit from mutual learning, and (2) to assess whether the LINE module can generate more effective proxies for clinical text using the list of clinical concepts. Specifically, we use the LINE-projected CODER embeddings for tasks involving clinical concepts and the LINE-projected BGE or GPT-4 embeddings for tasks involving clinical text. Given that two model configurations are used, we refer to each configuration as "LINE" in our tables, with "Teacher 1+Teacher 2" specified for clarity. Three types of downstream We evaluated the quality of the LINE-projected embeddings based on their ability to detect known relationships among 222K pairs of 159K concepts. These pairs were curated from several major UMLS relation classes, including "may treat or may prevent", "classifies", "differential diagnosis", "method of", and "causative". Detailed counts of positive pairs for each class are provided in the last row of Table 2 . To ensure a fair comparison with the other models, we removed the edges corresponding to these evaluation pairs from the relational adjacency matrices used in the LINE model. We then assessed the model's ability to capture different types of relationships by calculating cosine similarities between the embedding vectors of related pairs and comparing these to randomly selected pairs with similar semantic characteristics. For example, when evaluating the "may treat or may prevent" relationship, we used related disease-drug pairs as positive examples and compared them against negative pairs also composed of disease-drug pairs.We calculated the Area Under the Curve (AUC) of the cosine similarities to assess the models' ability to distinguish known pairs from random ones. As shown in Table 2 , GPT-4 and BGE outperform CODER and all other baseline clinical LLMs, serving as strong benchmarks. Notably, the LINEprojected embeddings achieve AUCs that not only exceed those of the original CODER model but also outperform the BGE benchmark in six of the seven relation classes and the GPT-4 benchmark in four of the seven. In the remaining classes, LINE's AUCs are comparable to their respective BGE and GPT-4 benchmarks, with a deviation less than 0.008. We evaluated our model on two standard biomedical named entity recognition (NER) benchmark tasks: the i2b2 2006 de-identification challenge [31] and the i2b2 2014 de-identification challenge [27] . We followed the train/validation/test splits specified in the original challenges, as detailed in Table 1 of [3] . The datasets from 2006 and 2014 contain 317 and 43 label classes, respectively. The NER tasks involve tokenizing sentences and then classifying each token within the sentence. For this task, we used token-level embeddings from each model and the LINE projection was applied to token-level BGE embeddings. GPT-4 was not included due to the limited access to its token-level embedding. Our fine-tuning process followed the setting in [16] . Specifically, a single linear layer was added on top of each model and trained for 20 epochs. We adopted Adam optimizer and the learning rates for CODER, BGE and LINE were set to 2 × 10 -5 , 2 × 10 -5 and 5 × 5 × 10 -4 , respectively. The mean of the F1 scores calculated over five random initializations are reported in Table 3 , and it can be observed that LINE achieves the best overall performance.The dataset used for this downstream task comes from Mass General Brigham and was previously curated, fully de-identified, and annotated for another ongoing project. Here, we briefly describe the curation and annotation process. First, a phenotyping algorithm [38] was applied to identify a pool of renal cell carcinoma (RCC) patients. From this pool, 300 RCC patients were randomly selected, and a clinician reviewed all notes to extract one diagnostic-relevant sentence per patient. Each sentence was annotated into one of three classes: uninformative of RCC recurrence, informative of past or current recurrence, and informative of no recurrence. We then used NILE [37] to extract clinical concepts from each sentence, excluding those where NILE failed to extract any concept. This resulted in a final set of 288 labeled sentences, each paired with a list of corresponding clinical concepts. The dataset comprises 136 uninformative sentences, 64 informative of recurrence, and 88 informative of no recurrence.This classification task is designed to test how effectively proxy sentences, generated from concept lists, can detect recurrence information. Specifically, we aim to understand how well the proxy sen-tences perform in classifying uninformative, recurrence-informative, and non-recurrence-informative cases, simulating scenarios where only concept lists, not full sentences, are available. To this end, we additionally include classification results using raw sentences as a benchmark for expected optimal performance. We used LINE-projected CODER to embed the concept lists and LINE-projected GPT-4 or BGE to embed the raw sentences. A single linear layer was added on top of both models, and we fine-tuned it for 2000 epochs, with early stopping when the change in loss was less than 10 -5 . We used Adam optimizer with a learning rate of 10 -3 . The same setup was applied to all models for fair comparison. On average, the fine-tuning process took approximately one or two minutes per model. The average performance metrics from five-fold cross-validation are reported in Table 4 . As shown, when raw sentences are available, GPT-4 and BGE achieve significantly better classification results than the other benchmarks. LINE-projected embeddings for GPT-4 and BGE manage to maintain a comparable level of classification performance. However, using only concept lists for GPT-4 and BGE results in a substantial drop in classification performance. Notably, adding the LINE projection reduces this performance gap, as shown in Table 5 , supporting the effectiveness of the LINE module in better approximating sentence embeddings using concept lists.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA]Answer: [No] Justification: The code is not current available, but we will try to work on that in the near future. For the downstream tasks, we use both publicly available data as well as a dataset which is not publicly available. Guidelines:• The answer NA means that paper does not include experiments requiring code. Justification: We conform with the code of ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
conclusion and discussion: 
 This paper introduces a teacher-teacher paradigm in which two pretrained LLMs align different forms of data to enable knowledge exchange and cross-form representation through a two-stage, few-epoch training of the LINE module, guided by a well-defined alignment objective. Our downstream analysis demonstrates that (1) the proposed LINE module effectively generates cross-form representations, and (2) alignment learning enhances performance for both models, even when one teacher model is more advanced than the other.Although our primary focus is on a specific clinical use case, this teacher-teacher framework has broad potential applications. One promising application is enhancing the indexing and searchability of non-textual data. For example, medical images (e.g., CT or MRI scans) are often accompanied by clinical notes from which clinical concepts can be extracted. By segmenting images into regions and associating them with clinical concepts, we create a concept-based index of image content. The teacher-teacher framework aligns each image segment with its concept list, allowing the concept list to serve as a proxy representation for image content. This enables concept-or keyword-based searches for specific image segments via embeddings, making non-textual data more searchable. The framework also supports alignment between modalities without direct pairing. For example, CT and MRI scans are seldom captured together for the same patient, but clinical notes serve as a potential intermediary. In this scenario, clinical concepts extracted from notes are embedded by a pretrained teacher model. Separate teacher models then embed CT and MRI images. The alignment objective splits into two tasks: aligning clinical concepts with CT images and aligning clinical concepts with MRI images. This dual alignment establishes an indirect connection between CT and MRI images, creating a unified representation without direct pairing. These examples illustrate the framework's broad applicability in aligning disparate modalities, supporting cross-form searches and enhancing the utility of diverse data sources.Our clinical use case further demonstrates the practical benefits of this framework in improving patient care and data security. By reducing the need for direct access to clinical notes, the framework minimizes privacy concerns and strengthens data protection. Additionally, it facilitates the integration of up-to-date LLMs, improving performance in tasks essential to patient care and potentially leading to better outcomes. The framework's use of external, factual knowledge also reduces misinformation risks, fostering trust in AI-assisted clinical tools.Despite its strengths, the framework has two notable limitations. First, its reliance on paired data may limit its applicability in situations where aligned datasets are scarce. Although intermediaries can help bridge certain gaps, the framework's effectiveness remains constrained in cases where no suitable intermediaries are available. Second, the framework requires access to a comprehensive knowledge graph for effective training. In cases where such resources are unavailable or incomplete, model performance and generalizability may be impacted. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We discuss our societal impact in the clinical domain in the introduction and discussion sections.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: We have not yet release either our model or the non-public dataset.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We have made appropriate citations in places needed.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.• If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA]Justification: The paper has not released any new assets at the moment.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: The only non-public dataset used was previously curated, fully de-identified and annotated for another ongoing project with an existing IRB approval. The dataset used in the paper has already been fully de-identified and annotated. Hence, no human subjects are involved.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: The only non-public dataset used was previously curated, fully de-identified and annotated for another ongoing project with an existing IRB approval. The dataset used in the paper has already been fully de-identified and annotated.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Here, we provide additional details on how we handle concepts that are negatively mentioned. For example, consider the sentence, "There is no focal consolidation, pleural effusion, or pneumothorax."The terms "focal consolidation", "pleural effusion", and "pneumothorax" are stated as negative, indicating their absence. This negation significantly alters the sentence's meaning compared to a positive assertion, such as "There is focal consolidation, pleural effusion, or pneumothorax." Therefore, these negative concepts must be treated differently. To address this, we propose creating a dictionary of negative concepts derived from the original positive concepts, treating them as distinct entities. For example, the positive concept "pneumothorax" would have a corresponding negative "concept pneumothorax unobserved". Since the Unified Medical Language System (UMLS) only includes relations for positive concepts, we update the negative concepts by first using the multihead graph attention module, as described in Section 2.3, to update their corresponding positive concepts. The negative concept embeddings then pass through a projection layer. To ensure differentiation, we introduce a loss function to maintain a cosine similarity between the updated negative concept and its corresponding positive concept below a predefined threshold, δ. For each positive-negative concept pair, denoted as (c p , c n ), the corresponding loss is calculated as follows:-log e δ-cos(cp,cn) 1 + e δ-cos(cp,cn) , with 0 < δ ≤ 0.5. This loss function is implemented throughout our two-stage training process.In Table A1 , we present some examples of crucial clinical concepts that remain undetected due to misspellings but can be subsequently recovered in the first stage of our training process. In this subsection, we compare the proposed teacher-teacher framework with Retrieval-Augmented Generation (RAG), a method that combines large language models with knowledge databases to improve generation quality. A key distinguishing feature of our framework is that, rather than directly incorporating a knowledge database into Teacher 1 to enhance generation, we introduce a second generative model, Teacher 2, which is inherently endowed with relational knowledge. Teacher 2 then transfers this knowledge to Teacher 1 through alignment learning. This structural difference also supports an additional objective of our framework: enabling Teacher 2 to generate a purely concept-based embedding that can serve as a proxy for the clinical text, i.e. cross-form generation.We include two additional benchmarks to our validation task on the alignment objective (Section 3.2) and the renal cancer recurrence detection task (Section 3.3.3), as both tasks utilize two teacher models. Specifically, we projected the CODER embeddings onto the BGE embedding space (denoted as "CODER→BGE") and also performed the reverse projection from BGE to CODER (denoted as "BGE→CODER") using a projection matrix. Compared to the two benchmarks, the proposed LINE model achieves better alignment and reduces the performance gap between using sentences and only concepts to detect recurrence information.Additionally, we report the standard deviation of the F1 scores over five random initializations on two i2b2 datasets in Table A4 ATo demonstrate the computational efficiency of the proposed framework, using our training data, we compare the increase in training time over several training epochs for the following three approaches:(1) directly fine-tuning BGE or CODER, (2) fine-tuning with low-rank adaptation (LoRA [10] ), andtraining the LINE module while keeping the pretrained LLMs frozen. The computation time was estimated using the tqdm function on a single NVIDIA RTX 8000 GPU with 48GB of VRAM. The rough estimates, shown in Figure A1 , indicate that even with LoRA, training large models like BGE takes several days per epoch on a single GPU, with increasing computational overhead as training progresses. In contrast, our proposed LINE module requires only 2 hours per epoch on the same hardware.Table A4 : The mean of the F1 scores for different models over five random initializations on two i2b2 datasets. The results for BioBERT, Clinical BioBERT and UmlsBERT were directly copied from [16] for comparison. Here, the LINE projection is applied to the token-level BGE embeddings. The best result under each metric is highlighted in bold font. Justification: We summarized our main contributions and scope in both the abstract and introduction.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: This paper introduces LINE framework, a novel teacher-teacher approach for clinical language representation learning. The paper presents an innovative method for harmonizing knowledge from two pre-existing large language models (LLMs) in the healthcare domain, resulting in improved performance on downstream clinical tasks.

The reviewers have highlighted several strengths of this work. The clear motivation and well-structured presentation make the paper accessible to both experts and newcomers in the field. The methodology is sound, demonstrating a unique approach to leveraging existing LLMs rather than training new models or continually pre-training existing ones. The use of the MIMIC-IV database for validation adds credibility to the results, and the performance improvements over strong baseline models underscore the robustness of the proposed framework.

During the rebuttal phase, the authors have addressed key concerns raised by reviewers. They expanded their experiments to include GPT4 as a teacher model, demonstrating that LINE (GPT4+CODER) generally outperforms GPT4 alone on various tasks. This addition significantly strengthens the paper by comparing against a state-of-the-art general LLM. The authors have also clarified their approach to handling negated concepts and provided insights into their choice of the NILE extractor.

The paper's weaknesses, such as the initial confusion in Figure 1 and the need for more detailed comparisons with recent LLMs, have been addressed in the rebuttal. The authors have committed to improving the figure and including additional experiments and discussions in the camera-ready version.

Given the innovative nature of the work, its potential impact on clinical NLP applications, and the authors' thorough responses to reviewer concerns, I recommend accepting this paper. The LINE framework presents a significant contribution to the field by introducing a novel approach to aligning and enhancing existing LLMs for clinical applications. The authors have demonstrated a clear understanding of the limitations and potential challenges of their approach and have shown commitment to addressing these in the final version.

In conclusion, this paper offers a technically sound and impactful contribution to the field of clinical NLP, with potential applications beyond healthcare. The authors' responsiveness to reviewer feedback and willingness to enhance the paper further strengthen its case for acceptance.

Reviews: review: paper_summary: paper_summary: The paper proposes a mutual learning framework, called LINE, between two pre-existing LLMs in the healthcare domains. By harmonizing the knowledge of two distinct LLMs into a unified representation space, the model achieves better performance on intrinsic and extrinsic downstream evaluations of clinical tasks.
strength_weakness: strength_weakness: 1. Only two LLMs (BGE and CODER) were aligned by LINE. It is unclear if LINE will work on combinations of other LLMs. 

2. LINE make downstream predictions based on clinical concepts only, rather than the full context. The concepts themselves can be negated, historical and hypothetical in context, but the proposed method does not seem to consider this.
questions: questions: 1. Why was NILE selected? Have any other extractors been compared? Does the selection of extractors have a significant impact on results?

2. Line 222, which contrastive loss function was used eventually?
limitations: limitations: See weakness
paper_summary: paper_summary: This paper introduce a teacher-teacher framework for clinical language representation learning. The framework uses a lightweight knowledge alignment module to harmonize the knowledge of both models within a unified space, which including two steps: The first step involves initial training to define residuals and capture complementary information. The second step focuses on refining the alignment by recovering residual information. The framework was validated using the MIMIC-IV database, where the LINE model outperformed baseline models in aligning concept and text representations.
strength_weakness: strength_weakness: 1. Figure 1 is somewhat confusing. From my understanding, Teacher 1 should be a strong LLM, while Teacher 2 should be an LLM with existing domain-specific knowledge. However, Figure 1 gives the impression that Teacher 2 serves merely as a database, making the framework resemble a RAG framework.

2. Although the paper compares the LINE model against several strong baseline models, it lacks a detailed comparison with the latest strong general LLMs, such as GPT-4, which should be considered a strong baseline. Consider adding a small comparative analysis or stating the advantages of the framework over simply using GPT-4.

3. The paper underscore the practical value of the framework, but it does not sufficiently address potential practical implementation challenges, such as computational requirements and scalability when applied in real-world clinical settings.
questions: questions: 1.From Figure 1, if Teacher 2 only serves to provide domain-specific knowledge, why not implement a RAG framework, which is training-free and potentially more reliable?

2. Have you addressed potential hallucination issues? Could one teacher potentially mislead the other during the knowledge exchange process?

3. What are the potential computational and scalability challenges of implementing the teacher-teacher framework in real-world clinical settings? How do you propose to mitigate these challenges?

4. How can regulatory mechanisms be incorporated into the framework for safety?
limitations: limitations: The paper has limited discussion on the broader implications of implementing the teacher-teacher framework in clinical settings. Consider to add the assessment of how the framework could impact patient care, data security, and trust in AI systems in healthcare.
paper_summary: paper_summary: The authors look to address the question representational alignment between language models trained on different textual domains to improve performance of potentially both models on their out-of-domain text. The authors propose to specifically investigate this in the context of EHR text, and choose as their models for this CODER and BGE. They propose a contrastive loss, and additionally propose to train an alignment module/project layer rather than end-to-end training of the teacher models.
strength_weakness: strength_weakness: Some of the benchmark tasks are older, and the comparisons could be more robust. Some ablations are missing.

The project's scope is incredibly narrow: encoder models on extractive medical tasks. While the authors claim that the technique is broadly generalizable, it would be nice to see proof-of-concept. 

The work seems to me to fit more into the realm of domain adaptation rather than learning by alignment. We aren't learning novel models here via alignment (like CLIP), but rather, pushing the learned representations of two different models into a common space. I'd strongly consider citing and discussing DA literature for this paper.
questions: questions: Would it be possible to further test the LINE model on other, more varied, benchmarks to see how well those newly aligned representations perform? 

It could also be exciting to explore this with generative models.

Were alternative frameworks considered for the concept alignment? Why not align directly in embedding space without the grounding concepts? This would be an interesting ablation to perform to assess the significance of the extracted concepts on the underlying learned representation. Conversely, could you just fine-tune the generalist model on the extracted concepts as a means of medically aligning it? How well does that perform? 

Why not also compare the BGE-->CODER projection (inverse direction of the BGE-->CODER projection)?

If it isn't technically feasible to do in an end-to-end fashion, perhaps this could be approximated by tuning LoRA on the base models?
limitations: limitations: No discussion of limitations. Without additional experiments at the minimum a stated limitation should be the highly restricted domain of application (purely encoder models on medical topics).
paper_summary: paper_summary: This paper presents an interesting topic on LLM but the importance of this problem is not convincing and the methods here is not novel.
strength_weakness: strength_weakness: 1. The problem's importance is not significant.
2. There lacks the inclusion of SOTA models like llama, gpt, etc.
3. The results improvement is limited as shown in Tab. 4,5.
4. The Fig1 lacks details of the proposed method.
questions: questions: None.
limitations: limitations: See details in weakness.
paper_summary: paper_summary: The paper introduces a novel teacher-teacher framework named LIghtweight kNowledge alignmEnt (LINE), which facilitates knowledge exchange between two pre-existing large language models (LLMs) to enhance clinical language representation. By leveraging complementary knowledge from general-purpose and domain-specific models, LINE aims to harmonize their knowledge within a unified representation space. The framework is validated through downstream tasks showing that the LINE model outperforms individual pre-existing models in understanding and processing clinical language. This approach allows for more efficient sharing of clinical pretrianed models.
strength_weakness: strength_weakness: **Data Requirements and Availability**: A notable limitation of the proposed LINE framework is its dependency on well-aligned and specific types of data sources, which may not be readily available or commonly found in practical settings. For example, integrating data from disparate modalities like CT and MRI requires the availability of cases that include both types of data, which may not always be feasible. This requirement could limit the framework's applicability across different clinical or real-world scenarios where such aligned data sets are scarce.
questions: questions: 1. See weakness, under such situation is it possible to apply your method?
limitations: limitations: No.
The authors discuss the potential improvement instead of the limitation of the current work. Bring more information and try other situations cannot be counted as an adequate discussion of limitations.

Comments: comment: Thank you for the response! The new figure 1 makes sense to me. Thank you for new results. I decide to increase soundness from 2 to 3.
Thank you for the rebuttal! I will keep my recommendation unchanged.
Dear Reviewer Yayo, 
I am a NeurIPS 2024 Area Chair of the paper that you reviewed.

This is a reminder that authors left rebuttals for your review. 
We need your follow up answers on that. Please leave comment for any un-answered questions you had, or how you think about the author's rebuttal. 
The author-reviewer discussion is closed on Aug 13 11:59pm AoE.

Best regards,
AC
Dear Reviewer Yopj, 
I am a NeurIPS 2024 Area Chair of the paper that you reviewed.

This is a reminder that authors left rebuttals for your review. 
We need your follow up answers on that. Please leave comment for any un-answered questions you had, or how you think about the author's rebuttal. 
The author-reviewer discussion is closed on Aug 13 11:59pm AoE.

Best regards,
AC
Dear Reviewer TNnd, 
I am a NeurIPS 2024 Area Chair of the paper that you reviewed.

This is a reminder that authors already left rebuttals for your review. 

We need your follow up answers on that. Please leave comment for any un-answered questions you had, or how you think about the author's rebuttal. 
The author-reviewer discussion is closed on Aug 13 11:59pm AoE.

Best regards, 
AC
Dear Reviewer Hfn8, 
I am a NeurIPS 2024 Area Chair of the paper that you reviewed.

This is a reminder that authors already left rebuttals for your review. 

We need your follow up answers on that. Please leave comment for any un-answered questions you had, or how you think about the author's rebuttal. 
The author-reviewer discussion is closed on Aug 13 11:59pm AoE.

Best regards, 
AC
Thanks for your comment. I will keep my score.
## Additional Experiment

Thank you once again for your suggestion and thank you for your patience! We have incorporated an additional benchmark dataset focused on the mental health domain to further evaluate the proposed LINE framework against other methods. Specifically, we retrieved suicide-related publications from PubMed, using the search term "suici" to capture both "suicide" and "suicidal" references. This resulted in a collection of 8,000 publications with "suicid" in the title. We excluded a small number of large files (>215M) that required more than 16GB of RAM, accounting for approximately 1% of the total dataset.

From these publications, we extracted titles and keywords, and further refined the dataset by removing any publications that did not contain keywords. We then conducted a rank-based retrieval task, using the keywords to retrieve the corresponding article. For each positive (title, keyword) pair, we generated 100 negative pairs by randomly substituting the title with one from a different article.

Next, we computed the cosine similarity between the mean embedding of the keyword list and the title embedding for all pairs. These pairs were then ranked based on their cosine similarity scores, from highest to lowest. To assess performance, we calculated the mean rank, mean reverse rank, and Top-10 accuracy for all positive pairs. The results, presented in the table below, show that both LINE models show significant improvements over their respective teacher models.

| Metric                | PubmedBERT | BioBERT | SapBERT | CODER | BGE  | CODER$\to$BGE | BGE$\to$CODER | GPT      | LINE(BGE+CODER) | LINE(GPT+CODER) |
| --------------------- | ---------- | ------- | ------- | ----- | ---- | ---------- | ---------- | -------- | --------------- | --------------- |
| **Mean Rank**         | 29.37      | 35.84   | 8.52    | 22.89 | 3.03 | 17.32      | 41.55      | 2.95     | 2.07            | **1.94**        |
| **Mean Reverse Rank** | 0.17       | 0.14    | 0.52    | 0.27  | 0.84 | 0.33       | 0.08       | **0.85** | 0.83            | **0.85**        |
| **Top10@Acc**         | 0.34       | 0.27    | 0.79    | 0.46  | 0.94 | 0.56       | 0.18       | 0.95     | 0.97            |        **0.98**         |