Large Language Models as Instructors: A Study on Multilingual Clinical Entity Extraction
Year: 2023

Abstract:
In clinical and other specialized domains, data are scarce due to their confidential nature. This lack of data is a major problem when finetuning language models. Nevertheless, very large language models (LLMs) are promising for the medical domain but cannot be used directly in healthcare facilities due to data confidentiality issues. We explore an approach of annotating training data with LLMs to train smaller models more adapted to our problem. We show that this method yields promising results for information extraction tasks.

text: The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks, pages 178–190 July 13, 2023 ©2023 Association for Computational Linguistics

heading: 1 Introduction
text: Clinical notes contain the interactions between the patient and healthcare staff. Professionals record their impressions, observations, and various medical procedures performed. Despite the computerization of clinical documents, notes should remain fairly expressive and in a free format to save time for healthcare personnel and allow for the description of unusual situations (Rosenbloom et al., 2011). Moreover, a large amount of crucial information is exclusively contained in clinical notes. According to a study by Escudié et al. (2017), approximately 80% of patient phenotypes (a set of observable biological and physical characteristics that can characterize a disease) are present only in free text. These documents are difficult to use without advanced methods such as deep learning in NLP. The use of such methods requires the collection and annotation of a significant amount of medical data. However, Fries et al. (2022) proposes the term "dataset debt," highlighting that learning data in the biomedical field is poorly accessible, poorly documented, and opaque as to its reusability in a commercial or a hospital context. According to the article, only 13% of the 167 analyzed datasets are accessible and downloadable, 22% use a standard structured format, and 40% are in the public domain. In recent years, large language models
(LLMs) have proved their ability to perform a wide range of tasks with high accuracy in a zero-shot or a few-shots contexts. This trend holds great potential for clinical NLP, as preliminary results show promise for information extraction tasks (Agrawal et al., 2022). However, the clinical domain presents unique challenges due to the confidential and linguistically specific nature of its data, which can make collection and annotation time-consuming and expensive. Using LLMs for efficient information extraction without training data could be attractive, but it raises confidentiality concerns. The model deployment should be controllable, and the model’s predictions should evolve to fit a specific and changing annotation guideline. Most multilingual LLMs are not freely available (Scao et al., 2022; Ouyang et al., 2022; Thoppilan et al., 2022), to the best of our knowledge, only BLOOM is opensource and deployable in a custom infrastructure. The computing resources to use these models remains challenging for healthcare establishments.
One approach to address these issues is to distil LLMs into a smaller model via weak supervision. Weak supervision has recently gained community attention because it alleviates the annotation task. This technique refers to annotating datasets using rule-based, heuristic, dictionary extraction or more advanced methods and then training the smaller model on this dataset. In the same way, knowledge distillation aims to transfer knowledge from a master model to a student model. It has often been used to compress large-scale models to improve memory footprint and the inference speed (Li et al., 2021). Moreover, student models trained through knowledge distillation can be more easily monitored and versioned. Hosting them increases the healthcare centre’s sovereignty, and they become more compliant with existing privacy policies, as input data or predictions don’t leave the building.
178

heading: 2 Motivation and Contributions
text: We study the use of LLMs in the knowledge distillation technique via weak supervision in the Multilingual Clinical domain, especially in clinical entity extraction. We extend the Agrawal et al. (2022) study in the sense that we propose an in-depth study of the use of InstructGPT-3 to annotate a training dataset. Our work1 mainly aims to compare the annotation quality using weak supervision tasks on a smaller model (Figure 1). Finally, we propose to combine annotations provided by InstructGPT-3 and the dictionary extraction method.
This takes form in these contributions:
• We show that InstructGPT-3 distillation (Figure 1 middle) is a competitive technique compared to classic weak-supervision techniques in a multilingual clinical domain;
• We propose a weak supervision approach (Figure 1 bottom) that combines annotations from dictionary extraction and InstructGPT-3, which outperform the approach with only InstructGPT-3 annotation.

heading: 3 Related Works
text: Weak Supervision deep learning approach has achieved remarkable success in several domains beyond NLP (Zhang et al., 2022). However, the main bottleneck is collecting massively annotated data. To address this issue, weak supervision replaces ground-truth annotation with automatic annotation based on heuristic rules, gazetteers or constraints linguistic rules to address. Some techniques called distant supervision exploit semantic links from knowledge bases or ontologies (Lison et al., 2021). Karamanolakis et al. (2021) proposes an iterative self-training method to combine classic weak supervision and inference of the learning model to extract entities not covered by the initial heuristic rules. In the clinical domain, weak supervision has already been used for specific use cases (Cusick et al., 2021; Fries et al., 2021; Wang et al., 2019).
Clinical Language Models In the clinical context, some specific terms are underrepresented or absent in the general domain. As a result, the clinical NLP community has pretrained Language Models (LMs) (Alsentzer et al., 2019; Lee et al., 2020; Alsentzer et al., 2019) over domain-specific
1codebase: https://github.com/arkhn/bio-nlp2023.
corpora (i.e. MIMIC-III (Johnson et al., 2016), Pubmed abstracts). These models could be trained from scratch or from checkpoint to specialize a domain-agnostic model (Gururangan et al., 2020).
Though, the performance gains are marginal compared to the general language model. The structure and the abbreviated text present in clinical notes hurt performance. Instead of pretraining a specialized clinical model, machine learning practitioners can fine-tune agnostic-domain LLMs such as the GPT family of models or T5 on the clinical task. Fine-tuned general-purpose models have proven effective in clinical question-answering, protected health information de-identification, and relation extraction (Lehman et al., 2023). But this approach requires an important infrastructure and a regular re-finetuning if the data distribution of the EHR shifts. Nevertheless, some LLMs have been trained from scratch over clinical domainspecific notes such as GatorTron (Yang et al., 2022), BioGPT (Luo et al., 2022) or ClinicalT5 (Lu et al., 2022) who achieved promising performance on several tasks. Additionally, in-context learning with agnostic LLMs such as InstructGPT-3 (Ouyang et al., 2022) where no weight is modified shows good results (Agrawal et al., 2022; Brown et al., 2020) and outperforms specialized smaller models on several clinical tasks.
Prompt-based Method Prompt-based learning for generative language model treats a downstream task as a language modelling problem where a language model predicts the next tokens of the instruction given a textual prompt (Sainz et al., 2021).
In this paradigm, instead of fine-tuning a model to a downstream task ("pre-train, fine-tune and predict"), we manipulate the behaviour of a pretrained LM using an appropriate prompt to give the desired output ("pre-train, prompt and predict"). prompt engineering explores the most suitable prompt method applied to a LM to solve a task. This way, an unsupervised pre-trained LM can be used for many tasks (Liu et al., 2023).
Among these methods, in-context learning is the most popular method for information extraction, question-answering or sentiment analysis. In the clinical domain, some works exist on information retrieval and question-answering. The prompt contains three components: the examples’ template, the set of examples and the ordering of prompts, such as present in Figure 5. The aim is to provide some training examples in the prompt before the
test example. However, the chosen examples and their ordering and the format could impact performance (Zhao et al., 2021); these three components must be tuned to optimize performance.
In another way, we can cite works around the chain of thoughts (CoT). This encourages the LLM to explain its reasoning to get more accurate results, especially in mathematical and logical reasoning (Wei et al., 2022; Cobbe et al., 2021). This technique could be used with reason edited manually in the prompt or with two separate prompts where the first involves a reasoning task, then concatenated with the second prompt involving the main tasks.
Other techniques involve generated knowledge similar to CoT. Instead of reasoning, the first prompt generates potentially useful information associated with the tasks concatenated in the final prompt (Liu et al., 2022).

heading: 4 Method

heading: 4.1 Creating Annotations and Knowledge Distillation via Weak Supervision
text: Annotation Extraction from LLM Output Our study is inspired by the method developed in this paper (Agrawal et al., 2022). Their works benchmark how InstructGPT-3 (Ouyang et al., 2022) per-
form clinical NLP tasks in English. They show that InstructGPT-3 performs well in several clinical tasks. They introduce 3 new datasets to benchmark few-shot clinical information extraction to achieve this. Also, they introduce guided prompt design to induce easy-to-structure output with resolvers (or parsers) to convert the output into a structured prediction easily. Our work differs in the following:
1. Our studies areas are knowledge distillation via weak supervision and the improvement of this technique combining annotations from LLMs and dictionary extraction;
2. our methods are applied in a multilingual context, the initial work was only done in English;
3. we focus on the clinical entities extraction task based on the E3C dataset guidelines.
In this work, the LLM is used only as a predictor; we only query the model, no additional fine-tuning step has been realized, and we can only access inference parameters such as temperature, top p, frequency or presence penalty. We set the temperature and top p to 0 to control randomness and have a deterministic behaviour. So as not to penalize repetitions, we set the presence penalty and
frequency penalty to 0. We use an InstructGPT-3 model (text-davinci-003) (Ouyang et al., 2022) to infer the whole annotations for all our experiments. We provide the model with an instruction concatenated by the example to be predicted (Figure 5). The output of InstructGPT-3 is a string of characters that we must structure to align the predicted clinical entities with the initial text (Figure 2).
The task is to annotate the words (or tokens) of a sentence x ∈ Σ∗ with a set of labels such that L = {O,Bclin, Iclin} where O denotes a word in the text without a label, Bclin the first word of a clinical entity and Iclin the following words according to the format IOB (Ramshaw and Marcus, 1995). The goal is to identify the labels O, Bclin, Iclin and their character offsets in the sentence x. The task output is defined as ŷ = [y1, y2, . . . , yn] ∈ Y , where ŷ is the set of predicted annotations, and yi = ⟨si, ei, li⟩ such that si is the start offset, ei is the end offset and li ∈ L of the ith annotation.
As mentioned, a prompt-based method requires concatenating a template t ∈ Σ∗ with our sentence x to give our prompt, such as p = concat(t, x). We produce our output o ∈ Σ∗ from our LLM model Φ such as o = Φ(p, θh), where θh represents the set of hyperparameters (temperature, top p, frequence penalty, presence penalty).
Then, we structure o such as Σ∗ → Y using a simple string-matching function to produce a set of labels ŷ where r is our resolver applying the string matching function: r(o, x) = ŷ.
Knowledge Distillation via Weak Supervision Finally, the annotations generated via InstructGPT-3 prediction are used as a training dataset to fine-tune a smaller language model to a NER downstream task. For smaller language models, we limit our study to encoder models as mentioned in Table 1.

heading: 4.2 Prompting
text: We prime the model with three annotated data points, each corresponding to a sentence from our corpus (Table 2). For each language, we try 3 sets of data points. For each of them, we test the F1-Score performance of InstructGPT-3 on the test dataset (gold standard), and we select the set with the best F1-Score to perform prediction on the unannotated dataset. We insert keywords associated with the E3C guideline definition of the clinical entities into prompts. We add guidance to explicit the response structure to facilitate parsing
the output (Agrawal et al., 2022) (Figure 5.2).

heading: 5 Experiments

heading: 5.1 Dataset
text: We use the annotated E3C multilingual dataset (Magnini et al., 2020) for our experiments, consisting of two annotation types: temporal and clinical entities. The languages supported are English (en), Basque (eu), Spanish (es), French (fr) and Italian (it). Clinical entities are identified as patient disorders which could map to the UMLS meta thesaurus. The annotators have linked extracted clinical entities and UMLS concepts. In our experiments, we only extract clinical entities without mapping UMLS concepts. The E3C dataset is organized into 3 layers. A layer consists of a subset of files from each language annotated in a certain way (manually, semi-automatically) depending on the layer:
• the first layer (gold standard) consists of the full manual annotation; we used this layer as a test set for our experiments;
• The second layer consists of semi-automatic annotation; we use this layer as a train set with the initial annotation or the annotation inferred by InstructGPT-3. Moreover, we have access to two states of this layer; the first is the layer entirely annotated with dictionary extraction (silver); the second is a subset of this layer (only 10%) that has been fixed manually (silver with fixed annotations). The dictionary contains terms from UMLS and terms extracted from gold standard;
• Finally, the third layer (layer 3) is unannotated, which we don’t use for our experiments.
As mentioned above, the E3C dataset is wellsuited for our weak-supervision studies. But, the dataset has limited data in its various layers (Table 2). To address this limitation, we divide silver into five parts using 5-fold cross-validation. For silver with fixed annotations, we use the entire data as the training set. Our experiments employ multiple models for each language and relied solely on xlm-roberta-base in a multilingual context. The results presented in our work are an aggregation of the means and standard deviations across models and folds. However, each experiment result and model are reported in Appendix 7.

heading: 5.2 Experimental Setup
text: We conduct experiments on the clinical entity extraction tasks. For each language, we use models mentioned in Table 1 as a student model for the knowledge distillation step. We conduct our experiments we use five different dataset settings. For Monolingual Setting (SMonoSilver), Gold Setting (SMonoGold), SMonoSilver ∩ SMonoGold (SMonoGold ∩ MonoSilver) and each language, we use the silver of the corresponding language. For the Multilingual Setting (SMultiSilver) and each language, we concatenate the silver of the whole languages in E3C to constitute the train set. Finally, for all settings and each language, we test our method on the gold standard of the language we experiment with.
• Monolingual Setting (SMonoSilver): We use a ratio r to control the mix of annotations, with r representing the proportion of annotations from dictionary extraction and (1 − r) representing the proportion of annotations
from InstructGPT-3. If r = 1, the models are trained using only InstructGPT-3 annotations, while if r = 0, the models are trained exclusively with dictionary extraction annotations. We test and compare the performance of the trained models using various ratio values of r.
• Gold Setting (SMonoGold): we use silver with fixed annotations as the train set, and we compare encoder models trained on manually corrected annotation (r = 0) and an encoder model trained on the same subset but using InstructGPT-3 prediction annotations (r = 1);
• SMonoGold ∩ MonoSilver: we use silver as the train set. Still, we replace weak-supervision annotations with the annotation fixed in silver with fixed annotations. So, a small part of the InstructGPT-3 prediction annotations and the dictionary extraction annotations has been replaced by manual annotations;
• Multilingual Setting (SMultiSilver): we use the same setting as SMonoSilver except we are on multilingual training context. For this setting, our trained models are multilingual language models. We use xlm-roberta-base.

heading: 5.3 Results
text: InstructGPT-3 Prediction Analysis For silver, we observe that InstructGPT-3 extracts more entities than original extraction (Table 2). This trend is reduced in English and Spanish even if we observed a more important quantity of Iclin in tokens annotated by InstructGPT-3 for all languages. For silver with fixed annotations, the quantity of tokens annotated by both methods (InstructGPT-3 vs
manually corrected annotation) is relatively equivalent.
Knowledge Distillation Evaluation We compare the distilled model and InstructGPT-3 on the gold standard (Table 3). Distillation is beneficial in terms of performance for Basque, French and Italian. Moreover, we denote a remarkable gap between InstructGPT-3 (0.63) and distilled models (0.75) in Italian. Spanish and English have reversed trends: InstructGPT-3 performs better than distilled models. This echoed the exception we observed in the InstructGPT-3 Prediction Analysis paragraph.
SMonoSilver with r ∈ {0, 1} If we compare the global F1-Score (Table 4), the distilled models (r = 1) perform better than the weak-supervised models (r = 0) trained. In detail, distilled models display a better recall and recognizes multi-word clinical terms more easily. Still, this flexibility, balanced by the too-biased detection of false positive
terms, lowered the precision score. In comparing distilled models versus weak-supervised models (Figure 3), we note a noticeable performance gain of almost 0.1 in Basque, followed by French and Italian. In English, the F1-score of both models is relatively equivalent. For Spanish, the weaksupervised models outperformed the distilled models and still has our highest F1-Score.
SMonoGold The amount of annotated tokens in silver with fixed annotations is relatively small compared to silver (Figure 2). This hurts the result (Table 4) of the distilled models (0.61 with silver with fixed annotations vs 0.70 with silver) in contrast to the weak-supervised models, where performance has gained 0.03 (0.70 with silver with fixed annotations vs 0.67 with silver). The weaksupervised models performance is relatively better than the distilled models. Moreover, the distilled models recall performance seems to be affected by the small amount of data and annotated tokens (0.68 for SMonoSilver with r = 1 vs 0.58 for SMonoGold with r = 1).
SMonoGold ∩ MonoSilver The results (Table 4) show better performance for both models when we mix a slight quantity of manually annotated data with silver. The distilled models outperforms the weak-supervised models with a gain of 0.03. In both cases, the F1-Score gain is due to the improvement of the recall: we obtain a gain of 0.05 compared to the SMonoSilver with r ∈ {0, 1}.
SMonoSilver For all languages except Basque, we obtained better results when we mixed weak supervised and InstructGPT-3 annotations. The local optimum for these languages is reached when r ∈ [0.4, 0.6] (Figure 4). The Basque doesn’t follow this trend; using a dataset with only InstructGPT-3 annotations (where r = 1) gives the best result among all tried r values.
SMultiSilver Using a multilingual train set and LM (xlm-roberta-base) gives inferior results compared to SMonoSilver (Table 5). Though we obtain better results in Italian than the SMonoSilver (+0.01); the optimum is set to r = 0.8. In the other case, mixing annotations described in 5.2 don’t affect results as observed in SMonoSilver due to the noise generated by the multilingual nature of the train set.

heading: 5.4 Discussion
text: Our experiments using the E3C dataset demonstrate the potential of knowledge distillation and weak supervision in the context of clinical entity extraction tasks. We observe that distilled models outperform classic weak supervision approaches, especially in Basque, French, and Italian languages. However, we notice some interesting trends in English and Spanish that require further analysis.
The trend is reversed in Spanish, with the weaksupervised models performing better than the distilled ones. For all the models we trained for Spanish (Table 1), we don’t distinguish any difference between monolingual, agnostic domain, multilingual, or medical monolingual language models. One possible explanation for these trends is the difference in data sources. While the corpora for other languages come from the Pan African Journal or Pubmed, the Spanish corpus is sourced from the SPACCC corpus. The clinical entity distribution and semantic differences from this source could bias our results. Moreover, additional data cleaning has been applied to layer 1, such as sentence and punctuation removal and capitalization, which may reinforce this difference between the languages.
In English, the difference in performance between distilled and weak-supervised models is relatively small compared to other languages. This can be attributed to the superior quality of annotations in the silver. The English lexicon resource (supplied by the UMLS meta-thesaurus and terms extracted in gold standard) employed for mapping clinical entities in the text is likely more exten-
sive and precise than those accessible for other languages with fewer linguistic resources.
Furthermore, SMonoSilver reveals that combining annotations from Dictionary extraction and InstructGPT-3 marginally outperforms when r = 1. Integrating various annotation sources shows promise and typically enhances model generalization. However, in the case of Basque, SMonoSilver does not yield the best results when we have only InstructGPT-3 annotations (r = 1). As we raise the ratio r, we observe a gradual improvement in F1-score. It can be explained by the original annotations from silver in Basque was created using a low-resource lexicon. As shown in Table 2, only 63 Iclin tokens were initially anno-
tated, in contrast to 482 tokens for InstructGPT-3 annotations.
In the case of SMultiSilver, we did not observe any significant results. The performance of Spanish, Italian, and French languages either experienced a slight improvement or was unaffected by the multilingual composition of the training dataset. However, this setting negatively impacted English and Basque. The predominance of Romance languages in the dataset could be the cause.
Moreover, Basque is a distinct and isolated language with unique linguistic structures. The other languages in the training dataset are linguistically distant, which may introduce noise during the training process and consequently affect the performance of the Basque model.
Another interesting observation is that InstructGPT-3 extracts almost twice as many entities as the original extraction method (Figure 2). This trend is more pronounced in silver, while the number of annotated tokens in silver with fixed annotations is almost equivalent between both annotation sets, likely due to human validation. This difference could be explained by the fact that InstructGPT-3 has no access to the guidelines, and the prompt mentioned to extract "disorders," "disease," or "symptoms" is less restrictive than the E3C guideline annotation.
Our results highlight the potential of knowledge distillation and weak supervision for clinical entity extraction, particularly for languages with more limited resources. Though, data sources, annotation quality, and the comprehensiveness of linguis-
tic resources influence the performance of these methods. Further research is needed to address these challenges and improve our methods.

heading: 6 Limitation
text: One limitation of our study is the small size of the test set, which may impact the generalizability of our results. Additionally, we restrained our work on clinical entity extraction; in future work, we would investigate more in several tasks using the E3C temporality layer to cover a task of Name Entity Recognition and Relation Extraction tasks.
Finally, the E3C guidelines have been designed for clinical entity extraction and entity-linking via UMLS entities. After the first step of manual annotation, some spans of the entities have been modified to fit as close as possible to the semantical concepts found in UMLS (Magnini et al., 2020). For instance, clinical entities could be split into separate disorder concepts, and the extent of a disorder candidate could be reduced to fit with a concept. These biases could induce additional difficulties in finding the correct span for a given model.

heading: 7 Conclusion
text: Our results demonstrate that the knowledge distillation with InstructGPT-3 outperforms the dictionary supervision for extracting clinical entities.
We show that mixing these approaches to build a training dataset brings diversity to the annotations and improves the distilled model performance.
Weak-supervision approach with LLMs is relatively promising for creating a training dataset. This reduces the annotation cost and, at the same time, focuses the manual annotation on the test set, which is one of the most prominent parts of highstake domains like healthcare. Furthermore, the interest of the approach is also to fine-tune a small to medium-sized LM that may be used locally without the leak of confidential medical data and with a reduced energy cost. In a low-resource context, such as Basque, LLMs offer a competitive alter-
native to the classic weak supervision technique, which requires linguistic resources.
Furthermore, we aim to investigate advanced techniques to combine various annotations by incorporating confidence measures from the different predictions. Using other LLMs predictions and ensemble, the difference could be pertinent because the annotation diversity can improve a model’s performance, as we observed on SMonoSilver (Figure 4). Additionally, we will consider utilising performance metrics (such as recall and precision) to decide which type of annotations (begin or innertokens) to retain for each prediction method.
Finally, adapting CoT or generated knowledge (Wei et al., 2022; Cobbe et al., 2021) for clinical entity extraction could improve LLM’s precision. To our knowledge, none of these techniques has been adapted to clinical information retrieval. We could craft a prompt with different annotation steps through different examples. At each annotation step, we describe a precise instruction and its result. For example, incorporating the three steps of the E3C annotation into the prompt could help encourage the LLM to better adhere to the guideline.

heading: Future Work
text: Additionally, we restrained our work on clinical entity extraction; in future work, we would investigate more in several tasks using the E3C temporality layer to cover a task of Name Entity Recognition and Relation Extraction tasks. Finally, the E3C guidelines have been designed for clinical entity extraction and entity-linking via UMLS entities. After the first step of manual annotation, some spans of the entities have been modified to fit as close as possible to the semantical concepts found in UMLS (Magnini et al. , 2020). For instance, clinical entities could be split into separate disorder concepts, and the extent of a disorder candidate could be reduced to fit with a concept. These biases could induce additional difficulties in finding the correct span for a given model