dataesr/scientific-paragraphs-categorization huggingface dataset:
74 on the CoNLL2003 dataset. For the deidentification task, Liu et al. (  [37] ) proposed a hybrid system that combines the Bi-LSTM -CRF architecture with a rule-based subsystem. The results of the proposed model on the i2b2 dataset achieve F1 scores of 96. 98%, 95. 11% and 98. 28% under the "token", "strict" and "binary token" criteria, respectively, thus outperforming the results of Dernoncourt et al. (  [11] ) on the same deidentification dataset. The hybrid Bi-LSTM -CRF architecture for sequence labeling was then improved by adding contextualized word embeddings, which significantly improved the task (  [38] ). Large pretrained models, such as BERT (Bidirectional Encoder Representations from Transformers) (  [39] ), CamemBERT (  [40] ) language model for French and FlauBERT (  [41] ) have rapidly become the state-of-the-art approach to model tasks in NLP.
(b. ) CamemBERT: Developed by Facebook, INRIA, and Sorbonne University, CamemBERT (  [40] ) is a stateof-the-art language model for French based on the RoBERTa architecture (  [46] ), which is a variant of BERT. Pretrained on large French corpora, this model has been evaluated on four NLP tasks: part-of-speech tagging (POS), dependency parsing, named entity recognition (NER), and natural language inference. CamemBERT improved the state of the art in the four previous tasks, confirming the effectiveness of large pretrained linguistic models for French (  [40] ). (c. ) FlauBERT: FlauBERT, French language understanding via Bidirectional Encoder Representations from Transformers, is a French language model published in the article FlauBERT: Unsupervised Language Model Pretraining for French by Hang  Le et al.
([41] ). FlauBERT has been trained on a very large and heterogeneous French corpus, with a configuration similar to BERT and CamemBERT. This model was evaluated on several NLP tasks, and the results demonstrate yet again that a French-language model gives better performance than multilingual BERT models as well as other French language models. ( (2017)(  [48] ). Trained on large corpora, this approach is based on the skipgram model, where each token of the text is represented by a sequence of n characters and n grams. For example, for n = 3 (trigram), the word "Patient" will be represented by <Pat, ati, tie, ien, ent, nt>, where '<' and '>' represent the beginning and end of the word. This n-gram information enriches word vectors with subword information and allows morphological information to be captured to construct vectors for unseen words or out-of-vocabulary words.
Methods In this section, we present an overview of the materials and methods used in this study. Figure  1  illustrates the process of deidentification of clinical notes in French. A full description of the datasets, knowledge bases, preprocessing techniques and automatic annotation procedure used to create the training dataset is presented. In addition, the NER methods used in this study and the experimental parameters are clarified. Finally, the manual annotation procedure used to create the test dataset for evaluating the trained models is described at the end of this section. Data sources: The databases used in our study consist of clinical data from the eHOP Clinical Data Warehouse (CDW) at Rennes University Hospital in France (  [51] ) and knowledge bases including national knowledge bases of French streets and city names from the Base Adresse Nationale (BAN) (  [52] ) and healthcare professionals practicing in France from the Health Directory (  [53] ).
• Knowledge bases We collected data from national open sources knowledge databases of French streets, city names, and health professionals practicing in France (  [52] ) (  [53] ). The Health Directory gathers identification data for healthcare professionals and their structures from various national repositories, including the Shared Directory of Professionals involved in the healthcare system (RPPS), Association for the Development of Computer Logic (ADELI), and National File of Healthcare and Social Institutions (FINESS). These data are classified into two categories: data accessible to all and restricted access data. The freely accessible data, for which extractions are published, include the following: • The RPPS or ADELI number. • Gender. • Full name of professional. • Professional category: civilian, military, student.
• Profession. • Mode of practice: private practice, employed, volunteer. • Qualifications: degrees, practice authorizations, skills. • Contact information for the practice structures. • Function and type of activity. The Base Adresse Nationale (BAN) is one of the nine databases in the public service for reference data. It is the only officially recognized address database by the administration. The BAN is accessible in the form of files and APIs and includes the following data: • Street name (nom_voie). • Postal code (code_postal). • INSEE code. • City name (nom_commune). • Old municipality INSEE code. • Old municipality name. • Longitude (lon). • Latitude (lat). • Position type. • Source street name (source_nom_voie) • Municipality certification (certification_commune) • … From these resources, we selected valuable data for our study, such as the first and last names of the health professionals ("Nom d'exercice" and "Prenom d'exercice"), the street names ("nom_voie"), the names of the cities ("nom_commune"), and the postal codes ("code_postal") in France.
• Data Preprocessing We first parsed HTML discharge summaries using a custom HTML parser based on the Python "html. parser" library. This parser allows tracking character positions of extracted raw texts in the original HTML documents so that we can mask potential PII at the end of the pipeline. Once the HTML is parsed, raw text is split into sentences and tokens using the NLTK library 5. This module allows us to split paragraphs into sentences according to punctuation and capitalization. At the end of the preprocessing steps, each token can be positioned in its original position in the HTML document.
The rule-based system includes a set of rules that exploit numerous components, including personal information already available in the dataset (e. g. , patient names and their dates of birth) and the results of the dictionaries, to recognize entities in the text. For each type of entity, we developed a Python function that combines the previous components with several characteristics: • Pattern specifications, which incorporate the classic lexical forms of certain types of entities appearing in medical reports, for instance dates (e. g. , dd-mm-yyyyy, dd/mm/yy), zip codes (00000), and phone numbers (e. g. , '00-00-00-00-00-00, '00 00 00 00', '00. 00. 00'). • Orthographic characteristics, which consist of word specifications such as words that are capitalized, words that begin with upper or lower case, and the length of tokens or words.
• Lexical and semantic indicators. As an example, street names often include terms such as "AVENUE", "BOULEVARD", "RUE", and "ALLEE". • Contextual cues that point to the presence of a particular type of entity in the clinical text. They include specific lexical expressions (e. g. , titles of healthcare practitioners, months/days and their abbreviations, common abbreviations used in French medical reports, etc. Finally, we selected only the sentences that contain at least one entity, and we divided the documents in such a way that for each patient, all his documents go to the same dataset (Train_auto: 80%, Valid_auto: 10%, and Test_auto: 10%) using the Permanent Patient Identifier (ID_PAT). System Architecture: We formulate the medical deidentification problem as a sequence labeling task. For example, given an input sentence "Monsieur le Docteur JEAN DUPONT ", the medical deidentification model will generate the following labeling sequence ".
Bi-LSTM -CRF Based Models We used the BERT multilingual base model (cased) (mBERT), which contains 12 layers, 768 hidden dimensions, 12 attention heads, and 179 M parameters. In addition, we used the PyTorch Deep Learning Framework to implement the mBERT+ Bi-LSTM+CRF model. The model was trained for 5 epochs using the Adam optimizer. To further explore the performance of our models, we used the Flair11 library ([55]) for the implementation of the Bi-LSTM and CRF model with the embeddings mentioned earlier. FLAIR is a natural language processing (NLP) framework built on the popular PyTorch deep learning library. FLAIR enables state-of-the-art performance and includes implemented architectures such as Bi-LSTM and CRF as well as very powerful embeddings such as Flair embeddings, FastText embeddings, BERT embeddings, etc.
Fine-tuning Transformers We used pretrained language models to generate word embeddings for the Bi-LSTM+CRF architecture. In addition, we tested fine-tuning the BERT model and other French versions of BERT, which is a common approach for various NLP tasks. This involved fine-tuning the transformers on the NER task and adding a linear layer for word-level predictions (  [42] ). The models we used included the BERT multilingual base model, the CamemBERTbased model, and the FlauBERT base model. We used the Hugging Face Transformers Framework ([57]) to fine-tune the models on the training sentences. The models were trained for 5 epochs using the Adam optimizer, the batch size was set to 16, and the learning rate was set to 5e-5. Evaluation: • Building the Test Dataset To evaluate the quality of our automatic annotation system, we manually annotated a set of documents representative of all types of documents included in the CDW.
Several types of documents were excluded because they did not mention identifying data, such as Diagnosis-related groups (DRGs) and drug administrations. Data sampling consisted of a random selection of 250 documents stratified in each original data format (that is, native HTML, CDA R2, XML, PDF). Therefore, the total evaluation dataset to be annotated includes a total of 1000 documents. The details of the distribution of document types in the sample are available in [Additional file 1]. Annotators were selected among the members of the Massive Data and Learning Information Systems in Health (DOMASIA) team of the Signal and Image Processing Laboratory (LTSI) and signed a confidentiality agreement. Five people with different backgrounds performed the annotation: three medical doctors in public health with medical informatics backgrounds, one data manager, one PhD student in NLP, and one Master's student in NLP.
Each annotator was randomly assigned 20% of the total sample (n1), that is, approximately 200 documents per annotator. Then, to evaluate the concordance and to measure the interannotator agreement, each annotator also annotated 20% of the documents assigned to each of the other annotators (n2). In total, each annotator annotated the following number of documents presented in Table  3 : The data selected for the annotation campaign were collected on a secure local server. Manual annotation was performed with the Prodigy annotation tool at the sentence level. The guidelines for annotations were shared with all annotators through a dedicated document. A demonstration was also performed on a test sample so that the annotators could become more familiar with the Prodigy (  [58] ) annotation tool.
Statistics of the deidentification dataset Our data source contains more than 900K EHRs. We split the documents into sentences and filtered out sentences that do not contain any of our targeted entities. The dataset is rich in terms of PII instances belonging to our eight classes. Some statistical data on the deidentification dataset are reported in Table  5. In addition, the frequency of entities in the training and manual test datasets is shown in Figure  4. Table  5  provides the number of sentences that contain at least one of the target entities and number of mentions for each entity class for different splits of the deidentification dataset. The entity distribution analysis reveals quite similar patterns in the training and manual test datasets. In both datasets, DATE entities are quite frequent, representing 27.
2% and 36. 2% respectively. This high frequency underlines the importance of dates in EHRs. Another important similarity is observed in the distribution of DOCTOR entities. In the training dataset, they represent 21. 4%, and in the test dataset, 21. 0%. This consistent representation of DOCTOR entities indicates the significant presence of health professional mentions in both datasets. This uniform distribution also extends to other entity classes such as VILLE, PHONE, PATIENT, ZIP, STR and EMAIL, for which the distribution percentages remain consistent between the two datasets.
Evaluation Results The microaveraged F1 scores of all tested models and embeddings on the manually annotated test dataset are presented in Figure  5. The detailed evaluation results obtained by the rule-based system are presented in Table  9. The performance of each class for the best model shows that the model performs exceptionally well for classes such as "DATE", "CITY", "ZIP" and "EMAIL" with F1 scores of 0. 9832, 0. 9954, 0. 9862 and 0. 9896, respectively. For other classes, such as "DOCTOR", "PHONE", "PATIENT" and "STR", the model achieves good F1 scores ranging from 0. 9256 to 0. 9414. The rule-based system performs very well for some classes, such as "DATE", "PHONE", "ZIP" and "EMAIL", with F1 scores of 0. 9950, 0. 9918, 0. 9585 and 0. 9548, respectively. However, the rule-based system's performance was weaker for "PATIENT" and "STR", with F1 scores of 0.
Discussion In this study, we proposed a largely automated approach for the deidentification of French clinical reports using automatic annotation of medical text data based on regular rules and knowledge bases. We also proposed a solution for assisted and semiautomatic annotation of EHRs using the Prodigy annotation tool to reduce the cost of creating a reference corpus for the evaluation of our NER models. It is important to note that our NER models were trained on only one type of document, namely, hospital discharge summaries. To study the generalizability of our models, we evaluated their performance on different types of clinical documents, such as Administrative Records, Emergency Room Passes, Appointment Notices, and Discharge Letters. The results indicate a high level of agreement among the annotators, confirming the quality and reliability of the annotations made on this set of documents and providing a solid foundation for the evaluation of our NER models.
Our experimental results showed that the proposed approach performed well in extracting PII from clinical notes in unstructured text, with an F1 score of 96. 96% on average for the eight entity types considered. Our results also showed that our best deep learning-based model outperformed rule-based systems in terms of F1 score (93. 65%). Additionally, the results of this evaluation showed that our models were able to achieve high performance on different types of clinical documents, strengthening their robustness and applicability in the healthcare domain. Neural networks can incorporate sensitive information, particularly that designed for deidentification tasks. Hence, it is currently difficult to share such models outside the institution that owns the data used to train the model. One advantage of our approach is that it is fully automated, and the whole pipeline can be shared and used by anyone who wants to apply the process to its own data, with little adaptation.
This analysis shows that our approach yields similar results to existing French deidentification tools who used manually annotated training bases (  [9] ), which demonstrates the effectiveness of our method. Limitations of our study: Our study demonstrated the feasibility and effectiveness of using distant supervised learning for the task of deidentifying clinical data, which can help overcome the challenge of limited annotated data in the medical domain. However, there are some limitations. First, the clinical documents used in our study were retrieved from different sources, and the quality of the reports can vary. Certain medical reports are generally long and poorly formatted documents. The preprocessing of the data is therefore crucial, especially the sentence splitting step. In fact, in poorly formatted documents, sentences can be incomplete or improperly split, which can affect the quality of the annotations and the ability of the model to understand the context in which an entity is mentioned.
This difficulty can be reflected in the quality of the deidentification results and the overall performance of the model. Data preprocessing is also crucial for model deployment to ensure that data have the same structure when given to the model for new predictions on unknown examples. Second, automatic labels are sometimes noisy, which can cause the model to learn incorrect contexts. We would like to improve the model with active learning steps to correct annotations that are incorrect or not detected during the initial training step (  [59] ). The final limitation is to train the model on a corpus from a single hospital, which may result in a lack of generalization. Next, we consider collaborative training across different hospitals by developing a federated method to solve our clinical deidentification problem in a distributed scenario (  [16] ).
Conclusion In conclusion, our study provides a promising approach for the deidentification of French clinical reports, which can facilitate the reuse of electronic health records for secondary purposes such as clinical research. Our study also highlights the importance of using advanced NLP techniques such as deep learning for effective deidentification, as well as the need for innovative solutions such as distant supervision to overcome the challenge of limited annotated data in the medical domain. Our work demonstrates that despite the level of noise in the automatically annotated dataset, the trained NER model using the stacked Flair and FastText embeddings combined with the Bi-LSTM+CRF achieves good performance. This is due to the large size of the automatically annotated dataset. Future work could further explore the scalability and generalization of the proposed approach and investigate its applicability in a collaborative study among multiple medical institutions.
Predictive Surprisal Using VMO The essential step in estimating the predictive surprisal is building a model called Variable Markov Oracle (VMO). Based on Factor Oracle (FO) string matching algorithm VMO was decveloped to allow generative improvisation for real-valued scalar or vector data, such as sequences of audio feature vectors, or data vectors extracted from human poses during dance movements. VMO uses suffix data structure for query-guided audio content generation  [16]  and multimedia query-matching  [17, 18]. VMO operates on multivariate time serie data, VMO symbolizing a signal X sampled at time t, X = x 1 , x 2 ,. , x t ,. , x T , into a sequence S = s 1 , s 2 ,. , s t ,. , s T , having T states and observation frame x t labeled by s t. The labels are formed by following suffix links along the states in an oracle structure, whose value is one of the symbols in a finite sized alphabet Σ.
Representation Using VQ-VAE Computing Information Rate and Mutual information for raw audio signals is an extremely challenging and computationally expensive task. We need some form of dimensionality reduction that preserves the semantic meaning of audio (style, musical rules, composer attributes etc) in the latent space. Then, we can estimate IR and MI in the lower dimensional space, quite easily. For our framework, we use a pre-trained Jukebox's Vector Quantized-Variational Autoencoder (VQ-VAE)  [20, 21]  to encode raw audio files to lowdimensional vectors. VQ-VAE is a type of variational autoencoder that encodes data into a discrete latent space. These discrete codes correspond to continuous vectors in a codebook. Using this, we transform our data into 8192 64-dimensional latent vectors.
Experiments and Results We show the advantage of our method compared to other baselines by running simulations on the Labrosa APT dataset  [22]. We construct a dataset with audio wav files of 4 different composers (Bach, Albeniz, Borodin, Mozart). We convert each audio file to the corresponding embeddings from a pre-trained Jukebox VQVAE  [20, 21]. For X i , we create a VMO for each composer, that can synthesize a sequence of embeddings for a given query/context signal C. For our simulations, we construct C as a segment of music (not included in the VMO construction) from any of the composers. Ideally, our SymTE measure should be high for the X i (VMO) of the same composer C. We evaluate the effectiveness of SymTE by measuring the accuracy and F1 score. We conducted 20 trials for all the experiments.
Discussion and Future Work The methods presented in the paper use sequence of latent vectors coming from pretrained neural models of audio. We use VQ-VAE's embeddings and not the quantized codes, so there is only one quantization happening in this work, which is the VMO's. The reason why we chose VQ-VAE over other models is that we need strong pre-trained models and the best one currently is considered to be jukebox's VQ-VAE. Other neural models can be explored as well, as the representation is important for estimation of TE. Our query signals are 256 dimensional (≈0. 71 s). Our method should work for longer queries, but the main bottleneck is the complexity of the generative model. We plan to extend this work with more elaborate results with a bigger data set and query size. We also plan to test the framework in terms of computational time, so as to enable real-time switching for music improvisation.
Reducing failed labor and emergency cesarean section (CS) rates is an important goal. A childbirth simulation tool (PREDIBIRTH software and SIM37 platform) that evaluates a 5-min magnetic resonance imaging (MRI) assessment performed at 37 weeks of gestation was developed to enhance the consulting obstetrician's ability to predict the optimal delivery mode. We aimed to determine the potential value of this childbirth simulation tool in facilitating the selection of an optimal delivery mode for both mother and infant. A retrospective cohort study was performed on all patients referred by their obstetricians to our level 2 maternity radiology department between December 15, 2015 and November 15, 2016, to undergo MRI pelvimetry at approximately 37 weeks of gestation. The childbirth simulation software was employed to predict the optimal delivery mode based on the assessment of cephalopelvic disproportion.
Materials and methods We conducted a retrospective observational study by analyzing the MRI results of 401 patients from the Northern Paris Imaging Network (RIPN, Réseau d'Imagerie Paris Nord). These patients were referred by their obstetricians for MRI pelvimetry at approximately 37 weeks of gestation. MRI examinations have replaced pelvic computed tomography scans because they protect pregnant women from unnecessary radiation exposure, in compliance with the recommendations of the Council Directive 97/43/EURATOM, Ordinance 2001-270 of March 28, 2001. Patient data files were obtained through an MR 003 procedure filed with the National Commission for Informatics and Freedoms (CNIL, Commission Nationale de l'Informatique et des Libertés) to obtain a database deposited with CNIL under number 2099722v0.