dataesr/scientific-paragraphs-categorization huggingface dataset:
Datasets The label distribution among the corpus and each train, validation and test dataset is presented in Table  2. The most common type of trauma was the class "Fall" followed by "Other trauma" and "Motor Vehicle Accident". An example of clinical notes translated from French is given in supplementary Figure  S2. The median age at the visit was 37 years (1st and 3rd quartiles [24-58]) and 58. 5% of patients were male. Electronic health record was introduced in year 2012 in Bordeaux University hospital, which explains the lower proportion of data for this particular year. Year 2019 saw a decrease in ED venues while in 2020 there have been a significant increase. Table  3  summarizes the characteristics of the train, validation and test datasets for the concerned population. Distribution of the variables age, sex and year of venues at the ED were comparable among the 3 datasets.
Fine-tuning performance of models Unlike machine learning methods such as TF-IDF, supervised fine-tuning of the transformer models is time-consuming and is greatly accelerated with the use of GPUs (Graphics Processing Unit). The self-supervised fine-tuning step for GPTanam model took approximately 12 hours. At that point, GPTanam could generate artificial clinical notes as seen in supplementary Figure  S3 , which could not easily be differentiated from the original ones. One epoch of supervised fine-tuning took 15, 16, 19 and 18 minutes for, respectively, CamemBERT, FlauBERT, BelGPT2 and GPTanam. When looking deeper into each transformer model's F1-scores on the validation dataset, Figure  1  shows that CamemBERT reached its maximum F1-score (0. 873) at Epoch 6, FlauBERT achieved 0. 874 at epoch 5, BelGPT2 was at its peak (0.
Performance of models Average macro precision and micro F1-scores were systematically higher with the transformers than with the TF-IDF/SVM couple on the complete test dataset, as seen in Table. 4. Among the transformers, GPTanam achieved an average micro F1-score of 0. 969, outperforming CamemBERT, FlauBERT and BelGPT2 for which F1-scores were 0. 878, 0. 873 and 0. 887 respectively. Macroaverage precision was higher than F1-score in almost all cases, except for TF-IDF/SVM where macro precision was lower than micro F1-score (macro precision = 0. 860, micro F1-score = 0. 864). The distribution of n clinical notes per class not being balanced, the micro-F1 scores were, in all cases, lower with the classes where n was lower. Concerning the micro F1-score of the different classes, GPTanam had higher scores than the other transformers and TF-IDF.
04 for TF-IDF/SVM, CamemBERT, FlauBERT and BelGPT2. The average gain of micro F1-score was 0. 01 for GPTanam, which seems more robust to complex and/or ambiguous content. Difference in performance when potentially complex/ambiguous content is taken into account was greater with TF-IDF/SVM, CamemBERT, FlauBERT and BelGPT2 than with GPTanam, especially with the classes MVA and Sport Accident where the average gain of micro F1-score per class was 0. 07 as seen in Figure  2. Performance for the class "Accident of exposure to bodily fluids" did not improve for TF-IDF/SVM, CamemBERT and FlauBERT when complex/ambiguous content was removed from the test dataset. Performance of GPTanam did not improve for the GPTanam with the classes "Foreign body on the eye" and "Other trauma" but F1-scores were already very high with, respectively, F1-scores of 0.
The use of ontologies developed in the field of emergencies could constitute an area for improvement. Transformers have also recently been tested for the identification and replacement of abbreviations with good results for BERT  42, 43  , however there has not yet been a test on data from a mixture of common language and medical terms in French. In addition, as the authors who proposed the CamemBERT model did not compare the different models from the OSCAR, CCNet and Wikipedia datasets on a classification task, a future work could compare the different sets on our database. In this logic, it would be appropriate, while we have only used the basic models of CamemBERT, FlauBERT and GPT-2, to test the different sizes of pretraining datasets on a classification task as well as the different sizes of models.
Indeed, Martin's team has shown that the standard CamemBERT model (110 million parameters) trained on all 138GB of OSCAR text, does not massively outperform the model trained "only" on the 4GB sample in morphosyntactic labeling, syntactic parsing, Named Entity Recognition (NER) and Natural Language Inference (NLI). 44 One perspective considered is to test different models of French transformers that have been released since CamemBERT, FlauBERT or BelGPT2 such as Pagnol or BARThez.
Taxonomy The performance of the models improved when we excluded the clinical notes that we considered to be the most complex and/or ambiguous from our test dataset. The classification errors analysis showed that when clinical notes encompassing two mechanisms of trauma (e. i. "fall from bike on the street") were removed from the test dataset, models performed better. This expected result shows that since the advent of transformers, the margin of progress in a free text classification task is nowadays low. This behavior was less important with GPTanam, which seems to have benefited from the selfsupervised pre-training phase for reducing classification errors by learning semantic representations beforehand. However, the annotation grid created for the project is partly responsible for some classification errors in the sense that there are areas of semantic overlap between classes.
Acknowledgment This work was carried out within the framework of the TARPON project (Traitement Automatique des Résumés de Passages aux urgences pour un Observatoire National) led by the Inserm team Injury epidemiology (project leader E. Lagarde) and the emergency department of the Bordeaux University Hospital in collaboration with the SISTM team, shared by Inria and Inserm. This project is the winner of the 2nd call for projects of the Health Data Hub, Grand Défi "Improving medical diagnosis through Artificial Intelligence" and Bpifrance. This study has been carried within the framework of PIA3 (Investment for the Future), (project No. 17-EURE-0019). We would like to thank all members of the labeling team. We thank the University Hospital of Bordeaux for providing the logistical support that allowed us to access and analyse the data needed for the manuscript in such a short period.
Data availability The dataset is not available due to patient privacy restrictions. However, the model may be shared with qualified researchers from academic or university institutions upon request via the corresponding author. Furthermore, identifying information was found in the dataset and a parallel work consisted in handling those using NER (Named Entity Recognition) and FlauBERT. Data processing and computing were conducted within the facilities of the Emergency Department of the University Hospital of Bordeaux which have received regulatory clearance to host and exploit databases with personal and medical data. All patients from which information were retrieved were 15 or more years old.
of Trauma Train dataset Validation dataset Test dataset Foreign 642 (4. 4% 180 (5%) 186 (4. 3% 100 (4. 5% Body in Eye ) ) 8 ) Fall 477 (32. 9 116 (32%) 155 (36%) 749 (33. 3 8 %) 2 4 4 %) Sport 131 (9%) 341 (9. 4% 371 (8. 6% 202 (9%) Accident 1 ) ) 3 Intentional 341 (2. 3% 73 (2%) 112 (2. 6% 526 (2. 3% Injury ) ) ) Other 371 (25. 6 950 (26. 1 985 (22. 8 564 (25. 1 trauma 3 %) %) %) 8 %) Total 145 (64. 6 363 (16. 2 431 (19. 2 224 (100 32 %) 4 %) 5 %) 81 %) Total n % n % n % n % Accident of 132 (0. 9% 40 (1. 1% 41 (1%) 213 (0. 9% Exposure ) ) ) to Bodily Fluids Assault 158 (10. 9 393 (10. 8 498 (11. 5 247 (11%) 7 %) %) %) 8 MVA 202 (14%) 495 (13. 6 568 (13. 2 309 (13. 7 8 %) %) 1 %).
Multi-omic data analysis and mechanistic diagram mapping 2. 1. 1 Footprint-based analysis and causal network contextualization in a SARS-CoV-2 infected A549 cell line We used a published transcriptome dataset (4) focusing on A549 cells and combined it with phosphoproteomic data of mock-treated and SARS-CoV-2-infected cells  (5). We thus contextualized the perturbed signaling events of the viral infection and inferred a causal network using the Carnival tool  (6)  with the COSMOS approach  (7)  based on a prior knowledge network assembled from OmniPath resources  (8). The Carnival-inferred network connected the top ten deregulated kinases with the top 30 deregulated transcription factors (TFs; Fig. S1A ). Among the deregulated proteins in the Carnival-inferred network, we found four kinases (TBK1, IKBKE, TICAM1, MAPK3), four TFs (IRF3, ATF4, ATF6, SMAD1), and one serine protease (MBTPS1) among seven diagrams of the C19DMap (Fig.
Transcription factor activity and gene expression analysis in SARS-CoV-2 infected NHBE and A549 cell lines Next, we expanded our TF study to include RNA-seq data from Normal Human Bronchial Epithelial (NHBE) cells. We used the same datasets (GSE147507)  (4, 9)  to detect differentially expressed genes (DEGs) between these two cell lines (Fig. S2A ). TFs that statistically significantly regulate these DEGs were detected by limitless arity multiple testing procedures (LAMP; Fig. S2B )  (10). Results showed that the number of TFs detected for A549 cells was higher than that for NHBE cells, similar to the results of the DEG analysis. Many TFs detected in both cell types were involved in immune responses. We also performed Gene Ontology (GO) enrichment analysis to determine the functions of the genes regulated by these TFs (Fig.
Extended pathway analysis in SARS-CoV-2 infected NHBE and A549 cell lines Next, we identified altered COVID-19-specific and general molecular pathways in NHBE and A549 infected cells using the same RNA-seq dataset (GSE147507)  (4, 9). Then, overrepresentation analysis was performed on a combined pathway collection from C19DMap (2), WikiPathways  (11) , and Reactome  (12)  with 1,840 human pathways containing 12,037 unique genes (Fig. 2A ). Over-representation analysis revealed 74 altered pathways in NHBE and 101 altered pathways in A549 cells of which 11 pathways were changed in both, including several immune-and metabolism-related pathways (Fig. 2B ). Interestingly, NHBE cells showed several C19DMap pathways altered after SARS-CoV-2 exposure including interferon and coagulation pathways (Fig.
Single-cell transcriptome analysis in epithelial cell types of COVID-19 patient groups with different severity profiles. Next, we wanted to expand our analyses using patient data. The single-cell RNA sequencing (single-cell RNAseq) dataset was composed of bronchoalveolar lavages from nine COVID-19 patients (GSE145926)  (13)  and epithelial cells isolated from the lungs of nine healthy subjects (GSE160664)  (14). Clustering analysis was carried out on the entire matrix and showed 44 distinct clusters as the best representation of cell types (Fig. S6 ). Five epithelial cell types were selected by cell sample size between groups and gene markers, following the classification of Okuda and collaborators  (27). For each cell type, data of moderate, severe, and critical COVID-19 cases were grouped as the category COVID-19, and differential expression analysis was performed between COVID-19 and healthy controls.
Among all the DEGs overexpressed in COVID-19 patients in each cell type (Table  S1 ), 26 were shared among all lung epithelial cell types (Table  S2 ). The overexpressed genes in five cell types of COVID-19 patient groups were reported on the C19DMap to evaluate the activation of specific pathways. In all the epithelia cell types of the COVID-19 group, the genes IFIH1, OAS1, STAT1, OAS2, OAS3, and IRF7, which belong to the type I interferon pathway (WP4868), are found to be overexpressed, meaning that they get activated during SARS-CoV-2 infection. In addition, evidence of direct infections of SARS-CoV-2 in these cell types was confirmed using the databases Reactome  (12)  and KEGG  (15) , with activation of pathways linked to Interferon and Influenza A infection, respectively. Interestingly, OAS1, OAS3, and IFIH1 were also found to be differentially expressed in NHBE and A549 infected cells, with OAS1 and OAS3 having an opposite expression profile.
Combining omics data with mechanistic pathway modeling To expand on patient data and use the available diagrams in a more active way than mapping, we decided to employ the HiPathia approach (  16 ) that effectively combines RNAseq data with mechanistic diagrams and pathway modeling. In this step, a public RNAseq dataset of nasopharyngeal swabs from 430 individuals with SARS-CoV-2 and 54 negative controls (17) (GSE152075) was used. Following the pipeline developed for this study, 16 of the 23 pathways were suitable for the HiPathia algorithm. We found that 47 of the 145 circuits analyzed using the HiPathia algorithm were differentially activated (adjusted p-value < 0. 05), showing global deregulation of the pathways involved in SARS-CoV-2 infection (Table  S3 ). The most representative pathways are shown in Table  3.
Integration of the Type 1 IFN, the RA system, and the NLRP3 inflammasome curated pathways into a macrophage-specific Boolean model The population of macrophages expands during SARS-CoV-2 infection, and hyperactivation of these cells can lead to severe immunopathologies  (40). To be able to computationally simulate the effects of SARS-CoV-2 on several COVID-related pathways in macrophages, we extended a previously built macrophage polarization model to incorporate biological processes related to SARS-CoV-2 infection, including the Type 1 (T1) IFN response, the Renin-Angiotensin (RA) system, and the NLRP3 inflammasome modules from the C19DMap. The resulting COVID19 Macrophage Model, named MacCOV (https://gitlab. lcsb. uni. lu/computational-modelling-and-simulation/macrophage-model), comprises 131 nodes and 271 interactions manually verified against the macrophage-specific literature.
Text-mining and AI-assisted drug target enrichment We used two AI assistants, INDRA and AILANI, to keep the C19DMap up to date and to expand and enrich it with new knowledge. All analyses were performed using a harmonized bipartite graph that included the diagrams from MINERVA, REACTOME, and WikiPathways (See Materials and Methods). In the C19DMap, we now have a collection of 21 MINERVA  (5)  hosted diagrams, two REACTOME (6) pathways, and 19 WikiPathways (7) diagrams. We created a list of drugs and drug targets using the repository's content and information from various sources. For example, we provide a list of content-related information from Clinical trials DB, Transcription Factors, drug and protein targets, and miRNA. The corresponding file can be accessed on our public repository. From an initial list of 3,573 proteins extracted from the C19DMap and the drug-target information compiled for the C19DMap, we obtained 1,476 drugs associated with 1,120 drug targets to populate our C19DMap drug target database.
We identified 54 targets from the omics data and C19DMap diagrams integrative analysis and the computational modeling analysis (Table  6 ). Using our C19DMap drug target database, we could infer drugs, chemicals, and miRNAs that target these identified nodes (Table  S5 ). Table  6. A list of 54 identified targets from the omics data and C19DMap diagrams integrative analysis and the computational modeling analysis. MAPK11 STAT1 TREML4 SMAD1 FOS TBK1 TICAM1 JUN ARL6IP6 TBK1 RELA CASP7 IKBKE NFKB1 LNPEP IRF3 STAT2 HMOX1 ATF4 IRF9 FADD ATF6 BACH1 AKT1 MBTPS1 TBP ALG5 TP53 TCF12 AGTR1/2 STAT3 IFIH1 EGFR ISG5 OAS1 KEAP1 JUND OAS2 CUL3 AHR OAS3 E FOSL1 IRF7 nsp15 MAS1 BAX nsp14 RBX1 IFNB1 RNA Nsp3 TLR9 TLR7 nsp13.
Among the drugs, pharmacogenomic data were available for cisplatin. Cisplatin has a higher DRP score for Latino/Admixed Americans, both sexes and a lower DRP score across Ashkenazi Jewish and East Asian populations (Fig. S14 ). Emricasan was tested in 18 clinical trials, targeting liver diseases, and recently the drug was tested for its efficacy in COVID-19 disease in 13 patients with mild symptoms; however, no results have been published (https://clinicaltrials. gov/ct2/show/NCT04803227?term=emricasan&draw=4&rank=4). Lastly, for EGFR, the CAP score is very low across all populations, with a slightly higher CAP score for African/African American populations. Using our internal drug-drug target database, we retrieved two drugs, namely zanubrutinib and abivertinib. Zanubrutinib is being tested in clinical trials for the treatment of lymphoma patients (88 clinical trials retrieved from https://clinicaltrials.
uni. lu/covid/models/) and FAIRDOMHub  (58). We have worked on tool interoperability and promoting community standards; therefore, most input formats are GML, SIF or SBML, and SBML Qual files in an effort to enhance model reusability  (59). All maps and models are available under a CC-BY license. Appropriate metadata associated with each of the analyses and modeling results presented in the article is registered and indexed on FairdomHub to facilitate accessibility. Furthermore, we plan to submit the models obtained to model repositories such as the Cell Collective (30), GINsim (60), and BioModels  (61). We have also built the C19DMap-Neo4j graph database by integrating the content of the C19DMap diagrams available in MINERVA into the Neo4j framework. This database is available for online exploration at https://c19dm-neo4j.
lcsb. uni. lu and is used as a backend solution for efficient access to the resource data. Biological concepts from the C19DMap diagrams available in MINERVA (such as macromolecules and processes) are stored in the database under Neo4j nodes. In contrast, relationships between these concepts (such as consumption and catalysis) are stored as Neo4j relationships. In addition, annotations, such as UniProt identifiers and PubMed publication IDs, are stored in the form of individual nodes that we can easily query (for an example, see Fig. S15 ).
Lastly, over-representation analysis revealed 58 affected pathways in NHBE cells and 39 enriched pathways in A549 cells, including pathways relevant to immune response, the NFkB pathway, glucocorticoid receptor and MAPK signaling pathway, and pathways related to interferon. The single-cell RNAseq data analysis of a small group of patients confirmed some of the previously identified TFs, DEGs and altered pathways pointed out by the cell line analysis. However, the number of patients in this analysis was relatively small. To expand our analysis, we used an extensive dataset of 450 patients and the HiPathia modeling algorithm to identify affected circuits in the mechanisms described in the repository. We found pathways, such as apoptosis, to be systematically up or downregulated, which means that the whole pathway is relevant to the progression of the disease.
Perspectives As mentioned in our previous report  (2) , most of the diagrams of the CD19DMap repository were initially built using the scientific literature on SARS-CoV-1 and other coronaviruses that were available during the onset of the pandemic. This corpus provided the foundation for rapid curation and a literature triage approach. Annotations for the SARS-CoV-1 viral infection process, including the viral life cycle, host interactions, and therapeutic pathways, were built on this foundation. After more than two and a half years since the appearance of the SARS-CoV-2 virus, the body of scientific literature specific to this type of coronavirus has reached a point where it can now be used to curate complete mechanisms. With the continuous update of pathway information and new datasets related to SARS-CoV-2, reproducible and automated data analysis workflows can be rerun to provide more accuracy and specificity.
Generation of Reactome's SARS-CoV-2 pathway leveraged the database's foundational manual curation, orthoinference projection, and the collaborative resources of the CD19DMap project. The SARS-CoV-2 infection pathway emerged from a computationally generated rough draft via the orthoinference process from the manually curated, peerreviewed Reactome SARS-CoV-1 infection pathway (see Materials and Methods). The community can adopt this approach that identifies SARS-CoV-2-specific interactions to increase viral specificity in the mechanisms in the C19DMap repository. We made considerable efforts to increase interoperability and communication across three different platforms, MINERVA, WikiPathways and Reactome, support Systems Biology standards such as SBGN  (62)  and SBML  (63) , and promote scientific openness with the use of public repositories and the adoption of FAIR (Findability, Accessibility, Interoperability, and Reusability) Data principles  (54).
Using the mechanistic diagrams for omics data analysis Footprint analysis We obtained the transcriptomics dataset from the GEO database with accession number GSE147507 (4). We extracted series number 5 from the dataset, consisting of 2 conditions, A549 cells either mock-treated or infected with SARS-CoV-2, measured in triplicate 24 hours after infection. Differential analysis of the transcript abundances was performed using DESeq2  (64). The resulting t-values of the differential analysis were used as input to estimate pathway activity deregulation using Progeny  (65). The differential analysis t-values were also used to estimate the deregulation of TF activities using Dorothea (66) as a source of TF-target regulon and the Viper algorithm  (67)  to estimate the TF activity score. Phosphoproteomic data of mocktreated and SARS-CoV-2 infected cells were extracted from  (5).
Phosphosite differential analysis log2FC was used to estimate the deregulation of kinase activities using https://github. com/indralab/protmapper as a source of kinase-substrate interactions and a ztest to estimate kinase activity score  (68, 69). Finally, we used Carnival  (6)  with the COSMOS approach  (7)  to connect the top 10 deregulated kinases with the top 30 deregulated TFs with a Prior Knowledge Network assembled from OmniPath resources  (8). Progeny pathway activity scores were used to weigh the PKN and facilitate the optimal network search to connect kinases and TFs. To place our results in the context of the whole study, we matched the genes obtained in carnival results with those included in the curated pathways by the Covid-19 Disease map community (https://covid. pages. uni.
TF activity and drug target identification In this analysis, we inferred the gene regulatory systems that are hijacked by COVID-19, especially the target transcription factors. In order to infer the target transcription factors, we detected transcription factors that statistically significantly regulate the genes whose expression changes were induced by COVID-19. First, the gene groups whose expression changes were induced by COVID-19 in NHBE cells and A549 cells were detected as the DEGs using DESeq2 (64) for the GSE147507 dataset (4, 9) described above. Next, we extracted all the regulatory relationships with Confidence "A", "B", and "C" from DoRothEA (66) as information on the regulatory relationships of transcription factors to each of these DEGs for NHBE cells and A549 cells. The transcription factors that regulated each of these DEGs for NHBE cells and A549 cells were detected by LAMP (10) (significance level < 0.
Pathway and network analysis in SARS-CoV-2 infected NHBE and A549 cells We demonstrate an automated and reproducible workflow for transcriptomics data analysis using pathway-and network-based approaches (see our GitLab repository for details;https://gitlab. lcsb. uni. lu/computational-modelling-and-simulation/pathway-analysisand-extension). The analyses are fully automated in R with clusterProfiler  (73)  and RCy3  (74)  to connect to the widely adopted network analysis software Cytoscape (75) for network visualization. We obtained the transcriptomics dataset from the GEO database with accession number GSE147507 (4). We extracted series numbers 1 (NHBE) and 5 (A549) from the dataset, consisting of 4 conditions in triplicate, NHBE and A549 cells treated with mock (two controls) and NHBE and A549 infected with SARS-CoV-2, measured 24 hours after infection.