dataesr/scientific-paragraphs-categorization huggingface dataset:
76 kcal mol -1 vs. 0. 01 kcal mol -1 for AMOEBA) which was anticipated as ANI-2X tends to disfavor the initial state. A solution would be to compute all possible states explored by ANI-2X/AMOEBA. Indeed, many things remain to be solved in the modeling of the SAMPL4 dataset. For example, in the SAMPL4 challenge overview, Muddana et al. 81  reviewed the experimental conditions and concluded that it could be important to take into account the salt conditions and to go beyond the simple box neutralization. Indeed, in the event of a proton transfer, a new ionic species being created, it would be interesting to study its interaction with different solutions of increasing ionic strength, especially in our case where the full simulation includes polarization effects. We have not done it at this stage as it would require a large number of additional simulations and we decided to retain the present C3 free energy prediction that could probably be improved in a forthcoming study.
In any case, with C3, ANI-2X brings additional interpretative insights on the nature of the ligand. In the near future, it will also be interesting to investigate further the reactivity capabilities of the ANI-2X/AMOEBA approach. Finally, it is worth noting that C3 is the weakest binder of the series. ANI-2X/ AMOEBA still predicts it as such in terms of the relative free energy of binding compared to the other compounds. Overall, the hybrid ANI-2X/AMOEBA model results are in good agreement with experimental results, reaching, as for the solvation free energy studies, an accuracy in the range of chemical accuracy (average error of 0. 94 kcal mol -1 vs. experiment on the dataset) and dividing the initial AMOEBA error by 2. ANI-2X/AMOEBA can accurately predict binding free energies of exible charged systems and the simulations clearly benet from the addition of ANI-2X.
The solvents considered, along with their dielectric permittivity values, are as follows: toluene (3 = 2. 38), acetonitrile (3 = 36. 64), DMSO (3 = 47. 24) and water (3 = 77. 16). Further details regarding the solutes can be found in the ESI. † We withdrew molecules from the dataset that contained chemical elements not available in ANI-2X, resulting in a total of 38 molecules solvated in water (taken from ref. 43), 20 molecules solvated in toluene, 6 in acetonitrile and 6 in DMSO (taken from Essex et al. ).
Acknowledgements This work has received funding from the  European Research Council (ERC)  under the European Union's  Horizon 2020 research and innovation program  (grant agreement no. 810367 ), project  EMC2  (JPP). Simulations have been performed at  GENCI  on the  Jean Zay machine (IDRIS, Orsay, France ) on grant no. A0070707671  and at  TGCC (Bruyères le Châtel, France ) on the  Irène Joliot Curie machine. The work performed by H. G. and O. I. (PI) was made possible by the  Office of Naval Research (ONR)  through support provided by the  Energetic Materials Program  ( MURI grant  no. N00014-21-1-2476 ). This research is part of the Frontera computing project at the  Texas Advanced Computing Center. Frontera is made possible by the  National Science Foundation  award  OAC-1818253.
72 and 73. 73 The solvents considered, along with their dielectric permittivity values, are as follows: toluene (3 = 2. 38), acetonitrile (3 = 36. 64), DMSO (3 = 47. 24) and water (3 = 77. 16). Further details regarding the solutes can be found in the ESI. † We withdrew molecules from the dataset that contained chemical elements not available in ANI-2X, resulting in a total of 38 molecules solvated in water (taken from ref. 43), 20 molecules solvated in toluene, 6 in acetonitrile and 6 in DMSO (taken from Essex et al. ).
Virtual/moving block concepts are known under the FVB and FMB principles (respectively Fixed Virtual Block -Full Moving Block) in the framework of ERTMS L3  [5]. However, the monitoring of virtual/moving block occupancy requires to track in a more precise way train position, which is under the responsibility of the localization function. Currently, in ERTMS, this function has another goal: to ensure the control of speed limits on-board train and to guarantee train stop prior to dangerous location. It relies on a number of embedded sensors interacting with trackside equipment, the balises, distributed punctually along the block sections. Virtualization techniques have also been extended to balises by using software applications and databases (geographical data and embedded telegrams), both emulating the role initially fulfilled by trackside equipment.
Such investigation raises a number of tricky issues, and requires tackling the interaction complexity between the existing physical components and the new virtual items that are part of the railway control-command system. In this context, the Shift2Rail Joint Undertaking that coordinates the research and innovation investments of the H2020 European program in the railway domain has launched several projects, some of which consider the evolution of ERTMS, such as, X2Rail-1-5, MOVINGRAIL, and PER-FORMINGRAIL. Among different innovative topics pertaining to the ERTMS railway standard, these research projects explore the concepts of virtual blocks, moving blocks, and virtual balises, particularly in terms of safety and performance evaluation. They have resulted in the definition of early and high-level specifications for ERTMS L3 that will be improved and detailed in the next European program supported by Europe's Rail, the successor of Shift2Rail  [7].
The distance between the train front-end and its allocated target point, associated with the permitted speed, is called 'Movement Authority' (MA), while the target point is called 'End of Authority' (EoA). MA data are sent by the trackside sub-system through a radio communication link. ETCS L2 uses the traditional train separation method based on dividing the line into fixed block sections. The block sections are delimited by physical devices (e. g. , track circuits, axle counters). Based on the block occupancy status reported by the Trackside Train Detection devices (TTD), the ETCS Trackside determines the 'train location'. With a TTD-based reporting, the system exactly knows the segment (or the segments, when the train is passing from one block to the next block) in which the train lies.
e. , the potential loss of wagons if a mechanical link is broken. Based on TPR with the associated train integrity data, ETCS L3 no longer needs TTD-related physical equipment. Therefore, both the Full Moving Block (FMB) and the Fixed Virtual block (FVB) principles can be applied. In FVB, although the blocks are fixed, they can be used for implementing an FMB-like operation. Indeed, as they are only represented in a logical form in the trackside databases, they can finely discretize a railway line in small fixed sections by adapting the digital track configuration. Nevertheless, under the FMB operation, theoretically EoA can be issued in any point of the railway line, while under the FVB operation, EoA must correspond to some block extremity. For lines on which a migration towards ETCS L3 is foreseen, a transition phase with mixed traffic (train equipped or not equipped with Moving Block system) is possible by using hybrid implementations.
4) updating the error bound when a PB or a VB is encountered. Concretely, this induces a punctual down jump, in nominal conditions, of the error bound due to the resetting function, while the corresponding residual error is kept. To meet the requirements of our modeling process as discussed above, we chose the UPPAAL model-checking  5 tool employed in several works mentioned in Subsection C. UPPAAL  [56]  allows for handling a network of parameterizable timed and probabilistic automata, hence making it possible to establish modular and configurable models. First, the behavioral modules in UPPAAL have to be established as a number of parameterizable timed automaton templates. Then, the actual behavioral model is generated as a product of timed automaton models instantiated from these template models.
Moreover, the tool includes a number of modelchecking algorithms that allow the evaluation of various types of properties expressed as temporal logic assertions. If some property is not satisfied on the model, a counterexample is generated showing a trace that violates the property, which offers a valuable feature for debugging. Finally, it is worth noting that UPPAAL also offers simulation facilities that can be advantageously used for both modeling and verification phases. Figure  4  and Figure  5  show the main modules that have been developed to describe the behavior of the localization function. Besides these five modules, a module allowing the initialization of PB and VB location on the track, and another allowing the time to be elapsed have been developed. Figure  4  exhibits a high-level view of the global model with the shared data and the synchronization messages among the interacting modules, while Figure  5  exhibits four detailed automata along with the module variables and their role.
C. ANALYSIS PHASE As explained earlier in the paper, our analysis is performed by means of the model-checking facilities offered by UP-PAAL. As a matter of fact, we should indicate that we used both the graphical TA models discussed in the previous subsections, but also a number of textual TA models (". xta" files) and ". q" query files that are generated automatically by means of Python scripts we have developed. Indeed,  since we seek to investigate different track (PB/VB) configurations, while considering various uncertainty levels, we took advantage of the possibility offered by UPPAAL to perform model-checking using command-lines on the basis of TA models and query textual files. It is also worth mentioning that depending on the available computational capabilities, we can easily adapt the levels of accuracy and confidence of the model-checking results (resp.
ACKNOWLEDGMENT This research has received funding from the  Shift2Rail Joint Undertaking (JU)  under the European Union's  Horizon 2020 research and innovation program  under Grant Agreement N. 101015416 (PERFORMINGRAIL). The JU receives support from the  European Union 's  Horizon 2020 research and innovation program  and the Shift2Rail JU members other than the Union. The information and views set out in this document are those of the authors and do not necessarily reflect the official opinion of  Shift2Rail JU. The JU does not guarantee the accuracy of the data included in this article. Neither the JU nor any person acting on the JU's behalf may be held responsible for the use which may be made of the information contained therein.
RELATED WORK Since the introduction of BERT  (Devlin et al. , 2019) , transformer-based models have recently had a major impact on most NLP tasks. Multiple models evolved from it with different design choices, like RoBERTa  (Liu et al. , 2019) , ELECTRA  (Clark et al. , 2020)  and ALBERT  (Lan et al. , 2019). These models are trained on a large amount of data from multiple sources and domains, which means that they are not necessarily prepared for the biomedical domain. In recent years, a great number of resources and NLP tools have been developed specifically for the biomedical domain. For entity extraction, the most popular datasets are BC4CHEMD  (Krallinger et al. , 2015) , B5CDR-Chem  (Li et al. , 2016) , NCBI-Disease  (Dogan et al. , 2014) , BC2GM  (Smith et al. , 2008) , JNLPBA  (Kim et al.
For example,  (Naseem et al. , 2021)  showed that pre-training the ALBERT model on a huge biomedical corpus ensured that the model captured better biomedical contextdependent NER. Results outperform non-specialized models obtaining SOTA results in a lot of datasets. Similar results can be seen in  (raj Kanakarajan et al. , 2021) , where the authors pre-train a biomedical language model using biomedical text and vocabulary with the technique proposed by ELECTRA. Other specialized models based on BERT have been proposed by  (Beltagy et al. , 2019) ,  (Lee et al. , 2020)  and  (Gu et al. , 2020)  and  BioMed-RoBERTa (Gururangan et al. , 2020)  based on RoBERTa. (Michalopoulos et al. , 2020)  propose UmlsBERT, a contextual embedding model that integrates domain knowledge from the Unified Medical Language System (UMLS)  (Bodenreider, 2004) , taking into consideration structured expert domain knowledge.
They show that UmlsBERT can associate different clinical terms with similar meanings in the UMLS knowledge base and create meaningful input embeddings by leveraging information from the semantic type of each word. In our work, we compare the representation of the symptoms found in the clinical case with different contextual embeddings with the goal to find a representation which matches the one provided in the Human Phenotype Ontology (HPO). Ngai et al. (Ngai and Rudzicz, 2022 ) also tackle the problem of finding relevant clinical information, where among the entities they also identify symptoms. In contrast to our work, they only focus on 6 specific diagnoses. Furthermore, their goal is to predict the correct diagnosis and explain these predictions using feature attribution methods, whilst ours is to generate high-quality explanations in natural language for educational purposes, i.
In our work, we propose a method that takes into account the context in which the layperson term is introduced, leading therefore to an accurate mapping to an HPO term. Natural language explanation generation has received a lot of attention in recent years, grounding on the progress of generative models to train specific models for explanations. (Camburu et al. , 2018)  generate explanations by justifying a relation (entailment, contradiction or neutral) for a premise-hypothesis pair by training a Bi-LSTM on their e-SNLI dataset, i. e. , the Stanford Natural Language Inference  (Bowman et al. , 2015)  dataset augmented with an explanation layer which explains the SLNI relations. (Kumar and Talukdar, 2020)  propose to generate short explanations with GPT-2  (Radford et al. , 2019) , learned together with the input by a classifier to improve the final label prediction, using e-SNLI  (Camburu et al.
DATASET To train and evaluate the proposed approach to build natural language explanatory arguments, we rely on the MEDQA dataset  (Jin et al. , 2021) , which contains a set of clinical case descriptions together with a set of possible questions and answers on the correct diagnosis. The questions and their associated answers were collected from the National Medical Board Examination in the USA (USMLE), Mainland China (MCMLE), and Taiwan (TWMLE). In this work, we only focus on the clinical cases and the questions in English (i. e. , USMLE). In total, the MEDQA-USMLE dataset consists of 12,723 unique questions on different topics, ranging from questions like "Which of the following symptoms belongs to schizophrenia?" to questions about the most probable diagnosis, treatment or outcomes for a certain clinical case which is described  (Jin et al.
Annotation of the MEDQA-USMLE Clinical Cases. To annotate the clinical cases from the MEDQA-USMLE dataset, we rely on the labels from the Unified Medical Language System (UMLS) (Bodenreider, 2004) Semantic Types, making it consistent with standard textual annotations in the medical domain  (Campillos-Llanos et al. , 2021; Albright et al. , 2013; Mohan and Li, 2019). In particular, we annotate the following elements in the clinical case descriptions: Sign or Symptom, Finding, No Symptom Occurrence, Population Group, Age Group, Location and Temporal Concept. In this paper, we use only the symptoms, but we addressed a complete annotation to employ these data for future work. Quantifiers defining a symptom have not been annotated (e. g. , we can find "moderate pain", where we only annotate "pain").
The labels Sign or Symptom and No Symptom Occurrence are associated only to the text snippet defining the symptom in a sentence. Findings consist of such information discovered by direct observation or measurement of an organism's attribute or condition. For instance, components in "Her tem-perature is 39. 3°C (102. 8°F), pulse is 104/min, respirations are 24/min, and blood pressure is 135/88 mm Hg". Location refers to the location of a symptom in the human body, and Temporal Concept is used to tag time-related information, including duration and time intervals. Population Group and Age Group highlight information on the age and gender of the patient. To address the annotation process of the MEDQA-USMLE dataset, we first carried out a semi-automatic annotation relying on the UMLS database.
We processed each clinical case through the UMLS database and obtained all the entities detected along their Concept Unique Identifiers (CUI) and their semantic type. The semantic type is then used to disambiguate the entities and generate the pre-annotated files. After the definition of the detailed annotation guidelines (summarized above) in collaboration with clinical doctors, three annotators with a background in computational linguistics carried out the annotation of the 314 clinical cases. To ensure the reliability of the annotation task, the inter-annotator agreement (IAA) has been calculated on an unseen shared subset of 10 clinical cases annotated by four annotators, obtaining a Fleiss' kappa  (Fleiss, 1971 ) of 0. 70 for all of the annotated labels, 0. 61 for Sign or Symptom, 0. 94 for Location, 0.
71 for Population Group, 0. 66 for Finding, 0. 96 for Age Group and 0. 96 for No Symptoms Occurrence. We can see a substantial agreement for Sign or Symptom, Finding and Population Group, and an almost perfect agreement for Location, Age Group and No Symptoms Occurrence. Table  1  reports on the statistics of the final dataset, named MEDQA-USMLE-Symp. 1 The accuracy of the annotations provided by the three annotators has been validated from a medical perspective with a clinical doctor. Of the seven entity labels, only three contain medical vocabulary (Sign or Symptom, Finding, and No Symptom Occurrence) and they have been evaluated by this expert. More specifically, we randomly sampled 10% of the data (i. e. , 30 cases) and we asked the clinician to verify whether the entity was correctly labeled and whether there were any missing or extra words.
EXPERIMENTS In this section, we report on the experimental setup, the obtained results and the error analysis for the symptom detection and symptom alignment methods. Setup. For the symptom detection task, we experimented with different transformer-based Language Models (LMs) such as BERT  (Devlin et al. , 2019) , SciBERT  (Beltagy et al. , 2019) , BioBERT  (Lee et al. , 2020) , PubMedBERT  (Gu et al. , 2020)  and Umls-BERT  (Michalopoulos et al. , 2020)  initialized with their respective pre-trained weights. All the models we employ are specialized in the biomedical domain, with the exception of BERT which will serve us as a baseline. We cast the symptom detection problem as a sequence tagging task. Following the BIOtagging scheme, each token is labeled as either being at the Beginning, Inside or Outside of a component.
This translates into a sequence tagging problem with three labels, i. e. , B-Sign-or-Symptom, I-Sign-or-Symptom and Outside. To fine-tune the LMs, we use the PyTorch implementation of huggingface  (Wolf et al. , 2020) (v4. 18). For BERT, we use the uncased base model with 12 transformer blocks, a hidden size of 768, 12 attention heads, and a learning rate of 2. 5e-5 with Adam optimizer for 3 epochs. The same configuration was used to fine-tune SciBERT BioBERT, PubMedBERT and UmlsBERT. For SciB-ERT, we use both the cased and uncased versions, and for BioBERT we use version 1. 2. Batch size was 8 with a maximum sequence length of 128 subword tokens per input example. Regarding the matching module, we experimented with two different methods to align our detected symptoms with terms in HPO by (i) directly comparing the computed embeddings of the detected symptoms with the embeddings of the terms in HPO, and (ii) by taking into account the context in which the symptoms are detected and applying the same context to every term in HPO.
To align our detected symptoms (in the clinical case) with the equivalent HPO terms, we calculate the cosine distance of each embedding of the HPO terms with respect to the embedding of the detected symptom. In the experimental setting of (i) and (ii), we use the static pre-trained embeddings GloVe 6B as well as BERT, SciBERT, BioBERT and UmlsBERT in the same configurations as in the symptom detection task. For (ii), it is necessary to calculate the context embeddings "on the fly" because each context is unique and depends on the clinical case where it was detected. It is not reasonable to recalculate all HPO term embeddings on the fly for each new context since the ontology contains 10,319 unique terms, so we propose to generate all the HPO terms embedding at once and save them. Therefore, this module takes as input the symptoms detected by the previous module and finds the context 5  of these symptoms in the clinical case.
, 2022)  offers a similar approach to translating layperson terms to medical terms in HPO. However, their work does not take into account the context in which a symptom is found. To the best of our knowledge, this system constitutes the stateof-the-art when translating layperson terms to HPO terms so we decided to compare our proposal with theirs. However, due to the unavailability of their model, we rely on their online demo, which outputs only the top 5 ranking of the HPO terms that are closest to the input symptom. To perform a comparison with our pipeline, we first compute the accuracy of the aligned symptoms using our symptoms alignment module and then replaced it with  (Manzini et al. , 2022)  proposed system (DASH). Results are shown in Table  4. Since a symptom can be composed of several words (e.
g. , "shortness of breath"), we split the symptom and the entire clinical case. symptom into words that we encode by either using each word as an input on Glove  (Pennington et al. , 2014) , or extracting directly from the contextualized models the representation of the symptom by summarizing the hidden states of the last four layers in the model. We then sum the vectors of each word to get an n-gram representation of the symptom. We also explore sentence embeddings, by making use of Sentence-BERT  (Reimers and Gurevych, 2019) , a new model that derives semantically meaningful sentence embeddings (i. e. , semantically similar sentences are close in vector space) that can be compared using cosine similarity. Sentence-BERT can be used with different pretrained models, in this work we focus on the models BERT  (Devlin et al.
, 2019)  , SciBERT  (Beltagy et al. , 2019) , UMLSBERT  (Michalopoulos et al. , 2020)  and S-PubMedBert by  (Deka et al. , 2022). The first represents a competitive baseline in our experiments since it is the SOTA model for comparing sentences crossdomain, while the three latter models are pre-trained on scientific or medical data or both. To tackle both tasks we make use of our annotated dataset (Section 3). The annotations are converted into two datasets, one for each part of the pipeline. The first dataset is used for the symptom detection task, and it is in the CoNLL format for token-wise labels. The second dataset, for the symptom alignment task, is converted into a csv format, where each symptom in the clinical case description and available related knowledge (i. e. , the list of symptoms and their frequencies for each possible diagnosis associated with the case) extracted from HPO are paired.