dataesr/scientific-paragraphs-categorization huggingface dataset:
Introduction Research in the biomedical domain is characterized by an exponential growth of the published scientific materials, e. g. , articles, patents, datasets, technical reports. Handling such a scale of information is a huge challenge, for the purpose of which multiple initiatives have been launched in order to organize biomedical knowledge formally. The use of ontologies is one of the most promising key aspects in this direction that has attracted a lot of interest  [1]. An ontology is a complex formal structure that can be decomposed into a set of logical axioms that state different relations between formal concepts. Together the axioms model the state of affairs in a domain. With the advances in Description Logics (DL), the process of designing, implementing and maintaining large-scale ontologies has been considerably facilitated  [2].
Patterns are mined through the dynamic alignment of relevant sentences that mention the interaction. Both the precision and the recall of the system reach 80%. Xu and Wang  [18]  use simple pattern-based approach to extract drug-disease relation instances from MEDLINE abstracts. The patterns are not complicated (e. g. "DRUGinduced DISEASE"), thus the approach exhibits a typical bias towards high precision at the expense of low recall: 90% precision and 13% recall. However, the majority of extracted instances do not yet exist in a structured way in biomedical databases, which proves the usefulness of the approach. The majority of work on pattern-based relation extraction rely on hand-crafted templates whose construction is a laborious task. In some cases the patterns are built automatically, nevertheless the approach lacks the ability to extract relations that are not explicitly stated in the text, i.
The task is to build a relation identification model that is able to recognize a certain SNOMED CT relation in text. The textual data used in  [24]  are the clinical phrases from SNOMED CT that describe the concept in natural language (e. g. , "acute gastric ulcer with perforation"). A separate machine learning classifier is trained for every typed version of every SNOMED CT relation, e. g. , find-ing_site(disorder, body_structure) and finding_site(finding, body_structure) yield two separate models. There are three main drawbacks of this work. Firstly, it uses only the data from SNOMED CT clinical phrases, which are formulated in a controlled language. However, the ultimate goal of the system is to be able to identify relations in various medical texts for new biomedical concepts, and these texts are not written in a controlled language.
The first three experiments give us insights about all major parameters of the relation extraction process that we outlined in the abstract, i. e. , the source of the input corpus, its size, the number of distinct relations and feature representation. Table  5  summarizes the main results of the first three experiments. It shows that with the most important lexical features (character 3-grams) and appropriate semantic types, we achieved F-score larger than 90% on all datasets using 10-fold cross-validation for evaluation. Furthermore, free texts extracted from the Web proved to give a competitive result. One may assume that it is due to In all experiments Support Vector Machines are used. the larger data size of the corpus (9,292 v. s. 1,357 or 424 instances). The fourth experiment shows that our approach can generate formal definitions with a precision that can reach up to 81%, as it was defined by Equation  3.
Experiment 2: Semantic features and the number of relations In this experiment we aimed at expanding the relation set from just three SNOMED CT roles to a larger set of diverge, semantically rich relations. The process of aligning MeSH and SNOMED CT definitions provided a dataset of moderate size even for the most frequent relations, and the number of relation instances that we are able to extract via the alignment for less populated relations is insufficient for the automatic learning. Thus we switched to another corpus of definitions, namely SemRep (Section 'Corpora'). The SemRep Gold Standard corpus contains both textual definitions and a set of relations and consists of 1,357 relation instances. In addition, we introduced semantic features, reducing every argument concept to its UMLS semantic type (Section 'Feature engineering').
This is translated into comparing the first and the second line of Table  6. As the results show, the lexical features cannot be neglected as they do offer important contribution in the cases where a relatively large number of relations is considered, e. g. , 10 or more. The second question that we would like to address is whether semantic types are generally effective learning features, or the performance boost was specific to SemRep dataset. For this purpose, we have tried adding semantic features to the Experiment 1, extending the feature representation of MeSH instances with the same semantic types from UMLS. The resulting F-measure of 73. 9% is a bit lower than the original one: the semantic types slightly deteriorated the performance, serving as noise to the classifier. However, adding semantic types of different origin had an opposite effect: upper level concepts of SNOMED CT taken as types gave an F-measure of 99.
1% for the classification of the three SNOMED CT relations compared to 74. 5% with lexical features only. From this we can conclude that semantic types are of great value for the classification, given that their modeling is consistent with the modeling of the relations. The size of each dataset is specified by the absolute number of instances and by the percentage of instances covered by the respective set of relations. The table reports F-Measure for two settings: including semantic types in the feature space, and excluding them.
Experiment 3: Web corpora and the dataset size The third experiment introduces Web-based corpora as the source of textual definitions from which formal definitions and relation instances are built. We used two different corpora, WIKI and D4D, automatically extracted from the Web and analyzed the impact of corpora for our task. The results are reported in Table  7. In accordance to the findings of the previous experiment, semantic types (the SNOMED CT types) improved the results notably, achieving an F-measure of 100% on both corpora. This is largely due to the disjointness of argument types for the three target roles from SNOMED CT. Moreover, Table  7  shows that the use of D4D always improved the final performance compared to the use of WIKI. An explanation for this can be that the sentences from D4D are filtered beforehand by syntactical patterns  [8].
Overall, WIKI and D4D corpora are much larger that SemRrep and MeSH corpora (Section 'Corpora'). In fact, having a dataset of more than 60 thousand sentences, it is possible to plot the performance of the relation classification as a function of the dataset size and to analyze the impact that the number of learning instances has on the learning. Using the settings of Experiment 3, we discuss this impact in Section 'Does data size matter?'.
Considering merely the remaining 1,407 concepts, we achieved a precision of 81. 3%. We further investigated if formal reasoning can be helpful for our task. For this, instead of using InfRB as done in all other experiments, we use ExpRB, as given in Table  2 , to construct training data. Note that ExpRB is a proper subset of InfRB, so less training data can be obtained in this setting compared to using InfRB. As a result, the average precision of definitions decreased to 61. 4% from 66. 5% for the 1,721 concepts. The 5. 1 p. p. precision difference shows that the dataset automatically enriched by formal reasoning (the use of InfRB) improves the system's quality in predicting formal definitions of SNOMED CT concepts. This is because the inferred relationship base brings more training examples to boost the whole learning procedure.
Do corpora matter? At a first glance, for extracting formal definitions from texts, the textual data should have a big effect on the system. For example, MeSH contains manually edited textual definitions for concepts, which should be of an obvious advantage for this task. However, from Experiment 3 and 4, the experiments on automatically extracted web corpora WIKI and D4D, we can still achieve formal definitions of a good quality based on both F-score and the definition precision. Set aside MeSH, WIKI and SemRep corpora, which are manually curated, this might be explained by two facts: (1) the alignment process, described in Section 'Alignment', does ensure that the aligned examples are descriptive of the target relation set, and, (2) the D4D plugin prioritizes the definitional sentences from the Web, giving higher scores to trustworthy resources.
Do ML algorithms matter? The choice of the machine learning algorithm is secondary for the task at hand. While in  [10]  Support Vector Machines consistently dominate over Logistic Regression, Random Forests and Multinomial Naïve Bayes, the difference in performance rate values are partly due to the small size of the dataset (424 instances). In several cases, SVM have been shown to match or dominate the performance of competitive techniques for major text mining exercises. For example, in  [38]  it is shown that SVM, Naïve Bayes and k-Nearest Neighbor are among the best performing classifiers. A later work by Mladenić et al. [39]  evaluates several classifiers on the Reuters news datasets showing that SVM tend to outperform other algorithms including Naïve Bayes. However, several studies on the use of Big Data suggest that the performance rates of various algorithms tend to converge given considerable amount of instances.
In  [40] , in the task of word sense disambiguation, the performance of several algorithms (Naïve Bayes, Winnow, Perceptron) gradually increases and eventually converges as the training dataset size increases. Colas and Brazdil [41]  conclude that, although SVM are among the most effective algorithms in the area of text processing, various other algorithms, e. g. , Naïve Bayes or k-NN, achieve comparable performance. Hence, in our work we shift the focus towards feature engineering and use only SVM.
Does data size matter? For the experiments introduced in Section 'Results', the corpora data sizes moved from 424 to 9,292, on which we achieved consistent results as analyzed above. Indeed, we conducted similar conclusions by sampling smaller data sets from the large corpora WIKI and D4D: (a) The performance increased with the increase of the data size if only lexical features were used: on D4D, F-score ranged from 59% (data size 300-500) to 63% (data size 750-1,000), and to 70% (data size 1,500-3,100). On WIKI, F-score ranged from 48% (data size 600-700) to 57% (data size 800-3,000), and to 67% (data size 4,000-6,000). (b) The performance stayed relatively constant if semantic features were involved too, no matter the size of the data: F-score fell into the range of 98% to 100% on all the sampled smaller data sets.
arff files and can be found here: http://www. db-net. aueb. gr/gbt/ download. html. For the implementation of the machine learning approaches and the representation of the training and test instances in machine format, we used Weka (Waikato Environment for Knowledge Analysis), which can be obtained from http://www. cs. waikato. ac. nz/ml/ weka/. The implementation of the full pipeline of formal definition generation is published online as a GitHub project: https://github. com/alifahsyamsiyah/learningDL.
7 mm standard cell (optimal for 9. 5 keV photons) suffices for most data collection requirements between 6 and 20 keV. The stainless steel pod that houses the capillary has been designed to be easily changeable in case of a leak or irreversible capillary fouling. The universal adaptors of the pod allow for the connection of multiple sample delivery devices, enabling both manual and automatic sample loading. Typically, samples (with a volume between 20 and 35 ml) are delivered to the capillary for continuous-flow SAXS measurements using a third-generation EMBL/ESRF sample changer (see below). Continuous flow significantly reduces the effects of X-ray radiation damage  (Lipfert et al. , 2006). Automated loading of very low sample volumes (5 ml) for static sample data collection is also possible as well as manual sample loading via syringe or using alternative sample delivery systems, for example, the eluent from size-exclusion columns used for component separation experiments (see below).
Samples are loaded using built-in protocols that drive the sample storage plate to the fixedposition needle as opposed to moving the needle to the sample plate. This design ensures that the length of the tubing between the needle entry point and the measurement cell can be kept constant and as short as possible. We have found these conditions to be critical for consistent and reproducible sample loading into the SAXS capillary. The dead volume of the system is of the order of 1-2 ml, depending on sample adsorption. The precision pump can be programmed to perform continuous flow or static sample SAXS measurements and, if necessary, can be driven in reverse to recover samples after data acquisition. The accurate positioning of the sample in the capillary is monitored by a camera and automated through image processing.
Data acquisition is triggered by the beamline meta server software (BMS) and, for most standard operations, the sample undergoes continuous flow during the measurement. The sample flow rate is determined automatically by the amount of sample and by the total exposure time, typically divided into 20 Â 50 ms exposures for a total of 1 s. After the SAXS data have been collected, the sample is either flushed to waste, or recovered, and a subsequent automated washing cycle cleans the sample capillary. The wash cycle consists of successive rapid-flow detergent-ethanol-water and MilliQwater flushes, followed by high-pressure drying with filtered air. Capillary drying is monitored by a camera using image processing techniques to prevent residual water from the wash cycle contaminating the subsequent samples.
4 ml min À1 ) using an isocratic pump (Viscotek VE-1122). Although the sample is diluted in the chromatography column, meaningful SAXS data can be acquired from the sample components as they elute from the column matrix by connecting the output to the sample capillary. The SAXS data collected before the solutes flow out of the column can be used for matched buffer subtraction. At P12, the SEC-SAXS experiments can additionally be combined with a triple detector array (TDA 305, Malvern Instruments) consisting of right-angle laser light scattering (RALLS), refractive index (RI) and ultraviolet (UV) detectors. This output can then be correlated with the SAXS forward scattering intensities [I(0)] to cross-validate the molecular weights (MW) of the eluting species. 3. 3. 1. SEC-SAXS example: ovalbumin.
The advantages offered by SEC-SAXS are illustrated in Fig. 4 , where ovalbumin data obtained from batch-mode SAXS and SEC-SAXS are compared. Chicken ovalbumin was acquired from GE Healthcare. For SEC-SAXS/TDA analysis the samples were dissolved in 100 mM Tris pH 7. 5, 150 mM NaCl, 5% glycerol at a concentration of 15 mg ml À1. The sample was filtered through a 0. 2 mm centrifugal filter unit (Millipore) prior to loading on to the Superdex 200 10/300 column. The final solute concentrations were determined by measuring the absorbance at 280 nm (Thermo Scientific NanoDrop ND-1000) and using Experimental SAXS curves of ovalbumin collected with the sample changer (top) and with online SEC (bottom; offset by one order of magnitude). The online SEC curve is noisier than the sample changer curve because of the much lower solute concentration, and the higher angle data (for s > 1.
5 nm À1 ) have been re-binned with a binning factor of five for better visualization. The grey lines represent in both cases the fits by the curve computed from the atomic structure. Inset: ab initio model of the ovalbumin monomer (orange) built from the SEC-SAXS experimental curve; the aligned atomic structure is represented in blue for comparison. calculated extinction coefficients expressed as " 1% (10 mg ml À1 ) from ProtParam: " ovalbumin = 7. 32. A clear misfit is observed between the experimental scattering obtained in batch mode and the scattering calculated using CRYSOL  (Svergun et al. , 1995)  from the crystallographic monomer of the protein [Protein Data Bank (PDB) code 1ova  (Stein et al. , 1991) ], yielding a discrepancy 2 = 5. 6. Notably, at low angles the experimental scattering intensity is systematically higher than that of the model, and the experimental radius of gyration R g significantly exceeds the model one (3 nm versus 2.
The automated averaging of the individual SAXS data frames spanning the peak produces a final SAXS profile yielding an R g value of 2. 5 nm, in agreement with the R g value computed from the monomer atomic structure. Furthermore, the calculated model intensity from the crystal structure (PDB ID: 1ova) shows a significant improvement in the fit against the experimental data ( 2 = 2. 02). A low-resolution ab initio bead model automatically generated by the pipeline using DAMMIF models for initial search and DAMMIN for refinement  (Svergun, 1999; Kozin & Svergun, 2001; Volkov & Svergun, 2003; Franke & Svergun, 2009)  aligns well with the previously solved crystal structure (Fig. 4 ). This example illustrates the utility of SEC-SAXS with TDA for extracting individual component scattering data from polydisperse samples.
Users may also bring their own customized sample environments; recently, devices such as a hexapod for scanning SAXS applications and a heating stage (Linkham, Tadworth, UK) to reach extreme temperatures were successfully utilized at P12 by users from HZG. 4. 1. 1. Time-resolved SAXS environments. In a timeresolved experiment, SAXS data are collected from the intermediate states of a system as they shift toward a new equilibrium position. Controlled triggering of a reaction can be realized in several ways, for example by mixing two solutions, by changing the temperature or the pressure, or by directly activating a system using a laser-controlled photo reaction  (No ¨lting, 2005; Callender & Dyer, 2006). The timescales for the experiments are broad, spanning from the submillisecond range for rapid protein folding events  (Kathuria et al.
Owing to the high flux and the low instrument background of P12, sufficient scattered photons can be collected in 30-50 ms to obtain an interpretable SAXS profile. Although the data are noisy, these measurements not only allow the radius of gyration R g to be monitored but also permit more sophisticated analyses (an example of a typical 50 ms frame is displayed in Fig. 6 ). The timescale of the stopped-flow experiments fits with the repetition rate (30 Hz) of the Pilatus 2M detector, and kinetic reactions can be followed with a time resolution of 33 ms. Several P12 user groups have employed the stopped-flow system for the analysis of their samples, including the analysis of the tRNA-modifying enzyme complex MnmE/MnmG  (Fislage et al. , 2014). A disadvantage of the stopped-flow apparatus is in its significant sample consumption.
) can be very bulky; thus converting the lab-on-a-chip concept into what could be described as a 'chip-in-a-lab'. Here, we present a novel device, a centrifugal-force-driven microfluidic screening disc for solution SAXS measurements. The SAXS disc forgoes the need for any external pump or valve attachments during SAXS data collection (Ducree et al. , 2007) and offers a wide range of experimental possibilities for the structural biology user community. 4. 2. 1. SAXS disc operation and sample handling. The SAXS disc is a flat compact-disc-sized sample environment used for the handling and mixing of microlitre quantities of sample. The disc is divided into identical and independent sextants that each constitute an individual sample screen (Fig. 7 ). Each sextant has three small reservoirs for depositing solutions: 2 ml of sample, 3 ml of mixing buffer and 3 ml of screening buffer.
SAXS disc case study: unfolding of ribonuclease A. SAXS data were obtained from urea-induced denaturation states of ribonuclease A (RNase A) using the SAXS disc. The unfolding of RNase A has been well characterized  (Pace et al. , 1990; Sosnick & Trewhella, 1992 ) and the protein is thus a suitable standard to test the efficiency of the disc, both as a tool for mixing samples and as a sample environment for SAXS data collection. The appropriate reservoirs were loaded with 2 ml of unfolded RNase A solution (25 mg ml À1 , in 6 M urea, formate 30 mM, pH 3. 5), 3 ml of 6 M of a matched urea buffer (formate 30 mM, pH 3. 5) and 3 ml of a refolding buffer Figure  8  Schematic diagram of the sample environment for the microfluidic disc. The disc (blue) is mounted on a motorized disc holder that can rotate the disc and translate it in vertical and horizontal directions for fine alignment.
At low urea concentrations, the experimental intensities both from the SAXS disc measurements and from the sample changer fit the intensities calculated from the crystal structure of RNase A as shown in Fig. 8  [PDB code 1fs3  (Chatani et al. , 2002) , 2 = 0. 61 and 0. 88 for SAXS disc and sample changer measurements, respectively]. The consistency between the SAXS data collected from the SAXS disc versus those data collected from the sample changer shows that the integrated microfluidics of the disc function as an efficient mixing platform to deliver samples in the correct ratios to the measurement cells. Although the SAXS disc data are of somewhat reduced quality compared with those collected with the sample changer, the disc data are still very much interpretable. Importantly, the SAXS disc affords an opportunity to rapidly screen multiple sample conditions using very low volumes.
Automation of SAXS experiment and data analysis Efficient work with the robotic sample changer at P12 requires full automation of the SAXS experiment and its analysis. Synchronized and automated sample loading, data collection, reduction, processing, analysis and modelling protocols are needed to improve the overall quality assurance and simplify user operations. A typical SAXS experiment utilizing the invacuum setup and the sample-changer robot typically requires a repetition of a well defined sequence of data collection steps. Each measurement comprises the following steps: (a) cleaning of the capillary where the specimen is exposed to X-rays; (b) transport of the specimen into the capillary; (c) opening of the shutter; (d) flow of the specimen through the capillary to minimize radiation damage; (e) collection of the scattering pattern; (f ) recording of monitor values, such as transmitted beam intensity, capillary temperature etc.
Automation of the SAXS experiments with a samplechanger robot Automated collection of SAXS data requires synchronized and coordinated actions of all beamline components. Reliable communication between the devices at P12 is based on the threefold integrated networking environment  (Bartkiewicz & Duval, 2007). This environment provides a programming interface to implement servers controlling the hardware devices over the network. These devices include the automatic sample changer, a beam shutter, an X-ray detector and various monitors such as those for transmitted beam intensity, storage ring current, pressure inside the fly tube and temperature of the capillary etc. To automate the measurement, the sequence of actions needs to run autonomously. This task is executed by the BMS, a key component of the automated SAXS experiment, first implemented at the EMBL X33 beamline  (Franke et al.