dataesr/scientific-paragraphs-categorization huggingface dataset:
Methods The diversity of challenges proposed by DREAM (see Available challenges section) and the plethora of languages that have been used in past challenges has led to a fragmentation of the software designed to score submissions. In order to tackle this problem, we chose Python as a glue language. In addition to a clear syntax and the ability to scale-up software, Python can include compiled codes (e. g. , Fortran and C) or call other scripting languages (Perl, R). Besides, languages such as Ruby or MATLAB can also be easily translated to Python, which was an invaluable asset to incorporate many of the earlier challenges that were originally encoded in MATLAB. DREAMTools is an open-source library. Consequently, it can be used directly to evaluate a method against the gold standard of the corresponding challenge, and can also be used as a framework to develop further scoring schemes within the DREAM umbrella or elsewhere.
Note that all challenges follow the same structure with three main functions: to retrieve a template example, to retrieve a gold standard, and to score a prediction. In addition, dreamtools may give access to more functions. For example, the D5C2 challenge 6 has a plot method to compare a prediction with the official submissions, that facilitates inspection of the results, as shown in Figure  2. The figure is generated with an IPython notebook  12  , which is available in the source code repository of DREAMTools.
The downloaded files are stored locally in a standard place (e. g. , /home/user/. config/dreamtools directory under Linux systems). Users can retrieve the location of the templates and gold standards with the dreamtools application as follows: dreamtools --challenge D3C1 --downloadtemplate If sub-challenges are available, a sub-challenge name must be provided. The valid sub-challenge names can be obtained with the --info argument: dreamtools --challenge D8C2 --info DREAMTools for challenge developers: an easily expandable framework DREAMTools library also provides a framework to ease the addition of other challenges by encouraging the usage of a consistent layout. In order to incorporate a new challenge, a developer can look at previous instances and create manually its own tree structure.
dreamtools-layout --challenge D10C10 This command creates a directory named after the challenge nickname. Inside the directory, sub-directories are created to store the templates, gold standards and possibly other data sets. For instance, data to compute p-values may be stored in the data directory. Code related to training data generation could be stored in the generator directory, and so on. In addition to the tree directory, some files are created amongst them a README file that should be filled with information about the challenge (e. g. Synapse identifier, acronym, summary) and a Python script called scoring. py. The basic structure of the scoring script is to provide the same interface for each challenge. In particular, we enforce the implementation of a function to download a template, a function to download a gold standard and a function to score the submission.
Here is an example of such a file, which needs to be filled by the developer: Using the code above, the challenge will be automatically available in the standalone application without extra costs to the developer. The download_template( ) method is not strictly speaking required; it helps a user to create a prediction though and is provided for all challenges. Developers should consider adding tests and documentation in the existing framework. The last release of DREAMTools contains a test suite (collection of test cases used to check the software) with a code coverage higher than 80%; it guarantees that the DREAMTools functionalities (especially the scoring functions) do work as expected.
Installation and dependencies The DREAMTools source code is available on GitHub. It can be downloaded and installed as follows: > git clone git@github. com:dreamtools/ dreamtools. git > cd dreamtools > python setup. py install The source code gives access to the latest version but releases are also provided on the Python repository website (Pypi) and consequently installation is also possible using the pip tool: > pip install dreamtools DREAMTools relies on established scientific libraries such as Pandas  13  for the data mangling, SciKit-learn  14  (e. g. , ROC curves) and more generally NumPy/SciPy 15 for statistical analysis. Those libraries are recognized in the scientific community and there is an ample set of online resources that cover installation procedure. Yet, the compilation of these libraries may take a while or lead to compilation errors on some machine configuration.
Consequently, we also provide a pre-compiled version of DREAMTools within the bio channel (http://bioconda. github. io), which is a channel of Anaconda (https://www. continuum. io/downloads). The latter provides about 400 scientific packages including Numpy and Pandas aforementioned. Finally, note that in order to keep a light-weight package, we store large data files in Synapse. DREAMTools will download files automatically on request. The download is achieved using the Python Synapse client (also available for the R language).
Scoring metric: As in the D7C1 challenge (See D7C1 section), submissions were scored based on a combination of their parameter and prediction distances (Equation  13 ). The parameter distance was computed as the average log ratio of the estimated and true parameter values (Equation  11 ). The prediction distance was computed as the average Euclidean distance between the estimated and true in silico phenotypes, scaled by their variances. This scoring function is not included in DREAMTools. This scoring function is implemented in MATLAB, and is available open-source at GitHub (https:// github. com/CovertLab/wholecell). A complete working example of this scoring function, including the gold standard, is available at Synapse (https://www. synapse. org/#!Synapse: syn1876068/wiki/232963).
DREAM8. 5 D8dot5C1: Rheumatoid Arthritis Responder Description: The goal of this challenge was to use a crowd-based competition framework to develop a validated molecular predictor of anti-TNF response in Rheumatoid Arthritis (RA). We used the whole genome SNP data derived from two cohorts: 2,706 anti-TNF treated RA patients combined across 13 collections of European ancestry  32  , and 591 patients in the CORRONA CERTAIN study  33. Treatment efficacy was measured using the absolute change in disease activity score in 28 joints 34 (DAS28) following 3-6 months of anti-TNF treatment. The challenge was devised into two components. Sub-challenge 1: predict treatment response as measured by the change in disease activity score (DAS28) in response to anti-TNF therapy. Sub-Challenge 2: identify poor responders as defined by EULAR 35 criteria for non-response (20% of the study population).
Scoring metric: In sub-challenge 1, the scoring methods are the AUROC and balanced accuracy (BAC), defined in Section 1. 1. In sub-challenge 2 and 3, the scoring methods are the concordance index (CI) and Pearson correlation coefficient (see Section 1. 2. 5). The Pearson correlation coefficient is used to measure correlation between predictions of remission duration and actual remission duration. In those sub-challenges, the final rank is the average of the two intermediate ranks. D9C3: Alzheimer's Disease Big Data Description: The goal of the Alzheimer's Disease (AD) challenge was to identify accurate predictive biomarkers that can be used to improve AD diagnosis and treatment. In order to build predictive models, participants were given genetics and brain imaging data in combination with cognitive assessments, biomarkers and demographic information from cohorts ranging from Cognitively Normal (CN) to Mild Cognitively Impaired (MCI) to individuals with Alzheimer's Disease (AD).
DREAM10 D10C1: ALS Stratification Prize4Life Description: This challenge is a follow-up on to the DREAM 7 ALS Prize 4 Life Challenge (see D7C3 for details). It focuses on predicting the progression and survival of ALS patients. One objective of the challenge is to leverage the PRO-ACT database of more than 8,000 cases as the challenge training set. The challenge will include several unpublished data sets to be used for model validation. Scoring metric: This is an on-going challenge. The scoringmetric has not been released yet (September 2015). D10C2: ICGC-TCGA DREAM Somatic Mutation Calling Tumor Heterogeneity (SMC-Het) Description: This challenge is a follow-up on to D9C4 challenge (somatic mutation calling). This challenge's focus is to identify the best subclonal reconstruction algorithms and to identify the conditions that affect their performance.
Conclusions The organization of a collaborative competition such as the DREAM challenges is a complex task that starts by identifying a currently important and unresolved scientific problem, acquiring relevant data sets, engaging a community of participants, and implementing an appropriate scoring methodology. Participants can submit their solutions (e. g. , predictive models or predictions) which are then scored and ranked, and the results are shown on a public leaderboard. Once the challenge is closed, those leaderboards can be used as a benchmark for further development of methods. To promote scientific reproducibility as well as post-challenge use, DREAM provides via Sage Bionetwork's Synapse platform the resources to help researchers access data and leaderboards of previous challenges.
In this paper, we presented DREAMTools to provide a uniform framework where researchers can easily assess and compare new methods against benchmarks. DREAMTools gathers most of the scoring functions used in previous DREAM challenges. DREAM-Tools uses Python as a glue language known for its flexibility and ability to call other languages. Currently, about 80% of the closed challenges are available. The remaining challenges are either in the process of being included or hosted on external websites. Future versions of DREAMTools will aim at making available as many closed challenges as possible including newly closed challenges. DREAMTools will help researchers who wish to test their algorithms against existing benchmarks. Indeed, templates can be downloaded and used to create predictions, which can then be tested.
The gold standards are also available together with the relevant scoring functions. Since DREAMTools makes use of an object oriented approach, it will ease the integration of future challenges thereby facilitating scoring in upcoming challenges. DREAMTools can also be used as a place to retrieve metadata and information about a challenge. DREAMTools can be used as a standalone application or as a library making it a useful tools to be included in other software or pipelines. Developers who use the proposed layout will not need to change anything regarding the standalone application that will automatically recognize the challenge. In summary, we hope that DREAMTools will be a useful tool for researchers interested in benchmarking their methods against the state-of-the-art as defined by previous DREAM challenges, and to those developing new collaborative competitions within DREAM or elsewhere.
( , ) ( , , , ) d d i j i j i j c G R h g g r r N N < = = -∑ c -index ( 1) where  > < ∨ < >    = =  > > ∨ < <   1, if ( & ) ( & ) ( , , , ) 0. 5, if ( ) 0, if ( & ) ( & ) i j i j i j i j i j i j i j i j i j i j i j g g r r g g r r h g g r r g g g g r r g g r r (40)   In the previous formulation of the c-index, the variance of the gold standard data set is not taken into account. A probabilistic c-index (denoted pc-index) was introduced in Costello et al. 27  , Bansal et al. 28  and calculated as follows: where  This paper describes DREAMTools, which is a package that enables scoring of submissions for DREAM challenges. The paper is well written and this is certainly a useful tool. Benchmarking is essential in all aspects of tool or algorithm development, and the package described here will be helpful for evaluating the challenges.
Software availability Software releases are available from the Pypi website https://pypi. python. org/pypi/dreamtools Latest source code is hosted on GitHub website http://github. com/dreamtools/dreamtools Archived source code at the time of publication http://www. dx. doi. org/10. 5281/zenodo. 47949  37 g report and feature requests https://github. com/dreamtools/dreamtools/issues Documentation http://dreamtools. readthedocs. org License BSD 3-clause license ("BSD NEW" or "BSD Simplified").
Small-angle scattering measurement and analysis SAXS experiments were carried out at the European Molecular Biology Laboratory beamlines X33 and P12 of DORIS and PETRAIII storage rings respectively, using the X-ray wavelengths of 1. 54 and 1. 24 Å and a sample-to-detector distance of 2. 7 and 3. 0 m, respectively  (38). The scattering profiles measured at the beamline X33 covered a momentum transfer range of 0. 0087 < s < 0. 60 Å-1 , and those obtained at the beamline P12 a range of 0. 0047 < s < 0. 5 Å-1. SAXS data on free PCNA were collected for two series corresponding to monomeric protein concentrations of 1. 3, 3. 0, 5. 1, 10. 6, 12. 7, 42. 4 mg/ml, and 1. 3, 5. 1, 10. 2, 20 mg/ml at X33 and P12 beamlines, respectively. SAXS data of the PCNA-p15 complex were collected at beamline X33 from multiple titration point mixtures at 25 • C.
The forward scattering intensity, I(0), and the radius of gyration, R g , were evaluated using Guinier's approximation  (40) , assuming that at very small angles (s < 1. 3/R g ), the intensity can be well represented as I(s) = I(0) exp(-(sR g ) 2 /3). The P(r) distribution function of free PCNA was calculated by indirect Fourier Transform using GNOM  (41) , applying a momentum transfer range of 0. 0068 < s < 0. 60 Å-1. Ten independent ab initio reconstructions of PCNA were built starting from a sphere of D max = 94. 1 Å with the program DAMMIN  (42) , and the reported structure was derived from their average. The capacity of several programs to predict the theoretical SAXS curve from the 3D structure of PCNA was evaluated. In addition to WAXSiS (see below), we tested CRYSOL  (43) , AXES  (44) , FOXS  (45) , AquaSAXS  (46)  and SASTBX  (47).
We first removed translations and rotations for the whole molecule and then calculated the rotational correlation function with GROMACS utilities trjconv and rotacf respectively  (48, 60). For each residue, three C(t) functions were computed, corresponding to each of the protomers of PCNA, and subsequently averaged. The final C(t) was fitted to a two-parameter model:  (61)  with the program Scipy  (62). S 2 + (1 -S 2 )e -t / τ c.
Short explicit-solvent MD simulations for WAXSiS calculations The SAXS predictions for the structural ensembles were based on a short explicit-solvent MD simulation conducted for each conformation of the different ensembles. Accordingly, each structure of the ensembles of p15, PCNA and PCNA-p15 complexes were pre-processed with Rosetta3. 5 fixbb module  (64)  to alleviate side-chain clashes. Each structure was placed in a simulation box of a dodecahedron, solvated with the TIP3P water model (  57 ), and neutralized with counter ions. After an energy minimization, the backbone atoms were constrained with a force constant of 100 kJ mol -1 nm -2. Each structure was simulated for 200 ps using Gromacs 4. 6  (48)  and the AMBER99sb-ILDN (56) force-field. NPT conditions were maintained using velocity rescaling  (55)  to 300 K and Berendsen barostats  (54).
This reconstruction displayed the expected ring-shape of the protein and precisely enveloped the crystallographic structure of PCNA (PDB code: 4D2G)  (27). The high quality of the reconstruction reflects the wealth of information encoded in the 1D scattering pattern of PCNA. Before analyzing the SAXS curves of the PCNA-p15 mixtures, we assessed how accurately we could reproduce the experimental SAXS profile of human PCNA by backcalculating the scattering intensity from the atomic coordinates of its crystal structure, using available predictors with default parameters (Supplementary Figure  S1 ). The various methods vary in the calculation of interatomic distances, in the treatment of the hydration layer, and in the calculation of the background contrast  (72). WAXSiS  (65, 66)  was the only predictor that accurately described the features present in the experimental curve.
WAXSiS analysis of the intrinsically discorded protein p15 The local structural information contained in Residual Dipolar Couplings (RDCs) and the global features reported by SAXS were recently combined to study the intrinsic conformational flexibility of p15 in solution  (37). Here, we tested the performance of WAXSiS, which accounts for the explicit dynamics of the hydration layer, in predicting the SAXS properties of p15. Flexible-Meccano (FM) was used to generate an ensemble of 800 conformations of p15 using the same structural bias as that derived from the RDC analysis  (37). The theoretical SAXS profiles for these conformations were computed using both WAXSiS and CRYSOL  (43) , the latter being the software most commonly used to compute SAXS properties of disordered proteins  (74, 75).
1 M (27). For K d values much lower than the total concentration of PCNA (i. e. K d < 1 M), most of the p15 molecules are bound to PCNA, thereby explaining the lack of sensitivity of 2 to the K d. However, at such high affinities there was poor agreement with the experimental data, presumably because the model does not account for the contribution from unbound p15 and PCNA. Similarly, for low-affinity scenarios, the species distribution differed greatly from the experimental one, with a clear underestimation of the high stoichiometry forms of the complex and an overestimation of the free forms of the two proteins. A correct description of relative populations of the five species was achieved only in the intermediate range where the individual and collective 2 minima were found. In the optimal range of K d values, our model encoded the polydispersity of the system and was able to describe all SAXS curves of the experimental dataset without systematic deviations in the residuals (Figure  5C ).
In particular, our model captured the smearing of the SAXS features upon addition of increasing amounts of p15. At the optimal K d , the averaged collective 2 value of the com-  plete dataset was 0. 95, and the individual collective 2 values varied between 0. 91 and 0. 98. All titration points at a K d of 30 M contained non-negligible populations of several species, thereby highlighting the complexity of the system under study (Figure  5D ). The robustness of the derived model was further evaluated by calculating the optimal amount of p15 able to reproduce the experimental curves at the optimal K d. For each curve, the minimum found was in excellent agreement with the amount of p15 experimentally added (Figure  5E ). Although some curves displayed a narrow collective 2 minimum, others showed very flat and broad profiles, thereby indicating their reduced discriminatory capacity.
p15 crosses the PCNA ring In all the models built for the structural analysis of the SAXS data for the PCNA-p15 complex, the disordered N-terminal region of p15 was modeled to cross the ring and emerge on the back face. This model, which we call in-model (see Figure  6A ), is supported by previous NMR and X-ray crystallography data  (27). However, this model was not unambiguously proven (see above), and the possibility that the N-terminus of p15 reverses its direction towards the front face of PCNA, which we call out-model (see Figure  6A ), could not be discarded. We used the SAXS dataset to study this structural feature. The K d screening analysis of the SAXS data was repeated but using the averaged curves computed from 800 models for each of the three complexes (3:1, 3:2 and 3:3), where the N-terminus of p15 sampled the front face of PCNA.
While MD simulations probe fast dynamics on globular proteins, the conformational exploration of disordered proteins such as p15 and its complexes requires other approaches. Applying integrative strategies, we used previous non-overlapping NMR and crystallographic information to build accurate models for these highly flexible species. These models allowed us to simultaneously reproduce the SAXS dataset probing all species present in the mixture, thereby substantiating the quality of the models constructed. Finally, the use of atomistic models to describe transient equilibria can be achieved only when accurate approaches for computing scattering profiles from 3D models are available. Our study demonstrates the importance of properly describing protein and hydration dynamics when reproducing SAXS data.
In line with this hypothesis, in a recent SAXS/SANS study using three mutants of the green fluorescent protein with very different net charges (+36, -6 and -29), it was demonstrated that the hydration shell is locally denser around acidic residues than around basic and hydrophobic ones  (81). Our study also shows that, for p15, WAXSiS performs as well as CRYSOL, the latter using an implicit homogeneous description of the water layer. The origin of the similarity of the resulting averaged curves for proteins with extensive solvent-accessible surfaces remains unclear. However, the observation that individual conformations provide different theoretical curves depending on the program used suggests that the effect of the hydration layer is smeared when averaging hundreds of theoretical scattering profiles.
At the concentrations normally used in SAXS experiments, the K d value for stronger affinities cannot be precisely determined and, in this scenario, SAXS can detect only the formation of the complex. The continuous improvement of SAXS beam lines and commercial detectors will decrease the amount of protein required to achieve a satisfactory signal to noise ratio, thus allowing the quantification of stronger biomolecular interactions by SAXS. Interestingly, individual curves presented distinct sensitivity to K d depending on the species composition. Consequently, the simultaneous fitting of multiple SAXS curves facilitates the determination of the optimal K d. Our approach could also potentially detect and quantify positive or negative cooperativity effects between binding sites. Analytical equations for distinct allosteric models can be derived and tested using a titration SAXS dataset.
Conversely, structural models where p15 does not cross the ring but places its N-and C-termini on the front face of PCNA presented a lower agreement with the SAXS dataset. NMR chemical shift perturbation experiments and the crystallographic structure of PCNA with an enlarged version of p15 PIP region already suggested that p15 crossed the ring. Our SAXS data analysis now validates these previous observations. This interaction mode is coherent with the enhanced protection of p15 from proteasomal degradation in the presence of PCNA and suggests that the disordered chain remains partially encircled by PCNA in the unbound state due to the entropic cost of unthreading p15 chains  (27). Additionally, this model provides solid bases to hypothesize that p15 act as a drag that slows down PCNA processivity along DNA facilitating actions of the repair machinery.
FUNDING  SPIN-HD -ANR Chaires  d'Excellence 2011; ATIP-Avenir;  French Infrastructure for Integrated Structural Biology Nucleic Acids Research , 2017, Vol. 45, No. 3 1513 [FRISBI -ANR-10-INSB-05-01  to P. B. , CTQ2014-56966-R] from the  Spanish Ministerio de Economía y Competitividad  (to F. J. B. );  Deutsche Forschungsgemeinschaft  [ HU 1971-1/1  to P. C. and J. S. H. ]. Funding for open access charge: chemREPEAT-ERC-Consolidator.