dataesr/scientific-paragraphs-categorization huggingface dataset:
Therefore, a comprehensive validation analysis of different aspects is needed in order to properly assess the performance of this technique (e. g. temporal and extreme aspects, as described in the companion papers of this special issue). Moreover, the present study cannot give a conclusive assessment of the skill of downscaling methods to simulate regional future climates, and further experiments  (Maraun et al. 2015)  will be soon performed in the framework of the EURO-CORDEX initiative, thus completing the analysis initiated in the present manuscript. Finally, in order to favor research reproducibility, the experimental framework is precisely defined, all datasets needed for this experiment are publicly distributed http://www. value-cost. eu/ datasets and, in some cases, the packages and/or code to reproduce the results are publicly available.
A metadata description vocabulary has been defined and implemented in the VALUE Validation Portal http://www. value-cost. eu/validationportal, which provides metadata information for all contributing methods (approach, technique, predictors, method configuration, etc. ). Transparency is also promoted by the VALUE Validation Portal http://www. value-cost. eu/validationportal, which provides public access to data and metadata information for all contributing methods and also for all validation results, as well as tools to filter and visualize (both in tabular and graphical formats) the results. In particular, most of the figures of the paper can be reproduced with these tools.
Acknowledgments This work have been performed in the framework of the VALUE is funded via the EU COST Action ES1102, under FP7 programme. We thank the VALUE community for their input to this framework, helping in the coding the validation routines and helping to select the representative stations at a national level. In particular we would like to thank Andreas Gobiet, Constantin Mares, Robertas Alzbutas, Mandy Vlachogianni, Adam Jaczewski, Peter Thejll, Patrick Willems, Ivan Pilaš, Meron Teferi Taye, and Fredrik Boberg. We acknowledge the data providers in the ECAD project (data and metadata are available at http://www. ecad. eu) and ECMWF for allowing us to re-distribute ERA-Interim daily data (regridded at a 2 • resolution) within VALUE (registration is public) for the standard set of predictors.
Implementation: ISI-MIP is implemented in the downscaleR  (Bedia et al. 2016 ) R package (isimip function). This package is freely available without restriction. • MOS-AN (only precipitation): MOS implementation of the analog method considering precipitation as the single predictor, and trained across different zones (similar to the Prudence regions) computing similarity using Euclidean distances of the precipitation fields. As a benchmark this method has been applied directly to ERA-Interim precipitation, with "perfect" (day-to-day) synchrony with observations. When applied to the ERA-Interim driven RCM simulation, this method exploits the marginal temporal synchrony within the RCM domain given by the synchronous forcing at the boundary  (Turco et al. 2011 (Turco et al. , 2017)). Note that this method is best suited for for nudged RCM simulations.
5 for maximal extrapolated value, compared to the last percentile, and a ratio of 3. 0 as a change between the last two percentile values). More details in  Štěpánek, P. et al. (2016). Implementation: In-house R code, incorporated in ProClimDB software (www. climahom. eu). • QMm: Equidistant empirical quantile mapping. Empirical CDFs are calculated for the observation and the calibration and validation periods. The probabilities are calculated for bins with widths set for the resolution of the observational data (e. g. 0. 1 • ). For each day in the validation period, the probability obtained from the validation CDF is used in the observational and calibration CDFs to obtain the corresponding data values  (Li et al. 2010 ). The difference between the observed and calibration data is used as the correction term for the validation.
g. from mid-April to Mid-June for the correction applied in May) in all these methods. Implementation: In-house Fortran code. • SB (only temperatures) A local scaling method where mean bias calculated separately for each season is subtracted from simulations in validating period. Implementation: In-house Matlab code. • VGLMGAMMA (only precipitation): A stochastic single-site MOS approach to predict precipitation occurrence and amounts conditionally on simulated daily precipitation as predictor. Precipitation occurrence is modeled via a logistic regression; precipitation amounts on wet days based on a vector generalised linear model that expresses the rate and shape parameters of the 2-parameter gamma distribution as a function of simulated daily precipitation. Temporal dependence is not explicitly modelled but only imprinted by the predictor, i.
es/downscaling, which builds on MeteoLab, and includes as illustrative examples the same standard predictor data and the VALUE regions used in this study. This package is freely available without restriction. • ANALOG-ANOM: For a given day to be downscaled, the ANALOG-ANOM  (Vaittinada Ayar et al. 2016)  determines the day in the calibration period which has the closest atmospheric situation. This is determined by a similarity metric (here a Euclidean distance) between the predictor set of the day to be downscaled and the predictor set of the day in the calibration period, considering the whole European domain. For this method, the predictors are fields of daily anomalies with respect to the annual cycle computed from cubic regression smoothing splines fitted on the empirical daily annual cycle.
Implementation: ESD is implemented in the esd R package  (Benestad et al. 2015a ). This package is freely available without restriction. • FIC01P/03P (only precipitation): FIC01P is a two-step analog methods. In a first step, the 30 closest analogs are computed for each test day based on Z1000 and Z500. Every analogue is defined in a three-windows nested grid (for short, medium, and large scale) with different weights. For this experiment we have used 42 main windows, each one with 3 nested windows, covering Europe. Instead of considering the weighted (according to similarity) mean observations of the analog days (p i ), the second step performs a pooling and ranking of the analog days month by month (900 values for each month) and computes the mean of consecutive blocks of 30 days q i according to their mean values.
1, NumberOfNearestNeighbours = 0, NumberOfPCs = 15, SimOccurrence = 'true', SimAmount = 'true', minrainydays = 5. These results (for the case of the ERA-Interim predictors) can be also reproduced (and modified) online using the statistical downscaling portal http://meteo. unican. es/downscaling, which builds on MeteoLab, and includes as illustrative examples the same standard predictor data and the VALUE regions used in this study. This package is freely available without restriction. • MLR/MLR-WT (only temperatures):  (Gutiérrez et al. 2013)  Multiple linear regression trained across different zones covering Europe (similar to the Prudence regions) with no seasonal component. The predictors are the 15 leading PCs of the joined predictor fields (which account for 75-90% of the explained variance across the different zones considered).
MLR-WT is a state-dependent version of MLR, conditioning the training to 12 Weather Types defined using a k-means algorithm (k=12) applied to the daily SLP over Europe (this variable is excluded from the predictor set). Implementation: MeteoLab public Matlab toolbox (http://meteo. unican. es/trac/ MLToolbox/wiki) using downTrain function with parameters type = 'linear regression', Num-berOfNearestNeighbours = 0, and NumberOfPCs = 15. These results (for the case of the ERA-Interim predictors) can be also reproduced (and modified) online using the statistical downscaling portal http://meteo. unican. es/downscaling, which builds on MeteoLab, and includes as illustrative examples the same standard predictor data and the VALUE regions used in this study. This package is freely available without restriction.
• MLR-PCA-ZRT (only temperatures): Linear regression model using s-mode PCs as predictors. The selection of predictors has been automated by iterating through all possible predictor combinations, minimizing the mean squared error and maximizing the time series correlation in calibration and validation  (Hertig and Jacobeit 2008; Hertig et al. 2013; Jacobeit et al. 2014). The selection was done for each station separately and models were developed for each month separately. Implementation: In-house Fortran (for PC calculation) and R code (using "lm" for regression). • MLR-RSN/RAN/AAN/AAI/AAW/ASI/ASW: Multiple linear pointwise regression (with stepwise screening) using gridpoint raw data (or anomalies), trained at an annual (or seasonal) basis and including optional variance corrections in the form of inflation or addition of white noise.
The first letter of the code refers to the raw (R) or anomaly (A) data used as predictors, the second letter refers to the annual (A) or seasonal (S) training, and the third letter refers to inflation (I) or white noise (W) variance correction (N for no correction). More details in  Huth (2002) ;  Huth et al. (2015). Implementation: In-house Fortran code. • MLR-T/GLM-P: These methods have been implemented following the Statistical DownScaling Method SDSM, which builds on linear regression  (Wilby et al. 2002). The parameters of the regression model are obtained by the least squares method from standardized variables. The link between the predictands and predictors is either an unconditional model, used for temperature (MLR-T), or a conditional model, used for precipitation (GLM-P), being the conditioning variable the probability of wet-day occurrence.
The multi-objective approach aims at a simultaneous optimization of RMSE, bias, standard deviation, selected quantiles and, for precipitation, the number of precipitation days. MOGP is applied individually for each station and variable. Except for precipitation, the predictors are interpolated from the four closest GCM grid cells to the location of the respective station. Precipitation is taken at the GCM grid box closest to a station. The MOGP code is based on the Strength Pareto Evolutionary Algorithm (SPEA) by  Zitzler and Thiele (1999)  and the GPLAB by  Silva and Almeida (2003). SPEA returns not one single regression model for each station and variable but a set of Pareto optimal models. From each set of potential downscaling models one has been selected that optimizes a trade-off between all objectives.
The automatic selection results in 8 predictors on average for precipitation and 6 for temperature. More details can be found in  Zerenner et al. (2016). Implementation: In-house MATLAB code (based on the GPLAB). • SWG: A two-step approach is implemented to model precipitation in a Vectorised Generalized Linear Models (VGLM). First, the rainfall occurrence is modeled through a logistic regression, allowing to characterize the probability of rainfall occurrence for a given day conditionally on atmospheric predictors. Then, the probability density function (pdf) of the rain intensity (given that it rains) is assumed to be a Gamma distribution whose logarithms of the shape and rate parameters are linear functions of the large-scale predictors. For temperature, a single step is used, where temperature is supposed to follow a Gaussian distribution with the mean and the logarithm of the standard deviation linearly dependent on the predictors  (Vaittinada Ayar et al.
JMG and SH acknowledge partial funding from MULTI-SDM project (MINECO/FEDER, CGL2015-66583-R). BH and DR acknowledge COMPLEX project (FP7-ENV-2012. Number: 308601). MT was supported by HOPE project (MINECO, CGL2014-52571-R). Participation of MD and RH was funded by the Ministry of Education, Youth, and Sports of the Czech Republic, contracts LD12029 and LD12059, respectively.
Results: We developed an expert system using paraconsistent logic, named GROOLS (Genomic Rule Object-Oriented Logic System), that evaluates the completeness and the consistency of predicted functions through biological processes like metabolic pathways. Using a generic and hierarchical representation of knowledge, biological processes are modeled in a graph from which observations (i. e. predictions and expectations) are propagated by rules. At the end of the reasoning, conclusions are assigned to biological process components and highlight uncertainties and inconsistencies. Results on 14 microbial organisms are presented. Conclusions: GROOLS software is designed to evaluate the overall accuracy of functional unit and pathway predictions according to organism experimental data like growth phenotypes.
Background Assigning functions to all predicted proteins from genome sequencing projects remains challenging despite improvements in bioinformatics methods as illustrated by the second critical assessment of functional annotation initiative (CAFA2)  [1]. Millions of protein entries are thus not assigned reliable functions in databanks due to the lack of trustworthy annotations and the drawbacks of homology-based predictions  [2]. Using this genomic data, genome-scale metabolic networks can be obtained and contain all of the known metabolic reactions in a given organism and the genes that encode each enzyme. High quality functional annotation is thus an essential step to correctly predict the chemical transformations that may occur in a living cell. From these networks, metabolic models can be derived to study metabolic and growth functions notably by using flux balance analysis  [3].
In the IMG system, additional rules based on biological processes are used to predict phenotypes like auxotrophy/prototrophy for biosynthetic pathways or compound utilization for degradation pathways. Here, we present an expert system, named GROOLS (Genomic Rule Object-Oriented Logic System), that uses paraconsistent logic to assist biocurators in the evaluation of the functional annotation of genes. The GROOLS method is inspired from a previous work about the HERBS software (A. Viari et al. , unpublished data) that was presented in 2003 during a workshop of the HAMAP project  [13]. Similarly to previously mentioned systems (i. e. Genome Properties and IMG), the objective is to automate human expert reasoning applied in the curation of genome annotation. Indeed, GROOLS does not predict functions but evaluates the completeness and the consistency of genome annotation through biological processes like metabolic pathways.
Biological data Prior Knowledge (PK) concepts represent biological processes and their components (e. g. a metabolic pathway and its reactions). Genome Properties  [10]  and UniPathway  [17]  resources are currently supported by GROOLS to reconstruct two distinct conceptual graphs. From the Genome Properties description file (version 3. 2, GenProp_3. 2_release. RDF), "Properties", "Components" and "Evidences" were extracted and transformed in PK concepts. 'Part' relations were established between "Properties" and "Components", and 'subtype' relations between "Components" and "Evidences". In the Genome Properties model, "Evidences" correspond either to pathway variants or predictors (i. e. leaf concepts in the graph). Predictors are protein family Hidden Markov Models (HMMs) from PFAM  [18]  and TIGRFAM  [11]  databases.
Other relations between PK concepts were converted to 'part' relations. To populate Genome Properties conceptual graph, Observations of Computation type can be extracted from UniProtKB database  [5]  in which each protein entry is linked to predicted PFAM  [18]  and TIGRFAM  [11]  families. For UniPathway, UniProtKB annotations can be used and inserted in the conceptual graph as Observations of UER (UniPathway Enzymatic Reactions) using their identifiers (when provided in the protein entry) or their cross-referenced EC (Enzyme Commision) numbers. Annotations from the MicroScope platform  [8]  can be used as a second resource of UniPathway Observations. MicroScope contains automatic and curated annotations of microbial genomes. Similarly to UniProtKB, protein annotated with EC numbers are converted in UER Observations.
This quantitative growth phenotype data should be first discretized using the omp R package  [21]  with "grofit" aggregation method and weak discretization (-a, -w and -z options of run_opm. R program). Using Observations with a true (growth) or false (no growth) truth value, the results obtained with carbon and nitrogen sources (PM1, 2 and 3 plates) are then linked to their corresponding Genome Properties (16 pathways) or Uni-Pathway (37 pathways) PK concepts (i. e. the metabolic pathway for the degradation of the tested compound). For ambiguous discretizations (NA values), both true and false Observation concepts are created. Regarding prototrophic organisms, true Observations on amino acid biosynthesis pathways can be added as a second source of Experimentation corresponding to 15 pathways in Genome Properties and 20 in UniPathway.
Implementation GROOLS is built on object-oriented logic programming. This paradigm facilitates the representation of biological knowledge with Prior Knowledge concepts and their relations. A first prototype of GROOLS (https://github. com/Grools/grools-drools-checker) was written using the DROOLS framework (https://www. drools. org, a Business Rules Management System based on an object-oriented rule language). In the present version of GROOLS software (https://github. com/Grools/grools-application), rules were directly implemented in Java with functional programming to tackle performance issues during the graph reasoning. The reasoner is available in a separate library (https://github. com/Grools/grools-reasoner). GROOLS contains plugins to construct the PK graph from two databases: UniPathway (in OBO format) and Genome Properties (in non-standard format).
Using this plugin system, database file content can be improved by users. Moreover, other resources on biological processes can be integrated without changing the reasoning library. For Observations, an unified CSV file format is used to represent Computation, Curation and Experimentation data. Each row is composed of seven fields to describe an Observation: "Name", "EvidenceFor" (name of the related Prior Knowledge), "Type" (Computation, Curation or Experimentation), "isPresent" (true or false), "Source" (e. g. UniProtKB, MicroScope, BIOLOG), "Label" and "Description". Three shell scripts are provided as example to run GROOLS on Genome Properties using UniProtKB proteomes ("uniprot_genpropToGrools. sh" script) and on UniPathway using MicroScope or UniProtKB annotations ("microscope_upaToGrools.
e. these pathways are expected in the organisms). Detailed results on the 14 studied organisms are available at this Web address: https://www. genoscope. cns. fr/agc/grools. Statistics on GROOLS conclusions are reported in Table  3. Accuracies of pathway and functional unit predictions were computed only for pathways with univocal expectations (i. e. pathway concepts with true-only or false-only Observations of Experimentation type). For Genome Properties, the evaluation of functional unit predictions in confrontation to pathway expectations (i. e. from BIOLOG data and amino acid biosynthesis pathways) gives high level of accuracy (79. 14%), reflecting the exquisitely sensitive searches of TIGRFAM models. In the falsehood mode, pathway prediction accuracy decreases from 78. 80 to 67. 93%. As expected, we observe that contradictions on pathway predictions (i.
UniPathway specific mode example In UniPathway, two variants made of one enzymatic reaction are described for the biosynthesis of asparagine (UPA00134). Using Acinetobacer baylyi UniProtKB annotations (Fig. 4  part a and see Additional file 3 for textual descriptions), any observation is linked to the two enzymatic reactions (UER00194 and UER00195) (i. e. there is any protein annotated with the corresponding UERs). Since this pathway is expected in A. baylyi (Exp_UPA00134 Observation), GROOLS reasoning concludes that both variants and reactions could be missing (i. e. GROOLS is not able to choose between those two). Unfortunately, UniPathway is wrong because a third pathway variant is known and reported in other metabolic databases like MetaCyc  [19]. It corresponds to a tRNAdependent transamidation mechanism for the conversion of aspartate to asparagine.
Conclusions GROOLS software is designed to evaluate the overall accuracy of functional unit and pathway predictions according to organism experimental data like growth phenotypes. It allows users to compare different sources of annotations for a genome. Furthermore, using conclusion values, biocurators can quickly focused on missing or contradictory predictions to improve protein functional annotation. With the aim of building a metabolic model of an organism, and before applying time-intensive gap filling methods, GROOLS could thus be used to improve the initial metabolic network reconstruction in regards to the genome annotation and growth phenotype data.
Many approaches for testing configurable software systems start from the same assumption: it is impossible to test all configurations. This motivated the definition of variability-aware abstractions and sampling techniques to cope with large configuration spaces. Yet, there is no theoretical barrier that prevents the exhaustive testing of all configurations by simply enumerating them if the effort required to do so remains acceptable. Not only this: we believe there is a lot to be learned by systematically and exhaustively testing a configurable system. In this case study, we report on the first ever endeavour to test all possible configurations of the industry-strength, open source configurable software system: JHipster, a popular code generator for web applications. We built a testing scaffold for the 26,000+ configurations of JHipster using a cluster of 80 machines during 4 nights for a total of 4,376 hours (182 days) CPU time.
This article builds on preliminary results  [25]  that introduced the JHipster case for research in configurable systems and described early experiments with the testing infrastructure on a very limited number of configurations (300). In addition to providing a quantitative assessment of sampling techniques on all the configurations, the present contribution presents numerous qualitative and quantitative insights on building the testing infrastructure itself and compares them with JHipster developers' current practice. In short, we report on the first ever endeavour to test all possible configurations of the industry-strength open-source configurable software system: JHipster. While there have been efforts in this direction for Linux kernels, their variability space forces to focus on subsets (the selection of 42,000+ ker-nels corresponds to one month of computation  [54] ) or to investigate bugs qualitatively  [1, 2].
an assessment of sampling techniques. Dissimilarity and t-wise sampling techniques are effective to find faults that cause a lot of failures while requiring small samples of configurations. Studying both fault and failure efficiencies provides a more nuanced perspective on the compared techniques; 5. a retrospective analysis of JHipster practice. The 12  configurations used in the continuous integration for testing JHipster were not able to find the defects. It took several weeks for the community to discover and fix the 6 faults; 6. a discussion on the future of JHipster testing based on collected evidence and feedback from JHipster's lead developers; 7. a feature model for JHipster v3. 6. 1 and a dataset to perform ground truth comparison of configuration sampling techniques, both available at https: //github.