dataesr/scientific-paragraphs-categorization huggingface dataset:
Epidemiological data Epidemiologic data on DF were obtained from two different sources, depending on the collection period. For the data from 1991-2006, a surveillance system was used, and data were based on a weekly census of biologically confirmed cases (BCCs) that were recorded by the Arbovirus National Reference Centre, which is based at the Pasteur Institute of French Guiana. In 2006, a multi-source surveillance system was implemented by the Regional Epidemiology Unit of the Institut de Veille Sanitaire that included all seven biological laboratories (public hospital and private laboratories) that are located in the coastal area. Concurrently, in 2006, a new dengue diagnostic test based on NS1 antigen detection was made available to all laboratories, and it contributed substantially to improving surveillance.
Cases were biologically confirmed by isolating the virus and detecting viral RNA using reverse-transcription PCR (RT-PCR), NS1 antigen detection methods or serological tests that are based on an immunoglobulin M (IgM)capture enzyme-linked immunosorbent assay (MAC-ELISA)  [13]. This surveillance system was authorized by the French Data Protection Agency (CNIL, N°1213498). The DF incidence rates in French Guiana, which are defined as the yearly number of cases/100 000 inhabitants, were calculated for the 1991-2013 period. A standardization procedure was performed separately for the 1991 -April 2006 and May 2006-2013 periods. For this procedure, we used a zscore scaling method to take into account the improvement in the epidemiological surveillance system that was observed in April 2006. This approach led to a trend toward increasing numbers of cases and enabled us to work with a single dataset.
Climate data We used sets of meteorological parameters and large-scale atmospheric and global SST data for 1990-2013 for this study. Meteorological records included rainfall, temperature and relative humidity and were obtained from Météo-France. We calculated monthly country means from these daily station data throughout the study period. A set of atmospheric and SST predictors was constructed from the ERA-Interim reanalysis data that were obtained from the European Centre for Medium-Range Weather Forecasts  [43]. The ERA-Interim system assimilates observations and outputs using a forecast model. The climate fields were available at a 0. 75°x0. 75°s patial resolution and 60 vertical levels.
The model that maximized the AUC (area under the curve) from the receiver operating characteristic (ROC) analysis  [45]  and minimized the AIC (Akaike information criterion)  [46]  was selected. The final model performances were evaluated by calculating ROC scores and cross-validating the data  [47]. The ROC is a method of testing the skill of categorical forecasts using the hit rate (HR) and false alarm rate (FAR). The HR indicates the proportion of epidemic years that were categorically forecast (sensitivity). It ranges from 0 to 1 (1 being desirable) and is calculated as follows: HR ¼ Hits Hits þ Misses The FAR is the proportion of non-epidemic years that were forecast as epidemic years (1-specificity). It ranges from 0 to 1 (0 being desirable) and is defined as follows: FAR ¼ False alarms Hits þ False alarms Second, a cross-validation on chunks of multiple years was performed to measure the model stability.
What is the epidemic risk in 2016? Forecasts for 2014 and 2015 (not included in the training dataset) indicated that the model predictions were consistent with the non-epidemic conditions that were observed in French Guiana (the DF IR/100 000 inhabitants was 350 in 2014 and 106 for January to September, 2015) (Table  2 ). In 2016, as a result of the warm SST conditions over the Equatorial Pacific Ocean that occurred in August and July (25. 26°C) and the high pressures over the Azores High in November (1021. 36 hPa), the model predicted that French Guiana would likely experience an outbreak (probability of 0. 92).
However, our analysis showed that anomalous SST patterns did not precisely correlate with El Niño events. Indeed, the SST anomalies of El Niño events were more intense in winter months. Importantly, certain epidemic years corresponded to strong  (1997) (1998)  and moderate  (1992, (2009) (2010)  El Niño events, although this association was not systematic. For example, an El Niño event did not occur from 2005-2006, but French Guiana did subsequently experience a dramatic epidemic. Sea-level pressure conditions that characterized an epidemic year. A composite analysis was performed by separately averaging the SLP data for the years in which the highest (HIGH) and lowest (LOW) DF incidence were recorded in French Guiana. The contours (at 0. 5°C intervals) show the HIGH minus the LOW differences in the SLP from July to December to illustrate the conditions that characterized a typical epidemic year.
Conclusions Among the wide panel of factors that can influence DF outbreaks, these results suggest that large-scale climate factors play an important role. We found that the climatic indices that were assessed in this study were important for DF monitoring and for predicting outbreaks in French Guiana over a period of 2-3 months. This delay may give public health authorities the ability to anticipate outbreaks and implement social communication and vector control measures, and to adapt healthcare capacity and increase preparedness in a timely manner. Importantly, this model could be easily and regularly updated using newly collected data that was retrieved from the ongoing dengue surveillance system. Because the identified climate indicators are simple and easy to access, they could be used to estimate the probability of future epidemics occurring according to climate change simulations and help to evaluate the effectiveness of potential intervention strategies.
Acknowledgments We are grateful to all of the collaborators who were involved in the surveillance system that was monitored by  CIRE-AG. We wish to thank  Luisiane CARVALHO  and  Martine LEDRANS  ( CIRE-AG ),  Dr. Philippe DUSSART  and  Dr. Séverine MATHEUS  ( IPG-CNR Arbovirus ) and  Philippe PALANY (Météo-France). We also thank  Maguelonne TEISSEIRE  (Irstea-TETIS) and  Sandra BRINGAY  ( LIRMM, CNRS, UMR 5506 ) for their support during the preparation of this project.
Background: In the context of high-throughput molecular data analysis it is common that the observations included in a dataset form distinct groups; for example, measured at different times, under different conditions or even in different labs. These groups are generally denoted as batches. Systematic differences between these batches not attributable to the biological signal of interest are denoted as batch effects. If ignored when conducting analyses on the combined data, batch effects can lead to distortions in the results. In this paper we present FAbatch, a general, model-based method for correcting for such batch effects in the case of an analysis involving a binary target variable. It is a combination of two commonly used approaches: location-and-scale adjustment and data cleaning by adjustment for distortions due to latent factors.
We compare FAbatch extensively to the most commonly applied competitors on the basis of several performance metrics. FAbatch can also be used in the context of prediction modelling to eliminate batch effects from new test data. This important application is illustrated using real and simulated data. We implemented FAbatch and various other functionalities in the R package bapred available online from CRAN. Results: FAbatch is seen to be competitive in many cases and above average in others. In our analyses, the only cases where it failed to adequately preserve the biological signal were when there were extremely outlying batches and when the batch effects were very weak compared to the biological signal. Conclusions: As seen in this paper batch effect structures found in real datasets are diverse.
Background In practical data analysis, the observations included in a dataset sometimes form distinct groups-denoted as "batches"; for example, measured at different times, under different conditions, by different persons or even in different labs. Such batch data is common in the context of high-throughput molecular data analysis, where experimental conditions typically have a high impact on the measurements and only few patients are considered at a time. Taking a more general point of view, different batches may also represent different studies concerned with the same biological question of interest. Independently of the particular scenario, in this paper all systematic differences between batches of data not attributable to the biological signal of interest are denoted as batch effects.
See for example  [2, 3]  for a general overview and for an overview of methods suitable in applications involving prediction, respectively. Two of the most commonly used methods are ComBat  [4] , a location-and-scale batch effect adjustment method and SVA  [5, 6] , a non-parametric method, in which the batch effects are assumed to be induced by latent factors. Even though the assumed form of batch effects underlying a location-and-scale adjustment as done by ComBat is rather simple, this method has been observed to greatly reduce batch effects  [7]. However, a location-and-scale model is often too simplistic to account for more complicated batch effects. SVA is, unlike ComBat, concerned with situations where it is unknown which observations belong to which batches. This method aims at removing inhomogeneities within the dataset that also distort its correlation structure.
Care has to be taken in the latent factor estimation in the context of data-cleaning. Inhomogeneities within the dataset are naturally not only induced by sources of unwanted noise but also by the biological signal of interest. If one would not take this interference between batch effects and signal into account, removing the corresponding estimated latent factor loadings would lead to removing a large part of the biological signal of interest. An obvious, yet problematic way, of protecting the signal of interest would be to remove it temporarily before estimating the latent factors by regressing each of the variables in the dataset on the variable representing the biological signal. However, this can lead to an artificially increased signal, as outlined in the Section "FAbatch". As a solution for the case of a binary variable representing the biological signal, in our method we fit preliminary L 2 -penalized logistic regression models and use them to predict the probabilities of the individual observations to belong to the first and the second class, respectively.
These predicted probabilities are then used in place of the actual values of the binary variable when protecting the signal of interest during latent factor estimation. See the Section "FAbatch" for details. In its current form our method is thus only applicable when the signal variable is binary, but extensions to other types of variables are possible, see the Section "Discussion". As an illustration, Fig. 1  shows plots of the first two principal components obtained by Principal Component Analysis (PCA) on a raw dataset (upper-left) and after running the three different batch effect adjustment methods described above, respectively. The dataset, composed of two batches, contains the gene expressions of 20 alcoholics and 19 healthy controls. It is downloadable from ArrayExpress  [8] , accession number: E-GEOD-44456.
FAbatch Fig. 1  Visualization of batch effect adjustment. First two principal components out of PCA performed on the covariate matrix of a microarray dataset studying alcoholism: raw, after batch effect adjustment according to ComBat, SVA using three factors and FAbatch using three factors. The first batch is depicted in bold and the numbers distinguish the two classes "alcoholic" (2) vs. "healthy control"  (1). The contour lines represent batch-wise two-dimensional kernel estimates and the diamonds represent the batch-wise centers of gravities of the points obtain the prediction rule often constitutes a different batch than the validation data the prediction rule is applied to. Batch effect adjustment can be used here to make the validation data more similar to the training data before applying a prediction rule that was previously fitted on the training data.
Such a procedure, termed "addon batch effect adjustment" in the following, is not specific to our method, but a general concept. Here, batch effect adjustment is first conducted based on the available original dataset. Some methods require that the values of the target variable are known in this dataset. Subsequently, batch effect adjustment for independent batches is performed. To facilitate this, it is required that several observations from each batch are available simultaneously ("frozen SVA" is an exception here, see the Section "Addon adjustment of independent batches"). This second phase does not affect the data prepared in the first phase. See the Section "Addon adjustment of independent batches" for details. We refer to such scenarios as cross-batch prediction in the rest of this paper.
Using estimated probabilities instead of actual classes As already noted in the Section "Background", a further peculiarity of our method is that we do not use the actual classes when protecting the biological signal of interest in the estimation algorithm. Instead, we estimate the probabilities of the observations to belong to either class and use these in place of the actual classes, see the next paragraph and the next subsection for details. This procedure has two major advantages. Firstly, it makes the batch effect correction method applicable to prediction problems involving new test observations with unknown classes. Secondly, using the actual classes might lead to an artificial increase of separation between the two classes in the dataset. This is because, as will be seen in the next subsection, it is necessary to use the estimated, instead of the true, but unknown, classspecific means when centering the data before factor estimation.
Due to sampling variance, these estimated class-specific means often lie further away from each other than the true means, in particular for variables for which the true means lie close to each other. Subtracting the estimated factors' influences leads to a reduction of the variance. Now, if centering the variable values within the classes before factor estimation, removing the estimated factor influences would lead to a reduction of the variance around the respective estimated classspecific means. In those-frequently occurring-cases, in which the estimated class-specific means lie further from each other than the corresponding true means, this would lead to an artificial increase of the discriminatory power of the corresponding variable in the adjusted dataset. All analyses which are concerned with the discriminatory power of the covariate variables with respect to the target variable would be biased if performed on data adjusted in this way.
The only, but important, exception where we perform ordinary cross-validation for estimating the π ij is when the data come from only one batch (this occurs in the context of cross-batch prediction, when the training data consist of one batch). The shrinkage intensity tuning parameter of the L 2 -penalized logistic regression model is optimized with the aid of cross-validation  [10]. For computational efficiency this optimization is not repeated in each iteration of the cross-batch prediction. Instead, it is performed beforehand on the complete dataset. The overoptimism resulting from this procedure compared to that of the gold-standard technique "nested cross-batch prediction" can be assumed to be negligible in the considered context. 3. Calculate the class adjusted values x ijg,S,CA , which should contain considerably less class signal than x ijg,S : x ijg,S,CA := x ijg,S -(1 -π ij ) μ g,S (1) -π ij μ g,S (2) ,  (6)  where μ g,S (c) = (1/#L c ) {i * ,j * }∈L c x i * j * g,S with L c = {{i, j} : y ij = c, i ∈ {1,.
Verification of model assumptions on the basis of real data Due to the flexibility of its model FAbatch should adapt well to real datasets. Nevertheless it is important to check its validity based on real data, because the behaviour of high-dimensional biomolecular data does not become apparent by mere theoretical considerations. Therefore, we demonstrate that our model is indeed suited for such data using the dataset BreastCancerConcatenation from Table  1. This dataset was chosen because here the batch effects can be expected to be especially strong due to the fact that the batches involved in this dataset are themselves independent datasets. We obtained the same conclusions for other datasets (results not shown). Because our model is an extension of the ComBat-model by batch-specific latent factor contributions, we compare the model fit of FAbatch to that of ComBat.
Comparison of FAbatch with existing methods A comprehensive evaluation of the ability of our method to adjust for batch effects in comparison to its competitors was performed-using both simulated as well as real datasets. The simulation enables us to study the performance, subject to basic settings and to use a large number of datasets. Nevertheless simulated data can never capture all properties found in real datasets from the area of the application. Therefore, in addition, we studied 14 publicly available real datasets, each consisting of at least two batches. The value of batch effect adjustment contains different aspects, which are connected with the adjusted data itself or with the results of certain analyses performed using the latter. Therefore, when comparing batch effect adjustment methods it is necessary to consider several criteria, where each is concerned with a certain aspect.
Performance metrics Here we describe the performance metrics used to assess batch effect adjustment. Several of them are, in their original form, restricted to the case of only two batches. For datasets with more than two batches they are extended as follows: 1) Calculate the original metric for all possible pairs of batches; 2) Calculate the weighted average of the values in 1) with weights proportional to the sum of the sample sizes in the two respective batches. Separation score (sepscore) We derived this metric from the mixture score presented in  [2]. The latter was not applicable here, because it depends on the relative sizes of the two involved batches j and j *. Roughly speaking the mixture score measures the degree of mixing between the observations belonging to the two batches after batch effect adjustment.
Performance of differential expression analysis (diffexpr) This metric is similar to the idea pre- sented in  [2]  which consists in comparing the list of genes deemed differentially expressed the strongest using a batch effect adjusted dataset to the corresponding list obtained using an independent dataset. Having no independent data available here we had to consider a slightly different approach: 1) For each batch j leave this batch out and perform batch effect adjustment using the rest of the dataset. Derive two lists of the 5 % of variables deemed differentially expressed the strongest (see next paragraph for details): one using the batch effect adjusted datasetwhere batch j was left out-and one using the data from batch j. Calculate the number of variables appearing in both lists and divide this number by the common length of the lists.
2) Calculate a weighted average of the values obtained in 1) with weights proportional to the number of observations in the corresponding left-out batches. Note that in the case of the simulated datasets we would be able to estimate the true discovery rate instead of calculating the metric described above. However, for the sake of comparability, we applied the procedure described above for the simulated data as well. Now we describe the procedure performed for estimating those 5 % of variables which are most differentially expressed. Our original idea to use the p-values of simple two-sample t-tests between the two classes was soon discarded. The reason for this was that this procedure might have favoured batch effect adjustment methods that produce more normally distributed values of the variables.
Simulation design Three basic scenarios were considered: 1) "ComCor": Common correlation structure in all batches; 2) "Batch-Cor": Batch-specific correlation structures; 3) "Batch-ClassCor": Batch-and class-specific correlation structures. For each of these the correlations were induced in two ways (see below for details): 1) simulating from a latent factor model with normally distributed residuals; 2) drawing from multivariate normal distributions with specified correlation matrices. The second scheme was considered to avoid favouring FAbatch and SVA by restricting the simulation to factor-based data generation mechanisms. We simulated datasets consisting of four batches with 25 observations each. The number of variables was 1000. For each of the six (3 × 2) settings 500 datasets were simulated.
The values of the parameters occurring in the simulation models were based on corresponding estimates obtained from two publicly available microarray datasets: a dataset also used in the real data study, denoted as AutismTranscr (Table  1 ) and a dataset studying colon cancer, denoted as ColoncbTranscr. The latter is downloadable from ArrayExpress  [8] , accession number: E-GEOD-44861. All six settings can be expressed using the following most general model: x ij = α + a ij β + γ j + * ij , * ij ∼ MVN(0, j,a ij ), ( 9 ) with x ij = (x ij1 ,. , x ijp ) T , α = (α 1 ,. , α p ) T , a ij ∈ {0, 1}, β = (β 1 ,. , β p ) T , γ j = (γ j1 ,. , γ jp ) T , * ij = ( * ij1 ,. , * ijp ) T , j ∈ {1,. , K} and p = 1000. The entries of α and γ j (j ∈ {1,. , K}) were drawn from normal distributions with means and variances based on corresponding estimates obtained from ColoncTranscr.
We used the 1000 genes from the AutismTranscr dataset, which showed themselves to be most related to the binary outcome according to variable-wise two-sample t-tests. Before estimating the correlation matrices, the data was further centered by class in each batch to adjust for excess correlations due to class differences. The variances are the same in all three scenarios. They were set to be equal to those in scenario "ComCor" of Design A, i. e. 5 m=1 b 2 0gm + δ 2 jg σ 2 g. The correlation matrices were obtained as follows for the three scenarios: 1. ComCor: A single correlation matrix was used for all batches here. It was estimated from the data of a single batch in AutismTranscr. 2. BatchCor: A separate correlation matrix was used for each batch here, each estimated from the data of a batch in AutismTranscr.
Datasets We used 14 high-dimensional datasets with a binary target variable and at least two batches. They were downloaded from the ArrayExpress database (www. ebi. ac. uk/ arrayexpress)  [8]  or the NCBI GEO database (www. ncbi. nlm. nih. gov/geo) [18]. In searching for suitable datasets on ArrayExpress and NCBI GEO, we entered the search term "batch" and manually surveyed the search hits. This proceeding was chosen in order to maximise the number of possibly eligible datasets. Exclusion criteria were: number of samples too low, abscence of a batch variable, and impossibility of forming a suitable binary target variable. We state that the selection of the datasets was not in any way based on the results they yielded with the different methods, thus following Rule 4 from  [19]  ("do not fish for datasets").