dataesr/scientific-paragraphs-categorization huggingface dataset:
Three datasets featured too many variables to be manageable for our systems. Therefore, in these cases, we randomly selected 50,000 variables. When missing values occurred in the measurements of datasets we took the following approach. First, we excluded variables with too many missing values. Consecutively the remaining missing values were simply imputed by the median of the observed values of the corresponding variable in the corresponding batch. This simplistic imputation procedure can be justified by the very low numbers of variables with missing values in all datasets. Outlier analysis was performed by visually inspecting the principal components out of PCA applied to the individual datasets. Here, suspicious samples were removed. Additional file 1: Figure  S7  shows the first two principal components out of PCA applied to each of the used datasets after imputation and outlier removal.
Ability to adjust for batch effects Additional file 1: Figure  S8  to S14 show the values of the individual metrics obtained on the simulated data and Fig. 2  shows the corresponding results obtained on the 14 real datasets. Additional file 1: Tables S1 to S7 for the simulated and Tables  2  and 3  for the real data, respectively show the means of the metric values separated by method (and simulation scenario) together with the mean ranks of the methods with respect to the individual metrics. In most cases, we observe that the simulation results differ only slightly between the settings with respect to the ranking of the methods by their performance. Therefore, we will only occasionally differentiate between the scenarios in the interpretations. Similarly, simulations and real data analyses often yield similar results.
Differences will be discussed whenever relevant. According to the values of the separation score (Additional file 1: Figure  S8  and Fig. 2 , Additional file 1: Table  S1  and Table 2 ) ComBat, FAbatch and standardization seem to lead to the best mixing of the observations across the batches. For the real datasets, however, standardization was only slightly better on average than other methods. The results with respect to avedist are less clear. The simulation with factors (Design A) suggests that FAbatch and SVA are associated with greater minimal distances to neighboring batches, compared to the other methods. However, we do not clearly observe this for Design B other than for the setting with common correlations. The real data results also suggest no clear ordering between the methods with respect to this metric; see in particular the means over the datasets in Table  2.
The values of this metric were not appreciably improved by batch effect adjustment in general on the real datasets. The values of klmetric, which is conceptionally very similar to the separation score, allows a very similar conclusion as the latter metric (Additional file 1: Figure  S10  and Fig. 2 , Additional file 1: Table  S3  and Table 2 ): Com-Bat, FAbatch and standardization performed best here. While this conclusion could be obtained on both simulated and real data, other results differed between the different simulation scenarios and the real data analyses: SVA performed considerably worse here for Design A than B and mean-centering performed better on the simulated data in general. The estimates of the proportions of the variation explained by the class signals obtained via Principal Variance Components Analysis (pvca) are depicted in the Additional file 1: Figure  S11  and Fig.
2  and summarized in the Table  S4  (Additional file 1) and Table  2. SVA appears to be associated with the highest proportion of variation induced by the class signal. However, the comparison to the other methods is not fair here: SVA makes use of the target variable and is therefore associated with an artificially increased class signal. See the Section "Artificial increase of measured class signal by applying SVA" for details on this mechanism related to overoptimism. FAbatch performed well only on the simulated data here, but not on the real datasets, where it had the lowest mean value with the exception of no batch effect adjustment. Figure  2  reveals that those three datasets for which pvca was considerably smaller after batch effect adjustment by FAbatch were, at the same time, the three datasets with the highest pvca-values before batch effect adjustment.
Datasets with high pvca-values are datasets where the biological signal is relatively strong in comparison to the batch effects. Our results suggest that for such datasets, Fig. 2  Metric values in real datasets. Boxplots of values for all 14 datasets separated into method for the following metrics: sepscore, avedist, klmetr, pvca, diffexpr, skewdiv and corbeaf. The grey lines connect values corresponding to the same datasets batch effect adjustment with FAbatch might be counterproductive. The distinguishing feature of FAbatch in comparison to a mere location-scale adjustment as performed by ComBat is that it aims at additionally adjusting for batch effects not explainable by location and scale shifts. While FAbatch aims at protecting the biological signal in the factor estimation, it cannot be protected entirely here due to the uncertainty in the estimation of the class probabilities.
For FAbatch we examined those datasets which yielded substantially worse diffexpr-values after batch effect adjustment than before. As can already be seen from Fig. 2 , two of these datasets have high diffexprvalues on the data before batch effect adjustment. This implies that for these datasets the biological signal is well In each row the results are listed in descending order according to mean performance in terms of the original values and their ranks, respectively. The results of FAbatch are printed in bold In each row the results are listed in descending order according to mean performance in terms of the original values and their ranks, respectively. The results of FAbatch are printed in bold preserved in the batches-in other words they seem to be less affected by batch effects. A possible reason why FAbatch performs worse for mild batch effects has already been outlined above.
The other datasets connected with worse diffexpr-values than "no batch effect adjustment" in the case of FAbatch were those datasets for which some "outlying" batches were very different from the others-according to the PCA plots given in (Additional file 1: Figure  S7 ). We conjecture that, in this case, pooling the data of the outlying batch(es) with the other batches and estimating the L 2 -penalized logistic regression model can result in a predictor with bad performance. The combined data might be too heterogeneous for the L 2 -penalized logistic regression model, which assumes that all observations follow the same distribution. If the predictions of the class probabilities by the L 2 -penalized logistic regression rule are bad, the biological signal is less protected in the latent factor estimation.
Therefore, the removal of the estimated latent factor influences will affect the biological signal more. There were no noteworthy differences between the other methods with respect to diffexpr. For the real datasets there were also no improvements over no batch effect adjustment. This indicates that differential expression analysis might not benefit from batch effect adjustment in general. For the skewness divergence score (Additional file 1: Figure  S13  and Fig. 2 , Additional file 1: Table  S6  and Table  3 ) no clear ranking between the methods is seen in the case of the simulated data. However, for the real datasets, SVA and FAbatch clearly outperform the other methods with respect to this metric. Finally, both in the simulated and real data, FAbatch and SVA have considerably lower corbeaf-values (Additional file 1: Figure  S14  and Fig.
Application in cross-batch prediction In this illustrative analysis we apply all batch effect adjustment methods considered above together with the corresponding addon procedures described in the Section "Addon adjustment of independent batches" in cross-batch prediction in a real data example and using simulated data. A more extensive real data study was conducted by Luo et al. [3]  who used several datasets to compare all of the methods considered here, except for frozen SVA ("fSVA") and FAbatch, with respect to their performance in cross-batch prediction. We use the dataset IUGRTranscr. The reasons for choosing this dataset were that it features a relatively strong class signal and is at the same time strongly affected by batch effects-judging from the principal component analysis plot in Additional file 1: Figure  S7.
This dataset contains miRNA-measurements obtained from 67 human placentas using the Illumina Human-6 v2 Expression BeadChip. Of these 67 samples, 27 were obtained from placentas of embryos suffering from intrauterine growth restriction (IUGR), the remaining 40 samples originate from placentas of healthy embryos. The dataset consists of two batches of sizes 20 and 47, where in the first batch 9 (45 %) and in the second batch 18 (≈ 38 %) samples originate from IUGR embryos. As classification algorithm for the dependent variable "IUGR (yes vs. no)" Linear Discriminant Analysis (LDA) using Partial Least Squares (PLS) components as covariates  [20]  was chosen, where the number of components used was tuned on the grid 1,2,. , 10 employing 3-fold CV. Just as Luo et al. [3]  in their extensive real data study, we use Matthews Correlation Coefficient (MCC) as performance metric.
The additional fourth setting was generated by simply setting the correlations in Design B to zero. For each setting we simulated 100 datasets and proceeded as in the analysis of the real dataset presented above-with two differences. The first difference was that in the simulation we have to consider 4 2 × 2 = 12 instead of two combinations of training and validation batches per dataset, because the simulated datasets feature four instead of only two batches. The second difference concerns the evaluation of the results, because the MCC values could not be calculated in cases where both the numerator and denominator in the calculation were zero. Therefore for each combination of setting and batch effect adjustment method we summed up the true positives, the true negatives, the false positives and the false negatives over all prediction iterations in all 100 datasets and calculated the MCC-value using the standard formula.
MCC Fig. 5  MCC-values from simulation study. The colors differentiate the methods: none ( ), fabatch ( ), combat ( ), fsvafast ( ), fsvaexact ( ), meanc ( ), stand ( ), ratiog ( ), ratioa ( ). For better interpretability the results to the same methods are connected respects the simulation results concur with the results obtained using the real dataset. The most striking difference is that standardization was best here, while it was bad for the real data analysis. The good performance of standardization in the simulation should however not be over-interpreted as it was the least performant method in the study of Luo et al. [3]. FAbatch was the second-best method in all settings except for that without correlation between the predictors. In the latter setting, FAbatch is outperformed by ComBat and mean-centering.
Artificial increase of measured class signal by applying SVA In the Section "FAbatch" we detailed why using the actual values of the target variable in protecting the biological signal during the latent factor estimation of FAbatch would lead to an artificially increased class signal. SVA does use the values of the target variable and indeed suffers from the problem of an artificially increased class signal. In the following, we will outline the reason why SVA suffers from this problem. A crucial problem with the weighting of the variable values by the estimated probabilities that the corresponding variable is associated with unmeasured confounders but not with the target variable is the following: these estimated probabilities depend on the values of the target variable, in particular for smaller datasets.
This phenomenon is pronounced more strongly in smaller datasets. The reason for this is that for larger datasets the measured signals of the variables get closer to the actual signals, wherefore the overoptimism due to working with the apparent instead of the actual signals becomes less pronounced here. Accordingly, in the real data example from the previous subsection fSVA performed considerably worse when using the smaller batch as training data. Using datasets with artificially increased signals in analyses can lead to over-optimistic results, which can have dangerous consequences. For example, when the result of cross-validation is over-optimistic, this may lead to overestimating the discriminatory power of a poor prediction rule. Another example is searching for differentially expressed genes.
Here, an artificially increased class signal could lead to an abundance of false-positive results. The observed deterioration of the MCC-values in the real data example by performing frozen SVA when training on the smaller batch may, admittedly, also be due to random error. In order to investigate whether the effects originating from the mechanism of artificially increasing the discriminative power of datasets by performing SVA are strong enough to have actual implications in data analysis, we performed a small simulation study. We generated datasets with 40 observations, 1000 variables, two equally sized batches, standard normally distributed variable values and a binary target variable with equal class probabilities. Note that there is no class signal in this data. Then using 5-fold cross-validation repeated two times we estimated the misclassification error rate of PLS followed by LDA for this data.
Consecutively, we applied SVA to this data and again estimated the misclassification error rate of PLS followed by LDA using the same procedure. We repeated this procedure for the number of factors to estimate set to 1, 2 and 3, respectively. In each case we simulated 50 datasets. The mean of the misclassification error rates was 0. 504 for the raw datasets and 0. 431, 0. 356 and 0. 306 after applying SVA with 1, 2 and 3 factors. These results confirm that the artificial increase of the class signal by performing SVA can be strong enough to have implications in data analysis. Moreover, the problem seems to be more severe for a higher number of factors estimated. We did the same analysis with FAbatch, again using 1, 2 and 3 factors, where we obtained the misclassification error rates 0. 505, 0.
e. that in each batch different sources of heterogeneity may operate. In Appendix A. 2 (Additional file 1) it is shown that in the SVA model it is implicitly assumed that the distribution of the vector of latent factors may be different for each observation. This is a very general assumption. However, it is unclear how well SVA can deal with specific datasets originating from such a general model, because the link between the singular value decomposition used in the estimation and this model is not evident. Our algorithm by contrast was explicitly motivated by its underlying model, which is quite general and reasonable. In cases in which the data in question is approximately uniform with this model, FAbatch should perform reasonably well. In the form presented here, FAbatch is only applicable in the presence of a binary target variable.
This idea may also be applied in the protection of the biological signal of SVA, i. e. when multiplying the variable values by the estimated probabilities that the corresponding variables are associated with unmeasured confounders, but not with the binary variable representing the biological signal. More precisely these probabilities could be estimated in a cross-validation procedure-taking up again the idea also used in FAbatch. All batch effect adjustment methods considered in this paper, together with the corresponding addon procedures and all metrics used in the comparisons of the methods, were implemented/adopted into the new R package bapred available online from CRAN  [21].
Conclusions FAbatch leads to a good mixing of the observations across the batches in comparison to other methods, which is reassuring given the diversity of batch effect structures in real datasets. In the case of very weak batch effects and in the case of strongly outlying batches, the observed biological signal may be slightly altered by FAbatch. In our extensive comparison study of existing and new batch effect correction methods, we found that no method was best with respect to all metrics. It is thus difficult to formulate general recommendations: the choice of the method may primarily depend on the goal of the researcher as reflected by the choice of the metric. Performing no batch effect correction at all is in any case not recommended.
40 transcription profiling ArrayExpr. : E-GEOD-35574 IBSTranscr 63 6 54671 0. 70 transcription profiling ArrayExpr. : E-GEOD-36701 SarcoidosisTranscr 58 3 54675 0. 66 transcription profiling NCBI GEO: GSE19314 pSSTranscr 49 3 54675 0. 63 transcription profiling ArrayExpr. : E-GEOD-40611 AlcoholismTranscr 39 2 28869 0. 51 transcription profiling ArrayExpr. : E-GEOD-44456 WestNileVirusTranscr 39 2 47323 0. 46 transcription profiling ArrayExpr. : E-GEOD-43190 LabelNum. of Num. of Num. of Prop. with Data type Source (Acc. num. )observ. batches variables y = 2.
Introduction This paper considers model selection for capture-recapture data that are obtained from open populations of wild animals. Capture-recapture studies involve the capture and unique marking of individuals, which are then released into the population and subsequent attempts are made to recapture them. The resulting data can be recorded as individual encounter histories for each animal, which take the form of vectors with elements 0 and 1, indicating non-capture and capture respectively. The encounter history data can often be conveniently summarized in terms of an upper triangular matrix, which is known as an m-array, with elements m i,j denoting the number of individuals released at occasion t i and next recaptured at occasion t j , concatenated with a column vector with elements v i denoting the numbers of individuals released at occasion t i which were never captured again.
Motivating examples We consider two motivating capture-recapture data sets. The first is a large study of breeding great cormorants Phalacrocorax carbo sinensis from Denmark. The cormorant data have been fully analysed in  Hénaux et al. (2007). The cormorants provide a complex case-study for which it is unknown a priori what behavioural traits may be exhibited by the population. The data consist of capture histories from 862 breeding birds, captured at an established single colony over a period of 11 breeding seasons. The cormorants are only initially captured at the time of marking and are then subsequently resighted in the breeding colony. The second is a set of capture-recapture data on the humpback whale Megaptera novaeangliae population in the South Pacific. These data have been analysed by  Madon et al.
(2013). The capture-recapture data are compiled from genetic records and here we consider just the female genetic data for illustration, which have capture histories from 101 individuals, collected over a period of seven encounter occasions. In both cases, identifying behavioural responses, such as transience or trap response, may provide important biological insight into the animals being studied. If such responses are ignored within a model, then biases would result in the estimates of the parameters of interest, and therefore it is essential to fit appropriate models to the data.
Equivalence of score tests and diagnostic tests Diagnostic tests for capture-recapture data have become a standard preliminary tool before model fitting and consist of a number of contingency table tests based on summary statistics. They are commonly used because of readily available computer software, RELEASE, which can be run from within program MARK  (White and Burnham, 1999)  and U-CARE  (Choquet et al. , 2009). Once the preliminary diagnostic tests have been conducted, the traditional approach then relies on fitting all biologically plausible models (excluding those which have been ruled out by the diagnostic tests), comprising a model set which can be prohibitively large for successful implementation. An alternative step-up model selection strategy using score tests has been successfully used for ring recovery models  (Catchpole and Morgan, 1996)  and multistate capture-recapture models  (McCrea and Morgan, 2011).
Acknowledgements We thank  Thomas Bregnballe  for providing the cormorant data and  Claire Garrigue  from  Opération Cétacés  (www. operationcetaces. nc) for providing the whale data. McCrea is funded by  Natural Environment Research Council   fellowship  grant  NE/J018473/1  and all three authors were funded by the  Royal Society international  joint project grant  JP090515 , 'New statistical methods for wildlife population demography'.
First author partially supported by  European Research Council (ERC)  grant # 259527  and the  Austrian Science Fund (FWF) : M1717-N25. Both authors partially supported by  Agence Nationale de la Recherche (ANR)  grant  ANR-2010-BLAN-116-01  GGAA. They also thank the  Erwin Schrödinger Institute workshop "Geometry and Computation in Groups ".
Background: Publication bias and other reporting bias have been well documented for journal articles, but no study has evaluated the nature of results posted at ClinicalTrials. gov. We aimed to assess how many randomized controlled trials (RCTs) with results posted at ClinicalTrials. gov report statistically significant results and whether the proportion of trials with significant results differs when no treatment effect estimate or p-value is posted. Methods: We searched ClinicalTrials. gov in June 2015 for all studies with results posted. We included completed RCTs with a superiority hypothesis and considered results for the first primary outcome with results posted. For each trial, we assessed whether a treatment effect estimate and/or p-value was reported at ClinicalTrials. gov and if yes, whether results were statistically significant.
If no treatment effect estimate or p-value was reported, we calculated the treatment effect and corresponding p-value using results per arm posted at ClinicalTrials. gov when sufficient data were reported. Results: From the 17,536 studies with results posted at ClinicalTrials. gov, we identified 2823 completed phase 3 or 4 randomized trials with a superiority hypothesis. Of these, 1400 (50%) reported a treatment effect estimate and/or p-value. Results were statistically significant for 844 trials (60%), with a median p-value of 0. 01 (Q1-Q3: 0. 001-0. 26). For the 1423 trials with no treatment effect estimate or p-value posted, we could calculate the treatment effect and corresponding p-value using results reported per arm for 929 (65%). For 494 trials (35%), p-values could not be calculated mainly because of insufficient reporting, censored data, or repeated measurements over time.
For the 929 trials we could calculate p-values, we found statistically significant results for 342 (37%), with a median p-value of 0. 19 (Q1-Q3: 0. 005-0. 59). Conclusions: Half of the trials with results posted at ClinicalTrials. gov reported a treatment effect estimate and/or p-value, with significant results for 60% of these. p-values could be calculated from results reported per arm at ClinicalTrials. gov for only 65% of the other trials. The proportion of significant results was much lower for these trials, which suggests a selective posting of treatment effect estimates and/or p-values when results are statistically significant.