dataesr/scientific-paragraphs-categorization huggingface dataset:
The most straightforward screening prioritization approach trains a machine learning model on the included and excluded references from previous iterations of the systematic review, and then uses this model to reduce the workload in future review updates  [8]. For natural reasons, this approach can only be used in review updates, and not in new systematic reviews. By contrast, in the active learning approach, the model is continuously retrained as more and more references are screened. In a new systematic review, active learning starts with no training data, and the process is typically bootstrapped ("seeded") by sampling the references randomly, by using unsupervised models such as clustering or topic modeling or by using information retrieval methods with the database query or review protocol as the query  [36].
Comparing the relative performance of different methods is difficult since most are evaluated on different datasets, under different settings, and often report different measures. There have been attempts to compare previous methods by replicating reported methods on the same datasets, but the replication of published methods is often difficult or impossible due to insufficient reporting  [37]. Another way to compare the relative performance of methods is through the use of a shared task, a community challenge where participating systems are trained on the same training data and evaluated blindly using predecided metrics  [38, 39]. The shared task model removes many of the problems of replication studies and also safeguards against cheating, mistakes, and the cherry-picking of metrics or data, as well as publication bias.
Data used in the study The Limsi-Cochrane dataset. This dataset consists of 1939 meta-analyses from 63 systematic reviews of diagnostic test accuracy from the Cochrane Library (the full dataset is available online: DOI: 10. 5281/zenodo. 1303259) [42]. The dataset comprises all studies that were included in the systematic reviews, from any database or from gray literature, as well as the 2 × 2 tables (the number of true positives, false positives, true negatives, and false negatives) extracted from each included study by the systematic review authors, grouped by meta-analysis. This dataset can be used to replicate the meta-analyses in these systematic reviews, in full or over subsets of the data, for instance, to evaluate heterogeneity or bias of subgroups. The CLEF dataset. This dataset consists of all references from PubMed considered for inclusion-both those included in the systematic review and those ultimately judged not relevant to the systematic review-in 80 systematic reviews of diagnostic test accuracy also from the Cochrane Library (the full dataset is available online: https://github.
com/CLEF-TAR/tar) [6, 7]. Due to the way the data was collected, this dataset only contains references from PubMed, but not from other databases or gray literature. The dataset only includes the PubMed identifiers for each reference and whether the studies were included in the reviews. Combined dataset. For our experiments, we combined the two datasets by collecting the reviews, meta-analyses, and references common to both. In total, this intersection comprises 48 systematic reviews and 1354 meta-analyses of diagnostic tests. All analyses in this study will be based on this intersection unless otherwise specified. Since the CLEF dataset only includes references from PubMed, the meta-analyses performed in this study will only be based on studies from PubMed. Some metaanalyses may therefore be smaller than they were in the original reviews.
Characteristics of the systematic reviews In the 63 systematic reviews in the Limsi-Cochrane dataset, the minimum number of meta-analyses was 1 (3 reviews), the mode was 2 (11 reviews), the median was 6, and the maximum was 170. We used the combined dataset for all analyses. This dataset comprises 48 systematic reviews and 1354 metaanalyses of diagnostic test accuracy, but only 400 of the meta-analyses were based on at least 3 primary studies in PubMed and thus included in our analysis. Ninety-six of the meta-analyses were based on ten or more studies in PubMed. While we only consider studies from PubMed in this study, which decreases the number of studies per meta-analysis, the large majority of meta-analyses in the original systematic reviews were based on only one or two studies collected from multiple databases  [42].
1 only during the last 10% of the screened references for 10 out of 41 meta-analyses based on at least 10 studies (random, displacement threshold (0. 01) in Fig. 5 ). However, the same was only true for 1 out of 41 meta-analyses when using ranked order (ranked, displacement threshold (0. 01) in Fig. 5 ). In case 1, we could arguably stop screening (and possibly refine the database search) as soon as it becomes clear that a sufficient number of relevant studies cannot be retrieved. We cannot know with absolute certainty how many remaining studies exist for us to find. However, the found/effort curve will typically be convex when the candidate list is ranked, and extrapolating from its current slope therefore provides a probabilistic upper bound of the number of remaining studies  [45]. Case 2 assumes a sufficiently large amount of evidence to base the summary estimates upon.
If data saturation fails to occur because too few studies exist to find, the screening will not be interrupted. We can however also interrupt screening if it becomes clear that no further studies will be uncovered by the screening process, i. e. , by using a stopping criterion like found vs effort. For instance, using a combination of stopping criteria (displacement (0. 01) OR relevant (n = 15) OR found/effort (1/1000)) would have reduced the screening effort by 21. 5-99. 9285% (mean 81. 7%, median 90. 56%) for the main meta-analysis in 33 out of 38 systematic reviews with an average 1. 2% estimation error (Fig. 6 ). The five systematic reviews where the effort would not have been reduced were among the smallest with a total number of candidate studies ranging from 64 to 981. Ten systematic reviews performed no meta-analysis with at least three studies in PubMed and were therefore excluded from this analysis.
Limitations of this study This study focused on systematic reviews of diagnostic test accuracy studies. Therefore, we do not know what the implications are for other types of systematic reviews. However, the methods in this study are applicable to systematic reviews estimating numerical values, and our results may therefore be applicable also to systematic reviews of interventions. Due to the nature of the datasets, we could only recalculate meta-analyses using data from studies indexed in PubMed. Previous studies examining the impact of only searching PubMed on meta-analyses of interventions demonstrated moderate changes in estimates, and observed changes were equally likely to favor controls as interventions  [22, 24]. In this study, we assume that searching only PubMed is similarly unbiased for diagnostic test accuracy, but we are aware of no studies examining this directly.
Conclusions Our results suggest that diagnostic summary sensitivity and specificity can be estimated within an accuracy of 2 percentage points while deliberately missing over 40% of the relevant studies within a single database. This is contrary to current guidelines which assume that an exhaustive search is necessary to produce reliable estimates with low bias. On the other hand, we find a clear relationship between the absolute size of the meta-analysis and the reliability and precision of the estimates. In other words, a reliable meta-analysis requires identifying a sufficient number of studies, but how large a fraction of relevant studies is identified is less important. In the simulations, a combination of stopping criteria reduced the screening effort by 71. 2% on average (median 86.
Acknowledgments The authors would like to thank the two reviewers for useful comments and suggestions on a preliminary version of the manuscript. G. P f ∆ i, j * t ε,i,α = 1 = α 2 and sup f ∈A i( R,C,s,r ε,i) P f ∆ i, j * t ε,i,α = 0 ≤ α 2 provided that C, which does not depend on ε, is large enough. Proof: Consider ε ∈ (0, ε o ). Suppose that f ∈ N i (R). Then ε -2 T i, j * has a Chi-Square distribution with 2 | j * | degrees of freedom. Hence P f ∆ i, j * t ε,i,α = 1 = P f T i, j * > t ε,i,α = P f ε -2 T i, j * > χ 1-α 2 2 | j * | = α 2. Suppose now that f ∈ A i (R, C, s, r ε,i ). Then ε -2 T i, j * is a noncentral Chi-Square distribution with 2 | j * | degrees of freedom. Moreover, if V i j * characterizes the following linear span of ψ i j,k : j ∈ J i : j u < max( j * u , 1), ∀u and k ∈ K j , then, E f (T i, j * ) = ε 2 2 | j * | + j∈J i : j u <max( j * u ,1),∀u k∈K j θ i j,k 2 = ε 2 2 | j * | + Pro j V i j * ( f i ) 2 2 , Var f (T i, j * ) = 2ε 4 2 | j * | + 4ε 2 j∈J i : j u <max( j * u ,1),∀u k∈K j θ i j,k 2 = 2ε 4 2 | j * | + 4ε 2 Pro j V i j * ( f i ) 2 2.
Claeskens and J. -M. Freyermuth acknowledge the support of the Fund for Scientific Research Flanders, KU Leuven grant GOA/12/14 and of the IAP Research Network P7/06 of the Belgian Science Policy. Jean-Marc Freyermuth's research was supported by the Engineering and Physical Sciences Research Council [EP/K021672/2]. The proof of Theorem 4. 1 is a direct consequence of Propositions 9. 1 and 9. 2 that respectively deal with the upper bound and the lower bound of the minimax result.
Claeskens and J. -M. Freyermuth acknowledge the support of the Fund for Scientific Research Flanders, KU Leuven grant GOA/12/14 and of the IAP Research Network P7/06 of the Belgian Science Policy. Jean-Marc Freyermuth's research was supported by the Engineering and Physical Sciences Research Council [EP/K021672/2]. The proof of Theorem 4. 1 is a direct consequence of Propositions 9. 1 and 9. 2 that respectively deal with the upper bound and the lower bound of the minimax result. 9 Appendix 9. 1 Proof of Theorem 4. 1 Proposition 9. 1 Under the same assumptions of Theorem 4. 1, for any ε ∈ (0, ε o ), sup f ∈N i (R).
Acknowledgements We thank  Colin McDiarmid  and  Sergey Norin  for comments on an early version of this paper, and  Mihyun Kang  for interesting feedback. We also thank  Colin McDiarmid  for detecting a technical flaw in a previous version of the proof of Proposition 5. We thank the organizers and participants of the workshops Graph Theory and Combinatorics, Probability and Geometry held in March-April 2015 in Bellairs institute,  McGill University, Barbados , where this result was announced.
Acknowledgements The authors thank  Alex Arvanitakis ,  Davide Cassani ,  Yolanda Lozano ,  Carlos Nuñez ,  Hagen Triendl ,  Alessandro Tomasiello ,  Christoph Uhlemann ,  Oscar Varela  and  Daniel Waldram  for helpful discussions. EM would also like to thank the organisers of the "2018  USU Workshop  on.
Application of the PCF to fixation locations In this section we demonstrate how to compute the pair correlation function (PCFs) for eye movement datain three steps. The PCF reveals whether the distribution of fixation locations during a single scanpath can be explained by the overall inhomogeneity observed across all observers or whether fixation locations of a single scanpath contain additional spatial correlations. All analyses and graphs reported have been implemented in R using the spatstat  (Baddeley & Turner, 2005; Baddeley et al. , 2015)  and ggplot2 packages  (Wickham, 2009). We provide R-code at http://www. rpubs. com/hans/PCF. 1. Simulate inhomogeneous and homogeneous control processes. To evaluate our PCF computations, we simulate two control point processes, namely a homogeneous and an inhomogeneous point process.
Points (fixation locations) are sampled independently from each other in both control processes and due to the independence of points, we do not expect to observe any correlations between points at distance r. Any observed correlations would be spurious and depend on the data structure (e. g. , length of fixation sequences) or a wrong parameterization of the method. Hence, both control processes ensure that correlations in the PCF arise from the empirical data and not by the method itself. In addition, the inhomogeneous point process is used in the second step to estimate an optimal bandwidth for the intensity estimation of the PCF. Figure  2  (left panel) shows fixations on an image from participants viewing the same scene for 10 seconds (see Methods for details). As expected fixation locations are not uniformly distributed and indicate inhomogeneity.
The estimated intensity λs (x) of all fixation locations is depicted by gray shading where darker areas represent higher intensities. We used Scott's rule of thumb to compute the smoothing bandwidth for the intensity estimation (R-function: bw. scott) from the spatstat package  (Baddeley & Turner, 2005)  and estimated an optimal bandwidth for each image. The estimated intensity λs (x) is used to simulate an inhomogeneous control point process. The inhomogeneous point process (central panel) samples points proportionally to the intensity λs (x). Hence, the resemblance of the experimental and simulated distributions. The homogeneous point process (right panel) samples from a uniform distribution across the entire image area, therefore the subsequently estimated intensity (gray shading) is approximately constant, λs (x) ≈ λ.
Apparatus Stimuli were presented on a 20" CRT monitor (Mitsubishi Diamond Pro 2070; refresh rate 120 Hz; resolution: 1280×1024 pixels). Eye movements were recorded binocularly using the video-based Eyelink 1000 system (SR Research, Osgoode, ON, Canada) with a sampling rate of 1000 Hz. In order to reduce head movements we asked participants to position their head on a chin-rest in front of the computer screen (viewing distance: 70 cm). Stimulus presentation and response collection were implemented in MATLAB (The MathWorks, Natick, MA, USA) using the Psychophysics Toolbox  (PTB-3; Brainard, 1997; Kleiner et al. , 2007; Pelli, 1997)  and Eyelink Toolbox  (Cornelissen, Peters, & Palmer, 2002).
Data preprocessing We detected saccades by using a velocity-based algorithm  (Engbert & Kliegl, 2003; Engbert & Mergenthaler, 2006). Saccades were defined as fast movements of both eyes that exceeded the average velocity during a trial by 6 standard deviations for at least 6 ms with a minimum amplitude of 0. 5 •. Eye traces between successive saccades were tagged as fixations. Fixation positions were computed by averaging the mean eye position of both eyes. Since trials started with a fixation check, first fixations on images were removed from the data set (N = 2100). In addition, fixations containing a blink or with a blink during an adjacent saccade were excluded from subsequent analyses (N = 2214). Overall, 55. 526 fixations remained for further analyses.
Statistical modeling For statistical analyses, we computed linear mixed effect models for each dependent variable using the lme4 package  (Bates, Mächler, Bolker, & Walker, 2015)  in R (R Core Team, 2018). We log-transformed both dependent variables, since they deviated considerably from normal distributions. For the statistical model of the empirical data, we used the maximal possible random effect structure  (Barr, Levy, Scheepers, & Tily, 2013)  and ensured that none of the models was degenerate  (Bates, Kliegl, Vasishth, & Baayen, 2015). For our results we interpret all |t|>2 as significant fixed effects  (Baayen, Davidson, & Bates, 2008). In R the linear mixed effect model for the empirical data can be written as log(dv) ∼ 1 + A + B + A:B fixed effects + ( 1 + A + B + A:B | id ) random effects id + ( 1 + A | img ) random effects image (11) with a fixed effects of presentation A, image type B and their interaction A:B and the corresponding random effect structure.
Acknowledgments This work was funded by  Deutsche Forschungsgemeinschaft (DFG, German Research Foundation , grant  EN 471/13-1  to R. E. ) and the  Collaborative Research Centre   SFB 1294 , project number  B05  and by  Bundesministerium für Bildung und Forschung (BMBF) via Bernstein Center for Computational Neuroscience Berlin  (Project  B3 ,  Förderkennzeichen 01GQ1001F  to R. E. & F. A. W. ). Simon Barthelme  was supported by an  ANR  grant ( GenGP ,  ANR-16-CE23-0008 ).
Data The dataset described in this paper provides an estimate of the statistical power of Goodness-Of-Fit (GoF) tests. Its analytical calculation is in fact usually not easy: for most of GoF tests a closed form expression does not even exist. This estimation is necessary to properly select the sample size for testing procedures, thus reducing the type-II errors, i. e. the inability to reject the null hypothesis when it is actually false. The availability of this dataset can be advantageous for several fields, where the selection of the sample size is often performed with empirical procedures and where the results are often interpreted in a too optimistic view  [1]. The GoF tests aim at identifying the deviation of data samples from a given distribution. However, if the test is not able to identify such null hypothesis violation, nothing can be stated and the statistical power becomes the only quantitative value that gives us the test result reliability information.
The GoF tests have not been studied in case 0 scenario (called also external validation) for EVT distributions, i. e. when the samples used to perform the test are a different set w. r. t. the samples used to estimate the reference distribution. In particular, to the best of our knowledge, quantitative information of only case 3 scenarios is available in literature  [2] , while no case 0 power analysis for such distribution classes is available in literature. This dataset wants exactly to fill this gap. The statistical power computation has been performed with Monte Carlo approximations on a very large number of samples (10 9 ). This guaranteed a high level of accuracy of the results. This, together with the external validation scenario, is an interesting feature for recent applications of the EVT.
One of the possible use-case of this dataset is probabilistic real-time computing  [3] , where EVT is used to estimate the probabilistic Worst-Case Execution Time (WCET) of the computer tasks. In this scenario, the confidence level of the statistical test is critical. A false-negative result may indeed lead to an underestimation of the WCET, which may be unacceptable for the production system  [4]. This is the reason why we decided to build the statistical power dataset with the highest possible accuracy, enabling the selection of suitable sample size and ensuring a sufficient test result reliability  [5].
Value of the data The dataset described in this paper provides an estimate of the statistical power of Goodness-Of-Fit (GoF) tests performed on Extreme Value distributions. Several fields can benefit from the availability of this dataset, especially where it is necessary to select a proper sample size for the execution of GoF tests. The statistical power data have been computed on a case 0 scenario (also called external validation), i. e. when the samples used to perform the test are a different set with respect to the samples used to estimate the reference distribution. The Monte Carlo approximation used to compute the statistical power has been performed on a very large number of sample (10 9 ) to guarantee a high level of accuracy of the results. actually false. This concept can be expressed with the following conditional probability: Pðnot reject H 0 jH 0 is falseÞ.
The first author is partially supported by the ANR project COSPIN (ANR-13-JS01-0008-01). The first and second authors are supported by the ERC grant Geodycon. The third author was supported by the grants KAW 2013. 0321 and KAW 2016. 0446 from the Knut and Alice Wallenberg Foundation. This definition only depends on the co-orientation of ξ and not on the parametrisation of Λ s , nor on the choice of contact form α (as long as it induces the positive co-orientation). The most basic example of a positive isotopy is the displacement of a Legendrian submanifold by the Reeb flow associated to a choice of contact form. A positive isotopy for which Λ 0 " Λ 1 will be called a positive loop of Legendrians containing Λ 0. Since Legendrian isotopies are realised by ambient contact isotopies, the property of admitting such a positive loop does not depend on the representative of the Legendrian isotopy class of Λ 0.
Finally, by dynamically combining estimates from ARGO and Net, they showed that an ensemble approach, named ARGONet, led to improved results. Our contribution. In this study, we propose a forecasting platform that combines multiple data sources and statistical models to track flu activity in France at a spatial resolution that, to our knowledge, has not been explored before. Our forecasting platform produces accurate region-specific real-time and short-term flu activity forecasts for the 12 continental French regions, by leveraging national-level flu-related Google searches, electronic health records data, Twitter data, and region-specific climate data. Additionally, historical synchronicities across regions are captured with a Network model. A machine learning ensemble approach is proposed to improve predictions by dynamically combining estimates from these two distinct approaches.
Google Data We obtained the frequency per week of the 100 most correlated internet queries (if correlation ≥ 0. 60) by French users from Google Correlate (https://www. google. com/trends/correlate). November 19, 2019 3/109 Because our prediction period spans 05 January 2015 to 20 February 2017, we utilized the ILI signal for each French region, from January 2004 to December 2014 to obtain the most highly correlated search terms using the tool Google Correlate. In this way, we obtained different search terms for each individual region. The signals obtained correspond to queries performed by French users at the national level. We retrieved Google Correlate data in August 2018 for the period going from 05 January 2004 to 13 March 2017.
Electronic Health Record Data We retrieved EHR data from the clinical data warehouse (CDW) of Rennes University Hospital (France),This CDW, called eHOP, integrates structured (laboratory test results, prescriptions, ICD-10 diagnoses) and unstructured (discharge letter, pathology reports, operative reports) patients' data. It includes data from 1. 2 million inpatients and outpatients and 45 million documents that correspond to 510 million structured elements. eHOP consists of a powerful search engine system that can identify patients with specific criteria by querying unstructured data with keywords, or structured data with querying codes based on terminologies. The first approach to obtain eHOP data connected with ILI was to perform different manual queries to retrieve patients who had at least one document in their EHR that matched the following search criteria: (1) Queries directly connected with flu or ILI with the keywords "flu" or "ILI"; (2) Queries connected with flu symptoms with the keywords "fever", "pyrexia", "body aches" or "muscular pain"; (3) Queries connected with flu drugs with the keyword "Tamiflu"; (4) Queries with the ICD-10 terminology; (5) Queries connected with flu tests, positive or negative results.
In total, we performed 34 manual queries. For each query, the eHOP search engine returned all documents containing the chosen keywords (often, several documents for one patient and one stay). For query aggregation, we kept the oldest document for one patient and one stay and then calculated, for each week, the number of stays with at least one document mentioning the keyword contained in the query. From the CDW eHOP, we built a database containing the time series constructed from the structured data. In all, we have 1 335 347 time series. As Google Correlate, the Pearson correlation between each signal of each region and the time series from the database was calculated. In this way, for each region, the second approach was to retrieve the 100 most correlated signals to ILI signal. Because our test period is from 05 January 2015 to 20 February 2017, we calculated the correlation between January 2004 and December 2014.