dataesr/scientific-paragraphs-categorization huggingface dataset:
Simulated datasets allowed the comparison of the partitioning similarity between pipelines' results and the ground truth. Clustering metrics (NMI and AMI) were computed between taxa and ground truth at the family and genus level: the closer to 1, the closer the partitioning is to ground truth. The NID (Normalized Information Distance) was also computed  [48]. To be concordant with the F-measure and other clustering indexes (the closer to 1, the closer to ground truth), the 1-NID value has been used in this paper. A diversity index (Inverse Simpson) and an abundance-based richness estimator (Chao1) were also computed. Clustering, diversity and richness indexes were calculated on OTUs immediately after OTU clustering for clustering-first pipelines. To compare their performance with assignment-first pipelines, they were also computed after a taxonomic-merging step: all OTUs assigned to the same taxon were merged into a single taxonomic unit.
Hierarchical clustering of all pipelines In the subsection "On a real dataset, pipelines also segregate according to their algorithmic approach" of the Results & Discussion section, all pipelines were clustered using R version 3. 0. 2 with an average linkage hierarchical clustering, based on a Euclidean distance matrix computation between all pipelines' results for the real dataset. The results consisted of the relative abundance of each family discovered by each pipeline (excluding unclassified reads).
First, manually annotated sample size data for 583 studies were obtained from published meta-analyses  14. Second, sample sizes were automatically extracted from the Neurosynth database  15  for 548 studies published between 2011 and 2015 (by searching for regular expressions reflecting sample size, e. g. "13 subjects", "n=24") and then manually annotated to confirm automatic estimates and identify single-group versus multiple-group studies. (Data and code to generate all figures in this paper are available from the Open Science Framework at https://osf. io/spr9a/. ) Figure  1a  shows that sample sizes have steadily increased over the past two decades, with the median estimated sample size for a single-group fMRI study in 2015 at 28. 5. A particularly encouraging finding from this analysis is that the number of recent studies with large samples (greater than 100) is rapidly increasing (from 8 in 2012 to 17 in 2015, in the studied sample), suggesting that the field may be progressing towards adequately powered research.
To calculate the voxelwise statistical significance threshold for the active region in this model statistical map, we used the function ptoz from the FSL  17  software package, which computes a FWE threshold for a given volume and smoothness using the Euler Characteristic derived from Gaussian random field theory  18. This approach ensures that the probability of a voxel in the non-active brain region exceeding this significance threshold is controlled at 5%; the resulting significance threshold, t α , is equal to 5. 12. ---Figure  1  about here ---The statistical power is defined as the probability that the local maximum peak of activation in the active region exceeds this significance threshold. This probability was computed using a shifted version of the null local maximum distribution, with shift of D*sqrt(n) to reflect a given effect size and sample size.
Within these intersection masks, we then determined the average task-related increases in BOLD signal -and the effect size (Cohen's D) -associated with each different task. Additional details are provided in Box 2. The figure in Box 2, which lists the resulting BOLD signal changes and inferred effect sizes, demonstrates that realistic effect sizes -i. e. BOLD changes associated with a range of cognitive tasks -in fMRI are surprisingly small: even for powerful tasks such as the motor task which evokes median signal changes of greater than 4%, 75% of the voxels in the masks have a standardised effect size smaller than 1. For more subtle tasks, such as gambling, only 10% of the voxels in our masks demonstrated standardised effect sizes larger than 0. 5. Thus the average fMRI study remains poorly powered for capturing realistic effects, particularly given that the HCP data are of particularly high quality, and thus the present estimates are likely greater than what would be found with most standard fMRI datasets.
Problem: Flexibility in data analysis The typical fMRI analysis workflow contains a large number of preprocessing and analysis operations, each with choices to be made about parameters and/or methods (see Box 3). Carp  35  applied 6,912 analysis workflows (using the SPM  36  and AFNI  37  software packages) to a single data set and quantified the variability in resulting statistical maps. This revealed that some brain regions exhibited more substantial variation across the different workflows than did other regions. This issue is not unique to fMRI; for example, similar issues have been raised in genetics  38. These "researcher degrees of freedom" can lead to substantial inflation of Type I error rates 8 whenever an analysis decision is made based partly on the observed data--even when there is no intentional "p-hacking", and only a single analysis is ever conducted  9.
Solutions: We recommend pre-registration of methods and analysis plans. The details to be pre-registered should include planned sample size, specific analysis tools to be used, specification of predicted outcomes, and definition of any ROIs that will be used for analysis. The Open Science Framework (http://osf. io) and AsPredicted. org provide established platforms for pre-registration; the former allows an embargo period in which the registration remains private, obviating some concerns about ideas being disclosed while still under investigation. In addition, some journals now provide the ability to submit a "Registered Report", in which the hypotheses and methods are reviewed prior to data collection, and the study is guaranteed publication regardless of the outcome  39  ; see  40, 41  for examples of such reports, and https://osf.
For the fMRI data, we simulated statistical values at each voxel for a comparison of activation and baseline conditions for each of the simulated subjects within the standard MNI152 mask, and then spatially smoothed the image with a 6mm Gaussian kernel, based on the common smoothing level of 3 times the voxel size. A univariate analysis was performed using FSL to assess the correlation between activation in each voxel and the simulated behavioural regressor across subjects, and the resulting statistical map was thresholded at p < 0. 001 and with a 10-voxel extent threshold (which is a common heuristic correction shown by Eklund et al. 44  to result in highly inflated levels of false positives). This approach revealed a cluster of false positive activation in the superior temporal cortex in which the simulated fMRI data are highly correlated with the simulated behavioural regressor (Fig.
io/spr9a/). Only three presented fully uncorrected results, with four others presenting a mixture of corrected and uncorrected results; this suggests that corrections for multiple comparisons are now standard. However, there is evidence that researchers may engage in "method-shopping" for techniques that provide greater sensitivity, at a potential cost of increased error rates. Nine of the 65 papers used the FSL or SPM software packages to perform their primary analysis, but then used the alphasim or 3dClustSim tools from the AFNI software package (7 papers) or other simulation-based approaches (2 papers) to correct for multiple comparisons. This is concerning, because both FSL and SPM offer well-established methods that use Gaussian random field theory or nonparametric analyses to correct for multiple comparisons.
Given the substantial degree of extra work (e. g. software installation, file reformatting) involved in using multiple software packages, the use of a different tool raises some concern that this might reflect analytic phacking. This concern is further amplified by the finding that until very recently, this AFNI program had slightly inflated inflated Type I error rates  44. Distressingly, whereas nonparametric (randomization/permutation) methods are known to provide the more accurate control over familywise error rates compared to parametric methods  45, 49  and are applicable for nearly all models, they were not used in any of these papers.
Problem: Software errors As the complexity of a software program increases, the likelihood of undiscovered bugs quickly reaches certainty  55  , which implies that all software used for fMRI analysis is likely to have bugs. Most fMRI researchers use one of several open-source analysis packages for preprocessing and statistical analyses; many additional analyses require custom programs. Because most researchers writing custom code are not trained in software engineering, there is insufficient attention to good software development practices that could help catch and prevent errors. This issue came to the fore recently, when a 15-year-old bug was discovered in the AFNI program 3dClustSim (and the older AlphaSim), which resulted in slightly inflated Type I error rates  44, 56  (the bug was fixed in May 2015).
While small in this particular case, the impact of such bugs could be widespread; for example, PubMed Central lists 1362 publications mentioning AlphaSim or 3dClustSim published prior to 2015 (query [("alphasim" OR "3DClustSim") AND 1992:2014[DP] ] performed July 14, 2016). Similarly, the analyses presented in a preprint of the present article contained two software errors that led to different results being presented in the final version of the paper. The discovery of these errors led us to perform a code review and to include software tests in order to reduce the likelihood of remaining errors. Even though software errors will happen in commonly used toolboxes as well as in-house code, they are much more likely to be discovered in widely used packages due to scrutiny by many more users. It is very likely that significant bugs exist in custom software built for individual projects, but due to the limited user base those bugs will never be unearthed.
Solutions: Researchers should avoid the trap of the "not invented here" philosophy: When the problem at hand can be solved using software tools from a well-established project, these should be chosen instead of re-implementing the same method in custom code. Errors are more likely to be discovered when code has a larger userbase, and larger projects are more likely to follow better software-development practices. Researchers should learn and implement good programming practices, including the judicious use of software testing and validation. Validation methodologies (such as comparing with another existing implementation or using simulated data) should be clearly defined. Custom analysis code should always be shared upon manuscript submission (for an example, see  57  ). It may be unrealistic to expect reviewers to evaluate code in addition to the manuscript itself, although this is standard in some journals such as the Journal of Statistical Software.
Solutions: The neuroimaging community should acknowledge replication reports as scientifically important research outcomes that are essential in advancing knowledge. One such attempt is the OHBM Replication Award to be awarded in 2017 for the best neuroimaging replication study in the previous year. In addition, in case of especially surprising findings (for example extra-sensory perception), findings that could have influence on public health policy or medical treatment decisions, or findings that could be tested using data from another existing dataset, reviewers should consider requesting replication of the finding by the group before accepting the manuscript.
Towards the neuroimaging paper of the future In the foregoing we have outlined a number of problems with current practice and made suggestions for improvements. Here we outline what we would like to see in the "neuroimaging paper of the future", inspired by related work in the geosciences  71. Planning. The sample size for the study would be determined in advance using formal statistical power analysis. The entire analysis plan would be formally pre-registered, including inclusion/exclusion criteria, software workflows including contrasts and multiple comparison methods, and specific definitions for all planned regions of interest. Implementation. All code for data collection and analysis would be stored in a version control system, and would include software tests to detect common problems.
The repository would use a continuous integration system to ensure that each revision of the code passes the appropriate software tests. The entire analysis workflow (including both successful and failed analyses) would be completely automated within a workflow engine and packaged within a software container or virtual machine in order to ensure computational reproducibility. All datasets and results would be assigned version numbers in order to allow explicit tracking of provenance. Automated quality control would assess the analysis at each stage to detect potential errors. Validation. For empirical papers, all exploratory results would be validated against an independent validation dataset that was not examined prior to validation. For methodological papers, the approach would follow best practices for reducing overly optimistic results  72.
Any new method would be validated against benchmark datasets and compared to other state-ofthe-art methods. Dissemination. All results would be clearly marked as either hypothesis driven (with a link to the appropriate pre-registration) or exploratory. All analyses performed on the dataset (including those that failed) would be reported. The paper would be written using a literate programming technique, in which the code for figure generation is embedded within the paper and the image data depicted in figures is transparently accessible. The paper would be distributed along with the full codebase to perform the analyses and the data necessary to reproduce the analyses, preferably in a container or virtual machine to allow direct reproducibility. Both unthresholded statistical maps and the raw data would be shared via appropriate community repositories, and the shared raw data would be formatted according to a community standard, such as the Brain Imaging Data Structure (BIDS)  73  , and annotated using an appropriate ontology to allow automated meta-analysis.
Finally, results need to be replicated to ensure robust discoveries. Box 2: Effect-size estimates for common neuroimaging experimental paradigms. The aim of this analysis is to estimate the magnitude of typical effect sizes of blood oxygen level-dependent changes in fMRI signal associated with common psychological paradigms. We focus on four experiments administered by the Human Connectome Project (HCP)  78  : an emotion task, gambling task, working memory task and motor task (detailed below). We chose data from the HCP for its large sample size, which allows computation of stable effect size estimates, and its diverse set of activation tasks. The data and code used for this analysis are available at https://osf. io/spr9a/. Briefly, the processing of data from the Human Connectome Project was carried out in 4 main steps: 1.
Group Analyses: The first-level analyses, which summarise the relation between the experimental design and the measured time series for each subject, were obtained from the Human Connectome Project. The processing and analysis pipelines for these analyses are shared together with the data. Here we perform second-level analyses -that is, an assessment of the average effect of the task on BOLD signal over subjects -using the FSL program flame1  17  which performs a linear mixed-effects regression at each voxel, using generalized least squares with a local estimate of random effects variance. This analysis averages over subjects, while separating within-subject and between-subject variability to ensure control of unobserved heterogeneity. The specific contrasts that were tested are: • Motor: average BOLD response for tongue, hand and foot movements versus rest • Emotion: viewing faces with a fearful expression versus viewing neutral faces • Gambling: monetary reward versus punishment • Working memory: a contrast between conditions in which the participants indicate whether the current stimulus matches the one from 2 trials earlier ("2-back"), versus a condition where the participants indicate whether the current stimulus matches a specific target ("0-back").
Box 3: Flexibility in fMRI data analysis In the early days of fMRI analysis, it was rare to find two laboratories that used the same software to analyze their data, with most using locally-developed custom software. Over time, a small number of open-source analysis packages have gained prominence (SPM, FSL, and AFNI being the most common), and now most laboratories use one of these packages for their primary data processing and analysis. Within each of these packages, there is a great deal of flexibility in how data are analyzed; in some cases there are clear best practices, but in other cases there is no consensus regarding the optimal approach. This leads to a multiplicity of analysis options. In Table  B1  we outline some of the major choices involved in performing analyses using one of the common software packages (FSL).
It divides practice into seven categories, and provides detailed checklists that can be consulted when planning, analyzing and writing up a study. The table below lists these categories with summaries of the topics covered in the checklists. we estimated the standardized effect size required to detect an effect with 80% power for a whole-brain linear mixed-effects analysis using a voxelwise 5% familywise error rate threshold from random field theory  16  (see main text for details). Median effect size for which studies were powered to find in 2015 was 0. 75. Data and code to generate these figures are available at https://osf. io/spr9a/ ; see Supplementary materials for a version with all datapoints depicted. Seemingly impressive brain-behavior association can arise from completely random data through the use of uncorrected statistics and circular ROI analysis to capitalize on the large sampling error arising from small samples.
Acknowledgements RP, JD, JBP and KG are supported by the  Laura and John Arnold Foundation. MRM is supported by the  MRC  ( MC UU 12013/6 ) and a member of the  UK Centre for Tobacco and Alcohol Studies , a  UK Clinical Research Council Public Health Research: Centre of Excellence. Funding from  British Heart Foundation ,  Cancer Research UK ,  Economic and Social Research Council, Medical Research Council , and the  National Institute for Health Research , under the auspices of the  UK Clinical Research Collaboration , is gratefully acknowledged. CIB is supported by the  Intramural Research Program  of  NIMH  ( ZIA-MH 002909 ). TY is supported by  NIMH  ( R01MH096906 ). PMM gratefully acknowledges personal support from the  Edmond J. Safra Foundation and Lily Safra  and research support from the  Medical Research Council , the  Imperial College Healthcare Trust Biomedical Research Centre  and the  Imperial EPSRC Mathematics in Healthcare Centre.
TEN is supported by the  Wellcome Trust  ( 100309/Z/12/Z ). JBP is supported by  NIH-NIBIB   P41-EB019936  and by  NIH-NIDA   U24-038653. Data were provided [in part] by the  Human Connectome Project ,  WU-Minn Consortium  (Principal Investigators:  David Van Essen  and  Kamil Ugurbil ;  1U54MH091657 ) funded by the 16 Institutes and Centers of the  US National Institutes of Health (NIH)  that support the  NIH Blueprint for Neuroscience Research ; and by the  McDonnell Center for Systems Neuroscience at Washington University. Thanks to  Joe Wexler  for performing annotation of Neurosynth data,  Sean David  for providing sample size data, and  Robert Cox  and  Paul Taylor  for helpful comments on a draft of the manuscript.
We present a logical relations model of a higher-order functional programming language with impredicative polymorphism, recursive types, and a Haskell-style ST monad type with runST. We use our logical relations model to show that runST provides proper encapsulation of state, by showing that effectful computations encapsulated by runST are heap independent. Furthermore, we show that contextual refinements and equivalences that are expected to hold for pure computations do indeed hold in the presence of runST. This is the first time such relational results have been proven for a language with monadic encapsulation of state. We have formalized all the technical development and results in Coq.
INTRODUCTION Haskell is often considered a pure functional programming language because effectful computations are encapsulated using monads. To preserve purity, values usually cannot escape from those monads. One notable exception is the ST monad, introduced by Launchbury and Peyton Jones  [1994]. The ST monad comes equipped with a function runST : (∀β, ST β τ ) → τ that allows a value to escape from the monad: runST runs a stateful computation of the monadic type ST β τ and then returns the resulting value of type τ. In the original paper  [Launchbury and Peyton Jones 1994] , the authors argued informally that the ST monad is "safe", in the sense that stateful computations are properly encapsulated and therefore the purity of the functional language is preserved.
64:2 Amin Timany, Léo Stefanesco, Morten  Krogh-Jespersen, and Lars Birkedal  In this paper we present a logical relations model of STLang, a call-by-value higher-order functional programming language with impredicative polymorphism, recursive types, and a Haskellstyle ST monad type with runST. In contrast to earlier work, the operational semantics of STLang uses a single global mutable heap, capturing how the language would be implemented in reality. We use our logical relations model to show for the first time that runST provides proper encapsulation of state. Concretely, we state a number of contextual refinements and equivalences that are expected to hold for pure computations and we then use our logical relations model to prove that they indeed hold for STLang, i. e. , in the presence of stateful computations encapsulated using runST.
Moreover, we show a State-Independence theorem that intuitively expresses that, for any well-typed expression e of type τ , the evaluation of e in a heap h is independent of the choice of h, i. e. , e cannot read from or write to locations in h but may allocate new locations (via encapsulated stateful computations). Note that this is the strong result one really wishes to have since it is proved for a standard operational semantics using a single global mutable heap allowing for updates in-place, not an abstract semantics partitioning memory into disjoint regions as some earlier work  [Launchbury and Peyton Jones 1995; Moggi and Sabry 2001]. In STLang, values of any type can be stored in the heap, and thus it is an example of a language with so-called higher-order store. It is well-known that it is challenging to construct logical relations for languages with higher-order store.
Indeed, we have used the Iris proof mode to formalize all the technical results in this paper in Iris in Coq. In the remainder of this Introduction, we briefly recap the Haskell ST monad and recall why runST intuitively encapsulates state. We emphasize that STLang, unlike Haskell, is call-by-value; we show Haskell code to make the examples easier to understand. Finally, we give an overview of the technical development and our new results.