dataesr/scientific-paragraphs-categorization huggingface dataset:
e. , the 500 best scoring models considering all 20 sampling runs). Second, the ensemble of solutions was assessed in terms of how well they satisfy the data from which they were computed, including the cross-links as well as the excluded volume, sequence connectivity, and the 3D EM restraints. We validated the ensemble of solutions by comparing it with the ensemble of solutions obtained jackknifing the cross-link dataset. The obtained localization maps are similar to the one computed without jackknifing the dataset (Figure  2 -figure supplement 4), suggesting that the cross-link data are accurate and the models are not a result of overfitting. The 3D shape implied by the EM restraint was satisfied by the ensemble. The excluded volume and sequence connectivity restraints were also satisfied.
Average distance between clusters (Figure  2 -figure supplement 4C-E) The average distance between two clusters is calculated as the average RMSD, computed on the whole complex, between every pair of solutions taken from the two clusters. RMSF (Figure  2 -figure supplement 5A) RMSF of a given residue is the standard deviation of distances between the position of the residue in each solution of the cluster and its position in the cluster center. The position of the residue is the center of the bead that was used to represent it. Molecular visualizations were prepared using the UCSF chimera package  (Pettersen et al. , 2004).
Selection of the cluster of solutions In the IMP modeling procedure we perform a large number of independent modeling runs, each starting from a random configuration and undergoing thousands of rounds of small random model perturbations, in order to achieve exhaustive sampling of conformational space. As we are interested only in models that best satisfy the restraints, we select a small fraction of the models representing the very best scoring solutions (Figure  2  ). A single cluster showed both consistency with earlier localizations and significantly better violation statistics and was therefore deemed to be the Mediator architecture with the highest confidence. Molecular architecture of the RNA Pol II complex revealed by integrative modeling Modeling studies of RNA pol II employed cross-link data from two sources: 156 cross-links from previous experiments conducted with the free pol II fraction isolated during our holoenzyme SEC purification  (Trnka et al.
, 2014) , and 108 cross-links from a previous study  (Chen et al. , 2010). These sources had 63 cross-links in common. Hence, the current study used 201 unique cross-links. The atomic structures for the twelve-subunit yeast pol II have been previously determined by X-ray crystallography (PDB accession ID: 1WCM,  [Armache et al. , 2005] ). The crystallographic structure covers 89% of the residues in the pol II complex. A 20. 9 Å resolution density map from single particle EM reconstruction of the pol II complex was also considered (EMDB accession id: 1883,  [Czeko et al. , 2011] ). The 12 pol II subunits were represented as 15 domains, where Rpb1 was decomposed into 3 domains (residues: 1-1140, 1141-1274, 1275-1733), Rpb2 was decomposed into 2 domains (residues 1-1102, 1103-1224), and the remaining subunits were represented as single domains.
Acknowledgements We thank  JH Morris ,  CC Huang ,  EC Meng  and other members of the  Resource for Biocomputing, Visualization, and Informatics at UCSF  for support with Cytoscape and Chimera. We acknowledge  B Webb  for his help with IMP,  J Baker-LePain  for his help with the cluster at  UCSF , as well as  D Saltzberg  and  P Cimermancic  for discussions. This work was supported by the  US National Institutes of Health (NIH)  grants  R01 AI21144  (to RK),  P41 GM109824  and  R01 GM083960  (to AS), and  P41 GM103481  (to AB). We also acknowledge support from  Human Frontier Science Program  long-term fellowship  LT00160  (to PJR) and the  US National Science Foundation  through partnership in the  BioXFEL Science Technology Center  supported by grant  NSF-1231306  (to RK). Yeast fermentation was performed using an instrument purchased using funds from the  NIH   S10  shared instrumentation grant  S10RR028096.
Introduction Because current research is based on previous published studies, being able to reproduce an experiment and replicate a result is paramount to scientific progress. The extent to which results agree when performed by different researchers defines this tenet of the scientific method  [2, 3]. Recently, a number of authors have questioned the validity of many findings in epidemiology or in neuroscience  [4, 5]. Results can be found by chance (winner's curse effect), more often in poorly powered studies  [6] , or be declared significant after too many variations of the analysis procedure  [7, 8]  without controlling appropriately for the overall risk of error (p-hacking effect  [6, 9] ). Additionally, errors in code or in data manipulation are easy to make  [10] : it is in general difficult to check for the correctness of neuroimaging analyses.
Although we aim for reproduction of results with other data and independent analysis methods, the first step is to ensure that results can be replicated within laboratories. This seems an easy task, but it is in fact common that results cannot be replicated after, say, a year or two, when the student or post-doc responsible for the analyses and the data management has left. Increasing our capacity to replicate the data analysis workflow has another crucial aspect: this will allow us to better document our work, and therefore communicate and share it much more easily. It is crucial that we remember that resources are limited, and part of our work is to make it easy for others to check and build upon our findings. In computer science and related communities, a number of informatics tools and software are available (databases, control version system, virtual machines, etc.
Improving the capacity of other researchers to reproduce one's results involves some degree of sharing, through journals, repositories or dedicated websites (Annex 1). These practices, if followed, should be sufficient to allow any researcher to replicate a published fMRI experiment. Here we define replication as the capacity of a colleague to re-execute the analyses on the same dataset  [14] , but note that this definition varies in the literature  [15]. In step 2 below ('Improving scripts and turning them into workflows'), we expand on good practice for writing and sharing code. Although this can seem daunting for people who do not often write code, our goal is to give some tips to improve everyone's analysis scripts.
In the face of such variability, some have argued that 'anything less than release of actual source code is an indefensible approach for any scientific results that depend on computation, because not releasing such code raises needless, and needlessly confusing, roadblocks to reproducibility'  [22]. In contrast with data analysts or software developers, many neuroimagers do not code their analysis from scratch -instead they rely on existing software and often reuse code gathered from others in the laboratory or on the web. Pressing buttons in a graphical user interface is not something that can be replicated, unless inputs and processing steps are saved in log files. To ensure reproducibility (even for oneself in a few months' time) one needs to set up an automatic workflow. Informatics and bioinformatics researchers have been discussing issues of code reproducibility for many years  [23, 24] , and lessons can be learnt from their experience.
Sandve et al. [24]  have a few simple recommendations. First, keep track of every step, from data collection to results, and whenever possible keep track with electronic records. Most neuroimaging software has a so-called batch mode (SPM  [25, 26] ) or pipeline engine (Nipype  [27, 28] ), or is made up of scripts (AFNI  [29, 30] , FSL  [31, 32] ), and saving these is the best way to ensure that one can replicate the analysis. At each step, record electronically, and if possible automatically, what was done with what software (and its version). Second, minimize, and if possible eliminate, manual editing. For instance, if one needs to convert between file formats, this is better done automatically with a script, and this script should be saved. Third, for analyses that involve a random number generator, save the seed or state of the system, so that the exact same result can be obtained.
As for the computer program used to run the experiment (step 1), the batch and scripts can be made available as supplementary material in a journal, and/or shared in repositories. If one ends up with a fully functional script that includes a new type of analysis, this can itself be registered as a tool on dedicated websites such as the NeuroImaging Tool and Resources Clearinghouse (NITRC  [33] ). Sharing the analysis batch and scripts is the only way to ensure reproducibility by allowing anyone to (i) check for potential errors that 'creep in' to any analyses  [10] ; (ii) reuse them on new data, possibly changing a few parameters to suit changes in scanning protocol -similar results should be observed if the effects were true  [14]  -and (iii) base new analysis techniques or further research on verifiable code.
Improving scripts and turning them into workflows Although these recommendations are, we hope, useful, they are not generally sufficient. Analysis code depends on software, operating systems, and libraries that are regularly updated (see, e. g. [34]  for an effect on imaging results). When the code is rerun, these changes should be tracked, and results attached to a specific version of the code and its environment. The only complete solution is to set up virtual machine or equivalent. For neuroimaging, the NeuroDebian project  [35]  integrates relevant software into the Debian operating system, where all software is unambiguously versioned and seamlessly available from a package repository. This makes it possible to define the whole environment and reconstruct it at any later time using snapshots of the Debian archive  [36].
While such a solution is the most complete, investing in good revision control software is a first step that goes a long way in handling code (Wikipedia lists 36 types of such software  [37] ). We argue here that this investment is a necessity for reproducible science. Although a simple text editor or word processing document could be used to precisely describe each analysis step, only an executable script and information on the associated software environment can give one a reasonable chance of reproducing an entire experiment. This implies that much more should be done to teach programming to students or researchers who need to work with neuroimaging data. Barriers to code sharing are not as great as for data, but they do exist. Researchers are often concerned that their code is too poor, and that there might be some errors.
These, and the fear of being 'scooped' , are some of the main reasons scientists give for not sharing code with others  [38]. Yet, as  Barnes [39]  puts it, "software in all trades is written to be good enough for the job intended. So if your code is good enough to do the job, then it is good enough to release". A few simple rules can be applied to improve scripts  [23]. First, make your code understandable to others (and yourself ). Add comments to scripts, providing information not just about what is computed, but also reflecting what hypothesis is being tested, or question answered, by that specific piece of code  [24]. Second, version control everything. Version control systems (VCSs) store and back up every previous version of the code, allowing one to 'roll back' to an older version of the code when things go wrong.
Two of the most popular VCSs are Git [40] (which we recommend) and Subversion  [41]. 'Social coding' platforms, such as GitHub  [42]  or Bitbucket  [43] , are also useful sharing and collaboration tools. Third, test your code effectively, to assure yourself and others that it does what it is supposed to. The software industry tells us that "untested code is broken code", but scientists lack incentives to invest time in this. For example, if you coded some statistical tests to be run on multiple voxels, compare the routine in one voxel against a prototype solution. Learning how to test and document one's code is a crucial skill to reduce bugs and ensure safe reuse of code, an aspect that is not sufficiently emphasized and taught in curricula. In fact, the experience of the authors is that it is hardly ever mentioned.
Neuroimagers can also take advantage of a few easyto-use tools to create complex scripts and make a workflow (a workflow consists of a repeatable pattern of activities that transform data and can be depicted as a sequence of operations, declared as work of a person or group (adapted from  [44] ). For Matlab-based analyses, we can recommend using Matlab-specific formatting a in the code, and a workflow engine such as the Pipeline System for Octave and Matlab (PSOM  [45, 46] ) or the Automatic Analysis pipeline (AA  [47, 48] ). For Pythonbased analyses, we recommend the IPython notebook  ([49]  now the Jupyter project) to sketch the analysis and explore results, along with the workflows provided in Nipype  [27, 28]. Packages such as SPM  [25, 26]  have batch systems that create scripts of the whole analysis workflow, which should be learned for efficiency, reproducibility and provenance tracking.
It is also possible to create entire workflows using general (e. g. Taverna  [50] , Kepler  [51] ) or dedicated libraries (LONI pipeline  [52] ) and thereby obtain analysis provenance information. Using these pipelines, one can create (via a graphical interface or a script) a workflow of the different steps involved in fMRI data processing, specifying parameters needed at each step, and save the workflow. Dedicated libraries or scripts can be called, and the impact of changing a parameter value in a specific implementation of a step can be studied. Most of these pipeline systems have ways to help distribute the processing using computers' multicore architectures, or job-scheduling systems installed on clusters, thereby reducing computation time. In general, these tools require some programming and software expertise (local installation and configuration issues seem to be largely underestimated issues) beyond what fMRI researchers can usually do (whereas PSOM, Nipype and using the SPM batch system are 'easy').
Organize and share data and metadata Besides replicating an analysis (running exactly the same code on the same data), sharing data provides guarantees of reproducibility by (i) allowing a comparison with newly collected data (are the patterns observed in the new dataset the same, independently of statistical significance?), (ii) allowing alternative analyses to be tested on the same data, and (iii) aggregating them with other data for meta-analyses  [54]. Many funders now request that data are made available, and researchers must be prepared to do this and to identify where the data will be archived. When the data have obvious potential for reuse (e. g. [55] ) or pose special challenges (e. g. [56] ), their publication in journals such as Data in Brief, Frontiers in Neuroscience, F1000 Research, GigaScience, Journal of Open Psychology Data, or Scientific Data allow the creators to be acknowledged by citation.
In any case, data can simply be put in a repository such as NITRC  [33]  or Open-fMRI  [57]  (task-based fMRI  [58] ). As of March 2015, OpenfMRI hosts 33 full datasets, and a more complete format describing the data is being developed. Previously, the major project that supported sharing of full fMRI datasets was the fMRI Data Center  [59, 60]. It currently has 107 datasets available on request, but has not accepted submission of additional datasets since 2007. The researcher must also be aware of the constraints involved in sharing MRI data. It is of course essential that consent forms indicate clearly that the data will be de-identified and shared anonymously, and it is the responsibility of the principal investigator to ensure proper de-identification  [61] , that is, not only removing any personal information from the image headers, but also removing facial (and possibly dental and ear) information from the T1-weighted image.
Fortunately, personal information is removed automatically by most fMRI packages when converting from DICOM to NIfTI file format. Removing facial information can be trickier, but automated tools exist for this too (SPM  [25, 26] , MBRIN defacer  [62, 63] , Open fMRI face removal Python script b ). Another important issue to consider when sharing data is the metadata (information describing the data). Data reuse is only practical and efficient when data, metadata, and information about the process of generating the data are all provided  [64]. Ideally, we would like all of the information about how the data came to existence (why and how) to be provided. The World Wide Web Consortium Provenance Group  [65]  defines information 'provenance' as the sum of all of the processes, people (institutions or agents), and documents (data included) that were involved in generating or otherwise influencing or delivering a piece of information.
For fMRI data, this means that raw data would need to be available, along with (i) initial project information and hypotheses leading to the acquired data, including scientific background as well as people and funders involved; (ii) experimental protocol and acquisition details; and (iii) other subject information, such as demographics and behavioral or clinical assessments. There are currently no tools to do this metatagging, but we recommend checking with the database that will host the data and using their format from the start (that is, store data on your computer or server using the same structure). Functional MRI can have a complex data structure, and reorganizing the data post-hoc can be time-consuming (several hours for posting on Open-fMRI, if the reorganization is done manually  [66] ).
Discussion Researchers are much more likely to be able to replicate experiments and reproduce results if material and procedures are shared, from the planning of an experiment to the fMRI result maps. This is also crucial if the global efficiency of our research field is to improve. To be able to do this, the single most important advice to consider would probably be to plan ahead, as lack of planning often prevents sharing c. Informed consent and ethics should be compliant with data sharing. When previous data are available, statistical power should be computed, sample size chosen accordingly and reported. Data, scripts and maps should be organized and written with the intention to share and allow reuse, and they should have licenses allowing redistribution. To increase fMRI reproducibility, neuroscientists need to be trained, and to train others, to plan, document and code in a much more systematic manner than is currently done.
Neuroimaging is a computational data science, and most biologists, medical doctors and psychologists lack appropriate programming, software and data science training. In that respect, sharing work has an additional educational value. By studying the code used by others, in order to replicate their results, one also learns what practices are useful when sharing. Piwowar et al. [77]  showed that sharing data and code increases the trust and interest in papers, and citation of them. This also makes new collaborations possible more easily. Openness improves both the code used by scientists and the ability of the public to engage with their work  [39]. Putting the code associated with a paper in a repository is likely to have as many benefits as sharing data or publications. For instance, the practice of self-archiving can increase citation impact by a dramatic 50 to 250%  [78].
Annex 1 -list of websites mentioned in the article that can be used for sharing Bitbucket (https://bitbucket. org/) is "a web-based hosting service for projects that use either the Mercurial or Git revision control system" and allows managing and sharing code. Dryad (http://datadryad. org/) "is a curated resource that makes the data underlying scientific publications discoverable, freely reusable, and citable" under a Creative Commons license. It is a nonprofit membership organization from an initiative among a group of leading journals and scientific societies in evolutionary biology and ecology. This repository now hosts any kind of biological data. FigShare (http://figshare. com/) is a repository that "allows researchers to publish all of their data in a citable, searchable and sharable manner" under a Creative Commons license.
It is supported by Digital Science, part of Macmillan Publishers Limited. This repository now hosts any kind of data. GitHub (https://github. com/) is "a web-based Git repository hosting service" and allows managing and sharing code. Kepler (https://kepler-project. org/) is a scientific workflow application "designed to help scientists, analysts, and computer programmers create, execute, and share models and analyses across a broad range of scientific and engineering disciplines". LONI pipeline (http://pipeline. bmap. ucla. edu/) is an application to "create workflows that take advantage of all the tools available in neuroimaging, genomics  [and]  bioinformatics". NeuroDebian (http://neuro. debian. net/) integrates neuroimaging and other related neuroscientific and computational software into Debian (Linux).
It includes a repository of over 60 software and data packages. Neuro-Debian also provides a virtual machine, simplifying deployment within any existing Linux, OS X or Windows environment. NeuroImaging Tool and Resources Clearinghouse (http://www. nitrc. org/), is a web resource that "facilitates finding and comparing neuroimaging resources for functional and structural neuroimaging analyses". OpenScience framework (https://osf. io/) is a project management system for an "entire research lifecycle: planning, execution, reporting, archiving, and discovery". It supports local archiving, but also links with other repositories. Multiple options for licensing are available. It is supported by the Center for Open Science. Taverna (http://www. taverna. org. uk/) is a "domain-independent workflow management system -a suite of tools used to design and execute scientific workflows".
Endnotes a Matlab Publishing Markup refers to specific keys such as %% or _ _ which allows not only inserting comments into your Matlab code, but also format it for then publish the code automatically into an executable and readable format, see http://uk. mathworks. com/help/matlab/matlab_prog/marking-up-matlab-comments-for-publishing. html. b When uploading data to OpenfMRI you need to ensure the structural data are defaced appropriatelythe website also offers to use their own defacing tool, see https://github. com/poldrack/openfmri/tree/master/ pipeline/facemask. c Thanks to Dorothy Bishop for pointing to this.
It is currently funded by the NIH Blueprint for Neuroscience Research, National Institute of Biomedical Imaging and Bioengineering, National Institute of Drug Addiction, National Institute of Mental Health, and National Institute of Neurological Disorders and Stroke. NeuroVault (http://neurovault. org/) is a "public repository of unthresholded brain activation maps" under a data common license. It is managed by Krzysztof Gorgolewski, and supported by INCF and the Max Planck Society. Open fMRI (https://openfmri. org/) is "a project dedicated to the free and open sharing of functional magnetic resonance imaging (fMRI) datasets, including raw data" under an open data common license. It is managed by Russ Poldrack and funded by a grant from the National Science Foundation.