dataesr/scientific-paragraphs-categorization huggingface dataset:
One solution is to run the entire compilation from the beginning, which wastes compilation and development time and resources. Make addresses these issues by introducing the notion of 'targets' (output files) which combine required inputs with a shell command to produce the desired output in the right sequence  [Feldman, 1979b]. For example, the following Makefile contains two targets (a csv file and a pdf report) being produced by the same shell script (compiling an R markdown report): Any of the following two commands will run the script to produce both targets make output / r e p o r t. pdf make output / t a b l e. c s v Repeated invocation of these commands will not result in a re-compilation of the pdf report unless any of the input files changed or the output is removed. Since make can run any shell script to produce a target, it is equally generic as a simple bash script but provides the additional convenience of caching.
Since a Makefile follows a clear target/inputs/steps structure and dependencies are resolved automatically, it is often easier to infer the structure of a workflow from a Makefile than a simple shell script. Another advantage is that make can facilitate faster parallel compilation, adapted to today's mostly multicore machines. One problem of using make as workflow engine is the fact that outputs are not explicitly checked after executing a shell script for a target. In other words, for complex workflows it can be difficult to identify problems early since the absence of a file might only be noticed at a later point when it is used as input to a subsequent target.
Dedicated scientific workflow engines: snakemake and friends Workflow engines geared specifically towards scientific use (primarily in bioinformatics) take a more formal approach to representing workflows. For example snakemake  [Köster and Rahmann, 2012]  internally represents a workflow as a directed acyclic graph (DAG) which allows inference on the optimal execution order given the available computing resources. By automatically running (conditionally) independent steps of a workflow in parallel, snakemake can easily speed up workflows with the appropriate structure (simulations, multi-model comparisons). As the name already suggests, snakemake was inspired by make but adds a few crucial new features. Snakemake files are written in pseudo python code, and since python is a widespread scripting language for data analysis, the syntax will be familiar to a wide range of users.
Additionally, snakemake supports powerful wildcard schemes to easily generalise rules to classes of input (or output) files. Workflow parameters are easily handled through json or yml configuration files and snakemake works seamlessly with the popular cluster workload manager, slurm. This means that a workflow can be executed on a laptop and a slurm cluster without modifying the workflow itself. Alternatives to snakemake include nextflow and the common workflow language (CWL, the actual workflow engine is cwltool or any alternative implementation). Within limits, it is possible to automatically translate workflows between these different systems (via CWL). The primary reason for putting forward snakemake here is the low level of abstraction required by the user and the similarity with potentially familiar tools like make and the use of Python, well-known in the (bio)statistics community.
Cloud collaboration: CyVerse and Gigantum CyVerse and Gigantum are two very different web-based solutions to the reproducibility of workflows. Their scope and type of analysis differ dramatically from each other but both have legitimate reasons to be named in this section. They both aim to reduce the complexity of software setup that a computational scientist is faced with. Both solve the issue of a portable compute context by the dockerisation of the work environment (see Section 5) and solve the issue of user friendliness through interface-based workflow building facilities. Both use cloud technology to some degree and both focus on collaborative working and data sharing, with CyVerse even providing free cloud computing resources to the users. However, the scope and type of analysis that can be made through these environments is very different, though overlaps occur.
CyVerse, together with DataStore is a big framework for integrating data and tools, running workflows and facilitating the publishing of data, workflow and results. CyVerse's particular strengths lies in what they call 'democratising access to supercomputing capabilities'. They enable small groups with restricted computing resources to run large analysis on the CyVerse HPC environment. Using a graphical interface called the DiscoveryEnvironment, the user can build and submit analysis workflows based on wellestablished software tools, for example a gene annotation pipeline. It is community driven, to the point that any experienced user can 'wrap' existing or newly developed apps into the CyVerse environment, write a documentation for it and share it with collaborators or the entire community.
Thanks to docker, the new app will then work seamlessly on the CyVerse HPC cluster. While CyVerse can be entirely user interface based, the same outcome (an analysis workflow) can be achieved using the commandline. Workflows with crucial metadata such as the tool versions, file input and outputs can then be saved and rerun, similar to snakemake and nextflow. While RStudio or Jupyter can also be accessed through the discovery environment, the authors find that Gigantum is a better resource for collaboratively working on reproducible workflows, or in this case larger scripts of R or Python at its heart. Gigantum is like a streamlined project framework, integrated with version control and the ability to share projects and updates with collaborators in real time. Similar to CyVerse, Gigantum 'Projects' can be integrated with datasets, in their case however, the Gigantum 'Dataset' usually hosts metadata only, instead of the actual file content.
Gigantum's strengths are in interactive, collaborative and exploratory work. A local project's code and quick visualisations can be shared easily, via the Gigantum cloud and tracking features. It is a browser-based app, where workflows can be coded up on Jupyter or R notebooks. By using these notebooks, Gigantum requires the knowledge of either Python or R, and some understanding of version control concept is also a bonus, (even though Git is automated and works in the background). By focusing on R and Jupyter scripts, Gigantum seems to have a smaller scope than what is commonly understood for workflow engines such as snakemake, nextflow or CWL. However, the aspect of automatically documenting workflows (even if in this case only a single notebook), which enhances reproducibility is well catered for in  Gigantum.
Key factors to consider are accessibility for the target audience, required features (complexity of the workflow at hand), and software requirements. Firstly, it is a good idea to use the simplest tool available to get the job done. In cases where a single, well-commented bash script suffices to run all steps of an analysis and produce the required outputs it might very well be the most portable and easy-to-understand way of defining a workflow since bash is ubiquitous. Make is a straight-forward extension of this approach whenever caching might be useful (mostly during development to avoid recompiling unchanged results). Snakemake (or similar scientific workflow engines) add tools for automatic parallel execution (potentially on a cluster) and more rigorous output checking but come at the cost of having to install the workflow engine on the executing system (requiring Python or Java).
Dependency Management Section 3 discussed the benefits of source code version control. Section 4 addressed the problem of bundling together complex workflows spread across several interdependent script files using workflow engines to produce more user-friendly workflows that are reproducible by eliminating the need for manual interventions to produce a desired output. Both aspects are crucial in achieving reproducibility but fail to address the issue of managing software dependencies. To make sure that the workflow can be executed as intended, all software dependencies must be installed with exactly the same versions. The factor that complicates reproducible research the most is thus not so much the sharing of the code itself but rather the challenge to reproduce the exact same execution environment in a portable way.
Within individual scripting languages like R or Python, language-specific package managers can be used to download and install specific versions of packages (e. g. packrat and pip). Meta-package managers like Anaconda even bring together packages from multiple different languages. This approach, however, fails to isolate operating system level dependencies and relies on the continued availability of legacy package versions for download to replicate the package ecosystem at the time of the initial analysis. A more robust way of ensuring that an instance of the exact same computing environment is preserved for future re-runs of analyses is isolation at the operating system level via virtualisation. Arguably the easiest way of achieving this is via virtual machines. Virtualisation is a tried and tested technology frequently employed in server environments to make software deployment independent from the underlying hardware it is running on.
At execution time, no software needs to be downloaded or installed and the analysis scripts can be run in exactly the same computational environment as during the initial analysis. Since they are also generic with regard to the type of software that can be installed (anything that can be installed on a Linux system) they also add to the 'realism' of the approach since there is no restriction to a specific set of software packages. A sign for the success of containerised analyses workflows and singularity in particular is the fact that all major workflow engines (e. g. snakemake and nextflow) natively support executing individual tasks or entire workflows in singularity containers. 6 Archiving and Citability.
Archiving Defining a clear time window is crucial for a reproducible research project. The steps and suggestions outlined in the sections above are intended to facilitate the technical aspects of reproducibility. However, it is challenging to ensure that the technical pipeline for reproducing results will work in the long term. Even a completely containerised environment depends on support for the container software and a compatible host operating system. Additional steps should therefore be undertaken to ensure long-term availability of both code and data. When it comes to the storage of sensitive patient data, popular generalist repositories such as Zenodo may not be suitable  [Banzi et al. , 2019]  -for example, such repositories may not have guidelines or perform checks related to de-identification of data, nor allow controlled data access.
Code, and if available, containers, on the other hand are much easier to archive for a prolonged period since they should not contain any sensitive data in the first place. It is important to realise though that popular Git repository hosting services such as GitHub, GitLab or Bitbucket are not necessarily suited for long-term storage of analysis-related code (or entire container images). Despite being extremely popular and almost constituting a gold standard for code sharing, as noted by  Jackson [2018] , they do not create unique or persistent digital identifiers. In addition, they are not committed to providing long-term storage, and their business model and terms of use might change in the future. This has started to change recently, with initiatives like the GitHub archive program  [GitHub Inc.
, nd] , where a large selection of open source projects will be backed up in multiple forms of storage to ensure their preservation. However, these initiatives remain limited to a small portion of current projects. It is thus advisable to back up important revisions/releases of publication-related repositories to a dedicated long-term storage solution. As already mentioned in Section 2, the choice of repository is critical in ensuring that both data and code can be accessed in the future. In the last decade, a large number of repositories have emerged, including general purpose repositories (such as Zenodo, Figshare, Dryad, Mendeley Data and Dataverse) and discipline-specific repositories (such as the Worldwide Protein Data Bank, UniprotKB and Genbank) which accept only a restricted number of data types and formats.
Further subject-specific data repositories can be found at Nature Scientific Data policies for example  [Nature Scientific Data, nd]. Repositories also exist specifically for software, with initiatives such as the Software Heritage project (https://www. softwareheritage. org/) where free/open source software can be uploaded and preserved for long-term storage  [Di Cosmo et al. , 2019]. Alternatively, there are language-specific repositories such as CRAN and Bioconductor for R, PyPI for Python, CPAN for Perl, and Maven for Java-based projects. Table  3  gives some advantages and disadvantages of different archiving locations. This heterogeneous landscape offers several opportunities but also introduces challenges in effectively finding the most appropriate repositories for either storing or accessing software and data.
This is where Re3data represents a valuable resource, having registered more than 2000 research data repositories so far, which can be searched (e. g. by name, discipline, content type or country) to identify the most appropriate data repositories  [Kindling, 2017]. Another useful resource is FAIRsharing (https://fairsharing. org/), which describes and interlinks data standards, databases and data policies  [Sansone et al. , 2019]. The information collected ensures that the FAIR data principles are followed, and allows users to easily search for repositories that fit with funder or journal data policies. Section 2 gives further factors that should be considered when choosing a research repository, including whether long-term data preservation is offered. Despite these resources, the large number of available repositories still risks limiting the ability of researchers to easily access or deposit data from the right location.
In our opinion, more focus should be placed on developing standards for field-specific databases rather then starting redundant repositories. An important initiative to tackle this issue is the ELIXIR-Europe project, which aims to create a single infrastructure to provide easy access to life science resources from across Europe  [ELIXIR, nd]. The project comprises of different platforms, including the data platform and the interoperability platform. The data platform aims to provide users with robust, long-term sustainable data resources within a coordinated data ecosystem. This includes Literature-Data integration, to streamline the process of moving from a scientific article to its underlying datasets. Meanwhile, the interoperability platform aims to establish standardised file formats, metadata, vocabularies and identifiers for the life-science community.
, 2016]  are critical for citability and visibility. The FAIR principles should also be followed for code and software, but their application may require some revisions and elaboration to account for the specificities of software, see  Lamprecht et al. [2019]  for further details. Initiatives such as these are enabling data and software to be recognised in their own right, instead of being completely overshadowed by journal articles and associated metrics. Digital Object Identifiers (DOIs) are one key way to provide unique and persistent identification, as well as specificity in terms of the version of data or software that was used. For code in particular, it is important to note that some commonly-used general-purposed repositories can provide DOIs. For example, Github accounts can be linked to Zenodo or Figshare, so that any new release on the GitHub repository will automatically be archived with a corresponding DOI.
The automatic creation of release-specific DOIs facilitates the discoverability of the repositories, as they can then be cited in a robust way, even if the underlying content is moved to a different infrastructure in the future. Metadata is also critical for enabling visibility, as previously mentioned for datasets in Section 2. Metadata standards have also being developed for code: formats such as the CodeMeta Project [CodeMeta contributors, nd] and the Citation File Format  [Druskat et al. , 2019]  assist accessibility, and allow for proper credit and attribution. A recent development that facilitates citation of complex projects and workflows is the tools platform of the ELIXIR-EUROPE project, which has a specific focus on software. As an example, bio. tools (https://bio. tools/) aims to provide a comprehensive registry of both software and databases  [Ison et al.
Outlook 7. 1 The last mile: pre-prints and seamless publication services In practice, it is often surprisingly difficult to format a paper for submission to a journal since many medical journals still require manuscripts to be submitted as. docx files which makes a seamless reproducible research process almost impossible and still requires extensive manual editing. A promising solution to this problem is the tight integration of online LaTeX editors such as Overleaf together with GitHub, and finally with pre-print services. Overleaf [2012]  takes away the complexity of setting up a LaTeX environment and provides a rich text view to facilitate editing LaTeX source files for collaborators not familiar with this complex but highly professional typesetting tool. The integration with GitHub allows the analysis code to be stored in the same repository as the final manuscript file with the only downside of having to store binary figure output files in the Git repository.
Several pre-print services already allow submission directly from Overleaf (e. g. Biostatstics) and some even organise their peer review process on Overleaf. Since pre-printing services generally rely on the submitting authors to do the final layout and formatting, the growing ecosystem of pre-print services could provide an interesting solution for bridging that last gap between reproducible analysis and publication. Authors would initially write and publish to a pre-print service in a completely reproducible manner before considering submitting the manuscript to a traditional journal. Besides smoothing the publication process, this approach has the added advantage that all research is publicly available as a pre-print (cf. the FAIR principles). With the rise of bioRxiv and, more recently, medRxiv, the only reason not to publish biostatistics related research output as pre-print first is the risk of traditional journals rejecting acceptance of a manuscript due to copyright issues or restrictive pre-print policies.
Make your analysis workflow future-proof Archive results, code and data, and make them citable Provide pre-prints and/or go open access Create (R or Python) packages and place your code in containers with all dependencies to make it operating system independent and reduce setup difficulties for others. Good for future access, but also for easy switching between local computers and clusters, and exchange between group members/collaborators during the project. Try to make public where possible, but adhere to GDPR and HIPPA. Choose the correct repository to increase findability and reusability (see Section 2). See Table  3  for further pros and cons to choose your archiving locations. Make your findings openly and freely available. Biostatisticians face a multitude of hurdles when wanting to perform realistic, robust and reproducible research.
Figure2: Overview of elements involved in a robust reproducible workflow. Orange circles represent data sources from a repository, where the data is openly accessible and citable, complies with format requirements and contains the minimum required metadata, and where the dataset license is legally adhered to during the process of data reuse. The blue circle represents all files necessary for the reproducibility of a research project, available at an online location such as a Git repository that is easily downloadable by the researcher. The repository includes scripts that retrieve and prepare the data, perform simulations and/or analysis, and create the figures used in the publication. Lastly, all results are integrated in the manuscript, which recompiles itself in the same run instant. This entire workflow is governed through a workflow engine script, a Makefile (e.
g. a Snakefile in case of snakemake), which not only compiles and runs all executing scripts and outputs in the correct order, but also prepares the environments and dependencies before running the scripts. The black lines between the shapes in the green box represent the schematics of the Makefile. This analysis could be executed on a local machine or an HPC (green box). Once the Git repository is downloaded, the entire workflow should build the manuscript with one simple command: 'snakemake'.
Data Management Plan Useful templates and funder-specific guidance https://dmponline. dcc. ac. uk Software Management Plan Guidance and checklist https://doi. org/10. 5281/zenodo. 2159713 Readme Advice on producing Readme files https://data. research. cornell. edu/ content/readme Metadata annotation List of life science-specific metadata vocabularies https://www. ebi. ac. uk/ols/ontologies Data Deposition Help to choose your discipline-specific repository. Also: watch out for the CoreTrustSeal https://www. openaire. eu/ opendatapilot-repository-guide https://www. coretrustseal. org.
This entire workflow is governed through a workflow engine script, a Makefile (e. g. a Snakefile in case of snakemake), which not only compiles and runs all executing scripts and outputs in the correct order, but also prepares the environments and dependencies before running the scripts. The black lines between the shapes in the green box represent the schematics of the Makefile. This analysis could be executed on a local machine or an HPC (green box). Once the Git repository is downloaded, the entire workflow should build the manuscript with one simple command: 'snakemake'.
MiRNA loss-of-function screen and transcriptome profiling of human muscle differentiation process The screening data from the study by A. Polesskaya et al. [30]  was taken as the hit list for terminal human skeletal muscle differentiation process. In this study, genome-wide miRNA loss-of-function screening on a late differentiating human muscle precursor cell line (LHCN) was performed in a two-step approach. The primary screening was done in duplicate with a miRNA antisense inhibitors (Locked Nucleic Acids, LNA) library targeting 870 miRNAs and a readout assay that detects Myosin Heavy Chain (MHC) positive and multinucleated myotubes. Those miRNAs whose depletion resulted in differences to the negative control ≥2 standard deviation (SD) were selected for confirmation in the secondary screen.
The fact that the comparison resulted only in a few number of paths might indicate, that although these experimental systems study one biological process, they characterize the biological machinery at two different levels: transcriptional (transcriptome) and posttranscriptional (miRNAs). We also found that 22 paths from the analysis of transcriptome profiling have miRNAs hits from the loss-offunction screening and 260 paths from the analysis of loss-of-function screening have hit genes from the transcriptome profiling on them. These are the paths from the analysis of transcriptome profiling that pass through hsa-mir-125b which controls IGF2 gene and the paths that pass through hsa-mir-145 that control TRIP10 protein which, according to OMIM database, has highest expression in skeletal muscle  [49]  and interacts with CDC42 protein.