dataesr/scientific-paragraphs-categorization huggingface dataset:
To satisfy the recorded history criterion, version control (currently implemented in Git) is another component of Maneage (see Figure  2 ). Maneage is a Git branch that contains the shared components (infrastructure) of all projects (e. g. , software tarball URLs, build recipes, common subMakefiles, and interface script). The core Maneage git repository is hosted at git. maneage. org/project. git (archived at Software Heritage). Derived projects start by creating a branch and customizing it (e. g. , adding a title, data links, narrative, and subMakefiles for its particular analysis, see Listing 2). There is a thoroughly elaborated customization checklist in README-hacking. md. The current project's Git hash is provided to the authors as a L A T E X macro (shown here at the end of the abstract), as well as the Git hash of the last commit in the Maneage branch (shown in the acknowledgments).
The upward arrows at the branch-tops are therefore in the direction of time. experience in testing them through Maneage and its increasing user-base (thanks to the support of RDA). Firstly, while most researchers are generally familiar with them, the necessary low-level tools (e. g. , Git, L A T E X, the command-line and Make) are not widely used. Fortunately, we have noticed that after witnessing the improvements in their research, many, especially early-career researchers, have started mastering these tools. Scientists are rarely trained sufficiently in data management or software development, and the plethora of high-level tools that change every few years discourages them. Indeed the fast-evolving tools are primarily targeted at software developers, who are paid to learn and use them effectively for short-term projects before moving on to the next technology.
Scientists, on the other hand, need to focus on their own research fields and need to consider longevity. Hence, arguably the most important feature of these criteria (as implemented in Maneage) is that they provide a fully working template or bundle that works immediately out of the box by producing a paper with an example calculation that they just need to start customizing. Using mature and time-tested tools, for blending version control, the research paper's narrative, the software management and a robust data management strategies. We have noticed that providing a clear checklist of the initial customizations is much more effective in encouraging mastery of these core analysis tools than having abstract, isolated tutorials on each tool individually. Secondly, to satisfy the completeness criterion, all the required software of the project must be built on various Unixlike OSs (Maneage is actively tested on different GNU/Linux distributions, macOS, and is being ported to FreeBSD also).
This requires maintenance by our core team and consumes time and energy. However, because the PM and analysis components share the same job manager (Make) and design principles, we have already noticed some early users adding, or fixing, their required software alone. They later share their low-level commits on the core branch, thus propagating it to all derived projects. Thirdly, Unix-like OSs are a very large and diverse group (mostly conforming with POSIX), so our completeness condition does not guarantee bit-wise reproducibility of the software, even when built on the same hardware. However our focus is on reproducing results (output of software), not the software itself. Well written software internally corrects for differences in OS or hardware that may affect its output (through tools like the GNU Portability Library, or Gnulib).
Combined with SoftwareHeritage, precise highlevel science components of the analysis can be accurately cited (e. g. , even failed/abandoned tests at any historical point). Many components of "machine-actionable" data management plans can also be automatically completed as a byproduct, useful for project PIs and grant funders. From the data repository perspective, these criteria can also be useful, e. g. , the challenges mentioned in  [7] :  (1)  The burden of curation is shared among all project authors and readers (the latter may find a bug and fix it), not just by database curators, thereby improving sustainability. (2) Automated and persistent bidirectional linking of data and publication can be established through the published and complete data lineage that is under version control. (3) Software management: with these criteria, each project comes with its unique and complete software management.
It does not use a third-party PM that needs to be maintained by the data center (and the many versions of the PM), hence enabling robust software management, preservation, publishing, and citation. For example, see zenodo. 1163746, zenodo. 3408481, zenodo. 3524937, zenodo. 3951151 or zenodo. 4062460 where we distribute the source code of all software used in each project in a tarball, as deliverables. (  4 ) "Linkages between documentation, code, data, and journal articles in an integrated environment", which effectively summarizes the whole purpose of these criteria. Mohammad Akhlaghi is a postdoctoral researcher at the Instituto de Astrofísica de Canarias (IAC), Spain. He has a PhD from Tohoku University (Japan) and was previously a CNRS postdoc in Lyon (France). Email: mohammad-AT-akhlaghi.
Another approach to having an isolated analysis environment is to use the same file system as the host, but installing the project's software in a nonstandard, project-specific directory that does not interfere with the host. Because the environment in this approach can be built in any custom location on the host, this solution generally does not require root permissions or extra low-level layers like containers or VMs. However, "moving" the built product of such solutions from one computer to another is not generally as trivial as containers or VMs. Examples of such thirdparty package managers (that are detached from the host OS's package manager) include (but are not limited to) Nix, GNU Guix, Python's Virtualenv package, Conda. Because it is highly intertwined with the way software is built and installed, third party package managers are described in more detail as part of Section A-B.
We call this method "blind" packaging because it is agnostic to how the software and its dependencies were built (which is important in a scientific context). Moreover, these types of packagers are designed for the Linux kernel (using its containerization and unique mounting features). They can therefore only be run on GNU/Linux operating systems. 3) Nix or GNU Guix: Nix 7   [22]  and GNU Guix 8   [23]  are independent package managers that can be installed and used on GNU/Linux operating systems, and macOS (only for Nix, prior to macOS Catalina). Both also have a fully functioning operating system based on their packages: NixOS and "Guix System". GNU Guix is based on the same principles of Nix but implemented differently, so we focus the review here on Nix. The Nix approach to package management is unique in that it allows exact dependency tracking of all the dependencies, and allows for multiple versions of software, for more details see  [22].
In summary, a unique hash is created from all the components that go into the building of the package (including the instructions on how to build the software). That hash is then prefixed to the software's installation directory. As an example from  [22] : if a certain build of GNU C Library 2. 3. 2 has a hash of 8d013ea878d0, then it is installed under /nix/store/8d013ea878d0-glibc-2. 3. 2 and all software that is compiled with it (and thus need it to run) will link to this unique address. This allows for multiple versions of the software to co-exist on the system, while keeping an accurate dependency tree. As mentioned in  [23] , one major caveat with using these package managers is that they require a daemon with root privileges (failing our completeness criteria). This is necessary "to use the Linux kernel container facilities that allow it to isolate build processes and maximize build reproducibility".
This is because the focus in Nix or Guix is to create bitwise reproducible software binaries and this is necessary for the security or development perspectives. However, in a noncomputer-science analysis (for example natural sciences), the main aim is reproducible results that can also be created with the same software version that may not be bit-wise identical (for example when they are installed in other locations, because the installation location is hard-coded in the software binary or for a different CPU architecture). Finally, while Guix and Nix do allow precisely reproducible environments, it requires extra effort on the user's side to ensure that the built environment is reproducible later. For example, simply running guix install gcc (the most common way to install a new software) will install the most recent version of GCC, that can be different at different times.
Therefore, a user or script may not notice that the software that is being used is actually coming from the operating system, and not from the controlled Conda installation. • Generally, by default, Conda relies heavily on the operating system and does not include core commands like mkdir (to make a directory), ls (to list files) or cp (to copy). Although a minimal functionality is defined for them in POSIX and generally behave similarly for basic operations on different Unix-like operating systems, they have their differences. For example, mkdir -p is a common way to build directories, but this option is only available with the mkdir of GNU Coreutils (default on GNU/Linux systems and installable in almost all Unixlike OSs). Running the same command within a Conda environment that does not include GNU Coreutils on a macOS would crash.
5) Spack: Spack is a package manager that is also influenced by Nix (similar to GNU Guix), see  [26]. But unlike Nix or GNU Guix, it does not aim for full, bit-wise reproducibility and can be built without root access in any generic location. It relies on the host operating system for the C library. Spack is fully written in Python, where each software package is an instance of a class, which defines how it should be downloaded, configured, built, and installed. Therefore if the proper version of Python is not present, Spack cannot be used and when incompatibilities arise in future versions of Python (similar to how Python 3 is not compatible with Python 2), software building recipes, or the whole system, have to be upgraded. Because of such bootstrapping problems (for example how Spack needs Python to build Python and other software), it is generally a good practice to use simpler, lower-level languages/systems for a low-level operation like package management.
In conclusion for all package managers, there are two common issues regarding generic package managers that hinder their usage for high-level scientific projects: • Pre-compiled/binary downloads: Most package managers primarily download the software in a binary (precompiled) format. This allows users to download it very fast and almost instantaneously be able to run it. However, to provide for this, servers need to keep binary files for each build of the software on different operating systems (for example Conda needs to keep binaries for Windows, macOS and GNU/Linux operating systems). It is also necessary for them to store binaries for each build, which includes different versions of its dependencies. Maintaining such a large binary library is expensive, therefore once the shelf-life of a binary has expired, it will be removed, causing problems for projects that depend on them.
As a result, they manually install their high-level software in an uncontrolled, or non-standard way, thus jeopardizing the reproducibility of the whole work. This is another consequence of the detachment of the package manager from the project doing the analysis. Addressing these issues has been the basic reason behind the proposed solution: based on the completeness criteria, instructions to download and build the packages are included within the actual science project, and no special/new syntax/language is used. Software download, built and installation is done with the same language/syntax that researchers manage their research: using the shell (by default GNU Bash in Maneage) for low-level steps and Make (by default, GNU Make in Maneage) for job management.
It is thus very important for the integrity of a scientific project that the state/version of its processing is recorded as the project evolves. For example, better methods are found or more data arrive. Any intermediate dataset that is produced should also be tagged with the version of the project at the time it was created. In this way, later processing stages can make sure that they can safely be used, i. e. , no change has been made in their processing steps. Solutions to keep track of a project's history have existed since the early days of software engineering in the 1970s and they have constantly improved over the last decades. Today the distributed model of "version control" is the most common, where the full history of the project is stored locally on different systems and can easily be integrated.
There are many existing version control solutions, for example, CVS, SVN, Mercurial, GNU Bazaar, or GNU Arch. However, currently, Git is by far the most commonly used in individual projects. Git is also the foundation upon which this paper's proof of concept (Maneage) is built. Archival systems aiming for long term preservation of software like Software Heritage  [27]  are also modeled on Git. Hence we will just review Git here, but the general concept of version control is the same in all implementations. 1) Git: With Git, changes in a project's contents are accurately identified by comparing them with their previous version in the archived Git repository. When the user decides the changes are significant compared to the archived state, they can be "committed" into the history/repository.
D. Job management Any analysis will involve more than one logical step. For example, it is first necessary to download a dataset and do some preparations on it before applying the research software on it, and finally to make visualizations/tables that can be imported into the final report. Each one of these is a logically independent step, which needs to be run before/after the others in a specific order. Hence job management is a critical component of a research project. There are many tools for managing the sequence of jobs, below we review the most common ones that are also used the existing reproducibility solutions of Appendix B and Maneage. 1) Manual operation with narrative: The most commonly used workflow system for many researchers is to run the commands, experiment on them, and keep the output when they are happy with it (therefore loosing the actual command that produced it).
When any of the prerequisite(s) is newer than a target, the recipe is re-run to re-build the target. When all the prerequisites are older than the target, that target does not need to be rebuilt. A recipe is just a bundle or shell commands that are executed if necessary. Going deeper into the syntax of Make is beyond the scope of this paper, but we recommend interested readers to consult the GNU Make manual for a very good introduction  10. 4) Snakemake: Snakemake is a Python-based workflow management system, inspired by GNU Make (discussed above). It is aimed at reproducible and scalable data analysis  [29]   11. It defines its own language to implement the "rule" concept of Make within Python. Technically, calling commandline programs within Python is very slow, and using complex shell scripts in each step will involve a lot of quotations that make the code hard to read.
Currently, Snakemake requires Python 3. 5 (released in September 2015) and above, while Snakemake was originally introduced in 2012. Hence it is not clear if older Snakemake source files can be executed today. As reviewed in many tools here, depending on high-level systems for low-level project components causes a major bootstrapping problem that reduces the longevity of a project. 5) Bazel: Bazel 12  is a high-level job organizer that depends on Java and Python and is primarily tailored to software developers (with features like facilitating linking of libraries through its high-level constructs). 6) SCons: Scons 13  is a Python package for managing operations outside of Python (in contrast to CGAT-core, discussed below, which only organizes Python functions). In many aspects it is similar to Make, for example, it is managed through a 'SConstruct' file.
Like a Makefile, SConstruct is also declarative: the running order is not necessarily the topto-bottom order of the written operations within the file (unlike the imperative paradigm which is common in languages like C, Python, or FORTRAN). However, unlike Make, SCons does not use the file modification date to decide if it should be remade. SCons keeps the MD5 hash of all the files in a hidden binary file and checks them to see if it is necessary to re-run. SCons thus attempts to work on a declarative file with an imperative language (Python). It also goes beyond raw job management and attempts to extract information from within the files (for example to identify the libraries that must be linked while compiling a program). SCons is, therefore, more complex than Make and its manual is almost double that of GNU Make.
Besides added complexity, all these "smart" features decrease its performance, especially as files get larger and more numerous: on every call, every file's checksum has to be calculated, and a Python system call has to be made (which is computationally expensive). Finally, it has the same drawback as any other tool that uses high-level languages, see Section A-F. We encountered such a problem while testing SCons: on the Debian-10 testing system, the python program pointed to Python 2. However, since Python 2 is now obsolete, SCons was built with Python 3 and our first run crashed. To fix it, we had to either manually change the core operating system path, or the SCons source hashbang. The former will conflict with other system tools that assume python points to Python-2, the latter may need root permissions for some systems.
This can also be problematic when a Python analysis library, may require a Python version that conflicts with the running SCons. 7) CGAT-core: CGAT-Core is a Python package for managing workflows, see  [30]. It wraps analysis steps in Python functions and uses Python decorators to track the dependencies between tasks. It is used in papers like  [31]. However, as mentioned in  [31]  it is good for managing individual outputs (for example separate figures/tables in the paper, when they are fully created within Python). Because it is primarily designed for Python tasks, managing a full workflow (which includes many more components, written in other languages) is not trivial. Another drawback with this workflow manager is that Python is a very high-level language where future versions of the language may no longer be compatible with Python 3, that CGAT-core is implemented in (similar to how Python 2 programs are not compatible with Python 3).
10) Generic workflow specifications (CWL and WDL): Due to the variety of custom workflows used in existing reproducibility solution (like those of Appendix B), some attempts have been made to define common workflow standards like the Common workflow language (CWL  15  , with roots in Make, formatted in YAML or JSON) and Workflow Description Language (WDL  16  , formatted in JSON). These are primarily specifications/standards rather than software. With these standards, ideally, translators can be written between the various workflow systems to make them more interoperable. In conclusion, shell scripts and Make are very common and extensively used by users of Unix-based OSs (which are most commonly used for computations). They have also existed for several decades and are robust and mature. Many researchers that use heavy computations are also already familiar with them and have already used them already (to different levels).
As we demonstrated above in this appendix, the list of necessary tools for the various stages of a research project (an independent environment, package managers, job organizers, analysis languages, writing formats, editors, etc) is already very large. Each software/tool/paradigm has its own learning curve, which is not easy for a natural or social scientist for example (who need to put their primary focus on their own scientific domain). Most workflow management tools and the reproducible workflow solutions that depend on them are, yet another language/paradigm that has to be mastered by researchers and thus a heavy burden. Furthermore as shown above (and below) high-level tools will evolve very fast causing disruptions in the reproducible framework. A good example is Popper  [33]  which initially organized its workflow through the HashiCorp configuration language (HCL) because it was the default in GitHub.
E. Editing steps and viewing results In order to reproduce a project, the analysis steps must be stored in files. For example Shell, Python, R scripts, Makefiles, Dockerfiles, or even the source files of compiled languages like C or FORTRAN. Given that a scientific project does not evolve linearly and many edits are needed as it evolves, it is important to be able to actively test the analysis steps while writing the project's source files. Here we review some common methods that are currently used. 1) Text editors: The most basic way to edit text files is through simple text editors which just allow viewing and editing such files, for example, gedit on the GNOME graphic user interface. However, working with simple plain text editors like gedit can be very frustrating since its necessary to save the file, then go to a terminal emulator and execute the source files.
Furthermore, they usually require a graphic user interface to run. In summary, IDEs are generally very specialized tools, for special projects and are not a good solution when portability (the ability to run on different systems and at different times) is required. 3) Jupyter: Jupyter (initially IPython)  [34]  is an implementation of Literate Programming  [35]. Jupyter's name is a combination of the three main languages it was designed for: Julia, Python, and R. The main user interface is a webbased "notebook" that contains blobs of executable code and narrative. Jupyter uses the custom built. ipynb format  17. The. ipynb format, is a simple, human-readable format that can be opened in a plain-text editor) and formatted in JavaScript Object Notation (JSON). It contains various kinds of "cells", or blobs, that can contain narrative description, code, or multimedia visualizations (for example images/plots), that are all stored in one file.
The cells can have any order, allowing the creation of a literal programming style graphical implementation, where narrative descriptions and executable patches of code can be intertwined. For example to have a paragraph of text about a patch of code, and run that patch immediately on the same page. The. ipynb format does theoretically allow dependency tracking between cells, see IPython mailing list (discussion started by Gabriel Becker from July 2013  18  ). Defining dependencies between the cells can allow non-linear execution which is critical for large scale (thousands of files) and complex (many dependencies between the cells) operations. It allows automation, run-time optimization (deciding not to run a cell if it is not necessary), and parallelization. However, Jupyter currently only supports a linear run of the cells: always from the start to the end.