dataesr/scientific-paragraphs-categorization huggingface dataset:
In this article, we describe additional guidelines on how to write the Nextflow code. We propose a set of best practices along the development life cycle of the pipeline and deployment for production operations which address different expert communities: i) the bioinformaticians and statisticians who prototype the pipeline with state-of-the-art methods in order to extract most of the hidden value from the data and provide the end-users with summary reports, ii) the software engineers who optimize the pipeline to reduce the amount of required informatics resources, to shorten the time to result delivery, iii) the data managers and core facility engineers who deploy and operate the pipeline for daily production analysis for the end-users. The guidelines were motivated by: i) being less invasive to the different expert communities, ii) reducing the overall development cycle from the prototyping stage to deployment in a production environment, iii) providing portable pipelines with containers (Docker and Singularity), and iv) automating (whenever possible) the building of containers from the source code of the pipeline.
General principle The ultimate goal of the proposed best practices of coding with Nextflow was to ensure the portability and reproducibility of the pipeline with containerization techniques (such as Docker and Singularity). Therefore, the rationale behind Geniac was motivated by the automation of the construction of the containers without explicitly writing the Dockerfile or the Singularity Definition File recipes (whenever possible). To do so, the pipeline source code is parsed by Geniac. The portability of the pipeline with containers can be achieved in two ways: either a single container including all the tools required by the pipeline is provided, or several containers (one container for each tool to be as granular as possible) are provided. We decided to retain this second way, as this one container -one tool strategy was strongly recommended by several experts  (Gruening et al.
As the Geniac toolbox automatically generates the configuration files used by these different profiles from the source code, it ensures that they are consistent with each other. Moreover, it saves time for the developer who does not have to write them one by one. All these different profiles are made available with the Geniac toolbox so that it covers pretty much all of the preferred work habits of a developer. However, we strongly encourage the use of multiconda during prototyping and singularity for production. Of note, both conda and path profiles may not work if the pipeline needs tools that are not compatible with each other. The one container -one tool strategy has to be translated in terms of coding best practices with Nextflow so that the source code can be automatically parsed to generate these Nextflow configuration files and build the containers.
Availability of the tool When implementing a new tool in a pipeline, the main question to answer is Where is the tool available?. Depending on the answer, the developer has to follow specific guidelines that are fully detailed in the Geniac documentation. Whenever possible, it is recommended to use the tool from the Conda packaging system  (Gruening et al. , 2018). However, not all tools are available in Conda. Therefore, we guide the developer through a series of chronological questions that redirect the developer to the appropriate section whenever an answer is Yes. The questions are the following: i) Is it just a standard Unix command?, ii) Is it available in Conda?, iii) Is it available only as a binary or as an executable script?, iv) Is the source code available?, v) Have you still not answered yes?.
Briefly, the developer has to perform some actions such as modify the conf/geniac. config file, copy the source code of the tool in the modules folder, write some recipes in the folder recipes, copy some files in the bin or recipes/dependencies folders, depending on the answer to the questions. In most of the cases, the tool should be available in Conda which is the most comfortable situation as it allows the automation of the creation of the containers and the configuration files for Nextflow profiles. However, the toolbox is able to tackle any other situations, although this requires more configuration on the developer's side. Figure  2  describes the overall workflow with Geniac.
Implementation It is important to note that the Geniac toolbox  (Hupé et al. , 2021a)  relies on the structure of the Geniac Template (Servant & Hupé, 2021) while the latter could work without Geniac (provided that the missing configuration files for the different Nextflow profiles are added manually). It also means that the Geniac documentation explains how the Geniac Template works with the Geniac toolbox. Therefore, the source code must be organized as shown in Figure  3A  with some mandatory files from the Geniac Template in blue and the Geniac toolbox itself in the eponymous folder in green. Based on this organization, the automatic generation of the Nextflow configuration files, container recipes and images are performed by some Nextflow scripts (geniac/install/singularity. nf and geniac/install/docker.
nf) which are invoked by Cmake scripts. The file nextflow. config includes all the configuration files that are automatically generated by Geniac to define the different Nextflow profiles. The main Cmake script is the file geniac/CMakeLists. txt which invokes a series of functions from the geniac/cmake folder. The command-line interface for the linter corresponds to a standalone python package within the geniac/src folder. The linter offers the possibility for the developer to check the code's compliance with the guidelines. The command includes the following tests: i) existence of mandatory or optional files and folders in the Nextflow project tree structure, ii) configuration and consistency of labels with process directives in the Nextflow pipeline, iii) validation of the geniac configuration file, iv) availability of tools and their dependencies.
Similarly to the nf-core linter, the output can be exploited manually or automatically within a continuous integration pipeline since any critical error reported by the linter changes the exit code from 0 to 1. As it will be illustrated in the Use Cases section, the file conf/geniac. config is essential as it allows registration of the tools available from Conda. This file also defines important options to ensure reproducibility with Singularity. Indeed, the default behavior of Singularity makes it possible to access the user's $HOME directory from the host on which the process is executed. Some programming languages such as python or R preemptively load libraries from the user $HOME if they exist rather than the libraries from the canonical installation. This is therefore a major issue that can be tackled by defining in the file conf/geniac.
config the option singularityRunOptions = "-B \"\$PWD\":/tmp --containall" which is passed on the Singularity command line when invoked by Nextflow. The --containall sets some directories as empty (including $HOME and /tmp) thus avoiding possible libraries installed in the user's $HOME to be loaded by these programming language. The -B option allowing the binding of the /tmp (inside the container) to the work directory of the Nextflow process (i. e. in the folder like work/2d/8202294 created by Nextflow during the execution) is also necessary. Indeed, the default overlay size of /tmp inside the Singularity image is few MB that makes the writing of temporary files required by many tools impossible due to space limitation. We preferred to bind /tmp to the work directory of the Nextflow process instead of on the actual /tmp of the host since it might use some RAM (tmpfs) that is not reported to the job scheduler (such as slurm or PBS).
As we defined the singularityRunOptions, it isolates the $HOME from the host whatever the programming language used inside a Nextflow process that makes this solution universal. However, this solves this issue only for the singularity profile. In the case of the conda or multiconda profiles, we addressed this issue by adding ad-hoc environment variables using the beforeScript directive beforeScript = "export R_LIBS_USER=\"-\"; export PYTHONNOUSERSITE=1" in the files which are automatically generated by Geniac. If any other programming language suffers from the same behavior, ad-hoc solutions will have to be added. Note that the default options when using Docker containers within Nextflow avoid access to $HOME, this profile is therefore not concerned by the reproducibility issue.
Operation The Geniac toolbox is intended to run on a Linux distribution with Nextflow (>= 20. 01. 0), git (>= 2. 0), Cmake (>= 3. 0), Make (>= 4. 1) and Conda (>= 4. 10. 1). To use the containers, either Singularity (>= 3. 2) or Docker (>= 18. 0) are needed. The Geniac linter requires python (>= 3. 9).
Use cases In this section, several use cases are described. They are available in the bash script data/useCases. bash from the Geniac repository so that the reader can reproduce them step-by-step. The use cases depend on each other meaning that they must be sequentially performed from the beginning. They also require the installation of Nextflow (>= 20. 01. 0), git (>= 2. 0), Cmake (>= 3. 0), Make (>= 4. 1), Conda (>= 4. 10. 1) and Singularity. The conducting line of this section is the deployment of a pipeline to be run in routine production with Singularity on a high-performance computing cluster having slurm as a job-scheduler. A comprehensive overview of the Geniac functionalities is available in the Geniac documentation.
Add a Nextflow process We explain here how to add a new process in the pipeline when the tool is available in Conda. First, the source code from the Geniac Demo  (Hupé et al. , 2021b)  repository can be downloaded as follows: export WORK_DIR="${HOME}/tmp/myPipeline" export SRC_DIR="${WORK_DIR}/src" export INSTALL_DIR="${WORK_DIR}/install" export BUILD_DIR="${WORK_DIR}/build" export GIT_URL="https://github. com/bioinfo-pf-curie/geniac-demo. git" mkdir -p ${INSTALL_DIR} ${BUILD_DIR} # clone the repository # the option --recursive is needed if you use geniac as a submodule git clone --recursive ${GIT_URL} ${SRC_DIR} Let's assume that the tool multiqc  (Ewels et al. , 2016)  must be added in the pipeline. The section params. geniac. tools of the file ${SRC_DIR}/conf/geniac. config must contain the following line: multiqc = "conda-forge::lzstring=1.
0. 4=py_1001 ↪ conda-forge::matplotlib-base=3. 1. 1=py37h250f245_2 ↪ conda-forge::spectra=0. 0. 11=py_1 bioconda::multiqc=1. 8=py_2" The syntax follows the pattern from the conda package naming convention, so that the version is explicit as recommended in some guidelines  (Gruening et al. , 2018) : toolName = "condaChannelName::softName=version=buildString" If other Conda dependencies are required, they are added according to the same convention as shown for multiqc. In order to use the tool in a Nextflow process, use the label directive using the exact same name as given in the params. geniac. tools section. For example, the process multiqc in the file ${SRC_DIR}/main. nf contains the label multiqc defined in the file ${SRC_DIR}/conf/geniac. config: process multiqc { label 'multiqc' // write your nextflow code.
Check the code with the linter We recommend installing the Geniac command-line interface using the package installer for python (pip) in a separate environment with Conda as follows: conda create -n geniac-cli python=3. 9 conda activate geniac-cli pip install git+https://github. com/bioinfo-pf-curie/geniac. git@release Run the linter on your repository: geniac lint ${SRC_DIR} Build the Singularity containers It consists of two main steps. First, the pipeline is configured with the required cmake options. The options with the pattern -DCMAKE_<UPPER_CASE>=<value> correspond to the variables available in the cmake language (CMAKE_INSTALL_PREFIX defines where the analysis pipeline will be deployed). The options with the pattern -Dap_<lower_case>=<value> correspond to the ad-hoc variables defined by the Geniac toolbox (see the Geniac documentation for the list of available options).
Compatibility with Nextflow DSL2 Geniac is fully compatible with Nextflow DSL2 provided that the process, workflows and subworkflows are organized in the ${SRC_DIR}/nf-modules folder as shown in red in Figure  3A. The Geniac Demo DSL2  (La Rosa et al. , 2021)  repository can be used instead of the Geniac Demo. To do so just set, at the very beginning of the Use cases section, the value of GIT_URL to: export GIT_URL="https://github. com/bioinfo-pf-curie/geniac-demo-dsl2. git".
Conclusions Geniac consists of a toolbox with three components: a technical documentation available at https://geniac. readthedocs. io, a linter to check that the code respects the guidelines, and an add-on to generate configuration files, build the container and deploy the pipeline. The Geniac Demo and Geniac Demo DSL2 pipelines are available so that the reader can practice the Geniac guidelines. The Geniac Template makes it possible to start a new pipeline from scratch. The Geniac toolbox is fully compatible with Nextflow DSL1 and DSL2. While it supports the automatic generation of Docker and Singularity containers, it is straightforward to add any other containerization framework such as podman or shifter. The development of the Geniac toolbox was strongly motivated by the context of production analysis.
Although technical, the manuscript explains clearly the philosophy, the goals, the usage and the usefulness of Geniac. The tools are usable in the provided workflow example, I was able to run the geniac-demo and the linter works well and is easy to use. I believe that Geniac is a useful tool that answers important questions. However, I think it would be easier to understand and use with a few clarifications in my opinion: I had some initial difficulties to run the geniac-demo workflow. I think the documentation (on readthedoc and on the git repositories) could be clearer about how to run the workflow in different situations, at least with docker or singularity. For example, I initially had an error when the install and the build directories were included in the geniac-demo directory. Before understanding the general philosophy of Geniac, I also had some difficulties to understand how to generate the singularity.
Testing the workflow is also generally a difficult part of the development and maintenance of workflows. It is not mentioned clearly in the introduction or in the discussion, though I think that tests can be performed using test profiles in Geniac. I think it would be great to give a little bit more insights on what tests consist of in geniac workflows. Reply: Testing is indeed a major component of software development. Therefore, the section "Run the pipeline" has been renamed "Test and run the pipeline". We highlighted the necessity to include a test profile and test data. We have also indicated that geniac integrates several Make commands to facilitate the test of the pipeline. The abstract states that workflow systems + containers increase usage complexity. I am not sure that Geniac tackles this problem, since it adds a new layer of complexity in the top of Nextflow, Docker, etc (which is unavoidable).
Toni Hermoso Pulido Centre for Genomic Regulation, The Barcelona Institute of Science and Technology, Barcelona, Spain In this article, the Geniac toolbox is presented. The software consists of several tools to address some common difficulties of deploying and adapting Nextflow pipelines to fit into different computing environments. Its major emphasis is ensuring both code reproducibility and production-level reliability. Technically speaking, it depends on Nextflow itself and CMake automation and packaging tool. A Python-based code analysis (linter) tool is also provided for checking pipeline setup in advance before the actual build and deployment process takes place. Provided software for the pipelines relies primarily on Conda package manager, which Geniac facilitates to adapt to different approaches (e.
g. , single or multi), and converts to handy container images such as Singularity-based ones. (build, install) , as depicted in Figure  3 , could be more clearly stated. Reply: The Figure  3B  and 3C have been modified to highlight where the files automatically generated in the build directory are copied in the install directory. Despite it being mentioned in the article and in the toolbox documentation as an actual possibility, a more explicit example case of a software step that does not rely on Conda could be a nice addition. Reply: We added a new section in the manuscript to describe how to "Add a Nextflow process for a tool available from source code". As a user experience suggestion for future releases, it could be interesting that the geniac executable could be already provided straight from the Python Package Index (PyPI).
Reply: The Geniac command line interface is now available on PyPI at https://pypi. org/project/geniac/. It would also help new adopters, specially if not so familiar with CMake, if geniac could wrap some of the execution calls that are included in the useCases. bash script. Reply: New options have been added to the Geniac command line interface to facilitate the use of geniac for developers not familiar with CMake commands. The new section "Geniac command line interface" has been added in the manuscript to illustrate what command lines can be used for the proposed use case. The documentation has been updated with these new functionalities.
Methods: "Other folders are also created which contains"=>"contain". ○ Is the rationale for developing the new software tool clearly explained? Yes Is the description of the software tool technically sound? Partly Are sufficient details of the code, methods and analysis (if applicable) provided to allow replication of the software development and its use by others? Partly Is sufficient information provided to allow interpretation of the expected output datasets and any results generated using the tool? Yes Competing Interests: No competing interests were disclosed. Reviewer Expertise: I work in the field of evolutionary bioinformatics where I develop tools and workflows. I am interested in the reproducibility of bioinformatics analyses, and I use Nextflow and container technologies on a daily basis.
Experienced practitioners familiar with Nextflow, Conda and container technologies can easily follow the instructions and customize the provided examples (which are based on both legacy and DSL2 Nextflow syntax). Maybe the final result and associated folders of the pipeline preparation automation and packaging tool. A Python-based code analysis (linter) tool is also provided for checking pipeline setup in advance before the actual build and deployment process takes place. Provided software for the pipelines relies primarily on Conda package manager, which Geniac facilitates to adapt to different approaches (e. g. , single or multi), and converts to handy container images such as Singularity-based ones. Experienced practitioners familiar with Nextflow, Conda and container technologies can easily follow the instructions and customize the provided examples (which are based on both legacy and DSL2 Nextflow syntax).
Software availability • Source code for geniac available from: https://github. com/bioinfo-pf-curie/geniac • Archived source code at time of publication: https://doi. org/10. 5281/zenodo. 4883661 (Hupé et al. , 2021a)    • Source code for geniac-demo available from: https://github. com/bioinfo-pf-curie/geniac-demo • Archived source code at time of publication: https://doi. org/10. 5281/zenodo. 4884250 (Hupé et al. , 2021b)    • Source code for geniac-demo-dsl2 available from: https://github. com/bioinfo-pf-curie/geniac-demo-dsl2 • Archived source code at time of publication: https://doi. org/10. 5281/zenodo. 4883720 (La Rosa et al. , 2021) • Source code for geniac-template available from: https://github. com/bioinfo-pf-curie/geniac-template • Archived source code at time of publication: https://doi.
○ Methods: "Other folders are also created which contains"=>"contain". ○ Is the rationale for developing the new software tool clearly explained? Yes Is the description of the software tool technically sound? Partly Are sufficient details of the code, methods and analysis (if applicable) provided to allow replication of the software development and its use by others? Partly Is sufficient information provided to allow interpretation of the expected output datasets and any results generated using the tool? Yes Competing Interests: No competing interests were disclosed. Reviewer Expertise: I work in the field of evolutionary bioinformatics where I develop tools and workflows. I am interested in the reproducibility of bioinformatics analyses, and I use Nextflow and container technologies on a daily basis.
Sample Collection A total of 482 venous blood samples, collected between 2015 to 2019 from patients hospitalized in France with imported malaria from an African malaria-endemic country, was sent within 72 h after collection from different civilian or military hospitals of the French National Reference Centre for Imported Malaria network to the French National Reference Centre for Malaria (IRBA, IHU Méditerranée Infection Marseille, Marseille, France). Epidemiological and clinical data were collected for each isolate. Species identification was determined on blood smear and confirmed by real-time PCR  [90].
Gene Sequencing Pfcrt  [94] , Pfdhfr  [92] , Pfmdr1  [37] , Pfmdr5  [36] , Pfmdr6  [42] , PfK13  [95] , Pfubq  [91] , Pfcarl  [74] , Pfugt  [74] , Pfact  [74] , and Pfcoronin  [66]  were sequenced by Sanger method as previously described. The copy number of Pfpm2 was quantified by quantitative PCR as previously described  [96, 97]. Sequencing was performed on ABI Prism 3100 analyzer (Applied Biosystems, Foster City, CA, USA). The Vector NTI advance TM software (version 11, Invitrogen, Cergy Pontoise, France) and Codon Code aligner (Codon Code corporation, Centerville, OH, USA) were used to analysis chromatograms.
Statistical Analysis R software with ggplot2 package for Dot plots, Inserm UPMC biostaTGV (http:// websenti. u707. jussieu. fr/ (accessed on 26 March 2021)) tool and Excel statistical resource pack Real statistics ANOVA (Charles Zaiontz-http://realstatistics. com (accessed on 26 March 2021)) were used in order to perform all statistical tests of this project. The Multiple correspondence analysis (MCA) was performed using R package Fac-toMineR  [98]  and Factoextra (http://www. sthda. com/english/rpkgs/factoextra (accessed on 26 March 2021)) on RStudio version 1. 3. 1093. The objective of MCA is to identify direction (the principal components) for which the variation of the data is maximal, and thus those directions which are linked to a genotype in particular can explain the behavior of the phenotype of our isolates.