Package: fastqwiper
FastqWiperFastqWiperis a Snakemake-enabled application that wipes out bad reads from broken FASTQ files. Additionally, the available and pre-designed Snakemakeworkflowsallowsrecoveringcorruptedfastq.gz,droppingorfixingpesky lines,removingunpaired reads, andfixingreads interleaving.Compatibility: Python ≥3.7, <3.11OS: Windows, Linux, Mac OS (Snakemake workflows through Docker for Windows)Contributions:bioinformatics@css-mendel.itDocker:https://hub.docker.com/r/mazzalab/fastqwiperSingularity:https://cloud.sylabs.io/library/mazzalab/fastqwiper/fastqwiper.sifBug report:https://github.com/mazzalab/fastqwiper/issuesUSAGECase 1. You have one or a couple (R1&R2) ofcomputer readableFASTQ files which contain pesky, unformatted, uncompliant lines: UseFastWiperto clean them;Case 2. You have one or a couple (R1&R2) ofcomputer readableFASTQ files that you want to drop unpaired reads from or fix reads interleaving: Use the FastqWiper'sSnakemake workflows;Case 3. You have onefastq.gzfile or a couple (R1&R2) offastq.gzfiles which are corrupted (unreadable) and you want to recover healthy reads and reformat them: Use the FastqWiper'sSnakemake workflows;InstallationCase 1This requires you to install FastqWiper and therefore does not require you to configureworkflowsalso. You can do it for all OSs:Use Condaconda create -n fastqwiper python=3.10
conda activate fastqwiper
conda install -c bfxcss -c conda-forge fastqwiper

fastqwiper --helpHint: for an healthier experience, usemambaUse Pypipip install fastqwiper

fastqwiper --helpfastqwiper  <options>options:
  --fastq_in TEXT          The input FASTQ file to be cleaned  [required]
  --fastq_out TEXT         The wiped FASTQ file                [required]
  --log_frequency INTEGER  The number of reads you want to print a status message
  --log_out TEXT           The file name of the final quality report summary
  --help                   Show this message and exit.It  accepts in input and outputsreadable*.fastqor*.fastq.gzfiles.Cases 2 & 3There areQUICKand aSLOWmethods to configureFastqWiper's workflows.One quick way (Docker)Pull the Docker image from DockerHub:docker pull mazzalab/fastqwiperOnce downloaded the image, type:CMD:docker run --rm -ti --name fastqwiper -v "YOUR_LOCAL_PATH_TO_DATA_FOLDER:/fastqwiper/data" mazzalab/fastqwiper paired 8 sample 50000000Another quick way (Singularity)Pull the Singularity image from the Cloud Library:singularity pull library://mazzalab/fastqwiper/fastqwiper.sifOnce downloaded the image (e.g., fastqwiper.sif_2023.2.70.sif), type:CMDsingularity run --bind /scratch/tom/fastqwiper_singularity/data:/fastqwiper/data --writable-tmpfs fastqwiper.sif_2023.2.70.sif paired 8 sample 50000000If you want to bind the.singularitycache folder and thelogsfolder, you can omit--writable-tmpfs, create the folders.singularityandlogs(mkdir .singularity logs) on the host system, and use this command instead:CMD:singularity run --bind YOUR_LOCAL_PATH_TO_DATA_FOLDER/:/fastqwiper/data --bind YOUR_LOCAL_PATH_TO_.singularity_FOLDER/:/fastqwiper/.snakemake --bind YOUR_LOCAL_PATH_TO_LOGS_FOLDER/:/fastqwiper/logs fastqwiper.sif_2023.2.70.sif paired 8 sample 50000000For bothDockerandSingularity:YOUR_LOCAL_PATH_TO_DATA_FOLDERis the path of the folder where the fastq.gz files to be wiped are located;pairedtriggers the cleaning of R1 and R2. Alternatively,singlewill trigger the wipe of individual FASTQ files;8is the number of your choice of computing cores to be spawned;sampleis part of the names of the FASTQ files to be wiped.Be awarethat: forpaired-endfiles (e.g., "sample_R1.fastq.gz" and "sample_R2.fastq.gz"), your files must finish with_R1.fastq.gzand_R2.fastq.gz. Therefore, the argument to pass is everything before these texts:samplein this case. Forsingle end/individual files (e.g., "excerpt_R1_001.fastq.gz"), your file must end with the string.fastq.gz; the preceding text, i.e., "excerpt_R1_001" in this case, will be the text to be passed to the command as an argument.50000000is the number of rows-per-chunk (used when cores>1. It must be a number multiple of 4)The slow way (Linux & Mac OS)To enable the use of preconfiguredpipelines, you need to installSnakemake. The recommended way to install Snakemake is via Conda, because it enablesSnakemaketohandle software dependencies of your workflow.
However, the default conda solver is slow and often hangs. Therefore, we recommend installingMambaas a drop-in replacement via$ conda install -c conda-forge mambaif you have anaconda/miniconda already installed, or directly installingMambaforgeas describedhere.Then, create and activate a clean environment as above:mamba create -n fastqwiper python=3.10
mamba activate fastqwiperFinally, install a few dependencies:$ mamba install -c bioconda snakemake
$ mamba install colorama clickUsageClone the FastqWiper repository in a folder of your choice and enter it:git clone https://github.com/mazzalab/fastqwiper.git
cd fastqwiperIt contains, in particular, a folderdatacontaining the fastq files to be processed, a folderpipelinecontaining the released pipelines and a folderfastq_wiperwith the source files ofFastqWiper.Input files to be processed should be copied into thedatafolder.Currently, to run theFastqWiperpipelines, the following packages need to be installed manually:required packages:gzrt(Linux build fron sourceinstructions, Ubuntu installinstructions, Mac OS installinstructions)BBTools(installinstructions)If installed from source,gzrtscripts need to be put on PATH.bbmapmust be installed in the root folder of FastqWiper, as the image belowCommands:Copy the fastq files you want to fix in thedatafolder.N.b.: In all commands above, you will pass to the workflow the name of the sample to be analyzed through the config argument:sample_name. Remember that your fastq files' names must finish with_R1.fastq.gzand_R2.fastq.gz, for paired fastq files, and with.fastq.gz, for individual fastq files, and, therefore, the text to be assigned to the variablesample_namemust be everything before them. E.g., if your files aremy_sample_R1.fastq.gzandmy_sample_R2.fastq.gz, then--config sample_name=my_sample.Paired-end filesGet a dry runof a pipeline (e.g.,fix_wipe_pairs_reads_sequential.smk):snakemake --config sample_name=my_sample -s pipeline/fix_wipe_pairs_reads_sequential.smk --use-conda --cores 4Generate the planned DAG:snakemake --config sample_name=my_sample -s pipeline/fix_wipe_pairs_reads_sequential.smk --dag | dot -Tpdf > dag.pdfRun the pipeline(n.b., during the first execution, Snakemake will download and install some required remote packages and may take longer). The number of computing cores can be tuned accordingly:snakemake --config sample_name=my_sample -s pipeline/fix_wipe_single_reads_sequential.smk --use-conda --cores 2Fixed files will be copied in thedatafolder and will be suffixed with the string_fixed_wiped_paired_interleaving.
We remind that thefix_wipe_pairs_reads_sequential.smkandfix_wipe_pairs_reads_parallel.smkpipelines perform the following actions:executegzrton corrupted fastq.gz files (i.e., that cannot be unzipped because of errors) and recover readable reads;executeFastqWiperon recovered reads to make them compliant with the FASTQ format (source:Wipipedia)executeTrimmomaticon wiped reads to remove residual unpaired readsexecuteBBmap (repair.sh)on paired reads to fix the correct interleaving and sort fastq files.Single-end filesfix_wipe_single_reads_parallel.smkandfix_wipe_single_reads_sequential.smkwill not executetrimmomaticand BBmap'srepair.sh.Get a dry runof a pipeline (e.g.,fix_wipe_single_reads_sequential.smk):snakemake --config sample_name=my_sample -s pipeline/fix_wipe_single_reads_sequential.smk --use-conda --cores 2 -npGenerate the planned DAG:snakemake --config sample_name=my_sample -s pipeline/fix_wipe_single_reads_sequential.smk --dag | dot -Tpdf > dag.pdfRun the pipeline(n.b., The number of computing cores can be tuned accordingly):snakemake --config sample_name=my_sample -s pipeline/fix_wipe_single_reads_sequential.smk --use-conda --cores 2AuthorTommaso MazzaLaboratory of BioinformaticsFondazione IRCCS Casa Sollievo della SofferenzaViale Regina Margherita 261 - 00198 Roma ITTel: +39 06 44160526 - Fax: +39 06 44160548E-mail:t.mazza@css-mendel.itWeb page:http://www.css-mendel.itWeb page:http://bioinformatics.css-mendel.it