Package: pandoo
PandooTo get help::pandoo -hAuthorsMark B Schultz, Anders Gonçalves da Silva, Jason Kwong, Torsten SeemannIntroductionPandoois a command-line tool for exploring and characterising bacterial whole genome DNA sequence data.  It is a computational pipeline written in python and is scalable via implementation using the ruffus pipeline library.Ruffushandles task-scheduling and task-parallelisation during the run.  Pandoo is particularly useful for pathogenic species as it performs AMR and MLST profiling but in theory could be used for any bacterial species against any sequence database.  It was originally written to characterise and QC assemblies and reads of bacterial isolates at the Microbiological Diagnostic Unit Public Health Laboratory, Victoria, Australia.  Input is a tab-delimited text file that points the software to the assembly and paired-end read files for each isolate.  Specifically, the file has no header, one line per isolate, and four columns per line in the following column order::+-------------+------------------------------------------------+----------------------------+---------------------------+
|isolate_name | full_path_to_assembly (i.e., a 'contigs' file) | full_path_to_paired_reads1 | full_path_to_paired_reads2|
+-------------+------------------------------------------------+----------------------------+---------------------------+An exampleisolates.tabfile looks like this::isolate-1	/path/to/isolate-1/contigs.fa	/path/to/isolate-1/reads-1.fq.gz	/path/to/isolate-1/reads-2.fq.gz
isolate-2	/path/to/isolate-2/contigs.fa	/path/to/isolate-2/reads-1.fq.gz	/path/to/isolate-2/reads-2.fq.gz
isolate-3	/path/to/isolate-3/contigs.fa	/path/to/isolate-3/reads-1.fq.gz	/path/to/isolate-3/reads-2.fq.gz
isolate-4	/path/to/isolate-4/contigs.fa	/path/to/isolate-4/reads-1.fq.gz	/path/to/isolate-4/reads-2.fq.gz
isolate-5	/path/to/isolate-5/contigs.fa	/path/to/isolate-5/reads-1.fq.gz	/path/to/isolate-5/reads-2.fq.gzInstalling dependenciesThe following packages need to be installed before running pandoo.  To install dependencies, do something like this (you may need to add '--user' after pip3 install commands if you don't have admin privileges)::cpanm -i Moo
cpanm -i List::MoreUtils
cpanm -i Bio::Perl
brew tap homebrew/science
brew tap tseemann/homebrew-bioinformatics-linux
brew update
brew install mlst
brew install abricate
brew install seqtk
brew install mummer
brew install bowtie2
brew install cd-hit
brew install kraken
brew install graphviz
pip3 install git+https://github.com/MDU-PHL/ngmaster.git #(see install instructions at https://github.com/MDU-PHL/ngmaster)
pip3 install git+https://github.com/MDU-PHL/meningotype.git #(see install instructions at https://github.com/MDU-PHL/meningotype)
pip3 install biopython
pip3 install ariba
#install sistr from http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0147101 (location?)
git clone https://github.com/MDU-PHL/legsta.git # then add it to your path so is executable with 'legsta'
git clone https://github.com/MDU-PHL/LisSero.git # then add it to your path so is executable with 'LisSero.py'
git clone https://github.com/lskatz/mashtree.git # then add it to your path so is executable with 'mashtree.pl'Follow the instructions athttps://ccb.jhu.edu/software/krakento set up the databases.Installing PandooTo perform any of these install stepsfor all users, remove '--user'.  The final symlink step is not required if installing for all users.Pandoo is written forpython3and installation requirespip3andsetuptools.  To install the latest 'stable' version of pandoo for the current user only, do::First updatepython3,pip3,setuptools.  Then try:
pip3 install pandoo --user -I #-I means ignore those already installed and reinstall the versions as stated in setup.py, especially needing ruffus version 2.6.2 instead of ruffus 2.6.3.To upgrade::pip3 install pandoo --upgradeTo install the latest, potentially unstable, bleeding-edge version::pip3 install --user https://github.com/schultzm/pandoo/zipball/masterIf installing via the '--user' optionCheck where the executable is::which pandoo  # ~/.local/bin/pandooCheck where the site-packages are::python3 -m site --user-site  # ~/.local/lib/python3.6/site-packagesNow, symlink the packaged databases in site-packages above to the folder containing the executable shown above::ln -s ~/.local/lib/python3.6/site-packages/Pandoo/CARD/ ~/.local/bin
ln -s ~/.local/lib/python3.6/site-packages/Pandoo/VFDB/ ~/.local/bin
ln -s ~/.local/lib/python3.6/site-packages/Pandoo/qacGenes/ ~/.local/binTo uninstall, do::pip3 uninstall pandoo
# Remove the symlinks
rm -r ~/.local/bin/CARD
rm -r ~/.local/bin/VFDB*
rm -r ~/.local/bin/plasmidfinderIf accidentally installed with pip2 or pip, do::
pip uninstall pandooCheck that the dependencies are in the path using::pandoo checkAdd dependencies as required.Quickstart tutorialFor large jobs, run in screen mode, otherwise skip this step::screen -SL screennameAll screen-output within the screen started above, will be saved to::screenlog.%n #  Where %n is the number of the screen.After exiting the screen, the screenlog.0 can be viewed as the run progresses using standard command line actions::tail screenlog.0
less screenlog.0
watch "tail screenlog.0"To get help for pandoo, just do::pandoo -hOutput looks like::usage: pandoo <command> <options>

This is a tool for exploring your bacterial genome data. Given some assemblies
and/or paired-end read sets, run a pipeline of software tools to generate an
NJ tree from assemblies and a complementary table of metadata/results (contig
and read QC, mlst, species ID, resistance genes, virulence factors, plasmid
replicon types).

optional arguments:
  -h, --help     show this help message and exit
  -v, --version  Print version and quit.

Commands:

    check        Check pipeline dependencies
    input        Generate input table
    run          Run the pandoo pipeline
    merge        Merge two metadata tablesNotice, above, four modules: one each forcheck,input,runandmerge.  Each can be run independently.check moduleThis module will check if the required softwares are installed and executable as per the calls to these programs used by pandoo.input moduleThis module is used to generate the isolates.tab file.  Final output from this command is sent tostandard out (stdout).  To capture the information from stdout, redirect it to a file using '> isolates.tab'::pandoo input -h
pandoo input -i isolates.txt > isolates.tabrun moduleThis module is used to run the analysis pipeline.  In the example below, output will be a folder calledresultsand we have selected to use thetree option with -t::pandoo run -h
pandoo run -i isolates.tab -o results -tIn the results folder there is a sub-folder for each isolate containing the results for each isolate.  Also within the results folder there are three files::results/isolates_metadataAll.csv
results/isolates_metadataAll_simplified.csv
results/isolates_mashtree.tremerge moduleThis module is used to join existing metadata tables (e.g., LIMS table) with one of the output tables from the run module.  In this example, an excel file (AMR_ongoing_20170307.xlsx, with a header on line 5 and first column containing isolate names that were expanded with a wildcard search during thepandoo inputstep above) is joined with the results/isolates_metadataAll_simplified.csv table.  Again, final output is sent to stdout, which in this example is redirected to file using '> results/AMR_ongoing_20170307_join_isolates_metadataAll_simplified.csv'::pandoo merge -h
pandoo merge -l AMR_ongoing_20170307.xlsx -r results/isolates_metadataAll_simplified.csv > results/AMR_ongoing_20170307_join_isolates_metadataAll_simplified.csvWhy use Pandoo?Pandoo provides a single table of metadata and a tree that the metadata can be plotted next to usingphandangohttps://jameshadfield.github.io/phandango/The single NJ tree for the whole isolate set is inferred using the programmashtree.plfrom a distance matrix computed bymash.  The tree will not include isolates for which the assembly file is missing.The summary table (csv) for all isolates combines the results from:inferred species from running kraken on the readsinferred species call from running kraken on the contigs (assemblies)Inferred consensus species call from a consensus of the best hit from kraken on the reads and kraken on the contigsBased on the species call, runs mlst using the appropriate scheme (if available) or autodetects schemeGene content profiles using unlimited number of user databases of resistance genes, plasmid rep genes, virulence genes, etc., using abricate (contigs, BLAST contigs against database, assembly based) and ariba (reads, MiniMapping to database, mapping based)QC metrics using seqtk for reads and contigsReports software versions and paths to databases used in the analysis (for repeatability)The pipeline is modular in that the user can choose not to perform the tree inference step and/or the user can choose not to perform the read mapping step using aribaA flowchart is produced for the run (however, if the total path length of the results folders combined exceeds 16384 characters then the flowchart cannot be drawn)The user can supply reads and/or contigs for each file.  The final tree will only include taxa for which contigs have been suppliedAssumptionsFirst you need to download assemblies or perform the assemblies yourself from readsets (using e.g., unicyclerhttps://github.com/rrwick/Unicycler, which uses SPAdes (http://bioinf.spbau.ru/spades) or MegaHit (https://github.com/voutcn/megahit)).  It doesn't necessarily make sense to supply reads for an isolate but contigs that have been assembled using a readset other than the one supplied.  If you don't have reads, leave the columns blank for that isolate (for example, if you just want to characterise assemblies downloaded from NCBI GenBank). If you don't have contigs and only have reads, leave the column blank for contigs (but without contigs there will be no tree).LicencePandoo is:| Copyright (C) 2017 Mark B Schultz|https://github.com/schultzm/| email:dr.mark.schultz@gmail.comThis program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as published
by the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU Affero General Public License for more details.
You should have received a copy of the GNU Affero General Public License
along with this program.  If not, seehttp://www.gnu.org/licenses/.