Package: mdmcleaner
MDMcleanerA pipeline for the assessment, classification and refinement of microbial dark matter SAGs and MAGsMDMcleaner is a reference-DB contamination aware pipeline for reliable contig classification of metagenome assembled (MAG) and single-cell amplified (SAG) genomes.
It is based on the GTDB taxonomic system and uses GTDB representative genomes, as well as SILVA SSU and LSU and RefSeq eukaryotic and viral datasets as references. Classification is based on a "least common ancestor" (LCA) approach, that is implemented in a way that can recognize potential contaminants not only in the analyzed genome, but also in the underlying reference datasets. Furthermore each contig is classified only up to taxlevels that are actually supported by the corresponding alignment identities, thereby avoiding overclassification for organisms that are underrepresented in the reference database.Dependencies:python 3.7+biopython v.1.78+wget v.1.19+ (downloading reference datasets)ncbi-blast v.2.10.1+ (aligning nucleotide sequences)diamond v.2.0.6+ (aligning amino acid sequences)hmmer v.3.3.1+ (detecting conserved marker genes)barrnap v.0.9+ (detecting ribosomal RNA genes)aragorn v 1.2.38+ (detecting tRNA genes)prodigal v 2.6.3 (ORF/CDS-prediction)This repository is hosted atgithuband mirrored atgitlab.Installation:The MDMcleaner is now installable viapip(without dependencies), andBioconda. In case of pip, all dependencies listed above need to be installed seperately.to install via pip:pip install mdmcleanerto install via conda:conda install -c bioconda mdmcleanerto install directly from this repository:either clone this repository or download and extract the latest release.enter the repository folder:cd mdmcleanercreate wheel: ```python setup.py bdist_wheel``´install from local wheel file:pip install dist/*.whlConfigurationSeveral options can be passed directly as commandline arguments (see usage below), but basic settings, such as database location, should be provided in the form ofmdmcleaner.configconfig files. The pipeline distinguishes between global (system/environment-wide settings) and local (individual) config files.The hierarchy is as follows:command line arguments override global and local settinglocal settings override global settingsglobal settings are set in themdmcleaner.configfile within the mdmcleanerlibfolder (best accessed viamdmcleaner.py set_configs -s globalthe settings that can be specified/adjusted in the config files are:location of blastn binaries (default is simply "blastn", which assumes it is present in PATH)location of blastp binaries (default is simply "blastp", which assumes it is present in PATH)location of barrnap binaries (default is simply "barrnap", which assumes it is present in PATH)location of aragorn binaries (default is simply "blastp", which assumes it is present in PATH)To create a local config file in the current working directory, simply usemdmcleaner set_configs -s local [SETTING_ARGUMENTS]This file can be moved and copied and will be automatically recognized if present in the current working directory when running MDMcleaner. Alternative the path to a local config file can be passed to mdmcleaner via the "-c" argument of the "clean" and "makedb" workflows.overview of MDMcleaner commandsA list of mdmcleaner commands is returned when invoking the help function of MCMcleaner as follows:mdmcleaner -h. Each command has it's own help function that can be invoked withmdmcleaner <COMMAND> -h. The available commands are:set_configscan be used to change global or local settings. Will modify or create 'mdmcleaner.config'-filesshow_configslists the currently applicable MDMcleaner settings/configurationsmakedbdownloads and processes reference data into a MDMclenaner reference database. May have a LONG run-time but can be aborted and resumedcleanthe major MDMcleaner workflow for assessing and filtering genome contaminationget_markersan accessory command for extracting marker gene sequences from input genomescompletenessan accessory command for "quick-and-dirty" assessment of bin completeness based on universally required types or tRNAsrefdb_contamsEXPERIMENTAL: evaluates refDBambiguity overviewfiles and adds obvious refDB contaminations to the blacklistacc2taxpathGet full taxonomic path associated with a specific input accession. Currently only works for MDMcleaner/GTDB accessions, but support for NCBI accession-numbers will follow sooncheck_dependenciesJust check if all dependencies are being metversionshow version info and quitusage ofmdmcleaner set_configs:usage: mdmcleaner set_configs [-h] [-s {local,global}] [--blastp BLASTP] [--blastn BLASTN] [--diamond DIAMOND] [--barrnap BARRNAP] [--hmmsearch HMMSEARCH] [--aragorn ARAGORN] [--db_basedir DB_BASEDIR]
                                 [--threads THREADS]

optional arguments:
  -h, --help            show this help message and exit
  -s {local,global}, --scope {local,global}
                        change settings in local or global config file. 'global' likely require admin privileges. 'local' will modify or create a mdmcleaner.config file in the current working directory.
                        default = 'local'
  --blastp BLASTP       path to blastp binaries (if not in PATH)
  --blastn BLASTN       path to blastn binaries (if not in PATH)
  --diamond DIAMOND     path to diamond binaries (if not in PATH)
  --barrnap BARRNAP     path to barrnap binaries (if not in PATH)
  --hmmsearch HMMSEARCH
                        path to hmmsearch binaries (if not in PATH)
  --aragorn ARAGORN     path to aragorn binaries (if not in PATH)
  --db_basedir DB_BASEDIR
                        path to basedirectory for reference database
  --threads THREADS     threads to use by defaultDownloading and creating the reference databaseThe reference database is created by runningmdmcleaner.py makedb. This downloads the most recent datasets from GTDB, RefSeq and Silva and processes them into the format used by MDMcleaner, ensuring that users can access the newest reference data independently of our ability to keep our database current. The download and processing may take a long time (>13h at 100 Mbit/s), but can be resumed from the last checkpoint, simply by running it again if aborted.Alternatively you can use the ready made database used during the work on the mdmcleaner publication. This has been deposited at zenodo for reproducability reasons. You find and download it atthis link, or use the--get_pub_dataargument ofmdmcleaner.py makedbRemember to add the location of the database directory to a config file!Preferrably to the global configuration file by runningmdmcleaner.py set_configs --db_basedir <path/to/databasefolder>If you specify the target database directory in a config file, you do not need to use the-ooption of makedb (instead the target folder will be automatically read from the configs file)usage ofmdmcleaner makedb:usage: mdmcleaner makedb [-h] [-o OUTDIR] [-c CONFIGFILE] [--verbose] [--quiet]

optional arguments:
  -h, --help            show this help message and exit
  -o OUTDIR, --outdir OUTDIR
                        target base directory for reference-data. may not be the current working directory. Needs >100GB space! Default = './db/gtdb'
  -c CONFIGFILE, --config CONFIGFILE
                        provide a local config file with the target location to store database-files. default: looks for config files named 'mdmcleaner.config' in current working directory. settings in
                        the local config file will override settings in the global config file '/home/ww5070/temp_binsaga/sagabin_refiner/lib/mdmcleaner.config'
  --get_pub_data        get the ready made reference dataset used during the work on the mdmcleaner publication (WARNING: is outdated and only provided for reproducability and/or testing purposes)
  --verbose             verbose output (download progress etc)
  --quiet               quiet mode (suppress any status messages except Errors and Warnings)Processing MAGs and SAGsTo process MAGs and SAGs, simply use the "clean" function of MDMcleaner. If you have multiple genomes, specify them all at once instead of calling MDMcleaner individually for each genome. This apeeds up processing time by allowing to reuse of database objects without needing to reinitialize them for every query genome.
For safety reasons the number of threads used is set to 1 by default. However many steps of the MDMcleaner pipeline can profit from using multiple threads. Remember to specify as many threads as you can safely use on your system.For each genome, intermediary results are stored in a seperate subfolder in "mdmcleaner_results". Delete them only when you are finished, as these allow resuming or rerunning analyses without having to redo all analyses from scratch...By default, mdmcleaner will try to resolve any potential reference-database ambiguities it encounters. This will increase run time and may not always be successful. If speed is of essence, this can be skipped by using the--fast_runargumentIf possible, we recommend to use at least 8 threads (-t 8) for mdmcleaner runs. With 8 threads it will take about 20-30 minutes per complete & average-sized bacterial genomeusage ofmdmcleaner clean:usage: mdmcleaner clean [-h] -i INPUT_FASTAS [INPUT_FASTAS ...] [-o OUTPUT_FOLDER] [-v] [-c CONFIGFILE] [-t THREADS] [-f] [--overview_files_basename OVERVIEW_BASENAME] [-I IGNORELISTFILE]
                           [--no_filterfasta]

optional arguments:
  -h, --help            show this help message and exit
  -i INPUT_FASTAS [INPUT_FASTAS ...], --input_fastas INPUT_FASTAS [INPUT_FASTAS ...]
                        input fastas of genomes and/or bins
  -o OUTPUT_FOLDER, --output_folder OUTPUT_FOLDER
                        output-folder for MDMcleaner results. Default = 'mdmcleaner_output'
  -v, --version         show program's version number and exit
  -c CONFIGFILE, --config CONFIGFILE
                        provide a local config file with basic settings (such as the location of database-files). default: looks for config files named 'mdmcleaner.config' in current working directory.
                        settings in the local config file will override settings in the global config file '/home/ww5070/temp_binsaga/sagabin_refiner/lib/mdmcleaner.config'
  -t THREADS, --threads THREADS
                        Number of threads to use. Can also be set in the mdmcleaner.config file
  -f, --force           Force reclassification of pre-existing blast-results
  --overview_files_basename OVERVIEW_BASENAME
                        basename for overviewfiles (default="overview"
  -b BLACKLISTFILE, --blacklistfile BLACKLISTFILE
                        File listing reference-DB sequence-names that should be ignored during blast-analyses (e.g. known refDB-contaminations...
  --no_filterfasta      Do not write filtered contigs to final output fastas (Default = False)
  --fast_run            skips evaluation of reference-database-ambiguities. Runs will be faster but ambiguities will not be resolved on the go!OUTPUT FILESRunning thecleanoption of MDMcleaner.py will create a mdmcleaner_results folder in the current orking directory. Individual results for each genome are saved in individual subfolders of "mdmcleaner_results". This includes a detailed report on the individual contig classifications, a input-table for vizualization with KRONA and (optionally) filtered contig fastas, divided into the four categories "keep", "evaluate_low", "evaluate_high" and "delete":individual output files per genome:fullcontiginfos_beforecleanup.tsv --> a tab seperated table with details for each contig<genome-name>_rRNA_lsu_rRNA.fasta --> 23s rRNA genes<genome-name>_rRNA_ssu_rRNA.fasta --> 16s rRNA genes<genome-name>_rRNA_tsu_rRNA.fasta --> 5s rRNA genes<genome-name>_tRNAs.fasta.gz --> tRNA genes<genome-name>_totalprots.faa --> total protein sequences<genome-name>_keep.fasta --> trusted contigs that are safe to submit<genome-name>_evaluate_low.fasta --> contigs that yielded some kind of refDB ambiguity andmayprofit from some individual re-evaluation (cross-blast against RefSeq etc)<genome-name>_evaluate_high.fasta --> contigs that yielded indications of refDB contamiations and should not be submitted before detailed re-evaluation<genome-name>_delete.fasta --> untrusted contigs that should definitively not be included in the genome submissionvarious intermediary files and progress markers...Additionally, general overview files are written to the current working directory: "overview_allbeforecleanup.tsv", "overview_refdb_ambiguities.tsv" and "overview_errorlist.txt"general overview filesoverview_allbeforecleanup.tsv --> tab seperated table listing majority classification and general metrics per analyzed binoverview_refdb_ambiguities.tsv --> tab seperated table listing detected refDB ambiguities together with evidence information and preliminary MDMcleaner assessmentsoverview_errorlist.txt --> list of genomes that yielded errors during MDMcleaner assessments and may need to be rerunHow to cite:please cite our publication:Vollmers, John, et al. "How clear is our current view on microbial dark matter?(Re-) assessing public MAG & SAG datasets with MDMcleaner." Nucleic Acids Research (2022)For help or feedback:please write an issue on thegithub repositoryor send us anemail