Package: metator
MetaTORMetagenomic Tridimensional Organisation-based Reassembly - A set of scripts that streamlines the processing and binning of metagenomic metaHiC datasets.Table of contentsInstallationRequirementsUsing pipUsing docker containerUsageOutput filesReferencesContactInstallationRequirementsPython 3.6 or later is required.The following librairies are required but will be automatically installed with the pip installation:numpy,scipy,sklearn,pandas,docopt,networkxbiopythonpyfastxandpysam.The following software should be installed separately if you used the pip installation:bowtie2samtoolslouvain(original
implementation).networkanalysis(not
necessary only if you want to use Leiden algorithm to partition the network)micompletepairix(not necessary if you do not use metator contact map)Using pippip3installmetatoror, to use the latest version:pip3install-egit+https://github.com/koszullab/metator.git@master#egg=metatorUsing condacondacreate-nmetator-y--log-levelwarning-fmetator.yamlLouvain or Leiden dependencyIn order to use Louvain or Leiden it's necessary to set a global variableLOUVAIN_PATHandLEIDEN_PATHdepending on which algorithm you wan to use with the absolute path where the executable are.For Louvain algorithm in the directory where you have the archive file (available in the external directory of this repository):YOUR_DIRECTORY=$(pwd)tar-xvzflouvain-generic.tar.gzcdgen-louvain
makeexportLOUVAIN_PATH=$YOUR_DIRECTORY/gen-louvain/For Leiden algorithm, clone the networkanalysis repository from github and build the Java script. Then you can export the Leiden path:exportLEIDEN_PATH=/networkanalysis_repository_path/build/libs/networkanalysis-1.2.0.jarUsing docker containerA dockerfile is also available if that is of interest. You may fetch the image by running the following:dockerpullkoszullab/metatorUsagemetator{network|partition|validation|pipeline}[parameters]A metaTOR command takes the formmetator action --param1 arg1 --param2 arg2 #etc.There are three actions/steps in the metaTOR pipeline, which must be run in the
following order:network: Generate metaHiC contigs network from fastq reads or bam files and normalize it.partition: Perform the Louvain or Leiden community detection algorithm many times to bin contigs according to the metaHiC signal between contigs.validation: Use CheckM to validate the bins, then do a recursive decontamination step to remove contamination.There are a number of other, optional, miscellaneous actions:pipeline: Run all three of the above actions sequentially or only some of them depending on the arguments given. This can take a while.contactmap: Generates a contact map from one bin from the final ouptut of metaTOR.version: display current version number.help: display help message.A tutorial is availablehereto explain how to use metaTOR. More advanced tutorials to analyze the output files are also available:Anviomanual curation of the contaminated bins. Availablehere.Visualization and scaffolding of the MAGs with the contactmap modules of MetaTOR. Availablehere.Principle of MetaTOR pipeline:Output filesThe output files will be in the ouput directory given as parmater or in working directory if no paramater were given. Depending on the command used, different files will be in the ouptut:Files/Commandsdescriptionnetworkpartitionvalidationpipelinealignment_N_for.bamBam file of the forward alignmentXXalignment_N_rev.bamBam file of the reverse alignmentXXalignment_N.pairsPairs of the merge alignmentXXnetwork.txtNormalized network of the metaHiC libraryXXcontig_data_network.txtInformation on contigs after network stepXclustering_matrix_partition.txtMatrix of clustering from the partition iterationsXcontig_data_partition.txtInformation on contigs after partition stepXoverlapping_checkm_results.txtCheckM results summary file from the partition stepXXoverlapping_checkm_taxonomy.txtCheckM taxonomy file from the partition stepXXrecursive_checkm_results.txtCheckM results summary file from the recursive stepXXrecursive_checkm_taxonomy.txtCheckM taxonomy file from the recursive stepXXclustering_matrix_validation.txtMatrix of clustering from the recursive iterationsXclustering_matrix.txtMatrix of clustering from the partition and recursive iterationsXcontig_data_final.txtInformation on contigs after whole pipelineXXbin_summary.txtInformation on the final binsXXbinning.txtFile with contigs names and their final clusteringXXoverlapping_binDirectory with the fasta of the partition binsXXXrecursive_binDirectory with the fasta of the recursive binsXXfinal_binDirectory with the fasta of the final binsXXBam alignment filesFor the bam alignments files, only the aligned reads are kept and the bam are sorted by name. The N value correspond to the id (order of the given fastq started at 0)Pairs aligment filesThis format is used to store the relevant information of mapping of the merged alignment. It's a s The N value correspond to the id (order of the given fastq started at 0). It is a tab-separated format holding informations about Hi-C pairs. It has anofficial specificationdefined by the 4D Nucleome data coordination and integration center. Here we kept 7 columns readID-chr1-pos1-chr2-pos2-strand1-strand2.Network fileThis is a tsv file of the network with edgelist form: Id of the first contig, id of the second contig and the weigth of edge normalized.Contig data filesThese are the files with all the informations from the contigs:IDNameSizeGC_contentHitShotgun_coverageRestriction_siteCore_bin_IDCore_bin_contigsCore_bin_sizeOverlapping_bin_IDOverlapping_bin_contigsOverlapping_bin_sizeRecursive_bin_IDRecursive_bin_contigsRecursive_bin_sizeFinal_bin1NODE_164231138.6876450815882383741.156520061652175226139663223531522158803MetaTOR_1_12NODE_257635630.235826468363303172424.5091256240173541924017354190--MetaTOR_2_03NODE_354057142.305266098255366218814.585534053127640948434311361548011126385126MetaTOR_3_1They have to have the header when they are use as input but the order of the columns nd if they are others columns doesn't matter when they are used as input files.
Depending on which step of the pipeline have been launch they just have some of these columns:contig_data_network.txt: columns: ID, Name, Size, GC content, Hit, Shotgun_coverage and Restriction Site only.contig_data_partition.txt: The same as the previous with the information of core bins and overlapping bins.contig_data_final.txt: All the columns.The shotgun coverage will be filled only if the depth.txt file is given, otherwise it will be filled with-. This column is only necessarry for theabundanceand thetheoritical_hitnormalization. The restriction will also be filled with-if no enzyme are given. This column is only necessary for theRSand thetheoritical_hitnormalization.
Moreover, if the contig is not binned (no HiC reads mapped on it) all the columns with binning information will be filled with-, and if a bin is not recursively decontamined because it's smaller than the size threshold or it doesn't have any contamination the recusive bin information will be filled0,-,-. Finally, if the bin is not in a final bin, it will be annotatedNDin the last column (for not determined).clustering matrix filesThe clustering matrix files are at the.npzformat which is a compresed foramt for sparsed matrix. This sparsed matrix contains the ratio of time each pairs of contigs are clusterize together by the algorithm of clustering (either Louvain or Leiden). The partition matrix contains the information for the partition step, the recursive one for the recursive step and the general one is the mean of both matrices. Be careful the index of the contigs arezero-basedand not one-based as in the contig data file.It's possible to read them in python using thescipy.sparse.load_npz()function. If the users wants a tsv file instead, he or she could load the matrix in python using load_npz and make sure to transform the matrix in thescipy.sparse.coo_matrixfunction and used the function from metatormetator.io.save_sparse_matrixto save it as tsv file.CheckM resultsFiles fromcheckMoutput. Two types of files one with the main results of checkMcheckm_results.txtand one with the taxonomycheckm_taxonomy.txtfor both the partition and the recurisve bins.binning.txt fileThis is a tsv file with two columns: the contig name and the final were the contig is. It only contains contigs which are binned. It could be use a an input to import a binning results in anvio.Bin summary fileThis is the summary of the data of the final bins build with all the step of metaTOR. The HiC coverage is the number of contacts (intra and inter contigs) per kilobase in the whole bin. The Shotgun coverage is the mean coverage normalized by the size of the shotgun reads from the depth file.lineagecompletnesscontaminationsizecontigsN50longest_contigGCcoding_densitytaxonomyCoverageMetaTOR_8_1o__Clostridiales68.292.4614316121511612929162026.3687.97k__Bacteria;p__Firmicutes;c__Clostridia;o__Clostridiales146.46719755483332MetaTOR_8_2o__Clostridiales58.422.011396934584129017468228.8983.70k__Bacteria;p__Firmicutes;c__Clostridia;o__Clostridiales22.252416224710686MetaTOR_8_3o__Clostridiales49.370.94142082182330958996430.2983.24k__Bacteria;p__Firmicutes;c__Clostridia;o__Clostridiales;f__Peptostreptococcaceae_3;g__Clostridium_344.27369196532141ReferencesMetagenomic chromosome conformation capture (meta3C) unveils the diversity of chromosome organization in microorganisms, Martial Marbouty, Axel Cournac, Jean-François Flot, Hervé Marie-Nelly, Julien Mozziconacci, and Romain Koszul, eLife, 2014Meta3C analysis of a mouse gut microbiome, Martial Marbouty, Lyam Baudry, Axel Cournac, Romain Koszul, 2015Scaffolding bacterial genomes and probing host-virus interactions in gut microbiome by proximity ligation (chromosome capture) assay, Martial Marbouty, Lyam Baudry, Axel Cournac, and Romain Koszul, Science Advances, 2017ContactAuthorsamaury.bignaud@pasteur.frlyam.baudry@pasteur.frthfoutel@pasteur.frmartial.marbouty@pasteur.frromain.koszul@pasteur.frResearch labSpatial Regulation of Genomes(Institut Pasteur, Paris)