Package: instagraal
instaGRAALLarge genome reassembly based on Hi-C data (continuation and partial rewrite ofGRAAL) and post-scaffolding polishing libraries.This work is under continuous development/improvement - seeGRAALfor information about the basic principles.InstallationInstall from PyPI:sudopip3install-Uinstagraalor, if you want to get the very latest version:sudopip3install-egit+https://github.com/koszullab/instagraal.git@master#egg=instagraalThis should automatically handle most dependencies.RequirementsThe scaffolder and polishing libraries are written in Python 3 and CUDA. As such, an NVIDIA graphics card is required for the scaffolder to run. The Python 2 version is available at thepython2branch of this repository, but be aware that development will mainly focus on the Python 3 version. The software has been tested for Ubuntu 17.04 and later, and most dependencies can be downloaded with its package manager (or Python'spip).External librariesYou will need to download and install theNVIDIA CUDA toolkit. Manual installation is recommended - installingnvidia-cuda-toolkitfrom Ubuntu's package manager has been known to cause glitches. It is fairly straightforward on OS X thanks to the installation wizard. Here is how to quickly do it on Ubuntu 18.04:wgethttps://developer.nvidia.com/compute/cuda/10.0/Prod/local_installers/cuda_10.0.130_410.48_linuxchmod+xcuda_10.0.130_410.48_linuxsudo./cuda_10.0.130_410.48_linuxNote to Ubuntu users: Be aware that the installation script will fail if it isn't run as root, or if a graphical instance (e.g. X) is running as well. You may need to temporarily shut it down, for instance by switching to tty1 and running the following (prior to the installation script):sudoservicelightdmstop(Replacelightdmwithmdm,gdmor whichever login manager is present on your machine if that fails; if all else fails as well, you may have to run something likesudo pkill Xorginstead.)Note to OS X users: There is currentlyno CUDA supporton Mojave (10.14) and it is unclear when it is going to be added, if it is to be added at all. This means instaGRAAL (or indeed any CUDA-based application) willnotwork on Mojave. If you wish to run it on OS X, the only solution for now is to downgrade to High Sierra (10.13).Recommended librariesBecause some Python dependencies (such aspyopenglorh5py) require to be built against specific files, it is recommended that you install the following packages if you encounter errors.OpenGL librarieslibglu1-mesalibxi-devlibxmu-devlibglu1-mesa-devfreeglut3-devHDF5 serialization libraryhdf5-tools(hdf5for OS X in brew)Boost librarieslibboost-all-dev(boostandboost-pythonfor OS X in brew)Python dependenciesPython package requirements should be handled automatically bypip, but should you
wish to install them manually, these are:numpyscipymatplotlibcodepyh5pypyopengldocoptbiopythonThey can also be handily installed using the supplied requirements file in the repo:pip3 install -Ur requirements.txtYou will also need to buildpycudawith OpenGL support anddisableits use of custom Boost libraries. Installing it directly from PyPI will cause errors at runtime. Here is how to do it manually with Git on Ubuntu or OS X:gitclone--recurse-submoduleshttps://github.com/inducer/pycuda.gitcdpycudapython3configure.py--cuda-enable-gl--no-use-shipped-boostsudopython3setup.pyinstallYou may run (as root)instagraal-setup, an all-in-one script to handle all the above dependencies on Ubuntu 17+.ContainerThere is experimental Docker support for instaGRAAL. You may fetch the corresponding image by running the following:dockerpullkoszullab/instagraalHow to useUnlike GRAAL, this is meant to be run from the command line.Usageinstagraal <hic_folder> <reference.fa> [<output_folder>]
           [--level=4] [--cycles=100] [--coverage-std=1]
           [--neighborhood=5] [--device=0] [--circular] [--bomb]
           [--save-matrix] [--pyramid-only] [--save-pickle] [--simple]
           [--quiet] [--debug]Options-h, --help              Display this help message.
--version               Display the program's current version.
-l 4, --level 4         Level (resolution) of the contact map.
                        Increasing level by one means a threefold smaller
                        resolution but also a threefold faster computation
                        time. [default: 4]
-n 100, --cycles 100    Number of iterations to perform for each bin.
                        (row/column of the contact map). A high number of
                        cycles has diminishing returns but there is a
                        necessary minimum for assembly convergence.
                        [default: 100]
-c 1, --coverage-std 1  Number of standard deviations below the mean.
                        coverage, below which fragments should be filtered
                        out prior to binning. [default: 1]
-N 5, --neighborhood 5  Number of neighbors to sample for potential
                        mutations for each bin. [default: 5]
--device 0              If multiple graphic cards are available, select
                        a specific device (numbered from 0). [default: 0]
-C, --circular          Indicates genome is circular. [default: False]
-b, --bomb              Explode the genome prior to scaffolding.
                        [default: False]
--pyramid-only          Only build multi-resolution contact maps (pyramids)
                        and don't do any scaffolding. [default: False]
--save-pickle           Dump all info from the instaGRAAL run into a
                        pickle. Primarily for development purposes, but
                        also for advanced post hoc introspection.
                        [default: False]
--save-matrix           Saves a preview of the contact map after each
                        cycle, in csv format. [default: False]
--simple                Only perform operations at the edge of the contigs.
                        [default: False]
--quiet                 Only display warnings and errors as outputs.
                        [default: False]
--debug                 Display debug information. For development purposes
                        only. Mutually exclusive with --quiet, and will
                        override it. [default: False]Input datasetsFormat specificationThe above<hic_folder>passed as an argument to instaGRAAL needs three files:A file namedabs_fragments_contacts_weighted.txt, containing the (sparse) Hi-C map itself. The first line must beid_frag_a    id_frag_b    n_contact. All subsequent lines must represent the map's contacts in coordinate format (id_frag_abeing the row indices,id_frag_bbeing the column indices,n_contactbeing the number of contacts between each locus or index pair,e.g.if 5 contacts are found between fragments #2 and #3, there should be a line reading2 3 5in the file).n_contactmust be an integer. The list should be sorted according toid_frag_afirst, thenid_frag_b. Fragment ids start at 0.A file namedfragments_list.txtcontaining information related to each fragment of the genome. The first line must beid    chrom    start_pos    end_pos    size    gc_content, and subsequent lines (representing the fragments themselves) should follow that template. The fields should be self-explanatory; notably,chromcan be any string representing the chromosome's name to which the fragment at a given line belongs, and fragment ids should start over at 1 when the chromosome name changes. Aside from thechromfield and thegcfield which is currently unused in this version and can be filled with any value, all fields should be integers. Note thatstart_posstarts at 0.A file namedinfo_contigs.txtcontaining information related to each contig/scaffold/chromosome in the genome. The first line must becontig    length_kb    n_frags    cumul_length. Field names should be again self-explanatory; naturally the contig field must contain names that are consistent with those found infragments_list.txt. Alsolength_kbshould be an integer (rounded up or down if need be), andn_fragsandcumul_lengthare supposed to be consistent with each other in that the cumulated length (in fragments) of contig N should be equal to the sum of the fields found inn_fragsfor the N-1 preceding lines. Note thatcumul_lengthstarts at 0.All fields (including those in the files' headers) must be separated by tabs.Minimal working templates are provided in theexamplefolder.Matrix generationIf you want to generate instaGRAAL-compatible matrices from scratch (i.e. from reads and a reference genome, as opposed to existing Hi-C data in one of the numerous existing formats), you may do so withhicstuff, which acts as both a Python library and a pipeline. Agraphical interfaceis also available. Instructions, parameters and optional arguments are detailed in the repo's readme.OutputAfter the scaffolder is done running, whatever path you specified as output will contain atest_mcmc_Xdirectory, where X is the level (resolution) at which scaffolding was performed. This directory, in turn, will contain the following:genome.fasta: the scaffolded genome. Scaffolds will be ordered by increasing sizein fragments, which roughly (but not always) translates into increasing size in bp.info_frags.txt: a file that contains, for each newly formed scaffold, the original coordinates of every single bin in that scaffold, in the formatchromosome, id, orientation, start, end. Each bin has a unique ID that provides a convenient way of tracking consecutive stretches. Orientations are relative to one another, and when "-1" is supplied, it is understood that the reverse complement should be taken.Other files are mostly for developmental purposes and keep track of the evolution of various metrics and model parameters.PolishingLingering artifacts found in output genomes can be corrected by editing theinfo_frags.txtfile, either by hand or with a script. Look at options by running the following:instagraal-polish -hThe most common use case is to run all polishing procedures at once:instagraal-polish -m polishing -i info_frags.txt -f reference.fasta -o polished_assembly.faTroubleshootingLoading CUDA librariesIf you encounter the following error,despitehaving installed the NVIDIA CUDA Toolkit:ImportError: libcurand.so.9.2: cannot open shared object file: No such file or directoryit probably means the CUDA-related libraries haven't been properly added to your$PATHfor some reason. A quick solution is to simply add this at the end of your.bashrcor.bash_profile(replace the paths with wherever you installed the toolkit and change the version number accordingly):export PATH=/usr/local/cuda-9.2/bin${PATH:+:${PATH}}
export LD_LIBRARY_PATH=/usr/local/cuda-9.2/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}Remote runningIf you encounter the following error:freeglut (instagraal.py): failed to open display ''it most likely means you attempted to run an instaGRAAL instance remotely (e.g. over ssh) but didn't configure a proper$DISPLAYvariable. In order to avoid this, simply run the following beforehand:export DISPLAY=:0Note that this will disable the movie (it will play on the remote machine instead).However, instaGRAAL is based on OpenGL, which means there has to be an X server of some kind running on your target machine no matter what. While this allows for pretty movies and visualizations, it may prove problematic on an environment you don't have total control over,e.g.a server cluster. Currently, your best bet is asking the system administrator of the target machine to set up an X instance if they haven't already.Codepy toolchainIf you encounter an error like the following :File "/usr/local/lib/python3.6/dist-packages/codepy/toolchain.py", line 382, in _guess_toolchain_kwargs_from_python_config
object_suffix = '.' + make_vars['MODOBJS'].split()[0].split('.')[1]
IndexError: list index out of rangeYou need to install codepy directly from the git repository to have a more recent version than the one on PyPI. Run the following commands:sudopip3uninstallcodepygitclonehttps://github.com/inducer/codepy.gitcdcodepysudopython3setup.pyinstallYou will also need to update to gcc/g++ 8:sudoaptinstallgcc-8g++-8It should work directly afterwards.General tipsinstaGRAAL will attempt to detect already-built pyramids in hdf5 format, but if building was interrupted for some reason, and you re-run the pyramid building step, the hdf5 files will still be there, but corrupted. You will need to manually delete thepyramidsfolder and try again.If there is a mismatch between the version of CUDA you installed and the one instaGRAAL seems to rely on (e.g.you installed CUDA 10 but instaGRAAL complains that it can't findlibcurand.so.9.1), try reinstallingpycudaandinstagraalwith the--no-cache-diroption.You may experience issues if you handle dependencies with conda, such aspycudafailing to build because some header files that would be present when you installedlibboost-all-devaren't automatically recognized. If you don't want to manually mess with your$PATH, it's probably best to just deactivate conda altogether and install everything with your OS's normal package manger (andpip).DocumentationAs a Python package, instaGRAAL provides both a scaffolding and polishing library, as well as a convenient Hi-C matrix handling framework, and we've tried to expose much of the API behind these onreadthedocs. If you wish to know more about how the scaffolder works, see thereferences, especially thesupplementary methoddelving deeper into the details of the model.ReferencesPrincipleHigh-quality genome assembly using chromosomal contact data, Hervé Marie-Nelly, Martial Marbouty, Axel Cournac, Jean-François Flot, Gianni Liti, Dante Poggi Parodi, Sylvie Syan, Nancy Guillén, Antoine Margeot, Christophe Zimmer and Romain Koszul, Nature Communications, 2014A probabilistic approach for genome assembly from high-throughput chromosome conformation capture data, Hervé Marie-Nelly, 2013, PhD thesisUse casesProximity ligation scaffolding and comparison of two Trichoderma reesei strains genomes, Etienne Jourdier, Lyam Baudry, Dante Poggi-Parodi, Yoan Vicq, Romain Koszul, Antoine Margeot, Martial Marbouty, and Frédérique Bidard, Biotechnology for Biofuels, 2017Scaffolding bacterial genomes and probing host-virus interactions in gut microbiome by proximity ligation (chromosome capture) assay, Martial Marbouty, Lyam Baudry, Axel Cournac, and Romain Koszul, Science Advances, 2017Contactlyam.baudry@pasteur.frromain.koszul@pasteur.fr