Package: hicstuff
hicstuffA lightweight library that generates and handles Hi-C contact maps in either cooler-compatible 2Dbedgraph orinstaGRAALformat. It is essentially a merge of theyahcppipeline, thehicstufflibrary and extra features illustrated in the3C tutorialand theDADE pipeline, all packaged together for extra convenience.The goal is to make generation and manipulation of Hi-C matrices as simple as possible and work for any organism.Table of contentsInstallationpipcondadockerUsageFull pipelineIndividual componentsLibraryConnecting the modulesFile formatsContributingInstallationpipTo install a stable version:pip3install-Uhicstuffor, for the latest development version:pip3install-egit+https://github.com/koszullab/hicstuff.git@master#egg=hicstuffbowtie2,bwaand/orminimap2as well assamtoolsare required by thepipelinecommand.You can install them via the conda package manager:condainstall-cbiocondabowtie2bwaminimap2samtoolsAlternatively, on ubuntu you can also install them along with additional dependencies through APT:apt-getinstallbowtie2bwaminimap2samtoolslibbz2-devliblzma-devCondahicstuff is available as a bioconda package. It can be installed, along with all dependencies using:condainstall-cbiocondahicstuffDockerA pre-built docker image is made available on quay.io viabiocontainersand can be ran using:dockerpullquay.io/biocontainers/hicstuff:<tag>UsageThe hicstuff command line interface is composed of multiple subcommands. You can always get a summary of all available commands by running:hicstuff--help

SimpleHi-Cpipelineforgeneratingandmanipulatingcontactmatrices.

usage:hicstuff[-hv]<command>[<args>...]options:-h,--helpshowsthehelp-v,--versionshowstheversion

Thesubcommandsare:convertConvertHi-Cdatabetweendifferentformats.digestDigestgenomeintoalistoffragments.cutsitePreprocessfastqfilesbydigestingreadsatreligationsite.distancelawAnalyseandplotdistancelaw.filterFiltersHi-Cpairstoexcludespuriousevents.iteralignIterativelyalignsreadstoareferencegenome.missviewPreviewmissingHi-Cbinsinbasedonthegenomeandreadlength.pipelineHi-Cpipelinetogeneratecontactmatrixfromfastqfiles.rebinBinthematrixandregeneratefilesaccordingly.subsampleBootstrapsubsamplingofcontactsfromaHi-Cmap.viewVisualizeaHi-Cmatrix.Full pipelineAll components of the pipeline can be run at once using thehicstuff pipelinecommand. This allows to generate a contact matrix from reads in a single command. By default, the output is in GRAAL compatible COO sparse matrix format, but it can be a 2D bedgraph or cool file instead using the--matfmtoption. More detailed documentation can be found on the readthedocs website:https://hicstuff.readthedocs.io/en/latest/index.htmlusage:
    pipeline [--aligner=bowtie2] [--centromeres=FILE] [--circular] [--distance-law]
             [--duplicates] [--enzyme=5000] [--filter] [--force] [--mapping=normal]
             [--matfmt=graal] [--no-cleanup] [--outdir=DIR] [--plot] [--prefix=PREFIX]
             [--quality-min=30] [--read-len=INT] [--remove-centromeres=0] [--size=0]
             [--start-stage=fastq] [--threads=1] [--tmpdir=DIR] --genome=FILE <input1> [<input2>]

arguments:
    input1:             Forward fastq file, if start_stage is "fastq", sam
                        file for aligned forward reads if start_stage is
                        "bam", or a .pairs file if start_stage is "pairs".
    input2:             Reverse fastq file, if start_stage is "fastq", sam
                        file for aligned reverse reads if start_stage is
                        "bam", or nothing if start_stage is "pairs".For example, to run the pipeline with minimap2 using 8 threads and generate a matrix in instagraal format in the directoryout:hicstuff pipeline -t 8 -a minimap2 -e DpnII -o out/ -g genome.fa reads_for.fq reads_rev.fqIf you have already aligned your reads, hicstuff pipeline can also take bam files as input. For example,
to generate a matrix in cool format with a fixed bin size of 5kb:# Note the bam files have to be name-sorted, this can be done using samtools
samtools sort -n aligned_for.bam -o namesorted_for.bam
samtools sort -n aligned_rev.bam -o namesorted_rev.bam
hicstuff pipeline -S bam -e 5000 -M cool -o out/ -g genome.fa namesorted_for.bam namesorted_rev.bamThe pipeline can also be run from python, using thehicstuff.pipelinesubmodule. For example, this would run the pipeline with bowtie2 (default) using cutsiste alignment and keep all intermediate files. For more examples using the API, see theAPI demofromhicstuffimportpipelineashpihpi.full_pipeline('genome.fa','end1.fq','end2.fq',no_cleanup=Truemapping='cutsite',out_dir='out',enzyme="DpnII")The general steps of the pipeline are as follows:Individual componentsFor more advanced usage, different scripts can be used independently on the command line to perform individual parts of the pipeline. This readme contains quick descriptions and example usages. To obtain detailed instructions on any subcommand, one can usehicstuff <subcommand> --help.Digestion of the readsGenerates new gzipped fastq files from original fastq. The function will cut the reads at their religation sites and creates new pairs of reads with the different fragments obtained after cutting at the digestion sites.usage:
    cutsite --forward=FILE --reverse=FILE  --prefix=STR --enzyme=STR
    [--mode=for_vs_rev] [--seed-size=20] [--threads=1]Iterative alignmentTruncate reads from a fastq file to 20 basepairs and iteratively extend and re-align the unmapped reads to optimize the proportion of uniquely aligned reads in a 3C library.usage:
    iteralign [--aligner=bowtie2] [--threads=1] [--min-len=20] [--read-len=INT]
              [--tempdir=DIR] --out-bam=FILE --genome=FILE <reads.fq>Digestion of the genomeDigests a fasta file into fragments based on a restriction enzyme or a
fixed chunk size. Generates two output files into the target directory
named "info_contigs.txt" and "fragments_list.txt"usage:
    digest [--plot] [--figdir=FILE] [--force] [--circular] [--size=0]
           [--outdir=DIR] --enzyme=ENZ <fasta>For example, to digest the yeast genome with MaeII and HinfI and show histogram of fragment lengths:hicstuff digest --plot --outdir output_dir --enzyme MaeII,HinfI Sc_ref.faFiltering of 3C eventsFilters spurious 3C events such as loops and uncuts from the library based on a minimum distance threshold automatically estimated from the library by default. Can also plot 3C library statistics. This module takes a pairs file with 9 columns as input (readID, chr1, pos1, chr2, pos2, strand1, strand2, frag1, frag2) and filters it.usage:
    filter [--interactive | --thresholds INT-INT] [--plot]
           [--figdir FILE] [--prefix STR] <input> <output>Viewing the contact mapVisualize a Hi-C matrix file as a heatmap of contact frequencies. Allows to
tune visualisation by binning and normalizing the matrix, and to save the
output image to disk. If no output is specified, the output is displayed.usage:
    view [--binning=1] [--despeckle] [--frags FILE] [--trim INT] [--n-mad 3.0] [--lines]
         [--normalize] [--min=0] [--max=99%] [--output=IMG] [--cmap=Reds] [--dpi=300]
         [--transform=STR] [--circular] [--region=STR] <contact_map> [<contact_map2>]

arguments:
    contact_map             Sparse contact matrix in bg2, cool or graal format
    contact_map2            Sparse contact matrix in bg2, cool or graal format,
                            if given, the log ratio of contact_map/contact_map2
                            will be shown.For example, to view a 1Mb region of chromosome 1 from a full genome Hi-C matrix rebinned at 10kb:hicstuffview--normalize--binning10kb--regionchr1:10,000,000-11,000,000--fragsfragments_list.txtcontact_map.tsvLibraryAll components of the hicstuff program can be used as python modules. See the documentation onreathedocs. The expected contact map format for the library is a simple CSV file, and the objects handled by the library are simplenumpyarrays. The various submodules of hicstuff contain various utilities.importhicstuff.cutsite# Functions to digest fastq librairiesimporthicstuff.digest# Functions to work with restriction fragmentsimporthicstuff.iteralign# Functions related to iterative alignmentimporthicstuff.hicstuff# Contains utilities to modify and operate on contact maps as numpy arraysimporthicstuff.filter# Functions for filtering 3C events by type (uncut, loop)importhicstuff.view# Utilities to visualise contact mapsimporthicstuff.io# Reading and writing hicstuff filesimporthicstuff.pipeline# Generation and processing of files to generate matrices.Connecting the modulesAll the steps described here are handled automatically when running thehicstuff pipeline. But if you want to connect the different modules manually, the intermediate input and output files can be processed using some python scripting.Aligning the readsYou can generate SAM files independently using your favorite read mapping software, use the command line utilityhicstuff iteralign, or use the helper functionalign_readsin the submodulehicstuff.pipeline. For example, to perform iterative alignment using bwa (instead of bowtie2 by default):Using the python function:fromhicstuffimportpipelineashpihpi.align_reads("end1.fastq","genome.fasta","end1.bam",iterative=True,aligner='bwa')Using the command line tool:hicstuffiteralign--alignerbwa--genomegenome.fasta--out-bamend1.bamend1.fastqExtracting contacts from the alignmentThe output fromhicstuff iteralignis a BAM file. In order to retrieve Hi-C pairs, you need to run iteralign separately on the two fastq files and process the resulting alignment files into a name-sorted BAM file as follows using thepipelinesubmodules of hicstuff.fromhicstuffimportpipelineashpiimportpysamasps# Sort alignments by read names and get into BAM formatps.sort("-n","-O","BAM","-o","end1.bam.sorted","end1.bam")ps.sort("-n","-O","BAM","-o","end2.bam.sorted","end2.bam")# Combine BAM fileshpi.bam2pairs("end1.sorted.bam","end2.sorted.bam","output.pairs","info_contigs.txt",min_qual=30)This will generate a "pairs" file containing all read pairs where both reads have been aligned with a mapping quality of at least 30.Attributing each read to a restriction fragmentTo build a a contact matrix, we need to attribute each read to a fragment in the genome. This is done under the hood by performing a binary search for each read position against the list of restriction sites in the genome.fromhicstuffimportdigestashcdfromBioimportSeqIO# Build a list of restriction sites for each chromosomerestrict_table={}forrecordinSeqIO.parse("genome.fasta","fasta"):# Get chromosome restriction tablerestrict_table[record.id]=hcd.get_restriction_table(record.seq,enzyme,circular=circular)# Add fragment index to pairs (readID, chr1, pos1, chr2,# pos2, strand1, strand2, frag1, frag2)hcd.attribute_fragments("output.pairs","output_indexed.pairs",restrict_table)Filtering pairsThe resulting pairs file can then be filtered, either in the command line using thehicstuff filtercommand, or in python using thehicstuff.filtersubmodule. Otherwise, the matrix can be built directly from the unfiltered pairs.Filtering on the command line:hicstufffilteroutput_indexed.pairsoutput_filtered.pairsFiltering in python:fromhicstuffimportfilterashcfuncut_thr,loop_thr=hcf.get_thresholds("output_indexed.pairs")hcf.filter_events("output_indexed.pairs","output_filtered.pairs",uncut_thr,loop_thr)Note that both the command and the python function have various options to generate figure or tweak the filtering thresholds. These options can be displayed usinghicstuff filter -hMatrix generationA Hi-C sparse contact matrix can then be generated using the python submodulehicstuff.pipeline. The matrix can be generated in GRAAL-compatible COO format, bedgraph2 or cool format.fromhicstuffimportpipelineashpin_frags=sum(1forlineinopen(fragments_list,"r"))-1hpi.pairs2matrix("output_filtered.pairs","abs_fragments_contacts_weighted.txt",'fragments_list.txt',mat_fmt="GRAAL")File formatspairs files: This format is used for all intermediate files in the pipeline and is also used byhicstuff filter. It is a tab-separated format holding informations about Hi-C pairs. It has anofficial specificationdefined by the 4D Nucleome data coordination and integration center.2D bedgraph: This is an optional output format ofhicstuff pipelinefor the sparse matrix. It has two fragment per line, and the number of times they are found together. It has the following fields:chr1, start1, end1, chr2, start2, end2, occurencesThose files can beloaded by coolerusingcooler load -f bg2 <chrom.sizes>:<binsize> in.bg2.gz out.coolwhere chrom.sizes is a tab delimited file with chromosome names and length on each line, and binsize is the size of bins in the matrix.GRAAL sparse matrix: This is a simple tab-separated file with 3 columns:frag1, frag2, contacts. The id columns correspond to the absolute id of the restriction fragments (0-indexed). The first row is a header containing the number of rows, number of columns and number of nonzero entries in the matrix. Example:564	564	6978
0	0	3
1	2	4
1	3	3fragments_list.txt: This tab separated file provides information about restriction fragments positions, size and GC content. Note the coordinates are 0 point basepairs, unlike the pairs format, which has 1 point basepairs. Example:id: 1 based restriction fragment index within chromosome.chrom: Chromosome identifier. Order should be the same as in info_contigs.txt or pairs files.start_pos: 0-based start of fragment, in base pairs.end_pos: 0-based end of fragment, in base pairs.size: Size of fragment, in base pairs.gc_content: Proportion of G and C nucleotide in the fragment.id	chrom	start_pos	end_pos	size	gc_content
1	seq1	0	21	21	0.5238095238095238
2	seq1	21	80	59	0.576271186440678
3	seq1	80	328	248	0.5201612903225806info_contigs.txt: This tab separated file gives information on contigs, such as number of restriction fragments and size. Example:contig: Chromosome identified. Order should be the same in pairs files or fragments_list.txt.length: Chromosome length, in base pairs.n_frags: Number of restriction fragments in chromosome.cumul_length: Cumulative length of previous chromosome, in base pairs.contig	length	n_frags	cumul_length
seq1	60000	409	0
seq2	20000	155	409ContributingAll contributions are welcome, in the form of bug reports, suggestions, documentation or pull requests.
We use thenumpy standardfor docstrings when documenting functions.The code formatting standard we use isblack, with --line-length=79 to follow PEP8 recommendations. We usepytestwith thepytest-doctestandpytest-pylintplugins as our testing framework. Ideally, new functions should have associated unit tests, placed in thetestsfolder.
To test the code, you can run:pytest--doctest-modules--pylint--pylint-error-types=EF--pylint-rcfile=.pylintrchicstufftestsCitationPlease cite hicstuff using the official DOI as follows:Cyril Matthey-Doret, Lyam Baudry, Amaury Bignaud, Axel Cournac, Remi-Montagne, Nadège Guiglielmoni, Théo Foutel Rodier and Vittore F. Scolari. 2020. hicstuff: Simple library/pipeline to generate and handle Hi-C data . Zenodo.http://doi.org/10.5281/zenodo.4066363Bibtex entry:@software{cyril_matthey_doret_2020_4066351,
  author       = {Cyril Matthey-Doret and
                  Lyam Baudry and
                  Amaury Bignaud and
                  Axel Cournac and
                  Remi-Montagne and
                  Nadège Guiglielmoni and
                  Théo Foutel-Rodier and
                  Vittore F. Scolari},
  title        = {hicstuff: Simple library/pipeline to generate and handle Hi-C data },
  month        = oct,
  year         = 2020,
  publisher    = {Zenodo},
  version      = {v2.3.1},
  doi          = {10.5281/zenodo.4066351},
  url          = {http://doi.org/10.5281/zenodo.4066363}
}