Package: pique
The pique package is a high efficiency peak finder for ChIP-seqexperiments that yield high coverage allignments to the referencegenome. It was developed for studying gene expression in Halobacteriumsalinarum sp. NRC1, sequenced using barcoded 40bp Illumina reads aspart of an ongoing project in Marc Facciotti's lab at UC Davis.AUTHOR & CONTACT INFORMATION----------------------------Pique was written by Russell Neches, a graduate student in JonathanEisen's laboratory.Russell's Blog    : http://vort.org/Russell's Twitter : @rynechesRussell's Email   : ryneches at ucdavis dot eduLab homepage      : http://phylogenomics.wordpress.com/Project Page      : http://ryneches.github.com/pique/INSTALLING----------General Instructions : You can do the usual python thing. We recommend usingvirtualenv, because it's nice.$ cd ~$ virtualenv --system-site-packages opt$ PATH=~/opt/bin:$PATH$ pip install numpy         # prerequisite$ pip install scipy         # prerequisite$ pip install cython        # prerequisite$ pip install pysam         # prerequisite$ pip install argcomplete   # suggested package$ pip install matplotlib    # suggested package$ pip install ipython       # suggested package$ pip install pique         # install Pique$ pique testDebian and Ubuntu : We recommend using apt-get for most of theprerequisites.$ sudo apt-get install  python-numpy        \python-scipy        \python-matplotlib   \python-virtualenv   \cython              \ipython$ cd ~$ virtualenv --system-site-packages opt$ PATH=~/opt/bin:$PATH$ pip install pysam$ pip install argcomplete$ pip install pique$ pique testIn your .bashrc file, we suggest adding ~/opt/bin to your PATH :$ cd ~$ cat >> .bashrcPATH=~/opt/bin:$PATHexport PATH[ctrl-D]MacOS : Unfortunately, MacOS doesn't come with virtualenv or any sortof useful global package management system. Here's how to fix that :$ sudo easy_install pip     # install pip globallyMake sure pip works :$ pipUsage: pip COMMAND [OPTIONS]pip: error: You must give a command (use "pip help" to see alist of commands)Now, install virtualenv :$ sudo pip install virtualenvMake sure that works :$ virtualenvYou must provide a DEST_DIR...Now, scroll back up and follow the General Instructions.Using argcomplete : If you installed argcomplete, you can enable tabcompletion for pique commands by placing this into your .bashrc :eval "$(register-python-argcomplete pique)"TESTING DATA------------A testing data set is [available onFigshare](http://figshare.com/articles/A_ChIP-seq_test_dataset/98106).It was made from a his-tagged tfbD pulldown in _Halobacteriumsalinarum_, sequenced on the Illumina GAII. The reads were mappedusing [bowtie](http://bowtie-bio.sourceforge.net/) to a cut-downchromosome and one plasmid. Unmapped reads were discarded. The datasetcan be cited separately as :A ChIP-seq test dataset. Russell Neches, Marc Facciotti, ElizabethG. Wilbanks, Phillip M. Seitzer. **figshare**.Retrieved 06:12, Nov 27, 2012 (GMT)http://dx.doi.org/10.6084/m9.figshare.98106If you want to try out the test data, simply run :$ pique testThe test data will automatically be downloaded to `/tmp/pique/`, andthe resulting output data will be written to your current workingdirectory.TESTS-----Tests are designed to be used with nose.http://somethingaboutorange.com/mrl/projects/nose/The tests (stupidly) depend on hard-coded paths to find the test datafiles, so you have to run nosetests from the main module directory(i.e., the one that has setup.py in it).$ cd pique$ nosetestsDEPENDENCIES------------This package requires the following packages :scipynumpypysamcythonSuggested :matplotlibipythonsamtoolsargcompleteGaggle Genome BrowserUSAGE-----This software can be used either as a stand-alone application, or as alibrary accessed by your own software.To use pique as a stand-alone application, there is a self-documentingcommand line interface :$ pique --helpusage: pique [-h] {batch,tk,test,bam2wav,mapmaker} ...Pique, a ChIP-seq analytical tool.optional arguments:-h, --help            show this help message and exitcommands:{batch,tk,test,bam2wav,mapmaker}batch               command-line mode for peak detectiontk                  GUI mode for peak detectiontest                testing mode, downloads online test databam2wav             BAM to WAV conversion utilitymapmaker            genome analysis region mapping utilityThe application has four modes, batch, tk, test and mapmaker. To accessthe GUI interface, run :$ pique tkThe application window will appear. Enter a name for your project inthe 'Project name' field; this will be the name prefix for the outputfiles. Choose files for the IP and BG (background) tracks; theseshould be indexed BAM files, created by mapping your IP and controlreads to the reference sequence. Many different tools can be used forthis, such as bowtie, bwa and SHRiMP.The IP and background BAM files must contain identical contig names.If desired, choose a genome map file ('Map (optional)'). See below forhow to create one.Enter the dominant fragment length of the sequencing library, and theread length used for the sequencing run. If you have performed qualitytrimming, you may wish to use the average or median read length aftertrimming for the read length.Then, click 'Run.' Your BAM files will be loaded and processed, andoutput files will be written in the directory from which pique.py wasinvoked. Run time for a bacterial genome depends on the number ofmapped reads and the length of the genome. For our projects, we havefound that a run takes about one to five minutes.BATCH MODE----------If you have many experiments to process, it may be more convenient torun pique from a script.$ pique batch -husage: pique batch [-h] -name NAME -ip IPFILE -bg BGFILE -map MAPFILE[-a ALPHA] [-l L_THRESH] [-p]optional arguments:-h, --help    show this help message and exit-name NAME    project name-ip IPFILE    BAM file for IP data-bg BGFILE    BAM file for control data-map MAPFILE  genome map file-a ALPHA      read length-l L_THRESH   expected binding footprint-p            make a pickel fileThe arguments are essentially the same as those found in the GUI,except for the option of making a pickle file. If this option isenabled, after completing the anaysis, pique will dump the entireanalysis workbench object into a pickle file. This can be very usefulfor debugging and data post-processing in Python. For example, thepickle file can be used to load the analysis workbench into aninteractive ipython session and used to generate figures.http://docs.python.org/2/library/pickle.htmlMAPMAKER MODE-------------$ pique mapmaker -husage: pique mapmaker [-h] -name NAME -bamfile BAMFILE [-windowWINDOW][-stride STRIDE] -highest HIGHEST -lowest LOWEST[-bins BINS]optional arguments:-h, --help        show this help message and exit-name NAME        project name-bamfile BAMFILE  BAM file of aligned data-window WINDOW    sliding window size-stride STRIDE    stride length-highest HIGHEST  highest histogram bin-lowest LOWEST    lowest histogram bin-bins BINS        number of histogram binsBAM2WAV MODE------------As an alternative to visualization, pique contains a utility forconverting ChIP-seq data into audio data. The background alignmentis subtracted from the IP alignment, and a separate WAV file iswritten for each contig. Coverage information for reads that mapin the forward and reverse orientations is represented on the leftand right audio channels, respectively.$ pique bam2wav -husage: pique bam2wav [-h] -name NAME -ip IPFILE -bg BGFILEoptional arguments:-h, --help  show this help message and exit-name NAME  project name-ip IPFILE  BAM file for IP data-bg BGFILE  BAM file for control dataGENOME MAP FILE---------------The Genome Map File is a GFF-formatted file describing three kinds offeatures; analysis regions, masking regions, and normalizationregions. Analysis and masking region annotations are optional;however, each analysis region MUST have at least one normalizationregion. (Note : If you do not describe any analysis regions, one willautomatically be created spanning each contig. So, if you do notdescribe any analysis regions for a contig, you must still describe atleast one normalization region for that contig.)Analysis regions should use the feature name 'analysis_region' in thefile, and describe sub-regions of the contig that you wish to processindependently. For example, if there are inconsistencies in coveragelevel due to gene dosage variations with respect to the reference,processing the dosage regions separately can improve peak detection.Masking regions should use the feature name 'mask' in the file, anddescribe sub-regions of the analysis regions that you wish to deletefrom the analysis. (Note: Masking regions *must* fall completelywithin an analysis region.) This is useful for hiding coverageaberrations due to repetitive regions, such as IS elements.By default, an analysis region describing the whole contig is createdfor each contig. When an analysis region is loaded by the user via themapping file, the default analysis region for that contig is deleted.Normalization regions should use the name 'norm_region' in the mapfile, and describe regions (within analysis regions) in which thereare thought to be no binding events. These regions will be used tocalibrate the sequencing coverage between the IP track and the controltrack. If the user chooses poor normalization regions, Pique willstill function, but will produce incorrect normalization factors andmay set the peak detection cutoff too stringently.Features described in the genome map file must use the same contignames as the IP and background BAM files. To see them, run :$ samtools view -H BAMfile.bamMost columns in the GFF file are unused, and may be left empty (aslong as the correct number of tabs are present). The required fieldsare :0   contig name2   'analysis_region', 'mask' or 'norm_region'3   start coordinate4   end coordinateAn example genome map file is provided (data/map.gff).OUTPUT FILES------------<name>.IP.track  : A GGB-formatted track file of the IP data<name>.BG.track  : A GGB-formatted track file of the BG data<name>.bookmark  : A list of GGB bookmarks for each peak<name>.qp        : GGB-formatted quantitative positional dataof binding coordinates<name>.gff       : A GFF file of each peak<name>.peak.tsv  : Peak statistical quantities in TSV formatThe GFF output file includes the following columns :0   contig name1   software version2   feature description3   start coordinate4   end coordinate5   enrichment ratio relative to backgroundThe TSV output includes the following columns :0   contig1   start2   stop3   binds_at            # binding coordinate4   enrichment_ratio5   average_contig_norm # mean of all norms6   std_contig_norm     # standard deviation of all norms7..                     # all subsequent colunms are the normsThe bookmark file contains some additional annotations in addition tothe required GGB bookmark information :binds_at            estimated binding coordinateenrichment_ratio    enrichment ratio relative to backgroundnorm_0, norm_1...   the normalization factors calculated from thenormilization regionsCAVEATS-------Binding coordinates : The peak calling step (pique.py) will attempt tocalculate a binding coordinate for the putative peaks it finds. Themethod used is extremely naive, but we find that it generally comeswithin about 5bp of the expected binding site.Double peaks : Some enriched regions contain two binding sites, butthe algorithm is unable to de-convolute them. We have tried runningthese regions through several other peak callers with little success.We have included a script for identifying sites that are likely to bedouble peaks to assist with curation.DOWNSTREAM ANALYSIS-------------------Pique does not attempt to filter peaks that are statisticallyinsignificant. We have found that this part of the analysis is ratherspecific to the data and to the experiment.To address this, Pique is designed to achieve a low false-negativerate at the cost of a potentially high false-positive rate (it appearsthat the false-positive rate is in practice rather low compared toother software, but we have not implemented any filtering to preventthem from happening). The idea is that it is painstaking work to finda putative peak, it is less difficult to discard unsatisfactory peaksonce they have been found.  Thus, some kind of post-filtering isnecessary. Pique provides the user with output that can be used tosupport a variety of such statistical tests.Some recommended filtering might include :[1] Eliminate peaks that are significantly narrower than the sizerange of the sequencing library, and perhaps other peaks thatcluster with them.[2] Eliminate peaks with a normalized enrichment ratio below 1.0,or peaks that cluster with them.[3] Eliminate peaks that have predicted binding sites that arevery skewed from the center of the enriched region.Depending on how many peaks are recovered, the user may wish to tryone or all of these, perhaps with clustering to avoid arbitrarycutoffs.However, we do not recommend trying to generate a "perfect" peak listby statistical analysis alone.MANUAL CURATION---------------Microbial genomes are fairly small. It is probably not worth theeffort to undertake a very complicated statistical effort to eliminateall false positives. We recommend applying a few basic statisticaltests, and then loading tracks, quantitative positionals and bookmarksinto the Gaggle Genome Browser.Once the data is loaded into the browser, view the bookmark list. Whenyou select a bookmark, the browser will automatically scan to thatregion of the genome and highlight the putative peak region. If youthink the peak is a false positive, press the delete key.When you are finished, save a copy of the bookmark file. We find thatcuration of a microbial genome in this way takes about ten to twentyminutes.INSPIRATIONAL QUOTE-------------------I met men at every turn who owned from one thousand to thirty thousand"feet" in undeveloped silver mines, every single foot of which theybelieved would shortly be worth from fifty to a thousand dollars—andas often as any other way they were men who had not twenty-fivedollars in the world. Every man you met had his new mine to boast of,and his "specimens" ready; and if the opportunity offered, he wouldinfallibly back you into a corner and offer as a favor to you, not tohim, to part with just a few feet in the "Golden Age," or the "SarahJane," or some other unknown stack of croppings, for money enough toget a "square meal" with, as the phrase went. And you were never toreveal that he had made you the offer at such a ruinous price, for itwas only out of friendship for you that he was willing to make thesacrifice. Then he would fish a piece of rock out of his pocket, andafter looking mysteriously around as if he feared he might be waylaidand robbed if caught with such wealth in his possession, he would dabthe rock against his tongue, clap an eyeglass to it, and exclaim:"Look at that! Right there in that red dirt! See it? See the specks ofgold? And the streak of silver? That's from the Uncle Abe. There's ahundred thousand tons like that in sight! Right in sight, mind you!And when we get down on it and the ledge comes in solid, it will bethe richest thing in the world! Look at the assay! I don't want you tobelieve me—look at the assay!"Then he would get out a greasy sheet of paper which showed that theportion of rock assayed had given evidence of containing silver andgold in the proportion of so many hundreds or thousands of dollars tothe ton.I little knew, then, that the custom was to hunt out the richest pieceof rock and get it assayed! Very often, that piece, the size of afilbert, was the only fragment in a ton that had a particle of metalin it—and yet the assay made it pretend to represent the average valueof the ton of rubbish it came from!- Mark Twain, Roughing ItLEGAL STUFF-----------Copyright (c) 2012, Russell NechesAll rights reserved.Redistribution and use in source and binary forms, with or withoutmodification, are permitted provided that the following conditions aremet:* Redistributions of source code must retain the above copyrightnotice, this list of conditions and the following disclaimer.* Redistributions in binary form must reproduce the above copyrightnotice, this list of conditions and the following disclaimer inthe documentation and/or other materials provided with thedistribution.* Neither the name of the University of California, Davis nor thenames of its contributors may be used to endorse or promoteproducts derived from this software without specific priorwritten permission.THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOTLIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FORA PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHTHOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOTLIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANYTHEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USEOF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.