BradnerLab/pipeline — pull request #5 by jdimatteo (2014-02-06 01:14:40+00:00)
Title: BamLiquidator: Added support for varying bin sizes and improved performance

Arbitrary bin sizes is now supported for hot spot analysis. Substantial performance improvements as well (e.g. 200x faster than prior version) by using HDF5 instead of mysql, and using multithreaded C++ instead of bash. The test plan is on #4 . I ran most of the test plan a couple weeks ago, and I'm going to re-run the test plan again comprehensively, but I was hoping that we could start the code review and I could start getting some feedback before I finished testing.

I started adding some documentation at https://github.com/BradnerLab/pipeline/wiki/bamliquidator . I'm still working on completing that, as well as putting together a release candidate that labs can use without having to install dev libraries.

Note that I tried to hide most of the bamliquidator stuff in bamliquidator_internal. Instead of running the make command in the pipeline folder, now it should be run in the bamliquidator_internal folder, and that adds the bamliquidator executable in the pipeline folder. The pipeline folder is getting pretty messy, but I think this helps a little.

Note that I put the license at the bottom of some of the files -- I find that it is less in the way there.

Here is a summary of the files changes:

deleted files:
* bam_liquidator_bin_count_tables.sql
* we are using HDF5 instead of mysql now, so it is no longer needed

renamed / replacement files
* bam_liquidator_bin_counter.sh -> bamliquidator_batch.py
* Main driver: counts the number of base pair reads in each bin of each chromosome in the bam file(s) in a given directory, and then normalize, plot, and summarize the counts in an output directory. Executes bamliquidator_batch to process the .bam files.
* bam_liquidator_bin_plot_by_cell_type.py -> bamliquidator_internal/normalize_plot_and_summarize.py
* normalize_plot_and_summarize.py is used by bamliquidator_batch.py and normally wouldn't be called directly
* creates the bokeh html plots, normalizes, and summarizes the hdf5 data generated by bamliquidator_batch.m.cpp
* bamliquidator.c -> bamliquidator.h, bamliquidator.cpp, bamliquidator.m.cpp
* bamliquidator.m.cpp contains the main function (hence the .m in the name), and just parses command line arguments and calls bamliquidator.cpp
* it was important to separate out the main from bamliquidator.cpp so that the liquidation logic could be used between two different executables (namely bamliquidator and bamliquidator_batch)
* bamliquidator.cpp contains the bulk of what was in bamliquidator.c
* the changes I made here fixed various memory leaks, which is important now that the liquidate function is called multiple times in a loop by bamliquidator_batch.m.cpp. I used C++ because using things like std::vector made it easier for me to fix the memory leaks -- my apologies if introducing C++ complicates things for anyone.
* bamliquidator.h is the public interface of bamliquidator.cpp, and contains javadoc style comments for the function signature
* makefile -> bamliquidator_internal/makefile

new files
* bamliquidator_batch.m.cpp
* calls liquidate in bamliquidator.cpp with multiple threads and writes the results out to HDF5
* bamliquidator_internal/__init__.py
* just needed to make the directory a python package, so normalize_plot_and_summarize.py can be imported into bamliquidator_batch.py