BradnerLab/pipeline — pull request #22 by jdimatteo (2014-06-28 20:30:29+00:00)
Title: Long file names

Resolves #19 .

This turned out to be a huge change to fix. I removed the arbitrary limit of 64 characters in a bam file name. The changes should also result in slightly faster execution and less memory usage. The change is using file integer keys in the data rows, and having the file names stored in just one table -- this was actually a change I wanted to make a while ago, because it is inefficient using strings in every data row in terms of both disk space and in terms of speed for table queries/indexing.

The hdf5 table representation changed, and the resulting .tab file format changed as well. Charles/@bradnerComputation : can you please help me evaluate the impact of the hdf5 table schema changes? Instead of the file name being included as a column in the bin_counts, region_counts, and normalized_counts, there is now a column "file_key" which is an integer that indexes into the file_names table. If nobody is using the counts.h5 file directly, I could update the .tab files generated with the -f/--flatten option to show the file_name like they used to and nobody will notice the change -- would that be helpful? Feel free to call me whenever if you'd prefer to discuss over the phone.

I added a bunch of automated tests, but there are a lot of changes here so some regression testing would be helpful.