freeman-lab/thunder — pull request #31 by industrial-sloth (2014-10-01 19:46:56+00:00)
Title: Custom rdds: initial writers module

Biggest change here is the addition of a new "writers" module, to abstract over writing to various filesystems (currently the local fs and S3) in the same way that readers attempts to abstract away the filesystem being read from. Also includes a number of bug fixes.
Smaller changes:
* set kryo buffer size to 1GB by default in thunder ec2 script
* add option in Images exportAsPngs to first collect the pngs to the driver before writing. Should make it easier to get useful output onto a local filesystem.
* add FileNotFound error in readers for additional abstraction of error cases, currently implemented only for local reader (boto reader is a todo item)
* fix IndexError in block size calc when requested block size is larger than data set
* change Images saveAsBinarySeries filename output so that default sorting will group together all files on matching z plane
* fix reading of conf.json file in SeriesLoader fromBinary()