freeman-lab/thunder — pull request #30 by industrial-sloth (2014-09-29 23:09:39+00:00)
Title: Custom rdds: s3 reader

Moving towards a unified i/o architecture. This PR adds a "readers" module, which currently provides an abstraction layer over access to the local filesystem and to s3 via boto.

Other changes:
* disable a failing test in test_series - not immediately clear what the cause is, but should get fixed
* expose a couple options from the spark ec2 script in the thunder ec2 script
* start moving towards more standardized handling of data paths in the series module
* add subsample and exportAsPngs methods in Images