joblib/joblib — pull request #115 by undercoveridiot (2014-02-01 01:51:02+00:00)
Title: Convert numpy_pickle zfile to use buffered read/write to support large objects

If you try to save large arrays using the current version of numpy_pickle you will get integer overflow errors because the object length will not fit in an integer. This change converts the zlib calls to use a buffer instead of calling compress with the full object. Tested to read/write 20GB arrays with near identical performance in Python 2.7 and Python 3.2.