FrancescAlted/python-blosc — pull request #16 by esc (2013-04-06 13:03:00+00:00)
Title: Feature/compress from address

At some point in the past Francesc and I were discussion the possibility of compressing a Numpy array by passing the data pointer to Blosc rather than by going via Pickle as it is currently done.

The problem at the time was that metadata needed for the reconstruction of the array could not be saved easily. Now, with the recent advances in Bloscpack to include a metadata section in the format, we have an easy way to store the metadata.

Hence we could now (at least in theory) do much faster compression of numpy arrays by pointing Blosc at the data pointer of the Numpy array. The reason is that we can bypass pickle and especially avoid the copy which it necessitates.

A quick-and-dirty benchmark can be found in:

https://gist.github.com/esc/5326039

And when running it on my old slow two core laptop (Intel(R) Core(TM)2 Duo CPU L7500 @ 1.60GHz) i get:

2.53254199028
1.03268194199

The first figure being the time (in seconds) it takes using the pickle approach on a ``numpy.arange(3e7)`` array. The second is the passing the pointer approach.

I realize this feature is probably still a bit experimental and it may take a few iterations on this pull-request to bring it into a good state. For example I am not sure if ``unsigned long`` is indeed the correct type for casting the ``void *`` too. Also I am not sure if using a non-exposed helper function is the correct way to avoid copy and pasting code.