scikit-learn/scikit-learn — comment on commit 1694a8f2d802c006d35c842d37de5d0610f9f57a
File:  line 0
By GaelVaroquaux:

----- Original message -----
> Before going further in that direction we should have a look at
> [PyTables](http://pytables.org) /
> [blosc](http://blosc.pytables.org/trac) to avoid reinventing the will.

Carray, that I mentionned earlier uses blosc internally. That said, I have found that for disk persistance lzo is probably a better alternative than blosc, as the need for speed ins't that critical.

> Optional support / integration for working with data managed by pytables
> level would be very nice.

I agree. My experiments with pytables showed that for big arrays, pytables + lzo is probably the best store. That said, pytables also has issues, and you need to massage a bit the data to make it effecient.

> This could be done either at the joblib or
> scikit-learn level. I don't know pytables enough to know which would
> make the most sense.

In my experience, it is better to do this at the joblib level, because some of the logic necessary to massage the data can be reused. This is planned, but I want to get the simple version working well first.