kcarnold/joblib — comment on commit ba6dcb3609975e35a78657fdc5a6308baff3b0de
File:  line 0
By kcarnold:

Unfortunately, the old behavior is actually incorrect for matrices and other subclasses:

```
In [1]: import joblib.numpy_pickle # I reverted to origin/0.5.X here.
In [2]: import numpy as np
In [3]: m=np.matrix(np.zeros(5))
In [4]: joblib.numpy_pickle.dump(m, 'test-matrix')
Out[4]: ['test-matrix', 'test-matrix_01.npy']
In [5]: joblib.numpy_pickle.load('test-matrix')
Out[5]: array([[ 0., 0., 0., 0., 0.]])
```

This isn't your fault; it's a numpy bug, since the .npy file format doesn't save information about what kind of object held the array data. You can verify that by using np.save / np.load directly. Pickling does round-trip the data type correctly here:

```
In [6]: import cPickle as pickle
In [7]: pickle.dump(m, open('test-mat.pkl','wb'), -1)
In [8]: pickle.load(open('test-mat.pkl'))
Out[8]: matrix([[ 0., 0., 0., 0., 0.]])
```

But since `np.matrix`'s `__reduce_ex__` already dumps the data as a string, joblib wouldn't have a chance to special-case it. The best of both worlds would result if `np.matrix.__reduce_ex__` were to return `(np.matrix, (np.asarry(self),))` or the like. Then the optimized special case would kick in for storing the actual array data, and the pickle layer would take care of wrapping it with the right data type.

This all feels like a design deficiency in NumPy; after we understand the issue more, we should report bugs there. However, the deficiency may be intentional; looking at https://github.com/numpy/numpy/blob/master/doc/neps/npy-format.txt#L144, the authors seem to have held that fully accurate round-tripping was "out of scope." So for people who care about accurate round-tripping, joblib may become the de facto standard.