scikit-learn/scikit-learn — pull request #935 by ogrisel (2012-07-05 23:15:59+00:00)
Title: WIP: Svmlight memmaped loader

Hi all,

I am working on an incremental data loader for the svmlight format that reads chunks of a big file that is not expected to all fit in memory in smaller CSR matrix to be dumped as set of memmapable files in folders to be later reconcatenated into a single, large CSR memmaped matrix.

The goal being to be able to load big svmlight files (multiple tens of GB) into an efficient memmaped CSR in an out-of-core manner (possible using several workers in //).

The first step is to augment the existing parser to be able to load chunks of a svmlight using seeks to bytes offsets. However while testing it I am encountering a dtype precision issue (see the new failing tests):

Ouput on my box:

```
original: [-0.78023183 0.24841428 0.91939807 0.09992654 0.1206722 -0.88162848
-1.53088036 -1.30152471 1.63974345 0.03541357]
parsed: [-0.780232 0.248414 0.919398 0.099927 0.120672 -0.881628 -1.53088
-1.301525 1.639743 0.035414]
svmlight: 0
1.000000 8:-0.780232 20:0.248414 52:0.919398 70:0.099927 72:0.120672 103:-0.881628 109:-1.530880 122:-1.301525 149:1.639743 164:0.035414
```

Could @mblondel or @larsmans confirm that the loss of precision is not expected and a bug of our serializer?