Added parameter float_precision to CSV parser
State: closed

closes #8002.


Labels:
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations
name: IO CSV
description: read_csv, to_csv

Comments:
- needs a release note / example
- pls update doc-string (in parser.pyx), and in io.rst
- this is ONLY for the c-parser yes? need to have tests that raise as an invalid option if the engine is not the c-engine (note that you can specifiy different tests for different engines, or have it run on all depending on how you call it)
- perf tests, need to add a benchmark for this (vb_suite/parser.py), and do a perf check on existing (`float_precision=None`) to see if it has changed

@jreback - Does this look okay? I added new benchmarks for the different floating-point converters, and the ordinary method (`float_precision=None`) doesn't seem to have changed:

```
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
read_csv_infer_datetime_format_ymd           |   1.2757 |   1.4106 |   0.9043 |
read_csv_vb                                  |  13.4887 |  13.9753 |   0.9652 |
read_csv_comment2                            |  11.1427 |  11.5426 |   0.9653 |
read_csv_infer_datetime_format_iso8601       |   1.1740 |   1.2066 |   0.9729 |
read_csv_thou_vb                             |  11.7860 |  12.1030 |   0.9738 |
read_csv_standard                            |   7.9970 |   8.1897 |   0.9765 |
frame_to_csv2                                |  65.0780 |  65.2781 |   0.9969 |
frame_to_csv                                 |  78.9930 |  78.8483 |   1.0018 |
read_csv_infer_datetime_format_custom        |   5.6247 |   5.5720 |   1.0095 |
packers_read_csv                             | 115.8530 | 114.4253 |   1.0125 |
frame_to_csv_date_formatting                 |   6.4060 |   6.2940 |   1.0178 |
packers_write_csv                            | 708.9777 | 696.3497 |   1.0181 |
frame_to_csv_mixed                           | 134.2530 | 130.5247 |   1.0286 |
write_csv_standard                           |  24.8680 |  23.9093 |   1.0401 |
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
```

pls rebase/squash and a couple of minor comments

Okay, rebased and squashed.

Looks like there's a Travis failure with Python 2.6, I'll look into it.

@amras1  status?

Hm.....right now the way I changed the tests to fix the error is basically a cop out. Apparently floating-point numbers are read as strings in Python 2.6, though I have no idea way. It's been a pain trying to test this out locally with Python 2.6, but hopefully I can figure this out if necessary...

can you rebase

Rebased

ok, this looked good.

@jorisvandenbossche ?

@amras1 pls rebase again

@jreback - rebased

looks good, ping when green

@jreback - Green

@amras1 thanks! nice feature!

pls take a look at the built docs (travis build them so pretty soon, here), and if necessary issue corrections. http://pandas-docs.github.io/pandas-docs-travis/ 

@jreback - No problem!

There's a reflink error in the docs, should I open another PR with the same branch?

just post the new I am making some edits anyhow

https://github.com/amras1/pandas/commit/c5b8c24221f03137bd3659498aca259e37ca0c51

@amras1 I have a question about the docs. In the section in the io docs (http://pandas-docs.github.io/pandas-docs-travis/io.html#specifying-method-for-floating-point-conversion), you use `val = '0.3066101993807095471566981359501369297504425048828125'` in the example. But, to me, such a long string gives the impression that you _can_ have such a high precision. Which is not the case I think? (or I must be completely missing the point here) Why not provinding a string that gives the precision that is possible? That would still show the point?

Furthermore, if I run the example, I get the same result for `float_precision=None` as for `float_precision='round_trip'` (so absolute difference of 0), which is different from the docs.

@jorisvandenbossche - Maybe I should clarify in the docs, but basically the whole point of having more accurate `float_precision` methods is to deal with strings _beyond_ the precision of floats. Well, actually, `float_precision=None` sometimes is incorrect even for strings at or below float precision (about 16 significant digits IIRC), but while `float_precision="high"` guarantees accuracy for these values, it will not necessarily return the closest approximation to a string beyond the float precision level.

Very interesting that you see a difference from the docs...I ran it myself and found the same thing as the docs. Can you post your output?

Linux, 32bit, python 2.7.8:

```
In [34]: pd.__version__ 
Out[34]: '0.14.1-474-g744daa7'

In [35]: val = '0.3066101993807095471566981359501369297504425048828125'

In [36]: data = 'a,b,c\n1,2,{0}'.format(val)

In [37]: abs(pd.read_csv(StringIO(data), engine='c', float_precision=None)['c'][0] - float(val))
Out[37]: 0.0

In [38]: abs(pd.read_csv(StringIO(data), engine='c', float_precision='high')['c'][0] - float(val))
Out[38]: 5.5511151231257827e-17

In [39]: abs(pd.read_csv(StringIO(data), engine='c', float_precision='round_trip')['c'][0] - float(val))
Out[39]: 0.0

```

I get exactly the same with `val = '0.30661019938070955'`. 

The name 'round_trip' suggests to me that if you write the dataframe back to a csv, you have the same output, which is of course not the case: 

```
In [41]: df = pd.read_csv(StringIO(data), engine='c', float_precision='round_trip')

In [42]: df.to_csv()
Out[42]: ',a,b,c\n0,1,2,0.30661019938070955\n'
```

Therefore, I did found using such a long string misleading, but maybe its is the name 'round trip' that is misleading?

Hm, that is really puzzling...I'm on a 64 bit machine, but I can't see why that would make a difference since `double` should be guaranteed to be 8 bytes. I'll look into that and see why `xstrtod()` would be more accurate here...

Ah, maybe we should change the name to something else? As you noticed the output isn't the same, but the point is to guarantee that the floating-point value will be the same. That is, the following should always occur:

```
In [22]: roundtrip = pd.read_csv(StringIO(val), engine='c', float_precision='round_trip', header=None)[0][0]

In [23]: roundtrip == float(val)
Out[23]: True
```

It depends how you interprete the round-trip, as read-write, or as write-read. And that last on does hold, so maybe the name is not that bad. But it could maybe somewhat clarified.

Do you think it makes more sense to change to change the parameter name or to just add a clarifying note in the docs? Either is fine by me.