Repository: pandas-dev/pandas

Issue:
BUG: float conversion in read_csv is inaccurate for precise input
This is coming from #2566, which noted that the `xstrtod()` method used by `read_csv` doesn't agree with standard numpy float conversion. I see that the priority is speed of parsing over complete accuracy (within 0.5 units in the last place, or ULP), but there are two issues here that (at least to me) actually appear to be buggy:
1. Low-precision values (i.e. less than about 15 significant figures) should be guaranteed to be within 0.5 ULP of the correct result, or at least within 1 ULP. However, I have found cases in which `xstrtod()` is off by _more_ than 1 ULP, although these don't come up often.
2. High-precision values of course can't be guaranteed to be within 0.5 ULP without a costly correction loop as in the ordinary `strtod()`, but the error in conversion increases linearly as the number of supplied significant figures increases. With 30 significant figures, the error in conversion can potentially be over 7 ULP.

[Here](http://nbviewer.ipython.org/github/amras1/ascii-profiling/blob/master/fast_conversion_profile.ipynb) is an IPython notebook analyzing the accuracy of `xstrtod()`. I think there are two problems here: `xstrtod()` keeps reading digits after the 17th, none of which should matter for conversion, and the scaling step at the end produces a compounded error by repeatedly multiplying/dividing by powers of 10. I have a solution for AstroPy that seems to fix these issues, so I can open a PR if it's agreed that `xstrtod()` should be changed.



Discussion:
how about an option for this (default is turned off), may `float_precision=None|high` ?

That could work--our decision in AstroPy was also to use a parameter named `use_fast_converter`, although we have it defaulting to `False`. Then should the updated `xstrtod()` be used if `float_precision='high'`, or just `strtod()` (which is guaranteed to be within 0.5 ULP)?

I would like to leave the existing code in-place. No objection to adding new options like this though. I don't think the intermediate (using `strtod()`) is _that_ useful, while a higher-precision prob is. The tradeoff of course is speed/accuracy.

Just to add another voice: I also ran into this issue recently.  We were serialising data to disk via `DataFrame.to_csv` and then re-reading with `read_csv`, and were surprised to discover that the re-read data didn't exactly match the original.  While the errors are small, and not an issue for calculation, it's important to us that the serialisation round-trips correctly.  So a `float_precision='high'` option would be useful.

ok @amras1 we'll put this on the table for a new option.

Sounds good--I have the altered `xstrtod()` ready, so let me know if I should open a PR. However, the new `xstrtod()` isn't guaranteed to fix the round-trip issue @mdickinson noticed; `strtod()` is probably the best solution for that case.

I suppose you could have 3 levels for `float_precision=None|high|round_trip` or something where you either: use current xstrtod, new one, strtod


Resolving patch:
diff --git a/pandas/io/parsers.py b/pandas/io/parsers.py
--- a/pandas/io/parsers.py
+++ b/pandas/io/parsers.py
@@ -303,7 +303,8 @@ def _read(filepath_or_buffer, kwds):
     'error_bad_lines': True,
     'warn_bad_lines': True,
     'dtype': None,
-    'decimal': b'.'
+    'decimal': b'.',
+    'float_precision': None
 }
 
 _fwf_defaults = {
@@ -369,6 +370,7 @@ def parser_f(filepath_or_buffer,
                  date_parser=None,
 
                  memory_map=False,
+                 float_precision=None,
                  nrows=None,
                  iterator=False,
                  chunksize=None,
@@ -437,6 +439,7 @@ def parser_f(filepath_or_buffer,
                     encoding=encoding,
                     squeeze=squeeze,
                     memory_map=memory_map,
+                    float_precision=float_precision,
 
                     na_filter=na_filter,
                     compact_ints=compact_ints,
@@ -1264,6 +1267,11 @@ def TextParser(*args, **kwds):
         If True and `parse_dates` is True for a column, try to infer the
         datetime format based on the first datetime string. If the format
         can be inferred, there often will be a large parsing speed-up.
+    float_precision : string, default None
+        Specifies which converter the C engine should use for floating-point
+        values. The options are None for the ordinary converter,
+        'high' for the high-precision converter, and 'round_trip' for the
+        round-trip converter.
     """
     kwds['engine'] = 'python'
     return TextFileReader(*args, **kwds)
diff --git a/vb_suite/parser_vb.py b/vb_suite/parser_vb.py
--- a/vb_suite/parser_vb.py
+++ b/vb_suite/parser_vb.py
@@ -79,3 +79,22 @@
 cmd = "read_table(StringIO(data), sep=',', header=None, parse_dates=[1])"
 sdate = datetime(2012, 5, 7)
 read_table_multiple_date_baseline = Benchmark(cmd, setup, start_date=sdate)
+
+setup = common_setup + """
+from cStringIO import StringIO
+data = '''\
+0.1213700904466425978256438611,0.0525708283766902484401839501,0.4174092731488769913994474336
+0.4096341697147408700274695547,0.1587830198973579909349496119,0.1292545832485494372576795285
+0.8323255650024565799327547210,0.9694902427379478160318626578,0.6295047811546814475747169126
+0.4679375305798131323697930383,0.2963942381834381301075609371,0.5268936082160610157032465394
+0.6685382761849776311890991564,0.6721207066140679753374342908,0.6519975277021627935170045020
+'''
+data = data * 200
+"""
+cmd = "read_csv(StringIO(data), sep=',', header=None, float_precision=None)"
+sdate = datetime(2014, 8, 20)
+read_csv_default_converter = Benchmark(cmd, setup, start_date=sdate)
+cmd = "read_csv(StringIO(data), sep=',', header=None, float_precision='high')"
+read_csv_precise_converter = Benchmark(cmd, setup, start_date=sdate)
+cmd = "read_csv(StringIO(data), sep=',', header=None, float_precision='round_trip')"
+read_csv_roundtrip_converter = Benchmark(cmd, setup, start_date=sdate)


Test patch:
diff --git a/pandas/io/tests/test_parsers.py b/pandas/io/tests/test_parsers.py
--- a/pandas/io/tests/test_parsers.py
+++ b/pandas/io/tests/test_parsers.py
@@ -2523,6 +2523,12 @@ def test_verbose_import(self):
         finally:
             sys.stdout = sys.__stdout__
 
+    def test_float_precision_specified(self):
+        # Should raise an error if float_precision (C parser option) is specified
+        with tm.assertRaisesRegexp(ValueError, "The 'float_precision' option "
+                                   "is not supported with the 'python' engine"):
+            self.read_csv(StringIO('a,b,c\n1,2,3'), float_precision='high')
+
     def test_iteration_open_handle(self):
         if PY3:
             raise nose.SkipTest("won't work in Python 3 {0}".format(sys.version_info))
@@ -3088,6 +3094,25 @@ def test_compact_ints(self):
         ex_dtype = np.dtype([(str(i), 'u1') for i in range(4)])
         self.assertEqual(result.dtype, ex_dtype)
 
+    def test_precise_conversion(self):
+        # GH #8002
+        from decimal import Decimal
+        normal_errors = []
+        precise_errors = []
+        for num in np.linspace(1., 2., num=500): # test numbers between 1 and 2
+            text = 'a\n{0:.25}'.format(num) # 25 decimal digits of precision
+            normal_val = float(self.read_csv(StringIO(text))['a'][0])
+            precise_val = float(self.read_csv(StringIO(text), float_precision='high')['a'][0])
+            roundtrip_val = float(self.read_csv(StringIO(text), float_precision='round_trip')['a'][0])
+            actual_val = Decimal(text[2:])
+            def error(val):
+                return abs(Decimal('{0:.100}'.format(val)) - actual_val)
+            normal_errors.append(error(normal_val))
+            precise_errors.append(error(precise_val))
+            self.assertEqual(roundtrip_val, float(text[2:])) # round-trip should match float()
+        self.assertTrue(sum(precise_errors) < sum(normal_errors))
+        self.assertTrue(max(precise_errors) < max(normal_errors))
+
     def test_pass_dtype(self):
         data = """\
 one,two