Repository: pandas-dev/pandas
Pull request: Make to_numeric default to correct precision
- [x] closes #31364
- [x] tests added / passed
  - tools/test_to_numeric.py:test_precision
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

This relates to a very old issue #8002 where the default precision for CSV files could create wrong answers in the last bit.  @jreback was involved in the PR review for that, which created the default to not be the high precision parser.

For `to_numeric()`, I switched it to use `precise_xstrtod` instead of `xstrtod`  by default.

Unknown whether there are performance implications for `to_numeric()`, although this comment https://github.com/pandas-dev/pandas/issues/17154#issuecomment-319917647 from @jorisvandenbossche indicates that maybe we should consider switching to the higher precision parser by default in `read_csv()` anyway. 

Open question as to whether performance analysis of *this* PR is needed beyond what is shown below, which seems to indicate that using `precise_xstrtod` is *faster* than `xstrtod` 

The other possibility is to use a keyword argument in `to_numeric` that would allow the higher precision parser.  

Problem statement:
to_numeric with errors = "coerce" is adding digits at the end
#### Code Sample, a copy-pastable example if possible

```python
# Your code here
#problem: pandas to_numeric might give some errors when using coerce 
# it is adding digits at the end.

import pandas as pd


#minimal example
data = [{'value': '.'}, {'value': '.'}, {'value': '.'}, {'value': '.'}, {'value': '243.164'}, {'value': '245.968'}, {'value': '249.585'}, {'value': '259.745'}, {'value': '265.742'}, {'value': '272.567'}]
df = pd.DataFrame(data,columns=['value'])

df.value = pd.to_numeric(df.value,errors='coerce')

#looks as if all is good:
df.value 

#but
df.value[4]


#this can be random:
data2 = [{'value': '.'}, {'value': '.'}, {'value': '.'}, {'value': '.'}, {'value': '243.164'}, {'value': '245.968'}, {'value': '249.585'}, {'value': '259.745'}, {'value': '265.742'}, {'value': '272.567'}, {'value': '279.196'}, {'value': '280.366'}, {'value': '275.034'}, {'value': '271.351'}, {'value': '272.889'}, {'value': '270.627'}, {'value': '280.828'}, {'value': '290.383'}, {'value': '308.153'}, {'value': '319.945'}, {'value': '336.0'}, {'value': '344.09'}, {'value': '351.385'}, {'value': '356.178'}, {'value': '359.82'}, {'value': '361.03'}, {'value': '367.701'}, {'value': '380.812'}, {'value': '387.98'}, {'value': '391.749'}, {'value': '391.171'}, {'value': '385.97'}, {'value': '385.345'}, {'value': '386.121'}, {'value': '390.996'}, {'value': '399.734'}, {'value': '413.073'}, {'value': '421.532'}, {'value':
'430.221'}, {'value': '437.092'}, {'value': '439.746'}, {'value': '446.01'}, {'value': '451.191'}, {'value': '460.463'}, {'value': '469.779'}, {'value': '472.025'}, {'value': '479.49'}, {'value': '474.864'}, {'value': '467.54'}, {'value': '471.978'}]


#now 4, 36, and 47 are wrong, with different endings.
df2 = pd.DataFrame(data2,columns=['value'])

df2.value = pd.to_numeric(df2.value,errors='coerce')


df2.value[[4,36,47]].to_list()
```
#### Problem description

Current behavior:  pandas to_numeric, when using errors='coerce' seem to randomly add 
decimals at the end of the number. 

#### Expected Output

no decimals should be added at the end.

#### Output of ``pd.show_versions()``

<details>

[paste the output of ``pd.show_versions()`` here below this line]


INSTALLED VERSIONS
------------------
commit           : None
python           : 3.7.3.final.0
python-bits      : 64
OS               : Windows
OS-release       : 10
machine          : AMD64
processor        : Intel64 Family 6 Model 85 Stepping 4, GenuineIntel
byteorder        : little
LC_ALL           : None
LANG             : en_US.UTF-8
LOCALE           : None.None

pandas           : 0.25.3
numpy            : 1.16.4
pytz             : 2019.1
dateutil         : 2.8.0
pip              : 19.1.1
setuptools       : 41.0.1
Cython           : 0.29.12
pytest           : 5.0.1
hypothesis       : None
sphinx           : 2.1.2
blosc            : None
feather          : None
xlsxwriter       : 1.1.8
lxml.etree       : 4.3.4
html5lib         : 1.0.1
pymysql          : None
psycopg2         : None
jinja2           : 2.10.1
IPython          : 7.6.1
pandas_datareader: 0.7.4
bs4              : 4.7.1
bottleneck       : 1.2.1
fastparquet      : None
gcsfs            : None
lxml.etree       : 4.3.4
matplotlib       : 3.1.2
numexpr          : 2.6.9
odfpy            : None
openpyxl         : 2.6.2
pandas_gbq       : None
pyarrow          : 0.15.1
pytables         : None
s3fs             : None
scipy            : 1.2.1
sqlalchemy       : 1.3.5
tables           : 3.5.2
xarray           : 0.12.3
xlrd             : 1.2.0
xlwt             : 1.3.0
xlsxwriter       : 1.1.8
</details>

Hints from issue discussion:
being more precise, I am getting:

df2.value[[4,36,47]].to_list()

to be
[243.16400000000002, 413.07300000000004, 474.86400000000003]

instead of 

[243.164, 413.073, 474.864]


Also, downcasting gives me different behavior in the two datasets:

```python
#smaller dataset, data
pd.to_numeric(df.value,errors='coerce',downcast="float")[4]
#243.164
pd.to_numeric(df.value,errors='coerce',downcast="integer")[4]
#243.16400000000002


#bigger dataset, data2
#downcast to integer works in the bigger dataset (dataset2)
pd.to_numeric(df2.value,errors='coerce',downcast="float")[[4,36,47]].to_list()
#output: [243.16400146484375, 413.072998046875, 474.864013671875]
pd.to_numeric(df2.value,errors='coerce',downcast="integer")[[4,36,47]].to_list()
#output: [243.16400000000002, 413.07300000000004, 474.86400000000003]







```
I see similar behavior for example:
```python
float('0.167')
0.167
type(float('0.167'))
<class 'float'>
```
vs 
```python
pd.to_numeric('0.167')
0.16699999999999998
type(pd.to_numeric('0.167'))
<class 'numpy.float64'>
```
I guess output of `to_numeric` should be `0.167` like the `float('0.167')`
Bug is in `pd.to_numeric`:
```
In [2]: s='243.164'

In [3]: float(s), pd.to_numeric(s)
Out[3]: (243.164, 243.16400000000002)
```
We are using internally `xstrtod` instead of `precise_xstrtod`.

Commit message: fix lint issue in parse_helper.h

Patch to review:
diff --git a/doc/source/whatsnew/v1.1.2.rst b/doc/source/whatsnew/v1.1.2.rst
index 6935a64c7572f..4606143c2301e 100644
--- a/doc/source/whatsnew/v1.1.2.rst
+++ b/doc/source/whatsnew/v1.1.2.rst
@@ -37,6 +37,7 @@ Bug fixes
 - Bug in :meth:`Series.dt.isocalendar` and :meth:`DatetimeIndex.isocalendar` that returned incorrect year for certain dates (:issue:`36032`)
 - Bug in :class:`DataFrame` indexing returning an incorrect :class:`Series` in some cases when the series has been altered and a cache not invalidated (:issue:`33675`)
 - Bug in :meth:`DataFrame.corr` causing subsequent indexing lookups to be incorrect (:issue:`35882`)
+- Bug in :func:`to_numeric` where precision was incorrect (:issue:`31364`)
 
 .. ---------------------------------------------------------------------------
 
diff --git a/pandas/_libs/src/parse_helper.h b/pandas/_libs/src/parse_helper.h
index 2ada0a4bd173d..d161c4e29fe15 100644
--- a/pandas/_libs/src/parse_helper.h
+++ b/pandas/_libs/src/parse_helper.h
@@ -18,7 +18,9 @@ int to_double(char *item, double *p_value, char sci, char decimal,
     char *p_end = NULL;
     int error = 0;
 
-    *p_value = xstrtod(item, &p_end, decimal, sci, '\0', 1, &error, maybe_int);
+    /* Switch to precise xstrtod GH 31364 */
+    *p_value = precise_xstrtod(item, &p_end, decimal, sci, '\0', 1,
+                               &error, maybe_int);
 
     return (error == 0) && (!*p_end);
 }
diff --git a/pandas/tests/tools/test_to_numeric.py b/pandas/tests/tools/test_to_numeric.py
index 263887a8ea36e..813457612f989 100644
--- a/pandas/tests/tools/test_to_numeric.py
+++ b/pandas/tests/tools/test_to_numeric.py
@@ -649,3 +649,10 @@ def test_failure_to_convert_uint64_string_to_NaN():
     ser = Series([32, 64, np.nan])
     result = to_numeric(pd.Series(["32", "64", "uint64"]), errors="coerce")
     tm.assert_series_equal(result, ser)
+
+
+def test_precision():
+    # GH 31364
+    result = to_numeric("243.164")
+
+    assert result == 243.164


Code review comments:
diff_hunk: @@ -649,3 +649,10 @@ def test_failure_to_convert_uint64_string_to_NaN():
     ser = Series([32, 64, np.nan])
     result = to_numeric(pd.Series(["32", "64", "uint64"]), errors="coerce")
     tm.assert_series_equal(result, ser)
+
+
+def test_precision():
path: pandas/tests/tools/test_to_numeric.py
text: @user1:
can you parameterize this and add the other examples from the OP (currently working an non-working ones that are fixed here).

can you rename this to test_precision_float_conversion

@author:
OK, I just copied the huge list of numbers that he had there and put it in the test.  It does fail in master, works with this patch.