Repository: pandas-dev/pandas
Pull request: BUG: regression in DataFrame.combine_first with integer columns (GH14687)
closes #14687


Problem statement:
combine_first throws ValueError: Cannot convert NA to integer
I do not understand why there is a need to convert NA to integer if the result does not have NAs. Perhaps the combine_first algo needs to do it under the hood?


#### A small, complete example of the issue

```python
from pandas import DataFrame
DataFrame({'a': [0, 1, 3, 5]}).combine_first(DataFrame({'a': [1, 4]}))

Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/IPython/core/interactiveshell.py", line 3066, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-16-12b973b1b150>", line 1, in <module>
    pd.DataFrame({'a': [0, 1, 3, 5]}).combine_first(pd.DataFrame({'a': [1, 4]}))
  File "/usr/local/lib/python3.4/dist-packages/pandas/core/frame.py", line 3787, in combine_first
    return self.combine(other, combiner, overwrite=False)
  File "/usr/local/lib/python3.4/dist-packages/pandas/core/frame.py", line 3714, in combine
    otherSeries = otherSeries.astype(new_dtype)
  File "/usr/local/lib/python3.4/dist-packages/pandas/core/generic.py", line 3054, in astype
    raise_on_error=raise_on_error, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/pandas/core/internals.py", line 3168, in astype
    return self.apply('astype', dtype=dtype, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/pandas/core/internals.py", line 3035, in apply
    applied = getattr(b, f)(**kwargs)
  File "/usr/local/lib/python3.4/dist-packages/pandas/core/internals.py", line 462, in astype
    values=values, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/pandas/core/internals.py", line 505, in _astype
    values = _astype_nansafe(values.ravel(), dtype, copy=True)
  File "/usr/local/lib/python3.4/dist-packages/pandas/types/cast.py", line 531, in _astype_nansafe
    raise ValueError('Cannot convert NA to integer')
ValueError: Cannot convert NA to integer
```

#### Expected Output
```
   a
0  0
1  1
2  3
3  5
```

It does work when at least one item is a float:

```python
DataFrame({'a': [0.0, 1, 3, 5]}).combine_first(DataFrame({'a': [1, 4]}))

     a
0  0.0
1  1.0
2  3.0
3  5.0
```

I am aware that integer series cannot have NAs but there is no need to introduce NAs here. I do like it that the series is not upcasted to float silently though.


#### Output of ``pd.show_versions()``
<details>

INSTALLED VERSIONS
------------------

commit: None
python: 3.4.3.final.0
python-bits: 64
OS: Linux
OS-release: 3.19.0-66-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.19.0
nose: None
pip: 1.5.4
setuptools: 3.3
Cython: 0.24.1
numpy: 1.11.2
scipy: 0.17.1
statsmodels: 0.6.1
xarray: None
IPython: 4.0.0
sphinx: None
patsy: 0.4.1
dateutil: 2.5.3
pytz: 2016.7
blosc: None
bottleneck: None
tables: 3.2.2
numexpr: 2.4.6
matplotlib: 1.5.1
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: 0.999
httplib2: 0.9.2
apiclient: None
sqlalchemy: None
pymysql: None
psycopg2: 2.6.1 (dt dec pq3 ext lo64)
jinja2: 2.8
boto: None
pandas_datareader: None


</details>

Hints from issue discussion:
This seems to be a regression from 0.18, as this worked before:

```
In [1]: DataFrame({'a': [0, 1, 3, 5]}).combine_first(DataFrame({'a': [1, 4]}))
Out[1]: 
   a
0  0
1  1
2  3
3  5

In [2]: pd.__version__
Out[2]: u'0.18.1'
```

@Dmitrii-I Thanks for the report! Always welcome to look into what could have caused this change.

Commit message: BUG: regression in DataFrame.combine_first with integer columns (GH14687)

Patch to review:
diff --git a/doc/source/whatsnew/v0.19.2.txt b/doc/source/whatsnew/v0.19.2.txt
index dabc6036fc9ba..291b0ffde145e 100644
--- a/doc/source/whatsnew/v0.19.2.txt
+++ b/doc/source/whatsnew/v0.19.2.txt
@@ -78,7 +78,7 @@ Bug Fixes
 - Bug in clipboard functions on linux with python2 with unicode and separators (:issue:`13747`)
 - Bug in clipboard functions on Windows 10 and python 3 (:issue:`14362`, :issue:`12807`)
 - Bug in ``.to_clipboard()`` and Excel compat (:issue:`12529`)
-
+- Bug in ``DataFrame.combine_first()`` for integer columns (:issue:`14687`).
 
 - Bug in ``pd.read_csv()`` in which the ``dtype`` parameter was not being respected for empty data (:issue:`14712`)
 - Bug in ``pd.read_csv()`` in which the ``nrows`` parameter was not being respected for large input when using the C engine for parsing (:issue:`7626`)
diff --git a/pandas/core/frame.py b/pandas/core/frame.py
index 0d4bcd781cf74..b441c17a34ab3 100644
--- a/pandas/core/frame.py
+++ b/pandas/core/frame.py
@@ -3665,10 +3665,8 @@ def combine(self, other, func, fill_value=None, overwrite=True):
                 otherSeries[other_mask] = fill_value
 
             # if we have different dtypes, possibily promote
-            if notnull(series).all():
-                new_dtype = this_dtype
-                otherSeries = otherSeries.astype(new_dtype)
-            else:
+            new_dtype = this_dtype
+            if this_dtype != other_dtype:
                 new_dtype = _find_common_type([this_dtype, other_dtype])
                 if not is_dtype_equal(this_dtype, new_dtype):
                     series = series.astype(new_dtype)
diff --git a/pandas/tests/frame/test_combine_concat.py b/pandas/tests/frame/test_combine_concat.py
index 5b5236843643d..c6b69dad3e6b5 100644
--- a/pandas/tests/frame/test_combine_concat.py
+++ b/pandas/tests/frame/test_combine_concat.py
@@ -725,3 +725,13 @@ def test_combine_first_period(self):
         exp = pd.DataFrame({'P': exp_dts}, index=[1, 2, 3, 4, 5, 7])
         tm.assert_frame_equal(res, exp)
         self.assertEqual(res['P'].dtype, 'object')
+
+    def test_combine_first_int(self):
+        # GH14687 - integer series that do no align exactly
+
+        df1 = pd.DataFrame({'a': [0, 1, 3, 5]}, dtype='int64')
+        df2 = pd.DataFrame({'a': [1, 4]}, dtype='int64')
+
+        res = df1.combine_first(df2)
+        tm.assert_frame_equal(res, df1)
+        self.assertEqual(res['a'].dtype, 'int64')


Code review comments:
diff_hunk: @@ -3665,10 +3665,8 @@ def combine(self, other, func, fill_value=None, overwrite=True):
                 otherSeries[other_mask] = fill_value
 
             # if we have different dtypes, possibily promote
-            if notnull(series).all():
-                new_dtype = this_dtype
-                otherSeries = otherSeries.astype(new_dtype)
-            else:
+            new_dtype = this_dtype
+            if this_dtype != other_dtype:
path: pandas/core/frame.py
text: @user1:
use ``not is_dtype_equal``