Repository: pandas-dev/pandas
Pull request: BUG: Allow all int types for merge (GH28870)
- [x] closes #28870
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Problem statement:
Merge_asof() Requires specific int type, not reflected in error or documentation
#### Code Sample, a copy-pastable example if possible

```python
import pandas as pd

left = pd.DataFrame({'ts_int': [0, 100, 200], 'left_val': [1, 2, 3]})
right = pd.DataFrame({'ts_int': [50, 150, 250], 'right_val': [1, 2, 3]})
left['ts_int'] = left['ts_int'].astype(int)
right['ts_int'] = right['ts_int'].astype(int)
pd.merge_asof(left, right, on='ts_int', tolerance=100)

> pandas.errors.MergeError: key must be integer, timestamp or float

print(left['ts_int'].dtype)
>>> int32
print(right['ts_int'].dtype)
>>> int32
```
#### Problem description

merge_asof() throws the MergeError `key must be integer, timestamp or float` even when the data used to merge on is a valid integer type.

This happens with all types of int with the exception of int64, as a result of the check performed when a tolerance is passed _(pandas/core/reshape/merge.py:1641)_. This check uses the `is_int64_dtype(lt)`, which will ofcourse return `False` for every type of int that isn't `int64`. This requirement of the `on` key being int64 is however not documented in the docs, nor is it reflected from the raised error.

#### Expected Output

The expected output can go either of two ways. If it should be possible to perform the merge on every integer type, the expected output would be:
```
   ts_int  left_val  right_val
0       0         1        NaN
1     100         2        1.0
2     200         3        2.0
```

If it shouldn't be possible, I'd expect the returned MergeError and/or the docs to reflect that the integer type that should be used as key to merge on has to be of int64 type.

#### Output of ``pd.show_versions()``

<details>
INSTALLED VERSIONS
------------------
commit           : None
python           : 3.7.4.final.0
python-bits      : 64
OS               : Windows
OS-release       : 10
machine          : AMD64
processor        : Intel64 Family 6 Model 158 Stepping 10, GenuineIntel
byteorder        : little
LC_ALL           : None
LANG             : en_US.UTF-8
LOCALE           : None.None

pandas           : 0.25.1
numpy            : 1.17.2
pytz             : 2019.2
dateutil         : 2.8.0
pip              : 19.2.3
setuptools       : 41.2.0
Cython           : None
pytest           : None
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : None
IPython          : None
pandas_datareader: None
bs4              : None
bottleneck       : None
fastparquet      : None
gcsfs            : None
lxml.etree       : None
matplotlib       : 3.1.1
numexpr          : None
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : None
pytables         : None
s3fs             : None
scipy            : 1.3.1
sqlalchemy       : None
tables           : None
xarray           : None
xlrd             : None
xlwt             : None
xlsxwriter       : None
</details>

Hints from issue discussion:
I think this is because `.astype(int)` will convert your data to your OS's (Windows) platform int which is 32 bit. `is_int64_dtype` should just be replaced with `is_integer_dtype`. Care to submit a PR?
@mroeschke I can pick this up if @jjlkant doesn't want to.
Go for it @amcnulty93 
Had some issues getting the PR to work, hope it is okay like this? Please let me know if there are any other things I'd have to do first

Commit message: Whatsnew: int dtype tolerance fix in merge_asof

Patch to review:
diff --git a/doc/source/whatsnew/v0.25.2.rst b/doc/source/whatsnew/v0.25.2.rst
index 9789c9fce3541..8ce67866002aa 100644
--- a/doc/source/whatsnew/v0.25.2.rst
+++ b/doc/source/whatsnew/v0.25.2.rst
@@ -86,7 +86,7 @@ Groupby/resample/rolling
 Reshaping
 ^^^^^^^^^
 
--
+- Fix to ensure all int dtypes can be used in :meth:`pandas.core.reshape.merge_asof` when using a tolerance value (:issue:`28870`).
 -
 -
 -
diff --git a/pandas/core/reshape/merge.py b/pandas/core/reshape/merge.py
index 910c7ea561929..7bfc8153da568 100644
--- a/pandas/core/reshape/merge.py
+++ b/pandas/core/reshape/merge.py
@@ -28,7 +28,6 @@
     is_dtype_equal,
     is_extension_array_dtype,
     is_float_dtype,
-    is_int64_dtype,
     is_integer,
     is_integer_dtype,
     is_list_like,
@@ -1641,7 +1640,7 @@ def _get_merge_keys(self):
                 if self.tolerance < Timedelta(0):
                     raise MergeError("tolerance must be positive")
 
-            elif is_int64_dtype(lt):
+            elif is_integer_dtype(lt):
                 if not is_integer(self.tolerance):
                     raise MergeError(msg)
                 if self.tolerance < 0:
diff --git a/pandas/tests/reshape/merge/test_merge_asof.py b/pandas/tests/reshape/merge/test_merge_asof.py
index caf2539a9e150..206380c5ea46b 100644
--- a/pandas/tests/reshape/merge/test_merge_asof.py
+++ b/pandas/tests/reshape/merge/test_merge_asof.py
@@ -1287,3 +1287,19 @@ def test_timedelta_tolerance_nearest(self):
         )
 
         assert_frame_equal(result, expected)
+
+    def test_int_type_tolerance(self):
+        # GH #28870
+
+        left = pd.DataFrame({"a": [0, 100, 200], "left_val": [1, 2, 3]})
+        right = pd.DataFrame({"a": [50, 150, 250], "right_val": [1, 2, 3]})
+        left["a"] = left["a"].astype(np.int32)
+        right["a"] = right["a"].astype(np.int32)
+
+        expected = pd.DataFrame(
+            {"a": [0, 100, 200], "left_val": [1, 2, 3], "right_val": [np.nan, 1.0, 2.0]}
+        )
+        expected["a"] = expected["a"].astype(np.int32)
+
+        result = pd.merge_asof(left, right, on="a", tolerance=100)
+        assert_frame_equal(result, expected)


Code review comments:
diff_hunk: @@ -86,7 +86,7 @@ Groupby/resample/rolling
 Reshaping
 ^^^^^^^^^
 
--
+- Fix to ensure all int dtypes can be used in :meth:`pandas.core.reshape.merge_asof` when using a tolerance value (:issue:`28870`).
path: doc/source/whatsnew/v0.25.2.rst
text: @user1:
`` :func:`merge_asof` `` and maybe clarify that any non-int64 integer previously raised an erroneous `MergeError`