Repository: pandas-dev/pandas

Issue:
pd.merge() doesn't merge int and str column dtypes but no warning or error
When merging an int dtype with a str dtype the join does not work:

``` python
>>> import pandas as pd
>>> df1 = pd.DataFrame({"A":[0]})
>>> df2 = pd.DataFrame({"A":["0"]})
>>> pd.merge(df1, df2, on=["A"])
Empty DataFrame
Columns: [A]
Index: []
```

I think it would be better to get a warning that the join is performed on incompatible column dtypes.

This is my pandas version:

```
>>> pd.show_versions()                                                                                                                                    

INSTALLED VERSIONS
------------------
commit: None
python: 2.7.9.final.0
python-bits: 64
OS: Darwin
OS-release: 13.4.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

pandas: 0.15.2
nose: 1.3.4
Cython: 0.21
numpy: 1.9.2
scipy: 0.15.1                                                                                                                                                          
statsmodels: 0.6.1
IPython: 2.2.0
sphinx: 1.2.3
patsy: 0.3.0
dateutil: 2.4.1
pytz: 2014.9
bottleneck: None
tables: 3.1.1
numexpr: 2.3.1
matplotlib: 1.4.3
openpyxl: 1.8.5
xlrd: 0.9.3
xlwt: 0.7.5
xlsxwriter: 0.5.7
lxml: 3.4.0
bs4: 4.3.2
html5lib: None
httplib2: None
apiclient: None
rpy2: 2.5.6
sqlalchemy: 0.9.7
pymysql: None
psycopg2: None
```

Thanks for all your work on pandas!



Discussion:
You are doing an inner merge, which doesn't match. Not sure if we could reliably detect this, as it involves a computation to figure out that you have strings that looks like numbers.

```
In [8]: >>> pd.merge(df1, df2, on=["A"],how='outer')
Out[8]: 
   A
0  0
1  0

In [9]: >>> pd.merge(df1, df2, on=["A"],how='outer').dtypes
Out[9]: 
A    float64
dtype: object
```

you can also do

`df.convert_objects(convert_numeric=True)` to force the objects to become numbers

Thanks for your reply and the quick fix. I actually don't think that one should check whether the string represents numbers or not. I think it is more about whether the dtypes of the columns with the same name match. If you do an inner merge on DataFrames with no matching columns you get a MergeError. I think it would make sense to also throw one if there are matching columns but their dtypes can't be silently cast to match. Especially if one specifically sets the option `on=["A"]`.

@inodb @jreback: I would like to take a stab at this. From what I can understand, I need to check the datatypes of the columns that are to be merged, and if they're unequal then throw an error. Could you tell me what type of error ? Thanks.

this should raise a `ValueError` if the dtypes for the on columns don't match

Suppose the case of an int and float column. I don't think it should raise if you want to merge on those columns? (can eg already typically occur when having NaNs in one of both)

right we care about 'obvious' mismatches here that by-definition cannot match. so `object` vs `int/float` (or datetimelike vs `int/float/object`). We will need to `lib.infer_dtype` to make this more specific (e.g. `string/unicode` cannot match `int`)

Personally, I would leave this as a responsibility of the user. 
Although I understand that the example above can be confusing at first, trying to guess when the user passed wrong dtypes by accident instead of on purpose seems like adding complexity that is not really worth it (but it's a trade-off of course) 

Something else, currently, as @jreback shows above (https://github.com/pydata/pandas/issues/9780#issuecomment-88883521), the strings of `df2` get coerced to numeric values on merging:

```
In [57]: pd.merge(df1, df2, on=["A"],how='outer').dtypes
Out[57]: 
A    int64
dtype: object
```

I suppose this happens on purpose? IMO it should return object array and keep the original values (like `pd.concat` does)

@jorisvandenbossche yes, we should certainly not coerce on mixed types in merging, unless they are losslessly convertible (e.g. int & float), so maybe make a separate issue.

But I think we should raise on str/numeric, and datetimelike/(str or numeric), it is simply not possible (and if the user _really_ wants that, then its just a concat).

> and if the user really wants that, then its just a concat

That's a good point :-)

> yes, we should certainly not coerce on mixed types in merging, unless they are losslessly convertible (e.g. int & float), so maybe make a separate issue.

On a second thought, if we disallow merging on str/numeric (the case of the initial example), I don't think are cases left that we would allow but  where no coercing should happen? (for which I wanted to open an new issue)

@jorisvandenbossche can you show example / elaborate on your last?

> On a second thought, if we disallow merging on str/numeric (the case of the initial example), I don't think are cases left that we would allow but where no coercing should happen? (for which I wanted to open an new issue)

Well, the original example above has a dataframe with integers in the key column, and another dataframe with strings in the key column. They are now coerced to integers, something I think should not happen:

```
In [62]: df1 = pd.DataFrame({"A":[0], "B":[1]})

In [63]: df2 = pd.DataFrame({"A":["0"], "B":[2]})

In [64]: pd.merge(df1, df2, on=["A"],how='outer')
Out[64]: 
   A  B_x  B_y
0  0  1.0  NaN
1  0  NaN  2.0

In [65]: pd.merge(df1, df2, on=["A"],how='outer').A.dtype
Out[65]: dtype('int64')

In [66]: pd.merge(df1, df2, on=["A"],how='outer').A.values
Out[66]: array([0, 0])
```

And I wanted to open an new issue for this. But, this is also an example where we would want to raise an error about "incompatible columns to merge on". 
So would there still be cases where we would not raise such an error, but still happens unwanted coercing?

ah I c. I think this should raise a nice errors message (maybe saying that you might want to use `pd.concat`), which will simply work.

concat

```
In [16]: pd.concat([df1,df2])
Out[16]:
   A
0  0
0  0

In [17]: pd.concat([df1,df2]).dtypes
Out[17]:
A    object
dtype: object

In [18]: pd.concat([df1,df2]).values
Out[18]:
array([[0],
       ['0']], dtype=object)
```

merge

```
In [19]: pd.merge(df1, df2, on ='A')
Out[19]:
Empty DataFrame
Columns: [A]
Index: []

In [20]: pd.merge(df1, df2, on ='A', how='outer')
Out[20]:
   A
0  0
1  0

In [21]: pd.merge(df1, df2, on ='A', how='outer').dtypes
Out[21]:
A    int64
dtype: object
```

so [20,21] are wrong (this should be `object`).

But I would actually simply raise `ValueError` on the merge. Its a mistake on the users part.


Resolving patch:
diff --git a/pandas/core/reshape/merge.py b/pandas/core/reshape/merge.py
--- a/pandas/core/reshape/merge.py
+++ b/pandas/core/reshape/merge.py
@@ -27,6 +27,7 @@
     is_dtype_equal,
     is_bool,
     is_list_like,
+    is_datetimelike,
     _ensure_int64,
     _ensure_float64,
     _ensure_object,
@@ -962,6 +963,33 @@ def _maybe_coerce_merge_keys(self):
                 elif lib.infer_dtype(lk) == lib.infer_dtype(rk):
                     pass
 
+            # Check if we are trying to merge on obviously
+            # incompatible dtypes GH 9780
+            elif is_numeric_dtype(lk) and not is_numeric_dtype(rk):
+                msg = ("You are trying to merge on {lk_dtype} and "
+                       "{rk_dtype} columns. If you wish to proceed "
+                       "you should use pd.concat".format(lk_dtype=lk.dtype,
+                                                         rk_dtype=rk.dtype))
+                raise ValueError(msg)
+            elif not is_numeric_dtype(lk) and is_numeric_dtype(rk):
+                msg = ("You are trying to merge on {lk_dtype} and "
+                       "{rk_dtype} columns. If you wish to proceed "
+                       "you should use pd.concat".format(lk_dtype=lk.dtype,
+                                                         rk_dtype=rk.dtype))
+                raise ValueError(msg)
+            elif is_datetimelike(lk) and not is_datetimelike(rk):
+                msg = ("You are trying to merge on {lk_dtype} and "
+                       "{rk_dtype} columns. If you wish to proceed "
+                       "you should use pd.concat".format(lk_dtype=lk.dtype,
+                                                         rk_dtype=rk.dtype))
+                raise ValueError(msg)
+            elif not is_datetimelike(lk) and is_datetimelike(rk):
+                msg = ("You are trying to merge on {lk_dtype} and "
+                       "{rk_dtype} columns. If you wish to proceed "
+                       "you should use pd.concat".format(lk_dtype=lk.dtype,
+                                                         rk_dtype=rk.dtype))
+                raise ValueError(msg)
+
             # Houston, we have a problem!
             # let's coerce to object if the dtypes aren't
             # categorical, otherwise coerce to the category


Test patch:
diff --git a/pandas/tests/reshape/merge/test_merge.py b/pandas/tests/reshape/merge/test_merge.py
--- a/pandas/tests/reshape/merge/test_merge.py
+++ b/pandas/tests/reshape/merge/test_merge.py
@@ -6,6 +6,7 @@
 from numpy import nan
 import numpy as np
 import random
+import re
 
 import pandas as pd
 from pandas.compat import lrange, lzip
@@ -1370,30 +1371,47 @@ def f():
         pytest.raises(NotImplementedError, f)
 
 
-@pytest.fixture
-def df():
-    return DataFrame(
-        {'A': ['foo', 'bar'],
-         'B': Series(['foo', 'bar']).astype('category'),
-         'C': [1, 2],
-         'D': [1.0, 2.0],
-         'E': Series([1, 2], dtype='uint64'),
-         'F': Series([1, 2], dtype='int32')})
-
-
 class TestMergeDtypes(object):
 
-    def test_different(self, df):
-
-        # we expect differences by kind
-        # to be ok, while other differences should return object
-
-        left = df
-        for col in df.columns:
-            right = DataFrame({'A': df[col]})
+    @pytest.mark.parametrize('right_vals', [
+        ['foo', 'bar'],
+        Series(['foo', 'bar']).astype('category'),
+        [1, 2],
+        [1.0, 2.0],
+        Series([1, 2], dtype='uint64'),
+        Series([1, 2], dtype='int32')
+    ]
+    )
+    def test_different(self, right_vals):
+
+        left = DataFrame({'A': ['foo', 'bar'],
+                          'B': Series(['foo', 'bar']).astype('category'),
+                          'C': [1, 2],
+                          'D': [1.0, 2.0],
+                          'E': Series([1, 2], dtype='uint64'),
+                          'F': Series([1, 2], dtype='int32')})
+        right = DataFrame({'A': right_vals})
+
+        # GH 9780
+        # We allow merging on object and categorical cols and cast
+        # categorical cols to object
+        if (is_categorical_dtype(right['A'].dtype) or
+           is_object_dtype(right['A'].dtype)):
             result = pd.merge(left, right, on='A')
             assert is_object_dtype(result.A.dtype)
 
+        # GH 9780
+        # We raise for merging on object col and int/float col and
+        # merging on categorical col and int/float col
+        else:
+            msg = ("You are trying to merge on "
+                   "{lk_dtype} and {rk_dtype} columns. "
+                   "If you wish to proceed you should use "
+                   "pd.concat".format(lk_dtype=left['A'].dtype,
+                                      rk_dtype=right['A'].dtype))
+            with tm.assert_raises_regex(ValueError, msg):
+                pd.merge(left, right, on='A')
+
     @pytest.mark.parametrize('d1', [np.int64, np.int32,
                                     np.int16, np.int8, np.uint8])
     @pytest.mark.parametrize('d2', [np.int64, np.float64,
@@ -1462,6 +1480,42 @@ def test_merge_on_ints_floats_warning(self):
             result = B.merge(A, left_on='Y', right_on='X')
             assert_frame_equal(result, expected[['Y', 'X']])
 
+    @pytest.mark.parametrize('df1_vals, df2_vals', [
+        ([0, 1, 2], ["0", "1", "2"]),
+        ([0.0, 1.0, 2.0], ["0", "1", "2"]),
+        ([0, 1, 2], [u"0", u"1", u"2"]),
+        (pd.date_range('1/1/2011', periods=2, freq='D'), ['2011-01-01',
+                                                          '2011-01-02']),
+        (pd.date_range('1/1/2011', periods=2, freq='D'), [0, 1]),
+        (pd.date_range('1/1/2011', periods=2, freq='D'), [0.0, 1.0]),
+        ([0, 1, 2], Series(['a', 'b', 'a']).astype('category')),
+        ([0.0, 1.0, 2.0], Series(['a', 'b', 'a']).astype('category')),
+    ])
+    def test_merge_incompat_dtypes(self, df1_vals, df2_vals):
+        # GH 9780
+        # Raise a ValueError when a user tries to merge on
+        # dtypes that are incompatible (e.g., obj and int/float)
+
+        df1 = DataFrame({'A': df1_vals})
+        df2 = DataFrame({'A': df2_vals})
+
+        msg = ("You are trying to merge on {lk_dtype} and "
+               "{rk_dtype} columns. If you wish to proceed "
+               "you should use pd.concat".format(lk_dtype=df1['A'].dtype,
+                                                 rk_dtype=df2['A'].dtype))
+        msg = re.escape(msg)
+        with tm.assert_raises_regex(ValueError, msg):
+            pd.merge(df1, df2, on=['A'])
+
+        # Check that error still raised when swapping order of dataframes
+        msg = ("You are trying to merge on {lk_dtype} and "
+               "{rk_dtype} columns. If you wish to proceed "
+               "you should use pd.concat".format(lk_dtype=df2['A'].dtype,
+                                                 rk_dtype=df1['A'].dtype))
+        msg = re.escape(msg)
+        with tm.assert_raises_regex(ValueError, msg):
+            pd.merge(df2, df1, on=['A'])
+
 
 @pytest.fixture
 def left():