Repository: pandas-dev/pandas

Issue:
Confusing "MergeError: incompatible merge keys [1] category and category, must be the same type"
I'm using `pd.merges_asof()` to merge two dataframes together. The `by` keys are categoricals, but not equal, apparently (one has a few values more than the other). It took me a little bit to figure that out, however, because the error message for this was a bit confusing: "incompatible merge keys [1] category and category, must be the same type". It'd be nice if it was clearer. I can do the pull request, once we figure out how to make this better.

I see the following possible solutions:
1. Special-case the error message for dtypes that take parameters and so are not necessarily all equal. Something a bit like: "incompatible merge keys: both are category, but not equal ones" (Easiest solution.)
2. Make a nicer error message for categories, by digging a little bit into what makes them not equal. Something like "incompatible merge keys: both categories, but the left one has 3 levels more", or "but they have different levels: ..."
3. Change the `__str__` method for CategoricalDType to print something a bit more informative than "category". (No guarantee though that what we would print would always allow people to distinguish two not-equal categoricals as not equal.. unless we were to print out all the levels.)

Anything that sounds good here?


Discussion:
This is pretty tricky. Considering both option 1 and 2 to be a form of special casing I wouldn't be in favor of either of them. Not sure how option 3 would end up looking or if it even makes sense.

What would even be the requirements here? That the categoricals being merged would have to be ordered, monotonic, and that the right categorical would have to be a subset of the left?
This is about categoricals on both sides of the `by` key of `pd.merge_asof()`. My superficial understanding is that they need to be equal, because there's the equivalent of a groupby on those happening somewhere under the hood. But I'll let someone who knows more chime in.
@chrish42 this is correct, categorical must be exactly equal. I suppose the error message could be enhanced if you wanted to do a pull request. soln 1 is the most reasonable. the others are non-trivial / hard.

Resolving patch:
diff --git a/pandas/core/reshape/merge.py b/pandas/core/reshape/merge.py
--- a/pandas/core/reshape/merge.py
+++ b/pandas/core/reshape/merge.py
@@ -1446,10 +1446,26 @@ def _get_merge_keys(self):
         # validate index types are the same
         for i, (lk, rk) in enumerate(zip(left_join_keys, right_join_keys)):
             if not is_dtype_equal(lk.dtype, rk.dtype):
-                raise MergeError("incompatible merge keys [{i}] {lkdtype} and "
-                                 "{rkdtype}, must be the same type"
-                                 .format(i=i, lkdtype=lk.dtype,
-                                         rkdtype=rk.dtype))
+                if (is_categorical_dtype(lk.dtype) and
+                        is_categorical_dtype(rk.dtype)):
+                    # The generic error message is confusing for categoricals.
+                    #
+                    # In this function, the join keys include both the original
+                    # ones of the merge_asof() call, and also the keys passed
+                    # to its by= argument. Unordered but equal categories
+                    # are not supported for the former, but will fail
+                    # later with a ValueError, so we don't *need* to check
+                    # for them here.
+                    msg = ("incompatible merge keys [{i}] {lkdtype} and "
+                           "{rkdtype}, both sides category, but not equal ones"
+                           .format(i=i, lkdtype=repr(lk.dtype),
+                                   rkdtype=repr(rk.dtype)))
+                else:
+                    msg = ("incompatible merge keys [{i}] {lkdtype} and "
+                           "{rkdtype}, must be the same type"
+                           .format(i=i, lkdtype=repr(lk.dtype),
+                                   rkdtype=repr(rk.dtype)))
+                raise MergeError(msg)
 
         # validate tolerance; must be a Timedelta if we have a DTI
         if self.tolerance is not None:
@@ -1462,7 +1478,7 @@ def _get_merge_keys(self):
             msg = ("incompatible tolerance {tolerance}, must be compat "
                    "with type {lkdtype}".format(
                        tolerance=type(self.tolerance),
-                       lkdtype=lt.dtype))
+                       lkdtype=repr(lt.dtype)))
 
             if is_datetime64_dtype(lt) or is_datetime64tz_dtype(lt):
                 if not isinstance(self.tolerance, Timedelta):


Test patch:
diff --git a/pandas/tests/reshape/merge/test_merge_asof.py b/pandas/tests/reshape/merge/test_merge_asof.py
--- a/pandas/tests/reshape/merge/test_merge_asof.py
+++ b/pandas/tests/reshape/merge/test_merge_asof.py
@@ -994,9 +994,8 @@ def test_on_float_by_int(self):
 
         assert_frame_equal(result, expected)
 
-    def test_merge_datatype_error(self):
-        """ Tests merge datatype mismatch error """
-        msg = r'merge keys \[0\] object and int64, must be the same type'
+    def test_merge_datatype_error_raises(self):
+        msg = r'incompatible merge keys \[0\] .*, must be the same type'
 
         left = pd.DataFrame({'left_val': [1, 5, 10],
                              'a': ['a', 'b', 'c']})
@@ -1006,6 +1005,18 @@ def test_merge_datatype_error(self):
         with pytest.raises(MergeError, match=msg):
             merge_asof(left, right, on='a')
 
+    def test_merge_datatype_categorical_error_raises(self):
+        msg = (r'incompatible merge keys \[0\] .* both sides category, '
+               'but not equal ones')
+
+        left = pd.DataFrame({'left_val': [1, 5, 10],
+                             'a': pd.Categorical(['a', 'b', 'c'])})
+        right = pd.DataFrame({'right_val': [1, 2, 3, 6, 7],
+                              'a': pd.Categorical(['a', 'X', 'c', 'X', 'b'])})
+
+        with pytest.raises(MergeError, match=msg):
+            merge_asof(left, right, on='a')
+
     @pytest.mark.parametrize('func', [lambda x: x, lambda x: to_datetime(x)],
                              ids=['numeric', 'datetime'])
     @pytest.mark.parametrize('side', ['left', 'right'])