Repository: pandas-dev/pandas

Issue:
BUG: merging with a boolean/int categorical column 
#### Code Sample, a copy-pastable example if possible

```python
dfA = pd.DataFrame({'id':[1,2,3,4,5,6,7,8,9,10],'colA':[3,4,2,4,3,4,5,4,5,6],'colB':[7,6,5,6,5,7,8,7,6,7],'colC':[False,True,True,False,False,True,False,True,True,True]})
dfA['colC'] = dfA['colC'].astype('category',categories=[True,False],ordered=True)
dfB = pd.DataFrame({'id':[2,5,7,8],'colD':[1,9,7,3]})

print("Before\n====")
print('dfA dtypes\n------')
print(dfA.dtypes)
print('\ndfA\n---')
print(dfA)
print('\ndfB\n---')
print(dfB)

dfA = pd.merge(left=dfA,right=dfB,how='left',on='id')
print("\nAfter\n=====")
print(dfA)
```
#### Problem description

This problem was asked on StackOverflow at https://stackoverflow.com/questions/45538092/merging-pandas-dataframes-containing-a-categorical-variable-fails-with-valueerr where it was suggested that it was a bug.

Two dataframes containing different columns can be combined using the pandas.merge() method. This works well but in the above example, converting one of the columns in the dataframe to a categorical variable causes the method to fail with error:

```python
/Users/.../env3/lib/python3.4/site-packages/pandas/core/internals.py in __init__(self, values, placement, ndim, fastpath)
    104             ndim = values.ndim
    105         elif values.ndim != ndim:
--> 106             raise ValueError('Wrong number of dimensions')
    107         self.ndim = ndim
    108 

ValueError: Wrong number of dimensions
```

Using df.ndim() indicates that both dataframes have 2 dimensions.

#### Expected Output

The expected output can be generated simply by commenting out the second line in the above code, the line that converts one of the columns to a categorical variable.

```python
   colA  colB   colC  id  colD
0     3     7  False   1   NaN
1     4     6   True   2   1.0
2     2     5   True   3   NaN
3     4     6  False   4   NaN
4     3     5  False   5   9.0
5     4     7   True   6   NaN
6     5     8  False   7   7.0
7     4     7   True   8   3.0
8     5     6   True   9   NaN
9     6     7   True  10   NaN
```
#### Output of ``pd.show_versions()``

<details>

[paste the output of ``pd.show_versions()`` here below this line]
INSTALLED VERSIONS
------------------
commit: None
python: 3.4.1.final.0
python-bits: 64
OS: Darwin
OS-release: 15.6.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_GB.UTF-8
LOCALE: en_GB.UTF-8

pandas: 0.20.1
pytest: None
pip: 9.0.1
setuptools: 34.1.0
Cython: None
numpy: 1.12.1
scipy: 0.16.1
xarray: None
IPython: 4.1.1
sphinx: None
patsy: None
dateutil: 2.6.0
pytz: 2017.2
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: 1.5.3
openpyxl: 2.4.7
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: 0.7.11.None
psycopg2: None
jinja2: 2.8
s3fs: None
pandas_gbq: None
pandas_datareader: None
</details>



Discussion:
@lvphj : Thanks for the issue!  I don't see why this shouldn't work given that only the `dtype` changed.  PR to patch this is welcome!
something odd going on in the actual merge. If we replace the boolean with strings it is ok; with ints looks like some odd type mapping.

```
In [24]: pd.merge(left=dfA.assign(colC=dfA.colC.cat.rename_categories(['foo', 'bar'])),right=dfB,how='left',on='id')
Out[24]: 
   colA  colB colC  id  colD
0     3     7  bar   1   NaN
1     4     6  foo   2   1.0
2     2     5  foo   3   NaN
3     4     6  bar   4   NaN
4     3     5  bar   5   9.0
5     4     7  foo   6   NaN
6     5     8  bar   7   7.0
7     4     7  foo   8   3.0
8     5     6  foo   9   NaN
9     6     7  foo  10   NaN

In [25]: pd.merge(left=dfA.assign(colC=dfA.colC.cat.rename_categories([1,2])),right=dfB,how='left',on='id')
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
ValueError: Buffer dtype mismatch, expected 'Python object' but got 'long'
Exception ignored in: 'pandas._libs.lib.is_bool_array'
ValueError: Buffer dtype mismatch, expected 'Python object' but got 'long'
Out[25]: 
   colA  colB colC  id  colD
0     3     7    2   1   NaN
1     4     6    1   2   1.0
2     2     5    1   3   NaN
3     4     6    2   4   NaN
4     3     5    2   5   9.0
5     4     7    1   6   NaN
6     5     8    2   7   7.0
7     4     7    1   8   3.0
8     5     6    1   9   NaN
9     6     7    1  10   NaN
```

Resolving patch:
diff --git a/pandas/core/internals.py b/pandas/core/internals.py
--- a/pandas/core/internals.py
+++ b/pandas/core/internals.py
@@ -5494,7 +5494,7 @@ def get_reindexed_values(self, empty_dtype, upcasted_na):
                     # preserve these for validation in _concat_compat
                     return self.block.values
 
-            if self.block.is_bool:
+            if self.block.is_bool and not self.block.is_categorical:
                 # External code requested filling/upcasting, bool values must
                 # be upcasted to object to avoid being upcasted to numeric.
                 values = self.block.astype(np.object_).values


Test patch:
diff --git a/pandas/tests/reshape/test_merge.py b/pandas/tests/reshape/test_merge.py
--- a/pandas/tests/reshape/test_merge.py
+++ b/pandas/tests/reshape/test_merge.py
@@ -1546,6 +1546,30 @@ def test_dtype_on_categorical_dates(self):
         result_inner = pd.merge(df, df2, how='inner', on=['date'])
         assert_frame_equal(result_inner, expected_inner)
 
+    @pytest.mark.parametrize('category_column,categories,expected_categories',
+                             [([False, True, True, False], [True, False],
+                               [True, False]),
+                              ([2, 1, 1, 2], [1, 2], [1, 2]),
+                              (['False', 'True', 'True', 'False'],
+                               ['True', 'False'], ['True', 'False'])])
+    def test_merging_with_bool_or_int_cateorical_column(self, category_column,
+                                                        categories,
+                                                        expected_categories):
+        # GH 17187
+        # merging with a boolean/int categorical column
+        df1 = pd.DataFrame({'id': [1, 2, 3, 4],
+                            'cat': category_column})
+        df1['cat'] = df1['cat'].astype('category',
+                                       categories=categories, ordered=True)
+        df2 = pd.DataFrame({'id': [2, 4], 'num': [1, 9]})
+        result = df1.merge(df2)
+        expected = pd.DataFrame({'id': [2, 4], 'cat': expected_categories,
+                                 'num': [1, 9]})
+        expected['cat'] = expected['cat'].astype('category',
+                                                 categories=categories,
+                                                 ordered=True)
+        assert_frame_equal(expected, result)
+
 
 @pytest.fixture
 def left_df():