Repository: pandas-dev/pandas

Issue:
Appending Pandas dataframes in for loop results in ValueError
I recently posted this on StackOverflow. It seems to be a bug so I am posting here as well.

I want to generate a dataframe that is created by appended several separate dataframes generated in a for loop. Each individual dataframe consists of a name column, a range of integers and a column identifying a category to which the integer belongs (e.g. quintile 1 to 5). If I generate each dataframe individually and then append one to the other to create a 'master' dataframe then there are no problems. However, when I use a loop to create each individual dataframe then trying to append a dataframe to the master dataframe results in:

`ValueError: incompatible categories in categorical concat`

A work-around (suggested by jezrael) involved appending each dataframe to a list of dataframes and concatenating them using pd.concat.

I've written a simplified loop to illustrate:
#### Code Sample, a copy-pastable example if possible

``` python
import numpy as np
import pandas as pd

# Define column names
colNames = ('a','b','c')

# Define a dataframe with the required column names
masterDF = pd.DataFrame(columns = colNames)

# A list of the group names
names = ['Group1','Group2','Group3']

# Create a dataframe for each group
for i in names:
    tempDF = pd.DataFrame(columns = colNames)
    tempDF['a'] = np.arange(1,11,1)
    tempDF['b'] = i
    tempDF['c'] = pd.cut(np.arange(1,11,1),
                        bins = np.linspace(0,10,6),
                        labels = [1,2,3,4,5])
    print(tempDF)
    print('\n')

    # Try to append temporary DF to master DF
    masterDF = masterDF.append(tempDF,ignore_index=True)

print(masterDF)
```
#### Expected Output

```
     a       b  c
 0   1  Group1  1
 1   2  Group1  1
 2   3  Group1  2
 3   4  Group1  2
 4   5  Group1  3
 5   6  Group1  3
 6   7  Group1  4
 7   8  Group1  4
 8   9  Group1  5
 9  10  Group1  5
10  11  Group2  1
11  12  Group2  1
12  13  Group2  2
13  14  Group2  2
...
28  29  Group3  5
29  30  Group3  5
```
#### output of `pd.show_versions()`
## INSTALLED VERSIONS

commit: None
python: 3.4.1.final.0
python-bits: 64
OS: Darwin
OS-release: 15.5.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_GB.UTF-8

pandas: 0.18.1
nose: None
pip: 1.5.6
setuptools: 20.1.1
Cython: None
numpy: 1.11.0
scipy: 0.16.1
statsmodels: None
xarray: None
IPython: 4.1.1
sphinx: None
patsy: None
dateutil: 2.5.3
pytz: 2016.4
blosc: None
bottleneck: None
tables: None
numexpr: None
matplotlib: 1.5.0
openpyxl: 2.3.2
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: None
pymysql: 0.7.4.None
psycopg2: None
jinja2: 2.8
boto: None
pandas_datareader: None



Discussion:
`cut` returns a categorcial. By design, you can't `append` new categories

``` python
In [6]: s1 = pd.Series(['a', 'b']).astype('category')

In [7]: s1.append(pd.Series(['b', 'c']).astype('category'))
...
ValueError: incompatible categories in categorical concat
```

I believe your code would work if you change the `pd.cut(...)` to `pd.cut(...).categories`.

If you change your example code slightly so there are no NEW categories being added:

``` python
In [6]: s1 = pd.Series(['a', 'b']).astype('category')
In [7]: s1.append(pd.Series(['b', 'a']).astype('category'))

```

then it runs OK. In the original problem, the pd.cut() function generates the same categories in each dataframe, namely 1 to 5, so no new categories are being added.

Gotcha, here's a simpler example.

``` python
In [44]: df = pd.DataFrame(columns=['a'])

In [45]: a = pd.DataFrame({"a": pd.Categorical([1, 2, 3], ordered=True)})

In [46]: a.a
Out[46]:
0    1
1    2
2    3
Name: a, dtype: category
Categories (3, int64): [1 < 2 < 3]

In [47]: df.append(a).a
Out[47]:
0    1
1    2
2    3
Name: a, dtype: category
Categories (3, int64): [1, 2, 3]
```

So the orderedness of `a` is lost in the `append`. @lvphj any interest in digging through the traceback to see where it's lost?

Certainly interested – but may not have the skill set.

👍 just post here if you have any questions. Either way, thanks for the report.

Just a hunch, but I would start looking in https://github.com/pydata/pandas/blob/1a9abc44bbfd65675fd99701fe33aad8805ab147/pandas/types/concat.py#L147

this is by definition. you need union_categorical 

@jreback I think my last example should work, no? E.g. `df.append(a)` should have an ordered categorical if `a` was ordered?

Yes, I think that is correct.

It only seems to happen when you start with an empty frame, or append an empty frame:

```
In [29]: a.append(df).a  # <-- append empty frame
Out[29]:
0    1
1    2
2    3
Name: a, dtype: category
Categories (3, int64): [1, 2, 3]

In [30]: a.append(a).a
Out[30]:
0    1
1    2
2    3
0    1
1    2
2    3
Name: a, dtype: category
Categories (3, int64): [1 < 2 < 3]
```

Hmm, is the empty set of categories ordered or not? 😄 `pd.Categorical([]).ordered` is False (by default).
Is this worth special casing so that `empty_categorical.append(ordered_categorical)` becomes ordered? I think so, but maybe not.

cc @janschulz, thoughts?

Well, if we say that an empty series is ordered=False, then it should actually raise an error instead of changing the order of the result :-) 
But actually, in this case, you don't have an empty categorical, but just an empty frame without dtype info, so in this case it should ignore the fact that that part is ordered or not.

The problem is here: https://github.com/pydata/pandas/blob/1a9abc44bbfd65675fd99701fe33aad8805ab147/pandas/types/concat.py#L201 When concat is not dealing with only categoricals, but with a mixture of categoricals and object arrays, it takes the categories from the first categorical to concat, but not the other properties like ordered or not. Should be an easy fix to also pass ordered there.

```
In [44]: df = pd.DataFrame(columns=['a'])
In [45]: a = pd.DataFrame({"a": pd.Categorical([1, 2, 3], ordered=True)})
In [47]: df.append(a).a
```

IMO there can be two interpretations:
- If the df in the above append is seen as already having `category` column (and this is `Category([])` then this should IMO error because `Category([])` is ordered false (default) and has different Categories (= empty) and therefore this two shouldn't be appendable (as with two different categoricals in the column).
- If the dataframe has no type information at all and this is seen as basically a `set the column to the new type` and it should retain the information in the appended categorical.

``` py
pandas.DataFrame({"a": pandas.Categorical([1,2,3])}).append(pandas.DataFrame({"a": pandas.Categorical([1,2])})) # -> Error as it should be...
pandas.DataFrame({"a": pandas.Categorical([])}).append(pandas.DataFrame({"a": pandas.Categorical([1,2])})) # -> Also errors...
```

The question is if an empty column is the same as a categorical column without any value

IMO that's the difference between this two dataframes:

```
# 1. defined column type, but no value
pandas.DataFrame({"a": [1]})[[False]].dtypes # -> dtype of a: int64
print(pandas.DataFrame(columns=["a"])).dtypes # -> dtype of a: object
as_float = pandas.DataFrame({"a": [1.]})[[False]] # -> float64
as_float.append(pandas.DataFrame({"a": [1]})).dtypes # -> kept as float64 as that takes both int and float
```

and

```
# 2. no defined value
empty_object = pandas.DataFrame(columns=["a"])
print(empty_object.dtypes) # -> object
empty_object.append(pandas.DataFrame({"a": [1]})).dtypes # -> Float64 ??? why not int64 or object?
```

the first is just the usual  "cast to something which can take both" which is the rule for everything but categorical.  The second seems to be the upcast rules for int + object? So if the second follows the "normal rules", then IMO appending a categorical should also follow the usual categorical rules, aka erroring.

I met the same problem in #13626 and wrote short summary of `Series` `Index`differences.

How about following spec:
- concat 2 categories -> use the rule of `union_categorical`
- concat category and other dtype (which values are all in the category, including empty) -> category
  - this rule is applied regardless of order (if there is at least one category in concatenating values)
  - the property like `ordered` should be preserved.
- concat category and other dtype (which values are not in the category) -> not category (dtype is infered)  


Resolving patch:
diff --git a/pandas/core/internals.py b/pandas/core/internals.py
--- a/pandas/core/internals.py
+++ b/pandas/core/internals.py
@@ -4787,10 +4787,9 @@ def concatenate_block_managers(mgrs_indexers, axes, concat_axis, copy):
         [get_mgr_concatenation_plan(mgr, indexers)
          for mgr, indexers in mgrs_indexers], concat_axis)
 
-    blocks = [make_block(concatenate_join_units(join_units, concat_axis,
-                                                copy=copy),
-                         placement=placement)
-              for placement, join_units in concat_plan]
+    blocks = [make_block(
+        concatenate_join_units(join_units, concat_axis, copy=copy),
+        placement=placement) for placement, join_units in concat_plan]
 
     return BlockManager(blocks, axes)
 
diff --git a/pandas/tools/merge.py b/pandas/tools/merge.py
--- a/pandas/tools/merge.py
+++ b/pandas/tools/merge.py
@@ -1290,9 +1290,12 @@ def concat(objs, axis=0, join='outer', join_axes=None, ignore_index=False,
     join_axes : list of Index objects
         Specific indexes to use for the other n - 1 axes instead of performing
         inner/outer set logic
-    verify_integrity : boolean, default False
-        Check whether the new concatenated axis contains duplicates. This can
-        be very expensive relative to the actual data concatenation
+    ignore_index : boolean, default False
+        If True, do not use the index values along the concatenation axis. The
+        resulting axis will be labeled 0, ..., n - 1. This is useful if you are
+        concatenating objects where the concatenation axis does not have
+        meaningful indexing information. Note the index values on the other
+        axes are still respected in the join.
     keys : sequence, default None
         If multiple levels passed, should contain tuples. Construct
         hierarchical index using the passed keys as the outermost level
@@ -1301,12 +1304,9 @@ def concat(objs, axis=0, join='outer', join_axes=None, ignore_index=False,
         MultiIndex. Otherwise they will be inferred from the keys
     names : list, default None
         Names for the levels in the resulting hierarchical index
-    ignore_index : boolean, default False
-        If True, do not use the index values along the concatenation axis. The
-        resulting axis will be labeled 0, ..., n - 1. This is useful if you are
-        concatenating objects where the concatenation axis does not have
-        meaningful indexing information. Note the index values on the other
-        axes are still respected in the join.
+    verify_integrity : boolean, default False
+        Check whether the new concatenated axis contains duplicates. This can
+        be very expensive relative to the actual data concatenation
     copy : boolean, default True
         If False, do not copy data unnecessarily
 
@@ -1512,10 +1512,9 @@ def get_result(self):
 
                 mgrs_indexers.append((obj._data, indexers))
 
-            new_data = concatenate_block_managers(mgrs_indexers,
-                                                  self.new_axes,
-                                                  concat_axis=self.axis,
-                                                  copy=self.copy)
+            new_data = concatenate_block_managers(
+                mgrs_indexers, self.new_axes, concat_axis=self.axis,
+                copy=self.copy)
             if not self.copy:
                 new_data._consolidate_inplace()
 
diff --git a/pandas/types/concat.py b/pandas/types/concat.py
--- a/pandas/types/concat.py
+++ b/pandas/types/concat.py
@@ -5,7 +5,6 @@
 import numpy as np
 import pandas.tslib as tslib
 from pandas import compat
-from pandas.compat import map
 from pandas.core.algorithms import take_1d
 from .common import (is_categorical_dtype,
                      is_sparse,
@@ -133,19 +132,21 @@ def is_nonempty(x):
 
     typs = get_dtype_kinds(to_concat)
 
-    # these are mandated to handle empties as well
     _contains_datetime = any(typ.startswith('datetime') for typ in typs)
     _contains_period = any(typ.startswith('period') for typ in typs)
 
-    if _contains_datetime or 'timedelta' in typs or _contains_period:
+    if 'category' in typs:
+        # this must be priort to _concat_datetime,
+        # to support Categorical + datetime-like
+        return _concat_categorical(to_concat, axis=axis)
+
+    elif _contains_datetime or 'timedelta' in typs or _contains_period:
         return _concat_datetime(to_concat, axis=axis, typs=typs)
 
+    # these are mandated to handle empties as well
     elif 'sparse' in typs:
         return _concat_sparse(to_concat, axis=axis, typs=typs)
 
-    elif 'category' in typs:
-        return _concat_categorical(to_concat, axis=axis)
-
     if not nonempty:
         # we have all empties, but may need to coerce the result dtype to
         # object if we have non-numeric type operands (numpy would otherwise
@@ -181,18 +182,14 @@ def _concat_categorical(to_concat, axis=0):
         A single array, preserving the combined dtypes
     """
 
-    from pandas.core.categorical import Categorical
-
-    def convert_categorical(x):
-        # coerce to object dtype
-        if is_categorical_dtype(x.dtype):
-            return x.get_values()
-        return x.ravel()
-
-    if get_dtype_kinds(to_concat) - set(['object', 'category']):
-        # convert to object type and perform a regular concat
-        return _concat_compat([np.array(x, copy=False, dtype=object)
-                               for x in to_concat], axis=0)
+    def _concat_asobject(to_concat):
+        to_concat = [x.get_values() if is_categorical_dtype(x.dtype)
+                     else x.ravel() for x in to_concat]
+        res = _concat_compat(to_concat)
+        if axis == 1:
+            return res.reshape(1, len(res))
+        else:
+            return res
 
     # we could have object blocks and categoricals here
     # if we only have a single categoricals then combine everything
@@ -200,25 +197,15 @@ def convert_categorical(x):
     categoricals = [x for x in to_concat if is_categorical_dtype(x.dtype)]
 
     # validate the categories
-    categories = categoricals[0]
-    rawcats = categories.categories
-    for x in categoricals[1:]:
-        if not categories.is_dtype_equal(x):
-            raise ValueError("incompatible categories in categorical concat")
-
-    # we've already checked that all categoricals are the same, so if their
-    # length is equal to the input then we have all the same categories
-    if len(categoricals) == len(to_concat):
-        # concating numeric types is much faster than concating object types
-        # and fastpath takes a shorter path through the constructor
-        return Categorical(np.concatenate([x.codes for x in to_concat],
-                                          axis=0),
-                           rawcats, ordered=categoricals[0].ordered,
-                           fastpath=True)
+    if len(categoricals) != len(to_concat):
+        pass
     else:
-        concatted = np.concatenate(list(map(convert_categorical, to_concat)),
-                                   axis=0)
-        return Categorical(concatted, rawcats)
+        # when all categories are identical
+        first = to_concat[0]
+        if all(first.is_dtype_equal(other) for other in to_concat[1:]):
+            return union_categoricals(categoricals)
+
+    return _concat_asobject(to_concat)
 
 
 def union_categoricals(to_union, sort_categories=False):


Test patch:
diff --git a/pandas/tests/series/test_combine_concat.py b/pandas/tests/series/test_combine_concat.py
--- a/pandas/tests/series/test_combine_concat.py
+++ b/pandas/tests/series/test_combine_concat.py
@@ -185,9 +185,9 @@ def test_concat_empty_series_dtypes(self):
                          'category')
         self.assertEqual(pd.concat([Series(dtype='category'),
                                     Series(dtype='float64')]).dtype,
-                         np.object_)
+                         'float64')
         self.assertEqual(pd.concat([Series(dtype='category'),
-                                    Series(dtype='object')]).dtype, 'category')
+                                    Series(dtype='object')]).dtype, 'object')
 
         # sparse
         result = pd.concat([Series(dtype='float64').to_sparse(), Series(
diff --git a/pandas/tests/test_categorical.py b/pandas/tests/test_categorical.py
--- a/pandas/tests/test_categorical.py
+++ b/pandas/tests/test_categorical.py
@@ -2088,8 +2088,8 @@ def test_series_functions_no_warnings(self):
 
     def test_assignment_to_dataframe(self):
         # assignment
-        df = DataFrame({'value': np.array(
-            np.random.randint(0, 10000, 100), dtype='int32')})
+        df = DataFrame({'value': np.array(np.random.randint(0, 10000, 100),
+                                          dtype='int32')})
         labels = ["{0} - {1}".format(i, i + 499) for i in range(0, 10000, 500)]
 
         df = df.sort_values(by=['value'], ascending=True)
@@ -3355,16 +3355,15 @@ def test_slicing_and_getting_ops(self):
     def test_slicing_doc_examples(self):
 
         # GH 7918
-        cats = Categorical(
-            ["a", "b", "b", "b", "c", "c", "c"], categories=["a", "b", "c"])
+        cats = Categorical(["a", "b", "b", "b", "c", "c", "c"],
+                           categories=["a", "b", "c"])
         idx = Index(["h", "i", "j", "k", "l", "m", "n", ])
         values = [1, 2, 2, 2, 3, 4, 5]
         df = DataFrame({"cats": cats, "values": values}, index=idx)
 
         result = df.iloc[2:4, :]
         expected = DataFrame(
-            {"cats": Categorical(
-                ['b', 'b'], categories=['a', 'b', 'c']),
+            {"cats": Categorical(['b', 'b'], categories=['a', 'b', 'c']),
              "values": [2, 2]}, index=['j', 'k'])
         tm.assert_frame_equal(result, expected)
 
@@ -3379,10 +3378,9 @@ def test_slicing_doc_examples(self):
         tm.assert_series_equal(result, expected)
 
         result = df.ix["h":"j", 0:1]
-        expected = DataFrame({'cats': Series(
-            Categorical(
-                ['a', 'b', 'b'], categories=['a', 'b', 'c']), index=['h', 'i',
-                                                                     'j'])})
+        expected = DataFrame({'cats': Categorical(['a', 'b', 'b'],
+                                                  categories=['a', 'b', 'c'])},
+                             index=['h', 'i', 'j'])
         tm.assert_frame_equal(result, expected)
 
     def test_assigning_ops(self):
@@ -3636,8 +3634,8 @@ def f():
         with tm.assertRaises(ValueError):
             # different values
             df = orig.copy()
-            df.ix["j":"k", 0] = pd.Categorical(
-                ["c", "c"], categories=["a", "b", "c"])
+            df.ix["j":"k", 0] = pd.Categorical(["c", "c"],
+                                               categories=["a", "b", "c"])
 
         # assign a part of a column with dtype != categorical ->
         # exp_parts_cats_col
@@ -3674,8 +3672,8 @@ def f():
         self.assertRaises(ValueError, f)
 
         # fancy indexing
-        catsf = pd.Categorical(
-            ["a", "a", "c", "c", "a", "a", "a"], categories=["a", "b", "c"])
+        catsf = pd.Categorical(["a", "a", "c", "c", "a", "a", "a"],
+                               categories=["a", "b", "c"])
         idxf = pd.Index(["h", "i", "j", "k", "l", "m", "n"])
         valuesf = [1, 1, 3, 3, 1, 1, 1]
         df = pd.DataFrame({"cats": catsf, "values": valuesf}, index=idxf)
@@ -3733,9 +3731,8 @@ def f():
         s = orig.copy()
         s.index = ["x", "y"]
         s["y"] = "a"
-        exp = Series(
-            pd.Categorical(["b", "a"],
-                           categories=["a", "b"]), index=["x", "y"])
+        exp = Series(pd.Categorical(["b", "a"], categories=["a", "b"]),
+                     index=["x", "y"])
         tm.assert_series_equal(s, exp)
 
         # ensure that one can set something to np.nan
@@ -3887,7 +3884,7 @@ def test_cat_equality(self):
         self.assertRaises(TypeError, lambda: a > b)
         self.assertRaises(TypeError, lambda: b > a)
 
-    def test_concat(self):
+    def test_concat_append(self):
         cat = pd.Categorical(["a", "b"], categories=["a", "b"])
         vals = [1, 2]
         df = pd.DataFrame({"cats": cat, "vals": vals})
@@ -3896,20 +3893,22 @@ def test_concat(self):
         exp = pd.DataFrame({"cats": cat2,
                             "vals": vals2}, index=pd.Index([0, 1, 0, 1]))
 
-        res = pd.concat([df, df])
-        tm.assert_frame_equal(exp, res)
+        tm.assert_frame_equal(pd.concat([df, df]), exp)
+        tm.assert_frame_equal(df.append(df), exp)
 
-        # Concat should raise if the two categoricals do not have the same
-        # categories
+        # GH 13524 can concat different categories
         cat3 = pd.Categorical(["a", "b"], categories=["a", "b", "c"])
         vals3 = [1, 2]
-        df_wrong_categories = pd.DataFrame({"cats": cat3, "vals": vals3})
+        df_different_categories = pd.DataFrame({"cats": cat3, "vals": vals3})
 
-        def f():
-            pd.concat([df, df_wrong_categories])
+        res = pd.concat([df, df_different_categories], ignore_index=True)
+        exp = pd.DataFrame({"cats": list('abab'), "vals": [1, 2, 1, 2]})
+        tm.assert_frame_equal(res, exp)
 
-        self.assertRaises(ValueError, f)
+        res = df.append(df_different_categories, ignore_index=True)
+        tm.assert_frame_equal(res, exp)
 
+    def test_concat_append_gh7864(self):
         # GH 7864
         # make sure ordering is preserverd
         df = pd.DataFrame({"id": [1, 2, 3, 4, 5, 6],
@@ -3926,41 +3925,44 @@ def f():
                                 df2['grade'].cat.categories)
 
         dfx = pd.concat([df1, df2])
-        dfx['grade'].cat.categories
         self.assert_index_equal(df['grade'].cat.categories,
                                 dfx['grade'].cat.categories)
 
+        dfa = df1.append(df2)
+        self.assert_index_equal(df['grade'].cat.categories,
+                                dfa['grade'].cat.categories)
+
+
     def test_concat_preserve(self):
 
-        # GH 8641
-        # series concat not preserving category dtype
+        # GH 8641  series concat not preserving category dtype
+        # GH 13524 can concat different categories
         s = Series(list('abc'), dtype='category')
         s2 = Series(list('abd'), dtype='category')
 
-        def f():
-            pd.concat([s, s2])
-
-        self.assertRaises(ValueError, f)
+        exp = Series(list('abcabd'))
+        res = pd.concat([s, s2], ignore_index=True)
+        tm.assert_series_equal(res, exp)
 
-        result = pd.concat([s, s], ignore_index=True)
-        expected = Series(list('abcabc')).astype('category')
-        tm.assert_series_equal(result, expected)
+        exp = Series(list('abcabc'), dtype='category')
+        res = pd.concat([s, s], ignore_index=True)
+        tm.assert_series_equal(res, exp)
 
-        result = pd.concat([s, s])
-        expected = Series(
-            list('abcabc'), index=[0, 1, 2, 0, 1, 2]).astype('category')
-        tm.assert_series_equal(result, expected)
+        exp = Series(list('abcabc'), index=[0, 1, 2, 0, 1, 2],
+                     dtype='category')
+        res = pd.concat([s, s])
+        tm.assert_series_equal(res, exp)
 
         a = Series(np.arange(6, dtype='int64'))
         b = Series(list('aabbca'))
 
         df2 = DataFrame({'A': a,
                          'B': b.astype('category', categories=list('cab'))})
-        result = pd.concat([df2, df2])
-        expected = DataFrame({'A': pd.concat([a, a]),
-                              'B': pd.concat([b, b]).astype(
-                                  'category', categories=list('cab'))})
-        tm.assert_frame_equal(result, expected)
+        res = pd.concat([df2, df2])
+        exp = DataFrame({'A': pd.concat([a, a]),
+                         'B': pd.concat([b, b]).astype(
+                                'category', categories=list('cab'))})
+        tm.assert_frame_equal(res, exp)
 
     def test_categorical_index_preserver(self):
 
@@ -3968,44 +3970,21 @@ def test_categorical_index_preserver(self):
         b = Series(list('aabbca'))
 
         df2 = DataFrame({'A': a,
-                         'B': b.astype('category', categories=list(
-                             'cab'))}).set_index('B')
+                         'B': b.astype('category', categories=list('cab'))
+                        }).set_index('B')
         result = pd.concat([df2, df2])
         expected = DataFrame({'A': pd.concat([a, a]),
                               'B': pd.concat([b, b]).astype(
-                                  'category', categories=list(
-                                      'cab'))}).set_index('B')
+                                  'category', categories=list('cab'))
+                             }).set_index('B')
         tm.assert_frame_equal(result, expected)
 
         # wrong catgories
         df3 = DataFrame({'A': a,
-                         'B': b.astype('category', categories=list(
-                             'abc'))}).set_index('B')
+                         'B': pd.Categorical(b, categories=list('abc'))
+                        }).set_index('B')
         self.assertRaises(TypeError, lambda: pd.concat([df2, df3]))
 
-    def test_append(self):
-        cat = pd.Categorical(["a", "b"], categories=["a", "b"])
-        vals = [1, 2]
-        df = pd.DataFrame({"cats": cat, "vals": vals})
-        cat2 = pd.Categorical(["a", "b", "a", "b"], categories=["a", "b"])
-        vals2 = [1, 2, 1, 2]
-        exp = pd.DataFrame({"cats": cat2,
-                            "vals": vals2}, index=pd.Index([0, 1, 0, 1]))
-
-        res = df.append(df)
-        tm.assert_frame_equal(exp, res)
-
-        # Concat should raise if the two categoricals do not have the same
-        # categories
-        cat3 = pd.Categorical(["a", "b"], categories=["a", "b", "c"])
-        vals3 = [1, 2]
-        df_wrong_categories = pd.DataFrame({"cats": cat3, "vals": vals3})
-
-        def f():
-            df.append(df_wrong_categories)
-
-        self.assertRaises(ValueError, f)
-
     def test_merge(self):
         # GH 9426
 
@@ -4470,27 +4449,22 @@ def test_dt_accessor_api_for_categorical(self):
 
     def test_concat_categorical(self):
         # See GH 10177
-        df1 = pd.DataFrame(
-            np.arange(18, dtype='int64').reshape(6,
-                                                 3), columns=["a", "b", "c"])
-
-        df2 = pd.DataFrame(
-            np.arange(14, dtype='int64').reshape(7, 2), columns=["a", "c"])
-        df2['h'] = pd.Series(pd.Categorical(["one", "one", "two", "one", "two",
-                                             "two", "one"]))
-
-        df_concat = pd.concat((df1, df2), axis=0).reset_index(drop=True)
-
-        df_expected = pd.DataFrame(
-            {'a': [0, 3, 6, 9, 12, 15, 0, 2, 4, 6, 8, 10, 12],
-             'b': [1, 4, 7, 10, 13, 16, np.nan, np.nan, np.nan, np.nan, np.nan,
-                   np.nan, np.nan],
-             'c': [2, 5, 8, 11, 14, 17, 1, 3, 5, 7, 9, 11, 13]})
-        df_expected['h'] = pd.Series(pd.Categorical(
-            [None, None, None, None, None, None, "one", "one", "two", "one",
-             "two", "two", "one"]))
-
-        tm.assert_frame_equal(df_expected, df_concat)
+        df1 = pd.DataFrame(np.arange(18, dtype='int64').reshape(6, 3),
+                           columns=["a", "b", "c"])
+
+        df2 = pd.DataFrame(np.arange(14, dtype='int64').reshape(7, 2),
+                           columns=["a", "c"])
+
+        cat_values = ["one", "one", "two", "one", "two", "two", "one"]
+        df2['h'] = pd.Series(pd.Categorical(cat_values))
+
+        res = pd.concat((df1, df2), axis=0, ignore_index=True)
+        exp = pd.DataFrame({'a': [0, 3, 6, 9, 12, 15, 0, 2, 4, 6, 8, 10, 12],
+                            'b': [1, 4, 7, 10, 13, 16, np.nan, np.nan,
+                                  np.nan, np.nan, np.nan, np.nan, np.nan],
+                            'c': [2, 5, 8, 11, 14, 17, 1, 3, 5, 7, 9, 11, 13],
+                            'h': [None] * 6 + cat_values})
+        tm.assert_frame_equal(res, exp)
 
 
 class TestCategoricalSubclassing(tm.TestCase):
diff --git a/pandas/tools/tests/test_concat.py b/pandas/tools/tests/test_concat.py
--- a/pandas/tools/tests/test_concat.py
+++ b/pandas/tools/tests/test_concat.py
@@ -450,6 +450,254 @@ def test_concatlike_common_period_mixed_dt_to_object(self):
         res = pd.concat([tds, ps1])
         tm.assert_series_equal(res, pd.Series(exp, index=[0, 1, 0, 1]))
 
+    def test_concat_categorical(self):
+        # GH 13524
+
+        # same categories -> category
+        s1 = pd.Series([1, 2, np.nan], dtype='category')
+        s2 = pd.Series([2, 1, 2], dtype='category')
+
+        exp = pd.Series([1, 2, np.nan, 2, 1, 2], dtype='category')
+        tm.assert_series_equal(pd.concat([s1, s2], ignore_index=True), exp)
+        tm.assert_series_equal(s1.append(s2, ignore_index=True), exp)
+
+        # partially different categories => not-category
+        s1 = pd.Series([3, 2], dtype='category')
+        s2 = pd.Series([2, 1], dtype='category')
+
+        exp = pd.Series([3, 2, 2, 1])
+        tm.assert_series_equal(pd.concat([s1, s2], ignore_index=True), exp)
+        tm.assert_series_equal(s1.append(s2, ignore_index=True), exp)
+
+        # completelly different categories (same dtype) => not-category
+        s1 = pd.Series([10, 11, np.nan], dtype='category')
+        s2 = pd.Series([np.nan, 1, 3, 2], dtype='category')
+
+        exp = pd.Series([10, 11, np.nan, np.nan, 1, 3, 2])
+        tm.assert_series_equal(pd.concat([s1, s2], ignore_index=True), exp)
+        tm.assert_series_equal(s1.append(s2, ignore_index=True), exp)
+
+    def test_concat_categorical_coercion(self):
+        # GH 13524
+
+        # category + not-category => not-category
+        s1 = pd.Series([1, 2, np.nan], dtype='category')
+        s2 = pd.Series([2, 1, 2])
+
+        exp = pd.Series([1, 2, np.nan, 2, 1, 2])
+        tm.assert_series_equal(pd.concat([s1, s2], ignore_index=True), exp)
+        tm.assert_series_equal(s1.append(s2, ignore_index=True), exp)
+
+        # result shouldn't be affected by 1st elem dtype
+        exp = pd.Series([2, 1, 2, 1, 2, np.nan])
+        tm.assert_series_equal(pd.concat([s2, s1], ignore_index=True), exp)
+        tm.assert_series_equal(s2.append(s1, ignore_index=True), exp)
+
+        # all values are not in category => not-category
+        s1 = pd.Series([3, 2], dtype='category')
+        s2 = pd.Series([2, 1])
+
+        exp = pd.Series([3, 2, 2, 1])
+        tm.assert_series_equal(pd.concat([s1, s2], ignore_index=True), exp)
+        tm.assert_series_equal(s1.append(s2, ignore_index=True), exp)
+
+        exp = pd.Series([2, 1, 3, 2])
+        tm.assert_series_equal(pd.concat([s2, s1], ignore_index=True), exp)
+        tm.assert_series_equal(s2.append(s1, ignore_index=True), exp)
+
+        # completelly different categories => not-category
+        s1 = pd.Series([10, 11, np.nan], dtype='category')
+        s2 = pd.Series([1, 3, 2])
+
+        exp = pd.Series([10, 11, np.nan, 1, 3, 2])
+        tm.assert_series_equal(pd.concat([s1, s2], ignore_index=True), exp)
+        tm.assert_series_equal(s1.append(s2, ignore_index=True), exp)
+
+        exp = pd.Series([1, 3, 2, 10, 11, np.nan])
+        tm.assert_series_equal(pd.concat([s2, s1], ignore_index=True), exp)
+        tm.assert_series_equal(s2.append(s1, ignore_index=True), exp)
+
+        # different dtype => not-category
+        s1 = pd.Series([10, 11, np.nan], dtype='category')
+        s2 = pd.Series(['a', 'b', 'c'])
+
+        exp = pd.Series([10, 11, np.nan, 'a', 'b', 'c'])
+        tm.assert_series_equal(pd.concat([s1, s2], ignore_index=True), exp)
+        tm.assert_series_equal(s1.append(s2, ignore_index=True), exp)
+
+        exp = pd.Series(['a', 'b', 'c', 10, 11, np.nan])
+        tm.assert_series_equal(pd.concat([s2, s1], ignore_index=True), exp)
+        tm.assert_series_equal(s2.append(s1, ignore_index=True), exp)
+
+        # if normal series only contains NaN-likes => not-category
+        s1 = pd.Series([10, 11], dtype='category')
+        s2 = pd.Series([np.nan, np.nan, np.nan])
+
+        exp = pd.Series([10, 11, np.nan, np.nan, np.nan])
+        tm.assert_series_equal(pd.concat([s1, s2], ignore_index=True), exp)
+        tm.assert_series_equal(s1.append(s2, ignore_index=True), exp)
+
+        exp = pd.Series([np.nan, np.nan, np.nan, 10, 11])
+        tm.assert_series_equal(pd.concat([s2, s1], ignore_index=True), exp)
+        tm.assert_series_equal(s2.append(s1, ignore_index=True), exp)
+
+    def test_concat_categorical_3elem_coercion(self):
+        # GH 13524
+
+        # mixed dtypes => not-category
+        s1 = pd.Series([1, 2, np.nan], dtype='category')
+        s2 = pd.Series([2, 1, 2], dtype='category')
+        s3 = pd.Series([1, 2, 1, 2, np.nan])
+
+        exp = pd.Series([1, 2, np.nan, 2, 1, 2, 1, 2, 1, 2, np.nan])
+        tm.assert_series_equal(pd.concat([s1, s2, s3], ignore_index=True), exp)
+        tm.assert_series_equal(s1.append([s2, s3], ignore_index=True), exp)
+
+        exp = pd.Series([1, 2, 1, 2, np.nan, 1, 2, np.nan, 2, 1, 2])
+        tm.assert_series_equal(pd.concat([s3, s1, s2], ignore_index=True), exp)
+        tm.assert_series_equal(s3.append([s1, s2], ignore_index=True), exp)
+
+        # values are all in either category => not-category
+        s1 = pd.Series([4, 5, 6], dtype='category')
+        s2 = pd.Series([1, 2, 3], dtype='category')
+        s3 = pd.Series([1, 3, 4])
+
+        exp = pd.Series([4, 5, 6, 1, 2, 3, 1, 3, 4])
+        tm.assert_series_equal(pd.concat([s1, s2, s3], ignore_index=True), exp)
+        tm.assert_series_equal(s1.append([s2, s3], ignore_index=True), exp)
+
+        exp = pd.Series([1, 3, 4, 4, 5, 6, 1, 2, 3])
+        tm.assert_series_equal(pd.concat([s3, s1, s2], ignore_index=True), exp)
+        tm.assert_series_equal(s3.append([s1, s2], ignore_index=True), exp)
+
+        # values are all in either category => not-category
+        s1 = pd.Series([4, 5, 6], dtype='category')
+        s2 = pd.Series([1, 2, 3], dtype='category')
+        s3 = pd.Series([10, 11, 12])
+
+        exp = pd.Series([4, 5, 6, 1, 2, 3, 10, 11, 12])
+        tm.assert_series_equal(pd.concat([s1, s2, s3], ignore_index=True), exp)
+        tm.assert_series_equal(s1.append([s2, s3], ignore_index=True), exp)
+
+        exp = pd.Series([10, 11, 12, 4, 5, 6, 1, 2, 3])
+        tm.assert_series_equal(pd.concat([s3, s1, s2], ignore_index=True), exp)
+        tm.assert_series_equal(s3.append([s1, s2], ignore_index=True), exp)
+
+    def test_concat_categorical_multi_coercion(self):
+        # GH 13524
+
+        s1 = pd.Series([1, 3], dtype='category')
+        s2 = pd.Series([3, 4], dtype='category')
+        s3 = pd.Series([2, 3])
+        s4 = pd.Series([2, 2], dtype='category')
+        s5 = pd.Series([1, np.nan])
+        s6 = pd.Series([1, 3, 2], dtype='category')
+
+        # mixed dtype, values are all in categories => not-category
+        exp = pd.Series([1, 3, 3, 4, 2, 3, 2, 2, 1, np.nan, 1, 3, 2])
+        res = pd.concat([s1, s2, s3, s4, s5, s6], ignore_index=True)
+        tm.assert_series_equal(res, exp)
+        res = s1.append([s2, s3, s4, s5, s6], ignore_index=True)
+        tm.assert_series_equal(res, exp)
+
+        exp = pd.Series([1, 3, 2, 1, np.nan, 2, 2, 2, 3, 3, 4, 1, 3])
+        res = pd.concat([s6, s5, s4, s3, s2, s1], ignore_index=True)
+        tm.assert_series_equal(res, exp)
+        res = s6.append([s5, s4, s3, s2, s1], ignore_index=True)
+        tm.assert_series_equal(res, exp)
+
+    def test_concat_categorical_ordered(self):
+        # GH 13524
+
+        s1 = pd.Series(pd.Categorical([1, 2, np.nan], ordered=True))
+        s2 = pd.Series(pd.Categorical([2, 1, 2], ordered=True))
+
+        exp = pd.Series(pd.Categorical([1, 2, np.nan, 2, 1, 2], ordered=True))
+        tm.assert_series_equal(pd.concat([s1, s2], ignore_index=True), exp)
+        tm.assert_series_equal(s1.append(s2, ignore_index=True), exp)
+
+        exp = pd.Series(pd.Categorical([1, 2, np.nan, 2, 1, 2, 1, 2, np.nan],
+                                       ordered=True))
+        tm.assert_series_equal(pd.concat([s1, s2, s1], ignore_index=True), exp)
+        tm.assert_series_equal(s1.append([s2, s1], ignore_index=True), exp)
+
+    def test_concat_categorical_coercion_nan(self):
+        # GH 13524
+
+        # some edge cases
+        # category + not-category => not category
+        s1 = pd.Series(np.array([np.nan, np.nan], dtype=np.float64),
+                       dtype='category')
+        s2 = pd.Series([np.nan, 1])
+
+        exp = pd.Series([np.nan, np.nan, np.nan, 1])
+        tm.assert_series_equal(pd.concat([s1, s2], ignore_index=True), exp)
+        tm.assert_series_equal(s1.append(s2, ignore_index=True), exp)
+
+        s1 = pd.Series([1, np.nan], dtype='category')
+        s2 = pd.Series([np.nan, np.nan])
+
+        exp = pd.Series([1, np.nan, np.nan, np.nan])
+        tm.assert_series_equal(pd.concat([s1, s2], ignore_index=True), exp)
+        tm.assert_series_equal(s1.append(s2, ignore_index=True), exp)
+
+        # mixed dtype, all nan-likes => not-category
+        s1 = pd.Series([np.nan, np.nan], dtype='category')
+        s2 = pd.Series([np.nan, np.nan])
+
+        exp = pd.Series([np.nan, np.nan, np.nan, np.nan], dtype=object)
+        tm.assert_series_equal(pd.concat([s1, s2], ignore_index=True), exp)
+        tm.assert_series_equal(s1.append(s2, ignore_index=True), exp)
+        tm.assert_series_equal(pd.concat([s2, s1], ignore_index=True), exp)
+        tm.assert_series_equal(s2.append(s1, ignore_index=True), exp)
+
+        # all category nan-likes => category
+        s1 = pd.Series([np.nan, np.nan], dtype='category')
+        s2 = pd.Series([np.nan, np.nan], dtype='category')
+
+        exp = pd.Series([np.nan, np.nan, np.nan, np.nan], dtype='category')
+
+        tm.assert_series_equal(pd.concat([s1, s2], ignore_index=True), exp)
+        tm.assert_series_equal(s1.append(s2, ignore_index=True), exp)
+
+    def test_concat_categorical_empty(self):
+        # GH 13524
+
+        s1 = pd.Series([], dtype='category')
+        s2 = pd.Series([1, 2], dtype='category')
+
+        tm.assert_series_equal(pd.concat([s1, s2], ignore_index=True), s2)
+        tm.assert_series_equal(s1.append(s2, ignore_index=True), s2)
+
+        tm.assert_series_equal(pd.concat([s2, s1], ignore_index=True), s2)
+        tm.assert_series_equal(s2.append(s1, ignore_index=True), s2)
+
+        s1 = pd.Series([], dtype='category')
+        s2 = pd.Series([], dtype='category')
+
+        tm.assert_series_equal(pd.concat([s1, s2], ignore_index=True), s2)
+        tm.assert_series_equal(s1.append(s2, ignore_index=True), s2)
+
+        s1 = pd.Series([], dtype='category')
+        s2 = pd.Series([])
+
+        # different dtype => not-category
+        tm.assert_series_equal(pd.concat([s1, s2], ignore_index=True), s2)
+        tm.assert_series_equal(s1.append(s2, ignore_index=True), s2)
+        tm.assert_series_equal(pd.concat([s2, s1], ignore_index=True), s2)
+        tm.assert_series_equal(s2.append(s1, ignore_index=True), s2)
+
+        s1 = pd.Series([], dtype='category')
+        s2 = pd.Series([np.nan, np.nan])
+
+        # empty Series is ignored
+        exp = pd.Series([np.nan, np.nan])
+        tm.assert_series_equal(pd.concat([s1, s2], ignore_index=True), exp)
+        tm.assert_series_equal(s1.append(s2, ignore_index=True), exp)
+
+        tm.assert_series_equal(pd.concat([s2, s1], ignore_index=True), exp)
+        tm.assert_series_equal(s2.append(s1, ignore_index=True), exp)
+
 
 class TestAppend(ConcatenateBase):