ENH: concat and append now can handle unordered categories
State: closed

- [x] closes #13524
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry

**on current master:**

```
# different categoricals
pd.concat([pd.Series([1, 2], dtype='category'),
           pd.Series([3, 4], dtype='category')])
# ValueError: incompatible categories in categorical concat

# categorical + normal (values are contained in categories) -> object dtype
pd.concat([pd.Series([1, 2], dtype='category'),
           pd.Series([1, 2])])
#0    1
#1    2
#0    1
#1    2
# dtype: object

# categorical + normal (values are not contained in categories) -> object dtype
pd.concat([pd.Series([1, 2], dtype='category'),
           pd.Series([3, 4])])
#0    1
#1    2
#0    3
#1    4
# dtype: object
```

**this PR (updated according to the discussion):**

```
# different categoricals
pd.concat([pd.Series([1, 2], dtype='category'),
           pd.Series([3, 4], dtype='category')])
#0    1
#1    2
#0    3
#1    4
# dtype: int64

# specifying union_categoricals keeps category if possible
pd.concat([pd.Series([1, 2], dtype='category'),
           pd.Series([3, 4], dtype='category')], union_categoricals=True)
#0    1
#1    2
#0    3
#1    4
# dtype: category
Categories (4, int64): [1, 2, 3, 4]

# categorical + normal (values are contained in categories) -> category dtype
pd.concat([pd.Series([1, 2], dtype='category'),
           pd.Series([1, 2])])
#0    1
#1    2
#0    1
#1    2
# dtype: int64

# categorical + normal (values are not contained in categories) -> int dtype (keep original dtype)
pd.concat([pd.Series([1, 2], dtype='category'),
           pd.Series([3, 4])])
#0    1
#1    2
#0    3
#1    4
# dtype: int64
```

CC: @janschulz, @chris-b1


Labels:
name: Enhancement
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Categorical
description: Categorical Data Type

Comments:
## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/13767?src=pr) is 85.25% (diff: 100%)

> Merging [#13767](https://codecov.io/gh/pydata/pandas/pull/13767?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will decrease coverage by **<.01%**

``` diff
@@             master     #13767   diff @@
==========================================
  Files           139        139          
  Lines         50496      50491     -5   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
- Hits          43049      43044     -5   
  Misses         7447       7447          
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [8023029...96a372e](https://codecov.io/gh/pydata/pandas/compare/80230292e19e377f5b11409e112500b6a180e18a...96a372e09005c29eba935b9d1778537cbfae50c9?src=pr)

What is the actual "problem" or use case here?

I'm opposed to this PR: IMO `concat()` should not change the categories. In my thinking `(pd.Series([1, 2], dtype='category'), pd.Series([3, 4], dtype='category')` is as (un)similar as `([1,2], ["A", "B"])`. The latter might cast to a type which can take both (`object`) but not to a new `int_or_string` type...

So, I find the master variant more in line with that thinking. What I'm not sure is whether it should raise or (as it does now) convert to object. Other cat handling of two different cats (apart from the special `union_categoricals`) raises but pandas in general casts to object. As cat handling deviates in other cases from normal pandas casting rules, IMO it should error here, too.

I can see a logic for "categorical + iterable of the same values as in the categories = new categorical with the same categories as the first one" as that can be thought similar as `cat[x] = value`.

@janschulz thx for sharing your opinion. Can you illustrate your intention covering all possible patterns? (I copied mine from #13524).
- concat 2 categories -> use the rule of `union_categorical`
- concat category and other dtype (which values are all in the category, including empty) -> category
  - this rule is applied regardless of order (if there is at least one category in concatenating values)
  - the property like `ordered` should be preserved.
- concat category and other dtype (which values are not in the category) -> not category (dtype is infered)  

I come from using catagoricals as things like lickert scales: "Fully agree ... full disagree" and such things. From that standpoint, I usually read data in, convert to cat and replace the categories with the full set.  In that case the categories have a specific meaning (e.g the lickert scale or in other cases the 50 US states or such things which are fixed). 

From that comes the rule that you shouldn't be able to set a value in the cat column to something not in categories (adding "Germany" to a cat encoding the 50 US states makes no sense). And from that the "error if combining two _different_ categories" rule. So I'm fully happy that there isn't any "default" (i.e. anything in the `cat` accessor or in the default "non-categorical-specific" API like `concat` and similar functions) way to combine two **different** categories and trying to do it would error.

I see `union_categorical` as a way to use a categorical as a memory saving string type and as in the other issues where such additions/changes were proposed, I would love to see such a type instead of changing categorical to fit that needs.

The second usecase is  as an internal helper for someone using it with reading in csv files _before_ letting the user do some "meaningful" replacements.

Thanks for sharing the usecase. Based on this, what the "rules" on your mind are? can u list up possible rules like: `category + category (same category) = category` covering possible patterns to discuss the expected behavior.

IMO these are the rules:
- category + setting with values in the categories = fine
- category + setting with values not in the categories = error (not converting to object/common "normal" type)
- category + same category configuration ("order" +  categories OR "unordered" + same categories (the unordered case could actually be less strict and only test if the set of categories is the same)) = fine
- category + different category = error

I agree with @janschulz rules, except where `error` -> just coerce to the base type (of the categorical) if there is one, if multiple categoricals and different base types then must upcast (potentially to object).

The idea is that `concat/append` would always work, whether you add new categoricals or not, but the dtype will be preserved in a strict way. 

I suppose that we could further add an argument to `concat` itself that just calls `union_categoricals`, IOW allows a non-strict interpretation. maybe `union_categoricals=False` as the default

@jreback 

> > category + setting with values not in the categories = error (not converting to object/common "normal" type)
> 
> except where error -> just coerce to the base type (of the categorical) if there is one, if multiple categoricals and different base types then must upcast (potentially to object).

If you also want to change setting as well, then this is a whole different thing... IMO the above rules are what currently applies.

I'm still not convinced:  So what happens if I have a Categorical which is encoded as `complete agree ... completely disagree` (5 step lickert scale) and now I concat a column which is encoded as a categorical which encode the 50 US States. It will now work (both are string dtypes), but the meaning is completely screwed. Or imagine things like two categoricals which have one category in common (50 US states + 50 states capitals -> washington is in both...): will that mean that the final categorical have `x+y-1` categories? In my opinion, each categorical configuration is it's own dtype (is_dtype_equal in Categorical actually tests just that: https://github.com/pydata/pandas/blob/master/pandas/core/categorical.py#L1791-L1809), so upcasting to a combined one is the same as removing all meaning from the categorical configuration = upcasting to object.

What is the actual usecase here: internal csv parsing of different files to a single dataframe? If so, then in my eyes this looks (again :-)) like the "pd-String" usecase: you encode it directly to save memory and then you are have the problem to concat two dfs because you don't know the strings (=you have no beforehand knowledge about the expected categories). 

If so, then there are multiple different ways to solve this:
- add a `pd.String` dtype :-)
- add a convenience method like `pd.concat_nonstrict()` which would basically do a `union_categorical` on the category columns.
- add a `nonstrict_category_checks=False` to `concat/append` which would basically switch to `pd.concat_nonstrict()` behaviour above.

I agree that `concat` shouldn't fail, as users understand what they're doing. Considering all dtypes (not only category), coercing to the object dtype should be a basic rule (such as `DatetimeTZ` with different tz).

Based on this, I've organized the rules as below table.

| left | right | result |
| --- | --- | --- |
| category | category (identical categories and ordered) | category |
| category | category (different categories or ordered) | object (dtype is inferred) |
| category | not category | object (dtype is inferred) |

Also, adding following options to `concat` should cover all cases:
- `union_categorical` (default `False`): work as @jreback suggested. Apply `union_categorical` rule to concat `Categorical`. 
- `strict` (default `False`): raise when different dtype / different categories are being concatenated.

once updated based on the discussion. Note that following fixed are not included:
- `union_categoricals` with `Index.append` (being included in #13660) 
- `strict` option (should be after #13660)

cc @janschulz 
cc @chris-b1

can you have a look.

@wesm ? 

This pair of results was unexpected - it seems like either both should be `Categorical`, or both the category's dtype?  Although I'm not sure which.

``` python
In [6]: pd.__version__
Out[6]: '0.18.1+314.gc82629c'

In [4]: pd.concat([pd.Series([1,2], dtype='category'),
   ...:            pd.Series([2,1])], union_categoricals=True)
Out[4]: 
0    1
1    2
0    2
1    1
dtype: category
Categories (2, int64): [1, 2]

In [5]: pd.concat([pd.Series([1,2], dtype='category'),
   ...:            pd.Series([2,1, 3])], union_categoricals=True)
Out[5]: 
0    1
1    2
0    2
1    1
2    3
dtype: int64
```

@chris-b1 Agreed the above is confusing. The first example keeps category dtype as 2nd values are all in 1st categories.

Maybe the following rule is simpler? (category + non-category always results in non-category, regardless of `union_categoricals`)

| left | right | result | `union_categoricals=True` |
| --- | --- | --- | --- |
| category | category (identical categories) | category | category |
| category | category (different categories, both not ordered) | object (dtype is inferred) | category |
| category | category (different categories, either one is ordered) | object (dtype is inferred) | object (dtype is inferred) |
| category | not category | object (dtype is inferred) | object (dtype is inferred) |

@sinhrks I like that table better.

Yeah, that makes sense to me. 

Thanks for the feedback. Updated the impl based on the above table.

cc @mrocklin

```
In [7]: s1 = Series(pd.Categorical(list('baca'),categories=list('abc')))

In [8]: s2 = Series(pd.Categorical(list('baca'),categories=list('bac')))

In [9]: s1
Out[9]: 
0    b
1    a
2    c
3    a
dtype: category
Categories (3, object): [a, b, c]

In [10]: s2
Out[10]: 
0    b
1    a
2    c
3    a
dtype: category
Categories (3, object): [b, a, c]

In [11]: s1.cat.codes
Out[11]: 
0    1
1    0
2    2
3    0
dtype: int8

In [12]: s2.cat.codes
Out[12]: 
0    0
1    1
2    2
3    1
dtype: int8

In [13]: pd.concat([s1,s2],union_categoricals=True)
Out[13]: 
0    b
1    a
2    c
3    a
0    b
1    a
2    c
3    a
dtype: category
Categories (3, object): [a, b, c]

In [14]: pd.concat([s1,s2],union_categoricals=True).cat.codes
Out[14]: 
0    1
1    0
2    2
3    0
0    1
1    0
2    2
3    0
dtype: int8
```

Cool.  I'm very glad to see this.  Dask.dataframe can definitely make use
of this feature.

On Sat, Aug 6, 2016 at 12:01 PM, Jeff Reback notifications@github.com
wrote:

> cc @mrocklin https://github.com/mrocklin
> 
> In [7]: s1 = Series(pd.Categorical(list('baca'),categories=list('abc')))
> 
> In [8]: s2 = Series(pd.Categorical(list('baca'),categories=list('bac')))
> 
> In [9]: s1
> Out[9]:
> 0    b
> 1    a
> 2    c
> 3    a
> dtype: category
> Categories (3, object): [a, b, c]
> 
> In [10]: s2
> Out[10]:
> 0    b
> 1    a
> 2    c
> 3    a
> dtype: category
> Categories (3, object): [b, a, c]
> 
> In [11]: s1.cat.codes
> Out[11]:
> 0    1
> 1    0
> 2    2
> 3    0
> dtype: int8
> 
> In [12]: s2.cat.codes
> Out[12]:
> 0    0
> 1    1
> 2    2
> 3    1
> dtype: int8
> 
> In [13]: pd.concat([s1,s2],union_categoricals=True)
> Out[13]:
> 0    b
> 1    a
> 2    c
> 3    a
> 0    b
> 1    a
> 2    c
> 3    a
> dtype: category
> Categories (3, object): [a, b, c]
> 
> In [14]: pd.concat([s1,s2],union_categoricals=True).cat.codes
> Out[14]:
> 0    1
> 1    0
> 2    2
> 3    0
> 0    1
> 1    0
> 2    2
> 3    0
> dtype: int8
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> https://github.com/pydata/pandas/pull/13767#issuecomment-238030521, or mute
> the thread
> https://github.com/notifications/unsubscribe-auth/AASszIoHtnmUQ3HtH5vpWATcLOeo52HKks5qdK_xgaJpZM4JTey6
> .

lgtm. @jorisvandenbossche @TomAugspurger @shoyer comments?

👍 to the last table.

anyone want to summarize where we are on this PR and what if any issues remain for discussion?

Remaining discussion is only one (pls allow me if i missed something)
- What the `concat` keyword name should be (`union_categoricals`, or descriptive name like `less_strict_categorical_checks`)

Now I feel `less_strict...` may not be enough clear for users, as it doesn't say what the result be (less strictly keep category or coerces to object). But leave decision as I'm not a native:)

This seems okay to me. As an aside, having a type-specific argument in `pd.concat` isn't ideal, but I'm not sure what's better. You could be more explicit / general with something like

``` python
options = {
    'types': {
        'category': {
            'union': True
        }
    }
}
pd.concat(chunks, options=options)
```

Ok with the option specification as proposed by @wesm . We should create an issue to make this a bit more general, maybe using: [attrdict](https://github.com/bcj/AttrDict).

But can change to using this for now (as in reality its used only internally ATM).

or remove `union_categoricals` arg ATM. It gets less important because we've decided concatenating different categories coerces to object (users can re-categorize the object if they want).

you need for read_csv

ah, i meant `union categoricals` option in `concat`. the function itself is useful.

OK, we have to take a decision here if we want to include this one for 0.19. I tried to summarise below, but it has gotten a bit long :-) 
But **maybe the conclusion is just that this can be merged as is?** (or at least part 1)

---

### **1. dtype of return value of `concat` (in the default case, without specifying kwargs)**

I think most people agreed with the table above: https://github.com/pydata/pandas/pull/13767#issuecomment-237806603? 
This comes down to returning object dtype where you now got "ValueError: incompatible categories in categorical concat"

Additionally:
- concatting a categorical Series with a plain Series that only has values included in the categories no longer returns a Categorical Series. Eg with current PR:
  
  ```
  In [20]: s1 = pd.Series(['a', 'b'], dtype='category')
  
  In [21]: pd.concat([s1, s1]).dtype
  Out[21]: category
  
  In [22]: pd.concat([s1, s1.astype(object)]).dtype
  Out[22]: dtype('O')
  ```
  
  On current master this gives two times `'category'`. 
  @sinhrks If we keep this, this should actually be mentioned in the whatsnew docs I think.
- you do not get the clear error any more when concatting categoricals with different categories, but a silent downcast to object dtype (or the dtype of the categories). This can possibly give surprises further down the road when users wonder why they have no longer a category dtype. 

As @sinhrks said, we can also already include this part, and leave part 2 (below) for a later PR

### **2. addition of new keyword to `concat` to union categoricals with different categories**

Current PR proposes to add a `union_categoricals` keyword to `concat`, which lets you concatenate categoricals with different categories (like the `union_categoricals` function does). Default would be False.

There was some discussion about the name to use. @wesm suggested to use a dict to pass type specific options. But that is maybe something we should first explore further to see if there are other use cases for such a construct?

As I understand, this part of the PR is less pressing, as users can also use the `union_categoricals` function directly (although IMO it would be nice to not have to expose this function, but let people use `concat`)

1) +1
2) I think we _do_ need to do something, but ok to wait and add this later. 

Right now `union_categoricals` is a good soln, it is a semi-private API, but likely to be exposed to a developer API at some point. Since we are adding API, can just push this back a bit.

ok, let's take out `union_categoricals` arg to `concat` for now and put it in a separate PR, @sinhrks ?