BUG: Fix Categorical comparsion with Series of dtype 'category'
State: closed

This is a fix attempt for issue #16659.

 - [ ] closes #xxxx
 - [ ] tests added / passed
 - [ ] passes ``git diff upstream/master --name-only -- '*.py' | flake8 --diff``
 - [ ] whatsnew entry


Labels:
name: Bug
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Categorical
description: Categorical Data Type

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/16667?src=pr&el=h1) Report
> Merging [#16667](https://codecov.io/gh/pandas-dev/pandas/pull/16667?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/96a527434a6138b291c1b4a782bd2793cff51f74?src=pr&el=desc) will **decrease** coverage by `0.42%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/16667/graphs/tree.svg?width=650&src=pr&token=eZ4WkYLtcO&height=150)](https://codecov.io/gh/pandas-dev/pandas/pull/16667?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #16667      +/-   ##
==========================================
- Coverage   91.43%   91.01%   -0.43%     
==========================================
  Files         163      161       -2     
  Lines       50091    49353     -738     
==========================================
- Hits        45800    44917     -883     
- Misses       4291     4436     +145
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `88.78% <100%> (-0.44%)` | :arrow_down: |
| #single | `40.26% <66.66%> (-0.16%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/16667?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/16667?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY2F0ZWdvcmljYWwucHk=) | `95.48% <100%> (-0.32%)` | :arrow_down: |
| [pandas/io/s3.py](https://codecov.io/gh/pandas-dev/pandas/pull/16667?src=pr&el=tree#diff-cGFuZGFzL2lvL3MzLnB5) | `0% <0%> (-85%)` | :arrow_down: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/16667?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :arrow_down: |
| [pandas/tests/plotting/\_\_init\_\_.py](https://codecov.io/gh/pandas-dev/pandas/pull/16667?src=pr&el=tree#diff-cGFuZGFzL3Rlc3RzL3Bsb3R0aW5nL19faW5pdF9fLnB5) | `77.77% <0%> (-22.23%)` | :arrow_down: |
| [pandas/core/tools/datetimes.py](https://codecov.io/gh/pandas-dev/pandas/pull/16667?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvdG9vbHMvZGF0ZXRpbWVzLnB5) | `66.94% <0%> (-17.55%)` | :arrow_down: |
| [pandas/util/\_decorators.py](https://codecov.io/gh/pandas-dev/pandas/pull/16667?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvX2RlY29yYXRvcnMucHk=) | `65.3% <0%> (-15.4%)` | :arrow_down: |
| [pandas/compat/pickle\_compat.py](https://codecov.io/gh/pandas-dev/pandas/pull/16667?src=pr&el=tree#diff-cGFuZGFzL2NvbXBhdC9waWNrbGVfY29tcGF0LnB5) | `69.51% <0%> (-6.1%)` | :arrow_down: |
| [pandas/core/generic.py](https://codecov.io/gh/pandas-dev/pandas/pull/16667?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ2VuZXJpYy5weQ==) | `92% <0%> (-3.72%)` | :arrow_down: |
| [pandas/core/dtypes/missing.py](https://codecov.io/gh/pandas-dev/pandas/pull/16667?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZHR5cGVzL21pc3NpbmcucHk=) | `87.03% <0%> (-3.59%)` | :arrow_down: |
| [pandas/core/indexes/range.py](https://codecov.io/gh/pandas-dev/pandas/pull/16667?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9yYW5nZS5weQ==) | `92.81% <0%> (-2.85%)` | :arrow_down: |
| ... and [99 more](https://codecov.io/gh/pandas-dev/pandas/pull/16667?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/16667?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/16667?src=pr&el=footer). Last update [96a5274...2771e57](https://codecov.io/gh/pandas-dev/pandas/pull/16667?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

can you update
I was thinking that an agreement was yet to achieve.

So, I will try to fix the two bugs mentioned above in this PR altogether.
can you rebase / update

https://github.com/pandas-dev/pandas/pull/16667#discussion_r122086426 could be done as a separate PR prior to this (or here).
Sorry for late reply. 

So, by "rebase", do you mean that I can make a new branch from current topic branch and fix [#16667 (comment)](https://github.com/pandas-dev/pandas/pull/16667#discussion_r122086426) on that new branch and rebase its content to the master first? Something like,
```
git rebase --onto master current_topic_branch new_branch
``` 

Or just create a new separate PR to fix [#16667 (comment)](https://github.com/pandas-dev/pandas/pull/16667#discussion_r122086426), and have it merged into master first. Then rebase the topic branch in this PR after that commit (on master)?

Thanks for your guidance.
> So, by "rebase", do you mean that I can make a new branch from current topic branch and fix #16667 (comment) on that new branch and rebase its content to the master first? 

Rebase means the following:

1) Take your current branch and rewind your changes
2) Fast-forward the rewinded branch to the current state of `master`
3) Reapply your changes on the updated version of the branch

To do this, just to the following:
~~~
git rebase master
~~~

No new branch is needed, nor is there any need for a new PR
> Or just create a new separate PR to fix #16667 (comment), and have it merged into master first. Then rebase the topic branch in this PR after that commit (on master)?

So this is an option that @jreback presented to you in terms of addressing his comment.  You are also welcome to address the issue directly in this PR (so you wouldn't need to create a new one for it).
> Sorry for late reply.

No worries!  We've had to wait MUCH longer than this for PR's before :smile:
@gfyoung

Thanks for you detailed explanation. I was overthinking about the rebase.

So what I have done is as follows.

Update my upstream/master with
```
git fetch upstream
```

Switch to local master branch and update it
```
git checkout master
git merge upstream/master
```
Switch back to local branch fix_bug_in_categorical_comparison
```
git checkout fix_bug_in_categorical_comparison
```
Rebase it to fresh master branch
```
git rebase master
```
Finally, push to origin/fix_bug_in_categorical_comparison with force 
```
git push --force origin fix_bug_in_categorical_comparison
```
Since I am the only one working on this branch, I believe the force push here should not cause any trouble. Please let me know if this is not considered a good practice. Thank you.
@funnycrab : That should be just fine!
@gfyoung 
OK. Thank you. I will then focus on fixing the real issue.
can you rebase
Hello @funnycrab! Thanks for updating the PR.

 - In the file [`pandas/core/categorical.py`](https://github.com/pandas-dev/pandas/blob/0697610479a3ef88be1ffd28133c3b71417c3bcf/pandas/core/categorical.py), following are the PEP8 issues :

> [Line 2029:1](https://github.com/pandas-dev/pandas/blob/0697610479a3ef88be1ffd28133c3b71417c3bcf/pandas/core/categorical.py#L2029): [E112](https://duckduckgo.com/?q=pep8%20E112) expected an indented block
> [Line 2029:1](https://github.com/pandas-dev/pandas/blob/0697610479a3ef88be1ffd28133c3b71417c3bcf/pandas/core/categorical.py#L2029): [E305](https://duckduckgo.com/?q=pep8%20E305) expected 2 blank lines after class or function definition, found 0
> [Line 2029:3](https://github.com/pandas-dev/pandas/blob/0697610479a3ef88be1ffd28133c3b71417c3bcf/pandas/core/categorical.py#L2029): [E227](https://duckduckgo.com/?q=pep8%20E227) missing whitespace around bitwise or shift operator
> [Line 2029:7](https://github.com/pandas-dev/pandas/blob/0697610479a3ef88be1ffd28133c3b71417c3bcf/pandas/core/categorical.py#L2029): [E225](https://duckduckgo.com/?q=pep8%20E225) missing whitespace around operator
> [Line 2031:3](https://github.com/pandas-dev/pandas/blob/0697610479a3ef88be1ffd28133c3b71417c3bcf/pandas/core/categorical.py#L2031): [E225](https://duckduckgo.com/?q=pep8%20E225) missing whitespace around operator
> [Line 2031:5](https://github.com/pandas-dev/pandas/blob/0697610479a3ef88be1ffd28133c3b71417c3bcf/pandas/core/categorical.py#L2031): [E225](https://duckduckgo.com/?q=pep8%20E225) missing whitespace around operator
> [Line 2031:7](https://github.com/pandas-dev/pandas/blob/0697610479a3ef88be1ffd28133c3b71417c3bcf/pandas/core/categorical.py#L2031): [E225](https://duckduckgo.com/?q=pep8%20E225) missing whitespace around operator
> [Line 2032:13](https://github.com/pandas-dev/pandas/blob/0697610479a3ef88be1ffd28133c3b71417c3bcf/pandas/core/categorical.py#L2032): [E113](https://duckduckgo.com/?q=pep8%20E113) unexpected indentation
> [Line 2037:3](https://github.com/pandas-dev/pandas/blob/0697610479a3ef88be1ffd28133c3b71417c3bcf/pandas/core/categorical.py#L2037): [E227](https://duckduckgo.com/?q=pep8%20E227) missing whitespace around bitwise or shift operator
> [Line 2037:7](https://github.com/pandas-dev/pandas/blob/0697610479a3ef88be1ffd28133c3b71417c3bcf/pandas/core/categorical.py#L2037): [E225](https://duckduckgo.com/?q=pep8%20E225) missing whitespace around operator
> [Line 2038:13](https://github.com/pandas-dev/pandas/blob/0697610479a3ef88be1ffd28133c3b71417c3bcf/pandas/core/categorical.py#L2038): [E113](https://duckduckgo.com/?q=pep8%20E113) unexpected indentation
> [Line 2040:7](https://github.com/pandas-dev/pandas/blob/0697610479a3ef88be1ffd28133c3b71417c3bcf/pandas/core/categorical.py#L2040): [E225](https://duckduckgo.com/?q=pep8%20E225) missing whitespace around operator
> [Line 2041:3](https://github.com/pandas-dev/pandas/blob/0697610479a3ef88be1ffd28133c3b71417c3bcf/pandas/core/categorical.py#L2041): [E225](https://duckduckgo.com/?q=pep8%20E225) missing whitespace around operator
> [Line 2041:5](https://github.com/pandas-dev/pandas/blob/0697610479a3ef88be1ffd28133c3b71417c3bcf/pandas/core/categorical.py#L2041): [E225](https://duckduckgo.com/?q=pep8%20E225) missing whitespace around operator
> [Line 2041:7](https://github.com/pandas-dev/pandas/blob/0697610479a3ef88be1ffd28133c3b71417c3bcf/pandas/core/categorical.py#L2041): [E225](https://duckduckgo.com/?q=pep8%20E225) missing whitespace around operator
> [Line 2042:13](https://github.com/pandas-dev/pandas/blob/0697610479a3ef88be1ffd28133c3b71417c3bcf/pandas/core/categorical.py#L2042): [E113](https://duckduckgo.com/?q=pep8%20E113) unexpected indentation
> [Line 2044:7](https://github.com/pandas-dev/pandas/blob/0697610479a3ef88be1ffd28133c3b71417c3bcf/pandas/core/categorical.py#L2044): [E225](https://duckduckgo.com/?q=pep8%20E225) missing whitespace around operator
> [Line 2044:80](https://github.com/pandas-dev/pandas/blob/0697610479a3ef88be1ffd28133c3b71417c3bcf/pandas/core/categorical.py#L2044): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (114 > 79 characters)




@jreback 

Sorry for the late reply. During the rebasing, there are some conflicts in the pandas/core/categorical.py. I just rebased without really resolving the conflicts. So the commit 0697610479a3ef88be1ffd28133c3b71417c3bcf is a broken commit. I will try to fix the issue as soon as possible.
Apologies for the conflicts @funnycrab. Let me know if you need help cleaning them up.
@TomAugspurger 

It is OK. Actually, it is my bad for keeping this issue open for such a long time. Hopefully, I can fix this early October.

Will let you know when clarification is needed. Thanks in advance!
can you rebase / update
can you rebase
closing as stale, but if you want to continue working ping and we can reopen