PERF: avoid unnecessary recoding in CategoricalIndex._create_categorical
State: closed

- [x] progress towards #20395
- [x] xref #21369
- [ ] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

This issue was found when looking for a solution to #20395. I've found that ``CategoricalIndex._create_categorical`` makes an unnecessary call to ``Categorical._set_dtype`` when passed a dtype that is equal to self.dtype:

```python
>>> n = 100_000
>>> ci = pd.CategoricalIndex(list('a'*n + 'b'*n + 'c'*n))
>>> %timeit ci._create_categorical(ci, ci)
197 µs  # master and this PR
>>> %timeit ci._create_categorical(ci, ci, dtype=ci.dtype)
1.92 ms  # master
197 µs  # this PR
```

Internally, some operations in Pandas pass self.dtype to ``CategoricalIndex._create_categorical`` and onwards to ``_set_dtype``, which is a very slow code path.

By avoiding calling ``_set_dtype`` unnecessarily, some operations become faster. For example:

```python
>>> df = pd.DataFrame(dict(A=range(n*3)), index=ci)
>>> %timeit df.loc['b']
3.55 ms # master
2.14 ms  # this PR
```

As ``Categoricalndex._create_categorical`` is called directly or indirectly by various methods (e.g. ``CategoricalIndex._shallow_copy``), there are probably other places where this speedup is relevant also.

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Performance
description: Memory or execution speed performance

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/21506?src=pr&el=h1) Report
> Merging [#21506](https://codecov.io/gh/pandas-dev/pandas/pull/21506?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/9e982e18678c47c83ce2bb5932f844e2d9ee21f3?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/21506/graphs/tree.svg?token=eZ4WkYLtcO&src=pr&width=650&height=150)](https://codecov.io/gh/pandas-dev/pandas/pull/21506?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #21506      +/-   ##
==========================================
+ Coverage   91.92%   91.92%   +<.01%     
==========================================
  Files         153      153              
  Lines       49587    49589       +2     
==========================================
+ Hits        45583    45585       +2     
  Misses       4004     4004
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.32% <100%> (ø)` | :arrow_up: |
| #single | `41.89% <0%> (-0.01%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/21506?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/indexes/category.py](https://codecov.io/gh/pandas-dev/pandas/pull/21506/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9jYXRlZ29yeS5weQ==) | `97.11% <100%> (+0.01%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/21506?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/21506?src=pr&el=footer). Last update [9e982e1...0db3e02](https://codecov.io/gh/pandas-dev/pandas/pull/21506?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

@topper-123 : Nice!  Can you post `asv` benchmark timings (to make sure there were no regressions introduced) if you have a chance?
Hmm, there seem to be introduced some performance regressions by this PR that I don't quite understand. I will close this PR until I've figured it out, so as not to clutter PR quoue...