Repository: pandas-dev/pandas

Issue:
PERF: df.loc is 100x slower for CategoricalIndex than for normal Index
ORIGINAL: **13.8 ms**
EDIT: After #21369 was merged the result of ``%timeit df2.loc['b']`` has improved to  **3.8 ms**.
EDIT: After #21618 was merged the result of ``%timeit df2.loc['b']`` has improved to  **3.3 ms**.
EDIT: After #21659 was merged the result of ``%timeit df2.loc['b']`` has improved to  **1.6 ms**.
EDIT: After #23235 was merged the result of ``%timeit df2.loc['b']`` has improved to  **159 µs**. Issue resolved.

#### Code Sample

```python
>>> n = 100_000
>>> df1 = pd.DataFrame(dict(A=range(n*3)), index=list('a'*n + 'b'*n + 'c'*n))
>>> df1.index.is_monotonic_increasing
True
>>> df2 = df1.copy()
>>> df2.index = pd.CategoricalIndex(df2.index)
>>> df2.index.is_monotonic_increasing
True
>>> %timeit df1.loc['b']
125 µs ± 2.95 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
>>> %timeit df2.loc['b']
13.8 ms ± 193 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```
#### Problem description

Selecting on a ``CategoricalIndex`` is 100x slower than selecting on a normal ``Index``.

I've tested this on master ( a few days old)  and on v0.22, with same result for both versions. The speed is even worse than the speed for a full columns scan:

```python
>>> df3 = df2.reset_index()
>>> %timeit df3[df3['index'] == 'b']
6.58 ms ± 25.3 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```

A guess is that the binary search is bypassed and a full index scan is being done + some extra stuff  so it's even slower than a normal full columns scan.

#### Expected Output

The output is as expected, but the speed is very slow for ``CategoricalIndex``.

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit: a7a7f8c1101aed1a9d37abbbcd80f77da414f0a8
python: 3.6.3.final.0
python-bits: 32
OS: Windows
OS-release: 10
machine: AMD64
processor: Intel64 Family 6 Model 78 Stepping 3, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.None

pandas: 0.22.0.dev0+870.ga7a7f8c
pytest: 3.3.1
pip: 9.0.1
setuptools: 38.2.5
Cython: 0.26.1
numpy: 1.13.3
scipy: 1.0.0
pyarrow: None
xarray: 0.10.0
IPython: 6.2.1
sphinx: 1.6.3
patsy: 0.4.1
dateutil: 2.6.1
pytz: 2017.3
blosc: None
bottleneck: None
tables: 3.4.2
numexpr: 2.6.4
feather: None
matplotlib: 2.1.0
openpyxl: 2.4.9
xlrd: 1.1.0
xlwt: 1.3.0
xlsxwriter: 1.0.2
lxml: None
bs4: None
html5lib: 1.0b10
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.9.6
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

</details>


Discussion:
I've retested this on RC2, and this issue is still there. Anyone can confirm this issue?

Is this a known limitation of using ``CategoricalIndex`` rather than ``Index``?
It looks to me like CategoricalIndex is slower every step of the way, even with the improvements in #21022



```python

key = df1.loc.obj.index._engine.get_loc('b')
result = df1.loc.obj.iloc[key]

```

```python
%timeit key = df1.loc.obj.index._engine.get_loc('b')
```

9.86 µs ± 635 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
    


```python
%timeit result = df1.loc.obj.iloc[key]
```

73.9 µs ± 1.35 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
    


```python
codes = df2.loc.obj.index.categories.get_loc('b')
key = df2.loc.obj.index._engine.get_loc(codes)
result = df2.loc.obj.iloc[key]
```


```python
%timeit codes = df2.loc.obj.index.categories.get_loc('b')
%timeit key = df2.loc.obj.index._engine.get_loc(codes)
```

 2.15 µs ± 156 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
857 µs ± 144 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
    


```python
%timeit result = df2.loc.obj.iloc[key]
```
429 µs ± 17.9 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
    

Resolving patch:
diff --git a/asv_bench/benchmarks/indexing_engines.py b/asv_bench/benchmarks/indexing_engines.py
--- a/asv_bench/benchmarks/indexing_engines.py
+++ b/asv_bench/benchmarks/indexing_engines.py
@@ -1,18 +1,30 @@
 import numpy as np
 
-from pandas._libs.index import (Int64Engine, UInt64Engine, Float64Engine,
-                                ObjectEngine)
+from pandas._libs import index as libindex
+
+
+def _get_numeric_engines():
+    engine_names = [
+        ('Int64Engine', np.int64), ('Int32Engine', np.int32),
+        ('Int16Engine', np.int16), ('Int8Engine', np.int8),
+        ('UInt64Engine', np.uint64), ('UInt32Engine', np.uint32),
+        ('UInt16engine', np.uint16), ('UInt8Engine', np.uint8),
+        ('Float64Engine', np.float64), ('Float32Engine', np.float32),
+    ]
+    return [(getattr(libindex, engine_name), dtype)
+            for engine_name, dtype in engine_names
+            if hasattr(libindex, engine_name)]
 
 
 class NumericEngineIndexing(object):
 
-    params = [[Int64Engine, UInt64Engine, Float64Engine],
-              [np.int64, np.uint64, np.float64],
+    params = [_get_numeric_engines(),
               ['monotonic_incr', 'monotonic_decr', 'non_monotonic'],
               ]
-    param_names = ['engine', 'dtype', 'index_type']
+    param_names = ['engine_and_dtype', 'index_type']
 
-    def setup(self, engine, dtype, index_type):
+    def setup(self, engine_and_dtype, index_type):
+        engine, dtype = engine_and_dtype
         N = 10**5
         values = list([1] * N + [2] * N + [3] * N)
         arr = {
@@ -26,7 +38,7 @@ def setup(self, engine, dtype, index_type):
         # code belows avoids populating the mapping etc. while timing.
         self.data.get_loc(2)
 
-    def time_get_loc(self, engine, dtype, index_type):
+    def time_get_loc(self, engine_and_dtype, index_type):
         self.data.get_loc(2)
 
 
@@ -44,7 +56,7 @@ def setup(self, index_type):
             'non_monotonic': np.array(list('abc') * N, dtype=object),
         }[index_type]
 
-        self.data = ObjectEngine(lambda: arr, len(arr))
+        self.data = libindex.ObjectEngine(lambda: arr, len(arr))
         # code belows avoids populating the mapping etc. while timing.
         self.data.get_loc('b')
 
diff --git a/pandas/core/indexes/category.py b/pandas/core/indexes/category.py
--- a/pandas/core/indexes/category.py
+++ b/pandas/core/indexes/category.py
@@ -84,7 +84,17 @@ class CategoricalIndex(Index, accessor.PandasDelegate):
     """
 
     _typ = 'categoricalindex'
-    _engine_type = libindex.Int64Engine
+
+    @property
+    def _engine_type(self):
+        # self.codes can have dtype int8, int16, int32 or int64, so we need
+        # to return the corresponding engine type (libindex.Int8Engine, etc.).
+        return {np.int8: libindex.Int8Engine,
+                np.int16: libindex.Int16Engine,
+                np.int32: libindex.Int32Engine,
+                np.int64: libindex.Int64Engine,
+                }[self.codes.dtype.type]
+
     _attributes = ['name']
 
     def __new__(cls, data=None, categories=None, ordered=None, dtype=None,
@@ -382,7 +392,7 @@ def argsort(self, *args, **kwargs):
     def _engine(self):
 
         # we are going to look things up with the codes themselves
-        return self._engine_type(lambda: self.codes.astype('i8'), len(self))
+        return self._engine_type(lambda: self.codes, len(self))
 
     # introspection
     @cache_readonly
@@ -450,6 +460,7 @@ def get_loc(self, key, method=None):
         array([False,  True, False,  True], dtype=bool)
         """
         code = self.categories.get_loc(key)
+        code = self.codes.dtype.type(code)
         try:
             return self._engine.get_loc(code)
         except KeyError:


Test patch:
diff --git a/pandas/tests/indexes/test_category.py b/pandas/tests/indexes/test_category.py
--- a/pandas/tests/indexes/test_category.py
+++ b/pandas/tests/indexes/test_category.py
@@ -1,16 +1,16 @@
 # -*- coding: utf-8 -*-
 
 import pytest
+import numpy as np
 
 import pandas.util.testing as tm
 from pandas.core.indexes.api import Index, CategoricalIndex
 from pandas.core.dtypes.dtypes import CategoricalDtype
+from pandas._libs import index as libindex
 from .common import Base
 
 from pandas.compat import range, PY3
 
-import numpy as np
-
 from pandas import Categorical, IntervalIndex, compat
 from pandas.util.testing import assert_almost_equal
 import pandas.core.config as cf
@@ -1117,3 +1117,23 @@ def test_take_invalid_kwargs(self):
         msg = "the 'mode' parameter is not supported"
         tm.assert_raises_regex(ValueError, msg, idx.take,
                                indices, mode='clip')
+
+    @pytest.mark.parametrize('dtype, engine_type', [
+        (np.int8, libindex.Int8Engine),
+        (np.int16, libindex.Int16Engine),
+        (np.int32, libindex.Int32Engine),
+        (np.int64, libindex.Int64Engine),
+    ])
+    def test_engine_type(self, dtype, engine_type):
+        if dtype != np.int64:
+            # num. of uniques required to push CategoricalIndex.codes to a
+            # dtype (128 categories required for .codes dtype to be int16 etc.)
+            num_uniques = {np.int8: 1, np.int16: 128, np.int32: 32768}[dtype]
+            ci = pd.CategoricalIndex(range(num_uniques))
+        else:
+            # having 2**32 - 2**31 categories would be very memory-intensive,
+            # so we cheat a bit with the dtype
+            ci = pd.CategoricalIndex(range(32768))  # == 2**16 - 2**(16 - 1)
+            ci.values._codes = ci.values._codes.astype('int64')
+        assert np.issubdtype(ci.codes.dtype, dtype)
+        assert isinstance(ci._engine, engine_type)
diff --git a/pandas/tests/indexing/conftest.py b/pandas/tests/indexing/conftest.py
new file mode 100644
--- /dev/null
+++ b/pandas/tests/indexing/conftest.py
@@ -0,0 +1,20 @@
+import numpy as np
+import pytest
+
+from pandas._libs import index as libindex
+
+
+@pytest.fixture(params=[
+    (libindex.Int64Engine, np.int64),
+    (libindex.Int32Engine, np.int32),
+    (libindex.Int16Engine, np.int16),
+    (libindex.Int8Engine, np.int8),
+    (libindex.UInt64Engine, np.uint64),
+    (libindex.UInt32Engine, np.uint32),
+    (libindex.UInt16Engine, np.uint16),
+    (libindex.UInt8Engine, np.uint8),
+    (libindex.Float64Engine, np.float64),
+    (libindex.Float32Engine, np.float32),
+], ids=lambda x: x[0].__name__)
+def numeric_indexing_engine_type_and_dtype(request):
+    return request.param
diff --git a/pandas/tests/indexing/test_indexing_engines.py b/pandas/tests/indexing/test_indexing_engines.py
new file mode 100644
--- /dev/null
+++ b/pandas/tests/indexing/test_indexing_engines.py
@@ -0,0 +1,168 @@
+import numpy as np
+
+import pandas.util.testing as tm
+from pandas import compat
+from pandas._libs import algos as libalgos, index as libindex
+
+
+class TestNumericEngine(object):
+    def test_is_monotonic(self, numeric_indexing_engine_type_and_dtype):
+        engine_type, dtype = numeric_indexing_engine_type_and_dtype
+        num = 1000
+        arr = np.array([1] * num + [2] * num + [3] * num, dtype=dtype)
+
+        # monotonic increasing
+        engine = engine_type(lambda: arr, len(arr))
+        assert engine.is_monotonic_increasing is True
+        assert engine.is_monotonic_decreasing is False
+
+        # monotonic decreasing
+        engine = engine_type(lambda: arr[::-1], len(arr))
+        assert engine.is_monotonic_increasing is False
+        assert engine.is_monotonic_decreasing is True
+
+        # neither monotonic increasing or decreasing
+        arr = np.array([1] * num + [2] * num + [1] * num, dtype=dtype)
+        engine = engine_type(lambda: arr[::-1], len(arr))
+        assert engine.is_monotonic_increasing is False
+        assert engine.is_monotonic_decreasing is False
+
+    def test_is_unique(self, numeric_indexing_engine_type_and_dtype):
+        engine_type, dtype = numeric_indexing_engine_type_and_dtype
+
+        # unique
+        arr = np.array([1, 3, 2], dtype=dtype)
+        engine = engine_type(lambda: arr, len(arr))
+        assert engine.is_unique is True
+
+        # not unique
+        arr = np.array([1, 2, 1], dtype=dtype)
+        engine = engine_type(lambda: arr, len(arr))
+        assert engine.is_unique is False
+
+    def test_get_loc(self, numeric_indexing_engine_type_and_dtype):
+        engine_type, dtype = numeric_indexing_engine_type_and_dtype
+
+        # unique
+        arr = np.array([1, 2, 3], dtype=dtype)
+        engine = engine_type(lambda: arr, len(arr))
+        assert engine.get_loc(2) == 1
+
+        # monotonic
+        num = 1000
+        arr = np.array([1] * num + [2] * num + [3] * num, dtype=dtype)
+        engine = engine_type(lambda: arr, len(arr))
+        assert engine.get_loc(2) == slice(1000, 2000)
+
+        # not monotonic
+        arr = np.array([1, 2, 3] * num, dtype=dtype)
+        engine = engine_type(lambda: arr, len(arr))
+        expected = np.array([False, True, False] * num, dtype=bool)
+        result = engine.get_loc(2)
+        assert (result == expected).all()
+
+    def test_get_backfill_indexer(
+            self, numeric_indexing_engine_type_and_dtype):
+        engine_type, dtype = numeric_indexing_engine_type_and_dtype
+
+        arr = np.array([1, 5, 10], dtype=dtype)
+        engine = engine_type(lambda: arr, len(arr))
+
+        new = np.array(compat.range(12), dtype=dtype)
+        result = engine.get_backfill_indexer(new)
+
+        expected = libalgos.backfill(arr, new)
+        tm.assert_numpy_array_equal(result, expected)
+
+    def test_get_pad_indexer(
+            self, numeric_indexing_engine_type_and_dtype):
+        engine_type, dtype = numeric_indexing_engine_type_and_dtype
+
+        arr = np.array([1, 5, 10], dtype=dtype)
+        engine = engine_type(lambda: arr, len(arr))
+
+        new = np.array(compat.range(12), dtype=dtype)
+        result = engine.get_pad_indexer(new)
+
+        expected = libalgos.pad(arr, new)
+        tm.assert_numpy_array_equal(result, expected)
+
+
+class TestObjectEngine(object):
+    engine_type = libindex.ObjectEngine
+    dtype = np.object_
+    values = list('abc')
+
+    def test_is_monotonic(self):
+
+        num = 1000
+        arr = np.array(['a'] * num + ['a'] * num + ['c'] * num,
+                       dtype=self.dtype)
+
+        # monotonic increasing
+        engine = self.engine_type(lambda: arr, len(arr))
+        assert engine.is_monotonic_increasing is True
+        assert engine.is_monotonic_decreasing is False
+
+        # monotonic decreasing
+        engine = self.engine_type(lambda: arr[::-1], len(arr))
+        assert engine.is_monotonic_increasing is False
+        assert engine.is_monotonic_decreasing is True
+
+        # neither monotonic increasing or decreasing
+        arr = np.array(['a'] * num + ['b'] * num + ['a'] * num,
+                       dtype=self.dtype)
+        engine = self.engine_type(lambda: arr[::-1], len(arr))
+        assert engine.is_monotonic_increasing is False
+        assert engine.is_monotonic_decreasing is False
+
+    def test_is_unique(self):
+        # unique
+        arr = np.array(self.values, dtype=self.dtype)
+        engine = self.engine_type(lambda: arr, len(arr))
+        assert engine.is_unique is True
+
+        # not unique
+        arr = np.array(['a', 'b', 'a'], dtype=self.dtype)
+        engine = self.engine_type(lambda: arr, len(arr))
+        assert engine.is_unique is False
+
+    def test_get_loc(self):
+        # unique
+        arr = np.array(self.values, dtype=self.dtype)
+        engine = self.engine_type(lambda: arr, len(arr))
+        assert engine.get_loc('b') == 1
+
+        # monotonic
+        num = 1000
+        arr = np.array(['a'] * num + ['b'] * num + ['c'] * num,
+                       dtype=self.dtype)
+        engine = self.engine_type(lambda: arr, len(arr))
+        assert engine.get_loc('b') == slice(1000, 2000)
+
+        # not monotonic
+        arr = np.array(self.values * num, dtype=self.dtype)
+        engine = self.engine_type(lambda: arr, len(arr))
+        expected = np.array([False, True, False] * num, dtype=bool)
+        result = engine.get_loc('b')
+        assert (result == expected).all()
+
+    def test_get_backfill_indexer(self):
+        arr = np.array(['a', 'e', 'j'], dtype=self.dtype)
+        engine = self.engine_type(lambda: arr, len(arr))
+
+        new = np.array(list('abcdefghij'), dtype=self.dtype)
+        result = engine.get_backfill_indexer(new)
+
+        expected = libalgos.backfill_object(arr, new)
+        tm.assert_numpy_array_equal(result, expected)
+
+    def test_get_pad_indexer(self):
+        arr = np.array(['a', 'e', 'j'], dtype=self.dtype)
+        engine = self.engine_type(lambda: arr, len(arr))
+
+        new = np.array(list('abcdefghij'), dtype=self.dtype)
+        result = engine.get_pad_indexer(new)
+
+        expected = libalgos.pad_object(arr, new)
+        tm.assert_numpy_array_equal(result, expected)