Repository: pandas-dev/pandas
Pull request: BUG: Fix Series.get() for ExtensionArray and Categorical
- [x] closes #20882
- [x] tests added / passed
  - pandas/tests/extension/base/getitem.py::test_get()
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry
  - Not created - assume will go in v0.23 as part of ExtensionArray support


Problem statement:
BUG: Series.get() on ExtensionArray series (and Categorical)  indexed by integer returns incorrect result
#### Code Sample, a copy-pastable example if possible

```python
In [1]: import pandas as pd

In [2]: import decimal

In [3]: from pandas.tests.extension.decimal.array import DecimalArray
   ...:
   ...: a = DecimalArray([decimal.Decimal(str(i)) for i in range(5)])
   ...: sa = pd.Series(a, index=[2*i for i in range(5)])
   ...:

In [4]: sa
Out[4]:
0    0
2    1
4    2
6    3
8    4
dtype: decimal

In [5]: sa.get(4)
Out[5]: Decimal('4')

In [6]: sb = pd.Series([i for i in range(5)], index=sa.index)

In [7]: sb
Out[7]:
0    0
2    1
4    2
6    3
8    4
dtype: int64

In [8]: sb.get(4)
Out[8]: 2

In [14]: cat = pd.Categorical(values=["a", "b", "c", "a", "b", "c"],
    ...: categories=["a", "b", "c"], ordered=True)

In [15]: s = pd.Series(cat, index=[2*i for i in range(6)])

In [16]: s
Out[16]:
0     a
2     b
4     c
6     a
8     b
10    c
dtype: category
Categories (3, object): [a < b < c]

In [18]: s.get(2)
Out[18]: 'c'

In [22]: s2 = pd.Series(list(s.values), index=s.index)

In [23]: s2
Out[23]:
0     a
2     b
4     c
6     a
8     b
10    c
dtype: object

In [24]: s2.get(2)
Out[24]: 'b'
```
#### Problem description

In the above, `sb` is a standard `Series` and `sb.get(4)` returns the element with index value 4.   But for `sa`, which is backed by an `ExtensionArray`, it is returning the 4th element of the array.

For the series `s` containing `Categorical`, `s.get(2)` is returning the 3rd element of the array, rather than the second.


#### Expected Output

`sa.get(4)` should be `Decimal('2')`

For the `Categorical` example, `s.get(2)` should return `'b'`, similar to the expression `s2.get(2)`.


#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit: None
python: 3.6.4.final.0
python-bits: 64
OS: Windows
OS-release: 10
machine: AMD64
processor: Intel64 Family 6 Model 60 Stepping 3, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.None

pandas: 0.23.0.dev0+811.g4afc75638
pytest: 3.4.0
pip: 9.0.1
setuptools: 38.5.1
Cython: 0.25.1
numpy: 1.14.1
scipy: 1.0.0
pyarrow: 0.8.0
xarray: None
IPython: 6.2.1
sphinx: 1.7.1
patsy: 0.5.0
dateutil: 2.6.1
pytz: 2018.3
blosc: 1.5.1
bottleneck: 1.2.1
tables: 3.4.2
numexpr: 2.6.4
feather: None
matplotlib: 2.2.0
openpyxl: 2.5.0
xlrd: 1.1.0
xlwt: 1.3.0
xlsxwriter: 1.0.2
lxml: 4.1.1
bs4: 4.6.0
html5lib: 1.0.1
sqlalchemy: 1.2.5
pymysql: 0.8.0
psycopg2: None
jinja2: 2.10
s3fs: 0.1.3
fastparquet: None
pandas_gbq: None
pandas_datareader: None

</details>

Hints from issue discussion:

Commit message: Add additional tests

Patch to review:
diff --git a/pandas/core/indexes/base.py b/pandas/core/indexes/base.py
index 2ceec1592d49b..6f96b78a9dbc6 100644
--- a/pandas/core/indexes/base.py
+++ b/pandas/core/indexes/base.py
@@ -3068,13 +3068,23 @@ def get_value(self, series, key):
         # if we have something that is Index-like, then
         # use this, e.g. DatetimeIndex
         s = getattr(series, '_values', None)
-        if isinstance(s, (ExtensionArray, Index)) and is_scalar(key):
-            try:
-                return s[key]
-            except (IndexError, ValueError):
+        if is_scalar(key):
+            if isinstance(s, Index):
+                try:
+                    return s[key]
+                except (IndexError, ValueError):
 
-                # invalid type as an indexer
-                pass
+                    # invalid type as an indexer
+                    pass
+            elif isinstance(s, ExtensionArray):
+                try:
+                    # This should call the ExtensionArray __getitem__
+                    iloc = self.get_loc(key)
+                    return s[iloc]
+                except (IndexError, ValueError):
+
+                    # invalid type as an indexer
+                    pass
 
         s = com._values_from_object(series)
         k = com._values_from_object(key)
diff --git a/pandas/tests/extension/base/getitem.py b/pandas/tests/extension/base/getitem.py
index 5c9ede1079079..5c9f56be66b91 100644
--- a/pandas/tests/extension/base/getitem.py
+++ b/pandas/tests/extension/base/getitem.py
@@ -117,6 +117,29 @@ def test_getitem_slice(self, data):
         result = data[slice(1)]  # scalar
         assert isinstance(result, type(data))
 
+    def test_get(self, data):
+        # GH 20882
+        s = pd.Series(data, index=[2 * i for i in range(len(data))])
+        assert s.get(4) == s.iloc[2]
+
+        result = s.get([4, 6])
+        expected = s.iloc[[2, 3]]
+        self.assert_series_equal(result, expected)
+
+        result = s.get(slice(2))
+        expected = s.iloc[[0, 1]]
+        self.assert_series_equal(result, expected)
+
+        s = pd.Series(data[:6], index=list('abcdef'))
+        assert s.get('c') == s.iloc[2]
+
+        result = s.get(slice('b', 'd'))
+        expected = s.iloc[[1, 2, 3]]
+        self.assert_series_equal(result, expected)
+
+        result = s.get('Z')
+        assert result is None
+
     def test_take_sequence(self, data):
         result = pd.Series(data)[[0, 1, 3]]
         assert result.iloc[0] == data[0]


Code review comments:
diff_hunk: @@ -3068,13 +3068,23 @@ def get_value(self, series, key):
         # if we have something that is Index-like, then
         # use this, e.g. DatetimeIndex
         s = getattr(series, '_values', None)
-        if isinstance(s, (ExtensionArray, Index)) and is_scalar(key):
-            try:
-                return s[key]
path: pandas/core/indexes/base.py
text: @user1:
if you just use ``.get_loc(key)`` on an Index as well does this work? (perf should be similar). That way we can avoid separating this.