Repository: pandas-dev/pandas

Issue:
ExtensionBlock.is_numeric is always False
#### Code Sample/Problem Description

Currently `ExtensionBlock.is_numeric` always returns `False`.  This can be problematic for extension arrays that are numeric, as this is used under the hood in places to filter to numeric columns in a `DataFrame`.  I'll be using `IntegerArray` as an example, but this in principle applies to any numeric extension array, e.g. `DecimalArray` in the testing suite, an extension array for units/uncertainties, etc.

Setup:
```python
In [2]: df = pd.DataFrame({'group': list('aaabbb'),
   ...:                    'val1': IntegerArray([0, 1, 2, np.nan, 3, 4]),
   ...:                    'val2': np.arange(6)})
   ...:                    

In [3]: df
Out[3]: 
  group val1  val2
0     a    0     0
1     a    1     1
2     a    2     2
3     b  NaN     3
4     b    3     4
5     b    4     5
```

The `IntegerArray` column is ignored by `DataFrame._get_numeric_data()`:
```python
In [4]: df._get_numeric_data()
Out[4]: 
   val2
0     0
1     1
2     2
3     3
4     4
5     5
```

This leads some numeric routines, such as `DataFrame.corr` ignoring the `IntegerArray` column:
```python
In [5]: df.corr()
Out[5]: 
      val2
val2   1.0
```

Likewise, `groupby` uses `ExtensionBlock.is_numeric` to filter to numeric columns for some operations, leading to the `IntegerArray` column being ignored, even if explicitly requested:
```python
In [6]: df.groupby('group').sum()
Out[6]: 
       val2
group      
a         3
b        12

In [7]: df.groupby('group')['val1', 'val2'].sum()
Out[7]: 
       val2
group      
a         3
b        12
```
#### Expected Output
I'd expect `ExtensionBlock.is_numeric` to return `True` when appropriate, and for behavior to be consistent with non-extension numeric dtypes.

My first impression is that this should be an attribute of the `ExtensionArray` or `ExtensionDtype` class that defaults to `False`, with numeric implementations setting the attribute to `True`, and `ExtensionBlock.is_numeric` would read the value from there.

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit: 0370740034978d3a63d4b8e5e2c96ff54e7e08ba
python: 3.6.5.final.0
python-bits: 64
OS: Linux
OS-release: 4.14.29-galliumos
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.24.0.dev0+456.g0370740
pytest: 3.5.1
pip: 10.0.1
setuptools: 39.1.0
Cython: 0.28.2
numpy: 1.14.3
scipy: 1.1.0
pyarrow: None
xarray: None
IPython: 6.4.0
sphinx: 1.7.4
patsy: 0.5.0
dateutil: 2.7.3
pytz: 2018.4
blosc: None
bottleneck: 1.2.1
tables: 3.4.3
numexpr: 2.6.5
feather: None
matplotlib: 2.2.2
openpyxl: 2.5.3
xlrd: 1.1.0
xlwt: 1.3.0
xlsxwriter: 1.0.4
lxml: 4.2.1
bs4: 4.6.0
html5lib: 1.0.1
sqlalchemy: 1.2.7
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
gcsfs: None

</details>



Discussion:
I think instead of hardcoding is_xyz attributes on the block subclasses they should be inferred like in Index.  Both for correctness (i.e. this Issue) and so we can share code between Block and Index, both of which are array-like.
On my WIP SparseArray branch, I get around this by adding 

```python
    @property
    def _is_numeric(self):
        return False
```

to `ExtensionDtype`. And then ExtensionBlock looks for that.

We *could* infer from `ExtensionDtype.kind`, and exclude things like `O`, `S`, and `U`
so for integer I made 2 changes (these are not pushed yet). These provide dispatch to blocks as needed for all ops.

```
+diff --git a/pandas/core/dtypes/common.py b/pandas/core/dtypes/common.py
index b8cbb4150..a40b54919 100644
--- a/pandas/core/dtypes/common.py
+++ b/pandas/core/dtypes/common.py
@@ -1487,7 +1487,8 @@ def is_numeric_dtype(arr_or_dtype):
     if arr_or_dtype is None:
         return False
     tipo = _get_dtype_type(arr_or_dtype)
-    return (issubclass(tipo, (np.number, np.bool_)) and
+    return (issubclass(tipo, (np.number, np.bool_)) or
+            getattr(tipo, 'is_numeric_dtype', False) and
             not issubclass(tipo, (np.datetime64, np.timedelta64)))
 
 
diff --git a/pandas/core/arrays/integer.py b/pandas/core/arrays/integer.py
index c12611706..38ff5bfad 100644
--- a/pandas/core/arrays/integer.py
+++ b/pandas/core/arrays/integer.py
@@ -37,6 +37,10 @@ class _IntegerDtype(ExtensionDtype):
     type = None
     na_value = np.nan
 
+    @property
+    def is_numeric_dtype(self):
+        return True
+
```

Resolving patch:
diff --git a/pandas/core/arrays/integer.py b/pandas/core/arrays/integer.py
--- a/pandas/core/arrays/integer.py
+++ b/pandas/core/arrays/integer.py
@@ -47,6 +47,10 @@ def is_signed_integer(self):
     def is_unsigned_integer(self):
         return self.kind == 'u'
 
+    @property
+    def _is_numeric(self):
+        return True
+
     @cache_readonly
     def numpy_dtype(self):
         """ Return an instance of our numpy dtype """
diff --git a/pandas/core/dtypes/base.py b/pandas/core/dtypes/base.py
--- a/pandas/core/dtypes/base.py
+++ b/pandas/core/dtypes/base.py
@@ -94,6 +94,18 @@ def is_dtype(cls, dtype):
         except TypeError:
             return False
 
+    @property
+    def _is_numeric(self):
+        # type: () -> bool
+        """
+        Whether columns with this dtype should be considered numeric.
+
+        By default ExtensionDtypes are assumed to be non-numeric.
+        They'll be excluded from operations that exclude non-numeric
+        columns, like (groupby) reductions, plotting, etc.
+        """
+        return False
+
 
 class ExtensionDtype(_DtypeOpsMixin):
     """A custom data type, to be paired with an ExtensionArray.
@@ -109,6 +121,11 @@ class ExtensionDtype(_DtypeOpsMixin):
     * name
     * construct_from_string
 
+    The following attributes influence the behavior of the dtype in
+    pandas operations
+
+    * _is_numeric
+
     Optionally one can override construct_array_type for construction
     with the name of this dtype via the Registry
 
diff --git a/pandas/core/internals/blocks.py b/pandas/core/internals/blocks.py
--- a/pandas/core/internals/blocks.py
+++ b/pandas/core/internals/blocks.py
@@ -665,7 +665,7 @@ def _astype(self, dtype, copy=False, errors='raise', values=None,
                     pass
 
             newb = make_block(values, placement=self.mgr_locs,
-                              klass=klass)
+                              klass=klass, ndim=self.ndim)
         except:
             if errors == 'raise':
                 raise
@@ -1950,6 +1950,10 @@ def is_view(self):
         """Extension arrays are never treated as views."""
         return False
 
+    @property
+    def is_numeric(self):
+        return self.values.dtype._is_numeric
+
     def setitem(self, indexer, value, mgr=None):
         """Set the value inplace, returning a same-typed block.
 


Test patch:
diff --git a/pandas/tests/extension/base/groupby.py b/pandas/tests/extension/base/groupby.py
--- a/pandas/tests/extension/base/groupby.py
+++ b/pandas/tests/extension/base/groupby.py
@@ -67,3 +67,16 @@ def test_groupby_extension_apply(self, data_for_grouping, op):
         df.groupby("B").A.apply(op)
         df.groupby("A").apply(op)
         df.groupby("A").B.apply(op)
+
+    def test_in_numeric_groupby(self, data_for_grouping):
+        df = pd.DataFrame({"A": [1, 1, 2, 2, 3, 3, 1, 4],
+                           "B": data_for_grouping,
+                           "C": [1, 1, 1, 1, 1, 1, 1, 1]})
+        result = df.groupby("A").sum().columns
+
+        if data_for_grouping.dtype._is_numeric:
+            expected = pd.Index(['B', 'C'])
+        else:
+            expected = pd.Index(['C'])
+
+        tm.assert_index_equal(result, expected)
diff --git a/pandas/tests/extension/base/interface.py b/pandas/tests/extension/base/interface.py
--- a/pandas/tests/extension/base/interface.py
+++ b/pandas/tests/extension/base/interface.py
@@ -67,3 +67,7 @@ def test_no_values_attribute(self, data):
         # code, disallowing this for now until solved
         assert not hasattr(data, 'values')
         assert not hasattr(data, '_values')
+
+    def test_is_numeric_honored(self, data):
+        result = pd.Series(data)
+        assert result._data.blocks[0].is_numeric is data.dtype._is_numeric
diff --git a/pandas/tests/extension/decimal/array.py b/pandas/tests/extension/decimal/array.py
--- a/pandas/tests/extension/decimal/array.py
+++ b/pandas/tests/extension/decimal/array.py
@@ -44,6 +44,10 @@ def construct_from_string(cls, string):
             raise TypeError("Cannot construct a '{}' from "
                             "'{}'".format(cls, string))
 
+    @property
+    def _is_numeric(self):
+        return True
+
 
 class DecimalArray(ExtensionArray, ExtensionScalarOpsMixin):
 
diff --git a/pandas/tests/extension/integer/test_integer.py b/pandas/tests/extension/integer/test_integer.py
--- a/pandas/tests/extension/integer/test_integer.py
+++ b/pandas/tests/extension/integer/test_integer.py
@@ -766,6 +766,22 @@ def test_cross_type_arithmetic():
     tm.assert_series_equal(result, expected)
 
 
+def test_groupby_mean_included():
+    df = pd.DataFrame({
+        "A": ['a', 'b', 'b'],
+        "B": [1, None, 3],
+        "C": IntegerArray([1, None, 3], dtype='Int64'),
+    })
+
+    result = df.groupby("A").sum()
+    # TODO(#22346): preserve Int64 dtype
+    expected = pd.DataFrame({
+        "B": np.array([1.0, 3.0]),
+        "C": np.array([1, 3], dtype="int64")
+    }, index=pd.Index(['a', 'b'], name='A'))
+    tm.assert_frame_equal(result, expected)
+
+
 def test_astype_nansafe():
     # https://github.com/pandas-dev/pandas/pull/22343
     arr = IntegerArray([np.nan, 1, 2], dtype="Int8")
diff --git a/pandas/tests/frame/test_block_internals.py b/pandas/tests/frame/test_block_internals.py
--- a/pandas/tests/frame/test_block_internals.py
+++ b/pandas/tests/frame/test_block_internals.py
@@ -11,7 +11,8 @@
 import numpy as np
 
 from pandas import (DataFrame, Series, Timestamp, date_range, compat,
-                    option_context)
+                    option_context, Categorical)
+from pandas.core.arrays import IntegerArray, IntervalArray
 from pandas.compat import StringIO
 import pandas as pd
 
@@ -436,6 +437,17 @@ def test_get_numeric_data(self):
         expected = df
         assert_frame_equal(result, expected)
 
+    def test_get_numeric_data_extension_dtype(self):
+        # GH 22290
+        df = DataFrame({
+            'A': IntegerArray([-10, np.nan, 0, 10, 20, 30], dtype='Int64'),
+            'B': Categorical(list('abcabc')),
+            'C': IntegerArray([0, 1, 2, 3, np.nan, 5], dtype='UInt8'),
+            'D': IntervalArray.from_breaks(range(7))})
+        result = df._get_numeric_data()
+        expected = df.loc[:, ['A', 'C']]
+        assert_frame_equal(result, expected)
+
     def test_convert_objects(self):
 
         oops = self.mixed_frame.T.T