Repository: pandas-dev/pandas
Pull request: ENH: Allow storing ExtensionArrays in containers
Followup number 1 to #19268 

This adds changes to frame, series, dtypes, etc. (all the places other that `core/internals.py`) to support 3rd party extension arrays.

These changes were developed to support IntervalArrays (https://github.com/TomAugspurger/pandas/compare/pandas-array-upstream+fu1...TomAugspurger:pandas-array-upstream+fu1+interval separate PR) and https://github.com/continuumio/pandas-ip (external).

Also closes https://github.com/pandas-dev/pandas/issues/19585

Problem statement:
Test Order Dependency between parquet tests and internals
I observe this locally (py2 only)

```
pytest pandas/tests/io/test_parquet.py::TestParquetPyArrow::test_basic  pandas/tests/internals/test_internals.py::test_deprecated_fastpath --tb=no -v
=========================================================== test session starts ============================================================
platform darwin -- Python 2.7.14, pytest-3.3.2, py-1.5.2, pluggy-0.6.0 -- /Users/taugspurger/miniconda3/envs/pandas-ip2/bin/python
cachedir: .cache
rootdir: /Users/taugspurger/sandbox/pandas-ip-2/pandas, inifile: setup.cfg
plugins: xdist-1.22.0, forked-0.2, hypothesis-3.38.5
collected 2 items

pandas/tests/io/test_parquet.py::TestParquetPyArrow::test_basic PASSED                                                               [ 50%]
pandas/tests/internals/test_internals.py::test_deprecated_fastpath FAILED                                                            [100%]

==================================================== 1 failed, 1 passed in 0.35 seconds ====================================================
```

We apparently have an environment variable for controlling deprecation warnings.

```
PANDAS_TESTING_MODE=deprecate pytest pandas/tests/io/test_parquet.py::TestParquetPyArrow::test_basic  panda
s/tests/internals/test_internals.py::test_deprecated_fastpath --tb=no -v
=========================================================== test session starts ============================================================
platform darwin -- Python 2.7.14, pytest-3.3.2, py-1.5.2, pluggy-0.6.0 -- /Users/taugspurger/miniconda3/envs/pandas-ip2/bin/python
cachedir: .cache
rootdir: /Users/taugspurger/sandbox/pandas-ip-2/pandas, inifile: setup.cfg
plugins: xdist-1.22.0, forked-0.2, hypothesis-3.38.5
collected 2 items

pandas/tests/io/test_parquet.py::TestParquetPyArrow::test_basic PASSED                                                               [ 50%]
pandas/tests/internals/test_internals.py::test_deprecated_fastpath PASSED                                                            [100%]

========================================================= 2 passed in 0.13 seconds =========================================================
 ```

It'd be nice if the tests didn't rely on this.

Hints from issue discussion:

Commit message: Add a test array

Patch to review:
diff --git a/pandas/core/arrays/base.py b/pandas/core/arrays/base.py
index 1556b653819a6..f51c3bb12fe45 100644
--- a/pandas/core/arrays/base.py
+++ b/pandas/core/arrays/base.py
@@ -1,4 +1,6 @@
 """An interface for extending pandas with custom arrays."""
+import numpy as np
+
 from pandas.errors import AbstractMethodError
 
 _not_implemented_message = "{} does not implement {}."
@@ -18,19 +20,20 @@ class ExtensionArray(object):
 
     * __getitem__
     * __len__
+    * __iter__
     * dtype
     * nbytes
     * isna
     * take
     * copy
-    * _formatting_values
     * _concat_same_type
 
     Some additional methods are required to satisfy pandas' internal, private
     block API.
 
-    * _concat_same_type
     * _can_hold_na
+    * _formatting_values
+    * _fill_value
 
     This class does not inherit from 'abc.ABCMeta' for performance reasons.
     Methods and properties required by the interface raise
@@ -51,9 +54,6 @@ class ExtensionArray(object):
     Extension arrays should be able to be constructed with instances of
     the class, i.e. ``ExtensionArray(extension_array)`` should return
     an instance, not error.
-
-    Additionally, certain methods and interfaces are required for proper
-    this array to be properly stored inside a ``DataFrame`` or ``Series``.
     """
     # ------------------------------------------------------------------------
     # Must be a Sequence
@@ -177,9 +177,9 @@ def take(self, indexer, allow_fill=True, fill_value=None):
 
         Examples
         --------
-        Suppose the extension array somehow backed by a NumPy structured array
-        and that the underlying structured array is stored as ``self.data``.
-        Then ``take`` may be written as
+        Suppose the extension array somehow backed by a NumPy array and that
+        the underlying structured array is stored as ``self.data``. Then
+        ``take`` may be written as
 
         .. code-block:: python
 
@@ -219,7 +219,7 @@ def _formatting_values(self):
         # type: () -> np.ndarray
         # At the moment, this has to be an array since we use result.dtype
         """An array of values to be printed in, e.g. the Series repr"""
-        raise AbstractMethodError(self)
+        return np.array(self)
 
     @classmethod
     def _concat_same_type(cls, to_concat):
diff --git a/pandas/core/dtypes/base.py b/pandas/core/dtypes/base.py
index c7c5378801f02..2f071a3b3cf71 100644
--- a/pandas/core/dtypes/base.py
+++ b/pandas/core/dtypes/base.py
@@ -1,4 +1,6 @@
 """Extend pandas with custom array types"""
+import inspect
+
 from pandas.errors import AbstractMethodError
 
 
@@ -106,7 +108,8 @@ def is_dtype(cls, dtype):
 
         Parameters
         ----------
-        dtype : str or dtype
+        dtype : str, object, or type
+            The dtype to check.
 
         Returns
         -------
@@ -118,12 +121,15 @@ def is_dtype(cls, dtype):
 
         1. ``cls.construct_from_string(dtype)`` is an instance
            of ``cls``.
-        2. 'dtype' is ``cls`` or a subclass of ``cls``.
+        2. ``dtype`` is an object and is an instance of ``cls``
+        3. 'dtype' is a class and is ``cls`` or a subclass of ``cls``.
         """
         if isinstance(dtype, str):
             try:
                 return isinstance(cls.construct_from_string(dtype), cls)
             except TypeError:
                 return False
-        else:
+        elif inspect.isclass(dtype):
             return issubclass(dtype, cls)
+        else:
+            return isinstance(dtype, cls)
diff --git a/pandas/core/dtypes/common.py b/pandas/core/dtypes/common.py
index c66e7fcfc6978..2344091f85a88 100644
--- a/pandas/core/dtypes/common.py
+++ b/pandas/core/dtypes/common.py
@@ -1708,9 +1708,10 @@ def is_extension_array_dtype(arr_or_dtype):
     """
     from pandas.core.arrays import ExtensionArray
 
-    # we want to unpack series, anything else?
     if isinstance(arr_or_dtype, ABCSeries):
         arr_or_dtype = arr_or_dtype._values
+    elif isinstance(arr_or_dtype, ABCIndexClass):
+        arr_or_dtype = arr_or_dtype._as_best_array()
     return isinstance(arr_or_dtype, (ExtensionDtype, ExtensionArray))
 
 
diff --git a/pandas/core/dtypes/missing.py b/pandas/core/dtypes/missing.py
index ffac702476af1..c7cd97d5ceb87 100644
--- a/pandas/core/dtypes/missing.py
+++ b/pandas/core/dtypes/missing.py
@@ -10,9 +10,10 @@
                      is_datetimelike_v_numeric, is_float_dtype,
                      is_datetime64_dtype, is_datetime64tz_dtype,
                      is_timedelta64_dtype, is_interval_dtype,
-                     is_complex_dtype, is_categorical_dtype,
+                     is_complex_dtype,
                      is_string_like_dtype, is_bool_dtype,
                      is_integer_dtype, is_dtype_equal,
+                     is_extension_array_dtype,
                      needs_i8_conversion, _ensure_object,
                      pandas_dtype,
                      is_scalar,
@@ -52,12 +53,15 @@ def isna(obj):
 
 
 def _isna_new(obj):
+    from ..arrays import ExtensionArray
+
     if is_scalar(obj):
         return libmissing.checknull(obj)
     # hack (for now) because MI registers as ndarray
     elif isinstance(obj, ABCMultiIndex):
         raise NotImplementedError("isna is not defined for MultiIndex")
-    elif isinstance(obj, (ABCSeries, np.ndarray, ABCIndexClass)):
+    elif isinstance(obj, (ABCSeries, np.ndarray, ABCIndexClass,
+                          ExtensionArray)):
         return _isna_ndarraylike(obj)
     elif isinstance(obj, ABCGeneric):
         return obj._constructor(obj._data.isna(func=isna))
@@ -124,17 +128,20 @@ def _use_inf_as_na(key):
 
 
 def _isna_ndarraylike(obj):
-
     values = getattr(obj, 'values', obj)
     dtype = values.dtype
 
-    if is_string_dtype(dtype):
-        if is_categorical_dtype(values):
-            from pandas import Categorical
-            if not isinstance(values, Categorical):
-                values = values.values
-            result = values.isna()
-        elif is_interval_dtype(values):
+    if is_extension_array_dtype(obj):
+        if isinstance(obj, ABCIndexClass):
+            values = obj._as_best_array()
+        elif isinstance(obj, ABCSeries):
+            values = obj._values
+        else:
+            values = obj
+        result = values.isna()
+    elif is_string_dtype(dtype):
+        if is_interval_dtype(values):
+            # TODO(IntervalArray): remove this if block
             from pandas import IntervalIndex
             result = IntervalIndex(obj).isna()
         else:
@@ -406,4 +413,7 @@ def remove_na_arraylike(arr):
     """
     Return array-like containing only true/non-NaN values, possibly empty.
     """
-    return arr[notna(lib.values_from_object(arr))]
+    if is_extension_array_dtype(arr):
+        return arr[notna(arr)]
+    else:
+        return arr[notna(lib.values_from_object(arr))]
diff --git a/pandas/core/frame.py b/pandas/core/frame.py
index 96d28581cfdd9..ea3e7b33fb5b6 100644
--- a/pandas/core/frame.py
+++ b/pandas/core/frame.py
@@ -39,6 +39,7 @@
     is_categorical_dtype,
     is_object_dtype,
     is_extension_type,
+    is_extension_array_dtype,
     is_datetimetz,
     is_datetime64_any_dtype,
     is_datetime64tz_dtype,
@@ -71,7 +72,7 @@
                                    create_block_manager_from_arrays,
                                    create_block_manager_from_blocks)
 from pandas.core.series import Series
-from pandas.core.arrays import Categorical
+from pandas.core.arrays import Categorical, ExtensionArray
 import pandas.core.algorithms as algorithms
 from pandas.compat import (range, map, zip, lrange, lmap, lzip, StringIO, u,
                            OrderedDict, raise_with_traceback)
@@ -511,7 +512,7 @@ def _get_axes(N, K, index=index, columns=columns):
             index, columns = _get_axes(len(values), 1)
             return _arrays_to_mgr([values], columns, index, columns,
                                   dtype=dtype)
-        elif is_datetimetz(values):
+        elif (is_datetimetz(values) or is_extension_array_dtype(values)):
             # GH19157
             if columns is None:
                 columns = [0]
@@ -2796,7 +2797,7 @@ def reindexer(value):
             # now align rows
             value = reindexer(value).T
 
-        elif isinstance(value, Categorical):
+        elif isinstance(value, ExtensionArray):
             value = value.copy()
 
         elif isinstance(value, Index) or is_sequence(value):
@@ -2804,7 +2805,7 @@ def reindexer(value):
 
             # turn me into an ndarray
             value = _sanitize_index(value, self.index, copy=False)
-            if not isinstance(value, (np.ndarray, Index)):
+            if not isinstance(value, (np.ndarray, Index, ExtensionArray)):
                 if isinstance(value, list) and len(value) > 0:
                     value = maybe_convert_platform(value)
                 else:
@@ -2826,7 +2827,7 @@ def reindexer(value):
             value = maybe_cast_to_datetime(value, value.dtype)
 
         # return internal types directly
-        if is_extension_type(value):
+        if is_extension_type(value) or is_extension_array_dtype(value):
             return value
 
         # broadcast across multiple columns if necessary
@@ -3355,12 +3356,9 @@ class    max    type
             new_obj = self.copy()
 
         def _maybe_casted_values(index, labels=None):
-            if isinstance(index, PeriodIndex):
-                values = index.astype(object).values
-            elif isinstance(index, DatetimeIndex) and index.tz is not None:
-                values = index
-            else:
-                values = index.values
+            values = index._as_best_array()
+            # TODO: Check if nescessary...
+            if not isinstance(index, (PeriodIndex, DatetimeIndex)):
                 if values.dtype == np.object_:
                     values = lib.maybe_convert_objects(values)
 
diff --git a/pandas/core/indexes/base.py b/pandas/core/indexes/base.py
index 626f3dc86556a..37a408a8f6c11 100644
--- a/pandas/core/indexes/base.py
+++ b/pandas/core/indexes/base.py
@@ -13,6 +13,7 @@
 from pandas import compat
 
 from pandas.core.accessor import CachedAccessor
+from pandas.core.arrays import ExtensionArray
 from pandas.core.dtypes.generic import (
     ABCSeries, ABCDataFrame,
     ABCMultiIndex,
@@ -1038,6 +1039,31 @@ def _to_embed(self, keep_tz=False, dtype=None):
 
         return self.values.copy()
 
+    def _as_best_array(self):
+        # type: () -> Union[ExtensionArray, ndarary]
+        """Return the underlying values as the best array type.
+
+        Indexes backed by ExtensionArrays will return the ExtensionArray.
+        Otherwise, an ndarray is returned.
+
+        Examples
+        --------
+        >>> pd.Index([0, 1, 2])._as_best_array()
+        array([0, 1, 2])
+
+        >>> pd.CategoricalIndex(['a', 'a', 'b'])._as_best_array()
+        [a, a, b]
+        Categories (2, object): [a, b]
+
+        >>> pd.IntervalIndex.from_breaks([0, 1, 2])._as_best_array()
+        IntervalArray([(0, 1], (1, 2]])
+        """
+        # We need this since CategoricalIndex.values -> Categorical
+        #                but IntervalIndex.values    -> ndarray[object]
+        # TODO: IntervalIndex defines _array_values. Would be nice to
+        # have an unambiguous way of getting an ndarray (or just use asarray?)
+        return self.values
+
     _index_shared_docs['astype'] = """
         Create an Index with values cast to dtypes. The class of a new Index
         is determined by dtype. When conversion is impossible, a ValueError
@@ -1946,6 +1972,12 @@ def _format_with_header(self, header, na_rep='NaN', **kwargs):
 
         if is_categorical_dtype(values.dtype):
             values = np.array(values)
+
+        elif isinstance(values, ExtensionArray):
+            # This is still un-exercised within pandas, since all our
+            # extension dtypes have custom indexes.
+            values = values._formatting_values()
+
         elif is_object_dtype(values.dtype):
             values = lib.maybe_convert_objects(values, safe=1)
 
@@ -2525,7 +2557,7 @@ def get_value(self, series, key):
         # if we have something that is Index-like, then
         # use this, e.g. DatetimeIndex
         s = getattr(series, '_values', None)
-        if isinstance(s, Index) and is_scalar(key):
+        if isinstance(s, (ExtensionArray, Index)) and is_scalar(key):
             try:
                 return s[key]
             except (IndexError, ValueError):
diff --git a/pandas/core/indexes/category.py b/pandas/core/indexes/category.py
index 2c7be2b21f959..90541c58b2ef9 100644
--- a/pandas/core/indexes/category.py
+++ b/pandas/core/indexes/category.py
@@ -297,6 +297,9 @@ def get_values(self):
         """ return the underlying data as an ndarray """
         return self._data.get_values()
 
+    def _as_best_array(self):
+        return self._data
+
     def tolist(self):
         return self._data.tolist()
 
diff --git a/pandas/core/indexes/datetimes.py b/pandas/core/indexes/datetimes.py
index e09fa87477122..2ba010aeba467 100644
--- a/pandas/core/indexes/datetimes.py
+++ b/pandas/core/indexes/datetimes.py
@@ -1034,6 +1034,13 @@ def _to_embed(self, keep_tz=False, dtype=None):
 
         return self.values.copy()
 
+    def _as_best_array(self):
+        # no-tz -> ndarray
+        # tz    -> DatetimeIndex (for now)
+        if self.tz is not None:
+            return self
+        return self.values
+
     def to_pydatetime(self):
         """
         Return DatetimeIndex as object ndarray of datetime.datetime objects
diff --git a/pandas/core/internals.py b/pandas/core/internals.py
index f553e1a02c9d6..feff9ef24d8bc 100644
--- a/pandas/core/internals.py
+++ b/pandas/core/internals.py
@@ -56,7 +56,10 @@
     is_null_datelike_scalar)
 import pandas.core.dtypes.concat as _concat
 
-from pandas.core.dtypes.generic import ABCSeries, ABCDatetimeIndex
+from pandas.core.dtypes.generic import (
+    ABCSeries,
+    ABCDatetimeIndex,
+    ABCIndexClass)
 import pandas.core.common as com
 import pandas.core.algorithms as algos
 
@@ -1854,6 +1857,20 @@ class ExtensionBlock(NonConsolidatableMixIn, Block):
 
     ExtensionArrays are limited to 1-D.
     """
+
+    def __init__(self, values, placement, ndim=None):
+        values = self._maybe_coerce_values(values)
+        super(ExtensionBlock, self).__init__(values, placement, ndim)
+
+    def _maybe_coerce_values(self, values):
+        # Unboxes Series / Index
+        # Doesn't change any underlying dtypes.
+        if isinstance(values, ABCSeries):
+            values = values.values
+        elif isinstance(values, ABCIndexClass):
+            values = values._as_best_array()
+        return values
+
     @property
     def _holder(self):
         # For extension blocks, the holder is values-dependent.
@@ -4101,7 +4118,8 @@ def set(self, item, value, check=False):
         # FIXME: refactor, clearly separate broadcasting & zip-like assignment
         #        can prob also fix the various if tests for sparse/categorical
 
-        value_is_extension_type = is_extension_type(value)
+        value_is_extension_type = (is_extension_type(value) or
+                                   is_extension_array_dtype(value))
 
         # categorical/spares/datetimetz
         if value_is_extension_type:
@@ -4834,13 +4852,10 @@ def form_blocks(arrays, names, axes):
     if len(items_dict['ExtensionBlock']):
 
         external_blocks = []
+
         for i, _, array in items_dict['ExtensionBlock']:
-            if isinstance(array, ABCSeries):
-                array = array.values
-            # Allow our internal arrays to chose their block type.
-            block_type = getattr(array, '_block_type', ExtensionBlock)
             external_blocks.append(
-                make_block(array, klass=block_type,
+                make_block(array, klass=ExtensionBlock,
                            fastpath=True, placement=[i]))
         blocks.extend(external_blocks)
 
diff --git a/pandas/core/series.py b/pandas/core/series.py
index 78b4c3a70a519..6bd6bfc1c8ae9 100644
--- a/pandas/core/series.py
+++ b/pandas/core/series.py
@@ -14,6 +14,7 @@
 import numpy.ma as ma
 
 from pandas.core.accessor import CachedAccessor
+from pandas.core.arrays import ExtensionArray
 from pandas.core.dtypes.common import (
     is_categorical_dtype,
     is_bool,
@@ -173,12 +174,15 @@ def __init__(self, data=None, index=None, dtype=None, name=None,
                 raise NotImplementedError("initializing a Series from a "
                                           "MultiIndex is not supported")
             elif isinstance(data, Index):
-                # need to copy to avoid aliasing issues
                 if name is None:
                     name = data.name
 
-                data = data._to_embed(keep_tz=True, dtype=dtype)
-                copy = False
+                if dtype is not None:
+                    data = data.astype(dtype)
+
+                # need to copy to avoid aliasing issues
+                data = data._as_best_array().copy()
+
             elif isinstance(data, np.ndarray):
                 pass
             elif isinstance(data, Series):
@@ -234,6 +238,10 @@ def __init__(self, data=None, index=None, dtype=None, name=None,
                                        copy=copy)
                 elif copy:
                     data = data.copy()
+            elif isinstance(data, ExtensionArray):
+                if copy:
+                    data = data.copy()
+                data = SingleBlockManager(data, index, fastpath=True)
             else:
                 data = _sanitize_array(data, index, dtype, copy,
                                        raise_cast_failure=True)
@@ -2570,7 +2578,11 @@ def _reindex_indexer(self, new_index, indexer, copy):
             return self
 
         # be subclass-friendly
-        new_values = algorithms.take_1d(self.get_values(), indexer)
+        if isinstance(self.values, ExtensionArray):
+            new_values = self.values.take(indexer)
+        else:
+            new_values = algorithms.take_1d(self.get_values(), indexer)
+
         return self._constructor(new_values, index=new_index)
 
     def _needs_reindex_multi(self, axes, method, level):
@@ -3117,11 +3129,8 @@ def _sanitize_index(data, index, copy=False):
         raise ValueError('Length of values does not match length of ' 'index')
 
     if isinstance(data, ABCIndexClass) and not copy:
-        pass
-    elif isinstance(data, PeriodIndex):
-        data = data.astype(object).values
-    elif isinstance(data, DatetimeIndex):
-        data = data._to_embed(keep_tz=True)
+        data = data._as_best_array()
+
     elif isinstance(data, np.ndarray):
 
         # coerce datetimelike types
@@ -3194,11 +3203,12 @@ def _try_cast(arr, take_fast_path):
             # we will try to copy be-definition here
             subarr = _try_cast(data, True)
 
-    elif isinstance(data, Categorical):
+    elif isinstance(data, ExtensionArray):
         subarr = data
 
         if copy:
             subarr = data.copy()
+        # XXX: This is the only early return. See if it can be avoided.
         return subarr
 
     elif isinstance(data, (list, tuple)) and len(data) > 0:
@@ -3221,6 +3231,7 @@ def _try_cast(arr, take_fast_path):
         start, stop, step = get_range_parameters(data)
         arr = np.arange(start, stop, step, dtype='int64')
         subarr = _try_cast(arr, False)
+
     else:
         subarr = _try_cast(data, False)
 
diff --git a/pandas/tests/extension_arrays/__init__.py b/pandas/tests/extension_arrays/__init__.py
new file mode 100644
index 0000000000000..e69de29bb2d1d
diff --git a/pandas/tests/extension_arrays/base.py b/pandas/tests/extension_arrays/base.py
new file mode 100644
index 0000000000000..2a8a3c314e142
--- /dev/null
+++ b/pandas/tests/extension_arrays/base.py
@@ -0,0 +1,312 @@
+import operator
+
+import numpy as np
+import pytest
+
+import pandas as pd
+import pandas.util.testing as tm
+from pandas.compat import StringIO
+from pandas.core.internals import ExtensionBlock
+from pandas.core.dtypes.common import is_extension_array_dtype
+from pandas.core.dtypes.dtypes import ExtensionDtype
+
+
+class BaseDtypeTests(object):
+    """Base class for ExtensionDtype classes"""
+
+    @pytest.fixture
+    def dtype(self):
+        """A fixture providing the ExtensionDtype to validate."""
+        raise NotImplementedError
+
+    def test_name(self, dtype):
+        assert isinstance(dtype.name, str)
+
+    def test_kind(self, dtype):
+        valid = set('biufcmMOSUV')
+        if dtype.kind is not None:
+            assert dtype.kind in valid
+
+    def test_construct_from_string_own_name(self, dtype):
+        result = dtype.construct_from_string(dtype.name)
+        assert type(result) is type(dtype)
+
+        # check OK as classmethod
+        result = type(dtype).construct_from_string(dtype.name)
+        assert type(result) is type(dtype)
+
+    def test_is_dtype_from_name(self, dtype):
+        result = type(dtype).is_dtype(dtype.name)
+        assert result is True
+
+    def test_is_dtype_from_self(self, dtype):
+        result = type(dtype).is_dtype(dtype)
+        assert result is True
+
+
+class BaseArrayTests(object):
+    """Base class for extension array classes.
+
+    Subclasses should implement the following fixtures
+
+    * data
+    * data_missing
+    """
+
+    @pytest.fixture
+    def data(self):
+        """Length-100 array for this type."""
+        raise NotImplementedError
+
+    @pytest.fixture
+    def data_missing(self):
+        """Length-2 array with [NA, Valid]"""
+        raise NotImplementedError
+
+    @pytest.fixture
+    def na_cmp(self):
+        """Binary operator for comparing NA values.
+
+        Should return a function of two arguments that returns
+        True if both arguments are (scalar) NA for your type.
+
+        By defult, uses ``operator.or``
+        """
+        return operator.is_
+
+    def test_len(self, data):
+        assert len(data) == 100
+
+    def test_ndim(self, data):
+        assert data.ndim == 1
+
+    def test_can_hold_na_valid(self, data):
+        assert data._can_hold_na() in {True, False}
+
+    def test_series_constructor(self, data):
+        result = pd.Series(data)
+        assert result.dtype == data.dtype
+        assert len(result) == len(data)
+        assert isinstance(result._data.blocks[0], ExtensionBlock)
+
+    @pytest.mark.parametrize("from_series", [True, False])
+    def dataframe_constructor(self, data, from_series):
+        if from_series:
+            data = pd.Series(data)
+        result = pd.DataFrame({"A": data})
+        assert result.dtypes['A'] == data.dtype
+        assert result.shape == (len(data), 1)
+        assert isinstance(result._data.blocks[0], ExtensionBlock)
+
+    def test_concat(self, data):
+        result = pd.concat([
+            pd.Series(data),
+            pd.Series(data),
+        ], ignore_index=True)
+        assert len(result) == len(data) * 2
+
+    def test_iloc(self, data):
+        ser = pd.Series(data)
+        result = ser.iloc[:4]
+        expected = pd.Series(data[:4])
+        tm.assert_series_equal(result, expected)
+
+        result = ser.iloc[[0, 1, 2, 3]]
+        tm.assert_series_equal(result, expected)
+
+    def test_loc(self, data):
+        ser = pd.Series(data)
+        result = ser.loc[:3]
+        expected = pd.Series(data[:4])
+        tm.assert_series_equal(result, expected)
+
+        result = ser.loc[[0, 1, 2, 3]]
+        tm.assert_series_equal(result, expected)
+
+    def test_repr(self, data):
+        ser = pd.Series(data)
+        assert data.dtype.name in repr(ser)
+
+        df = pd.DataFrame({"A": data})
+        repr(df)
+
+    def test_dtype_name_in_info(self, data):
+        buf = StringIO()
+        pd.DataFrame({"A": data}).info(buf=buf)
+        result = buf.getvalue()
+        assert data.dtype.name in result
+
+    def test_memory_usage(self, data):
+        s = pd.Series(data)
+        result = s.memory_usage(index=False)
+        assert result == s.nbytes
+
+    def test_is_extension_array_dtype(self, data):
+        assert is_extension_array_dtype(data)
+        assert is_extension_array_dtype(data.dtype)
+        assert is_extension_array_dtype(pd.Series(data))
+        assert isinstance(data.dtype, ExtensionDtype)
+
+    def test_array_interface(self, data):
+        result = np.array(data)
+        assert result[0] == data[0]
+
+    def test_getitem_scalar(self, data):
+        result = data[0]
+        assert isinstance(result, data.dtype.type)
+
+        result = pd.Series(data)[0]
+        assert isinstance(result, data.dtype.type)
+
+    def test_getitem_scalar_na(self, data_missing, na_cmp):
+        result = data_missing[0]
+        assert na_cmp(result, data_missing._fill_value)
+
+    def test_getitem_mask(self, data):
+        # Empty mask, raw array
+        mask = np.zeros(len(data), dtype=bool)
+        result = data[mask]
+        assert len(result) == 0
+        assert isinstance(result, type(data))
+
+        # Empty mask, in series
+        mask = np.zeros(len(data), dtype=bool)
+        result = pd.Series(data)[mask]
+        assert len(result) == 0
+        assert result.dtype == data.dtype
+
+        # non-empty mask, raw array
+        mask[0] = True
+        result = data[mask]
+        assert len(result) == 1
+        assert isinstance(result, type(data))
+
+        # non-empty mask, in series
+        result = pd.Series(data)[mask]
+        assert len(result) == 1
+        assert result.dtype == data.dtype
+
+    def test_getitem_slice(self, data):
+        # getitem[slice] should return an array
+        result = data[slice(0)]  # empty
+        assert isinstance(result, type(data))
+
+        result = data[slice(1)]  # scalar
+        assert isinstance(result, type(data))
+
+    def test_take_sequence(self, data):
+        result = pd.Series(data[[0, 1, 3]])
+        assert result.iloc[0] == data[0]
+        assert result.iloc[1] == data[1]
+        assert result.iloc[2] == data[3]
+
+    def test_isna(self, data_missing):
+        if data_missing._can_hold_na:
+            expected = np.array([True, False])
+        else:
+            expected = np.array([False, False])
+
+        result = pd.isna(data_missing)
+        tm.assert_numpy_array_equal(result, expected)
+
+        result = pd.Series(data_missing).isna()
+        expected = pd.Series(expected)
+        tm.assert_series_equal(result, expected)
+
+    @pytest.mark.parametrize("method", [
+        "mean", "sum", "prod", "mad", "sem", "var", "std",
+        "skew", "kurt", "median"
+    ])
+    def test_nuisance_dropped(self, data, method):
+        data = data[:5]
+        func = operator.methodcaller(method)
+        df = pd.DataFrame({"A": np.arange(len(data)),
+                           "B": data})
+        assert len(func(df)) == 1
+
+    @pytest.mark.parametrize("method", [min, max])
+    def test_reduction_orderable(self, data, method):
+        data = data[:5]
+        func = operator.methodcaller(method.__name__)
+        df = pd.DataFrame({"A": np.arange(len(data)),
+                           "B": data})
+        result = func(df)
+        assert len(result) == 2
+
+        expected = method(data)
+        assert result['B'] == expected
+
+    @pytest.mark.parametrize("method", ['cummax', 'cummin'])
+    @pytest.mark.xfail(reason="Assumes comparable to floating.")
+    def test_cumulative_orderable(self, data, method):
+        # Upcast to object
+        # https://github.com/pandas-dev/pandas/issues/19296
+        # assert result.dtypes['B'] == data.dtype
+        data = data[:5]
+        func = operator.methodcaller(method)
+        df = pd.DataFrame({"A": np.arange(len(data)),
+                           "B": data})
+        result = func(df)
+        assert result.shape == df.shape
+
+    @pytest.mark.parametrize("binop", [
+        operator.add,
+        operator.sub,
+        operator.lt,
+        operator.le,
+        operator.ge,
+        operator.gt,
+        operator.pow,
+    ], ids=lambda x: x.__name__)
+    def test_binops(self, data, binop):
+        # Assert that binops work between DataFrames / Series with this type
+        # if binops work between arrays of this type. Extra tests will be
+        # needed for, e.g., Array + scalar
+        data = data[:5]
+        df = pd.DataFrame({
+            "A": np.arange(len(data)),
+            "B": data
+        })
+
+        try:
+            expected = pd.DataFrame({
+                "A": binop(df['A'], df['A']),
+                "B": binop(df['B'].values, df['B'].values),
+            })
+        except Exception:
+            msg = "{} not supported for {}".format(binop.__name__,
+                                                   data.dtype.name)
+            raise pytest.skip(msg)
+
+        result = binop(df, df)
+        tm.assert_frame_equal(result, expected)
+
+        # homogeneous frame
+        result = binop(df[['B']], df[['B']])
+        tm.assert_frame_equal(result, expected[['B']])
+
+        # series
+        result = binop(df['B'], df['B'])
+        tm.assert_series_equal(result, expected['B'])
+
+    def test_as_ndarray(self, data):
+        np.array(data, dtype=data.dtype.kind)
+
+    def test_align(self, data):
+        a = data[:3]
+        b = data[2:5]
+        r1, r2 = pd.Series(a).align(pd.Series(b, index=[1, 2, 3]))
+
+        # TODO: assumes that the ctor can take a list of scalars of the type
+        e1 = pd.Series(type(data)(list(a) + [data._fill_value]))
+        e2 = pd.Series(type(data)([data._fill_value] + list(b)))
+        tm.assert_series_equal(r1, e1)
+        tm.assert_series_equal(r2, e2)
+
+    @pytest.mark.xfail(reason="GH-19342")
+    def test_series_given_index(self, data):
+        result = pd.Series(data[:3], index=[0, 1, 2, 3, 4])
+        assert result.dtype == data.dtype
+        assert len(result) == 5
+        assert len(result.values) == 5
+        assert pd.isna(result.loc[[3, 4]]).all()
diff --git a/pandas/tests/extension_arrays/test_json.py b/pandas/tests/extension_arrays/test_json.py
new file mode 100644
index 0000000000000..7685c7693b122
--- /dev/null
+++ b/pandas/tests/extension_arrays/test_json.py
@@ -0,0 +1,154 @@
+import collections
+import itertools
+import numbers
+import operator
+import random
+import string
+import sys
+
+import numpy as np
+import pytest
+
+
+from pandas.core.dtypes.base import ExtensionDtype
+from pandas.core.arrays import ExtensionArray
+
+from .base import BaseArrayTests, BaseDtypeTests
+
+
+class JSONDtype(ExtensionDtype):
+    type = collections.Mapping
+    name = 'json'
+
+    @classmethod
+    def construct_from_string(cls, string):
+        if string == cls.name:
+            return cls()
+        else:
+            raise TypeError("Cannot construct a '{}' from "
+                            "'{}'".format(cls, string))
+
+
+class JSONArray(ExtensionArray):
+    dtype = JSONDtype()
+
+    def __init__(self, values):
+        for val in values:
+            if not isinstance(val, collections.Mapping):
+                raise TypeError
+        self.data = values
+
+    def __getitem__(self, item):
+        # TDOO: fancy indexing
+        if isinstance(item, numbers.Integral):
+            return self.data[item]
+        elif isinstance(item, np.ndarray) and item.dtype == 'bool':
+            return type(self)([x for x, m in zip(self, item) if m])
+        elif isinstance(item, collections.Sequence):
+            return type(self)([self.data[i] for i in item])
+        else:
+            return type(self)(self.data[item])
+
+    def __len__(self):
+        return len(self.data)
+
+    def __iter__(self):
+        return iter(self.data)
+
+    def __repr__(self):
+        return 'JSONArary({!r})'.format(self.data)
+
+    @property
+    def nbytes(self):
+        return sys.getsizeof(self.data)
+
+    def isna(self):
+        return np.array([x == {} for x in self.data])
+
+    def take(self, indexer, allow_fill=True, fill_value=None):
+        output = [self.data[loc] if loc != -1 else {}
+                  for loc in indexer]
+        return type(self)(output)
+
+    def copy(self, deep=False):
+        return type(self)(self.data.copy(deep=deep))
+
+    @property
+    def _fill_value(self):
+        return {}
+
+    @classmethod
+    def _concat_same_type(cls, to_concat):
+        data = list(itertools.chain.from_iterable([x.data for x in to_concat]))
+        return cls(data)
+
+
+
+def make_data():
+    return [{random.choice(string.ascii_letters): random.randint(0, 100)
+             for _ in range(random.randint(0, 10))}
+             for _ in range(100)]
+
+
+class TestJSONDtype(BaseDtypeTests):
+    @pytest.fixture
+    def dtype(self):
+        return JSONDtype()
+
+
+class TestJSON(BaseArrayTests):
+
+    @pytest.fixture
+    def data(self):
+        """Length-100 PeriodArray for semantics test."""
+        return JSONArray(make_data())
+
+    @pytest.fixture
+    def data_missing(self):
+        """Length 2 array with [NA, Valid]"""
+        return JSONArray([{}, {'a': 10}])
+
+    @pytest.fixture
+    def na_cmp(self):
+        return operator.eq
+
+    @pytest.mark.skip(reason="Unorderable")
+    def test_reduction_orderable(self, data, method):
+        pass
+
+    
+
+# def test_concat_mixed_closed_raises():
+#     one = IntervalArray.from_breaks([0, 1, 2], closed='left')
+#     two = IntervalArray.from_breaks([1, 2, 3], closed='right')
+#
+#     with tm.assert_raises_regex(ValueError, "Intervals must all be closed"):
+#         IntervalArray._concat_same_type([one, two])
+#
+#
+# def test_series_constructor_intervalindex():
+#     result = pd.Series(pd.IntervalIndex.from_breaks([0, 1, 2]))
+#     assert result.dtype == 'interval[int64]'
+#
+#
+# def dataframe_constructor_intervalindex():
+#     result = pd.DataFrame({"A": pd.IntervalIndex.from_breaks([0, 1, 2])})
+#     assert result.dtypes['A'] == 'interval[int64]'
+#
+#
+# def dataframe_set_intervalarray():
+#     df = pd.DataFrame({"A": [1, 2]})
+#     arr = IntervalArray.from_breaks([0, 1, 2])
+#     df['B'] = arr
+#
+#     expected = pd.DataFrame({"A": [1, 2], "B": arr})
+#     tm.assert_frame_equal(df, expected)
+#
+#
+# def dataframe_set_intervalindex():
+#     df = pd.DataFrame({"A": [1, 2]})
+#     arr = pd.IntervalIndex.from_breaks([0, 1, 2])
+#     df['B'] = arr
+#
+#     expected = pd.DataFrame({"A": [1, 2], "B": arr})
+#     tm.assert_frame_equal(df, expected)
diff --git a/pandas/tests/indexes/datetimelike.py b/pandas/tests/indexes/datetimelike.py
index 7d01a2a70145d..64fc1ee8c9680 100644
--- a/pandas/tests/indexes/datetimelike.py
+++ b/pandas/tests/indexes/datetimelike.py
@@ -83,3 +83,16 @@ def test_asobject_deprecated(self):
         with tm.assert_produces_warning(FutureWarning):
             i = d.asobject
         assert isinstance(i, pd.Index)
+
+    def test_as_best_array(self):
+        result = pd.DatetimeIndex(['2017-01-01T00:00:00',
+                                   '2017-01-02T00:00:00'])._as_best_array()
+        expected = np.array(['2017-01-01T00:00:00',
+                             '2017-01-02T00:00:00'], dtype='M8[ns]')
+        tm.assert_numpy_array_equal(result, expected)
+
+    def test_as_best_array_tz(self):
+        arr = pd.DatetimeIndex(['2017-01-01T00:00:00',
+                                '2017-01-02T00:00:00'], tz='US/Central')
+        result = arr._as_best_array()
+        tm.assert_index_equal(arr, result)
diff --git a/pandas/tests/indexes/test_base.py b/pandas/tests/indexes/test_base.py
index 974099f1fbbe9..7e90260ae62c7 100644
--- a/pandas/tests/indexes/test_base.py
+++ b/pandas/tests/indexes/test_base.py
@@ -2282,6 +2282,11 @@ def test_comparison_tzawareness_compat(self, op):
         # TODO: implement _assert_tzawareness_compat for the reverse
         # comparison with the Series on the left-hand side
 
+    def test_as_best_array(self):
+        result = pd.Index([0, 1, 2])._as_best_array()
+        expected = np.array([0, 1, 2], dtype=np.int64)
+        tm.assert_numpy_array_equal(result, expected)
+
 
 class TestIndexUtils(object):
 
diff --git a/pandas/tests/indexes/test_category.py b/pandas/tests/indexes/test_category.py
index c2e40c79f8914..54dd0851ac2d5 100644
--- a/pandas/tests/indexes/test_category.py
+++ b/pandas/tests/indexes/test_category.py
@@ -1080,3 +1080,8 @@ def test_take_invalid_kwargs(self):
         msg = "the 'mode' parameter is not supported"
         tm.assert_raises_regex(ValueError, msg, idx.take,
                                indices, mode='clip')
+
+    def test_as_best_array(self):
+        result = pd.CategoricalIndex([0, 1, 2])._as_best_array()
+        expected = pd.Categorical([0, 1, 2])
+        tm.assert_categorical_equal(result, expected)


Code review comments:
diff_hunk: @@ -18,19 +20,20 @@ class ExtensionArray(object):
 
     * __getitem__
     * __len__
+    * __iter__
path: pandas/core/arrays/base.py
text: @author:
@user1 we removed this earlier since collections implementing `__len__` and `__getitem__` are iterable. But (strangely?) `isinstance(arr, collections.Iterable)` is False, so these aren't considered "list-like" unless they actually implement `__iter__`.

@user1:
But do we use this somewhere? (the expectation that isinstance(arr, collections.Iterable) is True) 

As the docs https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterable says this is not a correct way to test that something is iterable, exactly for the reason you just mentioned.

Anyway, if we add it, I would add a default implementation (`return iter(self)` ?) and not require subclasses to provide one manually.

@author:
> But do we use this somewhere?

Yes, sorry. We use it in `Series.__init__`, via `is_list_like(data)`.

Will add the default implementation.

@author:
Wait, `iter(self)` will cause infinite recursion, and we can't assume anything about how the data is stored. That's unfortunate.

@author:
I guess we could fall back to

```python
for i in range(len(self)):
   yield self[i]
```

@user1:
> Yes, sorry. We use it in Series.__init__, via is_list_like(data).

Or we could fix `is_list_like` ..

> Wait, iter(self) will cause infinite recursion, and we can't assume anything about how the data is stored.

Why would it cause infinite recursion? 
With `iter(self)` we don't assume anything about how the data is stored, I don't know how iter exactly works under the hood, but I assume it just uses `__getitem__` ?

@author:
> Or we could fix is_list_like 

Prior to https://github.com/pandas-dev/pandas/pull/17654 we did that, but it was fragile to things like `str` (the class, not an instance) being considered iterable. Not sure if changing it back is an improvement.

> Why would it cause infinite recursion?

```
def __iter__(self):
    return iter(self)
```

will call `self.__iter__`, which calls `self.__iter__`, ...

@user1:
ah, yes of course :-) 
Anyhow, the other way is a fine default implementation