Repository: pandas-dev/pandas
Pull request: BUG: Series construction with EA dtype and index but no data fails
- [ ] closes #26469
- [ ] closes #33559
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Problem statement:
BUG: Series construction with EA dtype and index but no data fails
Using master (0.25.0dev):

```
In [8]: pd.Series(None, index=[1, 2, 3], dtype='Int64')                                                                                                                                                             
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
~/scipy/pandas/pandas/core/internals/construction.py in _try_cast(arr, take_fast_path, dtype, copy, raise_cast_failure)
    691         if is_integer_dtype(dtype):
--> 692             subarr = maybe_cast_to_integer_array(arr, dtype)
    693 

~/scipy/pandas/pandas/core/dtypes/cast.py in maybe_cast_to_integer_array(arr, dtype, copy)
   1311         if not hasattr(arr, "astype"):
-> 1312             casted = np.array(arr, dtype=dtype, copy=copy)
   1313         else:

TypeError: data type not understood

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
<ipython-input-8-9447295feee6> in <module>
----> 1 pd.Series(None, index=[1, 2, 3], dtype='Int64')

~/scipy/pandas/pandas/core/series.py in __init__(self, data, index, dtype, name, copy, fastpath)
    202                 data = data._data
    203             elif isinstance(data, dict):
--> 204                 data, index = self._init_dict(data, index, dtype)
    205                 dtype = None
    206                 copy = False

~/scipy/pandas/pandas/core/series.py in _init_dict(self, data, index, dtype)
    295 
    296         # Input is now list-like, so rely on "standard" construction:
--> 297         s = Series(values, index=keys, dtype=dtype)
    298 
    299         # Now we just make sure the order is respected, if any

~/scipy/pandas/pandas/core/series.py in __init__(self, data, index, dtype, name, copy, fastpath)
    253             else:
    254                 data = sanitize_array(data, index, dtype, copy,
--> 255                                       raise_cast_failure=True)
    256 
    257                 data = SingleBlockManager(data, index, fastpath=True)

~/scipy/pandas/pandas/core/internals/construction.py in sanitize_array(data, index, dtype, copy, raise_cast_failure)
    620         subarr = _try_cast(arr, False, dtype, copy, raise_cast_failure)
    621     else:
--> 622         subarr = _try_cast(data, False, dtype, copy, raise_cast_failure)
    623 
    624     # scalar like, GH

~/scipy/pandas/pandas/core/internals/construction.py in _try_cast(arr, take_fast_path, dtype, copy, raise_cast_failure)
    711             # create an extension array from its dtype
    712             array_type = dtype.construct_array_type()._from_sequence
--> 713             subarr = array_type(arr, dtype=dtype, copy=copy)
    714         elif dtype is not None and raise_cast_failure:
    715             raise

~/scipy/pandas/pandas/core/arrays/integer.py in _from_sequence(cls, scalars, dtype, copy)
    305     @classmethod
    306     def _from_sequence(cls, scalars, dtype=None, copy=False):
--> 307         return integer_array(scalars, dtype=dtype, copy=copy)
    308 
    309     @classmethod

~/scipy/pandas/pandas/core/arrays/integer.py in integer_array(values, dtype, copy)
    110     TypeError if incompatible types
    111     """
--> 112     values, mask = coerce_to_array(values, dtype=dtype, copy=copy)
    113     return IntegerArray(values, mask)
    114 

~/scipy/pandas/pandas/core/arrays/integer.py in coerce_to_array(values, dtype, mask, copy)
    202 
    203     if not values.ndim == 1:
--> 204         raise TypeError("values must be a 1D list-like")
    205     if not mask.ndim == 1:
    206         raise TypeError("mask must be a 1D list-like")

TypeError: values must be a 1D list-like
```

while this works fine for non-EA dtypes. 

The problem is that the None / np.nan is eventually passed to the `_from_sequence` method, which expects a list-like. I don't think we should let people fix their `_from_sequence`, but we should rather make sure we never pass that to it.

BUG: Series.apply() throws an error on empty series when dtype = pd.StringDtype
- [x] I have checked that this issue has not already been reported.

- [x] I have confirmed this bug exists on the latest version of pandas.

- [ ] (optional) I have confirmed this bug exists on the master branch of pandas.

---

#### Code Sample, a copy-pastable example

```python
pd.Series(['a', 'b', 'c']).iloc[0:0].apply(lambda x: x) #works as expected
pd.Series(['a', 'b', 'c']).astype('string').iloc[0:0].apply(lambda x: x) #throws an exception
```

#### Problem description
Applying a function to an empty Series with dtype `pd.StringDtype` throws `ValueError: PandasArray must be 1-dimensional.`.

#### Expected Output
Apply does nothing.

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit           : None
python           : 3.8.1.final.0
python-bits      : 64
OS               : Linux
OS-release       : 5.6.3-arch1-1
machine          : x86_64
processor        : 
byteorder        : little
LC_ALL           : None
LANG             : en_US.UTF-8
LOCALE           : en_US.UTF-8

pandas           : 1.0.3
numpy            : 1.18.2
pytz             : 2019.3
dateutil         : 2.8.1
pip              : 20.0.2
setuptools       : 41.2.0
Cython           : None
pytest           : None
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : 2.8.5 (dt dec pq3 ext lo64)
jinja2           : 2.11.2
IPython          : 7.13.0
pandas_datareader: None
bs4              : None
bottleneck       : None
fastparquet      : None
gcsfs            : None
lxml.etree       : None
matplotlib       : None
numexpr          : None
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : None
pytables         : None
pytest           : None
pyxlsb           : None
s3fs             : None
scipy            : None
sqlalchemy       : None
tables           : None
tabulate         : None
xarray           : None
xlrd             : None
xlwt             : None
xlsxwriter       : None
numba            : None
</details>

Hints from issue discussion:
This problem also manifests when doing an `apply` on a Series of extended-array dtype, e.g.:

```
import pandas as pd
pd.Series([], dtype="Int64").apply(lambda f: f)
```

The reason is that the `apply` code contains a shortcut for the empty series case, which constructs a new series in exactly the problematic way described in this issue: with an EA datatype and an index, but no data:

https://github.com/pandas-dev/pandas/blob/2ae5152690cea598a7f377eb4cb264d6d42bb0cc/pandas/core/series.py#L3947-L3950

I'm on pandas 0.24.2
@vanschelven interested in working on this? I think the fix is to call `ExtensionArray._from_sequence([])` rather than `ExtensionArray._from_sequence(None)`.
I'm willing to do some work on this, but setting up a full development environment for pandas is a bit much for me now. I've made a "shoot from the hip" (completely untested)  PR #27475 with what I _think_ could be the solution though.
The problem with your suggestion is that at the moment of failure `arr` is already not an array (its a scalar, in particular `np.nan`). That is, there is no line `ExtensionArray._from_sequence(None)` to replace. In the PR referenced above I've gone to what I understand to be the moment `np.nan` gets introduced and removed that behavior.
As evidenced from the PR's travis logs, that's not a solution, so I'm letting this rest for now
Thanks for the report. Slightly smaller reproducer

```python
pd.Series(dtype="string", index=[])
```

We try to insert `np.array(<NA>)`.

In `_init_dict`

```python
    354         elif index is not None:
    355             # fastpath for Series(data=None). Just use broadcasting a scalar
    356             # instead of reindexing.
    357             values = na_value_for_dtype(dtype)
    358             keys = index
 ```

I'm not sure if that would need to change (special case length-0?) or whether `create_series_with_explicit_dtype` should change.

Commit message: add test_series_constructor_scalar_with_one_element_index

Patch to review:
diff --git a/doc/source/whatsnew/v1.1.0.rst b/doc/source/whatsnew/v1.1.0.rst
index 597a0c5386cf0..08b6abb39660e 100644
--- a/doc/source/whatsnew/v1.1.0.rst
+++ b/doc/source/whatsnew/v1.1.0.rst
@@ -732,7 +732,7 @@ ExtensionArray
 ^^^^^^^^^^^^^^
 
 - Fixed bug where :meth:`Series.value_counts` would raise on empty input of ``Int64`` dtype (:issue:`33317`)
--
+- Fixed bug in :class:`Series` construction with EA dtype and index but no data fails (:issue:`26469`)
 
 
 Other
diff --git a/pandas/core/construction.py b/pandas/core/construction.py
index 351ef1d0429da..3c5a5faeeb71e 100644
--- a/pandas/core/construction.py
+++ b/pandas/core/construction.py
@@ -516,7 +516,7 @@ def _try_cast(
 
     Parameters
     ----------
-    arr : ndarray, list, tuple, iterator (catchall)
+    arr : ndarray, scalar, list, tuple, iterator (catchall)
         Excludes: ExtensionArray, Series, Index.
     dtype : np.dtype, ExtensionDtype or None
     copy : bool
@@ -533,6 +533,10 @@ def _try_cast(
     if isinstance(dtype, ExtensionDtype) and dtype.kind != "M":
         # create an extension array from its dtype
         # DatetimeTZ case needs to go through maybe_cast_to_datetime
+
+        if lib.is_scalar(arr):
+            arr = [arr]
+
         array_type = dtype.construct_array_type()._from_sequence
         subarr = array_type(arr, dtype=dtype, copy=copy)
         return subarr
diff --git a/pandas/tests/extension/arrow/test_bool.py b/pandas/tests/extension/arrow/test_bool.py
index 94dd09d3eb053..7e3c9c6522697 100644
--- a/pandas/tests/extension/arrow/test_bool.py
+++ b/pandas/tests/extension/arrow/test_bool.py
@@ -55,6 +55,16 @@ def test_from_dtype(self, data):
     def test_from_sequence_from_cls(self, data):
         super().test_from_sequence_from_cls(data)
 
+    @pytest.mark.xfail(reason="GH-26469")
+    def test_series_constructor_no_data_with_index(self, data, na_value):
+        # pyarrow.lib.ArrowInvalid: only handle 1-dimensional arrays
+        super().test_series_constructor_no_data_with_index(data, na_value)
+
+    @pytest.mark.xfail(reason="GH-26469")
+    def test_series_constructor_scalar_na_with_index(self, data, na_value):
+        # pyarrow.lib.ArrowInvalid: only handle 1-dimensional arrays
+        super().test_series_constructor_scalar_na_with_index(data, na_value)
+
 
 class TestReduce(base.BaseNoReduceTests):
     def test_reduce_series_boolean(self):
diff --git a/pandas/tests/extension/base/constructors.py b/pandas/tests/extension/base/constructors.py
index 1ddc7af0f6268..a55379b96a198 100644
--- a/pandas/tests/extension/base/constructors.py
+++ b/pandas/tests/extension/base/constructors.py
@@ -33,6 +33,32 @@ def test_series_constructor(self, data):
         assert result2.dtype == data.dtype
         assert isinstance(result2._mgr.blocks[0], ExtensionBlock)
 
+    def test_series_constructor_no_data_with_index(self, data, na_value):
+        dtype = data.dtype
+        result = pd.Series(index=[1, 2, 3], dtype=dtype)
+        expected = pd.Series([na_value] * 3, index=[1, 2, 3], dtype=dtype)
+        self.assert_series_equal(result, expected)
+
+    def test_series_constructor_scalar_na_with_index(self, data, na_value):
+        dtype = data.dtype
+        result = pd.Series(na_value, index=[1, 2, 3], dtype=dtype)
+        expected = pd.Series([na_value] * 3, index=[1, 2, 3], dtype=dtype)
+        self.assert_series_equal(result, expected)
+
+    def test_series_constructor_scalar_with_index(self, data):
+        scalar = data[0]
+        dtype = data.dtype
+        result = pd.Series(scalar, index=[1, 2, 3], dtype=dtype)
+        expected = pd.Series([scalar] * 3, index=[1, 2, 3], dtype=dtype)
+        self.assert_series_equal(result, expected)
+
+    def test_series_constructor_scalar_with_one_element_index(self, data):
+        scalar = data[0]
+        dtype = data.dtype
+        result = pd.Series(scalar, index=["foo"], dtype=dtype)
+        expected = pd.Series([scalar], index=["foo"], dtype=dtype)
+        self.assert_series_equal(result, expected)
+
     @pytest.mark.parametrize("from_series", [True, False])
     def test_dataframe_constructor_from_dict(self, data, from_series):
         if from_series:
diff --git a/pandas/tests/extension/json/test_json.py b/pandas/tests/extension/json/test_json.py
index 06b59aaeff68d..dd117e61cc5d9 100644
--- a/pandas/tests/extension/json/test_json.py
+++ b/pandas/tests/extension/json/test_json.py
@@ -150,6 +150,26 @@ def test_from_dtype(self, data):
         # construct from our dtype & string dtype
         pass
 
+    @pytest.mark.xfail(reason="GH-26469")
+    def test_series_constructor_no_data_with_index(self, data, na_value):
+        # RecursionError: maximum recursion depth exceeded in comparison
+        super().test_series_constructor_no_data_with_index(data, na_value)
+
+    @pytest.mark.xfail(reason="GH-26469")
+    def test_series_constructor_scalar_na_with_index(self, data, na_value):
+        # RecursionError: maximum recursion depth exceeded in comparison
+        super().test_series_constructor_scalar_na_with_index(data, na_value)
+
+    @pytest.mark.xfail(reason="GH-26469")
+    def test_series_constructor_scalar_with_index(self, data):
+        # TypeError: All values must be of type <class 'collections.abc.Mapping'>
+        super().test_series_constructor_scalar_with_index(data)
+
+    @pytest.mark.xfail(reason="GH-26469")
+    def test_series_constructor_scalar_with_one_element_index(self, data):
+        # TypeError: All values must be of type <class 'collections.abc.Mapping'>
+        super().test_series_constructor_scalar_with_one_element_index(data)
+
 
 class TestReshaping(BaseJSON, base.BaseReshapingTests):
     @pytest.mark.skip(reason="Different definitions of NA")
diff --git a/pandas/tests/extension/test_datetime.py b/pandas/tests/extension/test_datetime.py
index e026809f7e611..45364bbe43b53 100644
--- a/pandas/tests/extension/test_datetime.py
+++ b/pandas/tests/extension/test_datetime.py
@@ -82,7 +82,15 @@ class TestDatetimeDtype(BaseDatetimeTests, base.BaseDtypeTests):
 
 
 class TestConstructors(BaseDatetimeTests, base.BaseConstructorsTests):
-    pass
+    @pytest.mark.xfail(reason="GH-26469")
+    def test_series_constructor_scalar_with_index(self, data):
+        # TypeError: data type not understood
+        super().test_series_constructor_scalar_with_index(data)
+
+    @pytest.mark.xfail(reason="GH-26469")
+    def test_series_constructor_scalar_with_one_element_index(self, data):
+        # TypeError: data type not understood
+        super().test_series_constructor_scalar_with_one_element_index(data)
 
 
 class TestGetitem(BaseDatetimeTests, base.BaseGetitemTests):
diff --git a/pandas/tests/extension/test_numpy.py b/pandas/tests/extension/test_numpy.py
index aa5a99282131a..89dcf752a3a5f 100644
--- a/pandas/tests/extension/test_numpy.py
+++ b/pandas/tests/extension/test_numpy.py
@@ -151,6 +151,16 @@ def test_array_from_scalars(self, data):
         # ValueError: PandasArray must be 1-dimensional.
         super().test_array_from_scalars(data)
 
+    @skip_nested
+    def test_series_constructor_scalar_with_index(self, data):
+        # ValueError: Length of passed values is 1, index implies 3.
+        super().test_series_constructor_scalar_with_index(data)
+
+    @skip_nested
+    def test_series_constructor_scalar_with_one_element_index(self, data):
+        # ValueError: PandasArray must be 1-dimensional.
+        super().test_series_constructor_scalar_with_one_element_index(data)
+
 
 class TestDtype(BaseNumPyTests, base.BaseDtypeTests):
     @pytest.mark.skip(reason="Incorrect expected.")
diff --git a/pandas/tests/extension/test_sparse.py b/pandas/tests/extension/test_sparse.py
index 694bbee59606f..8466d3485f64c 100644
--- a/pandas/tests/extension/test_sparse.py
+++ b/pandas/tests/extension/test_sparse.py
@@ -112,7 +112,15 @@ def test_view(self, data):
 
 
 class TestConstructors(BaseSparseTests, base.BaseConstructorsTests):
-    pass
+    @pytest.mark.xfail(reason="GH-26469", strict=False)
+    def test_series_constructor_no_data_with_index(self, data, na_value):
+        # ValueError: Cannot convert non-finite values (NA or inf) to integer
+        super().test_series_constructor_no_data_with_index(data, na_value)
+
+    @pytest.mark.xfail(reason="GH-26469", strict=False)
+    def test_series_constructor_scalar_na_with_index(self, data, na_value):
+        # ValueError: Cannot convert non-finite values (NA or inf) to integer
+        super().test_series_constructor_scalar_na_with_index(data, na_value)
 
 
 class TestReshaping(BaseSparseTests, base.BaseReshapingTests):


Code review comments:
diff_hunk: @@ -533,6 +533,10 @@ def _try_cast(
     if isinstance(dtype, ExtensionDtype) and dtype.kind != "M":
         # create an extension array from its dtype
         # DatetimeTZ case needs to go through maybe_cast_to_datetime
+
+        if lib.is_scalar(arr):
path: pandas/core/construction.py
text: @author:
we could look into how to identify a collection that could be considered a 'scalar' for some EA, eg JSONDtype. although I think out-of-scope for the issue that this PR attempts to fix (i.e. IntegerArray, where the scalars are scalars)

@user1:
rather than this I would call: `construct_1d_arraylike_from_scalar`

but I wouldn't do this right here, rather on L453, e.g. add an elif is_scalar(data)

@author:
that's option 3 in https://github.com/pandas-dev/pandas/pull/33846#issuecomment-621736781

do this just for EA types and keep the code path the same for non-EA types?

@user1:
no this will work generically

@author:
i'm getting a few failures in pandas/tests/series/test_constructors.py. i'll push the change anyway use the ci to see what else fails while I investigate. 

@user1:
kk

diff_hunk: @@ -33,6 +33,32 @@ def test_series_constructor(self, data):
         assert result2.dtype == data.dtype
         assert isinstance(result2._mgr.blocks[0], ExtensionBlock)
 
+    def test_series_constructor_no_data_with_index(self, data, na_value):
+        dtype = data.dtype
+        result = pd.Series(index=[1, 2, 3], dtype=dtype)
+        expected = pd.Series([na_value] * 3, index=[1, 2, 3], dtype=dtype)
+        self.assert_series_equal(result, expected)
+
+    def test_series_constructor_scalar_na_with_index(self, data, na_value):
+        dtype = data.dtype
+        result = pd.Series(na_value, index=[1, 2, 3], dtype=dtype)
+        expected = pd.Series([na_value] * 3, index=[1, 2, 3], dtype=dtype)
+        self.assert_series_equal(result, expected)
+
+    def test_series_constructor_scalar_with_index(self, data):
+        scalar = data[0]
+        dtype = data.dtype
+        result = pd.Series(scalar, index=[1, 2, 3], dtype=dtype)
+        expected = pd.Series([scalar] * 3, index=[1, 2, 3], dtype=dtype)
+        self.assert_series_equal(result, expected)
+
+    def test_series_constructor_scalar_with_one_element_index(self, data):
path: pandas/tests/extension/base/constructors.py
text: @user1:
I would maybe just combine this one with the test above (both are about scalar with index) ?

(and one test less to override in the subclasses)