Bugfix for constructing empty Series, with index, using ExtensionDtyp…
State: closed

…e with na_value of None

- [x] closes #44602 
- [ ] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Bug
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
Hello @venaturum! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2022-03-01 00:18:02 UTC
@phofl a test for this would require defining an ExtensionDtype and ExtensionArray which is quite a lot of code to be adding.  Are you aware of any existing "dummy" classes in the test suites that could be used?
Just use the example from the issue?
Although the example works for demonstrating the recursion error it doesn't end up giving the expected result due to implementations in the IntervalArray class.

I'll have a play and see how minimal I can make a test.
@phofl 

This is as small as I can make it

```
import pandas as pd
from pandas.testing import assert_series_equal
import numbers
from collections.abc import Iterable
from pandas.api.extensions import (
    ExtensionArray,
    ExtensionDtype,
    register_extension_dtype,
)

def test_empty_series_construction_with_na_value_None():

    class DummyClass:

        def __init__(self):
            pass

    @register_extension_dtype
    class DummyDtype(ExtensionDtype):

        type = DummyClass
        name = "dummy"
        na_value = None

        @classmethod
        def construct_from_string(cls, string):
            if string == cls.name:
                return cls()
            else:
                raise TypeError("Cannot construct a '{}' from " "'{}'".format(cls, string))

        @classmethod
        def construct_array_type(cls):
            return DummyArray


    class DummyArray(ExtensionArray):

        _dtype = DummyDtype()

        def __init__(self, values):
            self._values = values

        @property
        def dtype(self):
            return self._dtype

        def __len__(self):
            return len(self._values)

        def __getitem__(self, idx):
            if isinstance(idx, numbers.Integral):
                return self._values[idx]
            elif isinstance(idx, (Iterable, slice)):
                return DummyArray(self._values[idx])
            else:
                raise TypeError("Index type not supported", idx)

        @classmethod
        def _from_sequence(cls, scalars, dtype=None, copy=False):
            if isinstance(scalars, DummyClass):
                scalars = [scalars]
            return DummyArray(scalars)

    result = pd.Series(index=[1, 2, 3], dtype="dummy")
    expected = pd.Series([None] * 3, index=[1, 2, 3], dtype="dummy")
    assert_series_equal(result, expected)
 ```
 
 I would need some guidance on where it in the test suite it belongs.
Int64Dtype works for example
@phofl 

Unfortunately it won't pass the test, for the same reasons as IntervalArray

```
import pandas as pd
from pandas import Int64Dtype

Int64Dtype.na_value = None
dtype = Int64Dtype()

pd.Series(index=[1,2,3], dtype=dtype)
```

This results in 

```
1    <NA>
2    <NA>
3    <NA>
dtype: Int64
```

when the desired result is 

```
1    None
2    None
3    None
dtype: Int64
```


@jreback would it be better handled within the following function? https://github.com/pandas-dev/pandas/blob/fd9502654c1d35465262bb5d110a587d3310c0cc/pandas/core/construction.py#L807
> Unfortunately it won't pass the test, for the same reasons as IntervalArray [...]

Yah it'd be pretty convenient if subclasses could change Int64Dtype.na_value to, say, np.nan, which would be way less fragile.
closing as this needs tests and new implementation. ping when ready / open a new PR
@jreback 

Hey Jeff, the [test here](https://github.com/pandas-dev/pandas/pull/44615#issuecomment-979172694) does the job, I just wish I could make it slimmer but I think that would require a different approach 

As for a new implementation I'm happy to try something else but [need guidance on where in the code it belongs](https://github.com/pandas-dev/pandas/pull/44615#issuecomment-979465234).

Cheers.
@venaturum not averse to a test like that

but should be included as a first step
> @venaturum not averse to a test like that
> 
> but should be included as a first step

Would the test be best placed in [`pandas\tests\extension\test_common.py` ?](https://github.com/pandas-dev/pandas/blob/master/pandas/tests/extension/test_common.py) ?
> Would the test be best placed in pandas\tests\extension\test_common.py ? ?

try tests.extension.base.constructors
Note there is a test here https://github.com/pandas-dev/pandas/blob/de290f76815840549f273baa6ed7ba751dede871/pandas/tests/extension/base/constructors.py#L38 which tests what we need it to, it just needs to be used in conjunction with a Dtype with None na_value.  Perhaps all the tests in this suite should be tested with an appropriate Dtype?
@jreback @jbrockmendel 

A different approach to the tests can be found here

https://github.com/venaturum/pandas/blob/GH44602_dtype_na_value/pandas/tests/extension/test_xx.py

What is different:
- existing test code is left alone
- the BaseConstructorsTests suite is ran over the custom ExtensionArray, which includes a test for #GH44602
- only `na_value=None` is being tested as opposed to alternatives such as pd.NA or np.nan

If this is a better approach I will abandon this PR and start again, or if there is some better way of integrating this approach into the existing tests then please let me know.  Alternatively, I'm happy for someone else to take this and run with it.

Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, ping the current reviewers, and we can reopen.