Repository: pandas-dev/pandas

Issue:
ExtensionArray.__fillna__ should not use lib.is_scalar
`lib.is_scalar` is a bit too strict for the check we do in https://github.com/pandas-dev/pandas/blob/7273ea0709590e6264607f227bb8def0ef656c50/pandas/core/arrays/base.py#L272. For example, it doesn't condisder `ipaddress.IPv4Address` as a scalar.

```python
In [1]: from pandas._libs import lib

In [2]: import ipaddress

In [3]: lib.is_scalar(ipaddress.IPv4Address(1))
Out[3]: False
```

Replacing that check with a `hasattr(value, '__len__')` should be sufficient.



Discussion:
The same for geopandas.

However, in general, also the `len` check might not always be good, because eg for geopandas a MultiLineString has a length .. . Or for the json-array, a dict also has a length.

This might not be easy to solve in general, it's mainly due to our flexible API where we need to interpret a lot about what is passed in ..
Actually, even simpler example where a length check will not work: strings. 
(but of course, for only this one it would be easy to add to an additional check)
Hmm hadn't thought that through.

Perhaps we require that `ExtensionArray.fillna(array)` requires `array` to be an instance of the ExtensionArray? That should remove all ambiguity, at the cost of requiring `IPArray([]).fillna(['192.168.1.1'])` to be written as `IPArray([]).fillna(IPArray(['192.168.1.1']))`.
> at the cost of requiring IPArray([]).fillna(['192.168.1.1']) to be written as IPArray([]).fillna(IPArray(['192.168.1.1'])).

But in this case you can still fill with `IPArray([]).fillna('192.168.1.1')` (filling with a scalar) ?

But such a requirement might be a good idea. Or we could also check it being "array-like" (check if is has a shape and if that shape matches). 
In the current implementation, I don't think that a list (instead of array) would work anyhow, because we apply a boolean mask on it?
is_array_like seems to be what we want, since we do use boolean masking on
it.

On Mon, Mar 19, 2018 at 12:13 PM, Joris Van den Bossche <
notifications@github.com> wrote:

> at the cost of requiring IPArray([]).fillna(['192.168.1.1']) to be
> written as IPArray([]).fillna(IPArray(['192.168.1.1'])).
>
> But in this case you can still fill with IPArray([]).fillna('192.168.1.1')
> (filling with a scalar) ?
>
> But such a requirement might be a good idea. Or we could also check it
> being "array-like" (check if is has a shape and if that shape matches).
> In the current implementation, I don't think that a list (instead of
> array) would work anyhow, because we apply a boolean mask on it?
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/issues/20411#issuecomment-374292155>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIrUbFUoz-ormBr7bN5A8PzEJNsdVks5tf-c3gaJpZM4SwWBZ>
> .
>


Resolving patch:
diff --git a/pandas/core/arrays/base.py b/pandas/core/arrays/base.py
--- a/pandas/core/arrays/base.py
+++ b/pandas/core/arrays/base.py
@@ -261,7 +261,7 @@ def fillna(self, value=None, method=None, limit=None):
         -------
         filled : ExtensionArray with NA/NaN filled
         """
-        from pandas.api.types import is_scalar
+        from pandas.api.types import is_array_like
         from pandas.util._validators import validate_fillna_kwargs
         from pandas.core.missing import pad_1d, backfill_1d
 
@@ -269,7 +269,7 @@ def fillna(self, value=None, method=None, limit=None):
 
         mask = self.isna()
 
-        if not is_scalar(value):
+        if is_array_like(value):
             if len(value) != len(self):
                 raise ValueError("Length of 'value' does not match. Got ({}) "
                                  " expected {}".format(len(value), len(self)))


Test patch:
diff --git a/pandas/tests/extension/base/base.py b/pandas/tests/extension/base/base.py
--- a/pandas/tests/extension/base/base.py
+++ b/pandas/tests/extension/base/base.py
@@ -4,3 +4,6 @@
 class BaseExtensionTests(object):
     assert_series_equal = staticmethod(tm.assert_series_equal)
     assert_frame_equal = staticmethod(tm.assert_frame_equal)
+    assert_extension_array_equal = staticmethod(
+        tm.assert_extension_array_equal
+    )
diff --git a/pandas/tests/extension/base/missing.py b/pandas/tests/extension/base/missing.py
--- a/pandas/tests/extension/base/missing.py
+++ b/pandas/tests/extension/base/missing.py
@@ -47,6 +47,12 @@ def test_dropna_frame(self, data_missing):
         expected = df.iloc[:0]
         self.assert_frame_equal(result, expected)
 
+    def test_fillna_scalar(self, data_missing):
+        valid = data_missing[1]
+        result = data_missing.fillna(valid)
+        expected = data_missing.fillna(valid)
+        self.assert_extension_array_equal(result, expected)
+
     def test_fillna_limit_pad(self, data_missing):
         arr = data_missing.take([1, 0, 0, 0, 1])
         result = pd.Series(arr).fillna(method='ffill', limit=2)
diff --git a/pandas/util/testing.py b/pandas/util/testing.py
--- a/pandas/util/testing.py
+++ b/pandas/util/testing.py
@@ -20,6 +20,7 @@
 import numpy as np
 
 import pandas as pd
+from pandas.core.arrays import ExtensionArray
 from pandas.core.dtypes.missing import array_equivalent
 from pandas.core.dtypes.common import (
     is_datetimelike_v_numeric,
@@ -1083,6 +1084,32 @@ def _raise(left, right, err_msg):
     return True
 
 
+def assert_extension_array_equal(left, right):
+    """Check that left and right ExtensionArrays are equal.
+
+    Parameters
+    ----------
+    left, right : ExtensionArray
+        The two arrays to compare
+
+    Notes
+    -----
+    Missing values are checked separately from valid values.
+    A mask of missing values is computed for each and checked to match.
+    The remaining all-valid values are cast to object dtype and checked.
+    """
+    assert isinstance(left, ExtensionArray)
+    assert left.dtype == right.dtype
+    left_na = left.isna()
+    right_na = right.isna()
+    assert_numpy_array_equal(left_na, right_na)
+
+    left_valid = left[~left_na].astype(object)
+    right_valid = right[~right_na].astype(object)
+
+    assert_numpy_array_equal(left_valid, right_valid)
+
+
 # This could be refactored to use the NDFrame.equals method
 def assert_series_equal(left, right, check_dtype=True,
                         check_index_type='equiv',