Repository: geopandas/geopandas

Issue:
fillna error with 0.6
When performing `fillna` with `numpy.nan` on the entire dataframe, I get this error in version 0.6:
```
~/miniconda/envs/geocube/lib/python3.6/site-packages/geopandas/array.py in fillna(self, value, method, limit)
    831         elif not isinstance(value, BaseGeometry):
    832             raise NotImplementedError(
--> 833                 "fillna currently only supports filling with a scalar geometry"
    834             )
    835 

NotImplementedError: fillna currently only supports filling with a scalar geometry
```

Is this expected? Future plans?


Discussion:
Hmm, we should probably allow this (given that it worked before). 

But note that it would not change anything about the values in the GeoSeries in 0.6.0 (as we force the missing values to be None in the validation (any np.nan is changed to None when constructing a GeoSeries) to only have a single missing value indicator).

So curious, what is the use case for wanting to fill missing values with np.nan (which is already used as the missing value indicator in pandas to start with)?

> So curious, what is the use case for wanting to fill missing values with np.nan (which is already used as the missing value indicator in pandas to start with)?

Mostly being lazy - not entirely needed. Applying it to the whole dataframe to ensure any NaN-like values are converted to `NaN`.
```python
>>> pdf = pandas.DataFrame([None, 1, 2, "bob"])
>>> pdf
      0
0  None
1     1
2     2
3   bob
>>> import numpy
>>> pdf.fillna(numpy.nan)
     0
0  NaN
1    1
2    2
3  bob
```
Ah, yes, that's a good use case (it's actually what I was arguing myself for why we convert np.nan to None in geopandas, to have this consistent).

So it should be easy to re-allow this, but note that it would still not change None to nan for the geometry column (as there I would prefer to keep this limited to a single indicator, since we have control over what values to put in)
> So it should be easy to re-allow this, but note that it would still not change None to nan for the geometry column

That's fine with me.
I have the same issue with this code:

`# load the shape file for Chicago police beats
beat_orig = geopandas.read_file("Boundaries_beat.geojson", driver = "GeoJSON")
beat_data = beat_orig.join(beat_cn.set_index("beat_num"), how = "left", on = "beat_num")
beat_data.fillna(0, inplace = True)`

At least how must i change the code?

Thank you and sorry for the english
@NervenCid the problem is that you are trying to fill all columns with 0, which is not a geometry value. 

I suppose the reason this worked before is because there were no missing values in the geometry column (because if there were, and you filled it with 0, you would have ended up with an invalid geometry column, which would have resulted in errors once you did some spatial operation).

In the new version, we prevent you from filling with an invalid geometry. 
In principle, we could still allow an invalid geometry in the case that there are actually no missing values in the column (to support such a use case of filling the full dataframe).

The workaround is that you either only fill specific columns, or that you provide the value to fill with per column (`df.fillna({'col1': 0, 'col2': 0, ...}`), but if you have many columns, this can become tedious.
@jorisvandenbossche Thank you for your answer and sorry for my english

so i shall change 0 by Nan? I have many columns and i cn't change my originals inputs

my notebook stopped to work
Hi @jorisvandenbossche,
> I suppose the reason this worked before is because there were no missing values in the geometry column (because if there were, and you filled it with 0, you would have ended up with an invalid geometry column, which would have resulted in errors once you did some spatial operation).
> 
> In the new version, we prevent you from filling with an invalid geometry.
> In principle, we could still allow an invalid geometry in the case that there are actually no missing values in the column (to support such a use case of filling the full dataframe).
> 
> The workaround is that you either only fill specific columns, or that you provide the value to fill with per column (`df.fillna({'col1': 0, 'col2': 0, ...}`), but if you have many columns, this can become tedious.

I'm in the process of upgrading our codebase to use the latest geopandas and I can confirm that this is in fact quite tedious! IMO a missing geometry value is far less sensical/typical than `nan` values in the other non-geometry columns, so I would argue `gdf.fillna(0)` should definitely work if the geometry column has no missing values. Just my 2c but this has been the only major headache so far and I would consider the new behavior to be a regression rather than an improvement. Happy to make a PR if you think you could be convinced that this behavior is preferable :)


> I would argue gdf.fillna(0) should definitely work if the geometry column has no missing values

@bnaul how would you implement that? Check if the value is geometry and if not pass None to GeometryArray and original value to the rest of the df?
I was thinking you could just move this check https://github.com/geopandas/geopandas/blob/master/geopandas/array.py#L859 inside the `if mask.any()` block...think that would do it?
@bnaul That is good. Could you try a PR to check the behaviour? I'd be happy with this change.

Resolving patch:
diff --git a/geopandas/array.py b/geopandas/array.py
--- a/geopandas/array.py
+++ b/geopandas/array.py
@@ -858,18 +858,17 @@ def fillna(self, value=None, method=None, limit=None):
         if method is not None:
             raise NotImplementedError("fillna with a method is not yet supported")
 
-        if _isna(value):
-            value = None
-        elif not isinstance(value, BaseGeometry):
-            raise NotImplementedError(
-                "fillna currently only supports filling with a scalar geometry"
-            )
-
         mask = self.isna()
         new_values = self.copy()
 
         if mask.any():
             # fill with value
+            if _isna(value):
+                value = None
+            elif not isinstance(value, BaseGeometry):
+                raise NotImplementedError(
+                    "fillna currently only supports filling with a scalar geometry"
+                )
             new_values = new_values._fill(mask, value)
 
         return new_values


Test patch:
diff --git a/geopandas/tests/test_pandas_methods.py b/geopandas/tests/test_pandas_methods.py
--- a/geopandas/tests/test_pandas_methods.py
+++ b/geopandas/tests/test_pandas_methods.py
@@ -317,7 +317,7 @@ def test_select_dtypes(df):
 # Missing values
 
 
-def test_fillna(s):
+def test_fillna(s, df):
     s2 = GeoSeries([Point(0, 0), None, Point(2, 2)])
     res = s2.fillna(Point(1, 1))
     assert_geoseries_equal(res, s)
@@ -327,6 +327,21 @@ def test_fillna(s):
     res = s2.fillna(np.nan)
     assert_geoseries_equal(res, s2)
 
+    # raise exception if trying to fill missing geometry w/ non-geometry
+    df2 = df.copy()
+    df2["geometry"] = s2
+    res = df2.fillna(Point(1, 1))
+    assert_geodataframe_equal(res, df)
+    with pytest.raises(NotImplementedError):
+        df2.fillna(0)
+
+    # allow non-geometry fill value if there are no missing values
+    # https://github.com/geopandas/geopandas/issues/1149
+    df3 = df.copy()
+    df3.loc[0, "value1"] = np.nan
+    res = df3.fillna(0)
+    assert_geodataframe_equal(res.astype({"value1": "int64"}), df)
+
 
 def test_dropna():
     s2 = GeoSeries([Point(0, 0), None, Point(2, 2)])