Repository: geopandas/geopandas

Issue:
Indexing with an empty series results in dropping all column info
Ran into this problem while I was trying to work around the (recently fixed) problem of errors when loading an empty shapefile. I would then create a blank geodataframe and then pass it into a function which selects only the features within a bounding box:
```
import geopandas as gpd
import pandas as pd
from shapely.geometry import box
blankdata = \
            {'FIRENAME': pd.Series([], dtype=np.dtype('O')),
             'STATE': pd.Series([], dtype=np.dtype('O')),
             'ACRES': pd.Series([], dtype=np.dtype('int64')),
             'CONTAINDAT': pd.Series([], dtype=np.dtype('<M8[ns]')),
             'PERCENTCON': pd.Series([], dtype=np.dtype('int64')),
             'REPORTDATE': pd.Series([], dtype=np.dtype('<M8[ns]')),
             'DATECRNT': pd.Series([], dtype=np.dtype('<M8[ns]')),
             'IRWINID': pd.Series([], dtype=np.dtype('O'))}
points = gpd.GeoDataFrame(blankdata,
            crs='EPSG:4326', geometry=gpd.GeoSeries())
conus = box(-126, 24, -66, 50)
points = points[points.intersects(conus)]
```
Before the indexing, `points` is:
```
Empty GeoDataFrame
Columns: [ACRES, CONTAINDAT, DATECRNT, FIRENAME, IRWINID, PERCENTCON, REPORTDATE, STATE, geometry]
Index: []
```

The indexing array (result of the intersection) is:
```
Series([], dtype: float64)
```

After the indexing, points is:
```
Empty DataFrame
Columns: []
Index: []
```

So, the rest of the code is expecting an object with a bunch of columns that I guaranteed earlier in my code, and this intersection dropped them all unexpectedly. I eventually hit an AttributeError for a missing column name.

Tested with geopandas v0.3.0.


Discussion:
Thanks for the report!

This seems to be an issue with pandas as well:

```
In [36]: df = pd.DataFrame({'a': pd.Series([], dtype=int), 'b': pd.Series([], dtype=float)})

In [37]: df
Out[37]: 
Empty DataFrame
Columns: [a, b]
Index: []

In [38]: df[pd.Series()]
Out[38]: 
Empty DataFrame
Columns: []
Index: []
```

But, if you do eg a comparison operation in pandas on an empty, then it does work correctly:

```
In [39]: df[df['a'] > 10]
Out[39]: 
Empty DataFrame
Columns: [a, b]
Index: []
```

this is because pandas ensures it is still a boolean series although empty:

```
In [40]: df['a'] > 10
Out[40]: Series([], Name: a, dtype: bool)
```

So it seems the bug is in `GeoSeries.intersects` not being of type boolean when empty.

The fix will be to explicitly put the dtype on the series return value here:

https://github.com/geopandas/geopandas/blob/891c5cd73c604862fbefd014cc6536faf571d260/geopandas/base.py#L53-L54

(although we might need to make a separate version for this for the predicate methods that return boolean values)

PR always welcome.
I'll see what I can do today. Should we also file a bug report to pandas as
well?

On Thu, Mar 1, 2018 at 6:13 PM, Joris Van den Bossche <
notifications@github.com> wrote:

> Thanks for the report!
>
> This seems to be an issue with pandas as well:
>
> In [36]: df = pd.DataFrame({'a': pd.Series([], dtype=int), 'b': pd.Series([], dtype=float)})
>
> In [37]: df
> Out[37]:
> Empty DataFrame
> Columns: [a, b]
> Index: []
>
> In [38]: df[pd.Series()]
> Out[38]:
> Empty DataFrame
> Columns: []
> Index: []
>
> But, if you do eg a comparison operation in pandas on an empty, then it
> does work correctly:
>
> In [39]: df[df['a'] > 10]
> Out[39]:
> Empty DataFrame
> Columns: [a, b]
> Index: []
>
> this is because pandas ensures it is still a boolean series although empty:
>
> In [40]: df['a'] > 10
> Out[40]: Series([], Name: a, dtype: bool)
>
> So it seems the bug is in GeoSeries.intersects not being of type boolean
> when empty.
>
> The fix will be to explicitly put the dtype on the series return value
> here:
>
> https://github.com/geopandas/geopandas/blob/891c5cd73c604862fbefd014cc6536
> faf571d260/geopandas/base.py#L53-L54
>
> (although we might need to make a separate version for this for the
> predicate methods that return boolean values)
>
> PR always welcome.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/geopandas/geopandas/issues/685#issuecomment-369764439>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AARy-FNG8OYyQhCkqFHuPuqE17ztV6siks5taICRgaJpZM4SZF0b>
> .
>

> Should we also file a bug report to pandas as well?

I think not (but not fully sure). The empty series (because it is not boolean), is regarded as a selection on the columns. See eg example this case where the dataframe is emtpy but the series not:

```
In [41]: df = pd.DataFrame({'a': pd.Series([], dtype=int), 'b': pd.Series([], dtype=float)})

In [42]: df
Out[42]: 
Empty DataFrame
Columns: [a, b]
Index: []

In [43]: df[pd.Series(['a'])]
Out[43]: 
Empty DataFrame
Columns: [a]
Index: []
```

It selected on the columns. In case of an empty series, it is of course a bit ambiguous, as it could both be a selection on the rows as columns. But probably pandas looks at the dtype of the series then.
Looking at the code for `_series_op` I am guessing that the more pedantic solution is actually to set the dtype to the dtype of the null_value?
Or in addition to `null_val = False if op != 'distance' else np.nan` add a `dtype = bool if op != 'distance' else float`. 

I like my approach if only because it is then consistent with the change that would need to happen in `_series_unary_op()` as well, because it can take floats, bools or strings (which I think needs to be treated as objects, right?)
Ah, yes there the case exists as well. Doesn't really matter to me how exactly it is done. The 'None' case (for strings) would indeed be object. 

And it looks like the place for tests seems to be test_geom_methods.py. I think I will update it so that it will have a combination of (empty, something), (something, empty), and (empty, empty) operations.
anybody against me reworking that test file a bit to take advantage of pytest.parametrize? I won't go overboard, but this is a good place to use it to run tests on different combinations of data.
Hmm, reworking the test file is turning into quite a bear. I'll leave it off as a separate PR for later, and just add explicit tests for this.

Resolving patch:
diff --git a/geopandas/base.py b/geopandas/base.py
--- a/geopandas/base.py
+++ b/geopandas/base.py
@@ -40,7 +40,7 @@ def _geo_op(this, other, op):
 # TODO: think about merging with _geo_op
 def _series_op(this, other, op, **kwargs):
     """Geometric operation that returns a pandas Series"""
-    null_val = False if op != 'distance' else np.nan
+    null_val = False if op not in ['distance', 'project'] else np.nan
 
     if isinstance(other, GeoPandasBase):
         this = this.geometry
@@ -48,10 +48,11 @@ def _series_op(this, other, op, **kwargs):
         return Series([getattr(this_elem, op)(other_elem, **kwargs)
                     if not this_elem.is_empty | other_elem.is_empty else null_val
                     for this_elem, other_elem in zip(this, other)],
-                    index=this.index)
+                    index=this.index, dtype=np.dtype(type(null_val)))
     else:
         return Series([getattr(s, op)(other, **kwargs) if s else null_val
-                      for s in this.geometry], index=this.index)
+                       for s in this.geometry],
+                      index=this.index, dtype=np.dtype(type(null_val)))
 
 
 def _geo_unary_op(this, op):
@@ -63,7 +64,7 @@ def _geo_unary_op(this, op):
 def _series_unary_op(this, op, null_value=False):
     """Unary operation that returns a Series"""
     return Series([getattr(geom, op, null_value) for geom in this.geometry],
-                     index=this.index)
+                     index=this.index, dtype=np.dtype(type(null_value)))
 
 
 class GeoPandasBase(object):


Test patch:
diff --git a/geopandas/tests/test_geom_methods.py b/geopandas/tests/test_geom_methods.py
--- a/geopandas/tests/test_geom_methods.py
+++ b/geopandas/tests/test_geom_methods.py
@@ -20,6 +20,13 @@
 from pandas.util.testing import assert_series_equal, assert_frame_equal
 
 
+def assert_array_dtype_equal(a, b, *args, **kwargs):
+    a = np.asanyarray(a)
+    b = np.asanyarray(b)
+    assert a.dtype == b.dtype
+    assert_array_equal(a, b, *args, **kwargs)
+
+
 class TestGeomMethods:
 
     def setup_method(self):
@@ -56,6 +63,9 @@ def setup_method(self):
         self.l2 = LineString([(0, 0), (1, 0), (1, 1), (0, 1)])
         self.g5 = GeoSeries([self.l1, self.l2])
         self.g6 = GeoSeries([self.p0, self.t3])
+        self.empty = GeoSeries([])
+        self.empty.crs = {'init': 'epsg:4326', 'no_defs': True}
+        self.empty_poly = Polygon()
 
         # Crossed lines
         self.l3 = LineString([(0, 0), (1, 1)])
@@ -200,6 +210,8 @@ def test_crs_warning(self):
     def test_intersection(self):
         self._test_binary_topological('intersection', self.t1,
                                       self.g1, self.g2)
+        self._test_binary_topological('intersection', self.empty_poly,
+                                      self.g1, self.empty)
 
     def test_union_series(self):
         self._test_binary_topological('union', self.sq, self.g1, self.g2)
@@ -264,7 +276,7 @@ def test_unary_union(self):
 
     def test_contains(self):
         expected = [True, False, True, False, False, False]
-        assert_array_equal(expected, self.g0.contains(self.t1))
+        assert_array_dtype_equal(expected, self.g0.contains(self.t1))
 
     def test_length(self):
         expected = Series(np.array([2 + np.sqrt(2), 4]), index=self.g1.index)
@@ -277,48 +289,58 @@ def test_length(self):
 
     def test_crosses(self):
         expected = [False, False, False, False, False, False]
-        assert_array_equal(expected, self.g0.crosses(self.t1))
+        assert_array_dtype_equal(expected, self.g0.crosses(self.t1))
 
         expected = [False, True]
-        assert_array_equal(expected, self.crossed_lines.crosses(self.l3))
+        assert_array_dtype_equal(expected, self.crossed_lines.crosses(self.l3))
 
     def test_disjoint(self):
         expected = [False, False, False, False, False, True]
-        assert_array_equal(expected, self.g0.disjoint(self.t1))
+        assert_array_dtype_equal(expected, self.g0.disjoint(self.t1))
 
     def test_distance(self):
         expected = Series(np.array([np.sqrt((5 - 1)**2 + (5 - 1)**2), np.nan]),
                           self.na_none.index)
-        assert_array_equal(expected, self.na_none.distance(self.p0))
+        assert_array_dtype_equal(expected, self.na_none.distance(self.p0))
 
         expected = Series(np.array([np.sqrt(4**2 + 4**2), np.nan]),
                           self.g6.index)
-        assert_array_equal(expected, self.g6.distance(self.na_none))
+        assert_array_dtype_equal(expected, self.g6.distance(self.na_none))
 
     def test_intersects(self):
         expected = [True, True, True, True, True, False]
-        assert_array_equal(expected, self.g0.intersects(self.t1))
+        assert_array_dtype_equal(expected, self.g0.intersects(self.t1))
 
         expected = [True, False]
-        assert_array_equal(expected, self.na_none.intersects(self.t2))
+        assert_array_dtype_equal(expected, self.na_none.intersects(self.t2))
+
+        expected = np.array([], dtype=bool)
+        assert_array_dtype_equal(expected, self.empty.intersects(self.t1))
+
+        expected = np.array([], dtype=bool)
+        assert_array_dtype_equal(
+                expected, self.empty.intersects(self.empty_poly))
+
+        expected = [False] * 6
+        assert_array_dtype_equal(expected, self.g0.intersects(self.empty_poly))
 
     def test_overlaps(self):
         expected = [True, True, False, False, False, False]
-        assert_array_equal(expected, self.g0.overlaps(self.inner_sq))
+        assert_array_dtype_equal(expected, self.g0.overlaps(self.inner_sq))
 
         expected = [False, False]
-        assert_array_equal(expected, self.g4.overlaps(self.t1))
+        assert_array_dtype_equal(expected, self.g4.overlaps(self.t1))
 
     def test_touches(self):
         expected = [False, True, False, False, False, False]
-        assert_array_equal(expected, self.g0.touches(self.t1))
+        assert_array_dtype_equal(expected, self.g0.touches(self.t1))
 
     def test_within(self):
         expected = [True, False, False, False, False, False]
-        assert_array_equal(expected, self.g0.within(self.t1))
+        assert_array_dtype_equal(expected, self.g0.within(self.t1))
 
         expected = [True, True, True, True, True, False]
-        assert_array_equal(expected, self.g0.within(self.sq))
+        assert_array_dtype_equal(expected, self.g0.within(self.sq))
 
     def test_is_valid(self):
         expected = Series(np.array([True] * len(self.g1)), self.g1.index)
@@ -344,8 +366,8 @@ def test_xy_points(self):
         expected_x = [-73.9847, -74.0446]
         expected_y = [40.7484, 40.6893]
 
-        assert_array_equal(expected_x, self.landmarks.geometry.x)
-        assert_array_equal(expected_y, self.landmarks.geometry.y)
+        assert_array_dtype_equal(expected_x, self.landmarks.geometry.x)
+        assert_array_dtype_equal(expected_y, self.landmarks.geometry.y)
 
     def test_xy_polygons(self):
         # accessing x attribute in polygon geoseries should raise an error