Repository: geopandas/geopandas

Issue:
REGR: performance degrading after updating from version 0.8.2 to 0.9, using the "overlay"-method
- [x] I have searched the [geopandas] tag on [StackOverflow](https://stackoverflow.com/questions/tagged/geopandas) and [GIS StackExchange](https://gis.stackexchange.com/questions/tagged/geopandas) for similar questions.

- [x] I have asked my usage related question on [StackOverflow](https://stackoverflow.com) or [GIS StackExhange](https://gis.stackexchange.com).

---

#### Question about geopandas

Hello! The OS I'm working on is Microsoft Windows 10 LTSC. I often need to split GeoDataFrames with a minimum number of 350.000 Points into approximately 100 Polygons with:
```
cell_points = gpd.overlay(gdf_points, gdf_polygons, how='intersection')
```
So I was very happy when I found this bug report on the geopandas GitHub site: https://github.com/geopandas/geopandas/issues/1793. A bit further down in the thread, I found that geopandas 0.8.2 was released which works with pygeos. After I had updated from 0.8.1 to 0.8.2 my code executed much much faster. But when I recently updated to version 0.9, I quickly noticed that my code executed much much slower. Luckily, switching back to version 0.8.2 brought back the execution improvement.
What I did was to set up a virtual environment. Then I installed first of all geopandas from conda-forge, then I installed pygeos from conda-forge. This is the code that I used to measure the time:
```
import geopandas as gpd
import numpy as np
import pandas as pd
from shapely.geometry import Polygon
from time import perf_counter


print(gpd.__version__)


def create_Polygon(row):
   return Polygon([(i[0], i[1]) for i in row.reshape((4, 2))])


# #### Generate random Points inside (0, 10) ######
df_points = pd.DataFrame(data=np.random.rand(100000, 2) * 10, columns=['X', 'Y'])
gdf_points = gpd.GeoDataFrame(df_points, geometry=gpd.points_from_xy(df_points['X'], df_points['Y']))

# #### Generate 100 Polygons (Squares with edge length of 1) inside (0, 10) #####
step = 1
arr = np.zeros(shape=(100, 8))
counter = 0
for i in range(10):
   for j in range(10):
       arr[counter] = [i, j, i+step, j, i+step, j+step, i, j+step]
       counter += 1
gdf_polygons = gpd.GeoDataFrame(data=list(range(100)),
                               geometry=np.apply_along_axis(create_Polygon, axis=1, arr=arr))

# #### Intersect Polygons with Points #####
time_start = perf_counter()
cell_points = gpd.overlay(gdf_points, gdf_polygons, how='intersection')
print(f'Time: {(perf_counter() - time_start):.2f} seconds')
```
The execution time of version 0.9 was approximately 40 seconds. The execution time of version 0.8.2 was approximately 0.6 seconds! (Both with Python version 3.9.6, executed in PyCharm). My question is: Can someone recreate this? Is there maybe something else that I should have installed, when updating to geopandas version 0.9 to keep the performance improvements from version 0.8.2?



Discussion:
Wow, that is quite a performance hit.

I can confirm it in my environment. The issue comes from `keep_geom_type=True` option that checks geometry types but in a very sloppy way. 99% of the time is spent on `dissolve` in that check. If you pass `keep_geom_type=False` to overlay, you're back at 0.6s.

https://github.com/geopandas/geopandas/blob/8f08539b3f472c84daac52a847984584c63731db/geopandas/tools/overlay.py#L337

It was implemented in #1582 (cc @jdmcbr) and released in 0.9.0. I would say this can even be considered a regression that should be fixed before we release 0.10.

Thanks a lot for the report!
Oh gosh, that's terrible, sorry @Schulz3.

@martinfleis your response suggests you might already have an idea of a better way to perform this check without incurring the performance penalty? I admittedly am not immediately seeing a clearly more performant way to do this check, but can start looking for one if there wasn't something you already had in mind. 
No, I haven't thought about that. What is clear is that dissolve is not the way. 
The bottleneck is this groupby's aggregation in `dissolve`:

https://github.com/geopandas/geopandas/blob/8f08539b3f472c84daac52a847984584c63731db/geopandas/geodataframe.py#L1525-L1535

which initialises a `GeoSeries` for each block. I've removed the aggregation and iterated over the groupby's indices to select the geometries from the `GeometryArray` direclty, apply `unary_union` and append the resulting geometry to a list. With this list I can then initialise `aggregated_geometry`. I get a 3x performance improvement this way. This whole generation of `aggregated_geometry` can also be skipped if all groups are singletons, and thus recover more or less the original performance.

Shall I go ahead with a PR?
@TLouf That gives a performance improvement for any `dissolve` operation, right? If so, PR is welcome. 

I don't think it solves this regression though since we are  60x slower than before. 3x speedup will make it _"only"_ 20x slower :). I think we need to find another way here since `unary_union` will always be expensive.
Well in this case here the whole `dissolve` can be skipped because we're only doing unary_unions of single geometries, since no geometry is actually exploded. Actually I didn't think of it when I tested but in `dissolve` the unary_union part can be skipped for singletons too. The function is still costly because I need to initialise a `GeometryArray` from a list of geometries, and `__getitem__` many times though.
I think that the way forward here could be 

1. filter out results of a different geom type
2. filter geometry collections, loop through the collection and remove different types

that should do the trick while avoiding unions and tackling only those geometries that need to be altered.
On the actual issue of `overlay` needing to call (or not) the `dissolve` method or not: what's is it used for? 
So looking at the code and the PR that added it, we are doing 1) an explode, 2) filter out unwanted geometry types, and 3) dissolve again to reverse the explode.  

But AFAIU this was specifically done for GeometryCollections, and not for other Multi goemetries (eg if you have a MultiPolygon in the result, exploding it would give polygons, which won't be filtered out if a MultiPolygon is fine, and dissolving those polygons will give back the original MultiPolygon). 
So first, we should probably only do this explode/dissolve sequence in case there are actually geometry collections present (I suppose checking for this is relatively cheap compared to the actual overlay operation). And second, we also don't really need a "dissolve" (using `unary_union`) while we actually know that we just want to create a GeometryCollection. 
(reading Martin's post above again, so that's basically the same as what Martin already said .. :)).

One option could be to first filter out the geometry collections as a subset, and only do the exploding/filtering/recombining on this subset, and then assign the resulting filtered collections to the full geometry array. With `pygeos.geometrycollections` you can actually create GeometryCollections based on indices you get from exploding. So that could be used instead of dissolve/unary_union to recreate the collections. (EDIT: that might not be that useful, actually, as we *don't* want to recreate a collection if filtered result has only a single geometry type and could be represented as a Multi geometry)



Resolving patch:
diff --git a/geopandas/tools/overlay.py b/geopandas/tools/overlay.py
--- a/geopandas/tools/overlay.py
+++ b/geopandas/tools/overlay.py
@@ -185,8 +185,8 @@ def overlay(df1, df2, how="intersection", keep_geom_type=None, make_valid=True):
     1       2.0       1.0  POLYGON ((2.00000 2.00000, 2.00000 3.00000, 3....
     2       2.0       2.0  POLYGON ((4.00000 4.00000, 4.00000 3.00000, 3....
     3       1.0       NaN  POLYGON ((2.00000 0.00000, 0.00000 0.00000, 0....
-    4       2.0       NaN  MULTIPOLYGON (((4.00000 3.00000, 4.00000 2.000...
-    5       NaN       1.0  MULTIPOLYGON (((3.00000 2.00000, 3.00000 1.000...
+    4       2.0       NaN  MULTIPOLYGON (((3.00000 3.00000, 4.00000 3.000...
+    5       NaN       1.0  MULTIPOLYGON (((2.00000 2.00000, 3.00000 2.000...
     6       NaN       2.0  POLYGON ((3.00000 5.00000, 5.00000 5.00000, 5....
 
     >>> geopandas.overlay(df1, df2, how='intersection')
@@ -198,8 +198,8 @@ def overlay(df1, df2, how="intersection", keep_geom_type=None, make_valid=True):
     >>> geopandas.overlay(df1, df2, how='symmetric_difference')
        df1_data  df2_data                                           geometry
     0       1.0       NaN  POLYGON ((2.00000 0.00000, 0.00000 0.00000, 0....
-    1       2.0       NaN  MULTIPOLYGON (((4.00000 3.00000, 4.00000 2.000...
-    2       NaN       1.0  MULTIPOLYGON (((3.00000 2.00000, 3.00000 1.000...
+    1       2.0       NaN  MULTIPOLYGON (((3.00000 3.00000, 4.00000 3.000...
+    2       NaN       1.0  MULTIPOLYGON (((2.00000 2.00000, 3.00000 2.000...
     3       NaN       2.0  POLYGON ((3.00000 5.00000, 5.00000 5.00000, 5....
 
     >>> geopandas.overlay(df1, df2, how='difference')
@@ -213,7 +213,7 @@ def overlay(df1, df2, how="intersection", keep_geom_type=None, make_valid=True):
     1       2.0       1.0  POLYGON ((2.00000 2.00000, 2.00000 3.00000, 3....
     2       2.0       2.0  POLYGON ((4.00000 4.00000, 4.00000 3.00000, 3....
     3       1.0       NaN  POLYGON ((2.00000 0.00000, 0.00000 0.00000, 0....
-    4       2.0       NaN  MULTIPOLYGON (((4.00000 3.00000, 4.00000 2.000...
+    4       2.0       NaN  MULTIPOLYGON (((3.00000 3.00000, 4.00000 3.000...
 
     See also
     --------
@@ -317,32 +317,64 @@ def _make_valid(df):
             result = dfunion[dfunion["__idx1"].notnull()].copy()
 
     if keep_geom_type:
-        key_order = result.keys()
-        exploded = result.reset_index(drop=True).explode()
-        exploded = exploded.reset_index(level=0)
+        geom_type = df1.geom_type.iloc[0]
+
+        # First we filter the geometry types inside GeometryCollections objects
+        # (e.g. GeometryCollection([polygon, point]) -> polygon)
+        # we do this separately on only the relevant rows, as this is an expensive
+        # operation (an expensive no-op for geometry types other than collections)
+        is_collection = result.geom_type == "GeometryCollection"
+        if is_collection.any():
+            geom_col = result._geometry_column_name
+            collections = result[[geom_col]][is_collection]
+
+            exploded = collections.reset_index(drop=True).explode()
+            exploded = exploded.reset_index(level=0)
+
+            orig_num_geoms_exploded = exploded.shape[0]
+            if geom_type in polys:
+                exploded = exploded.loc[exploded.geom_type.isin(polys)]
+            elif geom_type in lines:
+                exploded = exploded.loc[exploded.geom_type.isin(lines)]
+            elif geom_type in points:
+                exploded = exploded.loc[exploded.geom_type.isin(points)]
+            else:
+                raise TypeError(
+                    "`keep_geom_type` does not support {}.".format(geom_type)
+                )
+            num_dropped_collection = orig_num_geoms_exploded - exploded.shape[0]
+
+            # level_0 created with above reset_index operation
+            # and represents the original geometry collections
+            # TODO avoiding dissolve to call unary_union in this case could further
+            # improve performance (we only need to collect geometries in their
+            # respective Multi version)
+            dissolved = exploded.dissolve(by="level_0")
+            result.loc[is_collection, geom_col] = dissolved[geom_col].values
+        else:
+            num_dropped_collection = 0
 
+        # Now we filter all geometries (in theory we don't need to do this
+        # again for the rows handled above for GeometryCollections, but filtering
+        # them out is probably more expensive as simply including them when this
+        # is typically about only a few rows)
         orig_num_geoms = result.shape[0]
-        geom_type = df1.geom_type.iloc[0]
         if geom_type in polys:
-            exploded = exploded.loc[exploded.geom_type.isin(polys)]
+            result = result.loc[result.geom_type.isin(polys)]
         elif geom_type in lines:
-            exploded = exploded.loc[exploded.geom_type.isin(lines)]
+            result = result.loc[result.geom_type.isin(lines)]
         elif geom_type in points:
-            exploded = exploded.loc[exploded.geom_type.isin(points)]
+            result = result.loc[result.geom_type.isin(points)]
         else:
             raise TypeError("`keep_geom_type` does not support {}.".format(geom_type))
+        num_dropped = orig_num_geoms - result.shape[0]
 
-        # level_0 created with above reset_index operation
-        # and represents the original geometry collections
-        result = exploded.dissolve(by="level_0")[key_order]
-
-        if (result.shape[0] != orig_num_geoms) and keep_geom_type_warning:
-            num_dropped = orig_num_geoms - result.shape[0]
+        if (num_dropped > 0 or num_dropped_collection > 0) and keep_geom_type_warning:
             warnings.warn(
                 "`keep_geom_type=True` in overlay resulted in {} dropped "
                 "geometries of different geometry types than df1 has. "
                 "Set `keep_geom_type=False` to retain all "
-                "geometries".format(num_dropped),
+                "geometries".format(num_dropped + num_dropped_collection),
                 UserWarning,
                 stacklevel=2,
             )


Test patch:
diff --git a/geopandas/tests/test_overlay.py b/geopandas/tests/test_overlay.py
--- a/geopandas/tests/test_overlay.py
+++ b/geopandas/tests/test_overlay.py
@@ -575,6 +575,11 @@ def test_keep_geom_type_geometry_collection():
     df1 = read_file(os.path.join(DATA, "geom_type", "df1.geojson"))
     df2 = read_file(os.path.join(DATA, "geom_type", "df2.geojson"))
 
+    with pytest.warns(UserWarning, match="`keep_geom_type=True` in overlay"):
+        intersection = overlay(df1, df2, keep_geom_type=None)
+    assert len(intersection) == 1
+    assert (intersection.geom_type == "Polygon").all()
+
     intersection = overlay(df1, df2, keep_geom_type=True)
     assert len(intersection) == 1
     assert (intersection.geom_type == "Polygon").all()
@@ -584,6 +589,44 @@ def test_keep_geom_type_geometry_collection():
     assert (intersection.geom_type == "GeometryCollection").all()
 
 
+def test_keep_geom_type_geometry_collection2():
+    polys1 = [
+        box(0, 0, 1, 1),
+        box(1, 1, 3, 3).union(box(1, 3, 5, 5)),
+    ]
+
+    polys2 = [
+        box(0, 0, 1, 1),
+        box(3, 1, 4, 2).union(box(4, 1, 5, 4)),
+    ]
+    df1 = GeoDataFrame({"left": [0, 1], "geometry": polys1})
+    df2 = GeoDataFrame({"right": [0, 1], "geometry": polys2})
+
+    result1 = overlay(df1, df2, keep_geom_type=True)
+    expected1 = GeoDataFrame(
+        {
+            "left": [0, 1],
+            "right": [0, 1],
+            "geometry": [box(0, 0, 1, 1), box(4, 3, 5, 4)],
+        }
+    )
+    assert_geodataframe_equal(result1, expected1)
+
+    result1 = overlay(df1, df2, keep_geom_type=False)
+    expected1 = GeoDataFrame(
+        {
+            "left": [0, 1, 1],
+            "right": [0, 0, 1],
+            "geometry": [
+                box(0, 0, 1, 1),
+                Point(1, 1),
+                GeometryCollection([box(4, 3, 5, 4), LineString([(3, 1), (3, 2)])]),
+            ],
+        }
+    )
+    assert_geodataframe_equal(result1, expected1)
+
+
 @pytest.mark.parametrize("make_valid", [True, False])
 def test_overlap_make_valid(make_valid):
     bowtie = Polygon([(1, 1), (9, 9), (9, 1), (1, 9), (1, 1)])