Repository: geopandas/geopandas

Issue:
BUG: Overlay function not working on certain geometries, but works with Shapely
- [x] I have checked that this issue has not already been reported.

- [x] I have confirmed this bug exists on the latest version of geopandas.

- [x] (optional) I have confirmed this bug exists on the main branch of geopandas.

---


#### Code Sample, a copy-pastable example

The following code does not work on the attached parquet files
```python
import geopandas as gpd

poly_gdf = gpd.read_parquet('polygon.parquet')
erase_gdf = gpd.read_parquet('erase_polygons.parquet')

overlay_gdf = gpd.overlay(poly_gdf, erase_gdf, how='difference')
```
The following shapely code works
```python
from shapely.ops import unary_union

erase_union= unary_union(erase_gdf.geometry)
poly_2= poly_gdf.copy()
poly_2['geometry'] = poly_gdf['geometry'].apply(lambda g: g.difference(erase_union))
```

#### Problem description

I am attaching a zip file with two parquet files. One is a multipolygon that I would like to erase features from using the erase_polygons parquet. However, when using the overlay difference function within geopandas, it does not properly erase the features. The shapely code above however does work. See the output below:
[gpd_difference_prob.zip](https://github.com/geopandas/geopandas/files/13299940/gpd_difference_prob.zip)

geopandas output:
<img width="809" alt="image" src="https://github.com/geopandas/geopandas/assets/133036282/a7262b50-910f-4674-ac9a-5f2ff5bf5247">

shapely output (correct output):
<img width="863" alt="image" src="https://github.com/geopandas/geopandas/assets/133036282/31d8b286-6f33-4170-9e9d-02125eda6366">


#### Output of ``geopandas.show_versions()``

<details>

SYSTEM INFO
-----------
python     : 3.10.12 | packaged by conda-forge | (main, Jun 23 2023, 22:34:57) [MSC v.1936 64 bit (AMD64)]
executable : [c:\Users\Eric\anaconda3\envs\assemblage_open_env\python.exe](file:///C:/Users/Eric/anaconda3/envs/assemblage_open_env/python.exe)
machine    : Windows-10-10.0.22621-SP0

GEOS, GDAL, PROJ INFO
---------------------
GEOS       : 3.12.0
GEOS lib   : None
GDAL       : 3.7.2
GDAL data dir: None
PROJ       : 9.3.0
PROJ data dir: [C:\Users\Eric\anaconda3\envs\assemblage_open_env\Library\share\proj](file:///C:/Users/Eric/anaconda3/envs/assemblage_open_env/Library/share/proj)

PYTHON DEPENDENCIES
-------------------
geopandas  : 0.14.0
numpy      : 1.26.0
pandas     : 2.1.1
pyproj     : 3.6.1
shapely    : 2.0.2
fiona      : 1.9.5
geoalchemy2: 0.14.2
geopy      : None
matplotlib : 3.8.0
mapclassify: 2.6.1
pygeos     : None
pyogrio    : None
psycopg2   : 2.9.9 (dt dec pq3 ext lo64)
pyarrow    : 13.0.0
rtree      : 1.1.0

</details>



Discussion:
@EricSamson-Tract thanks for the nice and reproducible report!

I initially couldn't reproduce it with my dev environment, and then could reproduce it with another environment with latest released geopandas, and was thinking it might be due to a difference in GEOS version (3.11 vs 3.12). But actually, it seems this is "fixed" in the dev version of geopandas because of a change we made in https://github.com/geopandas/geopandas/pull/3013 (specifically those lines: [link](https://github.com/geopandas/geopandas/commit/977f7904691138693bb408522d06a76a5caef760#diff-6de54d344de070ef579b8195793167cf4ffb9033a96d14ff72cc4434269e5b3aL97-R96) in overlay.py). 

Our `overlay` code for the "difference" operation has some logic to make the resulting geometries valid:

https://github.com/geopandas/geopandas/blob/0eb2a5ecdc3b7b595e406c9f7bf1e6435ad39828/geopandas/tools/overlay.py#L90-L100

I don't remember exactly what was the original motivation to do so (maybe in the past the difference method from shapely/GEOS often resulted in invalid results?). 
But so this does a `buffer(0)` on the result (if you have the latest shapely>=2), and it is this buffer operation that removes a part of your resulting geometry. Using your example dataframes, and doing the the difference vs difference+buffer on the individual shapely geometries:

```
poly1 = poly_gdf.geometry.item()
poly2 = erase_gdf.geometry.item()

poly1.difference(poly2)
poly1.difference(poly2).buffer(0)
```

![image](https://github.com/geopandas/geopandas/assets/1020496/8213853d-8a56-4774-8a62-264cca52c066)

In the meantime, on the main branch, we simplified that "make valid" step in the overlay difference function to always call `make_valid()` instead of `buffer(0)` (in the PR I mentioned above), and so this ensures that the buffer call doesn't invalidate the result of `overlay`. 

The fact that `buffer(0)` drops a significant part of the geometry might be a bug in Shapely/GEOS (and will report there). But it also has always been a bit of a hack to make polygons valid with this trick, and nowadays we should always use `make_valid()` instead, _if_ it's even needed to do this (we could also leave this up the user to do after the `overlay` operation, if they want to do so).
Accidental bugfix :). 
Opened https://github.com/geopandas/geopandas/issues/3073 to remove the remaining `buffer(0)` calls.

And it seems this is actually a regression in 0.14, because I see we accidentally wrongly update the if/else version check at https://github.com/geopandas/geopandas/commit/b51d43ba182d776cbf073c7a796010efc63eee1d#diff-6de54d344de070ef579b8195793167cf4ffb9033a96d14ff72cc4434269e5b3a 
Originally we were using `make_valid()` for pygeos and shapely 1.8+ (`if compat.USE_PYGEOS or compat.SHAPELY_GE_18:`), but that was wrongly replaced with `if compat.USE_PYGEOS:` 


(we can fix this if we do another 0.14.x release)

Resolving patch:
diff --git a/geopandas/tools/overlay.py b/geopandas/tools/overlay.py
--- a/geopandas/tools/overlay.py
+++ b/geopandas/tools/overlay.py
@@ -4,7 +4,6 @@
 import numpy as np
 import pandas as pd
 
-from geopandas import _compat as compat
 from geopandas import GeoDataFrame, GeoSeries
 from geopandas.array import _check_crs, _crs_mismatch_warn
 
@@ -94,10 +93,8 @@ def _overlay_difference(df1, df2):
         new_g.append(new)
     differences = GeoSeries(new_g, index=df1.index, crs=df1.crs)
     poly_ix = differences.geom_type.isin(["Polygon", "MultiPolygon"])
-    if compat.USE_PYGEOS:
-        differences.loc[poly_ix] = differences[poly_ix].make_valid()
-    else:
-        differences.loc[poly_ix] = differences[poly_ix].buffer(0)
+    differences.loc[poly_ix] = differences[poly_ix].make_valid()
+
     geom_diff = differences[~differences.is_empty].copy()
     dfdiff = df1[~differences.is_empty].copy()
     dfdiff[dfdiff._geometry_column_name] = geom_diff


Test patch: