Repository: shapely/shapely (language: python)

Issue:
Can shapely 2.0 support unpickling of shapely 1 object?

## Expected behavior and actual behavior.
shapely 2.0 can't unpickle objects generated from shapely 1.8. I am guessing this is intentional because shapely 2.0 objects are immutable. But we have many existing pickles that's also difficult to convert to new shapely 2.0 geometry class. Wondering if the library can provide a backward compatible method to load old pickles? 

## Steps to reproduce the problem.

in a shapely 1.8 environment:
```py
import shapely.geometry
import pickle
shape = shapely.geometry.Polygon([[0,0],[3,3],[2,2]])
bytes = pickle.dumps(shape)
```

in a shapely 2.0 enviroment
```py
import shapely
import pickle
pickle.loads(<the bytes from above>)
```
And it will throw error `_pickle.UnpicklingError: state is not a dictionary`, because `__setstate__` is not implemented. 

## Operating system
Ubuntu 20

## Shapely version and provenance
2.0b2 installed from PyPI using pip



Discussion / hints:
I think the simple explanation is that we didn't try to make unpickling backwards compatible, and indeed because the simple solution is not possible, as you mention: geometry objects are now immutable, and so the problem is that if we add a `__setstate__` to handle the state of 1.x shapely objects, at that moment the geometry object already has been created (as an empty geometry, by calling the class constructor without arguments), and so we can't update this object inplace with the WKB that is stored in the state of 1.x pickles. 

The pickle module provides quite some tools to tweak this process, or to provide custom overrides or Unpickler classes (https://docs.python.org/3/library/pickle.html), so it might be that there is something possible to provide a function that can load those older pickles. But I am not super familiar with this. Experimenting with this would certainly be welcome!

---

The pickle files are under the hood just using WKB as the serialization format. So one workaround is to load your pickle files in Shapely 1.8, store them as pure wkb instead, and then load that in Shapely 2.0. 

Something like:

```python
import pickle
import shapely.wkb

# creating a test pickle file
from shapely.geometry import Point
p = Point(1, 1)
with open("point1.pickle", "wb") as f:
    pickle.dump(p, f)

# convert all pickle files to pure wkb files (could be done in a loop)
with open("point1.pickle", "rb") as f:
    geom = pickle.load(f)
with open("point1.wkb", "wb") as f:
    f.write(shapely.wkb.dumps(geom))
```

And then with shapely 2.0, we can read that:

```python
import shapely
import pickle

# unpickling fails as expected
>>> with open("point1.pickle", "rb") as f:
...     geom = pickle.load(f)
..
UnpicklingError: state is not a dictionary

# but we can read the pure WKB file
>>> with open("point1.wkb", "rb") as f:
...     geom = shapely.from_wkb(f.read())
>>> geom
<POINT (1 1)>

we have many different pickles that are stored in nested data structures, so simply converting them to binary wkb is non-trivial either. 

I got around my issue without upgrading the shapely library, but for anyone who wants to investigate this in the future, I would say overwriting the [_Unpickler](https://github.com/python/cpython/blob/main/Lib/pickle.py) seemed to be a promising route, but at the expense of some performance. Inside `load_build`, pop the stack, create a new Shapely 2.0 object, and then append it back to the stack. 

the normal Unpickler only provides `find_class` method for overwrites, which allows you to define a custom class to load the wkb, but I got stuck in how to make the custom class behaves exactly like the new Shapely 2.0 classes. 
closing this issue for now. 
(going to keep this open for now in case other people run into this, or in case we can still provide a small script / documentation note and how to load such older pickle files)

Fix patch:
diff --git a/CHANGES.txt b/CHANGES.txt
index dc759e2dc..bad09f471 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,6 +1,13 @@
 Changes
 =======
 
+
+2.0rc4 (2022-10-26)
+-------------------
+
+- Added temporary for unpickling shapely<2.0 geometries.
+
+
 2.0rc1 (2022-10-26)
 -------------------
 
diff --git a/src/pygeom.c b/src/pygeom.c
index f95fbe2d4..9d29e9041 100644
--- a/src/pygeom.c
+++ b/src/pygeom.c
@@ -38,14 +38,14 @@ PyObject* GeometryObject_FromGEOS(GEOSGeometry* ptr, GEOSContextHandle_t ctx) {
   } else {
     self->ptr = ptr;
     self->ptr_prepared = NULL;
-    self->weakreflist = (PyObject *)NULL;
+    self->weakreflist = (PyObject*)NULL;
     return (PyObject*)self;
   }
 }
 
 static void GeometryObject_dealloc(GeometryObject* self) {
   if (self->weakreflist != NULL) {
-    PyObject_ClearWeakRefs((PyObject *)self);
+    PyObject_ClearWeakRefs((PyObject*)self);
   }
   if (self->ptr != NULL) {
     // not using GEOS_INIT, but using global context instead
@@ -291,7 +291,87 @@ static PyObject* GeometryObject_richcompare(GeometryObject* self, PyObject* othe
   return result;
 }
 
+
+static PyObject* GeometryObject_SetState(PyObject* self, PyObject* value) {
+  unsigned char* wkb = NULL;
+  Py_ssize_t size;
+  GEOSGeometry* geom = NULL;
+  GEOSWKBReader* reader = NULL;
+
+  PyErr_WarnFormat(PyExc_UserWarning, 0,
+                   "Unpickling a shapely <2.0 geometry object. Please save the pickle "
+                   "again; shapely 2.1 will not have this compatibility.");
+
+  /* Cast the PyObject bytes to char */
+  if (!PyBytes_Check(value)) {
+    PyErr_Format(PyExc_TypeError, "Expected bytes, found %s", value->ob_type->tp_name);
+    return NULL;
+  }
+  size = PyBytes_Size(value);
+  wkb = (unsigned char*)PyBytes_AsString(value);
+  if (wkb == NULL) {
+    return NULL;
+  }
+
+  PyObject* linearring_type_obj = PyList_GET_ITEM(geom_registry[0], 2);
+  if (linearring_type_obj == NULL) {
+    return NULL;
+  }
+  if (!PyType_Check(linearring_type_obj)) {
+    PyErr_Format(PyExc_RuntimeError, "Invalid registry value");
+    return NULL;
+  }
+  PyTypeObject* linearring_type = (PyTypeObject*)linearring_type_obj;
+
+  GEOS_INIT;
+
+  reader = GEOSWKBReader_create_r(ctx);
+  if (reader == NULL) {
+    errstate = PGERR_GEOS_EXCEPTION;
+    goto finish;
+  }
+  geom = GEOSWKBReader_read_r(ctx, reader, wkb, size);
+  if (geom == NULL) {
+    errstate = PGERR_GEOS_EXCEPTION;
+    goto finish;
+  }
+  if (Py_TYPE(self) == linearring_type) {
+    const GEOSCoordSequence* coord_seq = GEOSGeom_getCoordSeq_r(ctx, geom);
+    if (coord_seq == NULL) {
+      errstate = PGERR_GEOS_EXCEPTION;
+      goto finish;
+    }
+    geom = GEOSGeom_createLinearRing_r(ctx, (GEOSCoordSequence*)coord_seq);
+    if (geom == NULL) {
+      errstate = PGERR_GEOS_EXCEPTION;
+      goto finish;
+    }
+  }
+
+  if (((GeometryObject*)self)->ptr != NULL) {
+    GEOSGeom_destroy_r(ctx, ((GeometryObject*)self)->ptr);
+  }
+  ((GeometryObject*)self)->ptr = geom; 
+
+finish:
+
+  if (reader != NULL) {
+    GEOSWKBReader_destroy_r(ctx, reader);
+  }
+
+  GEOS_FINISH;
+
+  if (errstate == PGERR_SUCCESS) {
+    Py_INCREF(Py_None);
+    return Py_None;
+  }
+  return NULL;
+}
+
+
 static PyMethodDef GeometryObject_methods[] = {
+    {"__setstate__", (PyCFunction)GeometryObject_SetState, METH_O,
+     "For unpickling pre-shapely 2.0 pickles"},
     {NULL} /* Sentinel */
 };
 


Test patch:
diff --git a/tests/data/emptypoint_1.8.5.post1.pickle b/tests/data/emptypoint_1.8.5.post1.pickle
new file mode 100644
index 000000000..db17961a2
Binary files /dev/null and b/tests/data/emptypoint_1.8.5.post1.pickle differ
diff --git a/tests/data/emptypolygon_1.8.5.post1.pickle b/tests/data/emptypolygon_1.8.5.post1.pickle
new file mode 100644
index 000000000..971cc3ddb
Binary files /dev/null and b/tests/data/emptypolygon_1.8.5.post1.pickle differ
diff --git a/tests/data/geometrycollection_1.8.5.post1.pickle b/tests/data/geometrycollection_1.8.5.post1.pickle
new file mode 100644
index 000000000..c3fa5ae04
Binary files /dev/null and b/tests/data/geometrycollection_1.8.5.post1.pickle differ
diff --git a/tests/data/linearring_1.8.5.post1.pickle b/tests/data/linearring_1.8.5.post1.pickle
new file mode 100644
index 000000000..51ef63d64
Binary files /dev/null and b/tests/data/linearring_1.8.5.post1.pickle differ
diff --git a/tests/data/linestring_1.8.5.post1.pickle b/tests/data/linestring_1.8.5.post1.pickle
new file mode 100644
index 000000000..688f26fe9
Binary files /dev/null and b/tests/data/linestring_1.8.5.post1.pickle differ
diff --git a/tests/data/multilinestring_1.8.5.post1.pickle b/tests/data/multilinestring_1.8.5.post1.pickle
new file mode 100644
index 000000000..68313d1dd
Binary files /dev/null and b/tests/data/multilinestring_1.8.5.post1.pickle differ
diff --git a/tests/data/multipoint_1.8.5.post1.pickle b/tests/data/multipoint_1.8.5.post1.pickle
new file mode 100644
index 000000000..64dab2da4
Binary files /dev/null and b/tests/data/multipoint_1.8.5.post1.pickle differ
diff --git a/tests/data/multipolygon_1.8.5.post1.pickle b/tests/data/multipolygon_1.8.5.post1.pickle
new file mode 100644
index 000000000..db8090fde
Binary files /dev/null and b/tests/data/multipolygon_1.8.5.post1.pickle differ
diff --git a/tests/data/point2d_1.8.5.post1.pickle b/tests/data/point2d_1.8.5.post1.pickle
new file mode 100644
index 000000000..1dc7ae6d4
Binary files /dev/null and b/tests/data/point2d_1.8.5.post1.pickle differ
diff --git a/tests/data/point3d_1.8.5.post1.pickle b/tests/data/point3d_1.8.5.post1.pickle
new file mode 100644
index 000000000..bf3f2fc29
Binary files /dev/null and b/tests/data/point3d_1.8.5.post1.pickle differ
diff --git a/tests/data/polygon_1.8.5.post1.pickle b/tests/data/polygon_1.8.5.post1.pickle
new file mode 100644
index 000000000..d04ebfb7e
Binary files /dev/null and b/tests/data/polygon_1.8.5.post1.pickle differ
diff --git a/tests/test_pickle.py b/tests/test_pickle.py
index a20422b4d..dd0431a44 100644
--- a/tests/test_pickle.py
+++ b/tests/test_pickle.py
@@ -1,24 +1,71 @@
+import pathlib
+import pickle
+from pickle import dumps, loads, HIGHEST_PROTOCOL
+import warnings
+
+import shapely
+from shapely.geometry import Point, LineString, LinearRing, Polygon, MultiLineString, MultiPoint, MultiPolygon, GeometryCollection, box
+from shapely import wkt
+
 import pytest
-from shapely.geometry import Point, LineString, LinearRing, Polygon, MultiPoint
 
-from pickle import dumps, loads, HIGHEST_PROTOCOL
+
+HERE = pathlib.Path(__file__).parent
+
 
 TEST_DATA = {
-    "point2d": (Point, [(1.0, 2.0)]),
-    "point3d": (Point, [(1.0, 2.0, 3.0)]),
-    "linestring": (LineString, [(0.0, 0.0), (0.0, 1.0), (1.0, 1.0)]),
-    "linearring": (LinearRing, [(0.0, 0.0), (1.0, 0.0), (1.0, 1.0), (0.0, 0.0)]),
-    "polygon": (Polygon, [(0.0, 0.0), (1.0, 0.0), (1.0, 1.0), (0.0, 0.0)]),
-    "multipoint": (MultiPoint, [(1.0, 2.0), (3.0, 4.0), (5.0, 6.0)]),
+    "point2d": Point([(1.0, 2.0)]),
+    "point3d": Point([(1.0, 2.0, 3.0)]),
+    "linestring": LineString([(0.0, 0.0), (0.0, 1.0), (1.0, 1.0)]),
+    "linearring": LinearRing([(0.0, 0.0), (1.0, 0.0), (1.0, 1.0), (0.0, 0.0)]),
+    "polygon": Polygon([(0.0, 0.0), (1.0, 0.0), (1.0, 1.0), (0.0, 0.0)]),
+    "multipoint": MultiPoint([(1.0, 2.0), (3.0, 4.0), (5.0, 6.0)]),
+    "multilinestring": MultiLineString([[(0.0, 0.0), (1.0, 1.0)], [(1.0, 2.0), (3.0, 3.0)]]),
+    "multipolygon": MultiPolygon([box(0, 0, 1, 1), box(2, 2, 3, 3)]),
+    "geometrycollection": GeometryCollection([Point(1.0, 2.0), box(0, 0, 1, 1)]),
+    "emptypoint": wkt.loads("POINT EMPTY"),
+    "emptypolygon": wkt.loads("POLYGON EMPTY"),
 }
-TEST_NAMES, TEST_DATA = zip(*TEST_DATA.items())
-@pytest.mark.parametrize("cls,coords", TEST_DATA, ids=TEST_NAMES)
-def test_pickle_round_trip(cls, coords):
-    geom1 = cls(coords)
-    assert geom1.has_z == (len(coords[0]) == 3)
+TEST_NAMES, TEST_GEOMS = zip(*TEST_DATA.items())
+
+
+@pytest.mark.parametrize("geom1", TEST_GEOMS, ids=TEST_NAMES)
+def test_pickle_round_trip(geom1):
     data = dumps(geom1, HIGHEST_PROTOCOL)
-    geom2 = loads(data)
+    with warnings.catch_warnings():
+        warnings.simplefilter("error")
+        geom2 = loads(data)
     assert geom2.has_z == geom1.has_z
     assert type(geom2) is type(geom1)
     assert geom2.geom_type == geom1.geom_type
     assert geom2.wkt == geom1.wkt
+
+
+@pytest.mark.parametrize("fname", (HERE / "data").glob("*.pickle"), ids=lambda fname: fname.name)
+def test_unpickle_pre_20(fname):
+    from shapely.testing import assert_geometries_equal
+
+    geom_type = fname.name.split("_")[0]
+    expected = TEST_DATA[geom_type]
+
+    with open(fname, "rb") as f:
+        with pytest.warns(UserWarning):
+            result = pickle.load(f)
+
+    assert_geometries_equal(result, expected)
+
+
+if __name__ == "__main__":
+    datadir = HERE / "data"
+    datadir.mkdir(exist_ok=True)
+
+    shapely_version = shapely.__version__
+    print(shapely_version)
+    print(shapely.geos.geos_version)
+
+    for name, geom in TEST_DATA.items():
+        if name == "emptypoint" and shapely.geos.geos_version < (3, 9, 0):
+            # Empty Points cannot be represented in WKB
+            continue
+        with open(datadir / f"{name}_{shapely_version}.pickle", "wb") as f:
+            pickle.dump(geom, f)