Repository: geopandas/geopandas

Issue:
Error when loading pickled saved df with version < 0.6.3
I'm using geopandas for a while and am used to save my gdf as pickle files, and then open it directly with `pandas`. I have some old files that I want to open. I guess the files were saved as my pandas version was `0.6.2`, but I'm not sure. 

Now that I'm trying to re-open it I get the following error : 
`AttributeError: 'GeoDataFrame' object has no attribute '_item_cache'`

Here is a code sample which makes the problem reproducible : 
with `geopandas==0.6.3` run 
```
import pandas as pd
import geopandas as gpd
from shapely.geometry import Point

data = [[3, 45, 1], [3.1, 45.2, 1.5], [2.7, 44.5, 2.1]]
data = pd.DataFrame(data, columns=['Longitude', 'Latitude', 'value'])
data['geometry'] = list(zip(data['Longitude'], data['Latitude']))
data['geometry'] = data['geometry'].apply(Point)
data = gpd.GeoDataFrame(data)

data.to_pickle('df.pkl')
```

And then with `geopandas==0.8.0` (or `0.7.0`) try : 
```
import pandas as pd
df = pd.read_pickle('df.pkl')
```

It should raise the same error. 

- I'm not sure if it's a problem of `pickle` or `geopandas`, I naively tried to grep `__item_cache` in the gpd code but it wasn't successful either with `0.6.3` or with `0.8.0`
- This is troublesome because I have to keep `0.6.3` in the meantime unless someone sees a turnaround (I could make a "clean save" of all my pickled gdf with `0.6.3` which can be read with `0.8.0` but I don't know if this is possible)
- Maybe my workflow is terrible and then I'll gladly take some advice ! 



Discussion:
I am not sure if there is a way how to open old pickle in new pandas. Pickle as a format is not meant for a data storage, especially not with compatibility between versions. The nature of pickle itself does not allow that, if there are significant changes to pandas or geopandas in between the versions. These could be under the hood but still makes the pickle file unusable. My recommendation would be to save those pickle files as some standard geospatial data format with 0.6.3 so you can open that in any other version.
I did what you recommended @martinfleis, closing this issue
Pickle itself is indeed not necessarily stable across versions, but, when using the pandas `to_pickle`/`read_pickle` methods, pandas has a bunch of compatibility code to ensure reading in pickles from a range of versions is supported.  So this might be a bug in pandas.

@SylvainLan can you still check which versions of pandas you have in both environments?
I'm using 1.0.5
Also in the environment with geopandas 0.6.3?
Yes, when I run `pip install 'geopandas==0.6.3'` I don't get asked to downgrade pandas too
I ran the example above with `pandas==1.0.0` or `pandas==0.25.0` and it raises the same error

OK, thanks, I can indeed reproduce this with pandas 1.0 and geopandas 0.6 vs 0.8. 
I assume this is due to changes in the CRS attribute handling. It is now a property with a getter/setter, and not a simple attribute on the class. The problem seems to be that while unpickling, it tries to set `crs` which fails if the geodataframe object is not yet fully initialized.
The problem comes from the fact that older pandas versions (< 0.7) used `crs` as entry in `_metadata` and now we use `_crs`. So when unpickling, it tries to set `crs` instead of `_crs` (going through the property settter instead of the basic attribute assignment).

We might be able to solve this with some compat code in GeoDataFrame:

```python
    def __setstate__(self, state):
        # overriding DataFrame method for compat with older pickles (CRS handling)

        if isinstance(state, dict):
            if "_metadata" in state and "crs" in state["_metadata"]:
                metadata = state["_metadata"]
                metadata[metadata.index('crs')] = "_crs"
            if "crs" in state and "_crs" not in state:
                crs = state.pop("crs")
                state["_crs"] = pyproj.CRS.from_user_input(crs) if crs is not None else crs
        
        super().__setstate__(state)
```

this already reads in the pickle file into a GeoDataFrame fine, only it now still creates a GeometryArray without `_crs` .. so some more compat code would be needed there.
I tried resolving what you mentioned above @jorisvandenbossche but I couldn't do any better than to set the geometry's crs at the end of the method `self.geometry.crs = self._crs`

Resolving patch:
diff --git a/geopandas/array.py b/geopandas/array.py
--- a/geopandas/array.py
+++ b/geopandas/array.py
@@ -384,6 +384,13 @@ def __setstate__(self, state):
             self.data = geoms
             self.base = None
 
+    else:
+
+        def __setstate__(self, state):
+            if "_crs" not in state:
+                state["_crs"] = None
+            self.__dict__.update(state)
+
     # -------------------------------------------------------------------------
     # Geometry related methods
     # -------------------------------------------------------------------------
diff --git a/geopandas/geodataframe.py b/geopandas/geodataframe.py
--- a/geopandas/geodataframe.py
+++ b/geopandas/geodataframe.py
@@ -329,6 +329,28 @@ def crs(self, value):
                 # column called 'geometry' without geometry
                 self._crs = None if not value else CRS.from_user_input(value)
 
+    def __setstate__(self, state):
+        # overriding DataFrame method for compat with older pickles (CRS handling)
+        if isinstance(state, dict):
+            if "_metadata" in state and "crs" in state["_metadata"]:
+                metadata = state["_metadata"]
+                metadata[metadata.index("crs")] = "_crs"
+            if "crs" in state and "_crs" not in state:
+                crs = state.pop("crs")
+                state["_crs"] = CRS.from_user_input(crs) if crs is not None else crs
+
+        super().__setstate__(state)
+
+        # for some versions that didn't yet have CRS at array level -> crs is set
+        # at GeoDataFrame level with '_crs' (and not 'crs'), so without propagating
+        # to the GeoSeries/GeometryArray
+        try:
+            if self.crs is not None:
+                if self.geometry.values.crs is None:
+                    self.crs = self.crs
+        except Exception:
+            pass
+
     @classmethod
     def from_file(cls, filename, **kwargs):
         """Alternate constructor to create a ``GeoDataFrame`` from a file.


Test patch:
diff --git a/geopandas/io/tests/data/pickle/0.5.1_pd-0.25.3_py-3.7.3_x86_64_linux.pickle b/geopandas/io/tests/data/pickle/0.5.1_pd-0.25.3_py-3.7.3_x86_64_linux.pickle
new file mode 100644
Binary files /dev/null and b/geopandas/io/tests/data/pickle/0.5.1_pd-0.25.3_py-3.7.3_x86_64_linux.pickle differ
diff --git a/geopandas/io/tests/data/pickle/0.6.3_pd-0.25.3_py-3.8.0_x86_64_linux.pickle b/geopandas/io/tests/data/pickle/0.6.3_pd-0.25.3_py-3.8.0_x86_64_linux.pickle
new file mode 100644
Binary files /dev/null and b/geopandas/io/tests/data/pickle/0.6.3_pd-0.25.3_py-3.8.0_x86_64_linux.pickle differ
diff --git a/geopandas/io/tests/data/pickle/0.7.0_pd-1.0.4_py-3.7.6_x86_64_linux.pickle b/geopandas/io/tests/data/pickle/0.7.0_pd-1.0.4_py-3.7.6_x86_64_linux.pickle
new file mode 100644
Binary files /dev/null and b/geopandas/io/tests/data/pickle/0.7.0_pd-1.0.4_py-3.7.6_x86_64_linux.pickle differ
diff --git a/geopandas/io/tests/data/pickle/0.8.0_pd-1.0.5_py-3.8.3_x86_64_linux.pickle b/geopandas/io/tests/data/pickle/0.8.0_pd-1.0.5_py-3.8.3_x86_64_linux.pickle
new file mode 100644
Binary files /dev/null and b/geopandas/io/tests/data/pickle/0.8.0_pd-1.0.5_py-3.8.3_x86_64_linux.pickle differ
diff --git a/geopandas/io/tests/generate_legacy_storage_files.py b/geopandas/io/tests/generate_legacy_storage_files.py
new file mode 100644
--- /dev/null
+++ b/geopandas/io/tests/generate_legacy_storage_files.py
@@ -0,0 +1,98 @@
+"""
+Script to create the data and write legacy storage (pickle) files.
+
+Based on pandas' generate_legacy_storage_files.py script.
+
+To use this script, create an environment for which you want to
+generate pickles, activate the environment, and run this script as:
+
+$ python geopandas/geopandas/io/tests/generate_legacy_storage_files.py \
+    geopandas/geopandas/io/tests/data/pickle/ pickle
+
+This script generates a storage file for the current arch, system,
+
+The idea here is you are using the *current* version of the
+generate_legacy_storage_files with an *older* version of geopandas to
+generate a pickle file. We will then check this file into a current
+branch, and test using test_pickle.py. This will load the *older*
+pickles and test versus the current data that is generated
+(with master). These are then compared.
+
+"""
+import os
+import pickle
+import platform
+import sys
+
+import pandas as pd
+
+import geopandas
+from shapely.geometry import Point
+
+
+def create_pickle_data():
+    """ create the pickle data """
+
+    # custom geometry column name
+    gdf_the_geom = geopandas.GeoDataFrame(
+        {"a": [1, 2, 3], "the_geom": [Point(1, 1), Point(2, 2), Point(3, 3)]},
+        geometry="the_geom",
+    )
+
+    # with crs
+    gdf_crs = geopandas.GeoDataFrame(
+        {"a": [0.1, 0.2, 0.3], "geometry": [Point(1, 1), Point(2, 2), Point(3, 3)]},
+        crs="EPSG:4326",
+    )
+
+    return dict(gdf_the_geom=gdf_the_geom, gdf_crs=gdf_crs)
+
+
+def platform_name():
+    return "_".join(
+        [
+            str(geopandas.__version__),
+            "pd-" + str(pd.__version__),
+            "py-" + str(platform.python_version()),
+            str(platform.machine()),
+            str(platform.system().lower()),
+        ]
+    )
+
+
+def write_legacy_pickles(output_dir):
+    print(
+        "This script generates a storage file for the current arch, system, "
+        "and python version"
+    )
+    print("geopandas version: {}").format(geopandas.__version__)
+    print("   output dir    : {}".format(output_dir))
+    print("   storage format: pickle")
+
+    pth = "{}.pickle".format(platform_name())
+
+    fh = open(os.path.join(output_dir, pth), "wb")
+    pickle.dump(create_pickle_data(), fh, pickle.DEFAULT_PROTOCOL)
+    fh.close()
+
+    print("created pickle file: {}".format(pth))
+
+
+def main():
+    if len(sys.argv) != 3:
+        exit(
+            "Specify output directory and storage type: generate_legacy_"
+            "storage_files.py <output_dir> <storage_type> "
+        )
+
+    output_dir = str(sys.argv[1])
+    storage_type = str(sys.argv[2])
+
+    if storage_type == "pickle":
+        write_legacy_pickles(output_dir=output_dir)
+    else:
+        exit("storage_type must be one of {'pickle'}")
+
+
+if __name__ == "__main__":
+    main()
diff --git a/geopandas/io/tests/test_pickle.py b/geopandas/io/tests/test_pickle.py
new file mode 100644
--- /dev/null
+++ b/geopandas/io/tests/test_pickle.py
@@ -0,0 +1,60 @@
+"""
+See generate_legacy_storage_files.py for the creation of the legacy files.
+
+"""
+from distutils.version import LooseVersion
+import glob
+import os
+import pathlib
+
+import pandas as pd
+
+import pyproj
+
+import pytest
+from geopandas.testing import assert_geodataframe_equal
+from geopandas import _compat as compat
+
+
+DATA_PATH = pathlib.Path(os.path.dirname(__file__)) / "data"
+
+
+@pytest.fixture(scope="module")
+def current_pickle_data():
+    # our current version pickle data
+    from .generate_legacy_storage_files import create_pickle_data
+
+    return create_pickle_data()
+
+
+files = glob.glob(str(DATA_PATH / "pickle" / "*.pickle"))
+
+
+@pytest.fixture(params=files, ids=[p.split("/")[-1] for p in files])
+def legacy_pickle(request):
+    return request.param
+
+
+@pytest.mark.skipif(
+    compat.USE_PYGEOS or (str(pyproj.__version__) < LooseVersion("2.4")),
+    reason=(
+        "pygeos-based unpickling currently only works for pygeos-written files; "
+        "old pyproj versions can't read pickles from newer pyproj versions"
+    ),
+)
+def test_legacy_pickles(current_pickle_data, legacy_pickle):
+    result = pd.read_pickle(legacy_pickle)
+
+    for name, value in result.items():
+        expected = current_pickle_data[name]
+        assert_geodataframe_equal(value, expected)
+
+
+def test_round_trip_current(tmpdir, current_pickle_data):
+    data = current_pickle_data
+
+    for name, value in data.items():
+        path = str(tmpdir / "{}.pickle".format(name))
+        value.to_pickle(path)
+        result = pd.read_pickle(path)
+        assert_geodataframe_equal(result, value)