Repository: rasterio/rasterio

Issue:
support pickle for multiprocessing with rasterio.profiles.Profile data
Trying to do a bit of multiprocessing parallel, async IO to write out GeoTIFF data.  If there's a better way to do this, would like to hear about it.

Related to #909 - another concern with serializing the profile data.

## OS and versions
```
ubuntu 18.04

conda version : 4.6.2
conda-build version : not installed
python version : 3.7.2.final.0

rasterio==1.0.15
GDAL 2.4.0, released 2018/12/14
```

## example pseudo code
```python
def raster_to_geotiff(level_data):
    try:
        with rasterio.open(level_data.level_file, 'w', **level_data.level_profile) as dst:
            dst.write(level_data.data, 1)
        return True
    except Exception:
        return False

@dataclass
class LevelData:
    level: float
    data: np.ndarray
    level_file: Path
    level_profile: rasterio.profiles.Profile

level_data = (
        LevelData(
                level=d.level_value,
                data=d.ndarray.astype('uint8'),
                level_file=d.a_pathlib_path_instance,
                level_profile=an_instance_of_rasterio_profiles_Profile_from_src_geotiff
        )
        for d in datasets
    )

with multiprocessing.Pool() as pool:
    tiff_output_success = pool.map(raster_to_geotiff, level_data)
```

## error

It seems like `File "stringsource", line 2, in rasterio._crs._CRS.__reduce_cython__` cannot pickle in this error below.  This might be a problem with pickling the `rasterio.profiles.Profile` data.  If unit tests on that class could check it for pickle support, this might just work with the multiprocessing.Pool for async IO.

```
  File "/home/ubuntu/miniconda3/envs/gis-dataprocessing/lib/python3.7/multiprocessing/pool.py", line 290, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/home/ubuntu/miniconda3/envs/gis-dataprocessing/lib/python3.7/multiprocessing/pool.py", line 683, in get
    raise self._value
  File "/home/ubuntu/miniconda3/envs/gis-dataprocessing/lib/python3.7/multiprocessing/pool.py", line 457, in _handle_tasks
    put(task)
  File "/home/ubuntu/miniconda3/envs/gis-dataprocessing/lib/python3.7/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/home/ubuntu/miniconda3/envs/gis-dataprocessing/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
  File "stringsource", line 2, in rasterio._crs._CRS.__reduce_cython__
TypeError: no default __reduce__ due to non-trivial __cinit__
```



Discussion:
Yes, I can make CRS pickable. It used to be, but only by accident.

Here's a workaround for the meanwhile, assuming that you are able to stick to only the EPSG systems:

```
crs = an_instance_of_rasterio_profiles_Profile_from_src_geotiff['crs']
crs_string = 'EPSG:{}'.format(crs.to_epsg())
an_instance_of_rasterio_profiles_Profile_from_src_geotiff['crs'] = crs_string
```

rasterio.open's crs parameter can be a CRS instance or string like "EPSG:4326".
It seems like the `rasterio.crs.CRS.__repr__` or another dunder method `__str__` could wrap one of the `to_xyz()` representations like `to_proj4` or `to_wkt` or something - maybe whatever the general consensus is on the best default among the GDAL/OGR/PROJ4 community?  Would be nice if geopandas could play along too.

We have to be careful to avoid `None`, e.g. a GeoTIFF example we have gives:
```python
>>> src.crs.to_proj4()
'+proj=tmerc +lat_0=24.33333333333333 +lon_0=-81 +k=0.9999411764705882 +x_0=200000.0000000001 +y_0=0 +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +units=us-ft +no_defs=True'
>>> src.crs.to_wkt()
'PROJCS["unnamed",GEOGCS["GRS 1980(IUGG, 1980)",DATUM["unknown",SPHEROID["GRS80",6378137,298.257222101],TOWGS84[0,0,0,0,0,0,0]],PRIMEM["Greenwich",0],UNIT["degree",0.0174532925199433]],PROJECTION["Transverse_Mercator"],PARAMETER["latitude_of_origin",24.33333333333333],PARAMETER["central_meridian",-81],PARAMETER["scale_factor",0.9999411764705882],PARAMETER["false_easting",656166.6666666669],PARAMETER["false_northing",0],UNIT["US survey foot",0.3048006096012192,AUTHORITY["EPSG","9003"]]]'
>>> src.crs.to_epsg()
>>> src.crs.to_epsg() is None
```
WKT (version 1) is the best string representation to use now.

This is what CRS.to_wkt returns. It's what I'll use for pickling: `CRS.__getstate__` will return a WKT string and `CRS.__setstate__` will call CRS.from_wkt.
A preference for WKT-1 is also noted on a PDAL tutorial
- https://pdal.io/tutorial/las.html#spatial-reference-system
It's out of scope here, but geopandas docs and methods appear to be a light-on with regard to CRS support.  In https://github.com/geopandas/geopandas/blob/master/geopandas/geodataframe.py the code behaves like anything can be applied to the crs field.  All is not lost, however, it should raise somewhere by the time it dives down to Fiona/ogr code.

Resolving patch:
diff --git a/rasterio/crs.py b/rasterio/crs.py
--- a/rasterio/crs.py
+++ b/rasterio/crs.py
@@ -89,6 +89,12 @@ def __eq__(self, other):
         other = CRS.from_user_input(other)
         return (self._crs == other._crs)
 
+    def __getstate__(self):
+        return self.wkt
+
+    def __setstate__(self, state):
+        self._crs = _CRS.from_wkt(state)
+
     def to_proj4(self):
         """Convert CRS to a PROJ4 string
 


Test patch:
diff --git a/tests/test_crs.py b/tests/test_crs.py
--- a/tests/test_crs.py
+++ b/tests/test_crs.py
@@ -3,6 +3,7 @@
 import json
 import logging
 import os
+import pickle
 import subprocess
 
 import pytest
@@ -431,3 +432,11 @@ def test_empty_crs_str():
 def test_issue1620():
     """Different forms of EPSG:3857 are equal"""
     assert CRS.from_wkt('PROJCS["WGS 84 / Pseudo-Mercator",GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AUTHORITY["EPSG","4326"]],PROJECTION["Mercator_1SP"],PARAMETER["central_meridian",0],PARAMETER["scale_factor",1],PARAMETER["false_easting",0],PARAMETER["false_northing",0],UNIT["metre",1,AUTHORITY["EPSG","9001"]],AXIS["X",EAST],AXIS["Y",NORTH],EXTENSION["PROJ4","+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +wktext +no_defs"],AUTHORITY["EPSG","3857"]]') == CRS.from_dict(init='epsg:3857')
+
+
+@pytest.mark.parametrize('factory,arg', [(CRS.from_epsg, 3857), (CRS.from_dict, {'ellps': 'WGS84', 'proj': 'stere', 'lat_0': -90.0, 'lon_0': 0.0, 'x_0': 0.0, 'y_0': 0.0, 'lat_ts': -70, 'no_defs': True})])
+def test_pickle(factory, arg):
+    """A CRS is pickleable"""
+    crs1 = factory(arg)
+    crs2 = pickle.loads(pickle.dumps(crs1))
+    assert crs2 == crs1
diff --git a/tests/test_profile.py b/tests/test_profile.py
--- a/tests/test_profile.py
+++ b/tests/test_profile.py
@@ -1,3 +1,6 @@
+"""Profile tests"""
+
+import pickle
 import warnings
 
 import pytest
@@ -77,8 +80,8 @@ def test_blockysize_guard(tmpdir):
         rasterio.open(tiffname, 'w', **profile)
 
 
-def test_profile_overlay():
-    with rasterio.open('tests/data/RGB.byte.tif') as src:
+def test_profile_overlay(path_rgb_byte_tif):
+    with rasterio.open(path_rgb_byte_tif) as src:
         kwds = src.profile
     kwds.update(**default_gtiff_profile)
     assert kwds['tiled']
@@ -94,9 +97,9 @@ def test_dataset_profile_property_tiled(data):
         assert src.profile['tiled'] is True
 
 
-def test_dataset_profile_property_untiled(data):
+def test_dataset_profile_property_untiled(data, path_rgb_byte_tif):
     """An untiled dataset's profile has no block sizes"""
-    with rasterio.open('tests/data/RGB.byte.tif') as src:
+    with rasterio.open(path_rgb_byte_tif) as src:
         assert 'blockxsize' not in src.profile
         assert 'blockysize' not in src.profile
         assert src.profile['tiled'] is False
@@ -108,3 +111,14 @@ def test_profile_affine_set():
     profile['transform'] = 'foo'
     with pytest.raises(TypeError):
         profile['affine'] = 'bar'
+
+
+def test_profile_pickle():
+    """Standard profile can be pickled"""
+    assert pickle.loads(pickle.dumps(DefaultGTiffProfile())) == DefaultGTiffProfile()
+
+
+def test_dataset_profile_pickle(path_rgb_byte_tif):
+    """Dataset profiles can be pickled"""
+    with rasterio.open(path_rgb_byte_tif) as src:
+        assert pickle.loads(pickle.dumps(src.profile)) == src.profile