Repository: CartoDB/cartoframes
Pull request #1755 (2022-12-22 13:56:45)

Add compatibility with shapely>=1.7
### Context

`shapely` is a geopandas' dependency (`>1.7`), and with the version 2.0.0 released on December 12, some breaking changes were introduced. This means some parts of the code, like [this one]() are not working anymore, since `lgeos` is no longer defined: 

![Screenshot from 2022-12-22 12-57-54](https://user-images.githubusercontent.com/6042873/209149322-5dfceafd-0b3b-4f6e-ade9-aaf1a3322d79.png)

This makes some crucial parts of the core of CARTOFrames to fail, like the `to_carto` method.

### Relevant PR changes

- Adding `shapely` as requirement, with the following restrictions: `>1.7,<=1.7.1` to avoid the breaking changes.
- CI Changes:
  - Python 3.5 and 3.6 versions have reached end of life, so they are not available for `ubuntu-latest` versions. For that reason, we need to use `ubuntu-20.04`, and the last versions of `actions/checkout@v2` and `actions/setup-python@v4` for GitHub actions to be able to find them. [Job](https://github.com/CartoDB/cartoframes/actions/runs/3758298239/jobs/6386849233)
  - We need to disable `fail-fast`, otherwise we get a cancelled signal (`Error: The operation was canceled.`) every time we install Python dependencies. [Job](https://github.com/CartoDB/cartoframes/actions/runs/3758298239/jobs/6386849636)
- Failing tests:
  - If we execute the test with the last version of `markupsafe`, the following error is raised for every test. So we need to downgrade it to version `2.0.1` (or `1.1.1` for Python 3.5). [Job](https://github.com/CartoDB/cartoframes/actions/runs/3758693295/jobs/6387336328)
  ```
   E   ImportError: cannot import name 'soft_unicode' from 'markupsafe' (/home/runner/work/cartoframes/cartoframes/.tox/py37/lib/python3.7/site-packages/markupsafe/__init__.py)
  ```
  - The first approach was to use `<2.0` and that's enough to fix the reported issue. However, with that version, the following test was failing. To fix it, we would need to downgrade it to version `1.7.1`. [Job](https://github.com/CartoDB/cartoframes/actions/runs/3759544118/jobs/6389235514)
  ```
  _______________ test_read_carto_basegeometry_as_null_geom_value ________________
    
    mocker = <pytest_mock.plugin.MockerFixture object at 0x7f483cf208d0>
    
        def test_read_carto_basegeometry_as_null_geom_value(mocker):
            # Given
            cm_mock = mocker.patch.object(ContextManager, 'copy_to')
            cm_mock.return_value = GeoDataFrame({
                'cartodb_id': [1],
                'the_geom': [
                    None
                ]
            })
        
            # When
            gdf = read_carto('__source__', CREDENTIALS, null_geom_value=BaseGeometry())
        
            # Then
            expected = GeoDataFrame({
                'cartodb_id': [1],
                'the_geom': [
                    BaseGeometry()
                ]
            }, geometry='the_geom')
        
            cm_mock.assert_called_once_with('__source__', None, None, 3)
    >       assert expected.equals(gdf)
    
    tests/unit/io/test_carto.py:97: 
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
    .tox/py37/lib/python3.7/site-packages/geopandas/base.py:3284: in equals
        return self._data.equals(other._data)
    .tox/py37/lib/python3.7/site-packages/pandas/core/internals/base.py:119: in equals
        return self._equal_values(other)
    .tox/py37/lib/python3.7/site-packages/pandas/core/internals/managers.py:1310: in _equal_values
        return blockwise_all(self, other, array_equals)
    .tox/py37/lib/python3.7/site-packages/pandas/core/internals/ops.py:140: in blockwise_all
        res = op(info.lvals, info.rvals)
    .tox/py37/lib/python3.7/site-packages/pandas/core/dtypes/missing.py:518: in array_equals
        return left.equals(right)
    .tox/py37/lib/python3.7/site-packages/pandas/core/arrays/base.py:8[82](https://github.com/CartoDB/cartoframes/actions/runs/3759544118/jobs/6389235514#step:5:83): in equals
        equal_values = self == other
    .tox/py37/lib/python3.7/site-packages/geopandas/array.py:1313: in __eq__
        return self._binop(other, operator.eq)
    .tox/py37/lib/python3.7/site-packages/geopandas/array.py:1307: in _binop
        res = [op(a, b) for (a, b) in zip(lvalues, rvalues)]
    .tox/py37/lib/python3.7/site-packages/geopandas/array.py:1307: in <listcomp>
        res = [op(a, b) for (a, b) in zip(lvalues, rvalues)]
    .tox/py37/lib/python3.7/site-packages/shapely/geometry/base.py:281: in __eq__
        tuple(self.coords) == tuple(other.coords)
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
    
    self = <shapely.geometry.base.BaseGeometry object at 0x7f4[83](https://github.com/CartoDB/cartoframes/actions/runs/3759544118/jobs/6389235514#step:5:84)cb9[90](https://github.com/CartoDB/cartoframes/actions/runs/3759544118/jobs/6389235514#step:5:91)50>
    
        def _get_coords(self):
            """Access to geometry's coordinates (CoordinateSequence)"""
    >       raise NotImplementedError("set_coords must be provided by derived classes")
    E       NotImplementedError: set_coords must be provided by derived classes
    
    .tox/py37/lib/python3.7/site-packages/shapely/geometry/base.py:338: NotImplementedError
    ```
    - With Python 3.8, the following test fails if `numpy>1.22.4`. This only affects Python 3.8 because Python 3.7 was dropped with NumPy 1.22.0. So I've also added the `numpy<=1.22.4` requirement. [Job](https://github.com/CartoDB/cartoframes/actions/runs/3759768645/jobs/6389729223)
    ```
    ______________ ERROR collecting tests/unit/analysis/test_grid.py _______________
      tests/unit/analysis/test_grid.py:19: in <module>
          GDF_BOX = GeoDataFrame({'id': [1, 2], 'geom': [pol_1, pol_2]}, columns=['id', 'geom'], geometry='geom')
      .tox/py38/lib/python3.8/site-packages/geopandas/geodataframe.py:135: in __init__
          super().__init__(data, *args, **kwargs)
      .tox/py38/lib/python3.8/site-packages/pandas/core/frame.py:663: in __init__
          mgr = dict_to_mgr(data, index, columns, dtype=dtype, copy=copy, typ=manager)
      .tox/py38/lib/python3.8/site-packages/pandas/core/internals/construction.py:493: in dict_to_mgr
          return arrays_to_mgr(arrays, columns, index, dtype=dtype, typ=typ, consolidate=copy)
      .tox/py[38](https://github.com/CartoDB/cartoframes/actions/runs/3759652576/jobs/6389477230#step:5:39)/lib/python3.8/site-packages/pandas/core/internals/construction.py:123: in arrays_to_mgr
          arrays = _homogenize(arrays, index, dtype)
      .tox/py38/lib/python3.8/site-packages/pandas/core/internals/construction.py:617: in _homogenize
          val = sanitize_array(
      .tox/py38/lib/python3.8/site-packages/pandas/core/construction.py:6[42](https://github.com/CartoDB/cartoframes/actions/runs/3759652576/jobs/6389477230#step:5:43): in sanitize_array
          subarr = maybe_convert_platform(data)
      .tox/py38/lib/python3.8/site-packages/pandas/core/dtypes/cast.py:127: in maybe_convert_platform
          arr = construct_1d_object_array_from_listlike(values)
      .tox/py38/lib/python3.8/site-packages/pandas/core/dtypes/cast.py:1784: in construct_1d_object_array_from_listlike
          result[:] = values
      .tox/py38/lib/python3.8/site-packages/shapely/geometry/polygon.py:300: in __array_interface__
          raise NotImplementedError(
      E   NotImplementedError: A polygon does not itself provide the array interface. Its rings do.
  ```


Patch:
diff --git a/.github/workflows/cartoframes-ci.yml b/.github/workflows/cartoframes-ci.yml
index eb8b8ef7..f9f17221 100644
--- a/.github/workflows/cartoframes-ci.yml
+++ b/.github/workflows/cartoframes-ci.yml
@@ -10,18 +10,27 @@ on:
 jobs:
   test:
     strategy:
+      fail-fast: false
       matrix:
-        python-version: [3.5, 3.6, 3.7, 3.8]
+        include:
+          - os: ubuntu-20.04
+            python-version: 3.5
+          - os: ubuntu-20.04
+            python-version: 3.6
+          - os: ubuntu-latest
+            python-version: 3.7
+          - os: ubuntu-latest
+            python-version: 3.8
 
     name: Run tests on Python ${{ matrix.python-version }}
 
-    runs-on: ubuntu-latest
+    runs-on: ${{ matrix.os }}
 
     steps:
-    - uses: actions/checkout@v2
+    - uses: actions/checkout@v3
 
     - name: Set up Python ${{ matrix.python-version }}
-      uses: actions/setup-python@v1
+      uses: actions/setup-python@v4
       with:
         python-version: ${{ matrix.python-version }}
 
diff --git a/Makefile b/Makefile
index 5456d0a2..b6d32b9b 100644
--- a/Makefile
+++ b/Makefile
@@ -1,5 +1,9 @@
 init:
-	pip install -e .
+	pip install -U pip
+	pip install -e .[tests]
+
+lint:
+	flake8 cartoframes tests
 
 test:
 	pytest tests/unit/
diff --git a/cartoframes/data/observatory/catalog/dataset.py b/cartoframes/data/observatory/catalog/dataset.py
index 908786be..ced41526 100644
--- a/cartoframes/data/observatory/catalog/dataset.py
+++ b/cartoframes/data/observatory/catalog/dataset.py
@@ -500,8 +500,9 @@ class Dataset(CatalogEntity):
 
     @check_do_enabled
     def subscription_info(self, credentials=None):
-        """Get the subscription information of a Dataset, which includes the license, Terms of Service, rights, price, and
-        estimated time of delivery, among other metadata of interest during the :py:attr:`Dataset.subscription` process.
+        """Get the subscription information of a Dataset, which includes the license, Terms of Service, rights, price,
+        and estimated time of delivery, among other metadata of interest during the :py:attr:`Dataset.subscription`
+        process.
 
         Args:
             credentials (:py:class:`Credentials <cartoframes.auth.Credentials>`, optional):
diff --git a/cartoframes/io/managers/context_manager.py b/cartoframes/io/managers/context_manager.py
index 360998bd..ef2b11dc 100644
--- a/cartoframes/io/managers/context_manager.py
+++ b/cartoframes/io/managers/context_manager.py
@@ -168,7 +168,7 @@ class ContextManager:
     def delete_table(self, table_name):
         query = _drop_table_query(table_name)
         output = self.execute_query(query)
-        return not('notices' in output and 'does not exist' in output['notices'][0])
+        return not ('notices' in output and 'does not exist' in output['notices'][0])
 
     def _delete_function(self, function_name):
         query = _drop_function_query(function_name)
diff --git a/cartoframes/utils/geom_utils.py b/cartoframes/utils/geom_utils.py
index dbd8d4ce..b784fed8 100644
--- a/cartoframes/utils/geom_utils.py
+++ b/cartoframes/utils/geom_utils.py
@@ -218,7 +218,7 @@ def _load_ewkt(egeom):
     srid, geom = _extract_srid(egeom)
     ogeom = _load_wkt(geom)
     if srid:
-        shapely.geos.lgeos.GEOSSetSRID(ogeom._geom, int(srid))
+        ogeom = set_srid(ogeom, int(srid))
     return ogeom
 
 
@@ -241,7 +241,7 @@ def encode_geometry_ewkt(geom, srid=4326):
 
 def encode_geometry_ewkb(geom, srid=4326):
     if isinstance(geom, shapely.geometry.base.BaseGeometry):
-        shapely.geos.lgeos.GEOSSetSRID(geom._geom, srid)
+        geom = set_srid(geom, srid)
         return shapely.wkb.dumps(geom, hex=True, include_srid=True)
 
 
@@ -272,3 +272,18 @@ def get_crs(gdf):
         return gdf.crs['init']
     else:
         return str(gdf.crs)
+
+
+def get_srid(geom):
+    if shapely.__version__ < '2.0':
+        return shapely.geos.lgeos.GEOSGetSRID(geom._geom)
+    else:
+        return shapely.get_srid(geom)
+
+
+def set_srid(geom, srid):
+    if shapely.__version__ < '2.0':
+        shapely.geos.lgeos.GEOSSetSRID(geom._geom, int(srid))
+        return geom
+    else:
+        return shapely.set_srid(geom, int(srid))
diff --git a/setup.py b/setup.py
index 93b88715..705ab2b0 100644
--- a/setup.py
+++ b/setup.py
@@ -26,6 +26,7 @@ def get_version():
 REQUIRES = [
     'appdirs>=1.4.3,<2.0',
     'carto>=1.11.3,<2.0',
+    'markupsafe<=2.0.1',
     'jinja2>=2.10.1,<3.0',
     'pandas>=0.25.0',
     'geopandas>=0.6.0,<1.0',


Test patch:
diff --git a/tests/unit/io/test_carto.py b/tests/unit/io/test_carto.py
index b3638f21..3ff84c62 100644
--- a/tests/unit/io/test_carto.py
+++ b/tests/unit/io/test_carto.py
@@ -5,7 +5,6 @@ import random
 from pandas import Index
 from geopandas import GeoDataFrame
 from shapely.geometry import Point
-from shapely.geometry.base import BaseGeometry
 from shapely import wkt
 
 from carto.exceptions import CartoException
@@ -83,17 +82,18 @@ def test_read_carto_basegeometry_as_null_geom_value(mocker):
     })
 
     # When
-    gdf = read_carto('__source__', CREDENTIALS, null_geom_value=BaseGeometry())
+    gdf = read_carto('__source__', CREDENTIALS, null_geom_value=None)
 
     # Then
     expected = GeoDataFrame({
         'cartodb_id': [1],
         'the_geom': [
-            BaseGeometry()
+            None
         ]
     }, geometry='the_geom')
 
     cm_mock.assert_called_once_with('__source__', None, None, 3)
+    print(expected, gdf)
     assert expected.equals(gdf)
     assert gdf.crs == 'epsg:4326'
 
diff --git a/tests/unit/utils/test_geom_utils.py b/tests/unit/utils/test_geom_utils.py
index 9f0d3734..51d12730 100644
--- a/tests/unit/utils/test_geom_utils.py
+++ b/tests/unit/utils/test_geom_utils.py
@@ -3,12 +3,12 @@
 import pandas as pd
 import geopandas as gpd
 
-from shapely.geos import lgeos
 from shapely.geometry import Point
 
 from cartoframes.utils.geom_utils import (ENC_EWKT, ENC_SHAPELY, ENC_WKB,
                                           ENC_WKB_BHEX, ENC_WKB_HEX, ENC_WKT,
-                                          decode_geometry, decode_geometry_item, detect_encoding_type)
+                                          decode_geometry, decode_geometry_item,
+                                          detect_encoding_type, get_srid)
 
 
 class TestGeomUtils(object):
@@ -92,38 +92,38 @@ class TestGeomUtils(object):
     def test_decode_geometry_wkb(self):
         geom = decode_geometry_item(
             b'\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00H\x93@\x00\x00\x00\x00\x00\x9d\xb6@', ENC_WKB)
-        assert lgeos.GEOSGetSRID(geom._geom) == 0
+        assert get_srid(geom) == 0
         assert geom.wkb == b'\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00H\x93@\x00\x00\x00\x00\x00\x9d\xb6@'
 
         geom = decode_geometry_item(
             b'\x01\x01\x00\x00 \xe6\x10\x00\x00\x00\x00\x00\x00\x00H\x93@\x00\x00\x00\x00\x00\x9d\xb6@', ENC_WKB)  # ext
-        assert lgeos.GEOSGetSRID(geom._geom) == 4326
+        assert get_srid(geom) == 4326
         assert geom.wkb == b'\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00H\x93@\x00\x00\x00\x00\x00\x9d\xb6@'
 
     def test_decode_geometry_wkb_hex(self):
         geom = decode_geometry_item('0101000000000000000048934000000000009DB640', ENC_WKB_HEX)
-        assert lgeos.GEOSGetSRID(geom._geom) == 0
+        assert get_srid(geom) == 0
         assert geom.wkb_hex == '0101000000000000000048934000000000009DB640'
 
         geom = decode_geometry_item('0101000020E6100000000000000048934000000000009DB640', ENC_WKB_HEX)  # ext
-        assert lgeos.GEOSGetSRID(geom._geom) == 4326
+        assert get_srid(geom) == 4326
         assert geom.wkb_hex == '0101000000000000000048934000000000009DB640'
 
     def test_decode_geometry_wkb_bhex(self):
         geom = decode_geometry_item(b'0101000000000000000048934000000000009DB640', ENC_WKB_BHEX)
-        assert lgeos.GEOSGetSRID(geom._geom) == 0
+        assert get_srid(geom) == 0
         assert geom.wkb == b'\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00H\x93@\x00\x00\x00\x00\x00\x9d\xb6@'
 
         geom = decode_geometry_item(b'0101000020E6100000000000000048934000000000009DB640', ENC_WKB_BHEX)  # ext
-        assert lgeos.GEOSGetSRID(geom._geom) == 4326
+        assert get_srid(geom) == 4326
         assert geom.wkb == b'\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00H\x93@\x00\x00\x00\x00\x00\x9d\xb6@'
 
     def test_decode_geometry_wkt(self):
         geom = decode_geometry_item('POINT (1234 5789)', ENC_WKT)
-        assert lgeos.GEOSGetSRID(geom._geom) == 0
+        assert get_srid(geom) == 0
         assert geom.wkt == 'POINT (1234 5789)'
 
     def test_decode_geometry_ewkt(self):
         geom = decode_geometry_item('SRID=4326;POINT (1234 5789)', ENC_EWKT)  # ext
-        assert lgeos.GEOSGetSRID(geom._geom) == 4326
+        assert get_srid(geom) == 4326
         assert geom.wkt == 'POINT (1234 5789)'