Repository: geopandas/geopandas

Issue:
BUG: to_postgis() option if_exists="append" fails to find SRID
- [x] I have checked that this issue has not already been reported.

- [x] I have confirmed this bug exists on the latest version of geopandas.

- [ ] (optional) I have confirmed this bug exists on the main branch of geopandas.

---


#### Code Sample

```python
import geopandas
from shapely.geometry import Point
from sqlalchemy import create_engine

URI="postgresql:///user:password@localhost:5432/test"
eng=create_engine(URI)

d = {'col1': ['name1', 'name2'], 'geometry': [Point(1, 2), Point(2, 1)]}
gdf = geopandas.GeoDataFrame(d, crs="EPSG:6344")
with eng.connect() as conn:
    gdf.to_postgis('table_name', conn, if_exists="append", schema='schema_name')

```

#### Problem description

If the table does not already exist, this works fine, and creates a new table.

If the table does exist, the ``AttributeError`` below is thrown. I confirmed in standard pandas and sqlalchemy that the SRID can be found with: 
```python
with eng.connect() as conn:
    pd.read_sql(sqlalchemy.text("SELECT Find_SRID('schema_name', 'table_name', 'geom');"), conn)

Output:
   find_srid
0       6344
```

```
AttributeError: 'str' object has no attribute '_execute_on_connection'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/home/user/.local/share/virtualenvs/env/lib/python3.8/site-packages/geopandas/geodataframe.py", line 1931, in to_postgis
    geopandas.io.sql._write_postgis(
  File "/home/user/.local/share/virtualenvs/env/lib/python3.8/site-packages/geopandas/io/sql.py", line 428, in _write_postgis
    target_srid = connection.execute(
  File "/home/user/.local/share/virtualenvs/env/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1412, in execute
    raise exc.ObjectNotExecutableError(statement) from err
sqlalchemy.exc.ObjectNotExecutableError: Not an executable object: "SELECT Find_SRID('schema_name', 'table_name', 'geom');"
```

#### Expected Output

Should be able to find the SRID and append to the table as expected.

#### Output of ``geopandas.show_versions()``

``0.12.2``

</details>



Discussion:
This happens to me too, maybe the code to run sqlalchemy is not using ```sqlalchemy.text```?.

I have a workaround, use sqlalchemy 1 (not future) and pass the engine to geopandas, with that works for me.
Hi,
What do you mean by pass the engine to geopandas? I am also getting the exact same issue.

Thanks!
The main problem seems to be a piece of the code in the sql.py (entire path: "geopandas/io/sql.py") that is used to check if the SRID from the dataset you are trying to upload to PostGIS, has the same SRID of the table. If we remove the if statement from line 423 from the sql.py code, for example, the code works as it should (this is a workaround, but this does not solve the problem, and it will be a problem if the SRIDs are different). 

To solve the problem, as suggested here, we need to change the code like this:

From:
```
if if_exists == "append":
        # Check that the geometry srid matches with the current GeoDataFrame
        with _get_conn(con) as connection:
            # Only check SRID if table exists
            if connection.dialect.has_table(connection, name, schema):
                
                from sqlalchemy import text
                target_srid = connection.execute(
                    "SELECT Find_SRID('{schema}', '{table}', '{geom_col}');".format(
                        schema=schema_name, table=name, geom_col=geom_name
                    )
                ).fetchone()[0]

                if target_srid != srid:
                    msg = (
                        "The CRS of the target table (EPSG:{epsg_t}) differs from the "
                        "CRS of current GeoDataFrame (EPSG:{epsg_src}).".format(
                            epsg_t=target_srid, epsg_src=srid
                        )
                    )
                    raise ValueError(msg)
```
                    
To 

```
from sqlalchemy import text
if if_exists == "append":
        # Check that the geometry srid matches with the current GeoDataFrame
        with _get_conn(con) as connection:
            # Only check SRID if table exists
            if connection.dialect.has_table(connection, name, schema):
                
                target_srid = connection.execute(
                    text("SELECT Find_SRID('{schema}', '{table}', '{geom_col}');".format(
                        schema=schema_name, table=name, geom_col=geom_name
                    ))
                ).fetchone()[0]

                if target_srid != srid:
                    msg = (
                        "The CRS of the target table (EPSG:{epsg_t}) differs from the "
                        "CRS of current GeoDataFrame (EPSG:{epsg_src}).".format(
                            epsg_t=target_srid, epsg_src=srid
                        )
                    )
                    raise ValueError(msg)
```

It worked for my purposes here, but I could not create a new pull request, but it is the suggestion to solve this error.
I also ran into this error and other strange type errors (unrelated to geopandas) using sqlalchemy 2.0.1. Downgrading to an earlier version by specifying sqlalchemy<2.0.0 in my environment install solved this issue and others.
@felipempinto, why weren't you able to create a PR? I saw that there are no tests written for to_postgis()
@rietesh the tests are [here](https://github.com/geopandas/geopandas/blob/main/geopandas/io/tests/test_sql.py), the function that is called by `to_postgis` is called `_write_postgis` and is tested directly. See the implementation:

https://github.com/geopandas/geopandas/blob/00e3748c987f5b9a14b5df5233421710811d75bf/geopandas/geodataframe.py#L1948-L1950 
@rietesh I didn't have time to do that, I started to create a new PR, but I got stuck in some errors during the testing phase, so I thought this could take more time than I had. So, I wrote the solution I found here, and this could help someone to create the PR.

And yes, there are tests already created, as explained by @martinfleis, for the write_postgis function, but I haven't created a PR yet, due to some other errors that showed up.
I raised the PR to fix this bug #2807 

Resolving patch:
diff --git a/geopandas/io/sql.py b/geopandas/io/sql.py
--- a/geopandas/io/sql.py
+++ b/geopandas/io/sql.py
@@ -31,7 +31,10 @@ def _get_conn(conn_or_engine):
     from sqlalchemy.engine.base import Engine, Connection
 
     if isinstance(conn_or_engine, Connection):
-        with conn_or_engine.begin():
+        if not conn_or_engine.in_transaction():
+            with conn_or_engine.begin():
+                yield conn_or_engine
+        else:
             yield conn_or_engine
     elif isinstance(conn_or_engine, Engine):
         with conn_or_engine.begin() as conn:
@@ -392,6 +395,7 @@ def _write_postgis(
     """
     try:
         from geoalchemy2 import Geometry
+        from sqlalchemy import text
     except ImportError:
         raise ImportError("'to_postgis()' requires geoalchemy2 package.")
 
@@ -428,8 +432,10 @@ def _write_postgis(
             # Only check SRID if table exists
             if connection.dialect.has_table(connection, name, schema):
                 target_srid = connection.execute(
-                    "SELECT Find_SRID('{schema}', '{table}', '{geom_col}');".format(
-                        schema=schema_name, table=name, geom_col=geom_name
+                    text(
+                        "SELECT Find_SRID('{schema}', '{table}', '{geom_col}');".format(
+                            schema=schema_name, table=name, geom_col=geom_name
+                        )
                     )
                 ).fetchone()[0]
 


Test patch:
diff --git a/.github/workflows/tests.yaml b/.github/workflows/tests.yaml
--- a/.github/workflows/tests.yaml
+++ b/.github/workflows/tests.yaml
@@ -107,7 +107,7 @@ jobs:
           pytest -v -r a -n auto --color=yes --cov=geopandas --cov-append --cov-report term-missing --cov-report xml geopandas/
 
       - name: Test with PostGIS
-        if: contains(matrix.env, '39-pd13-conda-forge.yaml') && contains(matrix.os, 'ubuntu')
+        if: (contains(matrix.env, '39-pd13-conda-forge.yaml') || contains(matrix.env, '311-latest-conda-forge.yaml')) && contains(matrix.os, 'ubuntu')
         env:
           PGUSER: postgres
           PGPASSWORD: postgres
diff --git a/geopandas/io/tests/test_sql.py b/geopandas/io/tests/test_sql.py
--- a/geopandas/io/tests/test_sql.py
+++ b/geopandas/io/tests/test_sql.py
@@ -5,6 +5,7 @@
 see geopandas.tests.util for more information.
 """
 import os
+import warnings
 
 import pandas as pd
 
@@ -15,6 +16,13 @@
 from geopandas.tests.util import create_postgis, create_spatialite, validate_boro_df
 import pytest
 
+try:
+    from sqlalchemy import text
+except ImportError:
+    # Avoid local imports for text in all sqlalchemy tests
+    # all tests using text use engine_postgis, which ensures sqlalchemy is available
+    text = str
+
 
 @pytest.fixture
 def df_nybb():
@@ -43,8 +51,11 @@ def connection_postgis():
         )
     except OperationalError:
         pytest.skip("Cannot connect with postgresql database")
-
-    yield con
+    with warnings.catch_warnings():
+        warnings.filterwarnings(
+            "ignore", message="pandas only supports SQLAlchemy connectable.*"
+        )
+        yield con
     con.close()
 
 
@@ -73,7 +84,7 @@ def engine_postgis():
                 port=port,
             )
         )
-        con.begin()
+        con.connect()
     except Exception:
         pytest.skip("Cannot connect with postgresql database")
 
@@ -115,11 +126,15 @@ def drop_table_if_exists(conn_or_engine, table):
     sqlalchemy = pytest.importorskip("sqlalchemy")
 
     if sqlalchemy.inspect(conn_or_engine).has_table(table):
-        metadata = sqlalchemy.MetaData(conn_or_engine)
-        metadata.reflect()
+        metadata = sqlalchemy.MetaData()
+        with warnings.catch_warnings():
+            warnings.filterwarnings(
+                "ignore", message="Did not recognize type 'geometry' of column.*"
+            )
+            metadata.reflect(conn_or_engine)
         table = metadata.tables.get(table)
         if table is not None:
-            table.drop(checkfirst=True)
+            table.drop(conn_or_engine, checkfirst=True)
 
 
 @pytest.fixture
@@ -340,7 +355,7 @@ def test_write_postgis_default(self, engine_postgis, df_nybb):
         # Write to db
         write_postgis(df_nybb, con=engine, name=table, if_exists="fail")
         # Validate
-        sql = "SELECT * FROM {table};".format(table=table)
+        sql = text("SELECT * FROM {table};".format(table=table))
         df = read_postgis(sql, engine, geom_col="geometry")
         validate_boro_df(df)
 
@@ -355,7 +370,7 @@ def test_write_postgis_uppercase_tablename(self, engine_postgis, df_nybb):
         # Write to db
         write_postgis(df_nybb, con=engine, name=table, if_exists="fail")
         # Validate
-        sql = 'SELECT * FROM "{table}";'.format(table=table)
+        sql = text('SELECT * FROM "{table}";'.format(table=table))
         df = read_postgis(sql, engine, geom_col="geometry")
         validate_boro_df(df)
 
@@ -370,7 +385,7 @@ def test_write_postgis_sqlalchemy_connection(self, engine_postgis, df_nybb):
             # Write to db
             write_postgis(df_nybb, con=con, name=table, if_exists="fail")
             # Validate
-            sql = "SELECT * FROM {table};".format(table=table)
+            sql = text("SELECT * FROM {table};".format(table=table))
             df = read_postgis(sql, con, geom_col="geometry")
             validate_boro_df(df)
 
@@ -406,7 +421,7 @@ def test_write_postgis_replace_when_table_exists(self, engine_postgis, df_nybb):
         # Overwrite
         write_postgis(df_nybb, con=engine, name=table, if_exists="replace")
         # Validate
-        sql = "SELECT * FROM {table};".format(table=table)
+        sql = text("SELECT * FROM {table};".format(table=table))
         df = read_postgis(sql, engine, geom_col="geometry")
         validate_boro_df(df)
 
@@ -423,7 +438,7 @@ def test_write_postgis_append_when_table_exists(self, engine_postgis, df_nybb):
         write_postgis(df_nybb, con=engine, name=table, if_exists="replace")
         write_postgis(df_nybb, con=engine, name=table, if_exists="append")
         # Validate
-        sql = "SELECT * FROM {table};".format(table=table)
+        sql = text("SELECT * FROM {table};".format(table=table))
         df = read_postgis(sql, engine, geom_col="geometry")
         new_rows, new_cols = df.shape
 
@@ -449,13 +464,16 @@ def test_write_postgis_without_crs(self, engine_postgis, df_nybb):
         # Write to db
         df_nybb = df_nybb
         df_nybb.crs = None
-        write_postgis(df_nybb, con=engine, name=table, if_exists="replace")
+        with pytest.warns(UserWarning, match="Could not parse CRS from the GeoDataF"):
+            write_postgis(df_nybb, con=engine, name=table, if_exists="replace")
         # Validate that srid is -1
-        target_srid = engine.execute(
+        sql = text(
             "SELECT Find_SRID('{schema}', '{table}', '{geom_col}');".format(
                 schema="public", table=table, geom_col="geometry"
             )
-        ).fetchone()[0]
+        )
+        with engine.connect() as conn:
+            target_srid = conn.execute(sql).fetchone()[0]
         assert target_srid == 0, "SRID should be 0, found %s" % target_srid
 
     def test_write_postgis_with_esri_authority(self, engine_postgis, df_nybb):
@@ -471,11 +489,13 @@ def test_write_postgis_with_esri_authority(self, engine_postgis, df_nybb):
         df_nybb_esri = df_nybb.to_crs("ESRI:102003")
         write_postgis(df_nybb_esri, con=engine, name=table, if_exists="replace")
         # Validate that srid is 102003
-        target_srid = engine.execute(
+        sql = text(
             "SELECT Find_SRID('{schema}', '{table}', '{geom_col}');".format(
                 schema="public", table=table, geom_col="geometry"
             )
-        ).fetchone()[0]
+        )
+        with engine.connect() as conn:
+            target_srid = conn.execute(sql).fetchone()[0]
         assert target_srid == 102003, "SRID should be 102003, found %s" % target_srid
 
     def test_write_postgis_geometry_collection(
@@ -491,11 +511,14 @@ def test_write_postgis_geometry_collection(
         write_postgis(df_geom_collection, con=engine, name=table, if_exists="replace")
 
         # Validate geometry type
-        sql = "SELECT DISTINCT(GeometryType(geometry)) FROM {table} ORDER BY 1;".format(
-            table=table
+        sql = text(
+            "SELECT DISTINCT(GeometryType(geometry)) FROM {table} ORDER BY 1;".format(
+                table=table
+            )
         )
-        geom_type = engine.execute(sql).fetchone()[0]
-        sql = "SELECT * FROM {table};".format(table=table)
+        with engine.connect() as conn:
+            geom_type = conn.execute(sql).fetchone()[0]
+        sql = text("SELECT * FROM {table};".format(table=table))
         df = read_postgis(sql, engine, geom_col="geometry")
 
         assert geom_type.upper() == "GEOMETRYCOLLECTION"
@@ -516,10 +539,13 @@ def test_write_postgis_mixed_geometry_types(
         )
 
         # Validate geometry type
-        sql = "SELECT DISTINCT GeometryType(geometry) FROM {table} ORDER BY 1;".format(
-            table=table
+        sql = text(
+            "SELECT DISTINCT GeometryType(geometry) FROM {table} ORDER BY 1;".format(
+                table=table
+            )
         )
-        res = engine.execute(sql).fetchall()
+        with engine.connect() as conn:
+            res = conn.execute(sql).fetchall()
         assert res[0][0].upper() == "LINESTRING"
         assert res[1][0].upper() == "MULTILINESTRING"
         assert res[2][0].upper() == "POINT"
@@ -535,10 +561,13 @@ def test_write_postgis_linear_ring(self, engine_postgis, df_linear_ring):
         write_postgis(df_linear_ring, con=engine, name=table, if_exists="replace")
 
         # Validate geometry type
-        sql = "SELECT DISTINCT(GeometryType(geometry)) FROM {table} ORDER BY 1;".format(
-            table=table
+        sql = text(
+            "SELECT DISTINCT(GeometryType(geometry)) FROM {table} ORDER BY 1;".format(
+                table=table
+            )
         )
-        geom_type = engine.execute(sql).fetchone()[0]
+        with engine.connect() as conn:
+            geom_type = conn.execute(sql).fetchone()[0]
 
         assert geom_type.upper() == "LINESTRING"
 
@@ -558,15 +587,19 @@ def test_write_postgis_in_chunks(self, engine_postgis, df_mixed_single_and_multi
             chunksize=1,
         )
         # Validate row count
-        sql = "SELECT COUNT(geometry) FROM {table};".format(table=table)
-        row_cnt = engine.execute(sql).fetchone()[0]
+        sql = text("SELECT COUNT(geometry) FROM {table};".format(table=table))
+        with engine.connect() as conn:
+            row_cnt = conn.execute(sql).fetchone()[0]
         assert row_cnt == 3
 
         # Validate geometry type
-        sql = "SELECT DISTINCT GeometryType(geometry) FROM {table} ORDER BY 1;".format(
-            table=table
+        sql = text(
+            "SELECT DISTINCT GeometryType(geometry) FROM {table} ORDER BY 1;".format(
+                table=table
+            )
         )
-        res = engine.execute(sql).fetchall()
+        with engine.connect() as conn:
+            res = conn.execute(sql).fetchall()
         assert res[0][0].upper() == "LINESTRING"
         assert res[1][0].upper() == "MULTILINESTRING"
         assert res[2][0].upper() == "POINT"
@@ -579,15 +612,16 @@ def test_write_postgis_to_different_schema(self, engine_postgis, df_nybb):
 
         table = "nybb"
         schema_to_use = "test"
-        sql = "CREATE SCHEMA IF NOT EXISTS {schema};".format(schema=schema_to_use)
-        engine.execute(sql)
+        sql = text("CREATE SCHEMA IF NOT EXISTS {schema};".format(schema=schema_to_use))
+        with engine.begin() as conn:
+            conn.execute(sql)
 
         write_postgis(
             df_nybb, con=engine, name=table, if_exists="replace", schema=schema_to_use
         )
         # Validate
-        sql = "SELECT * FROM {schema}.{table};".format(
-            schema=schema_to_use, table=table
+        sql = text(
+            "SELECT * FROM {schema}.{table};".format(schema=schema_to_use, table=table)
         )
 
         df = read_postgis(sql, engine, geom_col="geometry")
@@ -603,16 +637,19 @@ def test_write_postgis_to_different_schema_when_table_exists(
 
         table = "nybb"
         schema_to_use = "test"
-        sql = "CREATE SCHEMA IF NOT EXISTS {schema};".format(schema=schema_to_use)
-        engine.execute(sql)
+        sql = text("CREATE SCHEMA IF NOT EXISTS {schema};".format(schema=schema_to_use))
+        with engine.begin() as conn:
+            conn.execute(sql)
 
         try:
             write_postgis(
                 df_nybb, con=engine, name=table, if_exists="fail", schema=schema_to_use
             )
             # Validate
-            sql = "SELECT * FROM {schema}.{table};".format(
-                schema=schema_to_use, table=table
+            sql = text(
+                "SELECT * FROM {schema}.{table};".format(
+                    schema=schema_to_use, table=table
+                )
             )
 
             df = read_postgis(sql, engine, geom_col="geometry")
@@ -627,8 +664,8 @@ def test_write_postgis_to_different_schema_when_table_exists(
             df_nybb, con=engine, name=table, if_exists="replace", schema=schema_to_use
         )
         # Validate
-        sql = "SELECT * FROM {schema}.{table};".format(
-            schema=schema_to_use, table=table
+        sql = text(
+            "SELECT * FROM {schema}.{table};".format(schema=schema_to_use, table=table)
         )
 
         df = read_postgis(sql, engine, geom_col="geometry")
@@ -645,7 +682,7 @@ def test_write_postgis_3D_geometries(self, engine_postgis, df_3D_geoms):
         write_postgis(df_3D_geoms, con=engine, name=table, if_exists="replace")
 
         # Check that all geometries have 3 dimensions
-        sql = "SELECT * FROM {table};".format(table=table)
+        sql = text("SELECT * FROM {table};".format(table=table))
         df = read_postgis(sql, engine, geom_col="geometry")
         assert list(df.geometry.has_z) == [True, True, True]
 
@@ -661,7 +698,7 @@ def test_row_order(self, engine_postgis, df_nybb):
         write_postgis(df_nybb, con=engine, name=table, if_exists="replace")
 
         # Check that the row order matches
-        sql = "SELECT * FROM {table};".format(table=table)
+        sql = text("SELECT * FROM {table};".format(table=table))
         df = read_postgis(sql, engine, geom_col="geometry")
         assert df["BoroCode"].tolist() == correct_order
 
@@ -678,7 +715,7 @@ def test_append_before_table_exists(self, engine_postgis, df_nybb):
         write_postgis(df_nybb, con=engine, name=table, if_exists="append")
 
         # Check that the row order matches
-        sql = "SELECT * FROM {table};".format(table=table)
+        sql = text("SELECT * FROM {table};".format(table=table))
         df = read_postgis(sql, engine, geom_col="geometry")
         validate_boro_df(df)