Repository: ibis-project/ibis

Issue:
bug(duckdb): newly created tables in an in-memory connection are not accessible
```
In [11]: con = ibis.duckdb.connect()

In [12]: con.raw_sql("CREATE TABLE t (x int)")
Out[12]: <sqlalchemy.engine.cursor.LegacyCursorResult at 0x7fe3bfa82620>                         

In [13]: con.list_tables()
Out[13]: []
```


Discussion:
This works if you call `.execute()` on the cursor, or if the data are in a file on disk.
Just confirming this isn't an issue on the `sqlite` backend

```
In [34]: con = ibis.sqlite.connect()

In [35]: con.con
Out[35]: Engine(sqlite:///:memory:)

In [36]: con.con.execute("CREATE TABLE t (t int32)")
Out[36]: <sqlalchemy.engine.cursor.LegacyCursorResult at 0x7fb07cdc4610>

In [37]: con.list_tables()
Out[37]: ['t']

In [38]: con = ibis.duckdb.connect()

In [39]: con.con
Out[39]: Engine(duckdb:///:memory:)

In [40]: con.con.execute("CREATE TABLE t (t int32)")
Out[40]: <sqlalchemy.engine.cursor.LegacyCursorResult at 0x7fb07ded1f30>

In [41]: con.list_tables()
Out[41]: []
```
This problem occurs without ibis, using just SQLAlchemy:

```
In [1]: import sqlalchemy as sa

In [2]: eng = sa.create_engine("duckdb:///:memory:")

In [3]: eng.execute("CREATE TABLE t (x int)")
Out[3]: <sqlalchemy.engine.cursor.LegacyCursorResult at 0x7f26134ae9b0>

In [4]: eng.execute("SHOW TABLES").fetchall()
Out[4]: []

In [5]: eng = sa.create_engine("duckdb:///test.ddb")

In [6]: eng.execute("CREATE TABLE t (x int)")
Out[6]: <sqlalchemy.engine.cursor.LegacyCursorResult at 0x7f26125ec970>

In [7]: eng.execute("SHOW TABLES").fetchall()
Out[7]: [('t',)]
```

I'll open an issue over on `duckdb-engine`.
Opened an issue here: https://github.com/Mause/duckdb_engine/issues/338

Resolving patch:
diff --git a/ibis/backends/duckdb/__init__.py b/ibis/backends/duckdb/__init__.py
--- a/ibis/backends/duckdb/__init__.py
+++ b/ibis/backends/duckdb/__init__.py
@@ -111,12 +111,13 @@ def do_connect(
         read_only
             Whether the database is read-only
         """
-        if path != ":memory:":
+        if not (in_memory := path == ":memory:"):
             path = Path(path).absolute()
         super().do_connect(
             sa.create_engine(
                 f"duckdb:///{path}",
                 connect_args={"read_only": read_only},
+                poolclass=sa.pool.SingletonThreadPool if in_memory else None,
             )
         )
         self._meta = sa.MetaData(bind=self.con)


Test patch:
diff --git a/ibis/backends/tests/test_client.py b/ibis/backends/tests/test_client.py
--- a/ibis/backends/tests/test_client.py
+++ b/ibis/backends/tests/test_client.py
@@ -343,3 +343,19 @@ def test_not_verify(alchemy_con, alchemy_backend):
 
     with pytest.warns(FutureWarning):
         assert not alchemy_backend.api.verify(expr)
+
+
+@pytest.mark.never(
+    ["postgres", "mysql"],
+    reason="postgres and mysql do not support in-memory tables",
+    raises=TypeError,
+)
+def test_in_memory(alchemy_backend):
+    con = getattr(ibis, alchemy_backend.name()).connect(path=":memory:")
+    table_name = f"t{guid()[:6]}"
+    con.raw_sql(f"CREATE TABLE {table_name} (x int)")
+    try:
+        assert table_name in con.list_tables()
+    finally:
+        con.raw_sql(f"DROP TABLE IF EXISTS {table_name}")
+        assert table_name not in con.list_tables()