Repository: ibis-project/ibis

Issue:
bug: listing tables with MotherDuck seems incorrect
### What happened?

When I connect to a DuckDB database in MotherDuck -- notice there can be many DBs -- and list tables, I get the list of tables for all the databases I have, instead of just the specific database.

Through the DuckDB CLI, connecting to the `imdb` database:

```
(venv) cody@dkdcvoda ibis % duckdb md:imdb
v0.8.1 6536a77232
Enter ".help" for usage hints.
D show tables;
┌───────────────────────┐
│         name          │
│        varchar        │
├───────────────────────┤
│ imdb_name_basics      │
│ imdb_title_akas       │
│ imdb_title_basics     │
│ imdb_title_crew       │
│ imdb_title_episode    │
│ imdb_title_principals │
│ imdb_title_ratings    │
│ penguins              │
└───────────────────────┘
D
```

then doing the same through Ibis:

```
(venv) cody@dkdcvoda ibis % ipy
Python 3.11.5 (main, Sep 14 2023, 13:17:51) [Clang 14.0.3 (clang-1403.0.22.14.1)]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.14.0 -- An enhanced Interactive Python. Type '?' for help.

[ins] In [1]: import ibis
con = i
[ins] In [2]: con = ibis.connect("duckdb://md:imdb")

[ins] In [3]: con.list_tables()
Out[3]:
['backends',
 'backends',
 'commits',
 'commits',
 'downloads',
 'downloads',
 'forks',
 'forks',
 'imdb_name_basics',
 'imdb_title_akas',
 'imdb_title_basics',
 'imdb_title_crew',
 'imdb_title_episode',
 'imdb_title_principals',
 'imdb_title_ratings',
 'issues',
 'issues',
 'penguins',
 'pulls',
 'pulls',
 'stars',
 'stars',
 'watchers',
 'watchers']
```

it seems to be, in addition to the IMDB database, listing out tables from the `prod_metrics` and `staging_metrics` databases (hence hte duplication):

```
(venv) cody@dkdcvoda ibis % duckdb md:imdb
v0.8.1 6536a77232
Enter ".help" for usage hints.
D show databases;
┌─────────────────┐
│  database_name  │
│     varchar     │
├─────────────────┤
│ imdb            │
│ my_db           │
│ prod_metrics    │
│ staging_metrics │
└─────────────────┘
D
```

### What version of ibis are you using?

master

### What backend(s) are you using, if any?

DuckDB, version 0.8.1 (MotherDuck doesn't support 0.9.0 yet)

### Relevant log output

```sh
see above
```


### Code of Conduct

- [X] I agree to follow this project's Code of Conduct


Discussion:
if you do try to query one of the tables, end up w/ this:

```python
[ins] In [1]: import ibis

[ins] In [2]: con = ibis.connect("duckdb://md:imdb")

[ins] In [3]: con.list_tables()
Out[3]:
['backends',
 'backends',
 'commits',
 'commits',
 'downloads',
 'downloads',
 'forks',
 'forks',
 'imdb_name_basics',
 'imdb_title_akas',
 'imdb_title_basics',
 'imdb_title_crew',
 'imdb_title_episode',
 'imdb_title_principals',
 'imdb_title_ratings',
 'issues',
 'issues',
 'penguins',
 'pulls',
 'pulls',
 'stars',
 'stars',
 'watchers',
 'watchers']

[ins] In [4]: t = con.table("issues")

[ins] In [5]: t
Out[5]:
DatabaseTable: issues
  title        string
  number       int64
  id           string
  url          string
  labels       edges: array<struct<node: struct<name: string>>>
  state        string
  state_reason string
  closed       boolean
  body         string
  comments     edges: array<struct<node: struct<body: string, author: struct<login: string>>>>
  created_at   timestamp
  updated_at   timestamp
  closed_at    timestamp
  login        string
  is_closed    boolean
  total_issues int64

[ins] In [6]: t.execute()
---------------------------------------------------------------------------
CatalogException                          Traceback (most recent call last)
File ~/repos/ibis/venv/lib/python3.11/site-packages/sqlalchemy/engine/base.py:1910, in Connection._execute_context(self, dialect, constructor, statement, parameters, execution_options, *args, **kw)
   1909     if not evt_handled:
-> 1910         self.dialect.do_execute(
   1911             cursor, statement, parameters, context
   1912         )
   1914 if self._has_events or self.engine._has_events:

File ~/repos/ibis/venv/lib/python3.11/site-packages/sqlalchemy/engine/default.py:736, in DefaultDialect.do_execute(self, cursor, statement, parameters, context)
    735 def do_execute(self, cursor, statement, parameters, context=None):
--> 736     cursor.execute(statement, parameters)

File ~/repos/ibis/venv/lib/python3.11/site-packages/duckdb_engine/__init__.py:144, in ConnectionWrapper.execute(self, statement, parameters, context)
    143     else:
--> 144         self.__c.execute(statement, parameters)
    145 except RuntimeError as e:

CatalogException: Catalog Error: Table with name issues does not exist!
Did you mean "staging_metrics.issues"?
LINE 2: FROM main.issues AS t0
             ^

The above exception was the direct cause of the following exception:

ProgrammingError                          Traceback (most recent call last)
Cell In[6], line 1
----> 1 t.execute()

File ~/repos/ibis/ibis/expr/types/core.py:322, in Expr.execute(self, limit, timecontext, params, **kwargs)
    295 def execute(
    296     self,
    297     limit: int | str | None = "default",
   (...)
    300     **kwargs: Any,
    301 ):
    302     """Execute an expression against its backend if one exists.
    303
    304     Parameters
   (...)
    320         Keyword arguments
    321     """
--> 322     return self._find_backend(use_default=True).execute(
    323         self, limit=limit, timecontext=timecontext, params=params, **kwargs
    324     )

File ~/repos/ibis/ibis/backends/base/sql/__init__.py:334, in BaseSQLBackend.execute(self, expr, params, limit, **kwargs)
    330 self._log(sql)
    332 schema = expr.as_table().schema()
--> 334 with self._safe_raw_sql(sql, **kwargs) as cursor:
    335     result = self.fetch_from_cursor(cursor, schema)
    337 return expr.__pandas_result__(result)

File ~/.local/share/rtx/installs/python/latest/lib/python3.11/contextlib.py:137, in _GeneratorContextManager.__enter__(self)
    135 del self.args, self.kwds, self.func
    136 try:
--> 137     return next(self.gen)
    138 except StopIteration:
    139     raise RuntimeError("generator didn't yield") from None

File ~/repos/ibis/ibis/backends/base/sql/alchemy/__init__.py:194, in BaseAlchemyBackend._safe_raw_sql(self, *args, **kwargs)
    191 @contextlib.contextmanager
    192 def _safe_raw_sql(self, *args, **kwargs):
    193     with self.begin() as con:
--> 194         yield con.execute(*args, **kwargs)

File ~/repos/ibis/venv/lib/python3.11/site-packages/sqlalchemy/engine/base.py:1385, in Connection.execute(self, statement, *multiparams, **params)
   1381     util.raise_(
   1382         exc.ObjectNotExecutableError(statement), replace_context=err
   1383     )
   1384 else:
-> 1385     return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)

File ~/repos/ibis/venv/lib/python3.11/site-packages/sqlalchemy/sql/elements.py:334, in ClauseElement._execute_on_connection(self, connection, multiparams, params, execution_options, _force)
    330 def _execute_on_connection(
    331     self, connection, multiparams, params, execution_options, _force=False
    332 ):
    333     if _force or self.supports_execution:
--> 334         return connection._execute_clauseelement(
    335             self, multiparams, params, execution_options
    336         )
    337     else:
    338         raise exc.ObjectNotExecutableError(self)

File ~/repos/ibis/venv/lib/python3.11/site-packages/sqlalchemy/engine/base.py:1577, in Connection._execute_clauseelement(self, elem, multiparams, params, execution_options)
   1565 compiled_cache = execution_options.get(
   1566     "compiled_cache", self.engine._compiled_cache
   1567 )
   1569 compiled_sql, extracted_params, cache_hit = elem._compile_w_cache(
   1570     dialect=dialect,
   1571     compiled_cache=compiled_cache,
   (...)
   1575     linting=self.dialect.compiler_linting | compiler.WARN_LINTING,
   1576 )
-> 1577 ret = self._execute_context(
   1578     dialect,
   1579     dialect.execution_ctx_cls._init_compiled,
   1580     compiled_sql,
   1581     distilled_params,
   1582     execution_options,
   1583     compiled_sql,
   1584     distilled_params,
   1585     elem,
   1586     extracted_params,
   1587     cache_hit=cache_hit,
   1588 )
   1589 if has_events:
   1590     self.dispatch.after_execute(
   1591         self,
   1592         elem,
   (...)
   1596         ret,
   1597     )

File ~/repos/ibis/venv/lib/python3.11/site-packages/sqlalchemy/engine/base.py:1953, in Connection._execute_context(self, dialect, constructor, statement, parameters, execution_options, *args, **kw)
   1950             branched.close()
   1952 except BaseException as e:
-> 1953     self._handle_dbapi_exception(
   1954         e, statement, parameters, cursor, context
   1955     )
   1957 return result

File ~/repos/ibis/venv/lib/python3.11/site-packages/sqlalchemy/engine/base.py:2134, in Connection._handle_dbapi_exception(self, e, statement, parameters, cursor, context)
   2132     util.raise_(newraise, with_traceback=exc_info[2], from_=e)
   2133 elif should_wrap:
-> 2134     util.raise_(
   2135         sqlalchemy_exception, with_traceback=exc_info[2], from_=e
   2136     )
   2137 else:
   2138     util.raise_(exc_info[1], with_traceback=exc_info[2])

File ~/repos/ibis/venv/lib/python3.11/site-packages/sqlalchemy/util/compat.py:211, in raise_(***failed resolving arguments***)
    208     exception.__cause__ = replace_context
    210 try:
--> 211     raise exception
    212 finally:
    213     # credit to
    214     # https://cosmicpercolator.com/2016/01/13/exception-leaks-in-python-2-and-3/
    215     # as the __traceback__ object creates a cycle
    216     del exception, replace_context, from_, with_traceback

File ~/repos/ibis/venv/lib/python3.11/site-packages/sqlalchemy/engine/base.py:1910, in Connection._execute_context(self, dialect, constructor, statement, parameters, execution_options, *args, **kw)
   1908                 break
   1909     if not evt_handled:
-> 1910         self.dialect.do_execute(
   1911             cursor, statement, parameters, context
   1912         )
   1914 if self._has_events or self.engine._has_events:
   1915     self.dispatch.after_cursor_execute(
   1916         self,
   1917         cursor,
   (...)
   1921         context.executemany,
   1922     )

File ~/repos/ibis/venv/lib/python3.11/site-packages/sqlalchemy/engine/default.py:736, in DefaultDialect.do_execute(self, cursor, statement, parameters, context)
    735 def do_execute(self, cursor, statement, parameters, context=None):
--> 736     cursor.execute(statement, parameters)

File ~/repos/ibis/venv/lib/python3.11/site-packages/duckdb_engine/__init__.py:144, in ConnectionWrapper.execute(self, statement, parameters, context)
    142         self.__c.execute(statement)
    143     else:
--> 144         self.__c.execute(statement, parameters)
    145 except RuntimeError as e:
    146     if e.args[0].startswith("Not implemented Error"):

ProgrammingError: (duckdb.CatalogException) Catalog Error: Table with name issues does not exist!
Did you mean "staging_metrics.issues"?
LINE 2: FROM main.issues AS t0
             ^
[SQL: SELECT t0.title, t0.number, t0.id, t0.url, t0.labels, t0.state, t0.state_reason, t0.closed, t0.body, t0.comments, t0.created_at, t0.updated_at, t0.closed_at, t0.login, t0.is_closed, t0.total_issues
FROM main.issues AS t0]
(Background on this error at: https://sqlalche.me/e/14/f405)
```
I haven't tried using motherduck yet, but I can't reproduce this locally:

```
In [9]: con = ibis.connect("duckdb://")

In [10]: con.raw_sql("create schema schema1")
Out[10]: <sqlalchemy.engine.cursor.LegacyCursorResult at 0x7ff7805a9090>

In [11]: con.raw_sql("create schema schema2")
Out[11]: <sqlalchemy.engine.cursor.LegacyCursorResult at 0x7ff769fd9cf0>

In [12]: con.raw_sql("create table schema1.t as select 1 a, 'b' b")
Out[12]: <sqlalchemy.engine.cursor.LegacyCursorResult at 0x7ff75885a380>

In [13]: con.raw_sql("create table schema2.t as select 1 a, 'b' b")
Out[13]: <sqlalchemy.engine.cursor.LegacyCursorResult at 0x7ff769fc72b0>

In [14]: con.list_tables()
Out[14]: []

In [16]: con.list_tables(database="schema1")
Out[16]: ['t']

In [17]: con.list_tables(database="schema2")
Out[17]: ['t']
I suspect it's specific to MotherDuck? let me know if anything I can do to help reproduce, you should just need multiple MotherDuck databases w/ some tables in them

Resolving patch:
diff --git a/ibis/backends/duckdb/__init__.py b/ibis/backends/duckdb/__init__.py
--- a/ibis/backends/duckdb/__init__.py
+++ b/ibis/backends/duckdb/__init__.py
@@ -15,6 +15,7 @@
 import duckdb
 import pyarrow as pa
 import sqlalchemy as sa
+import sqlglot as sg
 import toolz
 
 import ibis.common.exceptions as exc
@@ -25,6 +26,7 @@
 from ibis import util
 from ibis.backends.base import CanCreateSchema
 from ibis.backends.base.sql.alchemy import BaseAlchemyBackend
+from ibis.backends.base.sqlglot import C, F, lit
 from ibis.backends.duckdb.compiler import DuckDBSQLCompiler
 from ibis.backends.duckdb.datatypes import DuckDBType
 from ibis.expr.operations.relations import PandasDataFrameProxy
@@ -684,14 +686,69 @@ def read_delta(
             delta_table.to_pyarrow_dataset(), table_name=table_name
         )
 
-    def list_tables(self, like=None, database=None):
-        tables = self.inspector.get_table_names(schema=database)
-        views = self.inspector.get_view_names(schema=database)
-        # workaround for GH5503
-        temp_views = self.inspector.get_view_names(
-            schema="temp" if database is None else database
+    def list_tables(
+        self,
+        like: str | None = None,
+        database: str | None = None,
+        schema: str | None = None,
+    ) -> list[str]:
+        """List tables and views.
+
+        Parameters
+        ----------
+        like
+            Regex to filter by table/view name.
+        database
+            Database name. If not passed, uses the current database. Only
+            supported with MotherDuck.
+        schema
+            Schema name. If not passed, uses the current schema.
+
+        Returns
+        -------
+        list[str]
+            List of table and view names.
+
+        Examples
+        --------
+        >>> import ibis
+        >>> con = ibis.duckdb.connect()
+        >>> foo = con.create_table("foo", schema=ibis.schema(dict(a="int")))
+        >>> con.list_tables()
+        ['foo']
+        >>> bar = con.create_view("bar", foo)
+        >>> con.list_tables()
+        ['bar', 'foo']
+        >>> con.create_schema("my_schema")
+        >>> con.list_tables(schema="my_schema")
+        []
+        >>> with con.begin() as c:
+        ...     c.exec_driver_sql(
+        ...         "CREATE TABLE my_schema.baz (a INTEGER)"
+        ...     )  # doctest: +ELLIPSIS
+        ...
+        <...>
+        >>> con.list_tables(schema="my_schema")
+        ['baz']
+        """
+        database = F.current_database() if database is None else lit(database)
+        schema = F.current_schema() if schema is None else lit(schema)
+
+        sql = (
+            sg.select(C.table_name)
+            .from_(sg.table("tables", db="information_schema"))
+            .distinct()
+            .where(
+                C.table_catalog.eq(database).or_(C.table_catalog.eq(lit("temp"))),
+                C.table_schema.eq(schema),
+            )
+            .sql(self.name, pretty=True)
         )
-        return self._filter_with_like(tables + views + temp_views, like)
+
+        with self.begin() as con:
+            out = con.exec_driver_sql(sql).cursor.fetch_arrow_table()
+
+        return self._filter_with_like(out["table_name"].to_pylist(), like)
 
     def read_postgres(
         self, uri: str, table_name: str | None = None, schema: str = "public"


Test patch: