Repository: ibis-project/ibis

Issue:
perf(sqlalchemy): wide table experience suffers from performance problems
### What happened?

When playing with the World Health Organization data on world heathcare cost and expenditure, I noticed that the interactive experience suffers from performance problems.

The table in question has 4224 rows and 3220 columns.

I isolated this problem to ibis with this exploration.

First, I did the simple thing and used `ibis.read_csv` with the default backend, DuckDB.

```
In [1]: from ibis.interactive import *

In [2]: %time t = ibis.read_csv("GHED_data.csv")
CPU times: user 4.25 s, sys: 2.31 s, total: 6.56 s
Wall time: 4.14 s
```

Then I looked at DuckDB's time for this using `EXPLAIN ANALYZE`:

```
D explain analyze select * from read_csv_auto('GHED_data.csv');

┌─────────────────────────────┐
│┌───────────────────────────┐│
│└───────────────────────────┘│
└─────────────────────────────┘
┌─────────────────────────────────────┐
│┌───────────────────────────────────┐│
││    Query Profiling Information    ││
│└───────────────────────────────────┘│
└─────────────────────────────────────┘
explain analyze select * from read_csv_auto('GHED_data.csv');
┌─────────────────────────────────────┐
│┌───────────────────────────────────┐│
││         Total Time: 0.758s        ││
│└───────────────────────────────────┘│
└─────────────────────────────────────┘
```

So, ibis is already taking about 5.5x longer to construct a view, apparently not great.

That's not the full story though! 🧐 🔍 

DuckDB seems to be calculating the duration differently. Running that query on the command line with `time` I get something closer to ibis:

```
❯ time duckdb <<< "select * from read_csv_auto('GHED_data.csv')" > /dev/null
```

The real mystery is when you create a native DuckDB table from the CSV file, and then start performing basic operations.

Here's ibis, creating a table from the CSV and then counting the rows of that table:

```
In [7]: from ibis.interactive import *

In [8]: ddb = ibis.duckdb.connect()

In [9]: view = ddb.read_csv("GHED_data.csv")

In [10]: table = ddb.create_table("ghed", view, overwrite=True)
100% ▕████████████████████████████████████████████████████████████▏

In [11]: %time table.count().execute()
CPU times: user 1.69 s, sys: 99.3 ms, total: 1.79 s
Wall time: 1.71 s
Out[11]: 4224
```

Compare that to a raw SQL string doing the same thing:

```
In [12]: %time ddb.con.connect().exec_driver_sql('select count(*) from ghed').scalar()
CPU times: user 2.56 ms, sys: 258 µs, total: 2.82 ms
Wall time: 2.49 ms
Out[12]: 4224
```

Just rendering the SQL string is extremely slow as well:

```
In [14]: %time ibis.show_sql(table.count())
SELECT
  COUNT(*) AS "CountStar(ghed)"
FROM ghed AS t0
CPU times: user 1.64 s, sys: 45 ms, total: 1.68 s
Wall time: 1.63 s
```

My hunch is that this is due to extracting type information for all 3k columns in the `self.table` at the end of `read_csv`, but this warrants further investigation.

Data: [GHED_data.csv](https://github.com/ibis-project/ibis/files/12313409/GHED_data.csv)


### What version of ibis are you using?

`master`

### What backend(s) are you using, if any?

DuckDB

### Relevant log output

_No response_

### Code of Conduct

- [X] I agree to follow this project's Code of Conduct


Discussion:

Resolving patch:
diff --git a/ibis/backends/base/sql/alchemy/__init__.py b/ibis/backends/base/sql/alchemy/__init__.py
--- a/ibis/backends/base/sql/alchemy/__init__.py
+++ b/ibis/backends/base/sql/alchemy/__init__.py
@@ -700,13 +700,12 @@ def insert(
 
             from_table_expr = obj
 
-            with self.begin() as bind:
-                if from_table_expr is not None:
-                    compiled = from_table_expr.compile()
-                    columns = [
-                        self.con.dialect.normalize_name(c)
-                        for c in from_table_expr.columns
-                    ]
+            if from_table_expr is not None:
+                compiled = from_table_expr.compile()
+                columns = [
+                    self.con.dialect.normalize_name(c) for c in from_table_expr.columns
+                ]
+                with self.begin() as bind:
                     bind.execute(to_table.insert().from_select(columns, compiled))
         elif isinstance(obj, (list, dict)):
             to_table = self._get_sqla_table(table_name, schema=database)
diff --git a/ibis/backends/duckdb/__init__.py b/ibis/backends/duckdb/__init__.py
--- a/ibis/backends/duckdb/__init__.py
+++ b/ibis/backends/duckdb/__init__.py
@@ -443,6 +443,85 @@ def read_csv(
             con.exec_driver_sql(view)
         return self.table(table_name)
 
+    def _get_sqla_table(
+        self,
+        name: str,
+        schema: str | None = None,
+        database: str | None = None,
+        **_: Any,
+    ) -> sa.Table:
+        if schema is None:
+            schema = self.current_schema
+        *db, schema = schema.split(".")
+        db = "".join(db) or database
+        ident = ".".join(
+            map(
+                self._quote,
+                filter(None, (db if db != self.current_database else None, schema)),
+            )
+        )
+
+        s = sa.table(
+            "columns",
+            sa.column("table_catalog", sa.TEXT()),
+            sa.column("table_schema", sa.TEXT()),
+            sa.column("table_name", sa.TEXT()),
+            sa.column("column_name", sa.TEXT()),
+            sa.column("data_type", sa.TEXT()),
+            sa.column("is_nullable", sa.TEXT()),
+            sa.column("ordinal_position", sa.INTEGER()),
+            schema="information_schema",
+        )
+
+        where = s.c.table_name == name
+
+        if db:
+            where &= s.c.table_catalog == db
+
+        if schema:
+            where &= s.c.table_schema == schema
+
+        query = (
+            sa.select(
+                s.c.column_name,
+                s.c.data_type,
+                (s.c.is_nullable == "YES").label("nullable"),
+            )
+            .where(where)
+            .order_by(sa.asc(s.c.ordinal_position))
+        )
+
+        with self.begin() as con:
+            # fetch metadata with pyarrow, it's much faster for wide tables
+            meta = con.execute(query).cursor.fetch_arrow_table()
+
+        if not meta:
+            raise sa.exc.NoSuchTableError(name)
+
+        names = meta["column_name"].to_pylist()
+        types = meta["data_type"].to_pylist()
+        nullables = meta["nullable"].to_pylist()
+
+        ibis_schema = sch.Schema(
+            {
+                name: parse(typ).copy(nullable=nullable)
+                for name, typ, nullable in zip(names, types, nullables)
+            }
+        )
+        columns = self._columns_from_schema(name, ibis_schema)
+        return sa.table(name, *columns, schema=ident)
+
+    def drop_table(
+        self, name: str, database: str | None = None, force: bool = False
+    ) -> None:
+        name = self._quote(name)
+        # TODO: handle database quoting
+        if database is not None:
+            name = f"{database}.{name}"
+        drop_stmt = "DROP TABLE" + (" IF EXISTS" * force) + f" {name}"
+        with self.begin() as con:
+            con.exec_driver_sql(drop_stmt)
+
     def read_parquet(
         self,
         source_list: str | Iterable[str],
@@ -992,17 +1071,6 @@ def _register(name, table):
             except duckdb.NotImplementedException:
                 _register(name, data.to_pyarrow(schema))
 
-    def _get_sqla_table(
-        self, name: str, schema: str | None = None, **kwargs: Any
-    ) -> sa.Table:
-        with warnings.catch_warnings():
-            # We don't rely on index reflection, ignore this warning
-            warnings.filterwarnings(
-                "ignore",
-                message="duckdb-engine doesn't yet support reflection on indices",
-            )
-            return super()._get_sqla_table(name, schema, **kwargs)
-
     def _get_temp_view_definition(
         self, name: str, definition: sa.sql.compiler.Compiled
     ) -> str:


Test patch:
diff --git a/ibis/backends/duckdb/tests/test_datatypes.py b/ibis/backends/duckdb/tests/test_datatypes.py
--- a/ibis/backends/duckdb/tests/test_datatypes.py
+++ b/ibis/backends/duckdb/tests/test_datatypes.py
@@ -95,17 +95,17 @@ def test_null_dtype():
 def test_parse_quoted_struct_field():
     import ibis.backends.duckdb.datatypes as ddt
 
-    assert ddt.parse('STRUCT("a" INT, "a b c" INT)') == dt.Struct(
+    assert ddt.parse('STRUCT("a" INTEGER, "a b c" INTEGER)') == dt.Struct(
         {"a": dt.int32, "a b c": dt.int32}
     )
 
 
 def test_generate_quoted_struct():
     typ = sat.StructType(
-        {"in come": sa.TEXT(), "my count": sa.BIGINT(), "thing": sa.INT()}
+        {"in come": sa.VARCHAR(), "my count": sa.BIGINT(), "thing": sa.INTEGER()}
     )
     result = typ.compile(dialect=duckdb_engine.Dialect())
-    expected = 'STRUCT("in come" TEXT, "my count" BIGINT, thing INTEGER)'
+    expected = 'STRUCT("in come" VARCHAR, "my count" BIGINT, thing INTEGER)'
     assert result == expected
 
 
diff --git a/ibis/tests/benchmarks/test_benchmarks.py b/ibis/tests/benchmarks/test_benchmarks.py
--- a/ibis/tests/benchmarks/test_benchmarks.py
+++ b/ibis/tests/benchmarks/test_benchmarks.py
@@ -743,3 +743,13 @@ def test_snowflake_medium_sized_to_pandas(benchmark):
     )
 
     benchmark.pedantic(lineitem.to_pandas, rounds=5, iterations=1, warmup_rounds=1)
+
+
+def test_parse_many_duckdb_types(benchmark):
+    parse = pytest.importorskip("ibis.backends.duckdb.datatypes").parse
+
+    def parse_many(types):
+        list(map(parse, types))
+
+    types = ["VARCHAR", "INTEGER", "DOUBLE", "BIGINT"] * 1000
+    benchmark(parse_many, types)