Repository: ibis-project/ibis

Issue:
feat: DuckDB ASOF join in Ibis slower than native execution
### Is your feature request related to a problem?

I noticed that executing an ASOF join using the DuckDB backend is much slower than executing duckDB natively.

Ran benchmarks comparing Pandas, DuckDB, and DuckDB via Ibis from 10k to 100 million rows and 2 / 5 cols.
- DuckDB via Ibis seems to be quite a bit slower than executing the ASOF join via DuckDB natively

[Code for benchmarking](https://gist.github.com/tonysun9/dc395b08144f30dcfbf8a1f4827bcbf2)

[Results Spreadsheet](https://docs.google.com/spreadsheets/d/16q-Ld103Yi9iAxH4nA7Lz6SSzL1GgLEiCv1MjMbSZ54/edit?usp=sharing)

### Describe the solution you'd like

I would like to be able to execute an ASOF join in DuckDB quickly via Ibis. My understanding is that someone is actively implementing the merge ASOF join in DuckDB via Ibis.

Any insights into why executing `.sql()` takes additional time would also be appreciated.

### What version of ibis are you running?

6.0.0

### What backend(s) are you using, if any?

DuckDB

### Code of Conduct

- [X] I agree to follow this project's Code of Conduct


Discussion:
Hi, thanks for opening this. Can you please comment here with the example code? We don't currently have access to the example code linked in the spreadsheet you linked, and it's nice to keep everything on GitHub when possible. Thanks!
related issue tracking implementation of asof join for the DuckDB backend: https://github.com/ibis-project/ibis/issues/6487
> Can you please comment here with the example code? 

Thanks for the suggestion. Updated the original post with a gist for the code.
Thanks. I can confirm that there's a performance impact here. Still digging into the issue, but here's my cleaned up reproducible example:

<details>
<summary>Example Code</summary>

```python
from contextlib import contextmanager
import timeit

import pandas as pd
import numpy as np
import duckdb

import ibis


@contextmanager
def timer(name):
    start = timeit.default_timer()
    yield
    end = timeit.default_timer()
    elapsed = end - start

    print(f"'{name}' executed in {elapsed:.2f} seconds.")


def generate_data(
    prices_size: int, transactions_size: int, add_columns: bool = False
) -> (pd.DataFrame, pd.DataFrame):
    # Generate synthetic time data for prices
    prices_times = pd.date_range(start="09:00", periods=prices_size, freq="T")

    # Define potential transaction times (every 00 or 30 seconds)
    transaction_times = pd.date_range(
        start="09:00", periods=transactions_size * 2, freq="30S"
    )

    # Sample from the potential transaction times for transactions
    times_transactions = np.random.choice(
        transaction_times, size=transactions_size, replace=False
    )

    # Generate synthetic price data
    prices = (
        102.5 + np.sin(np.linspace(0, prices_size, prices_size) / 100) * 2.0
    )  # Sin wave around 102.5 with amplitude of 2

    # Generate synthetic transaction data
    shares = np.random.randint(5, 20, size=transactions_size)

    # Define DataFrames
    prices_df_data = {"time": prices_times, "price": prices}
    transactions_df_data = {"time": times_transactions, "shares": shares}

    if add_columns:
        # Add additional columns for prices_df
        prices_df_data["volume"] = np.random.randint(100, 1000, size=prices_size)
        prices_df_data["high"] = prices + np.random.rand(prices_size) * 0.5
        prices_df_data["low"] = prices - np.random.rand(prices_size) * 0.5

        # Add additional columns for transactions_df
        transactions_df_data["transaction_type"] = np.random.choice(
            ["BUY", "SELL"], size=transactions_size
        )
        transactions_df_data["order_id"] = np.arange(1, transactions_size + 1)
        brokers = ["BrokerA", "BrokerB", "BrokerC", "BrokerD"]
        transactions_df_data["broker"] = np.random.choice(
            brokers, size=transactions_size
        )

    # Create DataFrames
    prices_df = pd.DataFrame(prices_df_data)
    transactions_df = pd.DataFrame(transactions_df_data)

    # Sort transactions_df by time
    transactions_df = transactions_df.sort_values(by="time")

    return prices_df, transactions_df


def asof_pandas(transactions_df: pd.DataFrame, prices_df: pd.DataFrame) -> None:
    with timer("pandas"):
        return pd.merge_asof(transactions_df, prices_df, on="time")


def asof_duckdb(transactions_df, prices_df) -> None:
    con = duckdb.connect()
    con.sql("CREATE TABLE transactions AS SELECT * FROM transactions_df")
    con.sql("CREATE TABLE prices AS SELECT * FROM prices_df")

    sql = """
    SELECT
        t.time, t.shares, p.price
    FROM
        transactions t ASOF LEFT JOIN prices p
    ON
        t.time >= p.time
    """

    with timer("duckdb"):
        return con.sql(sql).to_arrow_table()


def asof_ibis_duckdb(transactions_df, prices_df) -> None:
    con = ibis.duckdb.connect()
    con.raw_sql("CREATE TABLE transactions AS SELECT * FROM transactions_df")
    con.raw_sql("CREATE TABLE prices AS SELECT * FROM prices_df")

    sql = """
    SELECT
        t.time, t.shares, p.price
    FROM
        transactions t ASOF LEFT JOIN prices p
    ON
        t.time >= p.time
    """

    with timer("ibis compile time"):
        expr = con.sql(sql)
        ibis.to_sql(expr)

    with timer("ibis execute time"):
        return expr.to_pyarrow()


prices_df, transactions_df = generate_data(5_000_000, 1_000_000, False)

asof_pandas(transactions_df, prices_df)
asof_duckdb(transactions_df, prices_df)
asof_ibis_duckdb(transactions_df, prices_df)
```

</details>

Timing results:

```python
$ python bench.py 
'pandas' executed in 0.06 seconds.
'duckdb' executed in 1.07 seconds.
'ibis compile time' executed in 0.01 seconds.
'ibis execute time' executed in 1.73 seconds.
```

I confirmed that the results for each execution engine are identical. For data sizes I've tested duckdb is always measurably slower than pandas (though it was mentioned offline that for larger inputs duckdb may be faster). Either way, ibis has surprisingly non-negligible overhead here isolated to execution only (not expression/compile time overhead). That overhead grows with data size; it's not a fixed cost. Still debugging the issue, but thought I'd post an update.
Ok, I have isolated the performance difference between `ibis + duckdb` and `duckdb` alone.

In short, this is due to a performance difference between `con.sql(...).to_arrow_table()` and `con.execute(...).fetch_arrow_table()`. Here's a reproducer that doesn't use ibis:

<details>
<summary>Benchmark Code</summary>

```python
from contextlib import contextmanager
import timeit

import pandas as pd
import numpy as np
import duckdb


@contextmanager
def timer(label):
    start = timeit.default_timer()
    yield
    end = timeit.default_timer()
    elapsed = end - start

    print(f"{label}: {elapsed:.2f} seconds.")


N_PRICES = 5_000_000
N_TRANSACTIONS = 2_000_000


prices = pd.DataFrame(
    {
        "time": pd.date_range(start="09:00", periods=N_PRICES, freq="T"),
        "price": np.linspace(0, N_PRICES, N_PRICES)
    }
)
transactions = pd.DataFrame(
    {
        "time": np.random.choice(
            pd.date_range(
                start="09:00", periods=N_TRANSACTIONS * 2, freq="30S"
            ),
            size=N_TRANSACTIONS,
            replace=False,
        ),
        "shares": np.random.randint(5, 20, size=N_TRANSACTIONS)
    }
)

con = duckdb.connect()
con.execute("CREATE TABLE transactions AS SELECT * FROM transactions")
con.execute("CREATE TABLE prices AS SELECT * FROM prices")

sql = """
SELECT
    t.time, t.shares, p.price
FROM
    transactions t ASOF LEFT JOIN prices p
ON
    t.time >= p.time
"""

with timer("Using sql and to_arrow_table"):
    con.sql(sql).to_arrow_table()


with timer("Using execute and fetch_arrow_table"):
    con.execute(sql).fetch_arrow_table()
```

</details>

Results:

```
$ python bench.py
Using sql and to_arrow_table: 3.26 seconds.
Using execute and fetch_arrow_table: 10.61 seconds.
```

I've opened an upstream discussion in the duckdb repo to investigate this: https://github.com/duckdb/duckdb/discussions/8582.
Thanks for looking into this @jcrist, appreciate the insight.

An unrelated Python question: I noticed you used a Python context manager rather than a decorator for timing the ASOF join. I assume the intent is so that you can pick and choose which lines of code to time rather than needing to time a whole function?
```python
@contextmanager
def timer(name):
    start = timeit.default_timer()
    yield
    end = timeit.default_timer()
    elapsed = end - start

    print(f"'{name}' executed in {elapsed:.2f} seconds.")
```
> I assume the intent is so that you can pick and choose which lines of code to time rather than needing to time a whole function?

Yep, that's correct. The first thing I did with your example was rearrange things to ensure the benchmarked operations were equivalent. Using a contextmanager to only time critical sections made this easier.

Resolving patch:
diff --git a/ibis/backends/duckdb/__init__.py b/ibis/backends/duckdb/__init__.py
--- a/ibis/backends/duckdb/__init__.py
+++ b/ibis/backends/duckdb/__init__.py
@@ -862,11 +862,20 @@ def to_pyarrow(
     ) -> pa.Table:
         self._run_pre_execute_hooks(expr)
         query_ast = self.compiler.to_ast_ensure_limit(expr, limit, params=params)
-        sql = query_ast.compile()
+
+        # We use `.sql` instead of `.execute` below for performance - in
+        # certain cases duckdb query -> arrow table can be significantly faster
+        # in this configuration. Currently `.sql` doesn't support parametrized
+        # queries, so we need to compile with literal_binds for now.
+        sql = str(
+            query_ast.compile().compile(
+                dialect=self.con.dialect, compile_kwargs={"literal_binds": True}
+            )
+        )
 
         with self.begin() as con:
-            cursor = con.execute(sql)
-            table = cursor.cursor.fetch_arrow_table()
+            cursor = con.connection.cursor()
+            table = cursor.sql(sql).to_arrow_table()
 
         return expr.__pyarrow_result__(table)
 


Test patch:
diff --git a/ibis/backends/tests/test_export.py b/ibis/backends/tests/test_export.py
--- a/ibis/backends/tests/test_export.py
+++ b/ibis/backends/tests/test_export.py
@@ -323,6 +323,7 @@ def test_table_to_csv(tmp_path, backend, awards_players):
             id="decimal256",
             marks=[
                 pytest.mark.notyet(["impala"], reason="precision not supported"),
+                pytest.mark.notyet(["duckdb"], reason="precision is out of range"),
                 pytest.mark.notyet(
                     ["druid", "duckdb", "snowflake", "trino"],
                     raises=sa.exc.ProgrammingError,