Repository: ibis-project/ibis

Issue:
feat: __dataframe__ protocol support
### Is your feature request related to a problem?

scikit-learn and other libraries (e.g. VegaFusion) are working to adding add for the standard [Python dataframe interchange protocol](https://data-apis.org/dataframe-protocol/latest/index.html) in order to be able to fit consume data from any dataframe library without having to first convert to pandas, see e.g.:

- https://github.com/scikit-learn/scikit-learn/issues/25896

### Describe the solution you'd like

Expose the standard `__dataframe__` method on any memory backed `Table` (e.g. those created by `ibis.memtable`).

This feature could start with the DuckDB backend and be latter generalized to other backends with in-memory data (e.g. pandas, polars, ...).

Note: the dataframe protocol was mentioned in this discussion:

- https://github.com/ibis-project/ibis/discussions/4542

But I figured it would be worth opening a dedicated feature request specifically for this need.

### What version of ibis are you running?

5.1.0 (master branch)

### What backend(s) are you using, if any?

DuckDB

### Code of Conduct

- [X] I agree to follow this project's Code of Conduct


Discussion:
@ogrisel Thanks for the issue!

Is there any reason not to support the dataframe protocol on every table expression?

It's not clear based on the linked issue or this one how implementing `__dataframe__` only for in-memory tables would be useful, since you wouldn't be able to rely on the protocol being implemented for an arbitrary expression, resulting in all sorts of annoying checks everywhere in downstream code.

That said, is there any expectation about the performance characteristics of the dataframe interchange protocol?

Would it be a problem for our implementation of the protocol to trigger arbitrary computation? A user would have to trigger compute regardless to get a DataFrame, pyarrow Table etc out.

There are also a bunch of things in the protocol that simply do not make sense for ibis, like buffers and memory layout.

I'd suggest that we implement `__dataframe__` like this to start:

```python
class Table(Expr):
	...  # existing APIs

	def __dataframe__(self):
		return self.to_pyarrow().__dataframe__()
```

and I guess hope that no one is calling `expr.__dataframe__()` in a tight loop!
Indeed you are right. I assumed restricting to in memory tables would avoid accidental memory exhaustion but this is not guaranteed for all expressions (some expressions might yield bigger results than the source data).

A generic implementation based on pyarrow or pandas is probably the way to go, depending on which library has the most complete coverage of the spec. Note that the spec is not yet finalized and all existing implementations are still lacking as far as I know.

Resolving patch:
diff --git a/ibis/expr/types/relations.py b/ibis/expr/types/relations.py
--- a/ibis/expr/types/relations.py
+++ b/ibis/expr/types/relations.py
@@ -109,6 +109,9 @@ class Table(Expr, _FixedTextJupyterMixin):
     def __array__(self, dtype=None):
         return self.execute().__array__(dtype)
 
+    def __dataframe__(self):
+        return self.to_pyarrow().__dataframe__()
+
     def as_table(self) -> Table:
         """Promote the expression to a table.
 


Test patch:
diff --git a/ibis/backends/tests/test_export.py b/ibis/backends/tests/test_export.py
--- a/ibis/backends/tests/test_export.py
+++ b/ibis/backends/tests/test_export.py
@@ -396,3 +396,19 @@ def test_arrow_timestamp_with_time_zone(alltypes):
 
     (batch,) = t.to_pyarrow_batches()
     assert batch.schema.types == expected
+
+
+@pytest.mark.notimpl(["dask", "druid"])
+@pytest.mark.notimpl(
+    ["mysql"],
+    raises=pa.ArrowInvalid,
+    reason="attempted conversion from decimal to double",
+)
+@pytest.mark.notimpl(
+    ["impala"], raises=AttributeError, reason="missing `fetchmany` on the cursor"
+)
+def test_dataframe_protocol(alltypes):
+    pytest.importorskip("pyarrow", minversion="12")
+    output = alltypes.__dataframe__()
+    assert list(output.column_names()) == alltypes.columns
+    assert alltypes.count().execute() == output.num_rows()