Repository: ibis-project/ibis

Issue:
feat: Implement Arrow PyCapsule Interface
### Is your feature request related to a problem?

Currently Ibis integrates with Arrow via the `to_pyarrow` method. The downside of this is that library consumers have to:

1. Be aware of ibis
2. Look for specific ibis data types
3. Know that they can call this `to_pyarrow` method, which tends to be named differently in different libraries. E.g. DuckDB calls it `.arrow()` and Polars calls it `.to_arrow()`.

### What is the motivation behind your request?

The [Arrow PyCapsule Interface](https://arrow.apache.org/docs/format/CDataInterface/PyCapsuleInterface.html) is a new standard for exchanging Arrow data in Python. Among other benefits, this defines a single method name (`__arrow_c_stream__`) that is public and standardized

This means that other libraries don't have to build specific connectors to Polars, DuckDB, Ibis, pyarrow, etc, but rather can implement support for _any_ input object with an `__arrow_c_stream__` method. For my particular use case, a geospatial visualization library I develop, [Lonboard](https://github.com/developmentseed/lonboard), is Arrow based and looks for this method.

### Describe the solution you'd like

Implement an `__arrow_c_stream__` method wherever there's currently a `to_pyarrow` method. This could be as simple as

```py
    def __arrow_c_stream__(self, requested_schema):
        return self.to_pyarrow().__arrow_c_stream__(requested_schema)
```

where it uses the fact that the pyarrow `Table` class implements this as of v14 or so (not 100% sure which version it was added in)

### What version of ibis are you running?

I haven't run ibis yet but `__arrow_c_stream__` is not found in a code search of the repo.

### What backend(s) are you using, if any?

_No response_

### Code of Conduct

- [X] I agree to follow this project's Code of Conduct


Discussion:
Makes sense to me! Would we also want to implement `__arrow_c_schema__` (as per the docs you linked)?
There's some ongoing discussion about this https://github.com/apache/arrow/issues/39689 but my own understanding is that if you're exporting a "table" or "stream of batches", then you'd only implement the `__arrow_c_stream__` method. While if you're defining your own arrow-compatible data types (which maybe ibis is), then on that data type or field object you could implement `__arrow_c_schema__`

Resolving patch:
diff --git a/ibis/expr/types/relations.py b/ibis/expr/types/relations.py
--- a/ibis/expr/types/relations.py
+++ b/ibis/expr/types/relations.py
@@ -188,6 +188,9 @@ def __dataframe__(self, nan_as_null: bool = False, allow_copy: bool = True):
 
         return IbisDataFrame(self, nan_as_null=nan_as_null, allow_copy=allow_copy)
 
+    def __arrow_c_stream__(self, requested_schema: object | None = None) -> object:
+        return self.to_pyarrow().__arrow_c_stream__(requested_schema)
+
     def __pyarrow_result__(
         self, table: pa.Table, data_mapper: type[PyArrowData] | None = None
     ) -> pa.Table:


Test patch:
diff --git a/ibis/backends/tests/test_export.py b/ibis/backends/tests/test_export.py
--- a/ibis/backends/tests/test_export.py
+++ b/ibis/backends/tests/test_export.py
@@ -4,6 +4,7 @@
 import pyarrow as pa
 import pyarrow.csv as pcsv
 import pytest
+from packaging.version import parse as vparse
 from pytest import param
 
 import ibis
@@ -44,6 +45,20 @@
 limit_no_limit = limit + no_limit
 
 
+@pytest.mark.skipif(
+    vparse(pa.__version__) < vparse("14"), reason="pyarrow >= 14 required"
+)
+def test_table___arrow_c_stream__(awards_players):
+    res = pa.table(awards_players)
+    sol = awards_players.to_pyarrow()
+    assert res.equals(sol)
+
+    # With explicit schema
+    schema = awards_players.schema().to_pyarrow()
+    res = pa.table(awards_players, schema=schema)
+    assert res.equals(sol)
+
+
 @pytest.mark.parametrize("limit", limit_no_limit)
 def test_table_to_pyarrow_batches(limit, awards_players):
     with awards_players.to_pyarrow_batches(limit=limit) as batch_reader: