Repository: narwhals-dev/narwhals

Issue:
[Enh]: Pass through the Arrow PyCapsule Interface
### We would like to learn about your use case. For example, if this feature is needed  to adopt Narwhals in an open source project, could you please enter the link to it below?


The [Arrow PyCapsule Protocol](https://arrow.apache.org/docs/format/CDataInterface/PyCapsuleInterface.html) makes it possible to share Arrow data between libraries without requiring a dependency on pyarrow. Many of the DataFrame libraries that narwhals wraps already support it (See https://github.com/apache/arrow/issues/39195 for ecosystem summary), so my thinking is that it wouldn't be too much work to support the protocol on the Narwhals DataFrame itself.

### Please describe the purpose of the new feature or describe the problem to solve.

Vega-Altair recently adopted Narwhals as a way to support polars without a pyarrow dependency, and I'm looking at how I could do the same in VegaFusion. The next VegaFusion release is going to support ingesting DataFrames that support the PyCapsule Interface (WIP by @kylebarron in https://github.com/vega/vegafusion/pull/501), so it would be nice to be able to pass the Narwhals DataFrame directly to VegaFusion and process it using this protocol.

### Suggest a solution if possible.

_No response_

### If you have tried alternatives, please describe them below.

_No response_

### Additional information that may help us understand your needs.

_No response_

Discussion:
MarcoGorelli: cool, thanks!

> If you had some way to access an arrow table from the underlying dataframe

we do have `narwhals.DataFrame.to_arrow`, which returns a pyarrow table - is that what you meant? If so, what else would you suggest adding?
kylebarron: In that case I'd suggest
```py
def __arrow_c_stream__(self, requested_schema: object | None = None) -> object:
    try:
        return self._compliant_frame._native_frame.__arrow_c_stream__(
            requested_schema=requested_schema
        )
    except AttributeError:
        pyarrow_table = self.to_arrow()
        return pyarrow_table.__arrow_c_stream__(requested_schema=requested_schema)
```
So in the first case, we can use the source's implementation of the pycapsule interface and pyarrow doesn't need to be installed, while in the second case we can still ensure the method never raises an exception.
MarcoGorelli: thanks! though I think the user could call this themselves? e.g.
```python
df: nw.DataFrame
try:
    result = df.__arrow_c_stream__(requested_schema=requested_schema)
except AttributeError:
    result = df.to_arrow().__arrow_c_stream__(requested_schema=requested_schema)
```
?
I think that'd be more explicit, not totally sure we should be calling `to_arrow` on behalf of the user. By 'user', I mean the library developer using Narwhals, I'd be inclined to leave it up to them whether or not to fallback to PyArrow
kylebarron: Well, one of my primary arguments for the pycapsule interface is that it allows an ecosystem of data producers and consumers to interoperate without any knowledge of the other, solely by looking for an `__arrow_c_stream__` dunder method. Calling `.to_arrow()` would indeed be more explicit, but it would require library consumers to _know about narwhals_, which I'd argue in the general case is not true. E.g. `pyarrow.table()` only knows to check for `__arrow_c_stream__`.
MarcoGorelli: I was thinking more of the Vegafusion case - I think it's better for them if they choose to explicitly call `to_arrow`. Otherwise they might be calling `__arrow_c_stream__` all over the places, thinking it's cheap, whereas it would've been better to do a single `to_arrow` upfront 😇 

I think any library developer using Narwhals (e.g. vegafusion) would and should know about Narwhals, whereas lower-level libraries like PyArrow shouldn't:
- PyArrow can just check for `__arrow_c_stream__` (as it currently does)
- Vegafusion can choose whether and when to call `to_arrow` before accessing `__arrow_c_stream__`
MarcoGorelli: In any case, as they often say, "in open source, 'no' is temporary but 'yes' is forever" - doubly so with our stable api policy 😆 

So, as the current implementation looks good to you, I'd say - let's start with that, we can always loosen it later if necessary

Thanks for your review and input, much appreciated 🙏 !
kylebarron: > Otherwise they might be calling `__arrow_c_stream__` all over the places, thinking it's cheap, whereas it would've been better to do a single `to_arrow` upfront

As a general note, consumers can't do this because consumers don't have a way to know whether the source is a _table_ that already exists in memory or whether it's a _stream_ that can only be called once. E.g. a pyarrow `RecordBatchReader` is a stream and you can only call `__arrow_c_stream__` once. So for a consumer like vegafusion, it would be important for it to import all the data once and then operate as it needs to on it.
MarcoGorelli: Ah nice, thanks for explaining!

In that case, I'm leaning more towards your suggestion - it would also mean being able to support this for versions of pandas prior to 2.2 but which support converting to pyarrow table
MarcoGorelli: thanks Kyle for your help!

cool, let's ship this 🚢 

Patch:
diff --git a/docs/api-reference/dataframe.md b/docs/api-reference/dataframe.md
index 676f6407..b251b2a5 100644
--- a/docs/api-reference/dataframe.md
+++ b/docs/api-reference/dataframe.md
@@ -4,6 +4,7 @@
     handler: python
     options:
       members:
+        - __arrow_c_stream__
         - __getitem__
         - clone
         - collect_schema
diff --git a/docs/api-reference/series.md b/docs/api-reference/series.md
index 7b7f62b8..f9cc2e6b 100644
--- a/docs/api-reference/series.md
+++ b/docs/api-reference/series.md
@@ -4,6 +4,8 @@
     handler: python
     options:
       members:
+        - __arrow_c_stream__
+        - __getitem__
         - abs
         - alias
         - all
diff --git a/narwhals/dataframe.py b/narwhals/dataframe.py
index 03949674..dfcdce87 100644
--- a/narwhals/dataframe.py
+++ b/narwhals/dataframe.py
@@ -15,6 +15,7 @@ from narwhals.dependencies import get_polars
 from narwhals.dependencies import is_numpy_array
 from narwhals.schema import Schema
 from narwhals.utils import flatten
+from narwhals.utils import parse_version
 
 if TYPE_CHECKING:
     from io import BytesIO
@@ -249,6 +250,30 @@ class DataFrame(BaseFrame[FrameT]):
             + "┘"
         )
 
+    def __arrow_c_stream__(self, requested_schema: object | None = None) -> object:
+        """
+        Export a DataFrame via the Arrow PyCapsule Interface.
+
+        - if the underlying dataframe implements the interface, it'll return that
+        - else, it'll call `to_arrow` and then defer to PyArrow's implementation
+
+        See [PyCapsule Interface](https://arrow.apache.org/docs/dev/format/CDataInterface/PyCapsuleInterface.html)
+        for more.
+        """
+        native_frame = self._compliant_frame._native_frame
+        if hasattr(native_frame, "__arrow_c_stream__"):
+            return native_frame.__arrow_c_stream__(requested_schema=requested_schema)
+        try:
+            import pyarrow as pa  # ignore-banned-import
+        except ModuleNotFoundError as exc:  # pragma: no cover
+            msg = f"PyArrow>=14.0.0 is required for `DataFrame.__arrow_c_stream__` for object of type {type(native_frame)}"
+            raise ModuleNotFoundError(msg) from exc
+        if parse_version(pa.__version__) < (14, 0):  # pragma: no cover
+            msg = f"PyArrow>=14.0.0 is required for `DataFrame.__arrow_c_stream__` for object of type {type(native_frame)}"
+            raise ModuleNotFoundError(msg) from None
+        pa_table = self.to_arrow()
+        return pa_table.__arrow_c_stream__(requested_schema=requested_schema)
+
     def lazy(self) -> LazyFrame[Any]:
         """
         Lazify the DataFrame (if possible).
diff --git a/narwhals/series.py b/narwhals/series.py
index a826e5dd..3c79024c 100644
--- a/narwhals/series.py
+++ b/narwhals/series.py
@@ -7,6 +7,8 @@ from typing import Literal
 from typing import Sequence
 from typing import overload
 
+from narwhals.utils import parse_version
+
 if TYPE_CHECKING:
     import numpy as np
     import pandas as pd
@@ -57,6 +59,32 @@ class Series:
     def __native_namespace__(self) -> Any:
         return self._compliant_series.__native_namespace__()
 
+    def __arrow_c_stream__(self, requested_schema: object | None = None) -> object:
+        """
+        Export a Series via the Arrow PyCapsule Interface.
+
+        Narwhals doesn't implement anything itself here:
+
+        - if the underlying series implements the interface, it'll return that
+        - else, it'll call `to_arrow` and then defer to PyArrow's implementation
+
+        See [PyCapsule Interface](https://arrow.apache.org/docs/dev/format/CDataInterface/PyCapsuleInterface.html)
+        for more.
+        """
+        native_series = self._compliant_series._native_series
+        if hasattr(native_series, "__arrow_c_stream__"):
+            return native_series.__arrow_c_stream__(requested_schema=requested_schema)
+        try:
+            import pyarrow as pa  # ignore-banned-import
+        except ModuleNotFoundError as exc:  # pragma: no cover
+            msg = f"PyArrow>=16.0.0 is required for `Series.__arrow_c_stream__` for object of type {type(native_series)}"
+            raise ModuleNotFoundError(msg) from exc
+        if parse_version(pa.__version__) < (16, 0):  # pragma: no cover
+            msg = f"PyArrow>=16.0.0 is required for `Series.__arrow_c_stream__` for object of type {type(native_series)}"
+            raise ModuleNotFoundError(msg)
+        ca = pa.chunked_array([self.to_arrow()])
+        return ca.__arrow_c_stream__(requested_schema=requested_schema)
+
     @property
     def shape(self) -> tuple[int]:
         """
diff --git a/utils/check_api_reference.py b/utils/check_api_reference.py
index 68c98008..f6e5303c 100644
--- a/utils/check_api_reference.py
+++ b/utils/check_api_reference.py
@@ -45,13 +45,13 @@ with open("docs/api-reference/dataframe.md") as fd:
 documented = [
     remove_prefix(i, "        - ")
     for i in content.splitlines()
-    if i.startswith("        - ")
+    if i.startswith("        - ") and not i.startswith("        - _")
 ]
 if missing := set(top_level_functions).difference(documented):
     print("DataFrame: not documented")  # noqa: T201
     print(missing)  # noqa: T201
     ret = 1
-if extra := set(documented).difference(top_level_functions).difference({"__getitem__"}):
+if extra := set(documented).difference(top_level_functions):
     print("DataFrame: outdated")  # noqa: T201
     print(extra)  # noqa: T201
     ret = 1
@@ -87,7 +87,7 @@ with open("docs/api-reference/series.md") as fd:
 documented = [
     remove_prefix(i, "        - ")
     for i in content.splitlines()
-    if i.startswith("        - ")
+    if i.startswith("        - ") and not i.startswith("        - _")
 ]
 if (
     missing := set(top_level_functions)


Test patch:
diff --git a/tests/frame/arrow_c_stream_test.py b/tests/frame/arrow_c_stream_test.py
new file mode 100644
index 00000000..7a3403f6
--- /dev/null
+++ b/tests/frame/arrow_c_stream_test.py
@@ -0,0 +1,42 @@
+import polars as pl
+import pyarrow as pa
+import pyarrow.compute as pc
+import pytest
+
+import narwhals.stable.v1 as nw
+from narwhals.utils import parse_version
+
+
+@pytest.mark.skipif(
+    parse_version(pl.__version__) < (1, 3), reason="too old for pycapsule in Polars"
+)
+def test_arrow_c_stream_test() -> None:
+    df = nw.from_native(pl.Series([1, 2, 3]).to_frame("a"), eager_only=True)
+    result = pa.table(df)
+    expected = pa.table({"a": [1, 2, 3]})
+    assert pc.all(pc.equal(result["a"], expected["a"])).as_py()
+
+
+@pytest.mark.skipif(
+    parse_version(pl.__version__) < (1, 3), reason="too old for pycapsule in Polars"
+)
+def test_arrow_c_stream_test_invalid(monkeypatch: pytest.MonkeyPatch) -> None:
+    # "poison" the dunder method to make sure it actually got called above
+    monkeypatch.setattr(
+        "narwhals.dataframe.DataFrame.__arrow_c_stream__", lambda *_: 1 / 0
+    )
+    df = nw.from_native(pl.Series([1, 2, 3]).to_frame("a"), eager_only=True)
+    with pytest.raises(ZeroDivisionError, match="division by zero"):
+        pa.table(df)
+
+
+@pytest.mark.skipif(
+    parse_version(pl.__version__) < (1, 3), reason="too old for pycapsule in Polars"
+)
+def test_arrow_c_stream_test_fallback(monkeypatch: pytest.MonkeyPatch) -> None:
+    # Check that fallback to PyArrow works
+    monkeypatch.delattr("polars.DataFrame.__arrow_c_stream__")
+    df = nw.from_native(pl.Series([1, 2, 3]).to_frame("a"), eager_only=True)
+    result = pa.table(df)
+    expected = pa.table({"a": [1, 2, 3]})
+    assert pc.all(pc.equal(result["a"], expected["a"])).as_py()
diff --git a/tests/series_only/arrow_c_stream_test.py b/tests/series_only/arrow_c_stream_test.py
new file mode 100644
index 00000000..9964d740
--- /dev/null
+++ b/tests/series_only/arrow_c_stream_test.py
@@ -0,0 +1,41 @@
+import polars as pl
+import pyarrow as pa
+import pyarrow.compute as pc
+import pytest
+
+import narwhals.stable.v1 as nw
+from narwhals.utils import parse_version
+
+
+@pytest.mark.skipif(
+    parse_version(pl.__version__) < (1, 3), reason="too old for pycapsule in Polars"
+)
+def test_arrow_c_stream_test() -> None:
+    s = nw.from_native(pl.Series([1, 2, 3]), series_only=True)
+    result = pa.chunked_array(s)
+    expected = pa.chunked_array([[1, 2, 3]])
+    assert pc.all(pc.equal(result, expected)).as_py()
+
+
+@pytest.mark.skipif(
+    parse_version(pl.__version__) < (1, 3), reason="too old for pycapsule in Polars"
+)
+def test_arrow_c_stream_test_invalid(monkeypatch: pytest.MonkeyPatch) -> None:
+    # "poison" the dunder method to make sure it actually got called above
+    monkeypatch.setattr("narwhals.series.Series.__arrow_c_stream__", lambda *_: 1 / 0)
+    s = nw.from_native(pl.Series([1, 2, 3]), series_only=True)
+    with pytest.raises(ZeroDivisionError, match="division by zero"):
+        pa.chunked_array(s)
+
+
+@pytest.mark.skipif(
+    parse_version(pl.__version__) < (1, 3), reason="too old for pycapsule in Polars"
+)
+def test_arrow_c_stream_test_fallback(monkeypatch: pytest.MonkeyPatch) -> None:
+    # Check that fallback to PyArrow works
+    monkeypatch.delattr("polars.Series.__arrow_c_stream__")
+    s = nw.from_native(pl.Series([1, 2, 3]).to_frame("a"), eager_only=True)["a"]
+    s.__arrow_c_stream__()
+    result = pa.chunked_array(s)
+    expected = pa.chunked_array([[1, 2, 3]])
+    assert pc.all(pc.equal(result, expected)).as_py()