Repository: narwhals-dev/narwhals
Pull request #520 (2024-07-14 10:52:51)

feat: pyarrow `item`, `is_empty`, `rename`, `write_parquet`
## What type of PR is this? (check all applicable)

- [ ] 💾 Refactor
- [x] ✨ Feature
- [ ] 🐛 Bug Fix
- [ ] 🔧 Optimization
- [ ] 📝 Documentation
- [ ] ✅ Test
- [ ] 🐳 Other

## Related issues 

- Related issue #372 

## Checklist

- [x] Code follows style guide (ruff)
- [x] Tests added 
- [x] Documented the changes

## If you have comments or can explain your changes, please do so below.

Can this be considered addiction 😂🙈?


Patch:
diff --git a/README.md b/README.md
index caf0f751..94bc8c66 100644
--- a/README.md
+++ b/README.md
@@ -9,12 +9,10 @@
 
 [![PyPI version](https://badge.fury.io/py/narwhals.svg)](https://badge.fury.io/py/narwhals)
 
-Extremely lightweight and extensible compatibility layer between Polars, pandas, Modin, and cuDF (and more!).
+Extremely lightweight and extensible compatibility layer between dataframe libraries!
 
-- [Read the documentation](https://narwhals-dev.github.io/narwhals/)
-- [Chat with us on Discord!](https://discord.gg/V3PqtB4VA4)
-- [Join our community call](https://calendar.google.com/calendar/embed?src=27ff6dc5f598c1d94c1f6e627a1aaae680e2fac88f848bda1f2c7946ae74d5ab%40group.calendar.google.com&ctz=Europe%2FLondon)
-- [Read the contributing guide](https://github.com/narwhals-dev/narwhals/blob/main/CONTRIBUTING.md)
+- **Full API support**: cuDF, Modin, pandas, Polars
+- **Interchange-level support**: Ibis, PyArrow, Vaex, anything else which implements the DataFrame Interchange Protocol
 
 Seamlessly support all, without depending on any!
 
@@ -30,6 +28,13 @@ Seamlessly support all, without depending on any!
 - ✅ **Perfect backwards compatibility policy**,
   see [stable api](https://narwhals-dev.github.io/narwhals/backcompat/) for how to opt-in
 
+Get started!
+
+- [Read the documentation](https://narwhals-dev.github.io/narwhals/)
+- [Chat with us on Discord!](https://discord.gg/V3PqtB4VA4)
+- [Join our community call](https://calendar.google.com/calendar/embed?src=27ff6dc5f598c1d94c1f6e627a1aaae680e2fac88f848bda1f2c7946ae74d5ab%40group.calendar.google.com&ctz=Europe%2FLondon)
+- [Read the contributing guide](https://github.com/narwhals-dev/narwhals/blob/main/CONTRIBUTING.md)
+
 ## Used by / integrates with
 
 Join the party!
@@ -68,20 +73,6 @@ There are three steps to writing dataframe-agnostic code using Narwhals:
    - if you started with Modin, you'll get Modin back (and compute will be distributed)
    - if you started with cuDF, you'll get cuDF back (and compute will happen on GPU)
 
-## What about Ibis?
-
-Like Ibis, Narwhals aims to enable dataframe-agnostic code. However, Narwhals comes with **zero** dependencies,
-is about as lightweight as it gets, and is aimed at library developers rather than at end users. It also does
-not aim to support as many backends, instead preferring to focus on dataframes. So, which should you use?
-
-- If you need a SQL frontend in Python: Ibis!
-- If you're a library maintainer and want a lightweight and minimal-overhead layer to get cross-dataframe library support: Narwhals!
-
-Here is the package size increase which would result from installing each tool in a non-pandas
-environment:
-
-![image](https://github.com/MarcoGorelli/narwhals/assets/33491632/a8dfba78-feb1-48c1-960a-5b9b03585fa5)
-
 ## Example
 
 See the [tutorial](https://narwhals-dev.github.io/narwhals/basics/dataframe/) for several examples!
diff --git a/docs/levels.md b/docs/levels.md
new file mode 100644
index 00000000..d632baa6
--- /dev/null
+++ b/docs/levels.md
@@ -0,0 +1,40 @@
+# Levels
+
+Narwhals comes with two levels of support: "full" and "interchange".
+
+Libraries for which we have full support can benefit from the whole
+[Narwhals API](https://narwhals-dev.github.io/narwhals/api-reference/).
+
+For example:
+
+```python exec="1" source="above"
+import narwhals as nw
+from narwhals.typing import FrameT
+
+@nw.narwhalify
+def func(df: FrameT) -> FrameT:
+    return df.group_by('a').agg(
+        b_mean=nw.col('b').mean(),
+        b_std=nw.col('b').std(),
+    )
+```
+will work for any of pandas, Polars, cuDF, and Modin.
+
+However, sometimes you don't need to do complex operations on dataframes - all you need
+is to inspect the schema a bit before making other decisions, such as which columns to
+select or whether to convert to another library. For that purpose, we also provide "interchange"
+level of support. If a library implements the
+[Dataframe Interchange Protocol](https://data-apis.org/dataframe-protocol/latest/), then
+a call such as
+
+```python exec="1" source="above"
+import narwhals as nw
+from narwhals.schema import Schema
+
+
+def func(df_any: Any) -> Schema:
+    df = nw.from_native(df, eager_or_interchange_only=True)
+    return df.schema
+```
+is also supported, meaning that, in addition to the libraries mentioned above, you can
+also pass Ibis, Vaex, PyArrow, and any other library which implements the protocol.
diff --git a/mkdocs.yml b/mkdocs.yml
index 101c1a7f..891e96ed 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -13,6 +13,7 @@ nav:
     - basics/complete_example.md
   - Other concepts:
     - other/pandas_index.md
+  - levels.md
   - overhead.md
   - backcompat.md
   - extending.md
diff --git a/narwhals/_arrow/dataframe.py b/narwhals/_arrow/dataframe.py
index dffa5500..7bfa3c7a 100644
--- a/narwhals/_arrow/dataframe.py
+++ b/narwhals/_arrow/dataframe.py
@@ -12,6 +12,7 @@ from narwhals._arrow.utils import validate_dataframe_comparand
 from narwhals._expression_parsing import evaluate_into_exprs
 from narwhals.dependencies import get_numpy
 from narwhals.dependencies import get_pyarrow
+from narwhals.dependencies import get_pyarrow_parquet
 from narwhals.utils import flatten
 
 if TYPE_CHECKING:
@@ -340,3 +341,33 @@ class ArrowDataFrame:
 
     def clone(self) -> Self:
         raise NotImplementedError("clone is not yet supported on PyArrow tables")
+
+    def is_empty(self: Self) -> bool:
+        return self.shape[0] == 0
+
+    def item(self: Self, row: int | None = None, column: int | str | None = None) -> Any:
+        if row is None and column is None:
+            if self.shape != (1, 1):
+                msg = (
+                    "can only call `.item()` if the dataframe is of shape (1, 1),"
+                    " or if explicit row/col values are provided;"
+                    f" frame has shape {self.shape!r}"
+                )
+                raise ValueError(msg)
+            return self._native_dataframe[0][0].as_py()
+
+        elif row is None or column is None:
+            msg = "cannot call `.item()` with only one of `row` or `column`"
+            raise ValueError(msg)
+
+        _col = self.columns.index(column) if isinstance(column, str) else column
+        return self._native_dataframe[_col][row].as_py()
+
+    def rename(self, mapping: dict[str, str]) -> Self:
+        df = self._native_dataframe
+        new_cols = [mapping.get(c, c) for c in df.column_names]
+        return self._from_native_dataframe(df.rename_columns(new_cols))
+
+    def write_parquet(self, file: Any) -> Any:
+        pp = get_pyarrow_parquet()
+        pp.write_table(self._native_dataframe, file)
diff --git a/narwhals/_arrow/series.py b/narwhals/_arrow/series.py
index 8c49bb39..dc29de96 100644
--- a/narwhals/_arrow/series.py
+++ b/narwhals/_arrow/series.py
@@ -248,6 +248,17 @@ class ArrowSeries:
         else:
             return self._from_native_series(ser.slice(abs(n)))
 
+    def item(self: Self, index: int | None = None) -> Any:
+        if index is None:
+            if len(self) != 1:
+                msg = (
+                    "can only call '.item()' if the Series is of length 1,"
+                    f" or an explicit index is provided (Series is of length {len(self)})"
+                )
+                raise ValueError(msg)
+            return self._native_series[0].as_py()
+        return self._native_series[index].as_py()
+
     @property
     def shape(self) -> tuple[int]:
         return (len(self._native_series),)
diff --git a/narwhals/dependencies.py b/narwhals/dependencies.py
index 90260903..cbb5eca3 100644
--- a/narwhals/dependencies.py
+++ b/narwhals/dependencies.py
@@ -51,6 +51,15 @@ def get_pyarrow_compute() -> Any:  # pragma: no cover
     return None
 
 
+def get_pyarrow_parquet() -> Any:  # pragma: no cover
+    """Get pyarrow.parquet module (if pyarrow has already been imported - else return None)."""
+    if "pyarrow" in sys.modules:
+        import pyarrow.parquet as pp
+
+        return pp
+    return None
+
+
 def get_numpy() -> Any:
     """Get numpy module (if already imported - else return None)."""
     return sys.modules.get("numpy", None)
diff --git a/pyproject.toml b/pyproject.toml
index 934a82cf..8bcd7ef0 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -8,7 +8,7 @@ version = "1.0.6"
 authors = [
   { name="Marco Gorelli", email="33491632+MarcoGorelli@users.noreply.github.com" },
 ]
-description = "Extremely lightweight compatibility layer between pandas, Polars, cuDF, and Modin"
+description = "Extremely lightweight compatibility layer between dataframe libraries"
 readme = "README.md"
 requires-python = ">=3.8"
 classifiers = [
diff --git a/utils/check_backend_completeness.py b/utils/check_backend_completeness.py
index cb9a6f39..a9b6ae73 100644
--- a/utils/check_backend_completeness.py
+++ b/utils/check_backend_completeness.py
@@ -13,14 +13,10 @@ from narwhals._arrow.dataframe import ArrowDataFrame
 
 MISSING = [
     "DataFrame.is_duplicated",
-    "DataFrame.is_empty",
     "DataFrame.is_unique",
-    "DataFrame.item",
     "DataFrame.iter_rows",
     "DataFrame.pipe",
-    "DataFrame.rename",
     "DataFrame.unique",
-    "DataFrame.write_parquet",
     "Series.drop_nulls",
     "Series.fill_null",
     "Series.from_iterable",
@@ -32,7 +28,6 @@ MISSING = [
     "Series.is_null",
     "Series.is_sorted",
     "Series.is_unique",
-    "Series.item",
     "Series.len",
     "Series.n_unique",
     "Series.quantile",


Test patch:
diff --git a/tests/frame/is_empty_test.py b/tests/frame/is_empty_test.py
index 06defe73..b49fbb11 100644
--- a/tests/frame/is_empty_test.py
+++ b/tests/frame/is_empty_test.py
@@ -8,9 +8,9 @@ import narwhals.stable.v1 as nw
 
 
 @pytest.mark.parametrize(("threshold", "expected"), [(0, False), (10, True)])
-def test_is_empty(constructor: Any, threshold: Any, expected: Any) -> None:
+def test_is_empty(constructor_with_pyarrow: Any, threshold: Any, expected: Any) -> None:
     data = {"a": [1, 3, 2], "b": [4, 4, 6], "z": [7.0, 8, 9]}
-    df_raw = constructor(data)
+    df_raw = constructor_with_pyarrow(data)
     df = nw.from_native(df_raw, eager_only=True)
     result = df.filter(nw.col("a") > threshold).is_empty()
     assert result == expected
diff --git a/tests/frame/rename_test.py b/tests/frame/rename_test.py
index ab901803..c857a92e 100644
--- a/tests/frame/rename_test.py
+++ b/tests/frame/rename_test.py
@@ -4,10 +4,9 @@ import narwhals.stable.v1 as nw
 from tests.utils import compare_dicts
 
 
-def test_rename(constructor: Any) -> None:
+def test_rename(constructor_with_pyarrow: Any) -> None:
     data = {"a": [1, 3, 2], "b": [4, 4, 6], "z": [7.0, 8, 9]}
-    df = nw.from_native(constructor(data), eager_only=True)
+    df = nw.from_native(constructor_with_pyarrow(data), eager_only=True)
     result = df.rename({"a": "x", "b": "y"})
-    result_native = nw.to_native(result)
     expected = {"x": [1, 3, 2], "y": [4, 4, 6], "z": [7.0, 8, 9]}
-    compare_dicts(result_native, expected)
+    compare_dicts(result, expected)
diff --git a/tests/frame/rows_test.py b/tests/frame/rows_test.py
index 371ff35d..9d60f48a 100644
--- a/tests/frame/rows_test.py
+++ b/tests/frame/rows_test.py
@@ -97,6 +97,7 @@ def test_rows(
         ):
             df.rows(named=named)
         return
+
     result = df.rows(named=named)
     assert result == expected
 
diff --git a/tests/frame/test_common.py b/tests/frame/test_common.py
index fe811d80..49f41916 100644
--- a/tests/frame/test_common.py
+++ b/tests/frame/test_common.py
@@ -334,7 +334,7 @@ def test_library(df_raw: Any, df_raw_right: Any) -> None:
         df_left.join(df_right, left_on=["a"], right_on=["a"], how="inner")
 
 
-@pytest.mark.parametrize("df_raw", [df_pandas, df_polars])
+@pytest.mark.parametrize("df_raw", [df_pandas, df_polars, df_pa])
 @pytest.mark.parametrize(
     ("row", "column", "expected"),
     [(0, 2, 7), (1, "z", 8)],
@@ -350,7 +350,7 @@ def test_item(
     assert df.select("a").head(1).item() == 1
 
 
-@pytest.mark.parametrize("df_raw", [df_pandas, df_polars])
+@pytest.mark.parametrize("df_raw", [df_pandas, df_polars, df_pa])
 @pytest.mark.parametrize(
     ("row", "column", "err_msg"),
     [
diff --git a/tests/frame/write_parquet_test.py b/tests/frame/write_parquet_test.py
index 88e35b3f..a00c02f7 100644
--- a/tests/frame/write_parquet_test.py
+++ b/tests/frame/write_parquet_test.py
@@ -15,7 +15,9 @@ data = {"a": [1, 2, 3]}
 @pytest.mark.skipif(
     parse_version(pd.__version__) < parse_version("2.0.0"), reason="too old for pyarrow"
 )
-def test_write_parquet(constructor: Any, tmpdir: pytest.TempdirFactory) -> None:
+def test_write_parquet(
+    constructor_with_pyarrow: Any, tmpdir: pytest.TempdirFactory
+) -> None:
     path = str(tmpdir / "foo.parquet")  # type: ignore[operator]
-    nw.from_native(constructor(data), eager_only=True).write_parquet(path)
+    nw.from_native(constructor_with_pyarrow(data), eager_only=True).write_parquet(path)
     assert os.path.exists(path)
diff --git a/tests/series/cast_test.py b/tests/series/cast_test.py
index 2c66ca92..37ae55a0 100644
--- a/tests/series/cast_test.py
+++ b/tests/series/cast_test.py
@@ -105,6 +105,7 @@ def test_cast_date_datetime_invalid() -> None:
         df.select(nw.col("a").cast(nw.Date))
 
 
+@pytest.mark.filterwarnings("ignore: casting period")
 def test_unknown_to_int() -> None:
     df = pd.DataFrame({"a": pd.period_range("2000", periods=3, freq="min")})
     assert nw.from_native(df).select(nw.col("a").cast(nw.Int64)).schema == {"a": nw.Int64}
diff --git a/tests/series/test_common.py b/tests/series/test_common.py
index a92b2f90..842883b7 100644
--- a/tests/series/test_common.py
+++ b/tests/series/test_common.py
@@ -7,6 +7,7 @@ from typing import Literal
 import numpy as np
 import pandas as pd
 import polars as pl
+import pyarrow as pa
 import pytest
 from numpy.testing import assert_array_equal
 from pandas.testing import assert_series_equal
@@ -40,6 +41,7 @@ else:  # pragma: no cover
     df_pandas_nullable = df_pandas
 df_polars = pl.DataFrame({"a": [1, 3, 2], "b": [4, 4, 6], "z": [7.0, 8, 9]})
 df_lazy = pl.LazyFrame({"a": [1, 3, 2], "b": [4, 4, 6], "z": [7.0, 8, 9]})
+df_pa = pa.table({"a": [1, 3, 2], "b": [4, 4, 6], "z": [7.0, 8, 9]})
 
 
 @pytest.mark.parametrize(
@@ -255,13 +257,13 @@ def test_cast_string() -> None:
 df_pandas = pd.DataFrame({"a": [1, 3, 2], "b": [4, 4, 6], "z": [7.0, 8, 9]})
 
 
-@pytest.mark.parametrize("df_raw", [df_pandas, df_polars])
+@pytest.mark.parametrize("df_raw", [df_pandas, df_polars, df_pa])
 @pytest.mark.parametrize(("index", "expected"), [(0, 1), (1, 3)])
 def test_item(df_raw: Any, index: int, expected: int) -> None:
     s = nw.from_native(df_raw["a"], series_only=True)
     result = s.item(index)
     assert result == expected
-    assert nw.from_native(df_raw["a"].head(1), series_only=True).item() == 1
+    assert s.head(1).item() == 1
 
     with pytest.raises(
         ValueError,