Repository: probabl-ai/skore

Issue:
feat: Add polars support
### Is your feature request related to a problem? Please describe.

Following https://github.com/probabl-ai/skore/issues/712, we should use polars more than pandas for data frames

### Describe the solution you'd like

The skore project should support Polars dataframes as items

Discussion:

Patch:
diff --git a/docs/design/0000-adr-template.md b/docs/design/0000-adr-template.md
new file mode 100644
index 0000000..72c69bf
--- /dev/null
+++ b/docs/design/0000-adr-template.md
@@ -0,0 +1,74 @@
+---
+# These are optional metadata elements. Feel free to remove any of them.
+status: "{proposed | rejected | accepted | deprecated | … | superseded by ADR-0123"
+date: {YYYY-MM-DD when the decision was last updated}
+decision-makers: {list everyone involved in the decision}
+consulted: {list everyone whose opinions are sought (typically subject-matter experts); and with whom there is a two-way communication}
+informed: {list everyone who is kept up-to-date on progress; and with whom there is a one-way communication}
+---
+
+# {short title, representative of solved problem and found solution}
+
+## Context and Problem Statement
+
+{Describe the context and problem statement, e.g., in free form using two to three sentences or in the form of an illustrative story. You may want to articulate the problem in form of a question and add links to collaboration boards or issue management systems.}
+
+<!-- This is an optional element. Feel free to remove. -->
+## Decision Drivers
+
+* {decision driver 1, e.g., a force, facing concern, …}
+* {decision driver 2, e.g., a force, facing concern, …}
+* … <!-- numbers of drivers can vary -->
+
+## Considered Options
+
+* {title of option 1}
+* {title of option 2}
+* {title of option 3}
+* … <!-- numbers of options can vary -->
+
+## Decision Outcome
+
+Chosen option: "{title of option 1}", because {justification. e.g., only option, which meets k.o. criterion decision driver | which resolves force {force} | … | comes out best (see below)}.
+
+<!-- This is an optional element. Feel free to remove. -->
+### Consequences
+
+* Good, because {positive consequence, e.g., improvement of one or more desired qualities, …}
+* Bad, because {negative consequence, e.g., compromising one or more desired qualities, …}
+* … <!-- numbers of consequences can vary -->
+
+<!-- This is an optional element. Feel free to remove. -->
+### Confirmation
+
+{Describe how the implementation of/compliance with the ADR can/will be confirmed. Are the design that was decided for and its implementation in line with the decision made? E.g., a design/code review or a test with a library such as ArchUnit can help validate this. Not that although we classify this element as optional, it is included in many ADRs.}
+
+<!-- This is an optional element. Feel free to remove. -->
+## Pros and Cons of the Options
+
+### {title of option 1}
+
+<!-- This is an optional element. Feel free to remove. -->
+{example | description | pointer to more information | …}
+
+* Good, because {argument a}
+* Good, because {argument b}
+<!-- use "neutral" if the given argument weights neither for good nor bad -->
+* Neutral, because {argument c}
+* Bad, because {argument d}
+* … <!-- numbers of pros and cons can vary -->
+
+### {title of other option}
+
+{example | description | pointer to more information | …}
+
+* Good, because {argument a}
+* Good, because {argument b}
+* Neutral, because {argument c}
+* Bad, because {argument d}
+* …
+
+<!-- This is an optional element. Feel free to remove. -->
+## More Information
+
+{You might want to provide additional evidence/confidence for the decision outcome here and/or document the team agreement on the decision and/or define when/how the decision should be realized and if/when it should be re-visited. Links to other decisions and resources might appear here as well.}
diff --git a/docs/design/0001-postpone-narwhals.md b/docs/design/0001-postpone-narwhals.md
new file mode 100644
index 0000000..b19edc3
--- /dev/null
+++ b/docs/design/0001-postpone-narwhals.md
@@ -0,0 +1,39 @@
+---
+# These are optional metadata elements. Feel free to remove any of them.
+status: "accepted"
+date: 2024-11-26
+decision-makers: @augustebaum @tuscland
+---
+
+# Postone usage of generic DataFrame library
+
+## Context and Problem Statement
+
+skore projects natively support inserting `pandas` DataFrames, and with PR [#792](https://github.com/probabl-ai/skore/pull/792) it will also support `polars` DataFrames.
+The question arises as to whether skore should use the `narwhals` library under the hood, or any another generic DataFrame library, as skore could then support many more DataFrame objects with little extra effort to integrate them. 
+
+## Considered Options
+
+* Postpone usage of generic DataFrame library
+* Use `narwhals`
+* Use `arrow`
+
+## Decision Outcome
+
+Chosen option: "Postpone usage of generic DataFrame library" 
+
+<!-- This is an optional element. Feel free to remove. -->
+### Consequences
+
+* Good, because there is not enough foresight to know which generic library serves our needs the best
+    - It is likely that we will explore generic formats in the near future in any case, for example:
+        - If the storage backend for skore changes from `diskcache` to something else
+        - if JSON as an exchange format is not robust enough for large data
+    - Since we don't have experience with `narwhals` or `arrow`, integrating one of them might result in unexpected roadblocks which would take time to resolve
+* Bad, because in the short term, adding support for a new DataFrame object will take more work
+    - At the same time, it might be unlikely that support for another DataFrame implementation will be requested in the short term.
+
+<!-- This is an optional element. Feel free to remove. -->
+## More Information
+
+This decision is being taken while reviewing PR [#792](https://github.com/probabl-ai/skore/pull/792), whose objective is to support inserting `polars` DataFrames.
diff --git a/docs/design/README.md b/docs/design/README.md
new file mode 100644
index 0000000..49616ad
--- /dev/null
+++ b/docs/design/README.md
@@ -0,0 +1,23 @@
+# Skore Architecture Decision Records (ADRs)
+
+Welcome to the Architecture Decision Records (ADR) for Skore.
+
+## What Are ADRs?
+
+Architecture Decision Records document the architectural and technical decisions made in a project. They help:
+
+* Preserve the reasoning behind key choices.
+* Guide new contributors in understanding the project's direction.
+* Enable revisiting or revising decisions as the project evolves in an open way.
+
+## How to Use This Folder
+
+Each ADR is stored as a markdown file and follows a sequential numbering system. The ADRs are organized to address different aspects of the backend architecture. New contributors and stakeholders can start by reviewing the decisions below to understand the foundational choices.
+
+## Updating ADRs
+
+If you propose a new decision or change an existing one:
+
+* Write a new ADR, referencing the previous one it supersedes (if applicable).
+* Follow the [established ADR template](https://github.com/adr/madr/blob/4.0.0/template/adr-template.md), [local version available](./0000-adr-template.md).
+* Submit it for team review and approval.
diff --git a/skore/pyproject.toml b/skore/pyproject.toml
index d30200d..1dd53db 100644
--- a/skore/pyproject.toml
+++ b/skore/pyproject.toml
@@ -12,6 +12,7 @@ dependencies = [
   "diskcache",
   "fastapi",
   "plotly>=5,<6",
+  "pyarrow",
   "rich",
   "skops",
   "uvicorn",
diff --git a/skore/src/skore/item/__init__.py b/skore/src/skore/item/__init__.py
index c6b879f..5b2d693 100644
--- a/skore/src/skore/item/__init__.py
+++ b/skore/src/skore/item/__init__.py
@@ -12,6 +12,8 @@ from skore.item.media_item import MediaItem
 from skore.item.numpy_array_item import NumpyArrayItem
 from skore.item.pandas_dataframe_item import PandasDataFrameItem
 from skore.item.pandas_series_item import PandasSeriesItem
+from skore.item.polars_dataframe_item import PolarsDataFrameItem
+from skore.item.polars_series_item import PolarsSeriesItem
 from skore.item.primitive_item import PrimitiveItem
 from skore.item.sklearn_base_estimator_item import SklearnBaseEstimatorItem
 
@@ -22,6 +24,8 @@ def object_to_item(object: Any) -> Item:
         PrimitiveItem,
         PandasDataFrameItem,
         PandasSeriesItem,
+        PolarsDataFrameItem,
+        PolarsSeriesItem,
         NumpyArrayItem,
         SklearnBaseEstimatorItem,
         MediaItem,
@@ -47,6 +51,8 @@ __all__ = [
     "NumpyArrayItem",
     "PandasDataFrameItem",
     "PandasSeriesItem",
+    "PolarsDataFrameItem",
+    "PolarsSeriesItem",
     "PrimitiveItem",
     "SklearnBaseEstimatorItem",
     "object_to_item",
diff --git a/skore/src/skore/item/item_repository.py b/skore/src/skore/item/item_repository.py
index d5a7175..45437ba 100644
--- a/skore/src/skore/item/item_repository.py
+++ b/skore/src/skore/item/item_repository.py
@@ -21,6 +21,8 @@ from skore.item.media_item import MediaItem
 from skore.item.numpy_array_item import NumpyArrayItem
 from skore.item.pandas_dataframe_item import PandasDataFrameItem
 from skore.item.pandas_series_item import PandasSeriesItem
+from skore.item.polars_dataframe_item import PolarsDataFrameItem
+from skore.item.polars_series_item import PolarsSeriesItem
 from skore.item.primitive_item import PrimitiveItem
 from skore.item.sklearn_base_estimator_item import SklearnBaseEstimatorItem
 
@@ -40,6 +42,8 @@ class ItemRepository:
         "NumpyArrayItem": NumpyArrayItem,
         "PandasDataFrameItem": PandasDataFrameItem,
         "PandasSeriesItem": PandasSeriesItem,
+        "PolarsDataFrameItem": PolarsDataFrameItem,
+        "PolarsSeriesItem": PolarsSeriesItem,
         "PrimitiveItem": PrimitiveItem,
         "CrossValidationItem": CrossValidationItem,
         "CrossValidationAggregationItem": CrossValidationAggregationItem,
diff --git a/skore/src/skore/item/polars_dataframe_item.py b/skore/src/skore/item/polars_dataframe_item.py
new file mode 100644
index 0000000..e7ad790
--- /dev/null
+++ b/skore/src/skore/item/polars_dataframe_item.py
@@ -0,0 +1,98 @@
+"""PolarsDataFrameItem.
+
+This module defines the PolarsDataFrameItem class,
+which represents a polars DataFrame item.
+"""
+
+from __future__ import annotations
+
+from functools import cached_property
+from typing import TYPE_CHECKING
+
+from skore.item.item import Item, ItemTypeError
+
+if TYPE_CHECKING:
+    import polars
+
+
+class PolarsToJSONError(Exception):
+    """Something happened while converting a polars DataFrame to JSON."""
+
+
+class PolarsDataFrameItem(Item):
+    """
+    A class to represent a polars DataFrame item.
+
+    This class encapsulates a polars DataFrame along with its
+    creation and update timestamps.
+    """
+
+    def __init__(
+        self,
+        dataframe_json: str,
+        created_at: str | None = None,
+        updated_at: str | None = None,
+    ):
+        """
+        Initialize a PolarsDataFrameItem.
+
+        Parameters
+        ----------
+        dataframe_json : str
+            The JSON representation of the dataframe, without its index.
+        created_at : str
+            The creation timestamp in ISO format.
+        updated_at : str
+            The last update timestamp in ISO format.
+        """
+        super().__init__(created_at, updated_at)
+
+        self.dataframe_json = dataframe_json
+
+    @cached_property
+    def dataframe(self) -> polars.DataFrame:
+        """
+        The polars DataFrame from the persistence.
+
+        Its content can differ from the original dataframe because it has been
+        serialized using polars' `to_json` function and not pickled, in order to be
+        environment-independent.
+        """
+        import io
+
+        import polars
+
+        with io.StringIO(self.dataframe_json) as df_stream:
+            dataframe = polars.read_json(df_stream)
+            return dataframe
+
+    @classmethod
+    def factory(cls, dataframe: polars.DataFrame) -> PolarsDataFrameItem:
+        """
+        Create a new PolarsDataFrameItem instance from a polars DataFrame.
+
+        Parameters
+        ----------
+        dataframe : polars.DataFrame
+            The polars DataFrame to store.
+
+        Returns
+        -------
+        PolarsDataFrameItem
+            A new PolarsDataFrameItem instance.
+
+        Notes
+        -----
+        The dataframe must be JSON serializable.
+        """
+        import polars
+
+        if not isinstance(dataframe, polars.DataFrame):
+            raise ItemTypeError(f"Type '{dataframe.__class__}' is not supported.")
+
+        try:
+            dataframe_json = dataframe.write_json()
+        except Exception as e:
+            raise PolarsToJSONError("Conversion to JSON failed") from e
+
+        return cls(dataframe_json=dataframe_json)
diff --git a/skore/src/skore/item/polars_series_item.py b/skore/src/skore/item/polars_series_item.py
new file mode 100644
index 0000000..71970cc
--- /dev/null
+++ b/skore/src/skore/item/polars_series_item.py
@@ -0,0 +1,86 @@
+"""PolarsSeriesItem.
+
+This module defines the PolarsSeriesItem class,
+which represents a polars Series item.
+"""
+
+from __future__ import annotations
+
+from functools import cached_property
+from typing import TYPE_CHECKING
+
+from skore.item.item import Item, ItemTypeError
+
+if TYPE_CHECKING:
+    import polars
+
+
+class PolarsSeriesItem(Item):
+    """
+    A class to represent a polars Series item.
+
+    This class encapsulates a polars Series along with its
+    creation and update timestamps.
+    """
+
+    def __init__(
+        self,
+        series_json: str,
+        created_at: str | None = None,
+        updated_at: str | None = None,
+    ):
+        """
+        Initialize a PolarsSeriesItem.
+
+        Parameters
+        ----------
+        series_json : str
+            The JSON representation of the series, without its index.
+        created_at : str
+            The creation timestamp in ISO format.
+        updated_at : str
+            The last update timestamp in ISO format.
+        """
+        super().__init__(created_at, updated_at)
+
+        self.series_json = series_json
+
+    @cached_property
+    def series(self) -> polars.Series:
+        """
+        The polars Series from the persistence.
+
+        Its content can differ from the original series because it has been serialized
+        using polars' `to_json` function and not pickled, in order to be
+        environment-independent.
+        """
+        import io
+
+        import polars
+
+        with io.StringIO(self.series_json) as series_stream:
+            series = polars.read_json(series_stream).to_series(0)
+
+            return series
+
+    @classmethod
+    def factory(cls, series: polars.Series) -> PolarsSeriesItem:
+        """
+        Create a new PolarsSeriesItem instance from a polars Series.
+
+        Parameters
+        ----------
+        series : polars.Series
+            The polars Series to store.
+
+        Returns
+        -------
+        PolarsSeriesItem
+            A new PolarsSeriesItem instance.
+        """
+        import polars
+
+        if not isinstance(series, polars.Series):
+            raise ItemTypeError(f"Type '{series.__class__}' is not supported.")
+
+        return cls(series_json=series.to_frame().write_json())
diff --git a/skore/src/skore/project/project.py b/skore/src/skore/project/project.py
index 8376ed4..fb33ddd 100644
--- a/skore/src/skore/project/project.py
+++ b/skore/src/skore/project/project.py
@@ -11,6 +11,8 @@ from skore.item import (
     NumpyArrayItem,
     PandasDataFrameItem,
     PandasSeriesItem,
+    PolarsDataFrameItem,
+    PolarsSeriesItem,
     PrimitiveItem,
     SklearnBaseEstimatorItem,
     object_to_item,
@@ -140,6 +142,10 @@ class Project:
             return item.dataframe
         elif isinstance(item, PandasSeriesItem):
             return item.series
+        elif isinstance(item, PolarsDataFrameItem):
+            return item.dataframe
+        elif isinstance(item, PolarsSeriesItem):
+            return item.series
         elif isinstance(item, SklearnBaseEstimatorItem):
             return item.estimator
         elif isinstance(item, CrossValidationItem):
@@ -154,7 +160,7 @@ class Project:
 
         Parameters
         ----------
-        key: str
+        key : str
             The key corresponding to the item to get.
 
         Returns
diff --git a/skore/src/skore/ui/project_routes.py b/skore/src/skore/ui/project_routes.py
index 39f2f80..86ae63a 100644
--- a/skore/src/skore/ui/project_routes.py
+++ b/skore/src/skore/ui/project_routes.py
@@ -15,6 +15,8 @@ from skore.item.media_item import MediaItem
 from skore.item.numpy_array_item import NumpyArrayItem
 from skore.item.pandas_dataframe_item import PandasDataFrameItem
 from skore.item.pandas_series_item import PandasSeriesItem
+from skore.item.polars_dataframe_item import PolarsDataFrameItem
+from skore.item.polars_series_item import PolarsSeriesItem
 from skore.item.primitive_item import PrimitiveItem
 from skore.item.sklearn_base_estimator_item import SklearnBaseEstimatorItem
 from skore.project import Project
@@ -58,6 +60,12 @@ def __serialize_project(project: Project) -> SerializedProject:
             elif isinstance(item, PandasSeriesItem):
                 value = item.series.to_list()
                 media_type = "text/markdown"
+            elif isinstance(item, PolarsDataFrameItem):
+                value = item.dataframe.to_pandas().to_dict(orient="tight")
+                media_type = "application/vnd.dataframe+json"
+            elif isinstance(item, PolarsSeriesItem):
+                value = item.series.to_list()
+                media_type = "text/markdown"
             elif isinstance(item, SklearnBaseEstimatorItem):
                 value = item.estimator_html_repr
                 media_type = "application/vnd.sklearn.estimator+html"


Test patch:
diff --git a/skore/tests/unit/item/test_pandas_dataframe_item.py b/skore/tests/unit/item/test_pandas_dataframe_item.py
index 09656da..e891327 100644
--- a/skore/tests/unit/item/test_pandas_dataframe_item.py
+++ b/skore/tests/unit/item/test_pandas_dataframe_item.py
@@ -53,6 +53,7 @@ class TestPandasDataFrameItem:
         dataframe = DataFrame([{"key": np.array([1])}], Index([0], name="myIndex"))
         item = PandasDataFrameItem.factory(dataframe)
 
+        # NOTE: isinstance would not work because numpy.array is an instance of list
         assert type(item.dataframe["key"].iloc[0]) is list
 
     @pytest.mark.order(1)
diff --git a/skore/tests/unit/item/test_pandas_series_item.py b/skore/tests/unit/item/test_pandas_series_item.py
index dbc6278..232a179 100644
--- a/skore/tests/unit/item/test_pandas_series_item.py
+++ b/skore/tests/unit/item/test_pandas_series_item.py
@@ -53,6 +53,7 @@ class TestPandasSeriesItem:
         series = Series([np.array([1])], Index([0], name="myIndex"))
         item = PandasSeriesItem.factory(series)
 
+        # NOTE: isinstance would not work because numpy.array is an instance of list
         assert type(item.series.iloc[0]) is list
 
     @pytest.mark.order(1)
diff --git a/skore/tests/unit/item/test_polars_dataframe_item.py b/skore/tests/unit/item/test_polars_dataframe_item.py
new file mode 100644
index 0000000..059c85f
--- /dev/null
+++ b/skore/tests/unit/item/test_polars_dataframe_item.py
@@ -0,0 +1,48 @@
+import numpy as np
+import pytest
+from polars import DataFrame
+from polars.testing import assert_frame_equal
+from skore.item import ItemTypeError, PolarsDataFrameItem
+from skore.item.polars_dataframe_item import PolarsToJSONError
+
+
+class TestPolarsDataFrameItem:
+    @pytest.fixture(autouse=True)
+    def monkeypatch_datetime(self, monkeypatch, MockDatetime):
+        monkeypatch.setattr("skore.item.item.datetime", MockDatetime)
+
+    def test_factory_exception(self):
+        with pytest.raises(ItemTypeError):
+            PolarsDataFrameItem.factory(None)
+
+    def test_factory(self, mock_nowstr):
+        dataframe = DataFrame([{"key": "value"}])
+
+        dataframe_json = dataframe.write_json()
+
+        item = PolarsDataFrameItem.factory(dataframe)
+
+        assert item.dataframe_json == dataframe_json
+        assert item.created_at == mock_nowstr
+        assert item.updated_at == mock_nowstr
+
+    def test_dataframe(self, mock_nowstr):
+        dataframe = DataFrame([{"key": "value"}])
+
+        dataframe_json = dataframe.write_json()
+
+        item1 = PolarsDataFrameItem.factory(dataframe)
+        item2 = PolarsDataFrameItem(
+            dataframe_json=dataframe_json,
+            created_at=mock_nowstr,
+            updated_at=mock_nowstr,
+        )
+
+        assert_frame_equal(item1.dataframe, dataframe)
+        assert_frame_equal(item2.dataframe, dataframe)
+
+    def test_dataframe_with_complex_object(self, mock_nowstr):
+        dataframe = DataFrame([{"key": np.array([1, 2])}])
+
+        with pytest.raises(PolarsToJSONError):
+            PolarsDataFrameItem.factory(dataframe)
diff --git a/skore/tests/unit/item/test_polars_series_item.py b/skore/tests/unit/item/test_polars_series_item.py
new file mode 100644
index 0000000..118fc36
--- /dev/null
+++ b/skore/tests/unit/item/test_polars_series_item.py
@@ -0,0 +1,50 @@
+import numpy as np
+import pytest
+from polars import Series
+from polars.testing import assert_series_equal
+from skore.item import ItemTypeError, PolarsSeriesItem
+
+
+class TestPolarsSeriesItem:
+    @pytest.fixture(autouse=True)
+    def monkeypatch_datetime(self, monkeypatch, MockDatetime):
+        monkeypatch.setattr("skore.item.item.datetime", MockDatetime)
+
+    def test_factory_exception(self):
+        with pytest.raises(ItemTypeError):
+            PolarsSeriesItem.factory(None)
+
+    @pytest.mark.order(0)
+    def test_factory(self, mock_nowstr):
+        series = Series([0, 1, 2])
+
+        series_json = series.to_frame().write_json()
+
+        item = PolarsSeriesItem.factory(series)
+
+        assert item.series_json == series_json
+        assert item.created_at == mock_nowstr
+        assert item.updated_at == mock_nowstr
+
+    @pytest.mark.order(1)
+    def test_series(self, mock_nowstr):
+        series = Series([0, 1, 2])
+
+        series_json = series.to_frame().write_json()
+
+        item1 = PolarsSeriesItem.factory(series)
+        item2 = PolarsSeriesItem(
+            series_json=series_json,
+            created_at=mock_nowstr,
+            updated_at=mock_nowstr,
+        )
+
+        assert_series_equal(item1.series, series)
+        assert_series_equal(item2.series, series)
+
+    @pytest.mark.order(1)
+    def test_series_with_complex_object(self, mock_nowstr):
+        series = Series([np.array([1, 2])])
+        item = PolarsSeriesItem.factory(series)
+
+        assert_series_equal(item.series, series, check_dtypes=False)
diff --git a/skore/tests/unit/test_project.py b/skore/tests/unit/test_project.py
index 3dea419..9383764 100644
--- a/skore/tests/unit/test_project.py
+++ b/skore/tests/unit/test_project.py
@@ -6,6 +6,8 @@ import numpy
 import numpy.testing
 import pandas
 import pandas.testing
+import polars
+import polars.testing
 import pytest
 from matplotlib import pyplot as plt
 from PIL import Image
@@ -78,6 +80,27 @@ def test_put_pandas_series(in_memory_project):
     pandas.testing.assert_series_equal(in_memory_project.get("pandas_series"), series)
 
 
+def test_put_polars_dataframe(in_memory_project):
+    dataframe = polars.DataFrame(
+        {
+            "A": [1, 2, 3],
+            "B": [4, 5, 6],
+            "C": [7, 8, 9],
+        },
+    )
+
+    in_memory_project.put("polars_dataframe", dataframe)
+    polars.testing.assert_frame_equal(
+        in_memory_project.get("polars_dataframe"), dataframe
+    )
+
+
+def test_put_polars_series(in_memory_project):
+    series = polars.Series("my_series", [0, 1, 2])
+    in_memory_project.put("polars_series", series)
+    polars.testing.assert_series_equal(in_memory_project.get("polars_series"), series)
+
+
 def test_put_numpy_array(in_memory_project):
     # Add a Numpy array
     arr = numpy.array([1, 2, 3, 4, 5])