Repository: narwhals-dev/narwhals
Pull request #2620 (2025-05-26 20:19:43)

refactor: Simplify `Implementation.to_native_namespace`
## What type of PR is this? (check all applicable)

- [x] 💾 Refactor
- [ ] ✨ Feature
- [ ] 🐛 Bug Fix
- [ ] 🔧 Optimization
- [ ] 📝 Documentation
- [ ] ✅ Test
- [ ] 🐳 Other

## Related issues

- Another piece of #2492

## Checklist

- [x] Code follows style guide (ruff)
- [x] Tests added
- [ ] Documented the changes

## If you have comments or can explain your changes, please do so below

Not sure how likable this is (see [discord](https://discord.com/channels/1235257048170762310/1276640597284884490/1379466092099145821))

dangotbanned: > > The phrasing of this tickled me 😂
> 
> I personally don't mind but who knows you guys don't have strong opinions on these hacks 😂
> 
> > @FBruzzesi what if we solved this and closed a related issue in one go?
> > 
> > * [Make `import_optional_dependency` util #1940 (comment)](https://github.com/narwhals-dev/narwhals/issues/1940#issuecomment-2780636862)
> 
> Sure, the polars implementation seems easy to adapt as well. What I am not too keen on is that `MIN_VERSIONS` is defined based on implementation, so that feels a bit circular, but definitly fixable 😂
> 
> > We'd also be able to get typing working (at least for the individual cases)
> 
> To my knowledge, `ModuleType` is not a generic, so how would that work? i.e. what should the following simplified version return?
> 
> ```python
> @overload
> def import_optional(module_name: Literal["pandas"]) -> ?
>     return __import__(module_name)
> ```

At the end of (https://github.com/narwhals-dev/narwhals/issues/1940#issuecomment-2780636862) ...

So you leave each `import_*` function return **un-annotated**, and let the the `Module(...)` part get inferred 😎 

> ![Image](https://github.com/user-attachments/assets/1085eb4b-8bc5-49b8-bacc-f4bd25961565)


Then you can have a mapping like:


```py
# or replace `Implementation` with the literal you resolve in `Implementation.value`
_IMPORT_FUNCTIONS: Mapping[Implementation, Callable[..., ModuleType]] = {
    Implementation.PANDAS: import_pandas,
    Implementation.POLARS: import_polars,
    ...: ...,
}
```
dangotbanned: The only way I've found to fix this was by removing the `sys.modules` monkeypatching.
But I don't really like that as it means `find_spec` isn't being tested against the case where the module hasn't been imported yet 🤔

https://github.com/narwhals-dev/narwhals/actions/runs/15491689078/job/43618476234?pr=2620
FBruzzesi: @MarcoGorelli @dangotbanned I tried to implement the idea I had mentioned in discord in https://github.com/narwhals-dev/narwhals/pull/2620/commits/96540c329c60c489e20404a7a8ad41d118f1a639, yet apparently everything breaks apart with ibis, so I am happy to revert unless you guys have some other ideas.

The reason to use `from importlib.metadata import version` was mostly because `sqlframe` does not have `__version__` that we can access (that's what polars does).
However, 

- `import_module("ibis")` -> ok
- `import_module("ibis-framework")` -> errors
- `version("ibis")` -> errors
- `version("ibis-framework")` -> ok

dangotbanned: @FBruzzesi here's a platter of thoughts 😄 

## This PR

### `importlib`
There's a very new [PEP 794 – Import Name Metadata](https://peps.python.org/pep-0794/) that seems relevant to the issue you had here

> * `version("ibis")` -> errors
> * `import_module("ibis-framework")` -> errors

`ibis-framework` is the name of the **distribution**, whereas `ibis` is the name of the **package**.
They're *usually* the same, but sadly not in this case 😞 

https://github.com/python/cpython/blob/24069fbca861a5904ee7718469919e84828f22e7/Lib/importlib/metadata/__init__.py#L980-L987
https://github.com/python/cpython/blob/24069fbca861a5904ee7718469919e84828f22e7/Lib/importlib/__init__.py#L71-L88

### `__version__`
> The reason to use `from importlib.metadata import version` was mostly because `sqlframe` does not have `__version__` that we can access (that's what polars does).

So this detail is already handled in here, you might notice it starts with a call to `to_native_namespace`

<details><summary><code>Implementation._backend_version</code></summary>
<p>

https://github.com/narwhals-dev/narwhals/blob/deaaf76b2a63dd6ad37d20714564712101b4a9f0/narwhals/utils.py#L617-L635

</p>
</details> 


## Minimum versions (*generally*)
I totally forgot to follow up on (https://discord.com/channels/1235257048170762310/1276640597284884490/1379470754051002389)
> I'll continue this over on that other issue so you've got something to come back to

We have two other methods for enforcing version constraints:
- [`utils.validate_backend_version`](https://github.com/narwhals-dev/narwhals/blob/deaaf76b2a63dd6ad37d20714564712101b4a9f0/narwhals/utils.py#L653-L658)
  - For the absolute minimum
- [`@requires.backend_version`](https://github.com/narwhals-dev/narwhals/blob/deaaf76b2a63dd6ad37d20714564712101b4a9f0/narwhals/utils.py#L1838-L1894)
  - For method-specific support

*If* there is another use-case these don't cover, I think it would make the most sense to **improve** those tools.
This PR would be adding a third way of handling package versions - leading to 3 different error messages for the same issue 😔
FBruzzesi: @dangotbanned I am quite happy with the state of this now. Let me know your thoughts whenever you have the time 🤞🏼
dangotbanned: > What's up with pyarrow nightly 🥲

Same!

I'm seeing that recently in #2572 as well

Patch:
diff --git a/narwhals/_utils.py b/narwhals/_utils.py
index 3d2c89ae..6d948260 100644
--- a/narwhals/_utils.py
+++ b/narwhals/_utils.py
@@ -4,7 +4,7 @@ import os
 import re
 from datetime import timezone
 from enum import Enum, auto
-from functools import wraps
+from functools import lru_cache, wraps
 from importlib.util import find_spec
 from inspect import getattr_static, getdoc
 from secrets import token_hex
@@ -17,6 +17,7 @@ from typing import (
     Iterable,
     Iterator,
     Literal,
+    Mapping,
     Protocol,
     Sequence,
     TypeVar,
@@ -30,7 +31,6 @@ from narwhals._enum import NoAutoEnum
 from narwhals._typing_compat import deprecated
 from narwhals.dependencies import (
     get_cudf,
-    get_dask,
     get_dask_dataframe,
     get_duckdb,
     get_ibis,
@@ -38,7 +38,6 @@ from narwhals.dependencies import (
     get_pandas,
     get_polars,
     get_pyarrow,
-    get_pyspark,
     get_pyspark_connect,
     get_pyspark_sql,
     get_sqlframe,
@@ -344,63 +343,20 @@ class Implementation(NoAutoEnum):
             else cls.from_native_namespace(backend)
         )
 
-    def to_native_namespace(self) -> ModuleType:  # noqa: C901, PLR0911
+    def to_native_namespace(self) -> ModuleType:
         """Return the native namespace module corresponding to Implementation.
 
         Returns:
             Native module.
         """
-        if self is Implementation.PANDAS:
-            import pandas as pd  # ignore-banned-import
-
-            return pd
-        if self is Implementation.MODIN:
-            import modin.pandas
-
-            return modin.pandas
-        if self is Implementation.CUDF:  # pragma: no cover
-            import cudf  # ignore-banned-import
-
-            return cudf
-        if self is Implementation.PYARROW:
-            import pyarrow as pa  # ignore-banned-import
-
-            return pa
-        if self is Implementation.PYSPARK:  # pragma: no cover
-            import pyspark.sql
-
-            return pyspark.sql
-        if self is Implementation.POLARS:
-            import polars as pl  # ignore-banned-import
-
-            return pl
-        if self is Implementation.DASK:
-            import dask.dataframe  # ignore-banned-import
-
-            return dask.dataframe
-
-        if self is Implementation.DUCKDB:
-            import duckdb  # ignore-banned-import
-
-            return duckdb
-
-        if self is Implementation.SQLFRAME:
-            import sqlframe  # ignore-banned-import
-
-            return sqlframe
-
-        if self is Implementation.IBIS:
-            import ibis  # ignore-banned-import
-
-            return ibis
-
-        if self is Implementation.PYSPARK_CONNECT:  # pragma: no cover
-            import pyspark.sql.connect  # ignore-banned-import
+        if self is Implementation.UNKNOWN:
+            msg = "Cannot return native namespace from UNKNOWN Implementation"
+            raise AssertionError(msg)
 
-            return pyspark.sql.connect
+        validate_backend_version(self, self._backend_version())
 
-        msg = "Not supported Implementation"  # pragma: no cover
-        raise AssertionError(msg)
+        module_name = _IMPLEMENTATION_TO_MODULE_NAME.get(self, self.value)
+        return _import_native_namespace(module_name)
 
     def is_pandas(self) -> bool:
         """Return whether implementation is pandas.
@@ -615,27 +571,38 @@ class Implementation(NoAutoEnum):
         return self is Implementation.SQLFRAME  # pragma: no cover
 
     def _backend_version(self) -> tuple[int, ...]:
-        native = self.to_native_namespace()
-        into_version: Any
-        if self not in {
-            Implementation.PYSPARK,
-            Implementation.PYSPARK_CONNECT,
-            Implementation.DASK,
-            Implementation.SQLFRAME,
-        }:
-            into_version = native
-        elif self in {Implementation.PYSPARK, Implementation.PYSPARK_CONNECT}:
-            into_version = get_pyspark()  # pragma: no cover
-        elif self is Implementation.DASK:
-            into_version = get_dask()
-        else:
+        """Returns backend version.
+
+        As a biproduct of loading the native namespace, we also store it as an attribute
+        under the `_native_namespace` name.
+        """
+        if self is Implementation.UNKNOWN:  # pragma: no cover
+            msg = "Cannot return backend version from UNKNOWN Implementation"
+            raise AssertionError(msg)
+
+        module_name = _IMPLEMENTATION_TO_MODULE_NAME.get(self, self.value)
+        native_namespace = _import_native_namespace(module_name)
+
+        into_version: ModuleType | str
+        if self.is_sqlframe():
             import sqlframe._version
 
             into_version = sqlframe._version
-        return parse_version(into_version)
+        elif self.is_pyspark() or self.is_pyspark_connect():  # pragma: no cover
+            import pyspark  # ignore-banned-import
+
+            into_version = pyspark
+        elif self.is_dask():
+            import dask  # ignore-banned-import
 
+            into_version = dask
+        else:
+            into_version = native_namespace
+
+        return parse_version(version=into_version)
 
-MIN_VERSIONS: dict[Implementation, tuple[int, ...]] = {
+
+MIN_VERSIONS: Mapping[Implementation, tuple[int, ...]] = {
     Implementation.PANDAS: (0, 25, 3),
     Implementation.MODIN: (0, 25, 3),
     Implementation.CUDF: (24, 10),
@@ -649,6 +616,14 @@ MIN_VERSIONS: dict[Implementation, tuple[int, ...]] = {
     Implementation.SQLFRAME: (3, 22, 0),
 }
 
+_IMPLEMENTATION_TO_MODULE_NAME: Mapping[Implementation, str] = {
+    Implementation.DASK: "dask.dataframe",
+    Implementation.MODIN: "modin.pandas",
+    Implementation.PYSPARK: "pyspark.sql",
+    Implementation.PYSPARK_CONNECT: "pyspark.sql.connect",
+}
+"""Stores non default mapping from Implementation to module name"""
+
 
 def validate_backend_version(
     implementation: Implementation, backend_version: tuple[int, ...]
@@ -658,6 +633,13 @@ def validate_backend_version(
         raise ValueError(msg)
 
 
+@lru_cache(maxsize=16)
+def _import_native_namespace(module_name: str) -> ModuleType:
+    from importlib import import_module
+
+    return import_module(module_name)
+
+
 def remove_prefix(text: str, prefix: str) -> str:  # pragma: no cover
     if text.startswith(prefix):
         return text[len(prefix) :]
diff --git a/pyproject.toml b/pyproject.toml
index bd76a54b..554d107e 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -284,7 +284,7 @@ exclude_also = [
   'if "modin" in str\(constructor',
   'if "pyspark" in str\(constructor',
   'if "pyspark_connect" in str\(constructor',
-  'pytest.skip\('
+  'pytest.skip\(',
 ]
 
 [tool.mypy]


Test patch:
diff --git a/tests/dependencies/imports_test.py b/tests/dependencies/imports_test.py
new file mode 100644
index 00000000..58ccb02e
--- /dev/null
+++ b/tests/dependencies/imports_test.py
@@ -0,0 +1,54 @@
+from __future__ import annotations
+
+from importlib.util import find_spec
+from types import ModuleType
+
+import pytest
+
+from narwhals.utils import Implementation
+
+implementations = [
+    Implementation.CUDF,
+    Implementation.DASK,
+    Implementation.DUCKDB,
+    Implementation.IBIS,
+    Implementation.MODIN,
+    Implementation.PANDAS,
+    Implementation.POLARS,
+    Implementation.PYARROW,
+    Implementation.PYSPARK,
+    Implementation.SQLFRAME,
+]
+
+
+@pytest.mark.parametrize("impl", implementations)
+def test_to_native_namespace(impl: Implementation) -> None:
+    if not find_spec(impl.value):
+        reason = f"{impl.value} not installed"
+        pytest.skip(reason=reason)
+
+    assert isinstance(impl.to_native_namespace(), ModuleType)
+
+
+@pytest.mark.parametrize("impl", implementations)
+def test_to_native_namespace_min_version(
+    monkeypatch: pytest.MonkeyPatch, impl: Implementation
+) -> None:
+    if not find_spec(impl.value):
+        reason = f"{impl.value} not installed"
+        pytest.skip(reason=reason)
+
+    monkeypatch.setattr(
+        "narwhals._utils.Implementation._backend_version", lambda _: (0, 0, 1)
+    )
+
+    with pytest.raises(ValueError, match="Minimum version"):
+        impl.to_native_namespace()
+
+
+def test_to_native_namespace_unknown() -> None:
+    impl = Implementation.UNKNOWN
+    with pytest.raises(
+        AssertionError, match="Cannot return native namespace from UNKNOWN Implementation"
+    ):
+        impl.to_native_namespace()