Repository: scipy/scipy
Pull request #23028: MAINT: create `_lib._array_api_override` to solve circular import
Author: login: lucascolley
Labels: name: scipy._lib
name: maintenance
description: Items related to regular maintenance tasks
name: array types
description: Items related to array API support and input array validation (see gh-18286)
Created: 2025-05-21T14:34:13Z | Merged: 2025-05-23T09:51:24Z

<!-- 
Thanks for contributing a pull request! Please ensure that
your PR satisfies the checklist before submitting:
https://scipy.github.io/devdocs/dev/contributor/development_workflow.html#checklist-before-submitting-a-pr

Also, please name and describe your PR as you would write a
commit message:
https://scipy.github.io/devdocs/dev/contributor/development_workflow.html#writing-the-commit-message.
However, please only include an issue number in the description, not the title,
and please ensure that any code names containing underscores are enclosed in backticks.

Depending on your changes, you can skip CI operations and save time and energy: 
https://scipy.github.io/devdocs/dev/contributor/continuous_integration.html#skipping

Note that we are a team of volunteers; we appreciate your
patience during the review process.

Again, thanks for contributing!
-->

#### Reference issue
<!--Example: Closes gh-WXYZ.-->

#### What does this implement/fix?
<!--Please explain your changes.-->

#### Additional information
<!--Any additional information you think is important.-->



Diff:
diff --git a/scipy/_lib/_array_api.py b/scipy/_lib/_array_api.py
index 5e1170ac51ef..4897f05df21c 100644
--- a/scipy/_lib/_array_api.py
+++ b/scipy/_lib/_array_api.py
@@ -9,10 +9,9 @@
 import contextlib
 import dataclasses
 import functools
-import os
 import textwrap
 
-from collections.abc import Generator, Iterable, Iterator
+from collections.abc import Generator
 from contextlib import contextmanager
 from contextvars import ContextVar
 from types import ModuleType
@@ -21,7 +20,6 @@
 import numpy as np
 import numpy.typing as npt
 
-from scipy._lib import array_api_compat
 from scipy._lib.array_api_compat import (
     is_array_api_obj,
     is_lazy_array,
@@ -35,13 +33,17 @@
     is_dask_namespace as is_dask,
     is_array_api_strict_namespace as is_array_api_strict
 )
-from scipy._lib._sparse import issparse
+from scipy._lib.array_api_extra.testing import lazy_xp_function
+from scipy._lib._array_api_override import (
+    array_namespace, SCIPY_ARRAY_API, SCIPY_DEVICE
+)
 from scipy._lib._docscrape import FunctionDoc
 
 __all__ = [
     '_asarray', 'array_namespace', 'assert_almost_equal', 'assert_array_almost_equal',
     'default_xp', 'eager_warns', 'is_lazy_array', 'is_marray',
     'is_array_api_strict', 'is_complex', 'is_cupy', 'is_jax', 'is_numpy', 'is_torch',
+    'np_compat',
     'SCIPY_ARRAY_API', 'SCIPY_DEVICE', 'scipy_namespace_for',
     'xp_assert_close', 'xp_assert_equal', 'xp_assert_less',
     'xp_copy', 'xp_device', 'xp_ravel', 'xp_size',
@@ -50,123 +52,16 @@
 ]
 
 
-# To enable array API and strict array-like input validation
-SCIPY_ARRAY_API: str | bool = os.environ.get("SCIPY_ARRAY_API", False)
-# To control the default device - for use in the test suite only
-SCIPY_DEVICE = os.environ.get("SCIPY_DEVICE", "cpu")
-
-_GLOBAL_CONFIG = {
-    "SCIPY_ARRAY_API": SCIPY_ARRAY_API,
-    "SCIPY_DEVICE": SCIPY_DEVICE,
-}
-
-
 Array: TypeAlias = Any  # To be changed to a Protocol later (see array-api#589)
 ArrayLike: TypeAlias = Array | npt.ArrayLike
 
 
-def _compliance_scipy(arrays: Iterable[ArrayLike]) -> Iterator[Array]:
-    """Raise exceptions on known-bad subclasses. Discard 0-dimensional ArrayLikes
-    and convert 1+-dimensional ArrayLikes to numpy.
-
-    The following subclasses are not supported and raise and error:
-    - `numpy.ma.MaskedArray`
-    - `numpy.matrix`
-    - NumPy arrays which do not have a boolean or numerical dtype
-    - Any array-like which is neither array API compatible nor coercible by NumPy
-    - Any array-like which is coerced by NumPy to an unsupported dtype
-    """
-    for array in arrays:
-        if array is None:
-            continue
-
-        # this comes from `_util._asarray_validated`
-        if issparse(array):
-            msg = ('Sparse arrays/matrices are not supported by this function. '
-                   'Perhaps one of the `scipy.sparse.linalg` functions '
-                   'would work instead.')
-            raise ValueError(msg)
-
-        if isinstance(array, np.ma.MaskedArray):
-            raise TypeError("Inputs of type `numpy.ma.MaskedArray` are not supported.")
-
-        if isinstance(array, np.matrix):
-            raise TypeError("Inputs of type `numpy.matrix` are not supported.")
-
-        if isinstance(array, np.ndarray | np.generic):
-            dtype = array.dtype
-            if not (np.issubdtype(dtype, np.number) or np.issubdtype(dtype, np.bool_)):
-                raise TypeError(f"An argument has dtype `{dtype!r}`; "
-                                f"only boolean and numerical dtypes are supported.")
-
-        if is_array_api_obj(array):
-            yield array
-        else:
-            try:
-                array = np.asanyarray(array)
-            except TypeError:
-                raise TypeError("An argument is neither array API compatible nor "
-                                "coercible by NumPy.")
-            dtype = array.dtype
-            if not (np.issubdtype(dtype, np.number) or np.issubdtype(dtype, np.bool_)):
-                message = (
-                    f"An argument was coerced to an unsupported dtype `{dtype!r}`; "
-                    f"only boolean and numerical dtypes are supported."
-                )
-                raise TypeError(message)
-            # Ignore 0-dimensional arrays, coherently with array-api-compat.
-            # Raise if there are 1+-dimensional array-likes mixed with non-numpy
-            # Array API objects.
-            if array.ndim:
-                yield array
-
-
 def _check_finite(array: Array, xp: ModuleType) -> None:
     """Check for NaNs or Infs."""
     if not xp.all(xp.isfinite(array)):
         msg = "array must not contain infs or NaNs"
         raise ValueError(msg)
 
-
-def array_namespace(*arrays: Array) -> ModuleType:
-    """Get the array API compatible namespace for the arrays xs.
-
-    Parameters
-    ----------
-    *arrays : sequence of array_like
-        Arrays used to infer the common namespace.
-
-    Returns
-    -------
-    namespace : module
-        Common namespace.
-
-    Notes
-    -----
-    Thin wrapper around `array_api_compat.array_namespace`.
-
-    1. Check for the global switch: SCIPY_ARRAY_API. This can also be accessed
-       dynamically through ``_GLOBAL_CONFIG['SCIPY_ARRAY_API']``.
-    2. `_compliance_scipy` raise exceptions on known-bad subclasses. See
-       its definition for more details.
-
-    When the global switch is False, it defaults to the `numpy` namespace.
-    In that case, there is no compliance check. This is a convenience to
-    ease the adoption. Otherwise, arrays must comply with the new rules.
-    """
-    if not _GLOBAL_CONFIG["SCIPY_ARRAY_API"]:
-        # here we could wrap the namespace if needed
-        return np_compat
-
-    api_arrays = list(_compliance_scipy(arrays))
-    # In case of a mix of array API compliant arrays and scalars, return
-    # the array API namespace. If there are only ArrayLikes (e.g. lists),
-    # return NumPy (wrapped by array-api-compat).
-    if api_arrays:
-        return array_api_compat.array_namespace(*api_arrays)
-    return np_compat
-
-
 def _asarray(
         array: ArrayLike,
         dtype: Any = None,
@@ -826,7 +721,6 @@ def _make_xp_pytest_marks(*funcs, capabilities_table=None):
     capabilities_table = (xp_capabilities_table if capabilities_table is None
                           else capabilities_table)
     import pytest
-    from scipy._lib.array_api_extra.testing import lazy_xp_function
 
     marks = []
     for func in funcs:
diff --git a/scipy/_lib/_array_api_compat_vendor.py b/scipy/_lib/_array_api_compat_vendor.py
index 85c720f57a90..64d844ff4641 100644
--- a/scipy/_lib/_array_api_compat_vendor.py
+++ b/scipy/_lib/_array_api_compat_vendor.py
@@ -3,7 +3,7 @@
 # to override functions of array_api_compat.
 
 from .array_api_compat import *  # noqa: F403
-from ._array_api import array_namespace as scipy_array_namespace
+from ._array_api_override import array_namespace as scipy_array_namespace
 
 # overrides array_api_compat.array_namespace inside array-api-extra
 array_namespace = scipy_array_namespace  # type: ignore[assignment]
diff --git a/scipy/_lib/_array_api_override.py b/scipy/_lib/_array_api_override.py
new file mode 100644
index 000000000000..7eb204099e73
--- /dev/null
+++ b/scipy/_lib/_array_api_override.py
@@ -0,0 +1,122 @@
+"""
+Override functions from array_api_compat, for use by array-api-extra
+and internally.
+
+See also _array_api_compat_vendor.py
+"""
+import os
+
+from collections.abc import Iterable, Iterator
+from types import ModuleType
+from typing import Any, TypeAlias
+
+import numpy as np
+import numpy.typing as npt
+
+from scipy._lib import array_api_compat
+import scipy._lib.array_api_compat.numpy as np_compat
+from scipy._lib.array_api_compat import is_array_api_obj
+from scipy._lib._sparse import issparse
+
+
+Array: TypeAlias = Any  # To be changed to a Protocol later (see array-api#589)
+ArrayLike: TypeAlias = Array | npt.ArrayLike
+
+# To enable array API and strict array-like input validation
+SCIPY_ARRAY_API: str | bool = os.environ.get("SCIPY_ARRAY_API", False)
+# To control the default device - for use in the test suite only
+SCIPY_DEVICE = os.environ.get("SCIPY_DEVICE", "cpu")
+
+
+def _compliance_scipy(arrays: Iterable[ArrayLike]) -> Iterator[Array]:
+    """Raise exceptions on known-bad subclasses. Discard 0-dimensional ArrayLikes
+    and convert 1+-dimensional ArrayLikes to numpy.
+
+    The following subclasses are not supported and raise and error:
+    - `numpy.ma.MaskedArray`
+    - `numpy.matrix`
+    - NumPy arrays which do not have a boolean or numerical dtype
+    - Any array-like which is neither array API compatible nor coercible by NumPy
+    - Any array-like which is coerced by NumPy to an unsupported dtype
+    """
+    for array in arrays:
+        if array is None:
+            continue
+
+        # this comes from `_util._asarray_validated`
+        if issparse(array):
+            msg = ('Sparse arrays/matrices are not supported by this function. '
+                   'Perhaps one of the `scipy.sparse.linalg` functions '
+                   'would work instead.')
+            raise ValueError(msg)
+
+        if isinstance(array, np.ma.MaskedArray):
+            raise TypeError("Inputs of type `numpy.ma.MaskedArray` are not supported.")
+
+        if isinstance(array, np.matrix):
+            raise TypeError("Inputs of type `numpy.matrix` are not supported.")
+
+        if isinstance(array, np.ndarray | np.generic):
+            dtype = array.dtype
+            if not (np.issubdtype(dtype, np.number) or np.issubdtype(dtype, np.bool_)):
+                raise TypeError(f"An argument has dtype `{dtype!r}`; "
+                                f"only boolean and numerical dtypes are supported.")
+
+        if is_array_api_obj(array):
+            yield array
+        else:
+            try:
+                array = np.asanyarray(array)
+            except TypeError:
+                raise TypeError("An argument is neither array API compatible nor "
+                                "coercible by NumPy.")
+            dtype = array.dtype
+            if not (np.issubdtype(dtype, np.number) or np.issubdtype(dtype, np.bool_)):
+                message = (
+                    f"An argument was coerced to an unsupported dtype `{dtype!r}`; "
+                    f"only boolean and numerical dtypes are supported."
+                )
+                raise TypeError(message)
+            # Ignore 0-dimensional arrays, coherently with array-api-compat.
+            # Raise if there are 1+-dimensional array-likes mixed with non-numpy
+            # Array API objects.
+            if array.ndim:
+                yield array
+
+
+def array_namespace(*arrays: Array) -> ModuleType:
+    """Get the array API compatible namespace for the arrays xs.
+
+    Parameters
+    ----------
+    *arrays : sequence of array_like
+        Arrays used to infer the common namespace.
+
+    Returns
+    -------
+    namespace : module
+        Common namespace.
+
+    Notes
+    -----
+    Thin wrapper around `array_api_compat.array_namespace`.
+
+    1. Check for the global switch: SCIPY_ARRAY_API.
+    2. `_compliance_scipy` raise exceptions on known-bad subclasses. See
+       its definition for more details.
+
+    When the global switch is False, it defaults to the `numpy` namespace.
+    In that case, there is no compliance check. This is a convenience to
+    ease the adoption. Otherwise, arrays must comply with the new rules.
+    """
+    if not SCIPY_ARRAY_API:
+        # here we could wrap the namespace if needed
+        return np_compat
+
+    api_arrays = list(_compliance_scipy(arrays))
+    # In case of a mix of array API compliant arrays and scalars, return
+    # the array API namespace. If there are only ArrayLikes (e.g. lists),
+    # return NumPy (wrapped by array-api-compat).
+    if api_arrays:
+        return array_api_compat.array_namespace(*api_arrays)
+    return np_compat
diff --git a/scipy/_lib/meson.build b/scipy/_lib/meson.build
index fcfb2a8f98e5..56ef7393f91c 100644
--- a/scipy/_lib/meson.build
+++ b/scipy/_lib/meson.build
@@ -112,6 +112,7 @@ python_sources = [
   '_array_api.py',
   '_array_api_compat_vendor.py',
   '_array_api_no_0d.py',
+  '_array_api_override.py',
   '_bunch.py',
   '_ccallback.py',
   '_disjoint_set.py',
diff --git a/scipy/_lib/tests/test_array_api.py b/scipy/_lib/tests/test_array_api.py
index f7f323adaa47..1a633a968ef5 100644
--- a/scipy/_lib/tests/test_array_api.py
+++ b/scipy/_lib/tests/test_array_api.py
@@ -4,7 +4,7 @@
 import pytest
 
 from scipy._lib._array_api import (
-    _GLOBAL_CONFIG, array_namespace, _asarray, xp_copy, xp_assert_equal, is_numpy,
+    SCIPY_ARRAY_API, array_namespace, _asarray, xp_copy, xp_assert_equal, is_numpy,
     np_compat, xp_default_dtype, xp_result_type, is_torch
 )
 from scipy._lib import array_api_extra as xpx
@@ -16,7 +16,7 @@
 lazy_xp_function(xp_copy)
 
 
-@pytest.mark.skipif(not _GLOBAL_CONFIG["SCIPY_ARRAY_API"],
+@pytest.mark.skipif(not SCIPY_ARRAY_API,
         reason="Array API test; set environment variable SCIPY_ARRAY_API=1 to run it")
 class TestArrayAPI:
 
@@ -25,11 +25,6 @@ def test_array_namespace(self):
         xp = array_namespace(x, y)
         assert 'array_api_compat.numpy' in xp.__name__
 
-        _GLOBAL_CONFIG["SCIPY_ARRAY_API"] = False
-        xp = array_namespace(x, y)
-        assert 'array_api_compat.numpy' in xp.__name__
-        _GLOBAL_CONFIG["SCIPY_ARRAY_API"] = True
-
     def test_asarray(self, xp):
         x, y = _asarray([0, 1, 2], xp=xp), _asarray(np.arange(3), xp=xp)
         ref = xp.asarray([0, 1, 2])


Reviews:
user: login: lucascolley
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-21T14:36:21Z
user: login: crusaderky
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-23T07:55:25Z
user: login: crusaderky
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-23T07:56:48Z

Review comments:
diff_hunk: @@ -16,7 +16,7 @@
 lazy_xp_function(xp_copy)
 
 
-@pytest.mark.skipif(not _GLOBAL_CONFIG["SCIPY_ARRAY_API"],
+@pytest.mark.skipif(not SCIPY_ARRAY_API,
path: scipy/_lib/tests/test_array_api.py
user: login: lucascolley
body: I've taken the liberty of removing `_GLOBAL_CONFIG` as it was unused—see gh-21532 for context
created_at: 2025-05-21T14:36:21Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,124 @@
+"""
+Override functions from array_api_compat, for use by array-api-extra
+and internally.
+
+See also _array_api_compat_vendor.py
+"""
+import os
+
+from collections.abc import Iterable, Iterator
+from types import ModuleType
+from typing import Any, TypeAlias
+
+import numpy as np
+import numpy.typing as npt
+
+from scipy._lib import array_api_compat
+from scipy._lib.array_api_compat import (
+    is_array_api_obj,
+    numpy as np_compat,
+)
path: scipy/_lib/_array_api_override.py
user: login: crusaderky
body: ```suggestion
import scipy._lib.array_api_compat.numpy as np_compat
from scipy._lib.array_api_compat import is_array_api_obj
```
created_at: 2025-05-23T07:52:20Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,124 @@
+"""
+Override functions from array_api_compat, for use by array-api-extra
+and internally.
+
+See also _array_api_compat_vendor.py
+"""
+import os
+
+from collections.abc import Iterable, Iterator
+from types import ModuleType
+from typing import Any, TypeAlias
+
+import numpy as np
+import numpy.typing as npt
+
+from scipy._lib import array_api_compat
+from scipy._lib.array_api_compat import (
+    is_array_api_obj,
+    numpy as np_compat,
+)
path: scipy/_lib/_array_api_override.py
user: login: crusaderky
body: you could also consider just using the `_compat` namespace like we already do in xpx
created_at: 2025-05-23T07:56:48Z
author_association: CONTRIBUTOR

PR comments:
user: login: ev-br
created_at: 2025-05-21T20:11:08Z
body: Thanks for the ping Lucas. 

Got to admit, I'm confused. If this PR is in response to https://github.com/scipy/scipy/pull/23020#issuecomment-2897476768, then something does not square up:

What I saw in gh-23020 was that https://github.com/scipy/scipy/pull/23020/commits/8a0426020bc2a788f65964f4103e37ec5503df70 failed across the CI, with diagnostics apparently hinting at an import cycle due to importing `array_api_extra` from `_lib._array_api`.

In this PR, you are importing from `array_api_extra.testing` from `_lib._array_api`, so either it should fail too, or I misdiagnosed the problem in gh-23020? And then there is no need to split `_array_api_override`, apparently.


  
author_association: MEMBER
user: login: lucascolley
created_at: 2025-05-21T20:24:15Z
body: I think you diagnosed it correctly, or at least got pretty close! What you may be missing is that `array_api_extra` imports from `_array_api_compat_vendor` when vendored like this. https://github.com/data-apis/array-api-extra/blob/27fbd9c6262ed7c547f17e247e9e21eaddf89576/src/array_api_extra/_lib/_utils/_compat.py#L6-L8

Before this PR:
`_array_api` -> `array_api_extra` -> `_array_api_compat_vendor` -> `_array_api` (for `array_namespace`) ⚠️ circle

After this PR:
`_array_api` -> `array_api_extra` -> `_array_api_compat_vendor` -> `_array_api_override` ✅ no circle
author_association: MEMBER
user: login: ev-br
created_at: 2025-05-21T20:41:35Z
body: Oh, I see.

TBH, I've a nagging feeling that the low-tech solution of https://github.com/scipy/scipy/pull/23020/commits/368748290e13269af590f2369ca4718e6b5ec1aa is a bit simpler :-).
On the other hand, if this all were done like this PR does, the issue would not have existed at all.
My preference for a one-liner is mild, and I'm going to go with the flow. If this PR lands, I'll update gh-23020; if it does not, I'll keep the local import.

The only tangentially related thing is that maybe, just maybe, we should take a long hard look at the whole state of `_lib.<array-api-related-libs-and-modules>. It does seem a bit complicated. Not that I have immediate specific suggestions though. 
author_association: MEMBER
user: login: lucascolley
created_at: 2025-05-23T09:51:30Z
body: thanks both
author_association: MEMBER