Repository: scipy/scipy
Pull request #22671: MAINT: enforce modularity with `tach`
Author: login: lucascolley
Labels: name: scipy.linalg
name: scipy._lib
name: maintenance
description: Items related to regular maintenance tasks
Created: 2025-03-12T00:54:25Z | Merged: 2025-03-21T10:15:18Z

Related to discussion in gh-22382, the bulk of the work here in setting up the initial `tach` config was done by @rgommers there.

This PR follows from a conversation on the Scientific Python Discord @mdhaber. Ideally we can enforce this in CI to stop new dependencies between submodules from being created accidentally.

Here we remove the dependency of `_lib` on `linalg`.


Diff:
diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml
index 90511fb347c4..a56bd450bb4b 100644
--- a/.github/workflows/lint.yml
+++ b/.github/workflows/lint.yml
@@ -39,7 +39,7 @@ jobs:
 
     - name: Install Python packages
       run: |
-        python -m pip install ruff cython-lint packaging
+        python -m pip install ruff cython-lint packaging tach
 
     - name: Lint
       run: |
@@ -52,3 +52,7 @@ jobs:
       shell: bash
       run: |
         python tools/check_python_h_first.py 
+
+    - name: Check module interdependencies
+      run: |
+        tach check --exact
diff --git a/scipy/_lib/meson.build b/scipy/_lib/meson.build
index 400bd6d27e07..e4a00f813d54 100644
--- a/scipy/_lib/meson.build
+++ b/scipy/_lib/meson.build
@@ -120,7 +120,6 @@ python_sources = [
   '_disjoint_set.py',
   '_docscrape.py',
   '_elementwise_iterative_method.py',
-  '_finite_differences.py',
   '_gcutils.py',
   '_pep440.py',
   '_sparse.py',
diff --git a/scipy/stats/_distn_infrastructure.py b/scipy/stats/_distn_infrastructure.py
index d3addcffce04..6e33ae116af5 100644
--- a/scipy/stats/_distn_infrastructure.py
+++ b/scipy/stats/_distn_infrastructure.py
@@ -26,7 +26,7 @@
 from scipy import integrate
 
 # to approximate the pdf of a continuous distribution given its cdf
-from scipy._lib._finite_differences import _derivative
+from scipy.stats._finite_differences import _derivative
 
 # for scipy.stats.entropy. Attempts to import just that function or file
 # have cause import problems
diff --git a/scipy/_lib/_finite_differences.py b/scipy/stats/_finite_differences.py
similarity index 99%
rename from scipy/_lib/_finite_differences.py
rename to scipy/stats/_finite_differences.py
index 506057b48b3f..6ec91654c4e9 100644
--- a/scipy/_lib/_finite_differences.py
+++ b/scipy/stats/_finite_differences.py
@@ -1,4 +1,5 @@
 from numpy import arange, newaxis, hstack, prod, array
+from scipy import linalg
 
 
 def _central_diff_weights(Np, ndiv=1):
@@ -54,7 +55,6 @@ def _central_diff_weights(Np, ndiv=1):
         )
     if Np % 2 == 0:
         raise ValueError("The number of points must be odd.")
-    from scipy import linalg
 
     ho = Np >> 1
     x = arange(-ho, ho + 1.0)
diff --git a/scipy/stats/_ksstats.py b/scipy/stats/_ksstats.py
index a25d07e3233b..8996745601ed 100644
--- a/scipy/stats/_ksstats.py
+++ b/scipy/stats/_ksstats.py
@@ -68,7 +68,7 @@
 import numpy as np
 import scipy.special
 import scipy.special._ufuncs as scu
-from scipy._lib._finite_differences import _derivative
+from scipy.stats._finite_differences import _derivative
 
 _E128 = 128
 _EP128 = np.ldexp(np.longdouble(1), _E128)
diff --git a/scipy/stats/meson.build b/scipy/stats/meson.build
index b127a8baeafb..e7bbc86cd830 100644
--- a/scipy/stats/meson.build
+++ b/scipy/stats/meson.build
@@ -120,6 +120,7 @@ py3.install_sources([
     '_distribution_infrastructure.py',
     '_distr_params.py',
     '_entropy.py',
+    '_finite_differences.py',
     '_fit.py',
     '_hypotests.py',
     '_kde.py',
diff --git a/tach.toml b/tach.toml
new file mode 100644
index 000000000000..0fdbb5fc024e
--- /dev/null
+++ b/tach.toml
@@ -0,0 +1,218 @@
+interfaces = []
+exclude = [
+    "**/*__pycache__",
+    "**/*egg-info",
+    "**/docs",
+    "**/tests",
+    "**/venv",
+    "benchmarks",
+    "build*",
+    "doc",
+    "dev.py",
+    "subprojects",
+    "tools",
+    "scipy/_build_utils",
+    "scipy/_lib/cobyqa/doc",
+    "scipy/_lib/cobyqa/examples",
+    "scipy/_lib/cobyqa/pyproject.toml",
+    "scipy/conftest.py",
+    "scipy/special/_precompute/*",
+]
+source_roots = [
+    ".",
+]
+
+[[modules]]
+path = "scipy"
+depends_on = ["scipy._lib"]
+
+[[modules]]
+path = "scipy._lib"
+depends_on = ["scipy"]
+
+[[modules]]
+path = "scipy._lib.cobyqa"
+unchecked = true
+
+[[modules]]
+path = "scipy.cluster"
+depends_on = ["scipy.cluster.vq", "scipy.cluster.hierarchy", "scipy._lib"]
+
+[[modules]]
+path = "scipy.cluster.hierarchy"
+depends_on = ["scipy._lib", "scipy.spatial.distance", "scipy.cluster"]
+
+[[modules]]
+path = "scipy.cluster.vq"
+depends_on = ["scipy.cluster", "scipy._lib", "scipy.spatial.distance"]
+
+[[modules]]
+path = "scipy.constants"
+depends_on = ["scipy._lib"]
+
+[[modules]]
+path = "scipy.datasets"
+depends_on = ["scipy._lib"]
+
+[[modules]]
+path = "scipy.differentiate"
+depends_on = ["scipy._lib"]
+
+[[modules]]
+path = "scipy.fft"
+depends_on = ["scipy.special", "scipy._lib"]
+
+[[modules]]
+path = "scipy.fftpack"
+depends_on = ["scipy._lib", "scipy.fft"]
+
+[[modules]]
+path = "scipy.integrate"
+depends_on = ["scipy._lib", "scipy.interpolate", "scipy.sparse", "scipy.stats", "scipy.sparse.linalg", "scipy.special", "scipy.linalg", "scipy.optimize"]
+
+[[modules]]
+path = "scipy.interpolate"
+depends_on = ["scipy._lib", "scipy.special", "scipy.spatial.distance", "scipy.linalg", "scipy.optimize", "scipy.spatial", "scipy.linalg.lapack", "scipy.sparse.linalg", "scipy.sparse", "scipy"]
+
+[[modules]]
+path = "scipy.io"
+depends_on = ["scipy.sparse", "scipy.io.wavfile", "scipy.io.arff", "scipy.io.matlab", "scipy._lib"]
+
+[[modules]]
+path = "scipy.io.arff"
+depends_on = ["scipy._lib"]
+
+[[modules]]
+path = "scipy.io.matlab"
+depends_on = ["scipy.sparse", "scipy._lib"]
+
+[[modules]]
+path = "scipy.io.wavfile"
+depends_on = []
+
+[[modules]]
+path = "scipy.linalg"
+depends_on = ["scipy._lib", "scipy.sparse", "scipy.special", "scipy.sparse.linalg", "scipy.linalg.blas", "scipy.linalg.lapack", "scipy.fft"]
+
+[[modules]]
+path = "scipy.linalg.blas"
+depends_on = ["scipy.linalg"]
+
+[[modules]]
+path = "scipy.linalg.interpolative"
+depends_on = ["scipy.linalg", "scipy.sparse.linalg"]
+
+[[modules]]
+path = "scipy.linalg.lapack"
+depends_on = ["scipy.linalg", "scipy.linalg.blas"]
+
+[[modules]]
+path = "scipy.misc"
+depends_on = []
+
+[[modules]]
+path = "scipy.ndimage"
+depends_on = ["scipy.special", "scipy._lib"]
+
+[[modules]]
+path = "scipy.odr"
+depends_on = ["scipy.linalg", "scipy._lib"]
+
+[[modules]]
+path = "scipy.optimize"
+depends_on = ["scipy.linalg", "scipy.spatial", "scipy._lib", "scipy.sparse", "scipy.sparse.linalg", "scipy.stats.qmc", "scipy.linalg.blas", "scipy.special", "scipy.linalg.interpolative", "scipy"]
+
+[[modules]]
+path = "scipy.signal"
+depends_on = ["scipy.interpolate", "scipy.fft", "scipy.signal.windows", "scipy.ndimage", "scipy.special", "scipy.optimize", "scipy._lib", "scipy.stats", "scipy.spatial", "scipy.linalg"]
+
+[[modules]]
+path = "scipy.signal.windows"
+depends_on = ["scipy.linalg", "scipy.special", "scipy._lib", "scipy.fft"]
+
+[[modules]]
+path = "scipy.sparse"
+depends_on = ["scipy", "scipy._lib", "scipy.sparse.csgraph", "scipy.sparse.linalg"]
+
+[[modules]]
+path = "scipy.sparse.csgraph"
+depends_on = ["scipy._lib", "scipy.sparse", "scipy.sparse.linalg"]
+
+[[modules]]
+path = "scipy.sparse.linalg"
+depends_on = ["scipy._lib", "scipy.linalg", "scipy.sparse"]
+
+[[modules]]
+path = "scipy.spatial"
+depends_on = ["scipy._lib", "scipy.linalg", "scipy.spatial.transform", "scipy", "scipy.spatial.distance"]
+
+[[modules]]
+path = "scipy.spatial.distance"
+depends_on = ["scipy.spatial", "scipy._lib", "scipy.linalg", "scipy.special"]
+
+[[modules]]
+path = "scipy.spatial.transform"
+depends_on = ["scipy.constants", "scipy.linalg", "scipy._lib", "scipy.interpolate"]
+
+[[modules]]
+path = "scipy.special"
+depends_on = ["scipy.optimize", "scipy.linalg", "scipy._lib"]
+
+[[modules]]
+path = "scipy.stats"
+depends_on = [
+    "scipy.linalg.blas",
+    "scipy.stats.distributions",
+    "scipy.fft",
+    "scipy.spatial.distance",
+    "scipy.stats.contingency",
+    "scipy.special",
+    "scipy._lib",
+    "scipy.sparse.csgraph",
+    "scipy.spatial",
+    "scipy.linalg",
+    "scipy.stats.qmc",
+    "scipy.integrate",
+    "scipy.optimize",
+    "scipy.stats.mstats",
+    "scipy.ndimage",
+    "scipy.interpolate",
+    "scipy.sparse",
+]
+
+[[modules]]
+path = "scipy.stats.contingency"
+depends_on = ["scipy.stats", "scipy._lib"]
+
+[[modules]]
+path = "scipy.stats.distributions"
+depends_on = ["scipy.stats"]
+
+[[modules]]
+path = "scipy.stats.mstats"
+depends_on = ["scipy.stats"]
+
+[[modules]]
+path = "scipy.stats.qmc"
+depends_on = ["scipy.stats"]
+
+[[modules]]
+path = "scipy.stats.sampling"
+depends_on = ["scipy.stats"]
+
+
+# Optional dependencies don't seem to be handled by tach yet
+[external]
+exclude = [
+    "array-api-compat",
+    "array-api-extra",
+    "cupy",
+    "cupyx",
+    "dask",
+    "jax",
+    "matplotlib",
+    "ndonnx",
+    "sparse",
+    "torch",
+    "uarray",
+]


Reviews:
user: login: ev-br
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-12T08:37:02Z
user: login: lucascolley
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-12T09:03:33Z
user: login: lucascolley
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-12T11:35:43Z
user: login: ev-br
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-12T12:41:03Z
user: login: mdhaber
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-12T12:57:48Z
user: login: mdhaber
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-12T13:00:55Z
user: login: j-bowhay
body: Seems reasonable to give this a go, can always revert if it becomes a pain!
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-21T10:14:56Z

Review comments:
diff_hunk: @@ -26,7 +26,7 @@
 from scipy import integrate
 
 # to approximate the pdf of a continuous distribution given its cdf
-from scipy._lib._finite_differences import _derivative
+from scipy.linalg._finite_differences import _derivative
path: scipy/stats/_distn_infrastructure.py
user: login: ev-br
body: Why does it move to linalg? AFAICS it's scalar finite differences formulas, have nothing to do with linalg.
created_at: 2025-03-12T08:37:02Z
author_association: MEMBER
diff_hunk: @@ -26,7 +26,7 @@
 from scipy import integrate
 
 # to approximate the pdf of a continuous distribution given its cdf
-from scipy._lib._finite_differences import _derivative
+from scipy.linalg._finite_differences import _derivative
path: scipy/stats/_distn_infrastructure.py
user: login: lucascolley
body: It uses `linalg` and is used in `stats`. Can put it in `stats` instead?
created_at: 2025-03-12T09:03:33Z
author_association: MEMBER
diff_hunk: @@ -26,7 +26,7 @@
 from scipy import integrate
 
 # to approximate the pdf of a continuous distribution given its cdf
-from scipy._lib._finite_differences import _derivative
+from scipy.linalg._finite_differences import _derivative
path: scipy/stats/_distn_infrastructure.py
user: login: lucascolley
body: Alternatively, from a modularity perspective we can put it in any of the modules which `stats` depends on and which already depend on `linalg` at the minute. So

```
"scipy.stats",
"scipy.special",
"scipy.spatial",
"scipy.linalg",
"scipy.integrate",
"scipy.optimize",
"scipy.interpolate",
```

Let me know if any of those seem better!
created_at: 2025-03-12T11:35:43Z
author_association: MEMBER
diff_hunk: @@ -26,7 +26,7 @@
 from scipy import integrate
 
 # to approximate the pdf of a continuous distribution given its cdf
-from scipy._lib._finite_differences import _derivative
+from scipy.linalg._finite_differences import _derivative
path: scipy/stats/_distn_infrastructure.py
user: login: ev-br
body: This is a small utility, originally from `misc`, used only in `stats` AFAIU. If we don't want to keep it in `_lib`, let's put it in `stats` I'd say.
created_at: 2025-03-12T12:41:03Z
author_association: MEMBER
diff_hunk: @@ -26,7 +26,7 @@
 from scipy import integrate
 
 # to approximate the pdf of a continuous distribution given its cdf
-from scipy._lib._finite_differences import _derivative
+from scipy.linalg._finite_differences import _derivative
path: scipy/stats/_distn_infrastructure.py
user: login: mdhaber
body: `linalg` seems appropriate to me. We wouldn't want it in `stats` if `linalg` doesn't already depend on `stats`. 
created_at: 2025-03-12T12:57:48Z
author_association: CONTRIBUTOR
diff_hunk: @@ -26,7 +26,7 @@
 from scipy import integrate
 
 # to approximate the pdf of a continuous distribution given its cdf
-from scipy._lib._finite_differences import _derivative
+from scipy.linalg._finite_differences import _derivative
path: scipy/stats/_distn_infrastructure.py
user: login: mdhaber
body: Oh, if it's only used in stats, then yeah, in stats.
created_at: 2025-03-12T13:00:55Z
author_association: CONTRIBUTOR

PR comments:
user: login: lucascolley
created_at: 2025-03-12T00:58:53Z
body: To try this out:
- Install `tach` (available on PyPI and conda-forge, however beware https://github.com/conda-forge/tach-feedstock/issues/8)
- run `tach check --exact`

Alternatively, you can use https://github.com/lucascolley/pixi-dev-scipystack/tree/tach and `pixi run tach`.
author_association: MEMBER
user: login: lucascolley
created_at: 2025-03-12T01:02:39Z
body: Example of nice discovery of dependencies:

```
scipy on 🎋 tach [🔨] is 📦 v1.16.0.dev0 via 🐍 
❯ git diff
diff --git a/tach.toml b/tach.toml
index 0fdbb5fc0..a322bbbf2 100644
--- a/tach.toml
+++ b/tach.toml
@@ -163,7 +163,6 @@ path = "scipy.stats"
 depends_on = [
     "scipy.linalg.blas",
     "scipy.stats.distributions",
-    "scipy.fft",
     "scipy.spatial.distance",
     "scipy.stats.contingency",
     "scipy.special",

scipy on 🎋 tach [🔨] is 📦 v1.16.0.dev0 via 🐍 
❯ pixi run tach
✨ Pixi task (tach in default): tach check --exact
Internal Dependencies
❌ scipy/stats/_hypotests.py[L13]: Cannot use 'scipy.fft.ifft'. Module 'scipy.stats' cannot depend on 'scipy.fft'.
❌ scipy/stats/_qmvnt.py[L37]: Cannot use 'scipy.fft.fft'. Module 'scipy.stats' cannot depend on 'scipy.fft'.
❌ scipy/stats/_qmvnt.py[L37]: Cannot use 'scipy.fft.ifft'. Module 'scipy.stats' cannot depend on 'scipy.fft'.

If you intended to add a new dependency, run 'tach sync' to update your module configuration.
Otherwise, remove any disallowed imports and consider refactoring.
```
author_association: MEMBER
user: login: rgommers
created_at: 2025-03-12T03:58:07Z
body: Thanks for moving this forward Lucas! I think it needs a CI check. Maybe easiest is to fold it into the `lint` task? Not sure, but if it's fast enough then I think that is a reasonable thing to do and no one notices anything until the check fails.
author_association: MEMBER
user: login: mdhaber
created_at: 2025-03-12T12:41:45Z
body: > I think it needs a CI check.

The conversation Lucas mentioned was about something like scikit-image/scikit-image#7749, which uses dependency data (with sub-package granularity) to speed up tests. As part of that, we would also check that new dependencies are not added, and I mentioned that we could easily eliminate some very weak dependencies. @steppi was going to work on a similar PR for SciPy, but if he hasn't started, perhaps that's no longer needed with Stefan and Lucas looking into similar things?

Perhaps `tach` can be used instead of my simple (but very sloppy!) script! 
author_association: CONTRIBUTOR
user: login: lucascolley
created_at: 2025-03-12T13:14:20Z
body: It sounds like the test speed-ups would be a welcome follow-up to this PR still @steppi !
author_association: MEMBER
user: login: lucascolley
created_at: 2025-03-12T15:53:29Z
body: Good to go I think! We could also expose this via `dev.py lint`, but I wouldn't personally find that useful.
author_association: MEMBER