Repository: dask/dask

Issue:
Scipy's `gmres` fails to converge with Dask Array
<!-- Please include a self-contained copy-pastable example that generates the issue if possible.

Please be concise with code posted. See guidelines below on how to provide a good bug report:

- Craft Minimal Bug Reports http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports
- Minimal Complete Verifiable Examples https://stackoverflow.com/help/mcve

Bug reports that follow these guidelines are easier to diagnose, and so are often handled much more quickly.
-->

**What happened**:
During a review of https://github.com/dask/dask/pull/8694, I noticed that scipy's `gmres` fails to converge on a solution with even a small dask array after several minutes. The task graph grows seemingly boundlessly

![image](https://user-images.githubusercontent.com/1163939/153312892-1b88d0f6-56f2-4d97-a8f4-748bab6c6324.png)

**What you expected to happen**:

`gmres` should converge seemingly instantly


**Minimal Complete Verifiable Example**:

```python
import numpy as np
from scipy.sparse.linalg import gmres
from scipy.sparse.linalg._interface import MatrixLinearOperator

from dask.distributed import Client
import dask.array as da

np.random.seed(0)
x_np = np.random.random(size=(40, 40))

with Client(n_workers=4) as client:
    x = da.from_array(x_np, chunks=(4, 4))
    A = MatrixLinearOperator(x)
    b = np.random.random(40)
    result =  gmres(A, b)
```

**Anything else we need to know?**:

**Environment**:

- Dask version: 2022.1.1
- Python version: 3.9.10
- Scipy version: 1.18.0
- Operating System: Mac OS 12.2 (21D49) with the M1 Pro chip
- Install method (conda, pip, source): conda (mambaforge in particular)

<details>
<summary>Full env</summary>

```text
# packages in environment at /Users/paul/mambaforge/envs/hydro:
#
# Name                    Version                   Build  Channel
aiohttp                   3.8.1            py39h5161555_0    conda-forge
aiosignal                 1.2.0              pyhd8ed1ab_0    conda-forge
anyio                     3.5.0            py39h2804cbe_0    conda-forge
appnope                   0.1.2            py39h2804cbe_2    conda-forge
argon2-cffi               21.3.0             pyhd8ed1ab_0    conda-forge
argon2-cffi-bindings      21.2.0           py39h5161555_1    conda-forge
asttokens                 2.0.5              pyhd8ed1ab_0    conda-forge
async-timeout             4.0.2              pyhd8ed1ab_0    conda-forge
attrs                     21.4.0             pyhd8ed1ab_0    conda-forge
babel                     2.9.1              pyh44b312d_0    conda-forge
backcall                  0.2.0              pyh9f0ad1d_0    conda-forge
backports                 1.0                        py_2    conda-forge
backports.functools_lru_cache 1.6.4              pyhd8ed1ab_0    conda-forge
black                     22.1.0             pyhd8ed1ab_0    conda-forge
bleach                    4.1.0              pyhd8ed1ab_0    conda-forge
bokeh                     2.4.2            py39h2804cbe_0    conda-forge
brotli                    1.0.9                h3422bc3_6    conda-forge
brotli-bin                1.0.9                h3422bc3_6    conda-forge
brotlipy                  0.7.0           py39h5161555_1003    conda-forge
bzip2                     1.0.8                h3422bc3_4    conda-forge
ca-certificates           2021.10.8            h4653dfc_0    conda-forge
certifi                   2021.10.8        py39h2804cbe_1    conda-forge
cffi                      1.15.0           py39h52b1de0_0    conda-forge
charset-normalizer        2.0.11             pyhd8ed1ab_0    conda-forge
click                     8.0.3            py39h2804cbe_1    conda-forge
cloudpickle               2.0.0              pyhd8ed1ab_0    conda-forge
cloudside                 0.1                       dev_0    <develop>
cryptography              36.0.1           py39hfb8cd70_0    conda-forge
cycler                    0.11.0             pyhd8ed1ab_0    conda-forge
cytoolz                   0.11.2           py39h5161555_1    conda-forge
dask                      2022.1.1           pyhd8ed1ab_0    conda-forge
dask-core                 2022.1.1           pyhd8ed1ab_0    conda-forge
dask-labextension         5.2.0              pyhd8ed1ab_0    conda-forge
dataclasses               0.8                pyhc8e2a94_3    conda-forge
debugpy                   1.5.1            py39hfb83b0d_0    conda-forge
decorator                 5.1.1              pyhd8ed1ab_0    conda-forge
defusedxml                0.7.1              pyhd8ed1ab_0    conda-forge
distributed               2022.1.1         py39h2804cbe_0    conda-forge
entrypoints               0.4                pyhd8ed1ab_0    conda-forge
executing                 0.8.2              pyhd8ed1ab_0    conda-forge
flit-core                 3.6.0              pyhd8ed1ab_0    conda-forge
fonttools                 4.29.1           py39h5161555_0    conda-forge
freetype                  2.10.4               h17b34a0_1    conda-forge
fribidi                   1.0.10               h27ca646_0    conda-forge
frozenlist                1.3.0            py39h5161555_0    conda-forge
fsspec                    2022.1.0           pyhd8ed1ab_0    conda-forge
giflib                    5.2.1                h27ca646_2    conda-forge
heapdict                  1.0.1                      py_0    conda-forge
idna                      3.3                pyhd8ed1ab_0    conda-forge
importlib-metadata        4.10.1           py39h2804cbe_0    conda-forge
importlib_resources       5.4.0              pyhd8ed1ab_0    conda-forge
iniconfig                 1.1.1              pyh9f0ad1d_0    conda-forge
ipykernel                 6.9.0            py39h32adebf_0    conda-forge
ipython                   8.0.1            py39h2804cbe_0    conda-forge
ipython_genutils          0.2.0                      py_1    conda-forge
jbig                      2.1               h3422bc3_2003    conda-forge
jedi                      0.18.1           py39h2804cbe_0    conda-forge
jinja2                    3.0.3              pyhd8ed1ab_0    conda-forge
jpeg                      9e                   h3422bc3_0    conda-forge
json5                     0.9.5              pyh9f0ad1d_0    conda-forge
jsonschema                4.4.0              pyhd8ed1ab_0    conda-forge
jupyter-server-proxy      3.2.1              pyhd8ed1ab_0    conda-forge
jupyter_client            7.1.2              pyhd8ed1ab_0    conda-forge
jupyter_core              4.9.1            py39h2804cbe_1    conda-forge
jupyter_server            1.13.5             pyhd8ed1ab_0    conda-forge
jupyterlab                3.2.9              pyhd8ed1ab_0    conda-forge
jupyterlab_pygments       0.1.2              pyh9f0ad1d_0    conda-forge
jupyterlab_server         2.10.3             pyhd8ed1ab_0    conda-forge
kiwisolver                1.3.2            py39h4d2d688_1    conda-forge
lcms2                     2.12                 had6a04f_0    conda-forge
lerc                      3.0                  hbdafb3b_0    conda-forge
libblas                   3.9.0           13_osxarm64_openblas    conda-forge
libbrotlicommon           1.0.9                h3422bc3_6    conda-forge
libbrotlidec              1.0.9                h3422bc3_6    conda-forge
libbrotlienc              1.0.9                h3422bc3_6    conda-forge
libcblas                  3.9.0           13_osxarm64_openblas    conda-forge
libcxx                    12.0.1               h168391b_1    conda-forge
libdeflate                1.8                  h3422bc3_0    conda-forge
libffi                    3.4.2                h3422bc3_5    conda-forge
libgfortran               5.0.0.dev0      11_0_1_hf114ba7_23    conda-forge
libgfortran5              11.0.1.dev0         hf114ba7_23    conda-forge
liblapack                 3.9.0           13_osxarm64_openblas    conda-forge
libopenblas               0.3.18          openmp_h5dd58f0_0    conda-forge
libpng                    1.6.37               hf7e6567_2    conda-forge
libsodium                 1.0.18               h27ca646_1    conda-forge
libtiff                   4.3.0                h74060c4_2    conda-forge
libwebp                   1.2.2                h0d20362_0    conda-forge
libwebp-base              1.2.2                h3422bc3_1    conda-forge
libxcb                    1.13              h9b22ae9_1004    conda-forge
libzlib                   1.2.11            hee7b306_1013    conda-forge
llvm-openmp               12.0.1               hf3c4609_1    conda-forge
locket                    0.2.0                      py_2    conda-forge
lz4-c                     1.9.3                hbdafb3b_1    conda-forge
markupsafe                2.0.1            py39h5161555_1    conda-forge
matplotlib                3.5.1            py39hdf13c20_0    conda-forge
matplotlib-base           3.5.1            py39h5aa4fe7_0    conda-forge
matplotlib-inline         0.1.3              pyhd8ed1ab_0    conda-forge
metar                     1.9.0              pyhd8ed1ab_0    conda-forge
mistune                   0.8.4           py39h5161555_1005    conda-forge
msgpack-python            1.0.3            py39h4d2d688_0    conda-forge
multidict                 6.0.2            py39h5161555_0    conda-forge
munkres                   1.1.4              pyh9f0ad1d_0    conda-forge
mypy_extensions           0.4.3            py39h2804cbe_4    conda-forge
nbclassic                 0.3.5              pyhd8ed1ab_0    conda-forge
nbclient                  0.5.10             pyhd8ed1ab_1    conda-forge
nbconvert                 6.4.1            py39h2804cbe_0    conda-forge
nbformat                  5.1.3              pyhd8ed1ab_0    conda-forge
ncurses                   6.3                  hc470f4d_0    conda-forge
nest-asyncio              1.5.4              pyhd8ed1ab_0    conda-forge
notebook                  6.4.8              pyha770c72_0    conda-forge
numpy                     1.22.2           py39h61a45d2_0    conda-forge
openjpeg                  2.4.0                h062765e_1    conda-forge
openssl                   1.1.1l               h3422bc3_0    conda-forge
packaging                 21.3               pyhd8ed1ab_0    conda-forge
pandas                    1.4.0            py39h7f752ed_0    conda-forge
pandocfilters             1.5.0              pyhd8ed1ab_0    conda-forge
parso                     0.8.3              pyhd8ed1ab_0    conda-forge
partd                     1.2.0              pyhd8ed1ab_0    conda-forge
pathspec                  0.9.0              pyhd8ed1ab_0    conda-forge
pexpect                   4.8.0              pyh9f0ad1d_2    conda-forge
pickleshare               0.7.5                   py_1003    conda-forge
pillow                    9.0.1            py39hcb29f89_0    conda-forge
pip                       22.0.3             pyhd8ed1ab_0    conda-forge
platformdirs              2.4.1              pyhd8ed1ab_1    conda-forge
pluggy                    1.0.0            py39h2804cbe_2    conda-forge
prometheus_client         0.13.1             pyhd8ed1ab_0    conda-forge
prompt-toolkit            3.0.26             pyha770c72_0    conda-forge
psutil                    5.9.0            py39h5161555_0    conda-forge
pthread-stubs             0.4               h27ca646_1001    conda-forge
ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
pure_eval                 0.2.2              pyhd8ed1ab_0    conda-forge
py                        1.11.0             pyh6c4a22f_0    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pygments                  2.11.2             pyhd8ed1ab_0    conda-forge
pyopenssl                 22.0.0             pyhd8ed1ab_0    conda-forge
pyparsing                 3.0.7              pyhd8ed1ab_0    conda-forge
pyrsistent                0.18.1           py39h5161555_0    conda-forge
pysocks                   1.7.1            py39h2804cbe_4    conda-forge
pytest                    7.0.0            py39h2804cbe_0    conda-forge
python                    3.9.10          hd16f9c5_2_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python_abi                3.9                      2_cp39    conda-forge
pytz                      2021.3             pyhd8ed1ab_0    conda-forge
pyyaml                    6.0              py39h5161555_3    conda-forge
pyzmq                     22.3.0           py39h02c6a76_1    conda-forge
readline                  8.1                  hedafd6a_0    conda-forge
requests                  2.27.1             pyhd8ed1ab_0    conda-forge
scipy                     1.8.0            py39h5060c3b_1    conda-forge
send2trash                1.8.0              pyhd8ed1ab_0    conda-forge
setuptools                59.8.0           py39h2804cbe_0    conda-forge
simpervisor               0.4                pyhd8ed1ab_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
sniffio                   1.2.0            py39h2804cbe_2    conda-forge
sortedcontainers          2.4.0              pyhd8ed1ab_0    conda-forge
sqlite                    3.37.0               h72a2b83_0    conda-forge
stack_data                0.1.4              pyhd8ed1ab_0    conda-forge
tblib                     1.7.0              pyhd8ed1ab_0    conda-forge
terminado                 0.13.1           py39h2804cbe_0    conda-forge
testpath                  0.5.0              pyhd8ed1ab_0    conda-forge
tk                        8.6.11               he1e0b03_1    conda-forge
tomli                     2.0.0              pyhd8ed1ab_1    conda-forge
toolz                     0.11.2             pyhd8ed1ab_0    conda-forge
tornado                   6.1              py39h5161555_2    conda-forge
traitlets                 5.1.1              pyhd8ed1ab_0    conda-forge
typed-ast                 1.5.2            py39h5161555_0    conda-forge
typing-extensions         4.0.1                hd8ed1ab_0    conda-forge
typing_extensions         4.0.1              pyha770c72_0    conda-forge
tzdata                    2021e                he74cb21_0    conda-forge
unicodedata2              14.0.0           py39h5161555_0    conda-forge
urllib3                   1.26.8             pyhd8ed1ab_1    conda-forge
wcwidth                   0.2.5              pyh9f0ad1d_2    conda-forge
webencodings              0.5.1                      py_1    conda-forge
websocket-client          1.2.3              pyhd8ed1ab_0    conda-forge
wheel                     0.37.1             pyhd8ed1ab_0    conda-forge
xorg-libxau               1.0.9                h27ca646_0    conda-forge
xorg-libxdmcp             1.1.3                h27ca646_0    conda-forge
xz                        5.2.5                h642e427_1    conda-forge
yaml                      0.2.5                h3422bc3_2    conda-forge
yarl                      1.7.2            py39h5161555_1    conda-forge
zeromq                    4.3.4                hbdafb3b_1    conda-forge
zict                      2.0.0                      py_0    conda-forge
zipp                      3.7.0              pyhd8ed1ab_1    conda-forge
zlib                      1.2.11            hee7b306_1013    conda-forge
zstd                      1.5.2                h861e0a7_0    conda-forge

```
</details>

<!-- If you are reporting an issue such as scale stability, cluster deadlock.
Please provide a cluster dump state with this issue, by running client.dump_cluster_state()

https://distributed.dask.org/en/stable/api.html?highlight=dump_cluster_state#distributed.Client.dump_cluster_state

-->

<details>
<summary>Cluster Dump State:</summary>

</details>


Discussion:

Resolving patch:
diff --git a/dask/array/stats.py b/dask/array/stats.py
--- a/dask/array/stats.py
+++ b/dask/array/stats.py
@@ -28,6 +28,7 @@
 # out of the use of this software, even if advised of the possibility of
 # such damage.
 import math
+from collections import namedtuple
 
 import numpy as np
 
@@ -42,16 +43,27 @@
     raise ImportError("`dask.array.stats` requires `scipy` to be installed.") from e
 from scipy import special
 from scipy.stats import distributions
-from scipy.stats.stats import (
-    F_onewayResult,
-    KurtosistestResult,
-    NormaltestResult,
-    Power_divergenceResult,
-    SkewtestResult,
-    Ttest_1sampResult,
-    Ttest_indResult,
-    Ttest_relResult,
-)
+
+# copied from https://github.com/scipy/scipy/blob/v1.8.0/scipy/stats/_stats_py.py since
+# these are all private after v1.8.0
+F_onewayResult = namedtuple("F_onewayResult", ("statistic", "pvalue"))
+KurtosistestResult = namedtuple("KurtosistestResult", ("statistic", "pvalue"))
+NormaltestResult = namedtuple("NormaltestResult", ("statistic", "pvalue"))
+Power_divergenceResult = namedtuple("Power_divergenceResult", ("statistic", "pvalue"))
+SkewtestResult = namedtuple("SkewtestResult", ("statistic", "pvalue"))
+Ttest_1sampResult = namedtuple("Ttest_1sampResult", ("statistic", "pvalue"))
+Ttest_indResult = namedtuple("Ttest_indResult", ("statistic", "pvalue"))
+Ttest_relResult = namedtuple("Ttest_relResult", ("statistic", "pvalue"))
+
+# Map from names to lambda_ values used in power_divergence().
+_power_div_lambda_names = {
+    "pearson": 1,
+    "log-likelihood": 0,
+    "freeman-tukey": -0.5,
+    "mod-log-likelihood": -1,
+    "neyman": -2,
+    "cressie-read": 2 / 3,
+}
 
 __all__ = [
     "ttest_ind",
@@ -140,14 +152,13 @@ def chisquare(f_obs, f_exp=None, ddof=0, axis=0):
 def power_divergence(f_obs, f_exp=None, ddof=0, axis=0, lambda_=None):
 
     if isinstance(lambda_, str):
-        # TODO: public api
-        if lambda_ not in scipy.stats.stats._power_div_lambda_names:
-            names = repr(list(scipy.stats.stats._power_div_lambda_names.keys()))[1:-1]
+        if lambda_ not in _power_div_lambda_names:
+            names = repr(list(_power_div_lambda_names.keys()))[1:-1]
             raise ValueError(
                 f"invalid string for lambda_: {lambda_!r}. "
                 f"Valid strings are {names}"
             )
-        lambda_ = scipy.stats.stats._power_div_lambda_names[lambda_]
+        lambda_ = _power_div_lambda_names[lambda_]
     elif lambda_ is None:
         lambda_ = 1
 


Test patch:
diff --git a/dask/array/tests/test_linearoperator.py b/dask/array/tests/test_linearoperator.py
deleted file mode 100644
--- a/dask/array/tests/test_linearoperator.py
+++ /dev/null
@@ -1,31 +0,0 @@
-import pytest
-
-pytest.importorskip("scipy")
-
-import numpy as np
-import scipy.sparse.linalg
-
-import dask.array as da
-
-
-def test_LinearOperator():
-    X = np.random.random(size=(3, 2))
-    y = np.random.random(size=(2, 1))
-    w = np.random.random(size=(3, 1))
-    square = np.random.random(size=(2, 2))
-
-    dX = da.from_array(X, chunks=(2, 1))
-
-    npLO = scipy.sparse.linalg.aslinearoperator(X)
-    daLO = scipy.sparse.linalg.interface.MatrixLinearOperator(dX)
-
-    functions = [lambda x, y: x.matvec(y), lambda x, y: x * y, lambda x, y: x.dot(y)]
-    for func in functions:
-        assert np.allclose(func(npLO, y), func(daLO, y))
-
-    assert np.allclose(npLO.matmat(square), daLO.matmat(square))
-
-    assert np.allclose(npLO.rmatvec(w), daLO.rmatvec(w))
-
-    assert npLO.dtype == daLO.dtype
-    assert npLO.shape == daLO.shape