Repository: dask/dask

Issue:
`test_tsqr_uncertain` failing on gpuCI
`dask/array/tests/test_cupy_linalg.py::test_tsqr_uncertain` [failed on gpuCI](https://gpuci.gpuopenanalytics.com/job/dask/job/dask/job/prb/job/dask-prb/297/CUDA_VER=11.2,LINUX_VER=ubuntu18.04,PYTHON_VER=3.8,RAPIDS_VER=21.10/console) with ``TypeError: Implicit conversion to a NumPy array is not allowed. Please use `.get()` to construct a NumPy array``

```python
11:34:33 =================================== FAILURES ===================================
11:34:33 ______________ test_tsqr_uncertain[10-5-chunks0-True-False-None] _______________
11:34:33 [gw3] linux -- Python 3.8.10 /opt/conda/envs/dask/bin/python
11:34:33 
11:34:33 m_min = 10, n_max = 5, chunks = (10, 5), vary_rows = True, vary_cols = False
11:34:33 error_type = None
11:34:33 
11:34:33     @pytest.mark.parametrize(
11:34:33         "m_min,n_max,chunks,vary_rows,vary_cols,error_type",
11:34:33         [
11:34:33             (10, 5, (10, 5), True, False, None),  # single block tall
11:34:33             (10, 5, (10, 5), False, True, None),  # single block tall
11:34:33             (10, 5, (10, 5), True, True, None),  # single block tall
11:34:33             (40, 5, (10, 5), True, False, None),  # multiple blocks tall
11:34:33             (40, 5, (10, 5), False, True, None),  # multiple blocks tall
11:34:33             (40, 5, (10, 5), True, True, None),  # multiple blocks tall
11:34:33             (
11:34:33                 300,
11:34:33                 10,
11:34:33                 (40, 10),
11:34:33                 True,
11:34:33                 False,
11:34:33                 None,
11:34:33             ),  # tall-skinny regular thin layers; recursion_depth=2
11:34:33             (
11:34:33                 300,
11:34:33                 10,
11:34:33                 (30, 10),
11:34:33                 True,
11:34:33                 False,
11:34:33                 None,
11:34:33             ),  # tall-skinny regular thin layers; recursion_depth=3
11:34:33             (
11:34:33                 300,
11:34:33                 10,
11:34:33                 (20, 10),
11:34:33                 True,
11:34:33                 False,
11:34:33                 None,
11:34:33             ),  # tall-skinny regular thin layers; recursion_depth=4
11:34:33             (
11:34:33                 300,
11:34:33                 10,
11:34:33                 (40, 10),
11:34:33                 False,
11:34:33                 True,
11:34:33                 None,
11:34:33             ),  # tall-skinny regular thin layers; recursion_depth=2
11:34:33             (
11:34:33                 300,
11:34:33                 10,
11:34:33                 (30, 10),
11:34:33                 False,
11:34:33                 True,
11:34:33                 None,
11:34:33             ),  # tall-skinny regular thin layers; recursion_depth=3
11:34:33             (
11:34:33                 300,
11:34:33                 10,
11:34:33                 (20, 10),
11:34:33                 False,
11:34:33                 True,
11:34:33                 None,
11:34:33             ),  # tall-skinny regular thin layers; recursion_depth=4
11:34:33             (
11:34:33                 300,
11:34:33                 10,
11:34:33                 (40, 10),
11:34:33                 True,
11:34:33                 True,
11:34:33                 None,
11:34:33             ),  # tall-skinny regular thin layers; recursion_depth=2
11:34:33             (
11:34:33                 300,
11:34:33                 10,
11:34:33                 (30, 10),
11:34:33                 True,
11:34:33                 True,
11:34:33                 None,
11:34:33             ),  # tall-skinny regular thin layers; recursion_depth=3
11:34:33             (
11:34:33                 300,
11:34:33                 10,
11:34:33                 (20, 10),
11:34:33                 True,
11:34:33                 True,
11:34:33                 None,
11:34:33             ),  # tall-skinny regular thin layers; recursion_depth=4
11:34:33         ],
11:34:33     )
11:34:33     def test_tsqr_uncertain(m_min, n_max, chunks, vary_rows, vary_cols, error_type):
11:34:33         mat = cupy.random.rand(m_min * 2, n_max)
11:34:33         m, n = m_min * 2, n_max
11:34:33         mat[0:m_min, 0] += 1
11:34:33         _c0 = mat[:, 0]
11:34:33         _r0 = mat[0, :]
11:34:33         c0 = da.from_array(_c0, chunks=m_min, name="c", asarray=False)
11:34:33         r0 = da.from_array(_r0, chunks=n_max, name="r", asarray=False)
11:34:33         data = da.from_array(mat, chunks=chunks, name="A", asarray=False)
11:34:33         if vary_rows:
11:34:33             data = data[c0 > 0.5, :]
11:34:33             mat = mat[_c0 > 0.5, :]
11:34:33             m = mat.shape[0]
11:34:33         if vary_cols:
11:34:33             data = data[:, r0 > 0.5]
11:34:33             mat = mat[:, _r0 > 0.5]
11:34:33             n = mat.shape[1]
11:34:33     
11:34:33         # qr
11:34:33         m_q = m
11:34:33         n_q = min(m, n)
11:34:33         m_r = n_q
11:34:33         n_r = n
11:34:33     
11:34:33         # svd
11:34:33         m_u = m
11:34:33         n_u = min(m, n)
11:34:33         n_s = n_q
11:34:33         m_vh = n_q
11:34:33         n_vh = n
11:34:33         d_vh = max(m_vh, n_vh)  # full matrix returned
11:34:33     
11:34:33         if error_type is None:
11:34:33             # test QR
11:34:33             q, r = da.linalg.tsqr(data)
11:34:33 >           q = q.compute()  # because uncertainty
11:34:33 
11:34:33 dask/array/tests/test_cupy_linalg.py:229: 
11:34:33 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
11:34:33 dask/base.py:286: in compute
11:34:33     (result,) = compute(self, traverse=False, **kwargs)
11:34:33 dask/base.py:568: in compute
11:34:33     results = schedule(dsk, keys, **kwargs)
11:34:33 dask/threaded.py:79: in get
11:34:33     results = get_async(
11:34:33 dask/local.py:517: in get_async
11:34:33     raise_exception(exc, tb)
11:34:33 dask/local.py:325: in reraise
11:34:33     raise exc
11:34:33 dask/local.py:223: in execute_task
11:34:33     result = _execute_task(task, data)
11:34:33 dask/core.py:121: in _execute_task
11:34:33     return func(*(_execute_task(a, cache) for a in args))
11:34:33 dask/optimization.py:969: in __call__
11:34:33     return core.get(self.dsk, self.outkey, dict(zip(self.inkeys, args)))
11:34:33 dask/core.py:151: in get
11:34:33     result = _execute_task(task, cache)
11:34:33 dask/core.py:121: in _execute_task
11:34:33     return func(*(_execute_task(a, cache) for a in args))
11:34:33 <__array_function__ internals>:5: in dot
11:34:33     ???
11:34:33 cupy/_core/core.pyx:1513: in cupy._core.core.ndarray.__array_function__
11:34:33     ???
11:34:33 /opt/conda/envs/dask/lib/python3.8/site-packages/cupy/linalg/_product.py:65: in dot
11:34:33     return a.dot(b, out)
11:34:33 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
11:34:33 
11:34:33 >   ???
11:34:33 E   TypeError: Implicit conversion to a NumPy array is not allowed. Please use `.get()` to construct a NumPy array explicitly.
11:34:33 
11:34:33 cupy/_core/core.pyx:1298: TypeError
```

cc @dask/gpu 


Discussion:
@charlesbluca can you take a look at this when you have a moment ?
Just wanted to note that this appears to only fail intermittently as the gpuCI build in https://github.com/dask/dask/pull/8102 (which only modified an unrelated test module) passed 
Yeah I'm not able to replicate this failure locally after several runs, so perhaps this is just a flaky test? I'm assuming the issue here is that either `b` or `out` are somehow numpy arrays, and I'll see if I can debug how this happens if I get a a failure.

In the meantime, do we have a preference on marking GPU tests as `flaky`?
Hrm interesting, just to confirm, you're using the same `cupy`, `numpy`, etc. versions as the gpuCI build? 
Sorry, should've bumped this - I can confirm that this is a flaky test, but the chances of failure are relatively low (only failed 2-3 times after running the specific test around 300 times). It looks like the failure is what I suspected - at some point, we are calling `np.dot` on a numpy and cupy array, although I still am not sure why this happens sometimes
Guessing you are already familiar with [pytest's ability to run a debugger during testing]( https://docs.pytest.org/en/6.2.x/usage.html#dropping-to-pdb-python-debugger-on-failures ), might be useful for exploring the stack and finding out where things go off the rails.
It looks like the erroneous numpy array is associated with these `getitem` task(s) being generated by `tsqr`:

https://github.com/dask/dask/blob/bfbce61a623587ca8693ba280c76387bb8060c22/dask/array/linalg.py#L146-L153

It's a little hard to get any additional info from the debugger though since it's just reraising the above error and I imagine whatever went wrong seemingly happened earlier on
Might be worth trying with [the single threaded scheduler]( https://docs.dask.org/en/latest/scheduling.html#single-thread )?
Thanks for the suggestion, I'll give that a try and bump this if I find anything else 🙂 
`pdb` also has commands like `up` and `down` which will let you go up or down the call stack to help figure out where things went wrong https://docs.python.org/3/library/pdb.html#debugger-commands

Resolving patch:
diff --git a/dask/array/linalg.py b/dask/array/linalg.py
--- a/dask/array/linalg.py
+++ b/dask/array/linalg.py
@@ -44,7 +44,7 @@ def _wrapped_qr(a):
     """
     # workaround may be removed when numpy stops rejecting edge cases
     if a.shape[0] == 0:
-        return np.zeros((0, 0)), np.zeros((0, a.shape[1]))
+        return np.zeros_like(a, shape=(0, 0)), np.zeros_like(a, shape=(0, a.shape[1]))
     else:
         return np.linalg.qr(a)
 


Test patch: