Repository: qutip/qutip

Issue:
Fix failing steadystate tests
**Description**

We have steadystate tests that fail almost always in CI on Python 3.9 with OpenMP and MKL, and sometimes with just Python 3.9 and MKL.

Issue is currently hard to reproduce locally.

**Related issues or PRs**
* Test failures seen in #1689 and elsewhere

**Progress so far**
* Fixed a small issue in steadystate tests so that I can use pytest-repeat to run `--count=100` on the steadystate tests in the hopes of reproducing the bug locally.
* Removed mutable default c_ops arguments for steadystate and liouvillian.
* Fix reference to method in _pseudo_inverse_sparse.
* ~~Only set method in pseudo_inverse if one is explicitly defined.~~ (revert)

**Changelog**
TODO: Write the changelog once we understand properly what is going on.


Discussion:
@Ericgig I've started this branch specifically to tackle the strange steadystate (and other) test failures.

[![Coverage Status](https://coveralls.io/builds/43683136/badge)](https://coveralls.io/builds/43683136)

Coverage remained the same at 16.832% when pulling **4b027c3c555bacd135e15bd21f842dd5c1962cf1 on hodgestar:fix/failing-steady-state-tests** into **091574d12a27a15f8d4861a6ae9c0c8a560dac42 on qutip:master**.

I think we finally have a concrete error and it's rather mystifying to me how it can happen:
```
            E = spla.expm(A.toarray())
            if np.isnan(E).any():
                print("A:", A)
                print("A data:", A.indices, A.indptr, A.shape)
                print("A toarray:", A.toarray())
                print("E:", E)
>               raise RuntimeError("NaNs generated by sp_expm.")
E               RuntimeError: NaNs generated by sp_expm.

qutip/sparse.py:408: RuntimeError
----------------------------- Captured stdout call -----------------------------
A:   (0, 1)	(-0.5+0j)
  (1, 0)	(0.5+0j)
  (1, 2)	(-0.7071067811865476+0j)
  (2, 1)	(0.7071067811865476+0j)
  (2, 3)	(-0.8660254037844386+0j)
  (3, 2)	(0.8660254037844386+0j)
  (3, 4)	(-1+0j)
  (4, 3)	(1+0j)
A data: [1 0 2 1 3 2 4 3] [0 1 3 5 7 8] (5, 5)
A toarray: [[ 0.        +0.j -0.5       +0.j  0.        +0.j  0.        +0.j
   0.        +0.j]
 [ 0.5       +0.j  0.        +0.j -0.70710678+0.j  0.        +0.j
   0.        +0.j]
 [ 0.        +0.j  0.70710678+0.j  0.        +0.j -0.8660254 +0.j
   0.        +0.j]
 [ 0.        +0.j  0.        +0.j  0.8660254 +0.j  0.        +0.j
  -1.        +0.j]
 [ 0.        +0.j  0.        +0.j  0.        +0.j  1.        +0.j
   0.        +0.j]]
E: [[nan+nanj nan+nanj nan+nanj nan+nanj nan+nanj]
 [nan+nanj nan+nanj nan+nanj nan+nanj nan+nanj]
 [nan+nanj nan+nanj nan+nanj nan+nanj nan+nanj]
 [nan+nanj nan+nanj nan+nanj nan+nanj nan+nanj]
 [nan+nanj nan+nanj nan+nanj nan+nanj nan+nanj]]
```
See https://github.com/qutip/qutip/runs/3966808806?check_suite_focus=true#step:6:1646
@Ericgig tracked this issue down to only occuring on numpy 1.21.X (and not 1.20.X) on CI workers with certain Intel CPUs (8171 and 8272). There are a number of changes in numpy 1.21 which could have caused this, but it might take awhile to track down.

The plan from here is to make a small PR for some of the tiny clean-ups from here that seem good to have anyway, and then to create a new PR off of master to try get us back onto 1.21.X somehow (probably this will require a numpy fix, but maybe there is another work around).
Minimal script to reproduce the error that only uses numpy:
```
# On a CPU with AVX512 extensions and numpy 1.21.2:
# (only tested on Ubuntu)
# It works again on numpy 1.21.4 (and maybe 1.21.3 -- I did not check because 1.21.3 was not conda installable)

import numpy as np

L = np.diag([1+0j, 1, 1, 1])
b = np.array([1+0j, 0, 0, 0])

# commenting out the line below makes everything work, with it solve returns nans.
np.exp(0)
# breakpoint()

v = np.linalg.solve(L, b)
np.testing.assert_allclose(v, b)
```
Numpy bug report -- https://github.com/numpy/numpy/issues/20356
Even smaller script for reproducing the issue:
```python
a = np.diag([1+0j, 1])
np.exp(0)
x = np.linalg.det(a)
```
@hodgestar do we close this now or do we wait for numpy's fix to be on conda.
@Ericgig I'm happy to leave this open until a new numpy is released and we can update the version of numpy used in CI tests. Probably also good to have an issue open in case users encounter this in the wild.
The bug fix is scheduled to be included in numpy 1.22.0 -- https://github.com/numpy/numpy/milestone/93.

Resolving patch:
diff --git a/qutip/steadystate.py b/qutip/steadystate.py
--- a/qutip/steadystate.py
+++ b/qutip/steadystate.py
@@ -81,7 +81,7 @@ def _default_steadystate_args():
     def_args = {'sparse': True, 'use_rcm': False,
                 'use_wbm': False, 'use_precond': False,
                 'all_states': False, 'M': None, 'x0': None, 'drop_tol': 1e-4,
-                'fill_factor': 100, 'diag_pivot_thresh': None, 'maxiter': 1000,
+                'fill_factor': 100, 'diag_pivot_thresh': 0.1, 'maxiter': 1000,
                 'permc_spec': 'COLAMD', 'ILU_MILU': 'smilu_2',
                 'restart': 20,
                 'max_iter_refine': 10,
@@ -1218,7 +1218,7 @@ def _pseudo_inverse_sparse(L, rhoss, w=None, **pseudo_args):
         LIQ = lu.solve(Q.toarray())
 
     else:
-        raise ValueError("unsupported method '%s'" % method)
+        raise ValueError("unsupported method '%s'" % pseudo_args['method'])
 
     R = sp.csr_matrix(Q * LIQ)
 


Test patch:
diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -64,6 +64,17 @@ jobs:
             python-version: "3.8"
             nocython: 1
 
+          # Python 3.10 and numpy 1.22
+          # Use conda-forge to provide numpy 1.22
+          # Ignore ImportWarning because pyximport registered an importer
+          # PyxImporter that does not have a find_spec method and this raises
+          # a warning on Python 3.10
+          - case-name: Python 3.10
+            os: ubuntu-latest
+            python-version: "3.10"
+            condaforge: 1
+            pytest-extra-options: "-W ignore::ImportWarning"
+
     steps:
       - uses: actions/checkout@v2
 
@@ -82,6 +93,10 @@ jobs:
             QUTIP_TARGET="$QUTIP_TARGET,runtime_compilation"
           fi
           export CI_QUTIP_WITH_OPENMP=${{ matrix.openmp }}
+          if [[ -n "${{ matrix.condaforge }}" ]]; then
+            conda config --prepend channels conda-forge
+            conda config --set channel_priority strict
+          fi
           if [[ -z "${{ matrix.nomkl }}" ]]; then
             conda install blas=*=mkl "numpy${{ matrix.numpy-requirement }}" "scipy${{ matrix.scipy-requirement }}"
           elif [[ "${{ matrix.os }}" =~ ^windows.*$ ]]; then
@@ -92,6 +107,7 @@ jobs:
             conda install nomkl "numpy${{ matrix.numpy-requirement }}" "scipy${{ matrix.scipy-requirement }}"
           fi
           python -m pip install -e .[$QUTIP_TARGET]
+          python -m pip install coverage==6.2
           python -m pip install pytest-cov coveralls
 
       - name: Package information
@@ -120,7 +136,7 @@ jobs:
             # truly being executed.
             export QUTIP_NUM_PROCESSES=2
           fi
-          pytest -Werror --strict-config --strict-markers --durations=0 --durations-min=1.0 --verbosity=1 --cov=qutip --cov-report= --color=yes qutip/tests
+          pytest -Werror --strict-config --strict-markers --durations=0 --durations-min=1.0 --verbosity=1 --cov=qutip --cov-report= --color=yes ${{ matrix.pytest-extra-options }} qutip/tests
           # Above flags are:
           #  -Werror
           #     treat warnings as errors