Repository: qutip/qutip

Issue:
propagator fails to dimension check for composite system
Hi, I report a strange behavior of `propagator` method.
`propagator` with collapse operator fails to dimension validation on composite system.

At first, single system works correctly.
```python
import qutip as qt
```
```python
H = qt.sigmaz()
c_ops = [np.sqrt(1) * qt.sigmam()]
tlist = np.linspace(0, 10, 201)
rho0 = qt.ket2dm(qt.basis(2, 0)).unit()

# mesolve
rho_f = qt.mesolve(H, rho0, tlist, c_ops=c_ops).states[-1]

# using propagator
F = qt.propagator(H, tlist, c_op_list=c_ops)[-1]
rho_f_prop = qt.vector_to_operator(F * qt.operator_to_vector(rho0))

rho_f.overlap(rho_f_prop)
```

The output is
```
0.9999092149599982
```
(I'm not sure why this is not 1. I guess it's limited by the numerical precision.)


However, same code on composite system fails.
The differences are only `H`, `c_ops`, and `rho0`, using `qt.tensor(*, qt.qeye(2))`
```python
H = qt.tensor(qt.sigmaz(), qt.qeye(2))
c_ops = [np.sqrt(1) * qt.tensor(qt.sigmam(), qt.qeye(2))]
tlist = np.linspace(0, 10, 201)
rho0 = qt.ket2dm(qt.tensor(qt.basis(2, 0), qt.basis(2, 0))).unit()

# mesolve
rho_f = qt.mesolve(H, rho0, tlist, c_ops=c_ops).states[-1]

# using propagator
F = qt.propagator(H, tlist, c_op_list=c_ops)[-1]
rho_f_prop = qt.vector_to_operator(F * qt.operator_to_vector(rho0))

rho_f.overlap(rho_f_prop)
```

```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-1-551bcc27bcfa> in <module>
      8 
      9 # using propagator
---> 10 F = qt.propagator(H, tlist, c_op_list=c_ops)[-1]
     11 rho_f_prop = qt.vector_to_operator(F * qt.operator_to_vector(rho0))
     12 

c:\users\mizuno\research\py38\lib\site-packages\qutip\propagator.py in propagator(H, t, c_op_list, args, options, unitary_mode, parallel, progress_bar, _safe_mode, **kwargs)
    239                 rho0 = Qobj(sp.csr_matrix(([1], ([row_idx], [col_idx])),
    240                                           shape=(N, N), dtype=complex))
--> 241                 output = mesolve(H, rho0, tlist, c_op_list, [], args, options,
    242                                  _safe_mode=False)
    243                 for k, t in enumerate(tlist):

c:\users\mizuno\research\py38\lib\site-packages\qutip\mesolve.py in mesolve(H, rho0, tlist, c_ops, e_ops, args, options, progress_bar, _safe_mode)
    263         raise Exception("Invalid H type")
    264 
--> 265     func, ode_args = ss.makefunc(ss, rho0, args, e_ops, options)
    266 
    267     if _safe_mode:

c:\users\mizuno\research\py38\lib\site-packages\qutip\mesolve.py in _qobjevo_set(HS, rho0, args, e_ops, opt)
    349         # Should be caught earlier in mesolve.
    350         raise ValueError("rho0 must be a ket, density matrix or superoperator")
--> 351     _test_liouvillian_dimensions(H_td.cte.dims, rho0.dims)
    352     return func, ()
    353 

c:\users\mizuno\research\py38\lib\site-packages\qutip\mesolve.py in _test_liouvillian_dimensions(L_dims, rho_dims)
    330         raise ValueError("Liouvillian had nonsquare dims: " + str(L_dims))
    331     if not ((L_dims[1] == rho_dims) or (L_dims[1] == rho_dims[0])):
--> 332         raise ValueError("".join([
    333             "incompatible Liouvillian and state dimensions: ",
    334             str(L_dims), " and ", str(rho_dims),

ValueError: incompatible Liouvillian and state dimensions: [[[2, 2], [2, 2]], [[2, 2], [2, 2]]] and [[4], [4]]
```


My environment is following.
```
QuTiP: Quantum Toolbox in Python
================================
Copyright (c) QuTiP team 2011 and later.
Current admin team: Alexander Pitchford, Nathan Shammah, Shahnawaz Ahmed, Neill Lambert, Eric Giguère, Boxi Li, Jake Lishman and Simon Cross.
Board members: Daniel Burgarth, Robert Johansson, Anton F. Kockum, Franco Nori and Will Zeng.
Original developers: R. J. Johansson & P. D. Nation.
Previous lead developers: Chris Granade & A. Grimsmo.
Currently developed through wide collaboration. See https://github.com/qutip for details.

QuTiP Version:      4.6.2
Numpy Version:      1.19.5
Scipy Version:      1.6.2
Cython Version:     0.29.22
Matplotlib Version: 3.4.1
Python Version:     3.8.9
Number of CPUs:     16
BLAS Info:          OPENBLAS
OPENMP Installed:   False
INTEL MKL Ext:      False
Platform Info:      Windows (AMD64)
Installation path:  c:\users\mizuno\research\py38\lib\site-packages\qutip
================================================================================
Please cite QuTiP in your publication.
================================================================================
For your convenience a bibtex reference can be easily generated using `qutip.cite()`
```



Discussion:
Thanks Kosuke-san!  This does look like a genuine bug, seems to be because when constructing the propagator, the dims on the initial state are not set, and this clashes with very recent fixes done on mesolve (e.g., https://github.com/qutip/qutip/pull/1459 )

This wasn't caught by the tests during the above fixes because it seems like the only composite system test is done on a noiseless system, so mesolve isn't called in the propagator construction for that test.

thanks for pointing this out.  
Yeah, this feels like a bug in `propagator` - I bet at some point we do `qeye(L.shape[0])` instead of `qeye(L.dims[0])`.  There's sometimes quite a bit of that in older code - it used to slip through because we weren't super strict about dimension handling, so in certain cases (like this one), we'd accept bad dimensions.

I might try and have a look tonight (UK time), if I have time - I've wanted to break up the large `propagator` monolith into smaller components for quite a while.

---

About numerical precision: if you want to try increasing the precision of the integrators, try passing `options=qutip.Options(nsteps=1_000_000, atol=1e-12, rtol=1e-10)` to `propagator`.  `atol` and `rtol` are absolute and relative tolerances respectively, and `nsteps` is the number of subdivisions the integrator is _allowed_ to take (if it reaches its tolerance goals, it won't take all of them).  You'll find that `atol` and `rtol` will have only a loose relation to the tolerance of your fidelity value, because there's a _lot_ of floating-point operations between those points in which the errors compound, and there's a hard limit on the tolerances the integrator can achieve, so you might not be able to get a perfect answer.  The fidelity calculation involves finding the square root of a density operator, so that's another large possible source of error.

About times: if you only care about the state at the last time, setting `tlist = np.linspace(0, x, 201)` is quite inefficient.  Instead, set `tlist = [0, x]`, and then pass `options=qutip.Options(nsteps=<very very big integer>)`.  It's more efficient because it doesn't need to achieve tolerance goals and construct/store `Qobj` instances at every intermediate timestep.

Temporary workaround: this is _very_ temporary, and generally a bad idea in QuTiP, but you can "flatten" all your operators into a single space for now, while we implement a fix.  To do that:
```python
# initial setup
H = qutip.tensor(...)
c_ops = [qutip.tensor(...)]

original_dimensions = H.dims.copy()
new_dimensions = list(H.shape)

H.dims = new_dimensions
for c_op in c_ops:
    c_op.dims = new_dimensions

outputs = qutip.propagator(H, times, c_ops=c_ops)
for op in outputs:
    # the ops are superoperators, so have expanded dimensions
    op.dims = [original_dimensions, original_dimensions]
```
@nwlambert: here's a thought - do you reckon it's worth implementing a test on the rank of the Hamiltonian/Liouvillian on input to `propagator`?  If it's significantly less than full-rank (which it almost invariably will be), there's probably a really good speed-up to be had by evolving the system in a rotated frame.
Ah no, never mind - time dependence would mess you up, and actually the sort of tri-diagonal systems I had in mind aren't so low-rank anyway.  Maybe I'll look into it, but it won't be anywhere near as good as I had first thought.

Resolving patch:
diff --git a/qutip/propagator.py b/qutip/propagator.py
--- a/qutip/propagator.py
+++ b/qutip/propagator.py
@@ -48,7 +48,7 @@
 from qutip.cy.sparse_utils import unit_row_norm
 from qutip.mesolve import mesolve
 from qutip.sesolve import sesolve
-from qutip.states import basis
+from qutip.states import basis, projection
 from qutip.solver import Options, _solver_safety_check, config
 from qutip.parallel import parallel_map, _default_kwargs
 from qutip.ui.progressbar import BaseProgressBar, TextProgressBar
@@ -196,22 +196,23 @@ def propagator(H, t, c_op_list=[], args={}, options=None,
         sqrt_N = int(np.sqrt(N))
         dims = H0.dims
 
-        u = np.zeros([N, N, len(tlist)], dtype=complex)
-
         if parallel:
-            output = parallel_map(_parallel_mesolve, range(N * N),
+            output = parallel_map(_parallel_mesolve, range(N),
                                   task_args=(
                                       sqrt_N, H, tlist, c_op_list, args,
                                       options),
+                                  task_kwargs={"dims": H0.dims[0]},
                                   progress_bar=progress_bar, num_cpus=num_cpus)
-            for n in range(N * N):
+
+            u = np.zeros([N, N, len(tlist)], dtype=complex)
+            for n in range(N):
                 for k, t in enumerate(tlist):
                     u[:, n, k] = mat2vec(output[n].states[k].full()).T
         else:
-            rho0 = qeye(N, N)
-            rho0.dims = [[sqrt_N, sqrt_N], [sqrt_N, sqrt_N]]
-            output = mesolve(H, psi0, tlist, [], args, options,
-                             _safe_mode=False)
+            rho0 = qeye(H0.dims[0])
+            output = mesolve(
+                H, rho0, tlist, args=args, options=options,
+                _safe_mode=False)
             return output.states[-1] if len(tlist) == 2 else output.states
 
     else:
@@ -227,6 +228,7 @@ def propagator(H, t, c_op_list=[], args={}, options=None,
             output = parallel_map(_parallel_mesolve, range(N * N),
                                   task_args=(
                                       N, H, tlist, c_op_list, args, options),
+                                  task_kwargs={"dims": H0.dims},
                                   progress_bar=progress_bar, num_cpus=num_cpus)
             for n in range(N * N):
                 for k, t in enumerate(tlist):
@@ -236,10 +238,11 @@ def propagator(H, t, c_op_list=[], args={}, options=None,
             for n in range(N * N):
                 progress_bar.update(n)
                 col_idx, row_idx = np.unravel_index(n, (N, N))
-                rho0 = Qobj(sp.csr_matrix(([1], ([row_idx], [col_idx])),
-                                          shape=(N, N), dtype=complex))
-                output = mesolve(H, rho0, tlist, c_op_list, [], args, options,
-                                 _safe_mode=False)
+                rho0 = projection(N, row_idx, col_idx)
+                rho0.dims = H0.dims
+                output = mesolve(
+                    H, rho0, tlist, c_ops=c_op_list, args=args,
+                    options=options, _safe_mode=False)
                 for k, t in enumerate(tlist):
                     u[:, n, k] = mat2vec(output.states[k].full()).T
             progress_bar.finished()
@@ -302,10 +305,11 @@ def _parallel_sesolve(n, N, H, tlist, args, options):
     return output
 
 
-def _parallel_mesolve(n, N, H, tlist, c_op_list, args, options):
+def _parallel_mesolve(n, N, H, tlist, c_op_list, args, options, dims=None):
     col_idx, row_idx = np.unravel_index(n, (N, N))
-    rho0 = Qobj(sp.csr_matrix(([1], ([row_idx], [col_idx])),
-                              shape=(N, N), dtype=complex))
-    output = mesolve(H, rho0, tlist, c_op_list, [], args, options,
-                     _safe_mode=False)
+    rho0 = projection(N, row_idx, col_idx)
+    rho0.dims = dims
+    output = mesolve(
+        H, rho0, tlist, c_ops=c_op_list, args=args, options=options,
+        _safe_mode=False)
     return output


Test patch:
diff --git a/qutip/tests/test_propagator.py b/qutip/tests/test_propagator.py
--- a/qutip/tests/test_propagator.py
+++ b/qutip/tests/test_propagator.py
@@ -127,5 +127,53 @@ def testPropHDims():
     U = propagator(H,1, unitary_mode='single')
     assert_equal(U.dims,H.dims)
 
+
+def testPropHSuperWithoutCops():
+    "Propagator: super operator without collapse operators"
+    H = tensor(sigmaz(), qeye(2))
+    H = liouvillian(H)
+    tlist = np.linspace(0, 10, 11)
+    Fs = propagator(H, tlist)
+    rho0 = qeye([[2, 2], [2, 2]])
+    expected_Fs = mesolve(H, rho0, tlist).states
+    assert Fs == expected_Fs
+
+
+def testPropHSuperWithoutCopsParallel():
+    "Propagator: super operator without collapse operators using parallel"
+    H = tensor(sigmaz(), qeye(2))
+    H = liouvillian(H)
+    tlist = np.linspace(0, 10, 11)
+    Fs = propagator(H, tlist, parallel=True)
+    rho0 = qeye([[2, 2], [2, 2]])
+    expected_Fs = mesolve(H, rho0, tlist).states
+    for k, _ in enumerate(tlist):
+        assert (Fs[k] - expected_Fs[k]).norm() < 1e-3
+
+
+def testPropHWithCops():
+    "Propagator: with collapse operators"
+    H = tensor(sigmaz(), qeye(2))
+    c_ops = [np.sqrt(1) * tensor(sigmam(), qeye(2))]
+    tlist = np.linspace(0, 10, 11)
+    Fs = propagator(H, tlist, c_op_list=c_ops)
+    rho0 = ket2dm(tensor(basis(2, 0), basis(2, 0))).unit()
+    rho_fs = [vector_to_operator(F * operator_to_vector(rho0)) for F in Fs]
+    expected_rho_fs = mesolve(H, rho0, tlist, c_ops=c_ops).states
+    assert rho_fs == expected_rho_fs
+
+
+def testPropHWithCopsParallel():
+    "Propagator: with collapse operators in parallel"
+    H = tensor(sigmaz(), qeye(2))
+    c_ops = [np.sqrt(1) * tensor(sigmam(), qeye(2))]
+    tlist = np.linspace(0, 10, 11)
+    Fs = propagator(H, tlist, c_op_list=c_ops, parallel=True)
+    rho0 = ket2dm(tensor(basis(2, 0), basis(2, 0))).unit()
+    rho_fs = [vector_to_operator(F * operator_to_vector(rho0)) for F in Fs]
+    expected_rho_fs = mesolve(H, rho0, tlist, c_ops=c_ops).states
+    assert rho_fs == expected_rho_fs
+
+
 if __name__ == "__main__":
     run_module_suite()