Repository: qutip/qutip

Issue:
correlation_2op_1t() and spectrum() giving the physically incorrect result unless solver='es'
**Describe the bug**
See discussion here:

https://groups.google.com/g/qutip/c/2lod1DuFZiE/m/y8Bj6dKHDAAJ?pli=1

To my surprise, my PhD student had this problem again today, I am a bit surprised nothing has been done about this. 

At least making `es` the default solver for correlation and spectrum, updating the docs to give a warning to only use the es solver (for now?), and maybe longer term figuring out why the other solvers are not handling it properly

**To Reproduce**

https://github.com/gsteele13/gary-exploring-qutip/blob/master/Correlations.ipynb

```
import numpy as np
import pylab as plt
from qutip import *

N = 5
w = 1 * 2 * np.pi              # 1 Hz oscillator frequency
Q = 40
kappa = w / Q

# Set up some operators
a = destroy(N)
n = num(N)
x = (a + a.dag())/np.sqrt(2)
p = -1j * (a - a.dag())/np.sqrt(2)
H = w * a.dag() * a

c_ops = []

psi0 = fock(N,0)
tlist = np.linspace(0, 10, 500)

corr = correlation_2op_1t(H, psi0, tlist, c_ops, x, x, solver='me')

wlist1, spec1 = spectrum_correlation_fft(tlist, corr)

plt.figure(figsize=(14,8))
plt.plot(tlist, np.real(corr))
plt.plot(tlist, np.imag(corr))
plt.show()

plt.figure(figsize=(14,8))
plt.plot(wlist1, np.abs(spec1))
plt.show()
```

**Expected behavior**
Correlation should have an imaginary component and quantum PSD should not be symmetric about f = 0

**Your Environment**
Software	Version
QuTiP	4.4.1
Numpy	1.17.4
SciPy	1.4.1
matplotlib	3.3.4
Cython	0.29.14
Number of CPUs	4
BLAS Info	INTEL MKL
IPython	7.9.0
Python	3.7.5 (default, Oct 25 2019, 10:52:18) [Clang 4.0.1 (tags/RELEASE_401/final)]
OS	posix [darwin]
Wed Mar 10 08:51:15 2021 CET

**Additional context**
Add any other context about the problem here.



Discussion:
hi gary,  thanks for making this an issue, i think it fell through the cracks on the google groups.  

I think the problem is with how the expectation values are calculated.  Mesolve() returns the correct states, but they are wrongly flagged with isherm=True, even when they are not actually hermitian, so when expect() is called it takes the real part.  I think this incorrect flagging happens here for the output states
https://github.com/qutip/qutip/blob/d285e96b3afc61afd1deceef61d9635f9d9aa505/qutip/mesolve.py#L520

and here for collating the expectation values
https://github.com/qutip/qutip/blob/d285e96b3afc61afd1deceef61d9635f9d9aa505/qutip/mesolve.py#L510
(where only the hermiticity of the operators are checked, not the state.   you can check this by setting x.isherm=False in your code example, which fixes everything)

@Ericgig is there any issue with just removing this isherm=True flag here, and fixing the hermiticity check in the expectation values output call? i guess all this is getting replaced with your solver rewrite anyway, but if not it might make a good ''first issue'' for gsoc?
> hi gary, thanks for making this an issue, i think it fell through the cracks on the google groups.
> 
> I think the problem is with how the expectation values are calculated. Mesolve() returns the correct states, but they are wrongly flagged with isherm=True, even when they are not actually hermitian, so when expect() is called it takes the real part. I think this incorrect flagging happens here for the output states
> https://github.com/qutip/qutip/blob/d285e96b3afc61afd1deceef61d9635f9d9aa505/qutip/mesolve.py#L520
> 

Actually sorry, this is just for the final state.  For the full output list this happens

https://github.com/qutip/qutip/blob/d285e96b3afc61afd1deceef61d9635f9d9aa505/qutip/mesolve.py#L503

where the mc-dm option forces an isherm=true for some reason?  (presumably for speed reasons)
`Qobj` has some fast-path constructors that are basically the solvers saying "this data is valid, just trust me", causing it to skip sanity checks.  There are a few slow points that can be skipped: copying unnecessary data, parsing/verifying the `dims` data, and determining the properties (`isherm`, `iscp`, etc).  All of those add on very non-negligible construction time, but at most points in QuTiP we _know_ the values when passing them.  Old-style code uses these magic "fast-path" constructors, but the new 5.0 branch gets rid of all of them - you can achieve much more, and more explicitly, by using the kwargs, and function-specific logic stays at the call location, rather than being bolted into `Qobj`.

We still want to maintain passing `isherm` to `Qobj` when we know it for speed reasons, especially when we know the most common use of the `Qobj` created by `mesolve` is for finding expectation values.  However, you can set the value based on the hermicity of the input density matrix; if you're passed a true density matrix, you'll always get a Hermitian matrix out, whereas you'll generally be in an unknown state if you're not.  The test can be
```python
def mesolve(H, rho0, tlist, ...):
    hermicity = rho0.isherm or None
    ...
    for t in tlist:
        output.states.append(Qobj(fdata, dims=dims, isherm=hermicity))
```
which forces `isherm=True` if that's _known_, but leaves it unset if not.  This is approximately how we handle things in 5.0, which is in part why we have huge reductions in `Qobj` construction time in the new version.
This is a little tricky, but yes this could be a good first issue.
With master using `fast="mc-dm"`, the `Qobj` constructor ignore `isherm` input.
Also, to add a hermicity check for the state in expect, you need to be careful about column-stacked dm.




Except for setting the type of the `Qobj` (which is deleted at the end of `Qobj.__init__`), there's no need for the 'mc' or 'mc-dm' fast-paths to exist in `Qobj.__init__` at all, even in `master`.  If we were to fix the handling of a passed `type` - nothing else is overwritten if passed explicitly - you could achieve all the same effects at approximately the same speed with correct application of the kwargs, such as
```python
Qobj(data, dims=dims, copy=False, type='oper', isherm=...)
```
providing `data` is actually of a valid (`fast_csr_matrix`) format. Technically the `Qobj.data` attribute will be a new object that wraps the same numpy arrays as was passed, rather than the same `fast_csr_matrix`.

In various forms, `expect` already does check the Hermicity, that's why `mc-dm` is careful to set it to avoid recalculating it.  The expect functions in `qutip.expect` do this directly, and aren't aware of column-stacked density matrices - they'll just raise a TypeError.  `mesolve` and `mcsolve` now bypass the `Qobj` stage and go direct to Cython - in 5.0 it might not be a terrible idea to route everything back through the `Qobj` form, since the performance issues there are solved (a column-stacked dense matrix can be directly wrapped by an f-ordered `Dense` type), and it's the natural central point of the code (and those parts are called from Python-space anyway, so no C concerns).

Here though, the problem is actually that the expectation generator in `mesolve` correctly calculates the dtype of the output expectation array using both the state and the expectation operator, but then it does it incorrectly in the loop, and passes the hermicity down to Cython using only the expectation operator, which promptly throws out the complex part.

So setting `fast='mc-dm'` in `Qobj` is a symptom of the same root cause as the problem here, rather than the cause itself.  At the start of `mesolve` we don't assume that the state is Hermitian, but then inside the integration loop we do.  We can still keep the speedup - if the output was created with a complex dtype, we can always safely say that we're passing in non-Hermitian operators, even if coincidentally at one timestep they happen to be anyway.

See https://github.com/qutip/qutip/blob/d285e96b3afc61afd1deceef61d9635f9d9aa505/qutip/mcsolve.py#L525-L529 and https://github.com/qutip/qutip/blob/d285e96b3afc61afd1deceef61d9635f9d9aa505/qutip/mesolve.py#L509-L511
For a first issue, it could be solved with little changes or by going down the rabbit-hole.
There could also be a discussion about always returning complex expectation values.
There's no rabbit hole - it's just a bug on line 511 of `mesolve.py` and 526 of `mcsolve.py`.  The output dtype is correctly calculated, it's just a bug in generating values for the output array.  There's a similar bug in the states output caused by the use of `fast='mc-dm'` that requires a more effort to fix, and should be at some point, but that's not so important right now.

Having `qutip.expect` return real values for Hermitian operators and states I'd say is the correct behaviour.  It's by far the most common use-case, and it allows them to be directly plotted without generating a load of warnings.

Resolving patch:
diff --git a/qutip/correlation.py b/qutip/correlation.py
--- a/qutip/correlation.py
+++ b/qutip/correlation.py
@@ -116,7 +116,7 @@ def correlation_2op_1t(H, state0, taulist, c_ops, a_op, b_op,
     Returns
     -------
     corr_vec : ndarray
-        An array of correlation values for the times specified by `tlist`.
+        An array of correlation values for the times specified by `taulist`.
 
     References
     ----------
diff --git a/qutip/mesolve.py b/qutip/mesolve.py
--- a/qutip/mesolve.py
+++ b/qutip/mesolve.py
@@ -510,22 +510,25 @@ def get_curr_state_data(r):
 
         if opt.store_states or expt_callback:
             cdata = get_curr_state_data(r)
+            fdata = dense2D_to_fastcsr_fmode(cdata, size, size)
 
-        if opt.store_states:
-            if issuper(rho0):
-                fdata = dense2D_to_fastcsr_fmode(cdata, size, size)
-                output.states.append(Qobj(fdata, dims=dims))
+            # Try to guess if there is a fast path for rho_t
+            if issuper(rho0) or not rho0.isherm:
+                rho_t = Qobj(fdata, dims=dims)
             else:
-                fdata = dense2D_to_fastcsr_fmode(cdata, size, size)
-                output.states.append(Qobj(fdata, dims=dims, fast="mc-dm"))
+                rho_t = Qobj(fdata, dims=dims, fast="mc-dm")
+
+        if opt.store_states:
+            output.states.append(rho_t)
 
         if expt_callback:
             # use callback method
-            output.expect.append(e_ops(t, Qobj(cdata, dims=dims)))
+            output.expect.append(e_ops(t, rho_t))
 
         for m in range(n_expt_op):
             output.expect[m][t_idx] = expect_rho_vec(e_ops_data[m], r.y,
-                                                     e_ops[m].isherm)
+                                                     e_ops[m].isherm
+                                                     and rho0.isherm)
 
         if t_idx < n_tsteps - 1:
             r.integrate(r.t + dt[t_idx])
@@ -534,6 +537,7 @@ def get_curr_state_data(r):
 
     if opt.store_final_state:
         cdata = get_curr_state_data(r)
-        output.final_state = Qobj(cdata, dims=dims, isherm=True)
+        output.final_state = Qobj(cdata, dims=dims,
+                                  isherm=rho0.isherm or None)
 
     return output


Test patch:
diff --git a/qutip/tests/test_correlation.py b/qutip/tests/test_correlation.py
--- a/qutip/tests/test_correlation.py
+++ b/qutip/tests/test_correlation.py
@@ -33,6 +33,7 @@
 
 import pytest
 import functools
+from itertools import product
 import numpy as np
 import qutip
 
@@ -58,6 +59,7 @@ def test_correlation_solver_equivalence(solver, start, legacy):
     system.
     """
     a = qutip.destroy(_equivalence_dimension)
+    x = ( a +a.dag() )/np.sqrt(2)
     H = a.dag() * a
     G1 = 0.75
     n_th = 2
@@ -251,3 +253,56 @@ def test_hamiltonian_order_unimportant():
     backwards = qutip.correlation_2op_2t(H[::-1], start, times, times, [sp],
                                          sp.dag(), sp)
     np.testing.assert_allclose(forwards, backwards, atol=1e-6)
+
+
+@pytest.mark.parametrize(['solver', 'state'], [
+    pytest.param('me', _equivalence_fock, id="me-ket"),
+    pytest.param('me', _equivalence_coherent, id="me-dm"),
+    pytest.param('me', None, id="me-steady"),
+    pytest.param('es', _equivalence_fock, id="es-ket"),
+    pytest.param('es', _equivalence_coherent, id="es-dm"),
+    pytest.param('es', None, id="es-steady"),
+    pytest.param('mc', _equivalence_fock, id="mc-ket",
+                 marks=[pytest.mark.slow]),
+])
+@pytest.mark.parametrize("is_e_op_hermitian", [True, False],
+                         ids=["hermitian", "nonhermitian"])
+@pytest.mark.parametrize("w", [1, 2])
+@pytest.mark.parametrize("gamma", [1, 10])
+def test_correlation_2op_1t_known_cases(solver,
+                                        state,
+                                        is_e_op_hermitian,
+                                        w,
+                                        gamma,
+                                       ):
+    """This test compares the output correlation_2op_1 solution to an analytical
+    solution."""
+
+    a = qutip.destroy(_equivalence_dimension)
+    x = (a + a.dag())/np.sqrt(2)
+
+    H = w * a.dag() * a
+
+    a_op = x if is_e_op_hermitian else a
+    b_op = x if is_e_op_hermitian else a.dag()
+    c_ops = [np.sqrt(gamma) * a]
+
+    times = np.linspace(0, 1, 30)
+
+    # Handle the case state==None when computing expt values
+    rho0 = state if state else qutip.steadystate(H, c_ops)
+    if is_e_op_hermitian:
+        # Analitycal solution for x,x as operators.
+        base = 0
+        base += qutip.expect(a*x, rho0)*np.exp(-1j*w*times - gamma*times/2)
+        base += qutip.expect(a.dag()*x, rho0)*np.exp(1j*w*times - gamma*times/2)
+        base /= np.sqrt(2)
+    else:
+        # Analitycal solution for a,adag as operators.
+        base = qutip.expect(a*a.dag(), rho0)*np.exp(-1j*w*times - gamma*times/2)
+
+    cmp = qutip.correlation_2op_1t(H, state, times, c_ops, a_op, b_op, solver=solver)
+
+    np.testing.assert_allclose(base, cmp, atol=0.25 if solver == 'mc' else 2e-5)
+
+
diff --git a/qutip/tests/test_mesolve.py b/qutip/tests/test_mesolve.py
--- a/qutip/tests/test_mesolve.py
+++ b/qutip/tests/test_mesolve.py
@@ -920,5 +920,55 @@ def f(t, args):
             msg="evolution with feedback not proceding as expected")
 
 
+def test_non_hermitian_dm():
+    """Test that mesolve works correctly for density matrices that are
+    not Hermitian.
+    See Issue #1460
+    """
+    N = 2
+    a = destroy(N)
+    x = (a + a.dag())/np.sqrt(2)
+    H = a.dag() * a
+
+    # Create non-Hermitian initial state.
+    rho0 = x*fock_dm(N, 0)
+
+    tlist = np.linspace(0, 0.1, 2)
+
+    options = Options()
+    options.store_final_state = True
+    options.store_states = True
+
+    result = mesolve(H, rho0, tlist, e_ops=[x], options=options)
+
+    msg = ('Mesolve is not working properly with a non Hermitian density' +
+       ' matrix as input. Check computation of '
+      )
+
+    imag_part = np.abs(np.imag(result.expect[0][-1]))
+    # Since we used an initial state that is not Hermitian, the expectation of
+    # x must be imaginary for t>0.
+    assert_(imag_part > 0,
+            msg + "expectation values. They should be imaginary")
+
+    # Check that the output state is not hermitian since the input was not
+    # Hermitian either.
+    assert_(not result.final_state.isherm,
+            msg + " final density  matrix. It should not be hermitian")
+    assert_(not result.states[-1].isherm,
+            msg + " states. They should not be hermitian.")
+
+    # Check that when suing a callable we get imaginary expectation values.
+    def callable_x(t, rho):
+        "Dummy callable_x expectation operator."
+        return expect(rho, x)
+    result = mesolve(H, rho0, tlist, e_ops=callable_x)
+
+    imag_part = np.abs(np.imag(result.expect[-1]))
+    assert_(imag_part > 0,
+            msg + "expectation values when using callable operator." +
+            "They should be imaginary.")
+
+
 if __name__ == "__main__":
     run_module_suite()