Repository: qutip/qutip

Issue:
Segmentation fault in mesolve if density matrix was constructed with flat aray
### Bug Description

Construct a density matrix, via some array creation method (in this case numpy - haven't checked with others). Create a `rho0` for mesolve to use with the flat array while explicitly specifying the matrix dimensions in the `dims` parameter. Upon passing the density matrix to `mesolve` a segmentation fault occurs. A simple proof of concept is included below. `np.random.random` was used to fill the matrix in this case, but the issue was encountered using valid density matricies, and the issue seems to be unrelated to whether the matrix is valid or not as a density matrix.

### Code to Reproduce the Bug

```shell
import qutip as qtip
import numpy as np

dim = 100 # Small dims don't work. I assume python has some buffer after the array
H = qtip.Qobj(np.identity(dim),dims=[[dim],[dim]])
rho0 = np.random.random([dim**2])
rho0 = qtip.Qobj(rho0.flatten(),dims=[[dim],[dim]]) # This fails
# rho0 = qtip.Qobj(rho0.reshape([dim,dim]),dims=[[dim],[dim]]) # This works
qtip.mesolve(rho0=rho0,H=H,tlist=np.linspace(0,10,1),progress_bar=qtip.ui.EnhancedTextProgressBar())
```


### Code Output

```shell
[          0%           ] Elapsed 0.00s / Remaining 00:00:00:00[1]    3281154 segmentation fault (core dumped)
```


### Expected Behaviour

The code should exit without failure or throw a python error elaborating on the issue

### Your Environment

```shell
QuTiP Version:      4.6.2
Numpy Version:      1.21.4
Scipy Version:      1.6.0
Cython Version:     0.29.21
Matplotlib Version: 3.4.3
Python Version:     3.9.7
Number of CPUs:     4
BLAS Info:          OPENBLAS
OPENMP Installed:   False
INTEL MKL Ext:      False
Platform Info:      Linux (x86_64)
Installation path:  /home/lala5th/.local/lib/python3.9/site-packages/qutip
```


### Additional Context

I ran the code through `gdb` to somewhat narrow down the cause of the issue. It seems that `dense2D_to_fastcsr_fmode` causes the segmentation fault, but the parameters passed to it seem fine at first glance, since I expect `ncols = 10000`. While I don't know how useful it is I will put the top of the stack trace (after this the only the call to `mesolve` is relevant in the stack trace).

```
#0  __pyx_pf_5qutip_2cy_9spconvert_6dense2D_to_fastcsr_fmode (__pyx_self=<optimised out>, __pyx_v_ncols=10000, __pyx_v_nrows=<optimised out>, __pyx_v_mat=...) at qutip/cy/spconvert.cpp:9472
#1  __pyx_pw_5qutip_2cy_9spconvert_7dense2D_to_fastcsr_fmode (__pyx_self=<optimised out>, __pyx_args=<optimised out>, __pyx_kwds=<optimised out>) at qutip/cy/spconvert.cpp:9237
#2  0x000000000053a8eb in cfunction_call (func=<built-in function dense2D_to_fastcsr_fmode>, args=<optimised out>, kwargs=<optimised out>) at ../Objects/methodobject.c:543
#3  0x000000000051c24b in _PyObject_MakeTpCall (tstate=0x9561b0, callable=<built-in function dense2D_to_fastcsr_fmode>, args=<optimised out>, nargs=<optimised out>, keywords=<optimised out>) at ../Objects/call.c:191
#4  0x0000000000516021 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimised out>, args=0x1c16930, callable=<built-in function dense2D_to_fastcsr_fmode>, tstate=<optimised out>) at ../Include/cpython/abstract.h:116
#5  _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimised out>, args=0x1c16930, callable=<built-in function dense2D_to_fastcsr_fmode>, tstate=<optimised out>) at ../Include/cpython/abstract.h:103
#6  PyObject_Vectorcall (kwnames=0x0, nargsf=<optimised out>, args=0x1c16930, callable=<built-in function dense2D_to_fastcsr_fmode>) at ../Include/cpython/abstract.h:127
#7  call_function (kwnames=0x0, oparg=<optimised out>, pp_stack=<synthetic pointer>, tstate=0x9561b0) at ../Python/ceval.c:5075
#8  _PyEval_EvalFrameDefault (tstate=<optimised out>, f=<optimised out>, throwflag=<optimised out>) at ../Python/ceval.c:3518
#9  0x000000000050f5e9 in _PyEval_EvalFrame (throwflag=0, 
    f=Frame 0x1c166f0, for file /home/lala5th/.local/lib/python3.9/site-packages/qutip/mesolve.py, line 779, in _generic_ode_solve (func=<built-in method mul_vec of qutip.cy.cqobjevo.CQobjCte object at remote 0x7fffd42248e0>, ode_args=(), rho0=<Qobj(_isherm=None, _type='oper', superrep=None, _isunitary=None, _data=<fast_csr_matrix(data=<numpy.ndarray at remote 0x7ffff3f918d0>, indices=<numpy.ndarray at remote 0x7ffff3f91570>, indptr=<numpy.ndarray at remote 0x7ffff3f91030>, _shape=(10000, 1), maxprint=50, format='csr') at remote 0x7fffe022a700>, dims=[[100], [100]]) at remote 0x7ffff7180df0>, tlist=<numpy.ndarray at remote 0x7ffff3f91930>, e_ops=[], opt=<Options(atol=<float at remote 0x7fffd51cfb70>, rtol=<float at remote 0x7fffd51cfb10>, method='adams', nsteps=1000, first_step=0, min_step=0, max_step=0, order=12, average_states=False, average_expect=True, ntraj=500, seeds=None, tidy=True, rhs_with_state=False, rhs_reuse=False, rhs_filename=None, mc_corr_eps=<float at remote 0x7fffd51cf9f0>, num_cpus=<numpy.int64 a...(truncated), tstate=0x9561b0) at ../Include/internal/pycore_ceval.h:40
```


Discussion:
@Lala5th Thank you for reporting this.

It looks like the issue is just hat `mesolve` calculates the `size` variable incorrectly for a flat array and the following diff removes the segfault for me:

```diff
diff --git a/qutip/mesolve.py b/qutip/mesolve.py
index f6527299..25e9f3cd 100644
--- a/qutip/mesolve.py
+++ b/qutip/mesolve.py
@@ -435,6 +435,8 @@ def _generic_ode_solve(func, ode_args, rho0, tlist, e_ops, opt,
     output.solver = "mesolve"
     output.times = tlist
     size = rho0.shape[0]
+    if rho0.shape[1] == 1:
+        size = int(np.sqrt(size))
 
     initial_vector = rho0.full().ravel('F')
```

I agree that we should not segfault. Probably the minimal patch is to raise an exception if `rho0.shape[0] != rho0.shape[1]`. If you agree, I can open a small PR (or you can if you would like to).


@hodgestar I agree, that an exception would suffice, my issue was mainly that segfaults are hard to debug, especially if the offending code can be removed from where the crash occurs. If you don't mind opening the PR I then I think that would be the best since it is your fix.
I think the bug is bigger than that. `Qobj` never ensure that the shape and dims match but in many places, solver check only the dims and not the shape. Here `rho` is shaped as an `operator-ket` but with `oper` dims, so it slip through the dims check. I wouldn't be surprised that we can cause other segfault with `Qobj` where the shape and dims don't match. 

It would be safer to add a check when entering cython code.
@Ericgig That is true. Possibly a check in `Qobj` constructor, would be the best to avoid mismatched `Qobj`s to be created in the first place. A rudimentary fix might be something like:

```diff
diff --git a/qutip/qobj.py b/qutip/qobj.py
index bf82a4a6..6c7c630d 100644
--- a/qutip/qobj.py
+++ b/qutip/qobj.py
@@ -310,6 +310,11 @@ class Qobj(object):
             )
             self.dims = [[int(inpt.shape[0])], [int(inpt.shape[1])]]
 
+        if self._data.shape[0] != np.prod(self.dims[0]) or \
+           self._data.shape[1] != np.prod(self.dims[1]):
+           raise ValueError(f"Qobj has mismatched dims {self.dims} " +\
+                            f"and shape {self._data.shape}")
+
         if type == 'super':
             # Type is not super, i.e. dims not explicitly passed, but oper-like
             # shape.
```
If you think this would be a good fix I can open a PR.
A check in `Qobj` would be great, but we have `enr_...` states and operator that have shape and dims not matching and are somewhat supported. This check would refuse those states. The best would be to improve dims to support these spaces, but it's only planned for v5.

If you can make a PR that do not break these `enr_` functions, it would be appreciated.
I am unfamiliar with the `enr_` functions, but I modified the check so that it does not have to be a perfect match. Now the requirement is that the shape of the underlying data has to be smaller (at each dimension) than the dimension it represents. I checked and `enr_fock` and `enr_thermal_dm` are not broken by this change, but I might be missing some other function that is broken by this change. I'll open a PR and attach it to this issue.

```diff
diff --git a/qutip/qobj.py b/qutip/qobj.py
index bf82a4a6..21461541 100644
--- a/qutip/qobj.py
+++ b/qutip/qobj.py
@@ -310,6 +310,11 @@ class Qobj(object):
             )
             self.dims = [[int(inpt.shape[0])], [int(inpt.shape[1])]]
 
+        if self._data.shape[0] > np.prod(self.dims[0]) or \
+           self._data.shape[1] > np.prod(self.dims[1]):
+           raise ValueError(f"Qobj has smaller dims {self.dims} " +\
+                            f"than underlying shape {self._data.shape}")
+
         if type == 'super':
             # Type is not super, i.e. dims not explicitly passed, but oper-like
             # shape.
```

Resolving patch:
diff --git a/qutip/qobj.py b/qutip/qobj.py
--- a/qutip/qobj.py
+++ b/qutip/qobj.py
@@ -328,6 +328,17 @@ def __init__(self, inpt=None, dims=None, shape=None,
             if self.type == 'super' and self.superrep is None:
                 self.superrep = 'super'
 
+        # While the obvious check would be != that would fail valid
+        # use cases such as enr_fock and other enr_ functions.
+        # This does leave open the possibility of data still being
+        # misused such as Qobj(complex[n**2][1], dims = [[n],[n]])
+        if (self._data.shape[0] > np.prod(np.hstack(self.dims[0])) or
+           self._data.shape[1] > np.prod(np.hstack(self.dims[1]))) and \
+           self.type != 'super':
+
+            raise ValueError(f"Qobj has smaller dims {self.dims} " +
+                             f"than underlying shape {self._data.shape}")
+
         # clear type cache
         self._type = None
 


Test patch:
diff --git a/qutip/tests/test_bofin_solvers.py b/qutip/tests/test_bofin_solvers.py
--- a/qutip/tests/test_bofin_solvers.py
+++ b/qutip/tests/test_bofin_solvers.py
@@ -179,13 +179,13 @@ def mk_ados(self, bath_dims, max_depth):
 
     def mk_rho_and_soln(self, ados, rho_dims):
         n_ados = len(ados.labels)
-        ado_soln = np.random.rand(n_ados, *[2**d for d in rho_dims])
-        rho = Qobj(ado_soln[0, :], dims=[2, 2])
+        ado_soln = np.random.rand(n_ados, *[np.product(d) for d in rho_dims])
+        rho = Qobj(ado_soln[0, :], dims=rho_dims)
         return rho, ado_soln
 
     def test_create(self):
         ados = self.mk_ados([2, 3], max_depth=2)
-        rho, ado_soln = self.mk_rho_and_soln(ados, [2, 2])
+        rho, ado_soln = self.mk_rho_and_soln(ados, [[2], [2]])
         ado_state = HierarchyADOsState(rho, ados, ado_soln)
         assert ado_state.rho == rho
         assert ado_state.labels == ados.labels
@@ -195,7 +195,7 @@ def test_create(self):
 
     def test_extract(self):
         ados = self.mk_ados([2, 3], max_depth=2)
-        rho, ado_soln = self.mk_rho_and_soln(ados, [2, 2])
+        rho, ado_soln = self.mk_rho_and_soln(ados, [[2], [2]])
         ado_state = HierarchyADOsState(rho, ados, ado_soln)
         ado_state.extract((0, 0)) == rho
         ado_state.extract(0) == rho