Repository: scipy/scipy

Issue:
Improvements to io.loadmat
scipy.io.loadmat is currently very difficult to work with, with confusing options, unusual names, and it does not work particularly well for matlab structs that include cell arrays. 

The second-highest answer in [this stackoverflow post](https://stackoverflow.com/a/29126361/1018861) contains an edit of the selected answer, making .mat files much easier to work with. It would be excellent if it, or a variation of it, could be included in the loadmat function.


### Reproducing code example:
As a simple example, I'm attaching a file produced by Matlab using the following code. The goal with importing into python would be accessing the value "a".
```matlab
field = 'mycell';
value = {{'a','b','c'}};
s = struct(field,value)
```
[matlab.zip (0.4KB)](https://github.com/scipy/scipy/files/1314287/matlab.zip)

Loading it into python using scipy.io's loadmat creates the following object:
```python
>>> import scipy.io as sio
>>> test_with_sio = sio.loadmat("matlab.mat")
>>> test_with_sio
{'__globals__': [],
 '__header__': b'MATLAB 5.0 MAT-file, Platform: PCWIN64, Created on: Tue Sep 19 14:52:28 2017',
 '__version__': '1.0',
 's': array([[ (array([[array(['a'],
       dtype='<U1'), array(['b'],
       dtype='<U1'),
         array(['c'],
       dtype='<U1')]], dtype=object),)]],
       dtype=[('mycell', 'O')])}
```
in order to access the entry containing "a", the following line is necessary:
```python
>>> test_with_sio["s"][0,0][0][0,0][0]
'a'
```

With the edit provided by the stackoverflow (second) answer, the object returned by loadmat is:
```python
{'__globals__': [],
 '__header__': b'MATLAB 5.0 MAT-file, Platform: PCWIN64, Created on: Tue Sep 19 14:52:28 2017',
 '__version__': '1.0',
 's': {'mycell': ['a', 'b', 'c']}}
```
and to access "a" one simply calls:
```python
test_edited["s"]["mycell"][0]
```

I hope I've made a case for cause for improvement.
### Scipy/Numpy/Python version information:
```
<<Output from 'import sys, scipy, numpy; print(scipy.__version__, numpy.__version__, sys.version_info)'>>
0.19.1 1.13.1 sys.version_info(major=3, minor=6, micro=2, releaselevel='final', serial=0)
```




Discussion:
I agree that the resulting object is a bit hard to work with, though I think this example is an especially weird case.

It's a little easier if you use the recarray by name, and if you use the `squeeze_me` keyword:
```python
foo = sio.loadmat('matlab.mat', squeeze_me=True)
foo['s']['mycell'].item()  # array([u'a', u'b', u'c'], dtype=object)
```

The only tricky part here is that `foo['s']['mycell']` is a 0-dimensional object array, wrapping the array of characters we actually want to get to. I'm not sure why that is.
I stumbled today over a much improved version of the one suggested here; it is the first improvement to `scipy.io.loadmat` that really worked smooth for me:

http://blog.nephics.com/2019/08/28/better-loadmat-for-scipy

The blog has no license to it, but I wrote the author, @jcbsv, and his reply is:

> "[...] the SciPy project, may use the code in the blog post
under a liberal OSS license (e.g., MIT or BSD style). But notice
that the code in the blog post is inspired by a Stack Overflow post:
https://stackoverflow.com/a/29126361/572908 (which is licensed
as CC by-sa, attribution required. I assume that a link in the
description/documentation is sufficient.)"

If anyone plans to implement this into SciPy (which would be awesome) I suggest to use Jacob's version over the original version on stackoverflow.

Thanks @jcbsv for making this awesome snippet available!
I think this would be a nice addition. Maybe adding this behavior with a new kwarg (which defaults to the current behavior) could be an option? If scipy devs agree that this is a welcome addition I'd volunteer to contribute a PR - unless someone else wants to do it, such as the original author of the snippet @jcbsv?
In case you need more examples, the [mat files from this website](http://bnci-horizon-2020.eu/database/data-sets) are cell arrays that are basically unreadable with `scipy.io.loadmat` (for humans I mean), whereas the proposed dict alternative makes it really nice and structured.
@cbrnr please go ahead and make the PR
Will do, but I'd like to get feedback from scipy devs first to discuss how to integrate this feature. Currently I see two options:

1. Add a new kwarg to `loadmat`, which if `True` will return the content of the mat file in the new format. This might require testing and excluding certain argument combinations, because currently the conversion only works if `struct_as_record=False` (and I think `squeeze_me=True`).
2. Add a new function which accepts a dict as generated by `loadmat` (with `struct_as_record=False` and `squeeze_me=True`) and returns the new format.

Also, your code snippet currently works only with `MatFile5Reader` objects, but maybe we should also add support for version 4 if that is not too much work.
Hey @cbrnr , that would be great. It might be worth to write an email to the SciPy-dev email list, you might get more feedback from there.
https://mail.python.org/pipermail/scipy-dev/2020-March/024102.html - no response yet unfortunately.
My impression is that a new keyword in `loadmat` would be better than a separate function. The interaction with `struct_as_record` and `squeeze_me` exists either way, so that doesn't look like a good reason for a new function.

@matthew-brett any opinion on this question?


Resolving patch:
diff --git a/scipy/io/matlab/mio.py b/scipy/io/matlab/mio.py
--- a/scipy/io/matlab/mio.py
+++ b/scipy/io/matlab/mio.py
@@ -128,6 +128,12 @@ def loadmat(file_name, mdict=None, appendmat=True, **kwargs):
         MATLAB variables to read from the file. The reader will skip any
         variable with a name not in this sequence, possibly saving some read
         processing.
+    simplify_cells : False, optional
+        If True, return a simplified dict structure (which is useful if the mat
+        file contains cell arrays). Note that this only affects the structure
+        of the result and not its contents (which is identical for both output
+        structures). If True, this automatically sets `struct_as_record` to
+        False and `squeeze_me` to True, which is required to simplify cells.
 
     Returns
     -------
diff --git a/scipy/io/matlab/mio5.py b/scipy/io/matlab/mio5.py
--- a/scipy/io/matlab/mio5.py
+++ b/scipy/io/matlab/mio5.py
@@ -68,7 +68,8 @@
 '''
 
 # Small fragments of current code adapted from matfile.py by Heiko
-# Henkelmann
+# Henkelmann; parts of the code for simplify_cells=True adapted from
+# http://blog.nephics.com/2019/08/28/better-loadmat-for-scipy/.
 
 import os
 import time
@@ -98,11 +99,55 @@
                           NP_TO_MXTYPES, miCOMPRESSED, miMATRIX, miINT8,
                           miUTF8, miUINT32, mxCELL_CLASS, mxSTRUCT_CLASS,
                           mxOBJECT_CLASS, mxCHAR_CLASS, mxSPARSE_CLASS,
-                          mxDOUBLE_CLASS, mclass_info)
+                          mxDOUBLE_CLASS, mclass_info, mat_struct)
 
 from .streams import ZlibInputStream
 
 
+def _has_struct(elem):
+    """Determine if elem is an array and if first array item is a struct."""
+    return (isinstance(elem, np.ndarray) and (elem.size > 0) and
+            isinstance(elem[0], mat_struct))
+
+
+def _inspect_cell_array(ndarray):
+    """Construct lists from cell arrays (loaded as numpy ndarrays), recursing
+    into items if they contain mat_struct objects."""
+    elem_list = []
+    for sub_elem in ndarray:
+        if isinstance(sub_elem, mat_struct):
+            elem_list.append(_matstruct_to_dict(sub_elem))
+        elif _has_struct(sub_elem):
+            elem_list.append(_inspect_cell_array(sub_elem))
+        else:
+            elem_list.append(sub_elem)
+    return elem_list
+
+
+def _matstruct_to_dict(matobj):
+    """Construct nested dicts from mat_struct objects."""
+    d = {}
+    for f in matobj._fieldnames:
+        elem = matobj.__dict__[f]
+        if isinstance(elem, mat_struct):
+            d[f] = _matstruct_to_dict(elem)
+        elif _has_struct(elem):
+            d[f] = _inspect_cell_array(elem)
+        else:
+            d[f] = elem
+    return d
+
+
+def _simplify_cells(d):
+    """Convert mat objects in dict to nested dicts."""
+    for key in d:
+        if isinstance(d[key], mat_struct):
+            d[key] = _matstruct_to_dict(d[key])
+        elif _has_struct(d[key]):
+            d[key] = _inspect_cell_array(d[key])
+    return d
+
+
 class MatFile5Reader(MatFileReader):
     ''' Reader for Mat 5 mat files
     Adds the following attribute to base class
@@ -133,8 +178,8 @@ def __init__(self,
                  matlab_compatible=False,
                  struct_as_record=True,
                  verify_compressed_data_integrity=True,
-                 uint16_codec=None
-                 ):
+                 uint16_codec=None,
+                 simplify_cells=False):
         '''Initializer for matlab 5 file format reader
 
     %(matstream_arg)s
@@ -152,8 +197,8 @@ def __init__(self,
             chars_as_strings,
             matlab_compatible,
             struct_as_record,
-            verify_compressed_data_integrity
-            )
+            verify_compressed_data_integrity,
+            simplify_cells)
         # Set uint16 codec
         if not uint16_codec:
             uint16_codec = sys.getdefaultencoding()
@@ -300,7 +345,10 @@ def get_variables(self, variable_names=None):
                 variable_names.remove(name)
                 if len(variable_names) == 0:
                     break
-        return mdict
+        if self.simplify_cells:
+            return _simplify_cells(mdict)
+        else:
+            return mdict
 
     def list_variables(self):
         ''' list variables from stream '''
diff --git a/scipy/io/matlab/miobase.py b/scipy/io/matlab/miobase.py
--- a/scipy/io/matlab/miobase.py
+++ b/scipy/io/matlab/miobase.py
@@ -340,8 +340,8 @@ def __init__(self, mat_stream,
                  chars_as_strings=True,
                  matlab_compatible=False,
                  struct_as_record=True,
-                 verify_compressed_data_integrity=True
-                 ):
+                 verify_compressed_data_integrity=True,
+                 simplify_cells=False):
         '''
         Initializer for mat file reader
 
@@ -365,6 +365,10 @@ def __init__(self, mat_stream,
             self.chars_as_strings = chars_as_strings
             self.mat_dtype = mat_dtype
         self.verify_compressed_data_integrity = verify_compressed_data_integrity
+        self.simplify_cells = simplify_cells
+        if simplify_cells:
+            self.squeeze_me = True
+            self.struct_as_record = False
 
     def set_matlab_compatible(self):
         ''' Sets options to return arrays as MATLAB loads them '''


Test patch:
diff --git a/scipy/io/matlab/tests/data/testsimplecell.mat b/scipy/io/matlab/tests/data/testsimplecell.mat
new file mode 100644
Binary files /dev/null and b/scipy/io/matlab/tests/data/testsimplecell.mat differ
diff --git a/scipy/io/matlab/tests/test_mio.py b/scipy/io/matlab/tests/test_mio.py
--- a/scipy/io/matlab/tests/test_mio.py
+++ b/scipy/io/matlab/tests/test_mio.py
@@ -1225,3 +1225,13 @@ def test_filenotfound():
     # Check the correct error is thrown
     assert_raises(IOError, loadmat, "NotExistentFile00.mat")
     assert_raises(IOError, loadmat, "NotExistentFile00")
+
+
+def test_simplify_cells():
+    # Test output when simplify_cells=True
+    filename = pjoin(test_data_path, 'testsimplecell.mat')
+    res1 = loadmat(filename, simplify_cells=True)
+    res2 = loadmat(filename, simplify_cells=False)
+    assert_(isinstance(res1["s"], dict))
+    assert_(isinstance(res2["s"], np.ndarray))
+    assert_array_equal(res1["s"]["mycell"], np.array(["a", "b", "c"]))