Repository: mne-tools/mne-bids

Issue:
write_raw_bids does not produce a valid SET file
#### Describe the bug

The function `write_raw_bids` does not produce a valid `.set` file that can be read by EEGLAB. When importing a `.set` file that was generated by `write_raw_bids` in EEGLAB, we systematically get the following error:

![Screen Shot 2022-03-30 at 10 34 08 AM](https://user-images.githubusercontent.com/1402456/160860302-e794476b-7a7b-4d52-9b06-10154efb5d92.png)


#### Steps to reproduce

This has been tested on the `eeglab_data.set` sample file that comes with EEGLAB sample data.

```
>>> import mne
>>> import os
>>> from mne_bids import write_raw_bids, BIDSPath
>>> file="/Users/cmadjar/Documents/MATLAB/eeglab2022.0/sample_data/eeglab_data.set"
>>> bids_directory = "/Users/cmadjar/Desktop/eeg_test_data/out_test/123"
>>> raw = mne.io.read_raw_eeglab(input_fname=file, preload=False, verbose=True)
Reading /Users/cmadjar/Documents/MATLAB/eeglab2022.0/sample_data/eeglab_data.fdt
>>> os.makedirs(bids_directory + os.path.sep, exist_ok=True)
>>> bids_directory = bids_directory + os.path.sep
>>> bids_root = bids_directory
>>> bids_basename = BIDSPath(subject='001', task='test', root=bids_root, acquisition='eeg', run=None)
>>> bids_basename.update(session='V1')
>>> write_raw_bids(raw, bids_basename, allow_preload=False, overwrite=False, verbose=True)
```


#### Additional information

This issue has been reproduced over several versions of mne and mne_bids and several OS.

>>> mne.sys_info()
Platform:         macOS-11.3-x86_64-i386-64bit
Python:           3.9.12 (main, Mar 26 2022, 15:52:10)  [Clang 13.0.0 (clang-1300.0.29.30)]
Executable:       /Users/cmadjar/Development/python_venv/python_intel_3.9.4/eeg2bids_py3.9.12/bin/python
CPU:              i386: 8 cores
Memory:           Unavailable (requires "psutil" package)
mne:              1.0.0
numpy:            1.21.5 {blas=openblas, lapack=openblas}
scipy:            1.8.0
matplotlib:       3.5.1 {backend=MacOSX}

sklearn:          1.0.2
numba:            0.55.1
nibabel:          Not found
nilearn:          Not found
dipy:             Not found
cupy:             Not found
pandas:           1.4.1
pyvista:          Not found
pyvistaqt:        Not found
ipyvtklink:       Not found
vtk:              Not found
PyQt5:            Not found
ipympl:           Not found
pooch:            v1.6.0

mne_bids:         0.10
mne_nirs:         Not found
mne_features:     0.2
mne_qt_browser:   Not found
mne_connectivity: Not found


Discussion:
Hello! 👋 Thanks for opening your first issue here! ❤️ We will try to get back to you soon. 🚴🏽‍♂️

Investigating a little deeper, I noticed that the `.set` file generated by `write_raw_bids` does not have the events properly encoded. Matlab expects the `EEG.event` variable to be a `structure` but they are `cells`, which causes EEGLAB to crash when loading the file.

Example of a file that works:
```
>> EEG.event

ans = 

  1×2670 struct array with fields:

    description
    begintime
    classid
    code
    duration
    label
    relativebegintime
    sourcedevice
    name
    tracktype
    latency
    type
    mffkeys
    mffkey_gidx
    mffkeysbackup
    mffkey_age_
    mffkey_exp_
    mffkey_hand
    mffkey_sex_
    mffkey_subj
    mffkey_cel
    mffkey_obs
    mffkey_spc
    mffkey_pos
    mffkey_argu
    mffkey_rsp
    mffkey_eval
    mffkey_rtim
    mffkey_trl
    mffkey_blk1
```

Example of EEG.event for a file generated by `write_raw_bids`:
```
ans =

  1×154 cell array

  Columns 1 through 8

    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}

  Columns 9 through 16

    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}

  Columns 17 through 24

    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}

  Columns 25 through 32

    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}

  Columns 33 through 40

    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}

  Columns 41 through 48

    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}

  Columns 49 through 56

    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}

  Columns 57 through 64

    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}

  Columns 65 through 72

    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}

  Columns 73 through 80

    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}

  Columns 81 through 88

    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}

  Columns 89 through 96

    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}

  Columns 97 through 104

    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}

  Columns 105 through 112

    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}

  Columns 113 through 120

    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}

  Columns 121 through 128

    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}

  Columns 129 through 136

    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}

  Columns 137 through 144

    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}

  Columns 145 through 152

    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}    {1×1 struct}

  Columns 153 through 154

    {1×1 struct}    {1×1 struct}
```

In case that piece of information helps with the debugging.
thanks for the report! maybe related to #654

In general, I guess it'd be best to switch over to using MNE-Python's ["export" functionality](https://mne.tools/stable/generated/mne.export.export_raw.html#mne.export.export_raw) ... one place to fix all issues and all downstream apps benefit from improvements.
I tested the export functionality you mentioned and it resolves the issue of the `.set` file. However, I believe we need the other functionalities of `write_raw_bids` that generates all the other BIDS files automatically.

Is there a way to modify the `write_raw_bids` function so that it calls the export eeglab for set files? Where should I look for that piece of code?

We could always run `write_raw_bids` and then re-export the set file with the export functionality of mne-python but that is not very pretty...
> I tested the export functionality you mentioned and it resolves the issue of the `.set` file. However, I believe we need the other functionalities of `write_raw_bids` that generates all the other BIDS files automatically.

What @sappelhoff meant is that in MNE-BIDS / `write_raw_bids`, we probably shouldn't do our own, custom EEGLAB file handling, but rather rely on what's already there in MNE-Python. It would require some code changes to MNE-BIDS, and should be transparent to users.
+1

> Message ID: ***@***.***>
>

> This has been tested on the eeglab_data.set sample file that comes with EEGLAB sample data.

@cmadjar one question about this: is `eeglab_data.set` accompanied by an `eeglab_data.fdt` oder `eeglab_data.dat` file? Because if **not**, then I am at a loss where the issue comes from. 

You see, for stand-alone `.set` files, we simply copy the file over -- **without** reading and manipulating with numpy or similar. 

relevant code snippet: https://github.com/mne-tools/mne-bids/blob/213c3789390dfacc285418d8f54c6575ca104e7f/mne_bids/copyfiles.py#L558-L560

we only do some editing if we encounter a pair of EEGLAB files (.set and .fdt), because here we need to edit the string in the .set file that points to the .fdt file.

Resolving patch:
diff --git a/mne_bids/copyfiles.py b/mne_bids/copyfiles.py
--- a/mne_bids/copyfiles.py
+++ b/mne_bids/copyfiles.py
@@ -29,6 +29,7 @@
 
 from mne_bids.path import BIDSPath, _parse_ext, _mkdir_p
 from mne_bids.utils import _get_mrk_meas_date, _check_anonymize
+import numpy as np
 
 
 def _copytree(src, dst, **kwargs):
@@ -495,12 +496,11 @@ def copyfile_edf(src, dest, anonymize=None):
 
 
 def copyfile_eeglab(src, dest):
-    """Copy a EEGLAB files to a new location and adjust pointer to '.fdt' file.
+    """Copy an EEGLAB file to a new location.
 
-    Some EEGLAB .set files come with a .fdt binary file that contains the data.
-    When moving a .set file, we need to check for an associated .fdt file and
-    move it to an appropriate location as well as update an internal pointer
-    within the .set file.
+    If the EEGLAB ``.set`` file comes with an accompanying ``.fdt`` binary file
+    that contains the actual data, this function will copy this file, too, and
+    update all internal pointers in the new ``.set`` file.
 
     Parameters
     ----------
@@ -529,27 +529,40 @@ def copyfile_eeglab(src, dest):
                          f' but got {ext_src}, {ext_dest}')
 
     # Load the EEG struct
+    # NOTE: do *not* simplify cells, because this changes the underlying
+    # structure and potentially breaks re-reading of the file
     uint16_codec = None
-    eeg = loadmat(file_name=src, simplify_cells=True,
+    eeg = loadmat(file_name=src, simplify_cells=False,
                   appendmat=False, uint16_codec=uint16_codec)
     oldstyle = False
     if 'EEG' in eeg:
         eeg = eeg['EEG']
         oldstyle = True
 
-    if isinstance(eeg['data'], str):
+    try:
         # If the data field is a string, it points to a .fdt file in src dir
-        fdt_fname = eeg['data']
-        assert fdt_fname.endswith('.fdt')
-        head, tail = op.split(src)
+        if isinstance(eeg['data'][0, 0][0], str):
+            has_fdt_link = True
+    except IndexError:
+        has_fdt_link = False
+
+    if has_fdt_link:
+        fdt_fname = eeg['data'][0, 0][0]
+
+        assert fdt_fname.endswith('.fdt'), f'Unexpected fdt name: {fdt_fname}'
+        head, _ = op.split(src)
         fdt_path = op.join(head, fdt_fname)
 
         # Copy the .fdt file and give it a new name
-        sh.copyfile(fdt_path, fname_dest + '.fdt')
+        fdt_name_new = fname_dest + '.fdt'
+        sh.copyfile(fdt_path, fdt_name_new)
 
         # Now adjust the pointer in the .set file
-        head, tail = op.split(fname_dest + '.fdt')
-        eeg['data'] = tail
+        # NOTE: Clunky numpy code is to match MATLAB structure for "savemat"
+        _, tail = op.split(fdt_name_new)
+        new_value = np.empty((1, 1), dtype=object)
+        new_value[0, 0] = np.atleast_1d(np.array(tail))
+        eeg['data'] = new_value
 
         # Save the EEG dictionary as a Matlab struct again
         mdict = dict(EEG=eeg) if oldstyle else eeg


Test patch:
diff --git a/mne_bids/tests/test_copyfiles.py b/mne_bids/tests/test_copyfiles.py
--- a/mne_bids/tests/test_copyfiles.py
+++ b/mne_bids/tests/test_copyfiles.py
@@ -11,7 +11,6 @@
 import pytest
 
 import mne
-from mne.fixes import _compare_version
 from mne.datasets import testing
 from mne_bids import BIDSPath
 from mne_bids.path import _parse_ext
@@ -198,16 +197,10 @@ def test_copyfile_edfbdf_uppercase(tmp_path):
                           'test_raw_2021.set'))
 def test_copyfile_eeglab(tmp_path, fname):
     """Test the copying of EEGlab set and fdt files."""
-    if (
-        fname == 'test_raw_chanloc.set' and
-        _compare_version(testing.get_version(), '<', '0.112')
-    ):
-        return
-
     bids_root = str(tmp_path)
     data_path = op.join(testing.data_path(), 'EEGLAB')
     raw_fname = op.join(data_path, fname)
-    new_name = op.join(bids_root, 'tested_conversion.set')
+    new_name = op.join(bids_root, f'CONVERTED_{fname}.set')
 
     # IO error testing
     with pytest.raises(ValueError, match="Need to move data with same ext"):