Repository: mne-tools/mne-bids

Issue:
copyfile_eeglab not working
### Description

I am currently attempting to copy EEGLAB files via copyfiles_eeglab with a new (BIDS compliant) name. The files are moved and renamed according to the specifications. However, when trying to open those files with MNE there is an error message:

```
Traceback (most recent call last):

  File "<ipython-input-17-8c31a528db9f>", line 1, in <module>
    raw_bids = mne.io.read_raw_eeglab('sub-001_ses-D1_task-RS2_run-02_eeg.set')

  File "C:\Users\Nils Rosjat\Anaconda3\lib\site-packages\mne\io\eeglab\eeglab.py", line 222, in read_raw_eeglab
    eog=eog, verbose=verbose, uint16_codec=uint16_codec)

  File "<decorator-gen-216>", line 24, in __init__

  File "C:\Users\Nils Rosjat\Anaconda3\lib\site-packages\mne\io\eeglab\eeglab.py", line 326, in __init__
    info, eeg_montage, update_ch_names = _get_info(eeg, eog=eog)

  File "C:\Users\Nils Rosjat\Anaconda3\lib\site-packages\mne\io\eeglab\eeglab.py", line 145, in _get_info
    ch_names, eeg_montage = _get_eeg_montage_information(eeg, has_pos)

  File "C:\Users\Nils Rosjat\Anaconda3\lib\site-packages\mne\io\eeglab\eeglab.py", line 123, in _get_eeg_montage_information
    ch_pos=dict(zip(ch_names, np.array(pos))),

TypeError: unhashable type: 'numpy.ndarray'
```

When comparing the original MATLAB structure:
![before](https://user-images.githubusercontent.com/34684014/101466558-93c8cd00-3941-11eb-9a90-c29db305b6a5.png)

with the newly created structure:
![after](https://user-images.githubusercontent.com/34684014/101466607-a216e900-3941-11eb-9979-26624a9e0f4b.png)

I saw that several entries were replaced by numpy array objects. This seems to happen already in

https://github.com/mne-tools/mne-bids/blob/3d96c5b2af004b57085b5c9bf884e18279707a3c/mne_bids/copyfiles.py#L531-L532

When replacing loadmat with `read_mat` (from mne.externals.pymatreader) this seems to work better, but still after saving the dict with `savemat` the same issue remains.


### Script used for conversion

**mne_bids version: 0.5**

```python
import os.path as op
import os

import mne

from mne_bids import BIDSPath
from mne_bids.copyfiles import copyfile_eeglab

parent_dir = op.abspath('..')
data_path = op.join(parent_dir,'data')

sub_dirs = ['001']
sessions = ['D1']
tasks = ['Tap']

for sub_dir in sub_dirs:
    for session in sessions:
        for task in tasks:
                sub_path = op.join(data_path, sub_dir, 'ses-' + session)
                sub_id = os.listdir(sub_path)[0].split('_')[0]
                set_name = sub_id + '_' + session + '_' + task + '_Nils_IntRef_CSD.set'
                set_file = op.join(data_path, sub_dir, 'ses-' + session,set_name)
                set_rename = 'sub-' + sub_dir + '_ses-' + session + '_task-' + task + '_desc-CSD.set'
                set_file_renamed = op.join(data_path, 'sub-' + sub_dir, 'ses-' + session, 'eeg', set_rename)
                copyfile_eeglab(set_file, set_file_renamed)
```




Discussion:
Thanks for the thorough report @nrosjat! What's surprising to me is, that in our test for the `copyfile_eeglab` function, this issue does not come up.

See: https://github.com/mne-tools/mne-bids/blob/3d96c5b2af004b57085b5c9bf884e18279707a3c/mne_bids/tests/test_copyfiles.py#L184-L205

and that's although in that test we also check a `.set` + `.fdt` combination of files :thinking: 
I also see that [`loadmat`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.io.loadmat.html) says that only: `v4 (Level 1.0), v6 and v7 to 7.2 matfiles are supported.`

So recent .mat files won't be supported by mne-bids :thinking: I don't know whether EEGLAB specified a specific MAT-file version when writing the .set and .fdt files.

Let's try our chance with @cbrnr: Do you have EEGLAB set+fdt experience and immediately see the issue here?


EDIT:

I am confused ... is this the function EEGLAB uses for saving `.set` files? --> https://github.com/sccn/eeglab/blob/e8e5dfb76777281e0513dc36306c4631afe2acf1/functions/popfunc/pop_saveset.m#L19-L22

because there they say that (i) v7.3 is being used by default, and (ii) `.dat` can saved ... but no mention of `.fdt`. --> I wasn't aware of `dat` until now. Is that a recent addition?

EDIT 2: 

Apparently that docstring is wrong, and `.dat` may have been an old extension? ... it's now `.fdt` anyhow. New files are written to v6 or v7.3(default) ... but due to EEGLAB being around for so long, there are obviously all kinds of mat versions for set+fdt around.

https://github.com/sccn/eeglab/issues/230
At least when reading in the original file with `read_mat` it seems to be a `MATLAB 5.0 MAT-file`.
Mhh so it *could* be an issue of `loadmat`, which does not explicitly state that it supports v5 ... (it looks like only v4, v6, v7-v7.2 are supported).

BUT you say that the issue also occurs with `pymatreader`. Did you make code adjustments?

I feel like we should generally use [pymatreader](https://pymatreader.readthedocs.io/en/latest/) rather than scipy loadmat because (i) it's what MNE uses and ships, so we do not add a new dependency to mne-bids, and (ii) because of this statement:

> pymatreader is a small python package that provides a consistent way of loading **all** versions of Matlab .mat files.

(emphasis added by me)

I didn't make any adjustments to pymatreader and loading seems to work properly there. However, after saving with `savemat` and reloading the same way using `pymatreader`, I get the same issue as before.
> I feel like we should generally use [pymatreader](https://pymatreader.readthedocs.io/en/latest/) rather than scipy loadmat 

+1 on this one

@nrosjat If you could share one of the problematic files with me, I can certainly have a look! My email address is richard.hoechenberger@gmail.com

Best wishes!
> However, after saving with savemat and reloading the same way using pymatreader, I get the same issue as before.

I see :thinking: 

Could you try to write the SET files as MAT v7.3, which is HDF5 --> if it can be done with the hdf5 version that MNE ships: https://github.com/mne-tools/mne-python/blob/master/mne/externals/h5io/_h5io.py

(... because we wouldn't want to add a dependency).

Then we can do:

- if everything is in the SET
    - read with pymatreader to check that EEG struct is present - if not: raise error
    - just copy the file without touching it
- if we have SET+FDT
    - read with pymatreader
    - save with the h5io that MNE ships (to MAT format v7.3 ... but with .set extension obviously)


EDIT:  I just realized that MNE does *not* ship HDF5 libraries.

I think the safest way forward then would be to add two optional dependencies to MNE-BIDS that are required when working with EEGLAB files:

- hdf5storage
- h5py
- (pymatreader can be used from mne.externals)

see: https://stackoverflow.com/questions/36362831/creating-a-mat-file-of-v7-3-in-python
> Could you try to write the SET files as MAT v7.3, which is HDF5 --> if it can be done with the hdf5 version that MNE ships: https://github.com/mne-tools/mne-python/blob/master/mne/externals/h5io/_h5io.py

I just tried, but got the following `TypeError`

```
Traceback (most recent call last):

  File "<ipython-input-11-e7c8076546c3>", line 1, in <module>
    write_hdf5('test_mat.set',test_mat,overwrite=True)

  File "C:\Users\Nils Rosjat\Anaconda3\envs\mne-bids\lib\site-packages\mne\externals\h5io\_h5io.py", line 113, in write_hdf5
    _triage_write(title, data, fid, comp_kw, str(type(data)),

  File "C:\Users\Nils Rosjat\Anaconda3\envs\mne-bids\lib\site-packages\mne\externals\h5io\_h5io.py", line 149, in _triage_write
    _triage_write(

  File "C:\Users\Nils Rosjat\Anaconda3\envs\mne-bids\lib\site-packages\mne\externals\h5io\_h5io.py", line 232, in _triage_write
    raise TypeError(err_str)

TypeError: unsupported type <class 'bytes'> (in <class 'dict'>["__header__"])
```
Of course, the quickest route to solving this would be to re-save your files in EEGLAB using the "onefile" approach, so that you only have `.set` and not `.set`+`.fdt`. 

I cannot work on a fix for this issue right now, so either you try to fix it on your own (and let us know :slightly_smiling_face: ) or perhaps @hoechenberger can spend a cycle on this?
For now, we just renamed the files with EEGLAB itself.

I'll try to find a fix for the issue, but any input from @hoechenberger is very appreciated.
It's been too long since I've last used the EEGLAB format, but you already discovered that currently they default to v7.3 files.

I'd also switch to `pymatreader` instead of `scipy.io.loadmat`. However, I don't understand why you need to write to .mat files again?
> I'd also switch to pymatreader instead of scipy.io.loadmat. However, I don't understand why you need to write to .mat files again?

the "copyfile" function is basically to rename existing files to BIDS convention. And for many file formats this turns out to be more complicated than renaming the files, because internal fields may have to be adjusted as well. As in BrainVision, EEGLAB, ...

@nrosjat did you share the problematic file with @hoechenberger it'd be invaluable to fix this issue.
> @nrosjat did you share the problematic file with @hoechenberger it'd be invaluable to fix this issue.

Yes, he should have received an email.
I will have a look later tonight, currently busy with something else. But thanks for sending the file so quickly!
> the "copyfile" function is basically to rename existing files to BIDS convention.

Well… only that nowhere does it enforce BIDS conformity. I'm honestly wondering if these functions currently do more harm than good… (i.e. I don't think they should be part of the public API unless they ensure they're actually producing valid BIDS paths)
Thanks @sappelhoff. I agree with @hoechenberger - EEGLAB is not part of the BIDS standard, so renaming some files and fields will still not produce a valid BIDS dataset. I think writing to .mat will be a challenge (or at least some work) because there is no readily available Python package for writing .mat files.
My point was more that if anything, this copying functionality should be part of `write_raw_bids` only, so we would ensure that the produced file names are BIDS-compatible. I'm okay with copying over EEGLAB data even if it's not (yet?) part of the standard
> EEGLAB is not part of the BIDS standard

>  I'm okay with copying over EEGLAB data even if it's not (yet?) part of the standard

it *is* part of BIDS: https://bids-specification.readthedocs.io/en/latest/04-modality-specific-files/03-electroencephalography.html#eeg-recording-data

however, it is not **recommended** ... it was included to improve adoption of BIDS, because there are many people that use the EEGLAB format.

Still this means, we should support it in mne-bids.

>  this copying functionality should be part of write_raw_bids only, so we would ensure that the produced file names are BIDS-compatible

I think that's a valid point. But I personally like the extra utility that I have to simply rename / move single files without making up a whole BIDS directory. --> or to use that function in my own BIDS conversion routine that I have --- for example because mne-bids does not yet (or never will) support my edge case perfectly. So I'd be +1 to leave the copyfile funcs public

> because there is no readily available Python package for writing .mat files.

yep, ... True. I think it could be done relatively safely by writing to .mat v7.3 exclusively and then use the "hdf5storage" package as detailed in this SO post: https://stackoverflow.com/questions/36362831/creating-a-mat-file-of-v7-3-in-python

(I edited my post up there, you probably didn't see it: https://github.com/mne-tools/mne-bids/issues/652#issuecomment-740557139 )
OK, got it. We don't even need `pymatreader` if we only use it to load files < v7.3 (non-HDF5). `scipy.io.loadmat` now has a `simplify_cells` parameter which produces a nice dict from cell arrays.
@nrosjat Ok starting to look into this now. Will report back once I have results.

On another side note, why are you using MNE-BIDS to process those EEGLAB data, anyway? Wouldn't it be more convenient for EEGLAB users to use something like this: https://github.com/sccn/bids-matlab-tools/wiki ?

Resolving patch:
diff --git a/mne_bids/copyfiles.py b/mne_bids/copyfiles.py
--- a/mne_bids/copyfiles.py
+++ b/mne_bids/copyfiles.py
@@ -17,13 +17,14 @@
 import os
 import os.path as op
 import re
-
 import shutil as sh
 from datetime import datetime
 
+from scipy.io import loadmat, savemat
+
 from mne.io import (read_raw_brainvision, read_raw_edf, read_raw_bdf,
                     anonymize_info)
-from scipy.io import loadmat, savemat
+from mne.utils import check_version
 
 from mne_bids.path import BIDSPath, _parse_ext, _mkdir_p
 from mne_bids.utils import _get_mrk_meas_date, _check_anonymize
@@ -520,41 +521,44 @@ def copyfile_eeglab(src, dest):
     copyfile_kit
 
     """
+    if not check_version('scipy', '1.5.0'):  # pragma: no cover
+        raise ImportError('SciPy >=1.5.0 is required handling EEGLAB data.')
+
     # Get extenstion of the EEGLAB file
-    fname_src, ext_src = _parse_ext(src)
+    _, ext_src = _parse_ext(src)
     fname_dest, ext_dest = _parse_ext(dest)
     if ext_src != ext_dest:
-        raise ValueError('Need to move data with same extension'
+        raise ValueError(f'Need to move data with same extension'
                          f' but got {ext_src}, {ext_dest}')
 
-    # Extract matlab struct "EEG" from EEGLAB file
-    mat = loadmat(src, squeeze_me=False, chars_as_strings=False,
-                  mat_dtype=False, struct_as_record=True)
+    # Load the EEG struct.
+    uint16_codec = None
+    mat = loadmat(file_name=src, simplify_cells=True,
+                  appendmat=False, uint16_codec=uint16_codec)
     if 'EEG' not in mat:
         raise ValueError(f'Could not find "EEG" field in {src}')
     eeg = mat['EEG']
 
-    # If the data field is a string, it points to a .fdt file in src dir
-    data = eeg[0][0]['data']
-    if all([item in data[0, -4:] for item in '.fdt']):
+    if isinstance(eeg['data'], str):
+        # If the data field is a string, it points to a .fdt file in src dir
+        fdt_fname = eeg['data']
+        assert fdt_fname.endswith('.fdt')
         head, tail = op.split(src)
-        fdt_pointer = ''.join(data.tolist()[0])
-        fdt_path = op.join(head, fdt_pointer)
-        fdt_name, fdt_ext = _parse_ext(fdt_path)
-        if fdt_ext != '.fdt':
-            raise IOError('Expected extension .fdt for linked data but found'
-                          f' {fdt_ext}')
-
-        # Copy the fdt file and give it a new name
+        fdt_path = op.join(head, fdt_fname)
+
+        # Copy the .fdt file and give it a new name
         sh.copyfile(fdt_path, fname_dest + '.fdt')
 
-        # Now adjust the pointer in the set file
+        # Now adjust the pointer in the .set file
         head, tail = op.split(fname_dest + '.fdt')
-        mat['EEG'][0][0]['data'] = tail
-        savemat(dest, mat, appendmat=False)
+        eeg['data'] = tail
 
-    # If no .fdt file, simply copy the .set file, no modifications necessary
+        # Save the EEG dictionary as a Matlab struct again
+        mdict = dict(EEG=eeg)
+        savemat(file_name=dest, mdict=mdict, appendmat=False)
     else:
+        # If no .fdt file, simply copy the .set file, no modifications
+        # necessary
         sh.copyfile(src, dest)
 
 
diff --git a/mne_bids/write.py b/mne_bids/write.py
--- a/mne_bids/write.py
+++ b/mne_bids/write.py
@@ -690,7 +690,7 @@ def _write_raw_brainvision(raw, bids_fname, events):
     """
     if not check_version('pybv', '0.4'):  # pragma: no cover
         raise ImportError('pybv >=0.4 is required for converting '
-                          'file to Brainvision format')
+                          'file to BrainVision format')
     from pybv import write_brainvision
     # Subtract raw.first_samp because brainvision marks events starting from
     # the first available data point and ignores the raw.first_samp


Test patch:
diff --git a/mne_bids/tests/test_copyfiles.py b/mne_bids/tests/test_copyfiles.py
--- a/mne_bids/tests/test_copyfiles.py
+++ b/mne_bids/tests/test_copyfiles.py
@@ -6,8 +6,9 @@
 # License: BSD (3-clause)
 import os.path as op
 import datetime
-import pytest
+from distutils.version import LooseVersion
 
+import pytest
 from scipy.io import savemat
 
 # This is here to handle mne-python <0.20
@@ -181,11 +182,17 @@ def _edf_get_real_date(fpath):
     assert rec_info == rec
 
 
-def test_copyfile_eeglab():
+@pytest.mark.parametrize('fname',
+                         ('test_raw.set', 'test_raw_chanloc.set'))
+def test_copyfile_eeglab(fname):
     """Test the copying of EEGlab set and fdt files."""
+    if (fname == 'test_raw_chanloc.set' and
+            LooseVersion(testing.get_version()) < LooseVersion('0.112')):
+        return
+
     bids_root = _TempDir()
     data_path = op.join(testing.data_path(), 'EEGLAB')
-    raw_fname = op.join(data_path, 'test_raw.set')
+    raw_fname = op.join(data_path, fname)
     new_name = op.join(bids_root, 'tested_conversion.set')
 
     # IO error testing
@@ -201,7 +208,14 @@ def test_copyfile_eeglab():
 
     # Test copying and reading a combined set+fdt
     copyfile_eeglab(raw_fname, new_name)
-    raw = mne.io.read_raw_eeglab(new_name)
+    if fname == 'test_raw_chanloc.set':
+        with pytest.warns(RuntimeWarning,
+                          match="The data contains 'boundary' events"):
+            raw = mne.io.read_raw_eeglab(new_name)
+            assert 'Fp1' in raw.ch_names
+    else:
+        raw = mne.io.read_raw_eeglab(new_name)
+        assert 'EEG 001' in raw.ch_names
     assert isinstance(raw, mne.io.BaseRaw)
 
 
diff --git a/test_requirements.txt b/test_requirements.txt
--- a/test_requirements.txt
+++ b/test_requirements.txt
@@ -1,5 +1,5 @@
 numpy>=1.14
-scipy>=0.18.1
+scipy>=1.5.0
 mne>=0.21
 nibabel>=2.2
 pybv>=0.4