Repository: mne-tools/mne-python

Issue:
mne.io.read_epochs_eeglab inconsistent behaviour 
Dear MNE expert, 

I have run mne.io.read_epochs_eeglab and everything went smoothly with the majority of my data sample. However, for some of them I got the following error: 

[Errno 2] No such file or directory: path/to/file.fdt

Apparently for some reason it doesn't see the file that it is of course in the same path. 

Do you have any tips?

Thanks 

Dave 


Discussion:
If you've copied the error exactly, then it means that Python is looking for a file called `file.fdt` in a location `./path/to/` (relative to the current working directory where Python was started).  That folder location is almost certainly not correct, and probably reflects some example code that you copy-pasted and forgot to comment out or remove from your script.

If that error message was just an example: well, Python is usually not wrong about whether or not a file exists.  Try commands like `os.getcwd()` and `os.listdir()` to make sure the Python interpreter is in fact working in the directory that you think it's in, and whether the file is in fact where Python is looking.


Dear Daniel, 

thanks for your reply. 

I double checked thousands times and the file is in the corresponding folder. All other files are in the same folder and the problem as I said is for only few subjects. 

Anyway I checked it out another time...I have run the `os.listdir()` and the file is in there. 

Any other suggestions? Shall I share both files? 

Thanks 




 
if the error is "no such file or directory" then sharing the file itself won't help.  Can you edit lines 3 and 4 below, then run this and tell us the output?

```python
import os
import mne
folder_where_files_are = os.path.join('path', 'to', 'files')  # <- edit this
full_file_path = os.path.join(folder_where_files_are, 'file.fdt')  # <- edit this
print(os.getcwd())
print(os.listdir(folder_where_files_are))
mne.io.read_epochs_eeglab(full_file_path)
```

I've run the command. 
This is the error I got

TypeError: Expecting miMATRIX type here, got 4002551232

I haven't shared the ouput of `print(os.listdir(folder_where_files_are))` because I have a lot of files in that folder and It would be unreadable. However I can assure you that the file is in there. 

Thanks
 


that is a different error from "no such file or directory". Anyway if you don't paste the full output (traceback, etc) there's little I can do to help.
```python
import os
import mne
folder_where_files_are = os.path.join('/Volumes/WD_Elements/BROAD/EEG/M1_IPL/Recep/M1_IPL/Visit_1/M1')
full_file_path = os.path.join(folder_where_files_are, '80LP_V1_SP_M1_S10_Final.fdt')
print(os.getcwd())
```                                                                                                                                                                                                                                    
/Volumes/WD_Elements/BROAD/EEG/M1_IPL/Recep/M1_IPL/Visit_1/M1

```python
print(os.listdir(folder_where_files_are))
```

['78TW_V1_150SP_M1_S10_Final.fdt', '._78TW_V1_150SP_M1_S10_Final.fdt', '78TW_V1_150SP_M1_S10_Final.set', '._78TW_V1_150SP_M1_S10_Final.set', '80LP_V1_SP_M1_S10_Final.fdt', '._80LP_V1_SP_M1_S10_Final.fdt', '80LP_V1_SP_M1_S10_Final.set', '._80LP_V1_SP_M1_S10_Final.set', '81SSF_V1_SP_M1_S10_Final.fdt', '._81SSF_V1_SP_M1_S10_Final.fdt', '81SSF_V1_SP_M1_S10_Final.set', '._81SSF_V1_SP_M1_S10_Final.set', '82YZ_V5_SP_M1_S10_Final.fdt', '._82YZ_V5_SP_M1_S10_Final.fdt', '82YZ_V5_SP_M1_S10_Final.set', '._82YZ_V5_SP_M1_S10_Final.set', '83KW_V1_SP_M1_S10_Final.fdt', '._83KW_V1_SP_M1_S10_Final.fdt', '83KW_V1_SP_M1_S10_Final.set', '._83KW_V1_SP_M1_S10_Final.set', '84LAL_V2_150sp_M1_S10_Final.fdt', '._84LAL_V2_150sp_M1_S10_Final.fdt', '84LAL_V2_150sp_M1_S10_Final.set', '._84LAL_V2_150sp_M1_S10_Final.set', '52SC_V1_120SP_M1_Merged_S11_GMFA.fdt', '._52SC_V1_120SP_M1_Merged_S11_GMFA.fdt', '52SC_V1_120SP_M1_Merged_S11_GMFA.set', '._52SC_V1_120SP_M1_Merged_S11_GMFA.set', '53SH_V3_120SP_M1_Merged_S10_Final.fdt', '._53SH_V3_120SP_M1_Merged_S10_Final.fdt', '53SH_V3_120SP_M1_Merged_S10_Final.set', '._53SH_V3_120SP_M1_Merged_S10_Final.set', '56YM_V2_120SP_M1_Merged_S11_GMFA.fdt', '._56YM_V2_120SP_M1_Merged_S11_GMFA.fdt', '56YM_V2_120SP_M1_Merged_S11_GMFA.set', '._56YM_V2_120SP_M1_Merged_S11_GMFA.set', '57AH_V2_120SP_M1_Merged_S10_Final.fdt', ]     

The error is different because in your  script you put the .fdt file instead of .set file into the mne.io.read_epochs_eeglab function. Otherwise the error is the same. 

Thanks                                                                                  



OK, thanks.  Sorry, but we had to rule out user error first.  But it seems this might be a bug.  I don't normally work with EEGLAB files so I'm pinging the other devs on [our gitter channel](https://gitter.im/mne-tools/mne-python) to see if one of them has seen this before.
please a short script and the files that are problematic so we can
investigate

>

Dear Alex, 

please find below the link to download both the .fdt and .set file. 

https://www.dropbox.com/s/0n8zs7t5gobldwu/80LP_V1_SP_M1_S10_Final.fdt?dl=0

https://www.dropbox.com/s/j5kbbyiecrju8rj/80LP_V1_SP_M1_S10_Final.set?dl=0

The files correspond to one among the problematic subjects and it is the same of the previous messages. 

Thanks 
@Davi1990 I'm looking into the files now, I will let you know what I find.
@Davi1990 @agramfort @drammock 
Ok, the error is related to the `EEG.data` value, which is different to the file name. Notice that the error says:
```python-traceback
Traceback (most recent call last):

  File "<ipython-input-24-da692532ba90>", line 1, in <module>
    epochs = mne.io.read_epochs_eeglab(op.join(data_dir, '80LP_V1_SP_M1_S10_Final.set'))

  File "c:\src\mne-python\mne\io\eeglab\eeglab.py", line 243, in read_epochs_eeglab
    uint16_codec=uint16_codec)

  File "<c:\src\mne-python\mne\externals\decorator.py:decorator-gen-195>", line 2, in __init__

  File "c:\src\mne-python\mne\utils\_logging.py", line 90, in wrapper
    return function(*args, **kwargs)

  File "c:\src\mne-python\mne\io\eeglab\eeglab.py", line 549, in __init__
    with open(data_fname, 'rb') as data_fid:

FileNotFoundError: [Errno 2] No such file or directory: 'C:\\tmp\\test_mne_7192\\80_LP_V1_SP_M1_S10_Final.fdt'
```
while the file name in the directory is `80LP_V1_SP_M1_S10_Final.fdt` (so there is `80LP`, not `80_LP` at the beginning).
There are two quick solutions on your end:
* you can change the file name to agree with the `EEG.data` value
* or you can edit the `EEG.data` to have the value that is the same to the actual file name.

On the mne side, because it is not the first time this error appears, I think we might change how fdt files are read: when the `EEG.data` filename is absent and it is different than `fdt_fname = set_fname.replace('.set', '.fdt')` then we could use `fdt_fname`.
This problem should be common when users rename the `.set` and `.fdt` files by hand but not through eeglab (which also changes the `EEG.data` value when saving under a different name). @Davi1990 - are you able to read the file in EEGLAB? I think it does the same that I am proposing here.
@mmagnuski thanks for your update. I've changed the name in MATLAB so that now everything match. Of course, now I was able to run ` mne.io.read_epochs_eeglab`with no issue. 

I am analyzing the same dataset in MATLAB for some convolutional stuffs but I've never faced this problem before. 

Thanks again for your help...really appreciate that! 
I  confirm that EEGLAB can read these files but raises a warning:

Warning: the name of the dataset has changed on disk, updating .dat & .fdt
data file to the new name
Reading float file '/Users/alex/Downloads/80LP_V1_SP_M1_S10_Final.fdt'...

+1 for this the behavior of MNE to match eeglab

>

Ok, I started doing that, I'll submit a PR later.

Resolving patch:
diff --git a/mne/io/eeglab/eeglab.py b/mne/io/eeglab/eeglab.py
--- a/mne/io/eeglab/eeglab.py
+++ b/mne/io/eeglab/eeglab.py
@@ -23,9 +23,14 @@
 CAL = 1e-6
 
 
-def _check_fname(fname):
-    """Check if the file extension is valid."""
-    fmt = str(op.splitext(fname)[-1])
+def _check_fname(fname, dataname):
+    """Check whether the filename is valid.
+
+    Check if the file extension is ``.fdt`` (older ``.dat`` being invalid) or
+    whether the ``EEG.data`` filename exists. If ``EEG.data`` file is absent
+    the set file name with .set changed to .fdt is checked.
+    """
+    fmt = str(op.splitext(dataname)[-1])
     if fmt == '.dat':
         raise NotImplementedError(
             'Old data format .dat detected. Please update your EEGLAB '
@@ -33,6 +38,21 @@ def _check_fname(fname):
     elif fmt != '.fdt':
         raise IOError('Expected .fdt file format. Found %s format' % fmt)
 
+    basedir = op.dirname(fname)
+    data_fname = op.join(basedir, dataname)
+    if not op.exists(data_fname):
+        fdt_from_set_fname = op.splitext(fname)[0] + '.fdt'
+        if op.exists(fdt_from_set_fname):
+            data_fname = fdt_from_set_fname
+            msg = ('Data file name in EEG.data ({}) is incorrect, the file '
+                   'name must have changed on disk, using the correct file '
+                   'name ({}).')
+            warn(msg.format(dataname, op.basename(fdt_from_set_fname)))
+        elif not data_fname == fdt_from_set_fname:
+            msg = 'Could not find the .fdt data file, tried {} and {}.'
+            raise FileNotFoundError(msg.format(data_fname, fdt_from_set_fname))
+    return data_fname
+
 
 def _check_load_mat(fname, uint16_codec):
     """Check if the mat struct contains 'EEG'."""
@@ -279,7 +299,6 @@ class RawEEGLAB(BaseRaw):
     @verbose
     def __init__(self, input_fname, eog=(),
                  preload=False, uint16_codec=None, verbose=None):  # noqa: D102
-        basedir = op.dirname(input_fname)
         eeg = _check_load_mat(input_fname, uint16_codec)
         if eeg.trials != 1:
             raise TypeError('The number of trials is %d. It must be 1 for raw'
@@ -291,8 +310,7 @@ def __init__(self, input_fname, eog=(),
 
         # read the data
         if isinstance(eeg.data, str):
-            data_fname = op.join(basedir, eeg.data)
-            _check_fname(data_fname)
+            data_fname = _check_fname(input_fname, eeg.data)
             logger.info('Reading %s' % data_fname)
 
             super(RawEEGLAB, self).__init__(
@@ -542,9 +560,7 @@ def __init__(self, input_fname, events=None, event_id=None, tmin=0,
                                  '(event id %i)' % (key, val))
 
         if isinstance(eeg.data, str):
-            basedir = op.dirname(input_fname)
-            data_fname = op.join(basedir, eeg.data)
-            _check_fname(data_fname)
+            data_fname = _check_fname(input_fname, eeg.data)
             with open(data_fname, 'rb') as data_fid:
                 data = np.fromfile(data_fid, dtype=np.float32)
                 data = data.reshape((eeg.nbchan, eeg.pnts, eeg.trials),


Test patch:
diff --git a/mne/io/eeglab/tests/test_eeglab.py b/mne/io/eeglab/tests/test_eeglab.py
--- a/mne/io/eeglab/tests/test_eeglab.py
+++ b/mne/io/eeglab/tests/test_eeglab.py
@@ -83,11 +83,10 @@ def test_io_set_raw(fname):
 def test_io_set_raw_more(tmpdir):
     """Test importing EEGLAB .set files."""
     tmpdir = str(tmpdir)
-    # test reading file with one event (read old version)
     eeg = io.loadmat(raw_fname_mat, struct_as_record=False,
                      squeeze_me=True)['EEG']
 
-    # test negative event latencies
+    # test reading file with one event (read old version)
     negative_latency_fname = op.join(tmpdir, 'test_negative_latency.set')
     evnts = deepcopy(eeg.event[0])
     evnts.latency = 0
@@ -103,6 +102,7 @@ def test_io_set_raw_more(tmpdir):
     with pytest.warns(RuntimeWarning, match="has a sample index of -1."):
         read_raw_eeglab(input_fname=negative_latency_fname, preload=True)
 
+    # test negative event latencies
     evnts.latency = -1
     io.savemat(negative_latency_fname,
                {'EEG': {'trials': eeg.trials, 'srate': eeg.srate,
@@ -126,6 +126,30 @@ def test_io_set_raw_more(tmpdir):
                appendmat=False, oned_as='row')
     shutil.copyfile(op.join(base_dir, 'test_raw.fdt'),
                     overlap_fname.replace('.set', '.fdt'))
+    read_raw_eeglab(input_fname=overlap_fname, preload=True)
+
+    # test reading file when the EEG.data name is wrong
+    io.savemat(overlap_fname,
+               {'EEG': {'trials': eeg.trials, 'srate': eeg.srate,
+                        'nbchan': eeg.nbchan, 'data': 'test_overla_event.fdt',
+                        'epoch': eeg.epoch,
+                        'event': [eeg.event[0], eeg.event[0]],
+                        'chanlocs': eeg.chanlocs, 'pnts': eeg.pnts}},
+               appendmat=False, oned_as='row')
+    with pytest.warns(RuntimeWarning, match="must have changed on disk"):
+        read_raw_eeglab(input_fname=overlap_fname, preload=True)
+
+    # raise error when both EEG.data and fdt name from set are wrong
+    overlap_fname = op.join(tmpdir, 'test_ovrlap_event.set')
+    io.savemat(overlap_fname,
+               {'EEG': {'trials': eeg.trials, 'srate': eeg.srate,
+                        'nbchan': eeg.nbchan, 'data': 'test_overla_event.fdt',
+                        'epoch': eeg.epoch,
+                        'event': [eeg.event[0], eeg.event[0]],
+                        'chanlocs': eeg.chanlocs, 'pnts': eeg.pnts}},
+               appendmat=False, oned_as='row')
+    with pytest.raises(FileNotFoundError, match="not find the .fdt data file"):
+        read_raw_eeglab(input_fname=overlap_fname, preload=True)
 
     # test reading file with one channel
     one_chan_fname = op.join(tmpdir, 'test_one_channel.set')