Repository: mne-tools/mne-python

Issue:
Error using mne.read_evoked_fieldtrip
I have a .mat file with averages for different conditions that were computed in Fieldtrip with FT_TIMELOCKANALYSIS. I am trying to use mne.read_evoked_fieldtrip, but when I run this code I get an error: `KeyError: 'data'`

```
#Sampling frequency
sfreq = 1000 
#Definition of channel types and names.
ch_types = ['eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eog','eog']
ch_names = ['FP1','FP2','F3', 'F4', 'C3', 'C4', 'P3', 'P4', 'O1', 'O2', 'F7', 'F8', 'T7', 'T8', 'P7', 'P8', 'FZ', 'CZ', 'PZ', 'OZ', 'FC1', 'FC2', 'CP1', 'CP2', 'FC5', 'FC6', 'CP5', 'CP6', 'FT9', 'FT10', 'TP9', 'TP10', 'F1', 'F2', 'C1', 'C2', 'P1', 'P2', 'AF3', 'AF4', 'FC3', 'FC4', 'CP3', 'CP4', 'PO3', 'PO4', 'F5', 'F6', 'C5', 'C6', 'P5', 'P6', 'AF7', 'AF8', 'FT7', 'FT8', 'TP7', 'TP8', 'PO7', 'PO8', 'FPZ', 'FCZ', 'CPZ', 'POZ', 'VEOG', 'HEOG']
#Creation of info object
info = mne.create_info(ch_names=ch_names, sfreq=sfreq, ch_types=ch_types)

file_path = r"D:/EEGdata/toFIELDTRIP";
filename = file_path + '/averages_for_ft.mat'

myepochs = mne.read_evoked_fieldtrip(filename, info)

```

I have also tried to specify the 'data_name', selecting only one of the conditions included in the 'averages_for_ft.mat', but again, I get an error: `TypeError: list indices must be integers or slices, not str`

`myepochs = mne.read_evoked_fieldtrip(filename, info, data_name='pretargetID_ft')`

I am using mne 0.20.dev0, and I am happy to send the 'averages_for_ft.mat' privately, if needed. Thanks.


Discussion:
maybe @thht or @DiGyt can have a look? can you share a file to replicate the pb?
Sure, but I would prefer to share it privately. Where can I send it to? It's about 465000KB.
Hi there, where could I send you the file? Thanks.
hi,
i am going to take a look.

@dede3de send me an email to thomas.hartmann@th-ht.de and i will provide you with a link to upload the file.
so, there are two/three problems here:

1. your file contains a cell array for each of the conditions. i assume each entry refers to the data of one participant. cell array are not supported for reader function at the moment. the solution for your would be to build one file for each participant. i will work on a better error message.
2. the channels in the data are lowercase while the channels in your info structure are uppercase. this needs to match.
3. there is one extra bug when dealing with these missing channels and evoked data.

i think, 1 and 2 have you covered to read your data. i will open a PR for the bugs you found.
Thanks very much for looking at the file.
Re (1), Each cell array is a condition, and within each condition I have a structure for each of the participants. In each structure for each participant, there are:
- label of channels (cell type)
- frequency sample
- electrode locations, with coordinates and labels (structure, with double and cell type)
- average for each channel (rows) and each time point (column) (single type) E.g., 66 channels x 900 timepoints
- variance for each channel (rows) and each time point (column) (single type)
- time points (double type)
- cfg
- dimord

Your suggestion is to create one file for each participant. Which information would I need to include? Only the average for each channel and time point (i.e., a file with 66x900 for each participant)?

Re (2), I am using this code, which works fine:
```
#Sampling frequency
sfreq = 1000 
#Definition of channel types and names.
ch_types = ['eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eeg','eog','eog']
ch_names = ['FP1','FP2','F3', 'F4', 'C3', 'C4', 'P3', 'P4', 'O1', 'O2', 'F7', 'F8', 'T7', 'T8', 'P7', 'P8', 'FZ', 'CZ', 'PZ', 'OZ', 'FC1', 'FC2', 'CP1', 'CP2', 'FC5', 'FC6', 'CP5', 'CP6', 'FT9', 'FT10', 'TP9', 'TP10', 'F1', 'F2', 'C1', 'C2', 'P1', 'P2', 'AF3', 'AF4', 'FC3', 'FC4', 'CP3', 'CP4', 'PO3', 'PO4', 'F5', 'F6', 'C5', 'C6', 'P5', 'P6', 'AF7', 'AF8', 'FT7', 'FT8', 'TP7', 'TP8', 'PO7', 'PO8', 'FPZ', 'FCZ', 'CPZ', 'POZ', 'VEOG', 'HEOG']
#Creation of info object
info = mne.create_info(ch_names=ch_names, sfreq=sfreq, ch_types=ch_types)
epochs.info = info
montage =  mne.channels.make_standard_montage('standard_1020')
epochs.set_montage(montage, match_case=False)#https://github.com/mne-tools/mne-python/issues/7169
```
Re (3) Not sure which ones are the missing channels and evoked data.

Thank you.
so, in your original file you have the following structure:

- condition_1
- condition_2

etc... (i use generic names here for your conditions). each of these fields is a cell-array containing one fieldtrip-structure per participant.

what you need to do is create a file for each participant. the general structure would be the same. i.e. there would be one variable per condition. but instead of containing a cell array with the data of all participants, the variable should only contain the fieldtrip-structure of one participant.

here is a small snippet i used to extract the data of the first participant:

```matlab
%% init
restoredefaultpath
clear all global

%% set variables
f_name = '/home/th/owncloud/sbg/upload/mne_issue_7293/averages_for_ft.mat';

%% load data
data = load(f_name);

%% convert fields
all_fields = fieldnames(data);

for idx_field = 1:length(all_fields)
  cur_field = all_fields{idx_field};
  data.(cur_field) = data.(cur_field){1};
end %for

%% save new data
save('new_data.mat', '-struct', 'data');
```

the only thing you would need to do is to do this for all participants, i.e. add another loop around it.

concerning the channel names: they are in lower case in your mat files. they are in uppercase in the MNE info structure. they need to match (case sensitive) in order for the import to work.

Resolving patch:
diff --git a/mne/datasets/utils.py b/mne/datasets/utils.py
--- a/mne/datasets/utils.py
+++ b/mne/datasets/utils.py
@@ -239,7 +239,7 @@ def _data_path(path=None, force_update=False, update_path=True, download=True,
     path = _get_path(path, key, name)
     # To update the testing or misc dataset, push commits, then make a new
     # release on GitHub. Then update the "releases" variable:
-    releases = dict(testing='0.79', misc='0.5')
+    releases = dict(testing='0.80', misc='0.5')
     # And also update the "md5_hashes['testing']" variable below.
 
     # To update any other dataset, update the data archive itself (upload
@@ -323,7 +323,7 @@ def _data_path(path=None, force_update=False, update_path=True, download=True,
         sample='12b75d1cb7df9dfb4ad73ed82f61094f',
         somato='f08f17924e23c57a751b3bed4a05fe02',
         spm='9f43f67150e3b694b523a21eb929ea75',
-        testing='7a8f1804a38e72394cd64aca1dad7cd3',
+        testing='de51a47739798064fd20bd3345d5eba0',
         multimodal='26ec847ae9ab80f58f204d09e2c08367',
         fnirs_motor='c4935d19ddab35422a69f3326a01fef8',
         opm='370ad1dcfd5c47e029e692c85358a374',
diff --git a/mne/io/fieldtrip/fieldtrip.py b/mne/io/fieldtrip/fieldtrip.py
--- a/mne/io/fieldtrip/fieldtrip.py
+++ b/mne/io/fieldtrip/fieldtrip.py
@@ -7,7 +7,7 @@
 import numpy as np
 
 from .utils import _create_info, _set_tmin, _create_events, \
-    _create_event_metadata
+    _create_event_metadata, _validate_ft_struct
 from .. import RawArray
 from ...epochs import EpochsArray
 from ...evoked import EvokedArray
@@ -52,6 +52,8 @@ def read_raw_fieldtrip(fname, info, data_name='data'):
     # load data and set ft_struct to the heading dictionary
     ft_struct = ft_struct[data_name]
 
+    _validate_ft_struct(ft_struct)
+
     info = _create_info(ft_struct, info)  # create info structure
     data = np.array(ft_struct['trial'])  # create the main data array
 
@@ -113,6 +115,8 @@ def read_epochs_fieldtrip(fname, info, data_name='data',
     # load data and set ft_struct to the heading dictionary
     ft_struct = ft_struct[data_name]
 
+    _validate_ft_struct(ft_struct)
+
     info = _create_info(ft_struct, info)  # create info structure
     data = np.array(ft_struct['trial'])  # create the epochs data array
     events = _create_events(ft_struct, trialinfo_column)
@@ -163,6 +167,8 @@ def read_evoked_fieldtrip(fname, info, comment=None,
                          variable_names=[data_name])
     ft_struct = ft_struct[data_name]
 
+    _validate_ft_struct(ft_struct)
+
     info = _create_info(ft_struct, info)  # create info structure
     data_evoked = ft_struct['avg']  # create evoked data
 
diff --git a/mne/io/fieldtrip/utils.py b/mne/io/fieldtrip/utils.py
--- a/mne/io/fieldtrip/utils.py
+++ b/mne/io/fieldtrip/utils.py
@@ -30,6 +30,12 @@
                  'source analysis, channel interpolation etc.'
 
 
+def _validate_ft_struct(ft_struct):
+    """Run validation checks on the ft_structure."""
+    if isinstance(ft_struct, list):
+        raise RuntimeError('Loading of data in cell arrays is not supported')
+
+
 def _create_info(ft_struct, raw_info):
     """Create MNE info structure from a FieldTrip structure."""
     if raw_info is None:
@@ -50,10 +56,18 @@ def _create_info(ft_struct, raw_info):
             new_chs = [ch for ch in ch_names if ch not in missing_channels]
             ch_names = new_chs
             ft_struct['label'] = ch_names
-            if ft_struct['trial'].ndim == 2:
-                ft_struct['trial'] = np.delete(ft_struct['trial'],
-                                               missing_chan_idx,
-                                               axis=0)
+
+            if 'trial' in ft_struct:
+                if ft_struct['trial'].ndim == 2:
+                    ft_struct['trial'] = np.delete(ft_struct['trial'],
+                                                   missing_chan_idx,
+                                                   axis=0)
+
+            if 'avg' in ft_struct:
+                if ft_struct['avg'].ndim == 2:
+                    ft_struct['avg'] = np.delete(ft_struct['avg'],
+                                                 missing_chan_idx,
+                                                 axis=0)
 
         info['sfreq'] = sfreq
         ch_idx = [info['ch_names'].index(ch) for ch in ch_names]


Test patch:
diff --git a/mne/io/fieldtrip/tests/test_fieldtrip.py b/mne/io/fieldtrip/tests/test_fieldtrip.py
--- a/mne/io/fieldtrip/tests/test_fieldtrip.py
+++ b/mne/io/fieldtrip/tests/test_fieldtrip.py
@@ -41,7 +41,8 @@
 @testing.requires_testing_data
 # Reading the sample CNT data results in a RuntimeWarning because it cannot
 # parse the measurement date. We need to ignore that warning.
-@pytest.mark.filterwarnings('ignore::RuntimeWarning')
+@pytest.mark.filterwarnings('ignore:.*parse meas date.*:RuntimeWarning')
+@pytest.mark.filterwarnings('ignore:.*number of bytes.*:RuntimeWarning')
 @pytest.mark.parametrize('cur_system, version, use_info',
                          all_test_params_epochs)
 def test_read_evoked(cur_system, version, use_info):
@@ -75,7 +76,8 @@ def test_read_evoked(cur_system, version, use_info):
 @testing.requires_testing_data
 # Reading the sample CNT data results in a RuntimeWarning because it cannot
 # parse the measurement date. We need to ignore that warning.
-@pytest.mark.filterwarnings('ignore::RuntimeWarning')
+@pytest.mark.filterwarnings('ignore:.*parse meas date.*:RuntimeWarning')
+@pytest.mark.filterwarnings('ignore:.*number of bytes.*:RuntimeWarning')
 @pytest.mark.parametrize('cur_system, version, use_info',
                          all_test_params_epochs)
 # Strange, non-deterministic Pandas errors:
@@ -128,7 +130,8 @@ def test_read_epochs(cur_system, version, use_info):
 @testing.requires_testing_data
 # Reading the sample CNT data results in a RuntimeWarning because it cannot
 # parse the measurement date. We need to ignore that warning.
-@pytest.mark.filterwarnings('ignore::RuntimeWarning')
+@pytest.mark.filterwarnings('ignore:.*parse meas date.*:RuntimeWarning')
+@pytest.mark.filterwarnings('ignore:.*number of bytes.*:RuntimeWarning')
 @pytest.mark.parametrize('cur_system, version, use_info', all_test_params_raw)
 def test_raw(cur_system, version, use_info):
     """Test comparing reading a raw fiff file and the FieldTrip version."""
@@ -228,3 +231,42 @@ def test_one_channel_elec_bug(version):
 
     with pytest.warns(**no_info_warning):
         mne.io.read_raw_fieldtrip(fname, info=None)
+
+
+@testing.requires_testing_data
+# Reading the sample CNT data results in a RuntimeWarning because it cannot
+# parse the measurement date. We need to ignore that warning.
+@pytest.mark.filterwarnings('ignore:.*parse meas date.*:RuntimeWarning')
+@pytest.mark.filterwarnings('ignore:.*number of bytes.*:RuntimeWarning')
+@pytest.mark.parametrize('version', all_versions)
+@pytest.mark.parametrize('type', ['averaged', 'epoched', 'raw'])
+@requires_h5py
+def test_throw_exception_on_cellarray(version, type):
+    """Test for a meaningful exception when the data is a cell array."""
+    fname = os.path.join(get_data_paths('cellarray'),
+                         '%s_%s.mat' % (type, version))
+
+    info = get_raw_info('CNT')
+
+    with pytest.raises(RuntimeError, match='Loading of data in cell arrays '
+                                           'is not supported'):
+        if type == 'averaged':
+            mne.read_evoked_fieldtrip(fname, info)
+        elif type == 'epoched':
+            mne.read_epochs_fieldtrip(fname, info)
+        elif type == 'raw':
+            mne.io.read_raw_fieldtrip(fname, info)
+
+
+@testing.requires_testing_data
+def test_evoked_with_missing_channels():
+    """Test _create_info on evoked data when channels are missing from info."""
+    cur_system = 'neuromag306'
+    test_data_folder_ft = get_data_paths(cur_system)
+    info = get_raw_info(cur_system)
+    del info['chs'][1:20]
+    info._update_redundant()
+
+    with pytest.warns(RuntimeWarning):
+        mne.read_evoked_fieldtrip(
+            os.path.join(test_data_folder_ft, 'averaged_v7.mat'), info)