Repository: mne-tools/mne-python

Issue:
Exclude multiple identically named channels when reading EDF
I am using mne 0.15.2 to load the EDF files in the CHB-MIT EEG database https://www.physionet.org/pn6/chbmit/.

[Note: I know there are other issues related to this dataset, in particular about the duplicate "T8-P8" channel. But I think this is a separate issue. It is rather small and there might be better solutions to it also.]

`read_raw_edf` happily reads the EDF files for about half of the subjects.
```
raw = mne.io.read_raw_edf('chb01/chb01_01.edf', stim_channel=None)

Extracting edf Parameters from chb01/chb01_01.edf...
EDF file detected
Setting channel info structure...
Created Raw.info structure...
Channel names are not unique, found duplicates for: {'T8-P8'}. Applying running numbers for duplicates.
Ready.
```

But it fails on the rest:
```
raw = mne.io.read_raw_edf('chb12/chb12_06.edf', stim_channel=None)

Extracting edf Parameters from chb12/chb12_06.edf...
EDF file detected
Setting channel info structure...
Created Raw.info structure...
Channel names are not unique, found duplicates for: {'T8-P8', '-'}. Applying running numbers for duplicates.

...

ValueError: Bad cals for channels {4: '--0', 9: '--1', 12: '--2', 17: '--3', 22: '--4'}
```

This seems to happen because some EDFs contain multiple "dummy" channels called "-".

Next I tried to exclude the "-" channels explicitly.

```
raw = mne.io.read_raw_edf('chb12/chb12_06.edf', stim_channel=None,
                          exclude=["-"])

Extracting edf Parameters from chb12/chb12_06.edf...
EDF file detected
Setting channel info structure...
Created Raw.info structure...
Channel names are not unique, found duplicates for: {'T8-P8', '-'}. Applying running numbers for duplicates.

....

ValueError: Bad cals for channels {8: '--0', 11: '--1', 16: '--2', 21: '--3'}
```

This excludes the first "-" channel but leaves the others in. So there is still a "bad cals" error.

I was able to solve this by modifying the code in `mne/io/edf/edf.py` to exclude all channels "-". I added this function:

```
def _find_exclude_idx(ch_names, exclude):
    # If there are several channels called "A" and we want to exclude "A",
    # then add (the index of) all "A" channels to the exclusion list
    #
    # Replace
    # return [ch_names.index(idx) for idx in exclude]
    # with
    return [idx for idx, ch in enumerate(ch_names)
            for ex in exclude if ch == ex]
```

And then I replaced all occurrences of

```
exclude = [ch_names.index(idx) for idx in exclude]
```
with 
```
exclude = _find_exclude_idx(ch_names, exclude)
```

Now I can read all EDFs in CHB-MIT as long as specify the list of channels to exclude appropriately:

```
if subj in ["chb12", "chb13", "chb14", "chb15",
            "chb16", "chb17", "chb18", "chb19",
            "chb21"]:
    exclude = ["-"]
else:
    exclude = []
```




Discussion:
the calibrations factors `cals` are 0 for these channels in this file...

```
In [5]: %debug
> /Users/alex/work/src/mne-python/mne/io/base.py(347)__init__()
    345         if len(bad) > 0:
    346             raise ValueError('Bad cals for channels %s'
--> 347                              % dict((ii, self.ch_names[ii]) for ii in bad))
    348         self.verbose = verbose
    349         self._cals = cals

ipdb> l
    342         for k in range(info['nchan']):
    343             cals[k] = info['chs'][k]['range'] * info['chs'][k]['cal']
    344         bad = np.where(cals == 0)[0]
    345         if len(bad) > 0:
    346             raise ValueError('Bad cals for channels %s'
--> 347                              % dict((ii, self.ch_names[ii]) for ii in bad))
    348         self.verbose = verbose
    349         self._cals = cals
    350         self._raw_extras = list(raw_extras)
    351         # deal with compensation (only relevant for CTF data, either CTF
    352         # reader or MNE-C converted CTF->FIF files)

ipdb> cals
array([ 6552000.,  6552000.,  6552000.,  6552000.,        0.,  6552000.,
        6552000.,  6552000.,  6552000.,        0.,  6552000.,  6552000.,
              0.,  6552000.,  6552000.,  6552000.,  6552000.,        0.,
        6552000.,  6552000.,  6552000.,  6552000.,        0.,  6552000.,
        6552000.,  6552000.,  6552000.,  6552000.])
```

the solution is to insert here : https://github.com/mne-tools/mne-python/blob/master/mne/io/edf/edf.py#L160

some code that changes the cals from 0 to 1 (or anything != 0). I wonder is we should do this by default with just a warning?

thoughts anyone?
It would be good to understand ideally is why these would be set to 0. in the file. But using `1.` with a warning seems okay to me.
According to the official CHB-MIT EEG Database website, 

> In some cases, up to 5 “dummy” signals (named "-") were interspersed among the EEG signals to obtain an easy-to-read display format; these dummy signals can be ignored.

And here is the list of channels from one EDF, the dummy ones included, to illustrate what they mean by easy-to-read display:

```
FP1-F7 F7-T7 T7-P7 P7-O1 - FP1-F3 F3-C3 C3-P3 P3-O1 - FZ-CZ CZ-PZ - FP2-F4 F4-C4 C4-P4 P4-O2 - FP2-F8 F8-T8 T8-P8 P8-O2 - P7-T7 T7-FT9 FT9-FT10 FT10-T8 T8-P8
```

So the dummy channels split the channels into "connected" groups. The choice of 0 as the dummy variable doesn't matter if this was done for display purposes only.

But then they didn't use these scheme for all their EDF files.

@dipetkov your fix looks reasonable to me in this case. Can you open a PR to fix it? One tweak, your check:
```
    return [idx for idx, ch in enumerate(ch_names)
            for ex in exclude if ch == ex]
```
Can be simplified to:
```
    return [idx for idx, ch in enumerate(ch_names) if ch not in exclude]
```
I would make exclude accept a regexp wdyt?
Why not but probably as a separate PR that does it for EDF and EGI (the two readers that use `exclude` kwargs).
ok fair enough


Resolving patch:
diff --git a/mne/io/edf/edf.py b/mne/io/edf/edf.py
--- a/mne/io/edf/edf.py
+++ b/mne/io/edf/edf.py
@@ -618,7 +618,7 @@ def _read_edf_header(fname, annot, annotmap, exclude):
         nchan = int(fid.read(4).decode())
         channels = list(range(nchan))
         ch_names = [fid.read(16).strip().decode() for ch in channels]
-        exclude = [ch_names.index(idx) for idx in exclude]
+        exclude = _find_exclude_idx(ch_names, exclude)
         for ch in channels:
             fid.read(80)  # transducer
         units = [fid.read(8).strip().decode() for ch in channels]
@@ -758,7 +758,7 @@ def _read_gdf_header(fname, stim_channel, exclude):
             units = [fid.read(8).decode('latin-1').strip(' \x00')
                      for ch in channels]
 
-            exclude = [ch_names.index(idx) for idx in exclude]
+            exclude = _find_exclude_idx(ch_names, exclude)
             include = list()
             for i, unit in enumerate(units):
                 if unit[:2] == 'uV':
@@ -938,7 +938,7 @@ def _read_gdf_header(fname, stim_channel, exclude):
             channels = list(range(nchan))
             ch_names = [fid.read(16).decode().strip(' \x00')
                         for ch in channels]
-            exclude = [ch_names.index(idx) for idx in exclude]
+            exclude = _find_exclude_idx(ch_names, exclude)
 
             fid.seek(80 * len(channels), 1)  # reserved space
             fid.seek(6 * len(channels), 1)  # phys_dim, obsolete
@@ -1158,6 +1158,15 @@ def _check_stim_channel(stim_channel, ch_names, include):
     return stim_channel
 
 
+def _find_exclude_idx(ch_names, exclude):
+    """Find the index of all channels to exclude.
+
+    If there are several channels called "A" and we want to exclude "A",
+    then add (the index of) all "A" channels to the exclusion list.
+    """
+    return [idx for idx, ch in enumerate(ch_names) if ch in exclude]
+
+
 def read_raw_edf(input_fname, montage=None, eog=None, misc=None,
                  stim_channel='auto', annot=None, annotmap=None, exclude=(),
                  preload=False, verbose=None):


Test patch: