Repository: mne-tools/mne-python

Issue:
IndexError when importing raw CNT file
#### Describe the bug
- Failure to import a particular Neuroscan .cnt file via `read_raw_cnt()` due to out of bounds indexing in `cnt.py` line 251
- About half of our .cnt files work, half do not due to this error.

#### Steps and/or code to reproduce
```
from mne.io import read_raw_cnt

raw_fname = 'raw_eeg_data.cnt'
montage_fname = 'standard-10-5-cap385.elp'
raw = read_raw_cnt(raw_fname, montage=montage_fname, eog=('HEOG', 'VEOG'), emg=('EMG',), preload=True, verbose=True)
```

#### Expected results
- raw variable successfully populated with a RawCNT object upon reading the .cnt file
- no error message

#### Actual results
```
Traceback (most recent call last):
  File "/Users/corey/Documents/GitLab/01mne_python_automation/bin/source_localization.py", line 210, in <module>
    raw = read_raw_cnt(raw_fname, montage=montage_fname, eog=('HEOG', 'VEOG'), emg=('EMG',), preload=True, verbose=True)
  File "/anaconda3/envs/mne/lib/python3.6/site-packages/mne/io/cnt/cnt.py", line 95, in read_raw_cnt
    preload=preload, verbose=verbose)
  File "/anaconda3/envs/mne/lib/python3.6/site-packages/mne/io/cnt/cnt.py", line 361, in __init__
    data_format, date_format)
  File "/anaconda3/envs/mne/lib/python3.6/site-packages/mne/io/cnt/cnt.py", line 251, in _get_cnt_info
    stim_channel[event_time - 1] = event_id
IndexError: index 295313 is out of bounds for axis 0 with size 294560
```

#### Additional information
`sys_info()` : 
```
Platform:      Darwin-17.7.0-x86_64-i386-64bit
Python:        3.6.6 |Anaconda, Inc.| (default, Jun 28 2018, 11:07:29)  [GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)]
Executable:    /anaconda3/envs/mne/bin/python
CPU:           i386: 8 cores
Memory:        16.0 GB

mne:           0.16.2
numpy:         1.15.0 {blas=mkl_rt, lapack=mkl_rt}
scipy:         1.1.0
matplotlib:    2.2.2 {backend=MacOSX}

sklearn:       0.19.1
nibabel:       2.3.0
mayavi:        4.6.1 {qt_api=pyqt5}
pycuda:        Not found
skcuda:        Not found
pandas:        0.23.3
```


Discussion:
@corscheid can you share one file with us? ideally a super tiny one?
Hi @agramfort, Thanks for the reply. 

My colleague @deepagupta has more authority than I concerning our experiment data; she will be by shortly to respond.
Dear @agramfort, Thank you for your response. 

Unfortunately, we deeply regret that we can't share our CNT data file because we are trying to process it for research and may eventually publish from that. So, sharing won't be ethically right or in agreement with our collaborators. 

However, we want to try our best to provide any other information if we can. For example, my teammate Corey posted the following error message:

```
File "/anaconda3/envs/mne/lib/python3.6/site-packages/mne/io/cnt/cnt.py", line 251, in _get_cnt_info
    stim_channel[event_time - 1] = event_id
IndexError: index 295313 is out of bounds for axis 0 with size 294560
```

Basically, we don't understand why mne python library is generating a stimulating channel and what are the numbers 900 and 75 in the equation below (file cnt.py at line#251):

```
           event_time = offset - 900 - 75 * n_channels
            event_time //= n_channels * n_bytes
            stim_channel[event_time - 1] = event_id
```

We believe this must be the source of the issue. However, we can't be entirely sure.
@deepagupta we cannot debug without access to a test file. You can share one file privately with one of us. @larsoner @massich or myself.

if you find the fix it's even better. Then please send us a pull request
If you can came up with a simulated MWE that has `np.empty(mydata.shape)` as data inside, that would be even better!

I also have an IndexError when importing raw CNT file. I can share the code and data, though not the same ones as the OP used, but hopefully it can help figuring out this issue. 

Data is available at http://predict.cs.unm.edu/downloads.php. I'm using the OCD Flankers one in the second row. The code below assumes that file 914flankers.cnt in OCI Flankers/EEG Data is downloaded and saved in the local folder data/.

My code for loading the file is:
```
montage = mne.channels.read_montage(kind='standard_alphabetic')
mne.io.read_raw_cnt('data/914flankers.cnt', montage)

```
The error message I got is:

```
load.py:7: RuntimeWarning:   Could not parse meas date from the header. Setting to None.
  mne.io.read_raw_cnt('data/914flankers.cnt', montage, verbose=True)
Traceback (most recent call last):
  File "load.py", line 7, in <module>
    mne.io.read_raw_cnt('data/914flankers.cnt', montage, verbose=True)
  File "/Users/fanyang1/anaconda3/envs/mne/lib/python3.6/site-packages/mne/io/cnt/cnt.py", line 95, in read_raw_cnt
    preload=preload, verbose=verbose)
  File "/Users/fanyang1/anaconda3/envs/mne/lib/python3.6/site-packages/mne/io/cnt/cnt.py", line 361, in __init__
    data_format, date_format)
  File "/Users/fanyang1/anaconda3/envs/mne/lib/python3.6/site-packages/mne/io/cnt/cnt.py", line 251, in _get_cnt_info
    stim_channel[event_time - 1] = event_id
IndexError: index 583988 is out of bounds for axis 0 with size 582560
```
I've downloaded the file and I can reproduce the error
```py
import mne
import os.path as op

file_name = op.join(op.dirname(mne.__file__), '..', 'sandbox', 'data',
                    '914flankers.cnt')

montage = mne.channels.read_montage(kind='standard_1020')
raw = mne.io.read_raw_cnt(file_name, montage)
```
**EDIT:** Reading the paper, the montage should be `standard_1020` not `standard_alphabetic`. But the error remains.
```sh
IndexError: index 583988 is out of bounds for axis 0 with size 582560
```

**EDIT:** The `'914flankers.cnt'` contains positions, so it should be read with `montage=None`

Resolving patch:
diff --git a/mne/io/cnt/cnt.py b/mne/io/cnt/cnt.py
--- a/mne/io/cnt/cnt.py
+++ b/mne/io/cnt/cnt.py
@@ -12,7 +12,7 @@
 from ...channels.layout import _topo_to_sphere
 from ..constants import FIFF
 from ..utils import (_mult_cal_one, _find_channels, _create_chs, read_str,
-                     _deprecate_stim_channel)
+                     _deprecate_stim_channel, _synthesize_stim_channel)
 from ..meas_info import _empty_info
 from ..base import BaseRaw, _check_update_montage
 from ...annotations import Annotations
@@ -291,35 +291,14 @@ def _get_cnt_info(input_fname, eog, ecg, emg, misc, data_format, date_format,
             cals.append(cal * sensitivity * 1e-6 / 204.8)
 
         if stim_channel_toggle:
-            if event_offset > data_offset:
-                fid.seek(event_offset)
-                event_type = np.fromfile(fid, dtype='<i1', count=1)[0]
-                event_size = np.fromfile(fid, dtype='<i4', count=1)[0]
-                teeg_offset = np.fromfile(fid, dtype='<i4', count=1)[0]
-                assert teeg_offset == 0  # documentation say this should be 0
-                if event_type == 1:
-                    event_bytes = 8
-                elif event_type in (2, 3):
-                    event_bytes = 19
-                else:
-                    raise IOError('Unexpected event size.')
-
-                # XXX long NumEvents is available, why are not we using it?
-                n_events = event_size // event_bytes
-            else:
-                n_events = 0
-
-            stim_channel = np.zeros(n_samples)  # Construct stim channel
-            for i in range(n_events):
-                fid.seek(event_offset + 9 + i * event_bytes)
-                event_id = np.fromfile(fid, dtype='u2', count=1)[0]
-                fid.seek(event_offset + 9 + i * event_bytes + 4)
-                offset = np.fromfile(fid, dtype='<i4', count=1)[0]
-                if event_type == 3:
-                    offset *= n_bytes * n_channels
-                event_time = offset - 900 - 75 * n_channels
-                event_time //= n_channels * n_bytes
-                stim_channel[event_time - 1] = event_id
+            data_format = 'int32' if n_bytes == 4 else 'int16'
+            annot = _read_annotations_cnt(input_fname, data_format=data_format)
+            events = (np.stack((annot.onset * sfreq,
+                                annot.duration * sfreq,
+                                annot.description.astype(int)))
+                      .astype(int)
+                      .transpose())
+            stim_channel = _synthesize_stim_channel(events, n_samples)
 
     info = _empty_info(sfreq)
     if lowpass_toggle == 1:


Test patch: