Repository: mne-tools/mne-python

Issue:
BUG: BV reader now fails for certain modern files
The most recent software by BrainProducts writes BrainVision files with integer codes. E.g.

```
Brain Vision Data Exchange Marker File, Version 1.0

[Common Infos]
Codepage=UTF-8
DataFile=BSN17_vw.eeg

[Marker Infos]
; Each entry: Mk<Marker number>=<Type>,<Description>,<Position in data points>,
; <Size in data points>, <Channel number (0 = marker is related to all channels)>
; Fields are delimited by commas, some fields might be omitted (empty).
; Commas in type or description text are coded as "\1".
Mk1=New Segment,,1,1,0,20160727190324526000
Mk2=Stimulus, 24,38759,1,0
Mk3=Stimulus,200,39169,1,0
Mk4=Stimulus,198,42426,1,0
Mk5=Stimulus,199,42843,1,0
Mk6=Stimulus, 99,45641,1,0
Mk7=Stimulus,201,46051,1,0
Mk8=Stimulus, 47,48690,1,0
Mk9=Stimulus,200,49100,1,0
```

As of #5426, Triggers such as ` 24` will currently not be parsed. Moreover, the warning string is somewhat deceptive, stating that "triggers such as [ 24, 200, 198 ..." will be dropped, but the whitespace matters here; e.g., only the last entry here would match: `{"24": 24, 24: 24, " 24": 24}`.

Given that Pycorder is a manufacturer-supplied and open, Python-based software, we should support such event codes by default.

@sappelhoff 


Discussion:
I agree, the commit introduces regressions so that loading brainvision files fails after updating to the current master.

In addition to the Pycorder issue, the new regex does not match response markers, which defeats the whole purpose of the ``trig_shift_by_type`` parameter (or the old ``response_trig_shift``). The tests only run because the ``R255`` marker was added to ``event_id`` (https://github.com/mne-tools/mne-python/blob/master/mne/io/brainvision/tests/test_brainvision.py#L65) .
@hekolk why does this defeat the purpose of `trig_shift_by_type`? If someone has response markers, they can add those to the `event_id` and then they will be shifted.



Resolving patch:
diff --git a/mne/io/brainvision/brainvision.py b/mne/io/brainvision/brainvision.py
--- a/mne/io/brainvision/brainvision.py
+++ b/mne/io/brainvision/brainvision.py
@@ -65,13 +65,13 @@ class RawBrainVision(BaseRaw):
         triggers will be ignored. Default is 0 for backwards compatibility,
         but typically another value or None will be necessary.
     event_id : dict | None
-        The id of special events to consider in addition to those that
-        follow the normal Brainvision trigger format ('S###').
-        If dict, the keys will be mapped to trigger values on the stimulus
-        channel. Example: {'SyncStatus': 1; 'Pulse Artifact': 3}. If None
-        or an empty dict (default), only stimulus and response events are added
-        to the stimulus channel. Keys are case sensitive. "New Segment" markers
-        are always dropped.
+        Special events to consider in addition to those that follow the normal
+        BrainVision trigger format ('###' with an optional single character
+        prefix). If dict, the keys will be mapped to trigger values on the
+        stimulus channel. Example: {'SyncStatus': 1; 'Pulse Artifact': 3}.
+        If None or an empty dict (default), only BrainVision format events are
+        added to the stimulus channel. Keys are case sensitive. "New Segment"
+        markers are always dropped.
     verbose : bool, str, int, or None
         If not None, override default verbose level (see :func:`mne.verbose`
         and :ref:`Logging documentation <tut_logging>` for more).
@@ -231,13 +231,13 @@ def _read_vmrk_events(fname, event_id=None, trig_shift_by_type=None):
     fname : str
         vmrk file to be read.
     event_id : dict | None
-        The id of special events to consider in addition to those that
-        follow the normal Brainvision trigger format ('S###').
-        If dict, the keys will be mapped to trigger values on the stimulus
-        channel. Example: {'SyncStatus': 1; 'Pulse Artifact': 3}. If None
-        or an empty dict (default), only stimulus and response events are added
-        to the stimulus channel. Keys are case sensitive. "New Segment" markers
-        are always dropped.
+        Special events to consider in addition to those that follow the normal
+        BrainVision trigger format ('###' with an optional single character
+        prefix). If dict, the keys will be mapped to trigger values on the
+        stimulus channel. Example: {'SyncStatus': 1; 'Pulse Artifact': 3}.
+        If None or an empty dict (default), only BrainVision format events are
+        added to the stimulus channel. Keys are case sensitive. "New Segment"
+        markers are always dropped.
     response_trig_shift : int | None
         Integer to shift response triggers by. None ignores response triggers.
 
@@ -323,7 +323,14 @@ def _read_vmrk_events(fname, event_id=None, trig_shift_by_type=None):
             trigger = event_id[mdesc]
         else:
             try:
-                trigger = int(re.findall(r'S([\s\d]{2}\d{1})', mdesc)[0])
+                # Match any three digit marker value (padded with whitespace).
+                # In BrainVision Recorder, the markers sometimes have a prefix
+                # depending on the type, e.g., Stimulus=S, Response=R,
+                # Optical=O, ... Note that any arbitrary stimulus type can be
+                # defined. So we match any single character that is not
+                # forbidden by BrainVision Recorder: [^a-z$%\-@/\\|;,:.\s]
+                marker_regexp = r'^[^a-z$%\-@/\\|;,:.\s]{0,1}([\s\d]{2}\d{1})$'
+                trigger = int(re.findall(marker_regexp, mdesc)[0])
             except IndexError:
                 trigger = None
         if mtype.lower() in trig_shift_by_type:
@@ -351,13 +358,10 @@ def _read_vmrk_events(fname, event_id=None, trig_shift_by_type=None):
 
     if len(dropped) > 0:
         dropped = list(set(dropped))
-        examples = ", ".join(dropped[:5])
-        if len(dropped) > 5:
-            examples += ", ..."
-        warn("Currently, {0} trigger(s) will be dropped, such as [{1}]. "
+        warn("Currently, {0} trigger(s) will be dropped, such as {1}. "
              "Consider using ``event_id`` to parse triggers that "
-             "do not follow the 'S###' pattern.".format(
-                 len(dropped), examples))
+             "do not follow the '###' pattern with an optional single "
+             "character prefix.".format(len(dropped), dropped[:5]))
 
     events = np.array(events).reshape(-1, 3)
     return events
@@ -872,13 +876,13 @@ def read_raw_brainvision(vhdr_fname, montage=None,
         triggers will be ignored. Default is 0 for backwards compatibility,
         but typically another value or None will be necessary.
     event_id : dict | None
-        The id of special events to consider in addition to those that
-        follow the normal Brainvision trigger format ('S###').
-        If dict, the keys will be mapped to trigger values on the stimulus
-        channel. Example: {'SyncStatus': 1; 'Pulse Artifact': 3}. If None
-        or an empty dict (default), only stimulus and response events are added
-        to the stimulus channel. Keys are case sensitive. "New Segment" markers
-        are always dropped.
+        Special events to consider in addition to those that follow the normal
+        BrainVision trigger format ('###' with an optional single character
+        prefix). If dict, the keys will be mapped to trigger values on the
+        stimulus channel. Example: {'SyncStatus': 1; 'Pulse Artifact': 3}.
+        If None or an empty dict (default), only BrainVision format events are
+        added to the stimulus channel. Keys are case sensitive. "New Segment"
+        markers are always dropped.
     verbose : bool, str, int, or None
         If not None, override default verbose level (see :func:`mne.verbose`
         and :ref:`Logging documentation <tut_logging>` for more).


Test patch:
diff --git a/mne/io/brainvision/tests/data/testv2.vmrk b/mne/io/brainvision/tests/data/testv2.vmrk
--- a/mne/io/brainvision/tests/data/testv2.vmrk
+++ b/mne/io/brainvision/tests/data/testv2.vmrk
@@ -22,8 +22,19 @@ Mk8=Stimulus,S255,3263,1,0
 Mk9=Stimulus,S253,4936,1,0
 Mk10=Stimulus,S255,4946,1,0
 Mk11=Response,R255,6000,1,0
-Mk12=Stimulus,S254,6620,1,0
-Mk13=Stimulus,S255,6630,1,0
+
+;Removing the prefix makes the marker "PyCorder" style
+;The removal of the prefix should not change the parsing
+Mk12=Stimulus,254,6620,1,0
+Mk13=Stimulus,255,6630,1,0
+
+;Do not parse by default if not in shape '###' + optional single char prefix
+Mk14=Comment,This will not be parsed by default 13,8010,1,0
+Mk15=Comment,Not parsed by default either S456 ms,8020,1,0
+
+;Certain prefixes are blacklisted by BrainVision recorder: [^a-z$%\-@/\\|;,:.\s]
+;Will not be parsed by default
+Mk16=$User_Spec,$ 18,8030,1,0
 
 [Marker User Infos]
 ; Each entry: Prop<Number>=Mk<Marker number>,<Type>,<Name>,<Value>,<Value2>,...,<ValueN>
diff --git a/mne/io/brainvision/tests/test_brainvision.py b/mne/io/brainvision/tests/test_brainvision.py
--- a/mne/io/brainvision/tests/test_brainvision.py
+++ b/mne/io/brainvision/tests/test_brainvision.py
@@ -62,7 +62,7 @@
 montage = op.join(data_dir, 'test.hpts')
 eeg_bin = op.join(data_dir, 'test_bin_raw.fif')
 eog = ['HL', 'HR', 'Vb']
-event_id = {'Sync On': 5, 'O  1': 1, 'R255': 255}
+event_id = {'Sync On': 5}
 
 
 def test_vmrk_meas_date():
@@ -509,7 +509,7 @@ def test_events():
     with pytest.warns(RuntimeWarning, match='channel types to misc'):
         raw = read_raw_brainvision(vhdr_v2_path)
     events = raw._get_brainvision_events()
-    assert events.shape == (10, 3)  # shape of events without comment/response
+    assert events.shape == (11, 3)  # shape of events without comment
 
     # with event_id specified, get that comment and assert it's there
     tmp_event_id = {'comment using [square] brackets': 999}
@@ -517,7 +517,7 @@ def test_events():
         raw = read_raw_brainvision(vhdr_v2_path, event_id=tmp_event_id)
     events = raw._get_brainvision_events()
     assert 999 in events[:, -1]
-    assert events.shape == (11, 3)  # shape of events without response
+    assert events.shape == (12, 3)  # shape of events with comment
 
     # check that events are read properly when event_id is specified for
     # auxiliary events