Repository: mne-tools/mne-python

Issue:
BrainVision .vmrk positions are 1-indexed but read as if they were 0-indexed
In a BrainVision `.vmrk` file, the marker positions are 1-indexed (i.e., the smallest possible marker is 1, not 0). You can see this in their documentation, e.g., where it says "Time points [msec] = (Data points – 1) * sampling interval [msec]":

http://pressrelease.brainproducts.com/markers/

You can also infer this from the files which always start with a "New Segment" marker at position "1" (not position "0"), e.g.:

https://github.com/mne-tools/mne-python/blob/master/mne/io/brainvision/tests/data/testv2.vmrk#L14

Currently, `read_raw_brainvision` assumes that the marker positions in a .vmrk file are 0-indexed, which means that all markers read by mne appear one sample later than they're supposed to. (I ran into this because I had data where there should have been 100 samples before my first marker of interest, but mne kept giving me 101 samples.) 

This happens because `_read_vmrk_events` reads the 1-indexed onsets without modifying them:

https://github.com/mne-tools/mne-python/blob/master/mne/io/brainvision/brainvision.py#L276

The documentation for events in mne is a little unclear to me, but I assume that events in mne are 0-indexed, since all the numpy arrays are, and so the 1-indexed onsets should be converted to 0-indexed onsets when they are read from the BrainVision file. The fix should be as simple as changing the line referenced above to `onset = int(onset) - 1`.

I looked into creating a pull request for this, but I'm a little uncertain what to do since the tests currently test the wrong behavior, e.g.:

https://github.com/mne-tools/mne-python/blob/master/mne/io/brainvision/tests/test_brainvision.py#L379

Is it okay to correct the incorrect onset numbers in the current tests?
BrainVision .vmrk positions are 1-indexed but read as if they were 0-indexed
In a BrainVision `.vmrk` file, the marker positions are 1-indexed (i.e., the smallest possible marker is 1, not 0). You can see this in their documentation, e.g., where it says "Time points [msec] = (Data points – 1) * sampling interval [msec]":

http://pressrelease.brainproducts.com/markers/

You can also infer this from the files which always start with a "New Segment" marker at position "1" (not position "0"), e.g.:

https://github.com/mne-tools/mne-python/blob/master/mne/io/brainvision/tests/data/testv2.vmrk#L14

Currently, `read_raw_brainvision` assumes that the marker positions in a .vmrk file are 0-indexed, which means that all markers read by mne appear one sample later than they're supposed to. (I ran into this because I had data where there should have been 100 samples before my first marker of interest, but mne kept giving me 101 samples.) 

This happens because `_read_vmrk_events` reads the 1-indexed onsets without modifying them:

https://github.com/mne-tools/mne-python/blob/master/mne/io/brainvision/brainvision.py#L276

The documentation for events in mne is a little unclear to me, but I assume that events in mne are 0-indexed, since all the numpy arrays are, and so the 1-indexed onsets should be converted to 0-indexed onsets when they are read from the BrainVision file. The fix should be as simple as changing the line referenced above to `onset = int(onset) - 1`.

I looked into creating a pull request for this, but I'm a little uncertain what to do since the tests currently test the wrong behavior, e.g.:

https://github.com/mne-tools/mne-python/blob/master/mne/io/brainvision/tests/test_brainvision.py#L379

Is it okay to correct the incorrect onset numbers in the current tests?


Discussion:

Resolving patch:
diff --git a/mne/io/brainvision/brainvision.py b/mne/io/brainvision/brainvision.py
--- a/mne/io/brainvision/brainvision.py
+++ b/mne/io/brainvision/brainvision.py
@@ -276,7 +276,7 @@ def _read_vmrk_events(fname, event_id=None, response_trig_shift=0):
     events, dropped = list(), list()
     for info in items:
         mtype, mdesc, onset, duration = info.split(',')[:4]
-        onset = int(onset)
+        onset = int(onset) - 1  # BrainVision is 1-indexed, not 0-indexed
         duration = (int(duration) if duration.isdigit() else 1)
         if mdesc in event_id:
             trigger = event_id[mdesc]


Test patch:
diff --git a/mne/io/brainvision/tests/test_brainvision.py b/mne/io/brainvision/tests/test_brainvision.py
--- a/mne/io/brainvision/tests/test_brainvision.py
+++ b/mne/io/brainvision/tests/test_brainvision.py
@@ -371,22 +371,24 @@ def test_brainvision_vectorized_data():
 def test_events():
     """Test reading and modifying events."""
     tempdir = _TempDir()
+    # Note: BrainVision event offsets are 1-based, mne offsets are 0-based.
+    # So in all tests below, the "onset" is 1 less than what's in the file
 
     # check that events are read and stim channel is synthesized correctly
     raw = read_raw_brainvision(vhdr_path, eog=eog, event_id=event_id)
     events = raw._get_brainvision_events()
     events = events[events[:, 2] != event_id['Sync On']]
-    assert_array_equal(events, [[487, 0, 253],
-                                [497, 1, 255],
-                                [1770, 1, 254],
-                                [1780, 1, 255],
-                                [3253, 1, 254],
-                                [3263, 1, 255],
-                                [4936, 1, 253],
-                                [4946, 1, 255],
-                                [6000, 1, 255],
-                                [6620, 1, 254],
-                                [6630, 1, 255]])
+    assert_array_equal(events, [[486, 0, 253],
+                                [496, 1, 255],
+                                [1769, 1, 254],
+                                [1779, 1, 255],
+                                [3252, 1, 254],
+                                [3262, 1, 255],
+                                [4935, 1, 253],
+                                [4945, 1, 255],
+                                [5999, 1, 255],
+                                [6619, 1, 254],
+                                [6629, 1, 255]])
 
     # check that events are read and stim channel is synthesized correctly and
     # response triggers are shifted like they're supposed to be.
@@ -394,17 +396,17 @@ def test_events():
                                response_trig_shift=1000, event_id=event_id)
     events = raw._get_brainvision_events()
     events = events[events[:, 2] != event_id['Sync On']]
-    assert_array_equal(events, [[487, 0, 253],
-                                [497, 1, 255],
-                                [1770, 1, 254],
-                                [1780, 1, 255],
-                                [3253, 1, 254],
-                                [3263, 1, 255],
-                                [4936, 1, 253],
-                                [4946, 1, 255],
-                                [6000, 1, 1255],
-                                [6620, 1, 254],
-                                [6630, 1, 255]])
+    assert_array_equal(events, [[486, 0, 253],
+                                [496, 1, 255],
+                                [1769, 1, 254],
+                                [1779, 1, 255],
+                                [3252, 1, 254],
+                                [3262, 1, 255],
+                                [4935, 1, 253],
+                                [4945, 1, 255],
+                                [5999, 1, 1255],
+                                [6619, 1, 254],
+                                [6629, 1, 255]])
 
     # check that events are read and stim channel is synthesized correctly and
     # response triggers are ignored.
@@ -413,16 +415,16 @@ def test_events():
                                    response_trig_shift=None)
     events = raw._get_brainvision_events()
     events = events[events[:, 2] != event_id['Sync On']]
-    assert_array_equal(events, [[487, 0, 253],
-                                [497, 1, 255],
-                                [1770, 1, 254],
-                                [1780, 1, 255],
-                                [3253, 1, 254],
-                                [3263, 1, 255],
-                                [4936, 1, 253],
-                                [4946, 1, 255],
-                                [6620, 1, 254],
-                                [6630, 1, 255]])
+    assert_array_equal(events, [[486, 0, 253],
+                                [496, 1, 255],
+                                [1769, 1, 254],
+                                [1779, 1, 255],
+                                [3252, 1, 254],
+                                [3262, 1, 255],
+                                [4935, 1, 253],
+                                [4945, 1, 255],
+                                [6619, 1, 254],
+                                [6629, 1, 255]])
 
     # check that events are read properly when event_id is specified for
     # auxiliary events
@@ -431,17 +433,17 @@ def test_events():
                                    response_trig_shift=None,
                                    event_id=event_id)
     events = raw._get_brainvision_events()
-    assert_array_equal(events, [[487, 0, 253],
-                                [497, 1, 255],
-                                [1770, 1, 254],
-                                [1780, 1, 255],
-                                [3253, 1, 254],
-                                [3263, 1, 255],
-                                [4936, 1, 253],
-                                [4946, 1, 255],
-                                [6620, 1, 254],
-                                [6630, 1, 255],
-                                [7630, 1, 5]])
+    assert_array_equal(events, [[486, 0, 253],
+                                [496, 1, 255],
+                                [1769, 1, 254],
+                                [1779, 1, 255],
+                                [3252, 1, 254],
+                                [3262, 1, 255],
+                                [4935, 1, 253],
+                                [4945, 1, 255],
+                                [6619, 1, 254],
+                                [6629, 1, 255],
+                                [7629, 1, 5]])
 
     assert_raises(TypeError, read_raw_brainvision, vhdr_path, eog=eog,
                   preload=True, response_trig_shift=0.1)
@@ -449,17 +451,17 @@ def test_events():
                   preload=True, response_trig_shift=np.nan)
 
     # to handle the min duration = 1 of stim trig (re)construction ...
-    events = np.array([[487, 1, 253],
-                       [497, 1, 255],
-                       [1770, 1, 254],
-                       [1780, 1, 255],
-                       [3253, 1, 254],
-                       [3263, 1, 255],
-                       [4936, 1, 253],
-                       [4946, 1, 255],
-                       [6620, 1, 254],
-                       [6630, 1, 255],
-                       [7630, 1, 5]])
+    events = np.array([[486, 1, 253],
+                       [496, 1, 255],
+                       [1769, 1, 254],
+                       [1779, 1, 255],
+                       [3252, 1, 254],
+                       [3262, 1, 255],
+                       [4935, 1, 253],
+                       [4945, 1, 255],
+                       [6619, 1, 254],
+                       [6629, 1, 255],
+                       [7629, 1, 5]])
 
     # Test that both response_trig_shit and event_id can be set
     read_raw_brainvision(vhdr_path, eog=eog, preload=False,