Repository: mne-tools/mne-python

Issue:
mne.read_events not reading all events
On current master, `mne.read_events` is omitting 20 rows.  MWE is below.  The file "sample-eve.txt" is [here](http://dan.mccloy.info/data/sample-eve.txt) and was generated by `mne.write_events`.

``` python
import mne
import pandas

mne_events = mne.read_events('sample-eve.txt')
pandas_df = pandas.read_csv('sample-eve.txt', sep=r'\s*', header=None)

print(mne_events.shape)  # (4620, 3) -- 20 rows missing
print(pandas_df.shape)   # (4640, 3)
```



Discussion:
oh, I get it.  I have re-coded the event values during processing to replace all `1` events with a stimulus code reconstructed from a string of preceding `4` and `8` triggers.  20 of those recoded event values are `0`, and default is `mask=0` so they're getting (silently) skipped.  Unfortunately `mask=-1` doesn't work, nor does `mask=None`.  Is my only option to re-do my stimulus codes?  That is very inconvenient, as they act as indices into several data structures.

One problem with a code of `0` is that, in the first line, it has a special meaning (or used to?) for the C tools, so it goes beyond the `mask` problem, see:

https://github.com/mne-tools/mne-python/blob/master/mne/event.py#L260

I suppose we could add a `mask=None` argument for this use case, and state clearly in the docstring that it could cause problems if events are used by the C tools.

@drammock, I'm not sure I see the problem, with when you do the re-coding to set those instead of to 0 to 1023 or some other value?

(Personally I do a uniform +1 shift and later -1 shift as needed, but it is annoying in some cases)

@dgwakeman of course, that is a work-around I considered (as is Eric's suggestion to shift all events by 1).  But such workarounds strike me as inelegant and error-prone. As someone who has only rarely used the C API and will likely never do so again, the [zero-as-a-special-case](https://github.com/mne-tools/mne-python/blob/master/mne/event.py#L260) that @Eric89GXL pointed out strikes me as, if not cruft, then at least something that should arguably not be the default, and at a bare minimum should at least be preventable via `mask=None` or some equivalent means.

(if devs/BDFL won't agree even to that, then perhaps at least a `warning` or `info` when reading in the events, saying how many were read and how many were skipped?)

I guess, I should of asked how the recoding is being done (personally, I've never needed to recode, I use the mask option and choose the values I want when I design the experiment). Maybe, we need better code for recoding stimuli? It seems pretty silly to have an event entry that reads `time 0 0`, if we stop creating events that are `time 0 0` it should avoid the problem. How will some other researcher read an event where the trigger went from 0 -> 0

> I should of asked how the recoding is being done

There are many potential use cases for re-coding event numbers -- just think about e.g. changing them based on subject response correctness, correcting values that were incorrect during acquisition, or inserting sub-trial-level stamps that weren't put in during the trial (e.g., word onsets in a continuous sentence). So I'm not really sure the particular re-coding motivation is relevant, since we don't want to force people to only use the triggers that the MEG machine recorded, plus masks later. But if you're just curious, we use a serially-stamped (before trial start) ID to avoid some hardware issues we had a long time ago at MGH, and are being paranoid that they might reoccur, so all our trials actually start with a 1 trigger, and thus need to be re-coded to be meaningful.

> How will some other researcher read an event where the trigger went from 0 -> 0

In my mind, event values are no longer this physical change you're talking about related to changes on TTL lines, but rather just numbers that are used to control the abstractions about an experiment. So 1 and 2 and 3 and even 0 could have meaning once you're at the point of reading and writing event lists. Somewhere in the code, 1, 2, 3 and just as easily (well maybe not currently) 0 are given more meaning by the `event_id` specific to the paradigm.

> then perhaps at least a warning or info when reading in the events

I agree at a minimum we need to warn when there are `zero` events. At this point it is probably reasonable to do it even for the zeroes that appear in the first line -- people working with legacy event lists will get a warning, but will hopefully know to disregard it.

+1 for warning


Resolving patch:
diff --git a/mne/event.py b/mne/event.py
--- a/mne/event.py
+++ b/mne/event.py
@@ -207,9 +207,9 @@ def read_events(filename, include=None, exclude=None, mask=0):
         A event id to exclude or a list of them.
         If None no event is excluded. If include is not None
         the exclude parameter is ignored.
-    mask : int
+    mask : int | None
         The value of the digital mask to apply to the stim channel values.
-        The default value is 0.
+        The default value is 0. ``None`` skips masking.
 
     Returns
     -------
@@ -257,12 +257,19 @@ def read_events(filename, include=None, exclude=None, mask=0):
             raise ValueError('Unknown number of columns in event text file')
 
         event_list = lines[:, goods]
-        if event_list.shape[0] > 0 and event_list[0, 2] == 0:
+        if (mask is not None and event_list.shape[0] > 0 and
+                event_list[0, 2] == 0):
             event_list = event_list[1:]
+            warn('first row of event file discarded (zero-valued)')
 
     event_list = pick_events(event_list, include, exclude)
-    event_list = _mask_trigs(event_list, mask)
-
+    unmasked_len = event_list.shape[0]
+    if mask is not None:
+        event_list = _mask_trigs(event_list, mask)
+        masked_len = event_list.shape[0]
+        if masked_len < unmasked_len:
+            warn('{0} of {1} events masked'.format(unmasked_len - masked_len,
+                                                   unmasked_len))
     return event_list
 
 


Test patch:
diff --git a/mne/tests/test_event.py b/mne/tests/test_event.py
--- a/mne/tests/test_event.py
+++ b/mne/tests/test_event.py
@@ -117,7 +117,10 @@ def test_io_events():
     write_events(op.join(tempdir, 'events.eve'), events)
     events2 = read_events(op.join(tempdir, 'events.eve'))
     assert_array_almost_equal(events, events2)
-    events2 = read_events(fname_txt_mpr)
+    with warnings.catch_warnings(record=True) as w:
+        warnings.simplefilter('always')
+        events2 = read_events(fname_txt_mpr)
+        assert_true(sum('first row of' in str(ww.message) for ww in w) == 1)
     assert_array_almost_equal(events, events2)
 
     # Test old format text file IO
@@ -138,6 +141,13 @@ def test_io_events():
     assert_array_equal(a, c)
     assert_array_equal(a, d)
 
+    # test reading file with mask=None
+    events2 = events.copy()
+    events2[:, -1] = range(events2.shape[0])
+    write_events(op.join(tempdir, 'events-eve.fif'), events2)
+    events3 = read_events(op.join(tempdir, 'events-eve.fif'), mask=None)
+    assert_array_almost_equal(events2, events3)
+
     # Test binary file IO for 1 event
     events = read_events(fname_1)  # Use as the new gold standard
     write_events(op.join(tempdir, 'events-eve.fif'), events)
@@ -172,7 +182,10 @@ def test_find_events():
     assert_array_almost_equal(events, events2)
     # now test with mask
     events11 = find_events(raw, mask=3)
-    events22 = read_events(fname, mask=3)
+    with warnings.catch_warnings(record=True) as w:
+        warnings.simplefilter('always')
+        events22 = read_events(fname, mask=3)
+        assert_true(sum('events masked' in str(ww.message) for ww in w) == 1)
     assert_array_equal(events11, events22)
 
     # Reset some data for ease of comparison