Repository: mne-tools/mne-python

Issue:
find_events with negative STI101 values
Hi,
I read events from STI101 using find_events
Because I want to import my data in SPM, I also create a trl struct with fieldtrip
And now, I am confused because events are different between mne-python and fieldtrip
Looking at the code, it seems to me that to deal with possible negative values in STI101, mne takes the absolute values whereas fieltrip casts the values from int16 to uint16 (read_trigger.m)
By example, with the value -32765, 
mne : -32765 => 32765 => ....101 => STI101_03
ft :      -32765 => 32771 => ....011 => STI101_02
What am I doing wrong?
Thanks



Discussion:
> What am I doing wrong?

nothing I guess. We just have a different default behavior. Can you fix
manually?

how can I know which "default behavior" is suited for my data?

Well you should know the paradigm right?

Just to make sure, are you talking about a mismatch in terms of event values, or in terms of event timing?

If you're talking about a time shift between the MNE and Fieldtrip triggers, it may be due to the fact that Fieldtrip doesn't read the `first_sample` information in the raw data, but always considers that the first time sample is 0.

You can check this by doing `raw.first_samp`.

i am talking about event coded values, not event timing
at sample N of channel STI101,
mne detects activation of bit#2 of STI101
while
ft detects activation of bit#3 of STI101
and at this point, this is independent of my paradigm
now if i introduce a paradigm, where cond2 is bit#2 of STI101 and cond3 is bit#3 of STI101, for sample N of STI101, 
mne will trigger cond2
and
ft will trigger cond3
which seems incoherent to me

From what I know, both MNE and Fieldtrip read the event channel as an array of  continuous (decimal) values.

If you encoded your TTL values in a binary format to code for your conditions (which is indeed often helpful), you'll still have to "decode" these values.
- If the TTL decoding error comes from a sign flip as you suggest, you can do:

```
import numpy as np
chan = np.where(np.array(raw.ch_names) == 'STI_101')[0][0]
raw._data[chan, :] *= -1
```
- Another possible source of error is that the TTL baseline may be 255, instead of 0, in which case you'd need to do

```
raw._data[chan, :] = 255 - raw._data[chan, :]
```
- Finally, note that in python, unlike matlab, indices start at 0 so `foo(1)` in matlab is the same as `foo[0]` in python.

i suggest the problem comes from the interpretation of the values of STI101
after the reading as an array of continuous decimal values, the values should be reinterpreted as an array of unsigned int (instead of taking the absolute value of an array of signed int)
[ i did it manually and retrieved the right count and timing for my events of interest ]

this difference of interpretation is a problem only with a few files, those which use trigger16 (to track MRI acquisition)
in this case, because trigger16 may correspond to a sign bit, negative values may appear in the array of continuous values and taking the absolute value lead to different events compared as reinterpretating the array of values as unsigned int
when trigger16 is not used, negative values are not present and taking the absolute has no effect

> the values should be reinterpreted as an array of unsigned int (instead of taking the absolute value of an array of signed int)

@agramfort  @Eric89GXL Does this seem reasonable to you?

@jean-didier My general impression is that your problem is quite specific to your usecase, and changing MNE default's behavior could have a negative impact on other's work.

Also, my feeling is that trigger values should not have negative signs in the first place. We may however want to make the documentation more specific to detail how negative values are handled, or indeed just prevent user to use negative values.

We have a model/api for dealing with non int events in brainvision. We use a dict, and the process goes:For each event,     - try looking it up in the dict, else    - see if it is an int, else    - discard it
This will also be implemented for the eeglab reader once I stop being really lazy. 

> the values should be reinterpreted as an array of unsigned int (instead of taking the absolute value of an array of signed int)

We could add an argument I suppose. Something like `casting='abs'` (current behavior) and `casting='uint'` might work. But the annoying part is that it would need to happen during the reading process, so it would mean another arg in the constructor. It doesn't seem all that clean.

@jean-didier are you reading in a .fif file, or some other format? How was the file created? I wonder if you could adjust the events before or during conversion yourself with the `uint` casting?

yes i am reading a _tsss.fif
yes i adjusted manually data of STI101 channel with uint cast before calling the find_event function (abs call has no more effect) but i am afraid the problem concern all users using bit16 of STI101

I would be fine adding an option to the find_events function.

@jean-didier would be ok to open an PR to help us with this as you're the
one reporting the pb?

FYI, I just tested on our Neuromag system, and we have the same problem (event on line 16 interpreted as -32768). I would argue that assuming the tests don't break switching to `uint` may be more appropriate (I see it as a bug fix on the fif reader (maybe just when you read STI channels on neuromag and avoid the more user facing `find_events`), as this doesn't seem to be at just a single site. thoughts? 

What do the C tools do?

There is a known bug in the acquisition code which makes this happen. 

In think

mne_fix_stim14 —32 <files> 

fixes this.
- Matti

> On Feb 29, 2016, at 6:30 PM, Eric Larson notifications@github.com wrote:
> 
> What do the C tools do?
> —
> Reply to this email directly or view it on GitHub https://github.com/mne-tools/mne-python/issues/2959#issuecomment-190449007.

The information in this e-mail is intended only for the person to whom it is
addressed. If you believe this e-mail was sent to you in error and the e-mail
contains patient information, please contact the Partners Compliance HelpLine at
http://www.partners.org/complianceline . If the e-mail was sent to you in error
but does not contain patient information, please contact the sender and properly
dispose of the e-mail.

@jean-didier can you try that function and see what it does to the data? Is it consistent with what FieldTrip does? If it differs, I suspect we should follow @mshamalainen's solution.

indeed, mne_fix_stim14 —32 <files> fixes the bug
ft give same events for raw and raw_fixed
mne give different events between raw and raw_fixed
mne using raw_fixed now gives same events as ft
Thanks !

cool. If you're not planning to fix the pb in python can you just share a
file so we can test our fix?

The fix is simple, see the C code. - Matti

> On Mar 1, 2016, at 8:07 AM, Alexandre Gramfort notifications@github.com wrote:
> 
> cool. If you're not planning to fix the pb in python can you just share a
> file so we can test our fix?
> —
> Reply to this email directly or view it on GitHub https://github.com/mne-tools/mne-python/issues/2959#issuecomment-190716549.

The information in this e-mail is intended only for the person to whom it is
addressed. If you believe this e-mail was sent to you in error and the e-mail
contains patient information, please contact the Partners Compliance HelpLine at
http://www.partners.org/complianceline . If the e-mail was sent to you in error
but does not contain patient information, please contact the sender and properly
dispose of the e-mail.

@agramfort I suspect we can "fake it" based on what @jean-didier told us about the `events` channel (we can put in appropriate values). I'll see if I have time to put in the C fix.

please go for it while I am in documentation mode :)


Resolving patch:
diff --git a/mne/event.py b/mne/event.py
--- a/mne/event.py
+++ b/mne/event.py
@@ -406,9 +406,9 @@ def find_stim_steps(raw, pad_start=None, pad_stop=None, merge=0,
                             pad_stop=pad_stop, merge=merge)
 
 
-@verbose
 def _find_events(data, first_samp, verbose=None, output='onset',
-                 consecutive='increasing', min_samples=0, mask=0):
+                 consecutive='increasing', min_samples=0, mask=0,
+                 uint_cast=False):
     """Helper function for find events"""
     if min_samples > 0:
         merge = int(min_samples // 1)
@@ -417,10 +417,15 @@ def _find_events(data, first_samp, verbose=None, output='onset',
     else:
         merge = 0
 
-    if np.any(data < 0):
-        warn('Trigger channel contains negative values, using absolute value.')
-        data = np.abs(data)  # make sure trig channel is positive
     data = data.astype(np.int)
+    if uint_cast:
+        data = data.astype(np.uint16).astype(np.int)
+    if data.min() < 0:
+        warn('Trigger channel contains negative values, using absolute '
+             'value. If data were acquired on a Neuromag system with '
+             'STI016 active, consider using uint_cast=True to work around '
+             'an acquisition bug')
+        data = np.abs(data)  # make sure trig channel is positive
 
     events = _find_stim_steps(data, first_samp, pad_stop=0, merge=merge)
     events = _mask_trigs(events, mask)
@@ -473,9 +478,9 @@ def _find_events(data, first_samp, verbose=None, output='onset',
 
 
 @verbose
-def find_events(raw, stim_channel=None, verbose=None, output='onset',
+def find_events(raw, stim_channel=None, output='onset',
                 consecutive='increasing', min_duration=0,
-                shortest_event=2, mask=0):
+                shortest_event=2, mask=0, uint_cast=False, verbose=None):
     """Find events from raw file
 
     Parameters
@@ -489,8 +494,6 @@ def find_events(raw, stim_channel=None, verbose=None, output='onset',
         etc. are read. If these are not found, it will fall back to
         'STI 014' if present, then fall back to the first channel of type
         'stim', if present.
-    verbose : bool, str, int, or None
-        If not None, override default verbose level (see mne.verbose).
     output : 'onset' | 'offset' | 'step'
         Whether to report when events start, when events end, or both.
     consecutive : bool | 'increasing'
@@ -509,6 +512,17 @@ def find_events(raw, stim_channel=None, verbose=None, output='onset',
     mask : int
         The value of the digital mask to apply to the stim channel values.
         The default value is 0.
+    uint_cast : bool
+        If True (default False), do a cast to ``uint16`` on the channel
+        data. This can be used to fix a bug with STI101 and STI014 in
+        Neuromag acquisition setups that use channel STI016 (channel 16
+        turns data into e.g. -32768), similar to ``mne_fix_stim14 --32``
+        in MNE-C.
+
+        .. versionadded:: 0.12
+
+    verbose : bool, str, int, or None
+        If not None, override default verbose level (see mne.verbose).
 
     Returns
     -------
@@ -597,7 +611,7 @@ def find_events(raw, stim_channel=None, verbose=None, output='onset',
 
     events = _find_events(data, raw.first_samp, verbose=verbose, output=output,
                           consecutive=consecutive, min_samples=min_samples,
-                          mask=mask)
+                          mask=mask, uint_cast=uint_cast)
 
     # add safety check for spurious events (for ex. from neuromag syst.) by
     # checking the number of low sample events


Test patch:
diff --git a/mne/tests/test_event.py b/mne/tests/test_event.py
--- a/mne/tests/test_event.py
+++ b/mne/tests/test_event.py
@@ -28,6 +28,20 @@
 raw_fname = op.join(base_dir, 'test_raw.fif')
 
 
+def test_fix_stim():
+    """Test fixing stim STI016 for Neuromag"""
+    raw = io.Raw(raw_fname, preload=True)
+    # 32768 (016) + 3 (002+001) bits gets incorrectly coded during acquisition
+    raw._data[raw.ch_names.index('STI 014'), :3] = [0, -32765, 0]
+    with warnings.catch_warnings(record=True) as w:
+        events = find_events(raw, 'STI 014')
+    assert_true(len(w) >= 1)
+    assert_true(any('STI016' in str(ww.message) for ww in w))
+    assert_array_equal(events[0], [raw.first_samp + 1, 0, 32765])
+    events = find_events(raw, 'STI 014', uint_cast=True)
+    assert_array_equal(events[0], [raw.first_samp + 1, 0, 32771])
+
+
 def test_add_events():
     """Test adding events to a Raw file"""
     # need preload