Repository: mne-tools/mne-python

Issue:
FIX/ENH: loading EDF annotations
I made sure that the stim channel does not get resampled, and that the int16 values (each containing two characters) are not clamped.

Parsing the stim channel to get events of the form [onset, duration, 'annotation'] remains to be done.

I'm not sure where to store these events, though. Currently I consider not to store the events at all. Instead, a member function would simply parse and return the event list. Any thoughts?



Discussion:
@t3on @christianbrodbeck I guess this is for you :)

[![Coverage Status](https://coveralls.io/builds/702670/badge)](https://coveralls.io/builds/702670)

Coverage increased (+0.12%) when pulling **a83f0c94e16d4b14374e7ef4fdbf58dec41b212e on kazemakase:fix-edf-annotations** into **9a88a37cf92c92eaad4a57c8bf5d2fa768be8afc on mne-tools:master**.

I've added a parser as a private member of RawEDF. Not sure how to expose the functionality, though.
What is the correct format of the event list?

BTW, how do you get coveralls to automatically comment on PRs? :)

we need the help of our EDF experts to review :) ping @t3on @christianbrodbeck 

I have no experience with this format...

in the MNE way events are used to generate a stim channel that can be parsed with find_events function

> in the MNE way events are used to generate a stim channel that can be parsed with find_events function

In that case I might have got it wrong. I thought the stim channel was specific to EDF.
Maybe it is necessary to convert the EDF annotation channel to an MNE compatible stim channel?

> In that case I might have got it wrong. I thought the stim channel was
> specific to EDF.
> Maybe it is necessary to convert the EDF annotation channel to an MNE
> compatible stim channel?
> 
> isn't that the case already?

find_events works on an EDF raw?

It does not work on data where events are stored as EDF+ annotations: http://www.edfplus.info/specs/edfplus.html#edfplusannotations

@kazemakase Coveralls will automatically comment once the build is passing (it's currently not). But it's actually a bit better to rely on your own local nose/coverage output, though, since Travis only runs a subset of the full test suite.

@t3on ? I've never worked with EDF either...

@eric89gxl I was just wondering because i can't get coveralls to comment on PRs in the SCoT repo even when the build is passing.

Ahh, for a different repo... you have to enable it on `coveralls.io` somewhere. 

Assuming you've seen that option and enabled it, you also have to have `.travis.yml` configured properly. For some reason some of the repos I work on it works properly, and others it doesn't. I haven't tracked down what subtle differences between the repos cause some to get commented on, and others not. It appears to work for this repo, so you could try tracking down the differences between your travis config and ours.

Yep, that option is enabled and coveralls receives coverage information from travis. Good to know it doesn't work everywhere. Maybe it doesn't like capital letters in the repo name? :) I'll have another look at .travis.yml.

sorry, just got notified. my email was silent on this ;) I am not clear on what the issue is. there is a private function, `_read_annot`, that takes care of the annotation file with the format described in the MNE manual. if the triggering is stored in the data, it is also capable of parsing it with the specification of the stim channel.

@t3on I assume you refer to the stim channel and event files as described here: http://martinos.org/mne/stable/manual/browse.html#events-and-annotations

Everything is fine if the EDF files in question were created according to that specification. However, I am trying to load data recorded by someone who never even heard of MNE. There is no stim channel or annotation file. Instead, there is a channel that contains ascii encoded tags (see my link above), which the current implementation is unable to read correctly.

> Instead, there is a channel that contains ascii encoded tags (see my link above), which the current implementation is unable to read correctly.

do you have an example of a file that has the text annotation as channel in the data?

Sure, I was working with that file: http://www.physionet.org/physiobank/database/eegmmidb/S001/S001R06.edf

If i got this right, MNE expects a certain stim_channel where events are encoded. There are two ways to obtain a stim_channel for an edf file:
a) it is stored in the edf
b) an annotation file is parsed that replaces the stim channel

I have added a parameter `tal_channel` which can be used to specify an EDF+ annotation channel. I plan to create the stim_channel from the tal_channel if no annotation file is specified.

Parsing the tal channel works already, but how are events encoded in the stim channel?

> Parsing the tal channel works already, but how are events encoded in the stim channel?

Have a look at the find_events function for example

[![Coverage Status](https://coveralls.io/builds/715965/badge)](https://coveralls.io/builds/715965)

Coverage remained the same when pulling **4a962143e1cbab5a7042462c88e2caafd23c3270 on kazemakase:fix-edf-annotations** into **9a88a37cf92c92eaad4a57c8bf5d2fa768be8afc on mne-tools:master**.

[![Coverage Status](https://coveralls.io/builds/716006/badge)](https://coveralls.io/builds/716006)

Coverage increased (+0.14%) when pulling **9c25cf7c3ef195a4c4495d82dabc5f24cdb180ef on kazemakase:fix-edf-annotations** into **9a88a37cf92c92eaad4a57c8bf5d2fa768be8afc on mne-tools:master**.

Is there a way to encode simultaneous events in the stim channel?

No. You need unique events

If you need to, you can conceptually do something like this using a binary representation. Each event number `n` becomes encoded as `2 ** (n - 1)`. So event type 3 is encoded as the number 4, while event number 5 is encoded as the number 16. Thus simultaneous 3 and 5 events would be encoded as 20 (their sum). It's not too tough to synthesize this representation, or use bit masking to get back the original values.

If this is too annoying, you can also include multiple stim channels in a FIFF file, they just need different channel names. IIRC usually `STI101` is just a binary combination (as above) of several other channels (which only take on values of zero or one) named something like `STI001`, `STI002` or so.

@kazemakase only a few STY comments from my side.

Thank you, @dengemann. Point taken :)
Btw, what is the maximum line length you prefer?

> Btw, what is the maximum line length you prefer?

strictly pep8 80 chars ;-)

[![Coverage Status](https://coveralls.io/builds/716989/badge)](https://coveralls.io/builds/716989)

Coverage increased (+0.14%) when pulling **1175ae773023fac3589cf1d6c0044f5c6760db6d on kazemakase:fix-edf-annotations** into **9a88a37cf92c92eaad4a57c8bf5d2fa768be8afc on mne-tools:master**.

[![Coverage Status](https://coveralls.io/builds/717131/badge)](https://coveralls.io/builds/717131)

Coverage increased (+0.23%) when pulling **3881f0e8f1d4e4699dba02657ed26449279ebe17 on kazemakase:fix-edf-annotations** into **9a88a37cf92c92eaad4a57c8bf5d2fa768be8afc on mne-tools:master**.

[![Coverage Status](https://coveralls.io/builds/717387/badge)](https://coveralls.io/builds/717387)

Coverage increased (+0.13%) when pulling **3fdac36c0b8189600aa96c329c39f6d5c96f1cd6 on kazemakase:fix-edf-annotations** into **9a88a37cf92c92eaad4a57c8bf5d2fa768be8afc on mne-tools:master**.

@kazemakase tell us when we shall review? can you achieve what you wanted now?

Yes. It's ready now.
I decided not to bother with simultaneous events. Maybe nobody needs that anyway :)

how shall test this code on real data?

given these data for example:

 S001R01.edf
 S001R01.edf.event

the tal_channel is the .event file? sorry for the noob question...

R01 is one of the rest conditions and contains no events.
I have tested on S001R06, but that file may be too big for unit tests?

The events are encoded in the last channel of the edf (tal_channel=-1). 
You can ignore the .event file. It contains redundant information and is not part of the edf+ spec.

> R01 is one of the rest conditions and contains no events.
> I have tested on S001R06, but that file may be too big for unit tests?

no it's just for me to test.

btw why tal? is there a more explicit name? stim_channel? is tal an
edf convention?

> The events are encoded in the last channel of the edf (tal_channel=-1).
> You can ignore the .event file. It contains redundant information and is not part of the edf+ spec.

ok. I will give it a try.

once this is merged do you plan to add a data loader for these data?
update the CSP example?

thanks heaps for helping on this

btw can you add your name + email on top of the files you changed? and
update whats_new.rst that generates:

http://martinos.org/mne/stable/whats_new.html

> btw why tal? is there a more explicit name? stim_channel? is tal an edf convention?

TAL is short for Time-stamped Annotations List. I took that term from the edf+ spec, but there may be a better name. Suggestions? :)

> once this is merged do you plan to add a data loader for these data?

what is a data loader?

> update the CSP example?

Yes, I could not correctly load the events when trying to update the CSP example. That's the main reason for this PR ;)

> btw can you add your name + email on top of the files you changed? and update whats_new.rst that generates.

Sure.

> TAL is short for Time-stamped Annotations List. I took that term from the edf+ spec, but there may be a better name. Suggestions? :)

if it's standard then it's ok. Maybe write in the docstring what tal stands for.

> btw can you add your name + email on top of the files you changed? and update whats_new.rst that generates.

let me know when it's done

I'll try now with a file

works like charm with 

```
raw = mne.fiff.edf.read_raw_edf('S001R06.edf', tal_channel=-1, preload=True)
events = mne.find_events(raw)
epochs = mne.Epochs(raw, event_id=2, events=events, tmin=-0.2, tmax=0.4)
evoked = epochs.average()
evoked.plot()
```

the only thing that does not work in evoked.plot_topomap() 

indeed it would be great to have a default layout for such EEG data

@kazemakase please just do the doc changes and I'll merge.

Wouldn't it be nice to have a small sample .edf for unit tests first?

Oh.. **don't merge yet!**

There is still an unresolved issue with stim channel resampling.. I'll comment in the code.

don't you test the tal argument already?

once we have the downloader we'll have an example that gets
a test when building the doc. I could live with that.

but we need to add soon the downloader :)

ok I'll wait then...

@t3on please review this part

> 1. insert equally spaced values (zeros or repeat previous sample)

+1

[![Coverage Status](https://coveralls.io/builds/720103/badge)](https://coveralls.io/builds/720103)

Coverage increased (+0.13%) when pulling **807162f4d6c9c4faccc1eb1a24f070aaa0309469 on kazemakase:fix-edf-annotations** into **9a88a37cf92c92eaad4a57c8bf5d2fa768be8afc on mne-tools:master**.

just curious, for the text annotation, how will it be stored once it is saved to disk? doesn't the data write in float?

> just curious, for the text annotation, how will it be stored once it is
> saved to disk? doesn't the data write in float?
> 
> yes it will but it's not a problem and easier this way.

> just curious, for the text annotation, how will it be stored once it is saved to disk? doesn't the data write in float?

As far i understand EDF data is stored as int16. Thus, each "sample" contains two ascii characters.

it will be store in float in a fif file if you use raw.save

The annotation channel is replaced with the stim channel, so that should not be a problem.

[![Coverage Status](https://coveralls.io/builds/722954/badge)](https://coveralls.io/builds/722954)

Coverage increased (+0.12%) when pulling **969334b3e5d50a4a0d27daffbe2fdf9112b65010 on kazemakase:fix-edf-annotations** into **9a88a37cf92c92eaad4a57c8bf5d2fa768be8afc on mne-tools:master**.

> Is there any way you can hack a solution with the files we have?

I added annotations to test.edf (Actually I rewrote the file, which changed some of the header information).
Some of the old tests had to be modified because they are not aware of the TAL channel.

> it will be store in float in a fif file if you use raw.save

Ah, yes.. if the TAL channel is not parsed the information is destroyed when stored as float. This does not really matter - the Raw class does not know what to do with that channel anyway.
On the other hand, if the TAL channel is converted to a stim channel first, everything is fine.

is the sampling problem solved?

@t3on are you happy with the changes?

> is the sampling problem solved?

solved, but there is no data file to test it with.

the PR cannot be merged automatically anymore. Can you rebase on master?
certainly a pb with the what's new page update.

[![Coverage Status](https://coveralls.io/builds/725717/badge)](https://coveralls.io/builds/725717)

Coverage increased (+0.2%) when pulling **12bc1fc9fcc3595defc457bed9ee7a6bd924251f on kazemakase:fix-edf-annotations** into **7a700d7c804d99f2c9c9ff0c07d71bd0740cff8e on mne-tools:master**.

@t3on please merge if you're happy

merged by rebase after a cosmit + pep8

https://github.com/mne-tools/mne-python/commit/49dda4dc9b9f71a996a629b364eb034b391c1bcf

good job and thanks for the contribution @kazemakase :)

Alexandre sent me here from the mne_analysis mailing list.

I normally use MNE for analysing MEG data from Elekta and CTF systems.

For an upcoming SEEG project, we have exported a couple of small test files from the acquisition system at the hospital here. **I can read the signals and events from a test EDF+ file**, although I get a lengthy error message regarding overlapping events (there are just 4 discrete events in the short test file, no overlapping events):

raw = mne.io.read_raw_edf('20160913_141540.EDF',preload=True)
Extracting edf Parameters from 20160913_141540.EDF...
Setting channel info structure...
Creating Raw.info structure...
Reading 0 ... 4999  =      0.000 ...     9.998 secs...
EDF+ with overlapping events are not fully supported
EDF+ with overlapping events are not fully supported
EDF+ with overlapping events are not fully supported
<ipython-input-156-6acf28b4f62c>:1: RuntimeWarning: EDF+ with overlapping events are not fully supported
....
<ipython-input-156-6acf28b4f62c>:1: RuntimeWarning: EDF+ with overlapping events are not fully supported
  raw = mne.io.read_raw_edf('20160913_141540.EDF',preload=True)
EDF+ with overlapping events are not fully supported

Ready.
<ipython-input-156-6acf28b4f62c>:1: RuntimeWarning: EDF+ with overlapping events are not fully supported
  raw = mne.io.read_raw_edf('20160913_141540.EDF',preload=True)

**The events ndarray seems fine** (the event codes are funky, but I think this was just the programmer of the system trying something; they are all the same class, from a button key press, and only the indices are important).

However, when I read the same file saved in a **BDF** file format, **there is no error message, signals are fine, but the events array is empty**:

raw = mne.io.read_raw_edf('20160913_141540.BDF',preload=True)
Extracting edf Parameters from 20160913_141540.BDF...
Setting channel info structure...
Interpolating stim channel. Events may jitter.
Creating Raw.info structure...
Reading 0 ... 5499  =      0.000 ...    10.998 secs...
Ready.
<ipython-input-157-72b96af5f50d>:1: RuntimeWarning: Interpolating stim channel. Events may jitter.
  raw = mne.io.read_raw_edf('20160913_141540.BDF',preload=True)

I enclose the test files.
[EDF BDF test files.zip](https://github.com/mne-tools/mne-python/files/562950/EDF.BDF.test.files.zip)

There is no script as such, I am just trying to read the data using mne.io.read_raw_edf()

can someone have a look?

maybe @teonbrooks @jasmainak or @jaeilepp ?

I'm not too familiar with the EDF event system, but it looks like the data has events with very long durations (like 1.84336258312e+13 seconds), thus emitting warnings. Is the data supposed to be structured this way (not returning to 0)? The warnings are there to indicate that MNE does not make a decision on how to handle these overlapping events at the moment.

Ok, I'll check with the programmer on how he implemented this, the events appear as transient in the original files.

However, the error messages on reading the EDF is a smaller problem. 

The bigger problem is that the event array is empty when I try to read the BDF file. Of course, the problem could also be related to the non-standard implementation of the timing of the events.

@jwessberg could you make this an issue and cross-ref this pr to better keep track of it? I will try to take a look at this in the next day or so.


Resolving patch:
diff --git a/mne/stats/cluster_level.py b/mne/stats/cluster_level.py
--- a/mne/stats/cluster_level.py
+++ b/mne/stats/cluster_level.py
@@ -1105,7 +1105,7 @@ def permutation_cluster_1samp_test(
         (n_vertices). Default is None, i.e, a regular lattice connectivity.
         Use square n_vertices matrix for datasets with a large temporal
         extent to save on memory and computation time. Can also be False
-        to assume no connectivity. Can also be False to assume no connectivity.
+        to assume no connectivity.
     %(verbose)s
     n_jobs : int
         Number of permutations to run in parallel (requires joblib package).


Test patch: