Repository: mne-tools/mne-python

Issue:
Expected values from find_edf_events: missing annotations for EDF+D, but not EDF+C in 0.18
Using MacOSX and conda environment, I downloaded two versions of MNE. Using mne==0.17.2 and 0.18 current version, you take two sets of EDF files (EDF+D and EDF+C). They can be manually converted using edfbrowser. 

For the EDF+D file it works for 0.17.2, but not 0.18. 

Both versions work when the file is in EDF+C format though. Sample code is below. Test data is available upon request (@larsoner has seen the data though)

```
import mne

# in these two runs, only 0.17.2 matches for both files
# in 0.18, only the first annotations gets the full annotation list

if __name__ == '__main__':
    fpath = "./test_edfc.edf"
    raw = mne.io.read_raw_edf(fpath, preload=True)
    annotations = mne.events_from_annotations(raw)
    print(annotations)

    fpath = "./test_edfd.edf"
    raw = mne.io.read_raw_edf(fpath, preload=True)
    annotations = mne.events_from_annotations(raw)
    print(annotations)
```


Discussion:
I will try looking at: mne/io/edf/edf.py to see how it's correctly finding the events and then look at differences with respect to version 0.18+.
On `gitter` @adam2392 mentioned that they might be able to look soon-ish. But @massich @cbrnr feel free to chime in if you have an idea of what the problem might be
I don't understand what the problem is. Are the annotations incorrect? Or is it the conversion to events?
The annotations are missing, yeah
@adam2392 can you share the two files with me as well?
@cbrnr https://gist.github.com/adam2392/8a9b687594eb440a0f6a9824a6aa6293
Something is wrong with both files. EDFbrowser can't read these files - the error mentions that the local patient identification field does not comply to the EDF+ standard.

SigViewer (which uses libbiosig) can load `test_edfc.edf` and shows 28 events (which correspond to what you expected). However, even though SigViewer can also load `test_edfd.edf`, it finds 2342 events - which is not what you expected.

I'm not sure if this should be fixed given that these files are likely invalid EDF files. However, you said that both files worked with 0.17.2, so I'm going to check if we can revert/adapt the behavior. 
Let's keep this issue open, I'm not sure if we're currently doing the right thing (even though the linked EDF files seem to be non-standard).

I don't understand what this code is doing:
https://github.com/mne-tools/mne-python/blob/8a4cef803db165e2f8a00ad0f304ab60a79bc6f3/mne/io/edf/edf.py#L128-L132

Apparently, `_read_segment_file` is overloaded in that it directly manipulates the input arguments, but in addition also returns the TAL channel data. In master, `tal_data` is a list of length 14, of which only the zeroth element is used in `_read_annotations_edf`. Interestingly, `tal_data[0]` has shape `(171, 158)`, whereas I expected that to be one-dimensional because there is only one TAL channel. This is also why annotations are incorrectly parsed - in v0.17.2, the input argument to `_read_annotations_edf` was an array of shape `(1, 370036)`, but now it is `(171, 58)`. This explains the difference, but I still don't know why this was changed and why the current master apparently works with other files.

@massich it seems like you authored the relevant code portions, do you know what's going on? Am I misunderstanding how this is supposed to work?
can you have a code snippet + a file + an expected output to be able to
debug this?
I assume expected output will come from another open source package

>

The problem is that pyEDFlib, which uses EDFlib, gives an error because at least one field in the header does not conform to the EDF standard. I don't know the exact problem, but it might just be an invalid (e.g. UTF8) character. But we can't compare with that lib because of this error.

Then there is BioSig, which also gives an incorrect number of events for the EDF-D file.

In any case, maybe we shouldn't waste time on this given that two other open source packages either give an error or output an unexpected number of events. Maybe these files are really invalid EDF. On the other hand, MNE 0.17.2 gave the expected number of events for both EDF-C and EDF-D.
+1 for low priority (on my side)

>


Resolving patch:
diff --git a/mne/io/edf/edf.py b/mne/io/edf/edf.py
--- a/mne/io/edf/edf.py
+++ b/mne/io/edf/edf.py
@@ -340,6 +340,9 @@ def _read_segment_file(data, idx, fi, start, stop, raw_extras, chs, filenames):
                               2**17 - 1)
         data[stim_channel_idx, :] = stim
 
+    if len(tal_data) > 1:
+        tal_data = np.concatenate([tal.ravel() for tal in tal_data])
+        tal_data = tal_data[np.newaxis, :]
     return tal_data
 
 


Test patch: