Repository: mne-tools/mne-python

Issue:
BUG: Reading EDF files can be very slow
I am trying to read an EDF file, but the .read_data() (edit, i mean` load_data()`) function fails. Same thing happens with this [sample file](http://www.teuniz.net/edf_bdf_testfiles/eeg_recording.zip) from [here](http://www.teuniz.net/edf_bdf_testfiles/)

Looking a bit into the code I find that there is a call to `_read_segment_file` in `io\base.py` which raises an `NotImplementedError`. The Problem seems to be the call itself, it doesn't even get to the called functions inside. If I comment the call out, it works fine.
```
self._read_segment_file(data[:, this_sl], idx, fi,
                                    int(start_file), int(stop_file),
                                    cals, mult)
```


Strangely enough running the sample file from the EDF-API works without problems


I hope I provided all the info needed. Unfortunately I'm not very experienced in debugging functions or classes. Thanks!
```
Platform:      Windows-10-10.0.14393-SP0
Python:        3.5.2 (same in 2.7.x) (v3.5.2:4def2a2901a5, Jun 25 2016, 22:18:55) [MSC v.1900 64 bit (AMD64)]
Executable:    C:\Program Files\WinPython-3.5\python-3.5.2.amd64\pythonw.exe

mne:           0.14.dev0
numpy:         1.11.2 {lapack=C:/Program Files (x86)/IntelSWTools/compilers_and_libraries/windows/mkl/include, blas=C:/Program Files (x86)/IntelSWTools/compilers_and_libraries/windows/mkl/include}
scipy:         0.18.1
matplotlib:    1.5.3

sklearn:       0.18
nibabel:       Not found
nitime:        Not found
mayavi:        Not found
nose:          1.3.7
pandas:        0.19.1
pycuda:        Not found
skcuda:        Not found
```


Discussion:
I'm not sure what you mean by ``read_data``. I can't find that in the code base. The file works fine for me. I read it with ``mne.io.read_raw_edf('ma0844az_1-1+.edf', preload=True)``.

``_read_segment_file`` in ``BaseRaw`` is an abstract method and the call should direct to [this](https://github.com/mne-tools/mne-python/blob/master/mne/io/edf/edf.py#L95) method.
Sorry, I meant `load_data`.
In the end `preload=True` produces the same results for me.
It does not on your system?
I'll have a look later to see where in the `_read_segment_file` it gets stuck.
Okay seems like I'm just not being patient enough.
After considerably amount of time the file loaded.

Strange, because files in other formats (brainvision) load much faster (1-2s).
I'm having SDD I/O of ~150MB/s for 15 seconds although the file is only 25MB large.
> I'm having SDD I/O of ~150MB/s for 15 seconds although the file is only 25MB large.

Yikes. That doesn't seem right. Which file are you testing with?

Perhaps there is some optimization we can do. I'll re-title the issue.
The file linked above (~25MB on disk) takes about 15 seconds to load.

I have another data file (~25MB on disk of sleep recordings ) that is not public and it takes 7.5 minutes to load.
It takes about 7 seconds on my system, which I agree is too long for a 20MB file (it will not scale well). It should be fixable, we loop in ways we could avoid.
That would be great. Any idea why my file takes 7.5 minutes despite the same size?

1. file: 1400s @ 250hz @ 38 channels -> 15 seconds
2. file: 30000s @ 100hz @ 5 channels -> 7.5 minutes
It probably has to do with inefficiencies in how we loop. Different channel/sample configurations will expose those weaknesses differently. Or your other file could require different processing, e.g. resampling if not all channels have the same sample rate, which can be slow.

Resolving patch:
diff --git a/mne/io/edf/edf.py b/mne/io/edf/edf.py
--- a/mne/io/edf/edf.py
+++ b/mne/io/edf/edf.py
@@ -130,59 +130,68 @@ def _read_segment_file(self, data, idx, fi, start, stop, cals, mult):
             for tal_channel in tal_channels:
                 offsets[tal_channel] = 0
 
+        # We could read this one EDF block at a time, which would be this:
+        ch_offsets = np.cumsum(np.concatenate([[0], n_samps]))
         block_start_idx, r_lims, d_lims = _blk_read_lims(start, stop, buf_len)
-        read_size = len(r_lims) * buf_len
+        # But to speed it up, we really need to read multiple blocks at once,
+        # Otherwise we can end up with e.g. 18,181 chunks for a 20 MB file!
+        # Let's do ~10 MB chunks:
+        n_per = max(10 * 1024 * 1024 // (ch_offsets[-1] * data_size), 1)
         with open(self._filenames[fi], 'rb', buffering=0) as fid:
             # extract data
             start_offset = (data_offset +
                             block_start_idx * buf_len * n_chan * data_size)
-            ch_offsets = np.cumsum(np.concatenate([[0], n_samps * data_size]))
-            this_data = np.empty((len(sel), buf_len))
-            for bi in range(len(r_lims)):
-                block_offset = bi * ch_offsets[-1]
-                d_sidx, d_eidx = d_lims[bi]
-                r_sidx, r_eidx = r_lims[bi]
-                n_buf_samp = r_eidx - r_sidx
+            for ai in range(0, len(r_lims), n_per):
+                block_offset = ai * ch_offsets[-1] * data_size
+                n_read = min(len(r_lims) - ai, n_per)
+                fid.seek(start_offset + block_offset, 0)
+                # Read and reshape to (n_chunks_read, ch0_ch1_ch2_ch3...)
+                many_chunk = _read_ch(fid, subtype, ch_offsets[-1] * n_read,
+                                      data_size).reshape(n_read, -1)
                 for ii, ci in enumerate(sel):
-                    n_samp = n_samps[ci]
-                    # bdf data: 24bit data
-                    fid.seek(start_offset + block_offset + ch_offsets[ci], 0)
-                    if n_samp == buf_len:
-                        # use faster version with skips built in
-                        fid.seek(r_sidx * data_size, 1)
-                        ch_data = _read_ch(fid, subtype, n_buf_samp, data_size)
-                    else:
-                        # read in all the data and triage appropriately
-                        ch_data = _read_ch(fid, subtype, n_samp, data_size)
+                    # This now has size (n_chunks_read, n_samp[ci])
+                    ch_data = many_chunk[:, ch_offsets[ci]:ch_offsets[ci + 1]]
+                    r_sidx = r_lims[ai][0]
+                    r_eidx = (buf_len * (n_read - 1) +
+                              r_lims[ai + n_read - 1][1])
+                    d_sidx = d_lims[ai][0]
+                    d_eidx = d_lims[ai + n_read - 1][1]
+                    if n_samps[ci] != buf_len:
                         if tal_channels is not None and ci in tal_channels:
-                            # don't resample tal_channels,
-                            # pad with zeros instead.
-                            n_missing = int(buf_len - n_samp)
-                            ch_data = np.hstack([ch_data, [0] * n_missing])
-                            ch_data = ch_data[r_sidx:r_eidx]
+                            # don't resample tal_channels, zero-pad instead.
+                            if n_samps[ci] < buf_len:
+                                z = np.zeros((len(ch_data),
+                                              buf_len - n_samps[ci]))
+                                ch_data = np.append(ch_data, z, -1)
+                            else:
+                                ch_data = ch_data[:, :buf_len]
                         elif ci == stim_channel:
                             if annot and annotmap or tal_channels is not None:
-                                # don't bother with resampling the stim ch
-                                # because it gets overwritten later on.
-                                ch_data = np.zeros(n_buf_samp)
+                                # don't resample, it gets overwritten later
+                                ch_data = np.zeros((len(ch_data, buf_len)))
                             else:
                                 # Stim channel will be interpolated
-                                oldrange = np.linspace(0, 1, n_samp + 1, True)
-                                newrange = np.linspace(0, 1, buf_len, False)
-                                newrange = newrange[r_sidx:r_eidx]
-                                ch_data = interp1d(
-                                    oldrange, np.append(ch_data, 0),
-                                    kind='zero')(newrange)
+                                old = np.linspace(0, 1, n_samps[ci] + 1, True)
+                                new = np.linspace(0, 1, buf_len, False)
+                                ch_data = np.append(
+                                    ch_data, np.zeros((len(ch_data), 1)), -1)
+                                ch_data = interp1d(old, ch_data,
+                                                   kind='zero', axis=-1)(new)
                         else:
-                            ch_data = resample(ch_data, buf_len, n_samp,
-                                               npad=0)[r_sidx:r_eidx]
-                    this_data[ii, :n_buf_samp] = ch_data
-                data[:, d_sidx:d_eidx] = this_data[:, :n_buf_samp]
+                            # XXX resampling each chunk isn't great,
+                            # it forces edge artifacts to appear at
+                            # each buffer boundary :(
+                            # it can also be very slow...
+                            ch_data = resample(
+                                ch_data, buf_len, n_samps[ci], npad=0, axis=-1)
+                    assert ch_data.shape == (len(ch_data), buf_len)
+                    data[ii, d_sidx:d_eidx] = ch_data.ravel()[r_sidx:r_eidx]
         data *= gains.T[sel]
         data += offsets[sel]
 
         # only try to read the stim channel if it's not None and it's
         # actually one of the requested channels
+        read_size = len(r_lims) * buf_len
         if stim_channel is not None and (sel == stim_channel).sum() > 0:
             stim_channel_idx = np.where(sel == stim_channel)[0]
             if annot and annotmap:


Test patch: