Repository: mne-tools/mne-python

Issue:
Epoch-ing too slow when #events is large..
So I have several ABR dataset where I have about 8000 trials.  Epoching is way too slow and I suspect that the issue is related to not pre-allocating memory in `_get_data_from_disk` and growing the `_data` array one epoch at a time. When I don't do `preload=True`,  when I eventually have to average, `epochs.average()` is too slow. 

Am I missing something or is there room to speed things up?



Discussion:
The problem is peak-to-peak artifact rejection. 8000 trials is really big,
I think you're hitting the limits of the technology that's currently used.
We hava lots of looping and single-trial operations which don't scale well
if you have many trials. Not sure what we can do. Epochs.average is slow
because it's built in a memory-saving way, accumulating (sum VS std) over
trials and subsequently dividing by the number of trials instead of using
numpy array methods directly. Of course you can build yourself a work-aroud
based on RawArray and EpochsArray, if all fits into memory.

2015-05-01 3:51 GMT+02:00 Hari Bharadwaj notifications@github.com:

> So I have several ABR dataset where I have about 8000 trials. Epoching is
> way too slow and I suspect that the issue is related to not pre-allocating
> memory in _get_data_from_disk and growing the _data array one epoch at a
> time. When I don't do preload=True, when I eventually have to average,
> epochs.average() is too slow.
> 
> Am I missing something or is there room to speed things up?
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/mne-tools/mne-python/issues/2049.

Ah Ok.. the thing is the trials are really short (15 ms)...so memory is not really an issue at all. I'll play with `EpochsArray`.  

That said, don't you think peak-to-peak based rejection can be vectorized?

> That said, don't you think peak-to-peak based rejection can be vectorized?

Not sure this is in reach.

> Ah Ok.. the thing is the trials are really short (15 ms)

Ah Ok ..... btw. what does  "long" mean, can you share benchmarks (time,
system environment).

2015-05-01 4:37 GMT+02:00 Hari Bharadwaj notifications@github.com:

> Ah Ok.. the thing is the trials are really short (15 ms)...so memory is
> not really an issue at all. I'll play with EpochsArray.
> 
> That said, don't you think peak-to-peak based rejection can be vectorized?
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/mne-tools/mne-python/issues/2049#issuecomment-98029457
> .

@haribharadwaj if you could share some minimal bad cases from the `sample` dataset, it would help us to tune into optimization targets. Iterating over 8000 things in Python doesn't have to be slow in principle, so we might be able to pull it off. You say you don't use `preload=True`, is it not an option in your case? If there were a `.preload_data()` you could call after instantiating your object, would it potentially fix your use case?

So I do have a `raw` object with `preload=True`. If I then construct `epochs  = mne.Epochs(raw, preload=False)` and then do `epochs.get_data()` or `epochs.average()`, either takes > 3 hours with Intel Quadcore Xeon(R) and 32 GB RAM running Ubuntu and Anaconda. Similarly if I do `epochs  = mne.Epochs(raw, preload=True)`, then this step itself takes > 3 hours.. 

Not sure how I could get the `sample` data to run into this issue. I think it it is specific to ABR type data where n_events is large. Happy to share a .fif file containing the ABR data though.

Just make a fake event list with 8000 events.

... Doesn't matter if epochs overlap

Ah ok.. will cook up a gist tomorrow! Thanks, @Eric89GXL, @dengemann 

I agree > 3 hours is pretty rough, BTW. Hopefully if you use 8000 events and the same number of time points with the sample data (ABR is usually sampled at a much higher rate so you'll probably have to adjust tmin/tmax appropriately based on `info['sfreq']`) you'll see similar performance degradation.

For a second I wondered if the reallocations were taking a long time with `get_data()`, but `.average()` does no such reallocation so that shouldn't be the culprit. The degradation should end up be linear with the number of epochs at least for `.average()`, so maybe shoot for the number of events that takes 3 minutes instead of 3 hours, and hopefully it's `180/3=60` times fewer than the number of epochs you actually use, so ~130. Then we can look at it deeper. If you feel like trying to investigate, I'd suggest using `line_prof` / `%lprun` with IPython. But I'm always happy to optimize if you don't have time :)

Actually... here goes:

```
import numpy as np

import mne
from mne import io
from mne.datasets import somato

# Load real somatosensory sample data.
data_path = somato.data_path()
raw_fname = data_path + '/MEG/somato/sef_raw_sss.fif'
n_events, tmin, tmax = 8000, -0.002, 0.015

# Setup for reading the raw data
raw = io.Raw(raw_fname, preload=True)
# Pick some single channel and match ABR sampling rate
ch_names = ['MEG 1142', ]
raw.pick_channels(ch_names)
raw.resample(sfreq=16384.)

baseline = (-0.002, 0.001)
event_samps = np.random.randint(raw.first_samp + 100, raw.last_samp - 300,
                                n_events)

# Making many event_ids to pool across (just in case it matters)
events = np.c_[event_samps, np.zeros(event_samps.shape, dtype=np.int16),
               np.random.randint(1, 100, n_events)]


epochs = mne.Epochs(raw, events, np.unique(events[:, 2]).tolist(), tmin, tmax,
                    baseline=baseline, reject=dict(grad=4000e-13),
                    preload=True, verbose='WARNING', proj=False)
```

Replacing the last statement with preload=False and then extracting data during averaging runs into the same issues..

```
epochs = mne.Epochs(raw, events, np.unique(events[:, 2]).tolist(), tmin, tmax,
                    baseline=baseline, reject=dict(grad=4000e-13),
                    preload=Flase, verbose='WARNING', proj=False)
evoked = epochs.average()
```

cool, that should be good to get things started

sounds like a great profiling exercise !

@haribharadwaj let me know if you have time for it, otherwise I might be able to look today or Monday

@Eric89GXL I won't be able to get to it before Tuesday if I did.. 

I tried to make your example more minimal, but in doing so I made something that takes only 4 sec on my machine:

``` Python
def test():
    # Load real somatosensory sample data.
    sfreq = 16384.
    n_events, tmin, tmax = 8000, -0.002, 0.015
    baseline = (-0.002, 0.001)

    data = np.zeros((1, int(sfreq)))
    info = mne.create_info(1, sfreq, 'eeg')
    raw = mne.io.RawArray(data, info)
    event_samps = np.random.randint(raw.first_samp + 100, raw.last_samp - 300,
                                    n_events)

    # Making many event_ids to pool across (just in case it matters)
    events = np.c_[event_samps, np.zeros(event_samps.shape, dtype=np.int32),
                   np.random.randint(1, 100, n_events)]
    epochs = mne.Epochs(raw, events, list(np.unique(events[:, 2])),
                        tmin, tmax, baseline=baseline, reject=dict(eeg=150e-6),
                        preload=False, verbose='WARNING', proj=False)
    epochs.average()


if __name__ == '__main__':
    import time
    t0 = time.time()
    test()
    print(time.time() - t0)
```

I'll gradually move back to the example you provided, but the speed of this one indicates that the problem is probably not in the `Epochs` machinery itself.

Did you try running my example to see if it indeed takes long?

Your example takes a long time, but I noticed it was hung up on the resampling. A few simple modifications of your script (to bypass the brutal resampling) is still fast for me (12 sec for 8000 epochs):

``` Python
import numpy as np

import mne
from mne import io
from mne.datasets import somato


def test():
    # Load real somatosensory sample data.
    data_path = somato.data_path()
    raw_fname = data_path + '/MEG/somato/sef_raw_sss.fif'
    n_events, tmin, tmax = 8000, -0.002, 0.015

    # Setup for reading the raw data
    raw = io.Raw(raw_fname, preload=True)
    # Pick some single channel and match ABR sampling rate
    ch_names = ['MEG 1142', ]
    raw.pick_channels(ch_names)
    raw.info['sfreq'] = 16384.

    baseline = (-0.002, 0.001)
    event_samps = np.random.randint(raw.first_samp + 100, raw.last_samp - 300,
                                    n_events)

    # Making many event_ids to pool across (just in case it matters)
    events = np.c_[event_samps, np.zeros(event_samps.shape, dtype=np.int16),
                   np.random.randint(1, 100, n_events)]


    epochs = mne.Epochs(raw, events, np.unique(events[:, 2]).tolist(),
                        tmin, tmax,
                        baseline=baseline, reject=dict(grad=4000e-13),
                        preload=False, verbose='WARNING', proj=False)
    print(epochs.average().nave)


if __name__ == '__main__':
    import time
    t0 = time.time()
    test()
    print(time.time() - t0)
```

Is it possible that you were hitting a slowdown with the use of `resample` in your code? If that's not it, then changing the latest example I posted so that it does have a slowdown (without adding a raw resample) should allow me to continue debugging.

Ah.. yes... Your example is fast for me too... Is there anything in the EDF case that would make it slower than the FIFF case? We don't have sample EDF data, do we?

(My ABR dataset is all BDF files)

Yeah, can you try to reproduce using a file from `mne/io/edf/tests/data/`?

You can just use a bunch of events occurring at the same time since the file will be too small to actually spread them out. And instead of changing the sample rate, getting the same number of samples by adjusting tmin and tmax should work.

@haribharadwaj I have tried with FIF, EDF, and BDF and always get the same result (quick loads) with 8000 epochs:

```
import numpy as np
from os import path as op

import mne
from mne import io
from mne.datasets import sample

edf_dir = op.join(op.dirname(mne.__file__), 'io', 'edf', 'tests', 'data')


def test(name='edf'):
    if name == 'fif':
        raw_fname = op.join(sample.data_path(), 'MEG', 'sample',
                            'sample_audvis_raw.fif')
        raw = io.Raw(raw_fname, preload=True)
        ch_names = ['MEG 1142']
    elif name == 'edf':
        raw = io.read_raw_edf(op.join(edf_dir, 'test.edf'), preload=True)
        ch_names = ['C2']
    else:
        raw = io.read_raw_edf(op.join(edf_dir, 'test.bdf'), preload=True)
        ch_names = ['Cz']

    ratio = 16384. / raw.info['sfreq']
    n_events = 8000
    tmin, tmax = -0.002 * ratio, 0.015 * ratio
    bmin, bmax = -0.002 * ratio, 0.001 * ratio

    raw.pick_channels(ch_names)
    events = np.zeros((n_events, 3), int)
    events[:, 2] = 1
    events[:, 0] = raw.first_samp + 100
    epochs = mne.Epochs(raw, events, 1, tmin, tmax, baseline=(bmin, bmax),
                        reject=dict(eeg=150e-6), preload=False, verbose=False, proj=False)
    if epochs.average().nave != n_events:
        raise RuntimeError


if __name__ == '__main__':
    import time
    t0 = time.time()
    test()
    print(time.time() - t0)
```

Any idea what's going on at your end?

You might try swapping in one of your files into the test above to see if it works or not.

Hmm.. So with my ABR file things are pretty slow.  Even if I just iterate over the epochs object from above as in:

```
for k, epoch in enumerate(epochs):
    print k

```

Each print takes about a second! Because `epochs.average()` iterates, you get the kind of delays I talk about. I think I am going to give you guys the file to play with :P 

Sure, I think I mentioned before you can record ~1 sec of data and then replicate with that, it will make the transfer less annoying. But the full file is also okay :)

So I pre-processed the data (essentially just filtering) and wrote a fif file:
http://nmr.mgh.harvard.edu/~hari/ABR_EPOCHING_WOES/I14L_3levels_ABR_raw.fif

Try the following on it

```
import mne

# Load pre-processed data from fif file
raw = mne.io.Raw('I14L_3levels_ABR_raw.fif', preload=True, proj=False)
eves = mne.find_events(raw)  # Should read about 30,000 events
cond = [3, 6]  # A condition of interest (2 polarities), about 10,000 events

epochs = mne.Epochs(raw, eves, cond, tmin=-0.001, proj=False,
                    tmax=0.014, baseline=(-0.001, 0.001),
                    reject=dict(eeg=25e-6), verbose='INFO')
abr = epochs.average()

```

Thanks!

Wow, all caps on the folder name! This must be keeping you up at night...

I think I might have found the problem in reconstructing `raw.times` on every access. In making the code more DRY I might have slowed it ... I'll try to fix it tonight or tomorrow morning. @dengemann @agramfort we might want to get this one into 0.9 since it is a pretty brutal slowdown.

> @dengemann @agramfort we might want to get this one into 0.9 since it is a pretty brutal slowdown.

Can we somehow consider this as a bug? ;) 

Ahh! I see.. Thanks for the catch @Eric89GXL 


Resolving patch:
diff --git a/mne/forward/forward.py b/mne/forward/forward.py
--- a/mne/forward/forward.py
+++ b/mne/forward/forward.py
@@ -1226,6 +1226,7 @@ def apply_forward_raw(fwd, stc, raw_template, start=None, stop=None,
     raw.info = _fill_measurement_info(raw.info, fwd, sfreq)
     raw.info['projs'] = []
     raw._projector = None
+    raw._update_times()
     return raw
 
 
diff --git a/mne/io/base.py b/mne/io/base.py
--- a/mne/io/base.py
+++ b/mne/io/base.py
@@ -233,7 +233,7 @@ def __init__(self, info, preload=False,
                                    'not %s' % preload.dtype)
             self._data = preload
             self.preload = True
-            self._last_samps = np.array([self._data.shape[1] - 1])
+            last_samps = [self._data.shape[1] - 1]
             load_from_disk = False
         else:
             if last_samps is None:
@@ -246,7 +246,8 @@ def __init__(self, info, preload=False,
                 raise ValueError('bad preload: %s' % preload)
             else:
                 load_from_disk = True
-            self._last_samps = np.array(last_samps)
+        self._last_samps = np.array(last_samps)
+        self._first_samps = np.array(first_samps)
         self.info = info
         cals = np.empty(info['nchan'])
         for k in range(info['nchan']):
@@ -257,13 +258,13 @@ def __init__(self, info, preload=False,
         self.comp = comp
         self._orig_comp_grade = orig_comp_grade
         self._filenames = list(filenames)
-        self._first_samps = np.array(first_samps)
         self.orig_format = orig_format
         self._projectors = list()
         self._projector = None
         # If we have True or a string, actually do the preloading
         if load_from_disk:
             self._preload_data(preload)
+        self._update_times()
 
     def _read_segment(start, stop, sel, data_buffer, projector, verbose):
         """Read a chunk of raw data
@@ -321,6 +322,12 @@ def _preload_data(self, preload):
         self.preload = True
         self.close()
 
+    def _update_times(self):
+        """Helper to update times"""
+        self._times = np.arange(self.n_times) / float(self.info['sfreq'])
+        # make it immutable
+        self._times.flags.writeable = False
+
     @property
     def first_samp(self):
         return self._first_samps[0]
@@ -868,6 +875,7 @@ def resample(self, sfreq, npad=100, window='boxcar',
         # adjust affected variables
         self._data = np.concatenate(new_data, axis=1)
         self.info['sfreq'] = sfreq
+        self._update_times()
 
     def crop(self, tmin=0.0, tmax=None, copy=True):
         """Crop raw data file.
@@ -922,6 +930,7 @@ def crop(self, tmin=0.0, tmax=None, copy=True):
         if raw.preload:
             # slice and copy to avoid the reference to large array
             raw._data = raw._data[:, smin:smax + 1].copy()
+        raw._update_times()
         return raw
 
     @verbose
@@ -1355,7 +1364,7 @@ def ch_names(self):
     @property
     def times(self):
         """Time points"""
-        return np.arange(self.n_times) / float(self.info['sfreq'])
+        return self._times
 
     @property
     def n_times(self):
@@ -1477,6 +1486,7 @@ def append(self, raws, preload=None):
             self._last_samps = np.r_[self._last_samps, r._last_samps]
             self._rawdirs += r._rawdirs
             self._filenames += r._filenames
+        self._update_times()
 
     def close(self):
         """Clean up the object.
diff --git a/mne/preprocessing/ica.py b/mne/preprocessing/ica.py
--- a/mne/preprocessing/ica.py
+++ b/mne/preprocessing/ica.py
@@ -666,6 +666,7 @@ def _sources_as_raw(self, raw, add_channels, start, stop):
 
         out._projector = None
         self._export_info(out.info, raw, add_channels)
+        out._update_times()
 
         return out
 


Test patch:
diff --git a/mne/realtime/tests/test_mockclient.py b/mne/realtime/tests/test_mockclient.py
--- a/mne/realtime/tests/test_mockclient.py
+++ b/mne/realtime/tests/test_mockclient.py
@@ -86,6 +86,7 @@ def test_find_events():
     raw._data[stim_channel_idx, 520:530] = 6
     raw._data[stim_channel_idx, 530:532] = 5
     raw._data[stim_channel_idx, 540] = 6
+    raw._update_times()
 
     # consecutive=False
     find_events = dict(consecutive=False)
diff --git a/mne/tests/test_event.py b/mne/tests/test_event.py
--- a/mne/tests/test_event.py
+++ b/mne/tests/test_event.py
@@ -150,6 +150,7 @@ def test_find_events():
     # Reset some data for ease of comparison
     raw._first_samps[0] = 0
     raw.info['sfreq'] = 1000
+    raw._update_times()
 
     stim_channel = 'STI 014'
     stim_channel_idx = pick_channels(raw.info['ch_names'],