Repository: mne-tools/mne-python

Issue:
equalize_epoch_count uses lots of RAM
`mne.epochs.equalize_epoch_counts` (using "mintime") takes 90GB of RAM for a single subject on some of our data sets, and is really slow. However, the "truncate" method seems undesirable. If `mne.epochs.equalize_epoch_counts` can't be optimized trivially, how about something like this?

```
larger, smaller = ((ep1, ep2) if len(ep1.events) > len(ep2.events) else (ep2, ep1))

from copy import deepcopy
arr = deepcopy(larger.events)

tts = []
for t in np.random.permutation(smaller.events[:,0]):
    idx = (np.abs(arr[:,0]-t)).argmin()
    tts.append(arr[idx,0])
    arr = np.delete(arr, idx, axis=0)

larger.drop_epochs(np.in1d(larger.events[:,0], tts) == False, 
                   reason='EQUALIZED_COUNT')
```



Discussion:
I don't think that the actual choice of the subset of epochs to use should affect the memory. If what you write above takes less memory, then we should indeed just optimize the existing method.

FWIW there is a principled reason behind the mintime. If noise properties vary as a function of time, you're selecting epochs as close in time as possible to minimize any effect of those changes.

I understand that, which is why I don't just use "truncate".

My code snippet should get epoch subsets that are at least somewhat similar, while at least for my test data being much faster. Maybe simply offer it as a 3rd method option ("mintime", "truncate", "approx_mintime")?

It seems like a better solution is to make the existing method more efficient. Have you looked to see why the existing one does not work as efficiently?

I tried ... I think it's just doing a lot of iterations by doing an all-vs-all check.
For my data, `_get_drop_indices` is what takes so long, and it calls `_minimize_time_diff`once per event, which in turn "# Check[s] every possible removal to see if it minimizes". The minimization in turn relies on np.interp. Too much maths + recursion for my feeble lib arts mind.

How long does it take? Are you sure it's the searching step that takes 90 GB? That's a _ton_ of memory for such a simple operation...

Can you save two problematic event lists to an npz and upload it somewhere? I can take a crack at optimizing it.

I think I misunderstood something - the _python process_ takes up a lot of RAM while this function is running, but when I run just the _get_drop_indices function in isolation on a fresh instance, it just takes up a lot of processor power for a few seconds. Seems the RAM usage is independent of the calculation.

Either way, here are two event sets taking surprisingly long on our machines.
https://www.dropbox.com/s/sb0klwra1z6v04u/ep1.npy?dl=0
https://www.dropbox.com/s/l25rqj769ahtqmp/ep2.npy?dl=0

And there does actually seem to be a decent benefit from the more involved calculation; here is "mintime" vs. my method:
![unknown-49](https://cloud.githubusercontent.com/assets/4321826/9039252/792956f0-39fa-11e5-9227-e05f3cc240b3.png)
![unknown-48](https://cloud.githubusercontent.com/assets/4321826/9039249/743dda26-39fa-11e5-9da9-2e9987175914.png)

What do you mean by surprisingly long? A few seconds doesn't seem bad to me, but maybe you're doing hundreds or thousands for some application that makes it painful?

(thanks for the files, I'll try to take a look today or tomorrow)

BTW for future reference you can use `np.savez_compressed` to save multiple arrays to a single file

I'm not sure this is a relevant comment, but if you're encountering this
issue because you want to avoid biased decoding predictions, have you tried
using 'class_weight' instead of data truncation ? It should be better
because you don't drop data.

On Monday, 3 August 2015, Eric Larson notifications@github.com wrote:

> BTW for future reference you can use np.savez_compressed to save multiple
> arrays to a single file
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/mne-tools/mne-python/issues/2379#issuecomment-127374377
> .

Indeed @kingjr - that is, 

```
from mne.decoding.time_gen import GeneralizationAcrossTime as GAT
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
scaler = StandardScaler()
estimator = LogisticRegression(class_weight="auto")
clf = Pipeline([('scaler', scaler), ('estimator', estimator)])
gat = GAT(clf=clf)
```

something like this?

Yes. Most linear classifiers have this option, although they don't solve it
in the same way.

If you have a complex dataset (e.g hierarchical classes) the best is to use
sample_weight on the true class (and adapt the stratfied folding using the
latter too) which can be distinct from the fitting class. This is a bit
tricky to use it with the gat class but I can send you some code if you
need.

On Monday, 3 August 2015, jona-sassenhagen notifications@github.com wrote:

> Indeed @kingjr https://github.com/kingjr - that is,
> 
> from mne.decoding.time_gen import GeneralizationAcrossTime as GAT
> from sklearn.preprocessing import StandardScaler
> from sklearn.linear_model import LogisticRegression
> from sklearn.pipeline import Pipeline
> scaler = StandardScaler()
> estimator = LogisticRegression(class_weight="auto")
> clf = Pipeline([('scaler', scaler), ('estimator', estimator)])
> gat = GAT(clf=clf)
> 
> something like this?
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/mne-tools/mne-python/issues/2379#issuecomment-127383703
> .


Resolving patch:
diff --git a/mne/epochs.py b/mne/epochs.py
--- a/mne/epochs.py
+++ b/mne/epochs.py
@@ -11,9 +11,12 @@
 import copy as cp
 import warnings
 import json
-
+import inspect
 import os.path as op
+from distutils.version import LooseVersion
+
 import numpy as np
+import scipy
 
 from .io.write import (start_file, start_block, end_file, end_block,
                        write_int, write_float_matrix, write_float,
@@ -1946,29 +1949,45 @@ def _get_drop_indices(event_times, method):
     return indices
 
 
+def _fix_fill(fill):
+    """Helper to fix bug on old scipy"""
+    if LooseVersion(scipy.__version__) < LooseVersion('0.12'):
+        fill = fill[:, np.newaxis]
+    return fill
+
+
 def _minimize_time_diff(t_shorter, t_longer):
     """Find a boolean mask to minimize timing differences"""
+    from scipy.interpolate import interp1d
     keep = np.ones((len(t_longer)), dtype=bool)
     scores = np.ones((len(t_longer)))
-    for iter in range(len(t_longer) - len(t_shorter)):
+    x1 = np.arange(len(t_shorter))
+    # The first set of keep masks to test
+    kwargs = dict(copy=False, bounds_error=False)
+    # this is a speed tweak, only exists for certain versions of scipy
+    if 'assume_sorted' in inspect.getargspec(interp1d.__init__).args:
+        kwargs['assume_sorted'] = True
+    shorter_interp = interp1d(x1, t_shorter, fill_value=t_shorter[-1],
+                              **kwargs)
+    for ii in range(len(t_longer) - len(t_shorter)):
         scores.fill(np.inf)
+        # set up the keep masks to test, eliminating any rows that are already
+        # gone
+        keep_mask = ~np.eye(len(t_longer), dtype=bool)[keep]
+        keep_mask[:, ~keep] = False
         # Check every possible removal to see if it minimizes
-        for idx in np.where(keep)[0]:
-            keep[idx] = False
-            scores[idx] = _area_between_times(t_shorter, t_longer[keep])
-            keep[idx] = True
+        x2 = np.arange(len(t_longer) - ii - 1)
+        t_keeps = np.array([t_longer[km] for km in keep_mask])
+        longer_interp = interp1d(x2, t_keeps, axis=1,
+                                 fill_value=_fix_fill(t_keeps[:, -1]),
+                                 **kwargs)
+        d1 = longer_interp(x1) - t_shorter
+        d2 = shorter_interp(x2) - t_keeps
+        scores[keep] = np.abs(d1, d1).sum(axis=1) + np.abs(d2, d2).sum(axis=1)
         keep[np.argmin(scores)] = False
     return keep
 
 
-def _area_between_times(t1, t2):
-    """Quantify the difference between two timing sets"""
-    x1 = list(range(len(t1)))
-    x2 = list(range(len(t2)))
-    xs = np.concatenate((x1, x2))
-    return np.sum(np.abs(np.interp(xs, x1, t1) - np.interp(xs, x2, t2)))
-
-
 @verbose
 def _is_good(e, ch_names, channel_type_idx, reject, flat, full_report=False,
              ignore_chs=[], verbose=None):


Test patch: