Repository: mne-tools/mne-python

Issue:
implement robust averaging the SPM way
cf mailing list



Discussion:
Are you convinced the SPM way would be useful? I have trouble finding in which kind of data distribution it would be sig. better than a mean of 15-85 percentile.

no I am not. I agree we should test and compare first

As by the no free lunch theorem, wouldn't it make sense to offer multiple options?

I think @jona-sassenhagen is offering his help :)

On 23 March 2016 at 15:42, jona-sassenhagen notifications@github.com
wrote:

> As by the no free lunch theorem, wouldn't it make sense to offer multiple
> options?
> 
> —
> You are receiving this because you commented.
> Reply to this email directly or view it on GitHub
> https://github.com/mne-tools/mne-python/issues/3044#issuecomment-200514532

Nope - the next thing I'll be doing is implement the regularization for linear_regression_raw you suggested to me months ago. 

OK don't hesitate to ping me, this is close to what we're planning on doing
for the gsoc.

On Wednesday, 23 March 2016, jona-sassenhagen notifications@github.com
wrote:

> Nope - the next thing I'll be doing is implement the regularization for
> linear_regression_raw you suggested to me months ago.
> 
> On Wed, Mar 23, 2016 at 1:17 PM -0700, "Jean-Rémi KING" <
> notifications@github.com
> <javascript:_e(%7B%7D,'cvml','notifications@github.com');>> wrote:
> 
> I think @jona-sassenhagen is offering his help :)
> 
> On 23 March 2016 at 15:42, jona-sassenhagen <notifications@github.com
> <javascript:_e(%7B%7D,'cvml','notifications@github.com');>>
> 
> wrote:
> 
> > As by the no free lunch theorem, wouldn't it make sense to offer multiple
> > 
> > options?
> > 
> > —
> > 
> > You are receiving this because you commented.
> > 
> > Reply to this email directly or view it on GitHub
> > 
> > <
> > https://github.com/mne-tools/mne-python/issues/3044#issuecomment-200514532
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly or view it on GitHub
> 
> —
> You are receiving this because you commented.
> Reply to this email directly or view it on GitHub
> https://github.com/mne-tools/mne-python/issues/3044#issuecomment-200527469

I will be happy to write a comparison over the weekend if that can be a help. Then we can discuss what and how to implement afterwards.

I think added a param to the average call could make sense, with options for percentile in non-SPM robust averaging and weight option for SPM robust average  

+1, I would pass a 'median' option too, which is the fastest. Thanks @MadsJensen 

+1 for the benchmark. It would be great if you can show a comparison on our
testing datasets (sample, somato and spm faces)

I have some progress but I am still working on the SPM robust averaging function. So far there is a mean, median, and robust averaging (kingjr's function) implemented in the analysis. 

I would like to hear if anybody has any thoughts on the best way to quantify the difference between the different methods, I feel that just inspecting plots does not suffice.

[Link to nbviewer](https://nbviewer.jupyter.org/github/MadsJensen/mne_compare_averages/blob/master/Compare_different_approaches_to_averaging.ipynb)
[Link to notebook](https://github.com/MadsJensen/mne_compare_averages/blob/master/Compare_different_approaches_to_averaging.ipynb)

thanks heaps

did I miss but did you also include the results obtained with SPM?

to facilitate comparisons it would better to display side by side results
obtained
with the different methods. Can you do it?

thanks again

To compare the results, I would make up synthetic data with an imbalanced mixture of data and noise with different SNR), then plot individual (1D e.g. 1 time point, 1 channels) samples together with the estimated mean. e.g.

```
o o o oo oo oo o       o o         o
            |
```

@agramfort  Sure, i'll do that. No there is no SPM results yet, when I got the a python version of the SPM robust averaging working I will add SPM results. Or did you just think about comparing with the SPM version to the robust, median, mean already available?

@kingjr  thanks for the suggest I will do something like that.  

> @agramfort https://github.com/agramfort Sure, i'll do that. No there is
> no SPM results yet, when I got the a python version of the SPM robust
> averaging working I will add SPM results.
> 
> I would just call the matlab code if it's for checking. If there is a good
> reason to add it we'll port it. That's my motivation behind these tests.
> 
> Or did you just think about comparing with the SPM version to the robust,
> median, mean already available?

what you call robust is more a trimmed mean. You're not doing the same as
what SPM does when it says "robust“ averaging.

Can we have something like

```
epochs.average(method=fun)
```

where `fun` can be e.g. `np.mean, np.median`, or something more complex we provide (e.g. `mne.stats._robust_mean`? Defaulting to `np.mean`.

sure but why would we want to do this ? I am just provoking but I'd like to
see evidence on data to justify complexifying the API.

I think _if_ we decide on implementing more ways of combining evokeds/TFRs (e.g. robust averaging), API wise, this would be very clean and allow users to easily plug in their own.

I'm actually arguing that it's simple enough _not_ to need to add to the
API :)

> I'm actually arguing that it's simple enough _not_ to need to add to the
> API :)

I have to admit I agree but it's better if we allow to do it without having
to hack
private attributes etc.

I am failing to see any remarkable differences between the averaging approaches  in @MadsJensen notebook, is that a good/bad/moot point with respect to plain old averaging? Is it the case that sample datasets have little variability to make the point? is that what @kingjr implying? 

@kambysese you need to test it on artefacted data (with blinks etc) where the noise distribution is of a different order of magnitude than the signal.

Any interest in seeing outcomes on infant auditory data (if you believe in that sort of thing:)? 

Yes. (And yes)

On Tuesday, 12 April 2016, Kambiz notifications@github.com wrote:

> Any interest in seeing outcomes on infant auditory data (if you believe in
> that sort of thing:)?
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly or view it on GitHub
> https://github.com/mne-tools/mne-python/issues/3044#issuecomment-209175605


Resolving patch:
diff --git a/mne/epochs.py b/mne/epochs.py
--- a/mne/epochs.py
+++ b/mne/epochs.py
@@ -813,14 +813,22 @@ def __next__(self, *args, **kwargs):
         """Provide a wrapper for Py3k."""
         return self.next(*args, **kwargs)
 
-    def average(self, picks=None):
-        """Compute average of epochs.
+    def average(self, picks=None, method="mean"):
+        """Compute an average over epochs.
 
         Parameters
         ----------
         picks : array-like of int | None
             If None only MEG, EEG, SEEG, ECoG, and fNIRS channels are kept
             otherwise the channels indices in picks are kept.
+        method : str | callable
+            How to combine the data. If "mean"/"median", the mean/median
+            are returned.
+            Otherwise, must be a callable which, when passed an array of shape
+            (n_epochs, n_channels, n_time) returns an array of shape
+            (n_channels, n_time).
+            Note that due to file type limitations, the kind for all
+            these will be "average".
 
         Returns
         -------
@@ -837,8 +845,18 @@ def average(self, picks=None):
         are selected, resulting in an error. This is because ICA channels
         are not considered data channels (they are of misc type) and only data
         channels are selected when picks is None.
+
+        The `method` parameter allows e.g. robust averaging.
+        For example, one could do:
+
+            >>> from scipy.stats import trim_mean  # doctest:+SKIP
+            >>> trim = lambda x: trim_mean(x, 10, axis=0)  # doctest:+SKIP
+            >>> epochs.average(method=trim)  # doctest:+SKIP
+
+        This would compute the trimmed mean.
+
         """
-        return self._compute_mean_or_stderr(picks, 'ave')
+        return self._compute_aggregate(picks=picks, mode=method)
 
     def standard_error(self, picks=None):
         """Compute standard error over epochs.
@@ -854,12 +872,10 @@ def standard_error(self, picks=None):
         evoked : instance of Evoked
             The standard error over epochs.
         """
-        return self._compute_mean_or_stderr(picks, 'stderr')
+        return self._compute_aggregate(picks, "std")
 
-    def _compute_mean_or_stderr(self, picks, mode='ave'):
+    def _compute_aggregate(self, picks, mode='mean'):
         """Compute the mean or std over epochs and return Evoked."""
-        _do_std = True if mode == 'stderr' else False
-
         # if instance contains ICA channels they won't be included unless picks
         # is specified
         if picks is None:
@@ -876,10 +892,29 @@ def _compute_mean_or_stderr(self, picks, mode='ave'):
 
         if self.preload:
             n_events = len(self.events)
-            fun = np.std if _do_std else np.mean
-            data = fun(self._data, axis=0)
+
+            if mode == "mean":
+                def fun(data):
+                    return np.mean(data, axis=0)
+            elif mode == "std":
+                def fun(data):
+                    return np.std(data, axis=0)
+            elif callable(mode):
+                fun = mode
+            else:
+                raise ValueError("mode must be mean, median, std, or callable"
+                                 ", got %s (type %s)." % (mode, type(mode)))
+            data = fun(self._data)
             assert len(self.events) == len(self._data)
+            if data.shape != self._data.shape[1:]:
+                    raise RuntimeError("You passed a function that resulted "
+                                       "in data of shape {}, but it should be "
+                                       "{}.".format(data.shape,
+                                                    self._data.shape[1:]))
         else:
+            if mode not in {"mean", "std"}:
+                raise ValueError("If data are not preloaded, can only compute "
+                                 "mean or standard deviation.")
             data = np.zeros((n_channels, n_times))
             n_events = 0
             for e in self:
@@ -893,18 +928,18 @@ def _compute_mean_or_stderr(self, picks, mode='ave'):
 
             # convert to stderr if requested, could do in one pass but do in
             # two (slower) in case there are large numbers
-            if _do_std:
+            if mode == "std":
                 data_mean = data.copy()
                 data.fill(0.)
                 for e in self:
                     data += (e - data_mean) ** 2
                 data = np.sqrt(data / n_events)
 
-        if not _do_std:
-            kind = 'average'
-        else:
+        if mode == "std":
             kind = 'standard_error'
             data /= np.sqrt(n_events)
+        else:
+            kind = "average"
 
         return self._evoked_from_epoch_data(data, self.info, picks, n_events,
                                             kind, self._name)
@@ -1572,7 +1607,7 @@ def _getitem(self, item, reason='IGNORED', copy=True, drop_event_id=True,
             return epochs
 
     def crop(self, tmin=None, tmax=None):
-        """Crop a time interval from epochs object.
+        """Crop a time interval from the epochs.
 
         Parameters
         ----------
@@ -1590,6 +1625,8 @@ def crop(self, tmin=None, tmax=None):
         -----
         Unlike Python slices, MNE time intervals include both their end points;
         crop(tmin, tmax) returns the interval tmin <= t <= tmax.
+
+        Note that the object is modified in place.
         """
         # XXX this could be made to work on non-preloaded data...
         _check_preload(self, 'Modifying data of epochs')
diff --git a/mne/viz/epochs.py b/mne/viz/epochs.py
--- a/mne/viz/epochs.py
+++ b/mne/viz/epochs.py
@@ -15,7 +15,7 @@
 
 import numpy as np
 
-from ..utils import verbose, get_config, set_config, logger, warn
+from ..utils import verbose, get_config, set_config, logger, warn, _pl
 from ..io.pick import pick_types, channel_type, _get_channel_types
 from ..time_frequency import psd_multitaper
 from .utils import (tight_layout, figure_nobar, _toggle_proj, _toggle_options,
@@ -313,14 +313,15 @@ def _get_picks_and_types(picks, ch_types, group_by, combine):
             n_picks = len(picks_)
             if n_picks < 2:
                 raise ValueError(" ".join(
-                    (name, "has only ", str(n_picks), "sensors.")))
+                    (name, "has only ", str(n_picks),
+                     "sensor{}.".format(_pl(n_picks)))))
         all_ch_types = list()
         for picks_, name in zip(all_picks, names):
             this_ch_type = list(set((ch_types[pick] for pick in picks_)))
             n_types = len(this_ch_type)
             if n_types > 1:  # we can only scale properly with 1 type
                 raise ValueError(
-                    "Roi {} contains {} sensor types!".format(
+                    "ROI {} contains more than one sensor type ({})!".format(
                         name, n_types))
             all_ch_types.append(this_ch_type[0])
             names.append(name)
@@ -354,19 +355,18 @@ def _pick_and_combine(epochs, combine, all_picks, all_ch_types, names):
     if combine == "gfp":
         def combine(data):
             return np.sqrt((data * data).mean(axis=1))
-    elif combine == "mean":
-        def combine(data):
-            return np.mean(data, axis=1)
-    elif combine == "std":
-        def combine(data):
-            return np.std(data, axis=1)
-    elif combine == "median":
+
+    elif combine in {"mean", "median", "std"}:
+        func = getattr(np, combine)
+
         def combine(data):
-            return np.median(data, axis=1)
+            return func(data, axis=1)
+
     elif not callable(combine):
         raise ValueError(
             "``combine`` must be None, a callable or one out of 'mean' "
             "or 'gfp'. Got " + str(type(combine)))
+
     for ch_type, picks_, name in zip(all_ch_types, all_picks, names):
         if len(np.atleast_1d(picks_)) < 2:
             raise ValueError("Cannot combine over only one sensor. "


Test patch: