Repository: mne-tools/mne-python

Issue:
ENH/FIX: allow for reject API  that is more consistent with MNE C
In MNE-C it is possible to specify epoching parameters using a config file like this:

```
average {
outfile __OUTFILE__
eventfile __EVEFILE__
logfile __LOGFILE__
grdReject 200e-12
magReject 4e-12
eegReject -1
eogReject -1
emgReject -1
name "WordOnsets"

category {
name "VerbNP"
event 71 
tmin -0.3
tmax 1.2
}
}
```

MNE-Python new comers might be tempted to construct a reject dict like this:

``` Python

reject = dict(mag=4e-12, grad=200e-12, eog=-1, ecg=-1)

# or

reject = dict(mag=4e-12, grad=200e-12, eog=None, ecg=None)
```

Which can / will lead to unexpected results, e.g. in Python comparisons None is always smaller than any numerical value ...

I would suggest to either allow for the MNE C style API or throw nice + concise errors once values like -1 or None are passed.

cc @agramfort @Eric89GXL 

Thanks to @bmaess to bringing this to my awareness 



Discussion:
I would be fine with :

reject = dict(mag=4e-12, grad=200e-12, eog=None, ecg=None)

-1 is really non pythonic.

any volunteer?

maybe @t3on ? :)

So -1 should elict a warning at least .... None should be ignored. Fine with me.

@agramfort btw I always found it rather annoying that reject keys that are not present in picks lead to errors. Depending on picks reject needs to be updated. I would find it super charming to ignore keys that are not picked. So one could setup the reject dict only once.

> So -1 should elict a warning at least
> 
> +1 an error

> +1 an error

whenever it's unexpected given the physical unit. Not for e.g. misc or stim ....

> @agramfort https://github.com/agramfort btw I always found it rather
> annoying that reject keys that are not present in picks lead to errors.
> Depending on picks reject needs to be updated. I would find it super
> charming to ignore keys that are not picked. So one could setup the reject
> dict only once.

it was to avoid errors ... maybe we can just throw warnings ...

any opinion on this?

> it was to avoid errors ... maybe we can just throw warnings ...
> 
> any opinion on this?

Maybe some logging would also do it.
`"Performing epochs rejection for channels types (%s)" % types_present` 
Warning would also be fine with me but I know it would lead to warnigns all the time ....

I can help out. :) warnings are good but you're right that they would occur everything.

On 07/23/2014 01:48 PM, Teon wrote:

> I can help out. :) warnings are good but you're right that they would 
> occur everything.
> 
> —
> Reply to this email directly or view it on GitHub 
> https://github.com/mne-tools/mne-python/issues/1452#issuecomment-49863450.
> 
> I think there is no warning needed. Once you have selected certain 
> channels via the picks, it does not make sense to me to check the 
> non-selected channels for something or to produce a warning if a 
> threshold for them is provided.

Hermann Sonntag - also working in Leipzig - created this function and I 
added the replacement '-1' into 'None'. However, as the decision now is 
to leave out the undefined rejection thresholds - this function needs to 
remove those 'empty' entries from the tree completely. If the function 
somehow fits into MNE python - please feel free to incorporate it.

any action needed here?

do we allow reject keys for absent channels?

Moving to 0.10, haven't heard responses or found volunteer


Resolving patch:
diff --git a/mne/epochs.py b/mne/epochs.py
--- a/mne/epochs.py
+++ b/mne/epochs.py
@@ -444,49 +444,57 @@ def apply_baseline(self, baseline, verbose=None):
     def _reject_setup(self, reject, flat):
         """Sets self._reject_time and self._channel_type_idx"""
         idx = channel_indices_by_type(self.info)
+        reject = deepcopy(reject) if reject is not None else dict()
+        flat = deepcopy(flat) if flat is not None else dict()
         for rej, kind in zip((reject, flat), ('reject', 'flat')):
-            if not isinstance(rej, (type(None), dict)):
+            if not isinstance(rej, dict):
                 raise TypeError('reject and flat must be dict or None, not %s'
                                 % type(rej))
-            if isinstance(rej, dict):
-                bads = set(rej.keys()) - set(idx.keys())
-                if len(bads) > 0:
-                    raise KeyError('Unknown channel types found in %s: %s'
-                                   % (kind, bads))
+            bads = set(rej.keys()) - set(idx.keys())
+            if len(bads) > 0:
+                raise KeyError('Unknown channel types found in %s: %s'
+                               % (kind, bads))
 
         for key in idx.keys():
-            if (reject is not None and key in reject) \
-                    or (flat is not None and key in flat):
-                if len(idx[key]) == 0:
-                    raise ValueError("No %s channel found. Cannot reject based"
-                                     " on %s." % (key.upper(), key.upper()))
-            # now check to see if our rejection and flat are getting more
-            # restrictive
-            old_reject = self.reject if self.reject is not None else dict()
-            new_reject = reject if reject is not None else dict()
-            old_flat = self.flat if self.flat is not None else dict()
-            new_flat = flat if flat is not None else dict()
-            bad_msg = ('{kind}["{key}"] == {new} {op} {old} (old value), new '
-                       '{kind} values must be at least as stringent as '
-                       'previous ones')
-            for key in set(new_reject.keys()).union(old_reject.keys()):
-                old = old_reject.get(key, np.inf)
-                new = new_reject.get(key, np.inf)
-                if new > old:
-                    raise ValueError(bad_msg.format(kind='reject', key=key,
-                                                    new=new, old=old, op='>'))
-            for key in set(new_flat.keys()).union(old_flat.keys()):
-                old = old_flat.get(key, -np.inf)
-                new = new_flat.get(key, -np.inf)
-                if new < old:
-                    raise ValueError(bad_msg.format(kind='flat', key=key,
-                                                    new=new, old=old, op='<'))
+            if len(idx[key]) == 0 and (key in reject or key in flat):
+                # This is where we could eventually add e.g.
+                # self.allow_missing_reject_keys check to allow users to
+                # provide keys that don't exist in data
+                raise ValueError("No %s channel found. Cannot reject based on "
+                                 "%s." % (key.upper(), key.upper()))
+
+        # check for invalid values
+        for rej, kind in zip((reject, flat), ('Rejection', 'Flat')):
+            for key, val in rej.items():
+                if val is None or val < 0:
+                    raise ValueError('%s value must be a number >= 0, not "%s"'
+                                     % (kind, val))
+
+        # now check to see if our rejection and flat are getting more
+        # restrictive
+        old_reject = self.reject if self.reject is not None else dict()
+        old_flat = self.flat if self.flat is not None else dict()
+        bad_msg = ('{kind}["{key}"] == {new} {op} {old} (old value), new '
+                   '{kind} values must be at least as stringent as '
+                   'previous ones')
+        for key in set(reject.keys()).union(old_reject.keys()):
+            old = old_reject.get(key, np.inf)
+            new = reject.get(key, np.inf)
+            if new > old:
+                raise ValueError(bad_msg.format(kind='reject', key=key,
+                                                new=new, old=old, op='>'))
+        for key in set(flat.keys()).union(old_flat.keys()):
+            old = old_flat.get(key, -np.inf)
+            new = flat.get(key, -np.inf)
+            if new < old:
+                raise ValueError(bad_msg.format(kind='flat', key=key,
+                                                new=new, old=old, op='<'))
 
         # after validation, set parameters
         self._bad_dropped = False
         self._channel_type_idx = idx
-        self.reject = reject
-        self.flat = flat
+        self.reject = reject if len(reject) > 0 else None
+        self.flat = flat if len(flat) > 0 else None
 
         if (self.reject_tmin is None) and (self.reject_tmax is None):
             self._reject_time = None
@@ -501,7 +509,6 @@ def _reject_setup(self, reject, flat):
             else:
                 idxs = np.nonzero(self.times <= self.reject_tmax)[0]
                 reject_imax = idxs[-1]
-
             self._reject_time = slice(reject_imin, reject_imax)
 
     @verbose
@@ -1790,14 +1797,12 @@ def __init__(self, raw, events, event_id, tmin, tmax, baseline=(None, 0),
         proj = proj or raw.proj
 
         # call _BaseEpochs constructor
-        super(Epochs, self).__init__(info, None, events, event_id, tmin, tmax,
-                                     baseline=baseline, raw=raw, picks=picks,
-                                     name=name, reject=reject, flat=flat,
-                                     decim=decim, reject_tmin=reject_tmin,
-                                     reject_tmax=reject_tmax, detrend=detrend,
-                                     add_eeg_ref=add_eeg_ref, proj=proj,
-                                     on_missing=on_missing,
-                                     preload_at_end=preload, verbose=verbose)
+        super(Epochs, self).__init__(
+            info, None, events, event_id, tmin, tmax, baseline=baseline,
+            raw=raw, picks=picks, name=name, reject=reject, flat=flat,
+            decim=decim, reject_tmin=reject_tmin, reject_tmax=reject_tmax,
+            detrend=detrend, add_eeg_ref=add_eeg_ref, proj=proj,
+            on_missing=on_missing, preload_at_end=preload, verbose=verbose)
 
     @verbose
     def _get_epoch_from_raw(self, idx, verbose=None):


Test patch:
diff --git a/mne/tests/test_epochs.py b/mne/tests/test_epochs.py
--- a/mne/tests/test_epochs.py
+++ b/mne/tests/test_epochs.py
@@ -76,6 +76,11 @@ def test_reject():
                   picks=picks, preload=False, reject='foo')
     assert_raises(ValueError, Epochs, raw, events, event_id, tmin, tmax,
                   picks=picks_meg, preload=False, reject=dict(eeg=1.))
+    for val in (None, -1):  # protect against older MNE-C types
+        for kwarg in ('reject', 'flat'):
+            assert_raises(ValueError, Epochs, raw, events, event_id,
+                          tmin, tmax, picks=picks_meg, preload=False,
+                          **{kwarg: dict(grad=val)})
     assert_raises(KeyError, Epochs, raw, events, event_id, tmin, tmax,
                   picks=picks, preload=False, reject=dict(foo=1.))