Repository: mne-tools/mne-python

Issue:
ENH: make mne python for CTF data preprocessing more friendly

@NataKozh and I have been doing some data preprocessing with CTF data and we found that mne-python needs a few PR upgrades to integrate common preprocessing steps done when maxfilter is not used. 

Namely:
- Muscle artifacts detection `detect_artifact_muscle`
   Detect data segments with muscle artifacts by bandpassing the data between 110-140 Hz, taking the envelope, z-scoreing and low pass filtering to smooth it. Then annotate the segments above a given threshold. This was written by @bloyl and is very similar to how fieldtrip detects muscle artifacts.

- Head movement detection and head position correction, `detect_artifact_movement` & `correct_head_pos`
  Detect segments where the head position is further away from a given threshold distance from the median head position during the recording. It iterates twice to find the median position during the segments with an acceptable head position. @bloyl was our source of inspiration and we are looking forward to his comments ;)
 
Another function that I would like to add:
- `Detect_bad_channels` 
It depends on the threshold used and it is useful to highlight the channels that might be bad, rather than an automatic step that does not require data visualization and inspection. It z-scores the data to find channel outliers based on power and/or does local correlation to find channels that do not capture similar magnetic fields as it's neighbour peers. 

I would like to add these functions and make an example tutorial. Let me know what do you think and if you have any suggestions. @jasmainak 

If you want to have a look at the functions we have them on: https://github.com/AdoNunes/ASD_MNE_pipe/blob/master/MNEprepro.py



Discussion:
I found some old code too
https://github.com/mne-tools/mne-python/blob/master/mne/preprocessing/bads.py#L8

it would be great if you break down this script in smaller functions.

if you find bad segment then return an Annotations with proper description
eg BAD_MUSCLE or BAD_MOVEMENT etc.

do we have some public data we could use to demo / validate this?

>

What about the brainstorm auditory dataset? There are bad channels and maybe the subject moved at least a bit...

I will call the `find_outliers` func from bads.


what about making a separate repository for these issues as we did with `pyprep` in #7049?
At least start by an example so we see what function could be interesting for mne core library 

I think that at least the muscle artifact and movement annotation would be good to have it in MNE. 

I have added a script example and the functions script on gist: https://gist.github.com/AdoNunes/b8edec72916543acdde716b6b23e926f

After computing the head position the head pos time series can be plotted (the horizontal line is the threshold, each line is the distance of one of the three cHPI to the median head position):
![image](https://user-images.githubusercontent.com/17873774/69351813-e3126180-0c49-11ea-8c80-9c0c1bbf139a.png)

In this case, the participant moved slightly only during the brake

After computing muscle artifacts, the z-scores can be plotted:
![image](https://user-images.githubusercontent.com/17873774/69351947-1228d300-0c4a-11ea-987f-33d221cb3b19.png)


and the annotations inspected:
![image](https://user-images.githubusercontent.com/17873774/69351301-156f8f00-0c49-11ea-816b-89059d24b84b.png)

could you open a PR with this code putting the function in an example for
now? So at least we can comment on it.

>


Resolving patch:
diff --git a/examples/preprocessing/plot_artifact_detection.py b/examples/preprocessing/plot_artifact_detection.py
new file mode 100644
--- /dev/null
+++ b/examples/preprocessing/plot_artifact_detection.py
@@ -0,0 +1,81 @@
+"""
+======================================================
+Annotate movement artifacts and reestimate dev_head_t
+======================================================
+
+Periods, where the participant moved considerably, are contaminated by low
+amplitude artifacts. When averaging the magnetic fields, the more spread the
+head position, the bigger the cancellation due to different locations.
+Similarly, the covariance will also be affected by severe head movement,
+and source estimation will suffer low/smeared coregistration accuracy.
+
+This example uses the continuous head position indicators (cHPI) times series
+to annotate periods of head movement, then the device to head transformation
+matrix is estimated from the artifact-free segments. The new head position will
+be more representative of the actual head position during the recording.
+
+"""
+# Authors: Adonay Nunes <adonay.s.nunes@gmail.com>
+#          Luke Bloy <luke.bloy@gmail.com>
+# License: BSD (3-clause)
+
+import os.path as op
+
+import mne
+from mne.datasets.brainstorm import bst_auditory
+from mne.io import read_raw_ctf
+from mne.preprocessing import annotate_movement, compute_average_dev_head_t
+
+# Load data
+data_path = bst_auditory.data_path()
+data_path_MEG = op.join(data_path, 'MEG')
+subject = 'bst_auditory'
+subjects_dir = op.join(data_path, 'subjects')
+trans_fname = op.join(data_path, 'MEG', 'bst_auditory',
+                      'bst_auditory-trans.fif')
+raw_fname1 = op.join(data_path_MEG, 'bst_auditory', 'S01_AEF_20131218_01.ds')
+raw_fname2 = op.join(data_path_MEG, 'bst_auditory', 'S01_AEF_20131218_02.ds')
+
+raw = read_raw_ctf(raw_fname1, preload=False)
+
+mne.io.concatenate_raws([raw, read_raw_ctf(raw_fname2, preload=False)])
+raw.crop(350, 410).load_data()
+raw.resample(100, npad="auto")
+
+###############################################################################
+# Plot continuous head position with respect to the mean recording position
+# --------------------------------------------------------------------------
+
+# get cHPI time series and compute average
+pos = mne.chpi.calculate_head_pos_ctf(raw)
+original_head_dev_t = mne.transforms.invert_transform(
+    raw.info['dev_head_t'])
+average_head_dev_t = mne.transforms.invert_transform(
+    compute_average_dev_head_t(raw, pos))
+fig = mne.viz.plot_head_positions(pos)
+for ax, val, val_ori in zip(fig.axes[::2], average_head_dev_t['trans'][:3, 3],
+                            original_head_dev_t['trans'][:3, 3]):
+    ax.axhline(1000 * val, color='r')
+    ax.axhline(1000 * val_ori, color='g')
+
+# The green horizontal lines represent the original head position, whereas the
+# red lines are the new head position averaged over all the time points.
+
+###############################################################################
+# Plot raw data with annotated movement
+# ------------------------------------------------------------------
+
+mean_distance_limit = .0015  # in meters
+annotation_movement, hpi_disp = annotate_movement(
+    raw, pos, mean_distance_limit=mean_distance_limit)
+raw.set_annotations(annotation_movement)
+raw.plot(n_channels=100, duration=20)
+
+##############################################################################
+# After checking the annotated movement artifacts, calculate the new transform
+# and plot it:
+new_dev_head_t = compute_average_dev_head_t(raw, pos)
+raw.info['dev_head_t'] = new_dev_head_t
+mne.viz.plot_alignment(
+    raw.info, show_axes=True, subject=subject, trans=trans_fname,
+    subjects_dir=subjects_dir)
diff --git a/mne/preprocessing/__init__.py b/mne/preprocessing/__init__.py
--- a/mne/preprocessing/__init__.py
+++ b/mne/preprocessing/__init__.py
@@ -23,3 +23,4 @@
 from .xdawn import Xdawn
 from ._csd import compute_current_source_density
 from . import nirs
+from .artifact_detection import (annotate_movement, compute_average_dev_head_t)
diff --git a/mne/preprocessing/artifact_detection.py b/mne/preprocessing/artifact_detection.py
new file mode 100644
--- /dev/null
+++ b/mne/preprocessing/artifact_detection.py
@@ -0,0 +1,212 @@
+# Authors: Adonay Nunes <adonay.s.nunes@gmail.com>
+#          Luke Bloy <luke.bloy@gmail.com>
+# License: BSD (3-clause)
+
+
+import numpy as np
+
+from ..annotations import (Annotations, _annotations_starts_stops)
+from ..chpi import _apply_quat
+from ..transforms import (quat_to_rot, _average_quats, _angle_between_quats)
+from .. import Transform
+from ..utils import (_mask_to_onsets_offsets, logger)
+
+
+def annotate_movement(raw, pos, rotation_velocity_limit=None,
+                      translation_velocity_limit=None,
+                      mean_distance_limit=None):
+    """Detect segments with movement.
+
+    Detects segments periods further from rotation_velocity_limit,
+    translation_velocity_limit and mean_distance_limit. It returns an
+    annotation with the bad segments.
+
+    Parameters
+    ----------
+    raw : instance of Raw
+        Data to compute head position.
+    pos : array, shape (N, 10)
+        The position and quaternion parameters from cHPI fitting. Obtained
+        with `mne.chpi` functions.
+    rotation_velocity_limit : float
+        Head rotation velocity limit in radians per second.
+    translation_velocity_limit : float
+        Head translation velocity limit in radians per second.
+    mean_distance_limit : float
+        Head position limit from mean recording in meters.
+
+    Returns
+    -------
+    annot : mne.Annotations
+        Periods with head motion.
+    hpi_disp : array
+        Head position over time with respect to the mean head pos.
+    """
+    sfreq = raw.info['sfreq']
+    hp_ts = pos[:, 0].copy()
+    hp_ts -= raw.first_samp / sfreq
+    dt = np.diff(hp_ts)
+    seg_good = np.append(dt, 1. / sfreq)
+    hp_ts = np.concatenate([hp_ts, [hp_ts[-1] + 1. / sfreq]])
+
+    annot = Annotations([], [], [], orig_time=None)  # rel to data start
+
+    # Mark down times that are bad according to annotations
+    onsets, ends = _annotations_starts_stops(raw, 'bad')
+    for onset, end in zip(onsets, ends):
+        seg_good[onset:end] = 0
+
+    # Annotate based on rotational velocity
+    t_tot = raw.times[-1]
+    if rotation_velocity_limit is not None:
+        assert rotation_velocity_limit > 0
+        # Rotational velocity (radians / sec)
+        r = _angle_between_quats(pos[:-1, 1:4], pos[1:, 1:4])
+        r /= dt
+        bad_mask = (r >= np.deg2rad(rotation_velocity_limit))
+        onsets, offsets = _mask_to_onsets_offsets(bad_mask)
+        onsets, offsets = hp_ts[onsets], hp_ts[offsets]
+        bad_pct = 100 * (offsets - onsets).sum() / t_tot
+        logger.info(u'Omitting %5.1f%% (%3d segments): '
+                    u'ω >= %5.1f°/s (max: %0.1f°/s)'
+                    % (bad_pct, len(onsets), rotation_velocity_limit,
+                       np.rad2deg(r.max())))
+        annot += _annotations_from_mask(hp_ts, bad_mask, 'BAD_mov_rotat_vel')
+
+    # Annotate based on translational velocity limit
+    if translation_velocity_limit is not None:
+        assert translation_velocity_limit > 0
+        v = np.linalg.norm(np.diff(pos[:, 4:7], axis=0), axis=-1)
+        v /= dt
+        bad_mask = (v >= translation_velocity_limit)
+        onsets, offsets = _mask_to_onsets_offsets(bad_mask)
+        onsets, offsets = hp_ts[onsets], hp_ts[offsets]
+        bad_pct = 100 * (offsets - onsets).sum() / t_tot
+        logger.info(u'Omitting %5.1f%% (%3d segments): '
+                    u'v >= %5.4fm/s (max: %5.4fm/s)'
+                    % (bad_pct, len(onsets), translation_velocity_limit,
+                       v.max()))
+        annot += _annotations_from_mask(hp_ts, bad_mask, 'BAD_mov_trans_vel')
+
+    # Annotate based on displacement from mean head position
+    disp = []
+    if mean_distance_limit is not None:
+        assert mean_distance_limit > 0
+        # Get static head pos from file, used to convert quat to cartesian
+        chpi_pos = sorted([d for d in raw.info['hpi_results'][-1]
+                          ['dig_points']], key=lambda x: x['ident'])
+        chpi_pos = np.array([d['r'] for d in chpi_pos])
+        # CTF: chpi_pos[0]-> LPA, chpi_pos[1]-> NASION, chpi_pos[2]-> RPA
+        # Get head pos changes during recording
+        chpi_pos_mov = np.array([_apply_quat(quat, chpi_pos, move=True)
+                                for quat in pos[:, 1:7]])
+
+        # get average position
+        chpi_pos_avg = np.average(chpi_pos_mov, axis=0, weights=seg_good)
+
+        # get movement displacement from mean pos
+        hpi_disp = chpi_pos_mov - np.tile(chpi_pos_avg, (len(seg_good), 1, 1))
+        # get positions above threshold distance
+        disp = np.sqrt((hpi_disp ** 2).sum(axis=2))
+        bad_mask = np.any(disp > mean_distance_limit, axis=1)
+        onsets, offsets = _mask_to_onsets_offsets(bad_mask)
+        onsets, offsets = hp_ts[onsets], hp_ts[offsets]
+        bad_pct = 100 * (offsets - onsets).sum() / t_tot
+        logger.info(u'Omitting %5.1f%% (%3d segments): '
+                    u'disp >= %5.4fm (max: %5.4fm)'
+                    % (bad_pct, len(onsets), mean_distance_limit, disp.max()))
+        annot += _annotations_from_mask(hp_ts, bad_mask, 'BAD_mov_dist')
+    return annot, disp
+
+
+def compute_average_dev_head_t(raw, pos):
+    """Get new device to head transform based on good segments.
+
+    Segments starting with "BAD" annotations are not included for calculating
+    the mean head position.
+
+    Parameters
+    ----------
+    raw : instance of Raw
+        Data to compute head position.
+    pos : array, shape (N, 10)
+        The position and quaternion parameters from cHPI fitting.
+
+    Returns
+    -------
+    dev_head_t : array
+        New trans matrix using the averaged good head positions.
+    """
+    sfreq = raw.info['sfreq']
+    seg_good = np.ones(len(raw.times))
+    trans_pos = np.zeros(3)
+    hp = pos.copy()
+    hp_ts = hp[:, 0] - raw._first_time
+
+    # Check rounding issues at 0 time
+    if hp_ts[0] < 0:
+        hp_ts[0] = 0
+        assert hp_ts[1] > 1. / sfreq
+
+    # Mask out segments if beyond scan time
+    mask = hp_ts <= raw.times[-1]
+    if not mask.all():
+        logger.info(
+            '          Removing %d samples > raw.times[-1] (%s)'
+            % (np.sum(~mask), raw.times[-1]))
+        hp = hp[mask]
+    del mask, hp_ts
+
+    # Get time indices
+    ts = np.concatenate((hp[:, 0], [(raw.last_samp + 1) / sfreq]))
+    assert (np.diff(ts) > 0).all()
+    ts -= raw.first_samp / sfreq
+    idx = raw.time_as_index(ts, use_rounding=True)
+    del ts
+    if idx[0] == -1:  # annoying rounding errors
+        idx[0] = 0
+        assert idx[1] > 0
+    assert (idx >= 0).all()
+    assert idx[-1] == len(seg_good)
+    assert (np.diff(idx) > 0).all()
+
+    # Mark times bad that are bad according to annotations
+    onsets, ends = _annotations_starts_stops(raw, 'bad')
+    for onset, end in zip(onsets, ends):
+        seg_good[onset:end] = 0
+    dt = np.diff(np.cumsum(np.concatenate([[0], seg_good]))[idx])
+    assert (dt >= 0).all()
+    dt = dt / sfreq
+    del seg_good, idx
+
+    # Get weighted head pos trans and rot
+    trans_pos += np.dot(dt, hp[:, 4:7])
+
+    rot_qs = hp[:, 1:4]
+    best_q = _average_quats(rot_qs, weights=dt)
+
+    trans = np.eye(4)
+    trans[:3, :3] = quat_to_rot(best_q)
+    trans[:3, 3] = trans_pos / dt.sum()
+    assert np.linalg.norm(trans[:3, 3]) < 1  # less than 1 meter is sane
+    dev_head_t = Transform('meg', 'head', trans)
+    return dev_head_t
+
+
+def _annotations_from_mask(times, art_mask, art_name):
+    """Construct annotations from boolean mask of the data."""
+    from scipy.ndimage.measurements import label
+    comps, num_comps = label(art_mask)
+    onsets, durations, desc = [], [], []
+    n_times = len(times)
+    for l in range(1, num_comps + 1):
+        l_idx = np.nonzero(comps == l)[0]
+        onsets.append(times[l_idx[0]])
+        # duration is to the time after the last labeled time
+        # or to the end of the times.
+        if 1 + l_idx[-1] < n_times:
+            durations.append(times[1 + l_idx[-1]] - times[l_idx[0]])
+        else:
+            durations.append(times[l_idx[-1]] - times[l_idx[0]])
+        desc.append(art_name)
+    return Annotations(onsets, durations, desc)


Test patch:
diff --git a/mne/preprocessing/tests/test_artifact_detection.py b/mne/preprocessing/tests/test_artifact_detection.py
new file mode 100755
--- /dev/null
+++ b/mne/preprocessing/tests/test_artifact_detection.py
@@ -0,0 +1,49 @@
+# Author: Adonay Nunes <adonay.s.nunes@gmail.com>
+#
+# License: BSD (3-clause)
+
+
+import os.path as op
+import numpy as np
+from numpy.testing import assert_allclose
+from mne.chpi import read_head_pos
+from mne.datasets import testing
+from mne.io import read_raw_fif
+from mne.preprocessing import (annotate_movement, compute_average_dev_head_t)
+
+data_path = testing.data_path(download=False)
+sss_path = op.join(data_path, 'SSS')
+pre = op.join(sss_path, 'test_move_anon_')
+raw_fname = pre + 'raw.fif'
+pos_fname = op.join(data_path, 'SSS', 'test_move_anon_raw.pos')
+
+
+@testing.requires_testing_data
+def test_movement_annotation_head_correction():
+    """Test correct detection movement artifact and dev_head_t."""
+    raw = read_raw_fif(raw_fname, allow_maxshield='yes').load_data()
+    pos = read_head_pos(pos_fname)
+
+    # Check 5 rotation segments are detected
+    annot_rot, [] = annotate_movement(raw, pos, rotation_velocity_limit=5)
+    assert(annot_rot.duration.size == 5)
+
+    # Check 2 transalation vel. segments are detected
+    annot_tra, [] = annotate_movement(raw, pos, translation_velocity_limit=.05)
+    assert(annot_tra.duration.size == 2)
+
+    # Check 1 movement distance segment is detected
+    annot_dis, disp = annotate_movement(raw, pos, mean_distance_limit=.02)
+    assert(annot_dis.duration.size == 1)
+
+    # Check correct trans mat
+    raw.set_annotations(annot_rot + annot_tra + annot_dis)
+    dev_head_t = compute_average_dev_head_t(raw, pos)
+
+    dev_head_t_ori = np.array([
+                              [0.9957292, -0.08688804, 0.03120615, 0.00698271],
+                              [0.09020767, 0.9875856, -0.12859731, -0.0159098],
+                              [-0.01964518, 0.1308631, 0.99120578, 0.07258289],
+                              [0., 0., 0., 1.]])
+
+    assert_allclose(dev_head_t_ori, dev_head_t['trans'], rtol=1e-5, atol=0)
diff --git a/mne/tests/test_import_nesting.py b/mne/tests/test_import_nesting.py
--- a/mne/tests/test_import_nesting.py
+++ b/mne/tests/test_import_nesting.py
@@ -22,7 +22,7 @@
                        and sys.modules[x] is not None)
 bad = scipy_submodules - ok_scipy_submodules
 if len(bad) > 0:
-    out.append('scipy submodules: %s' % list(bad))
+    out |= {'scipy submodules: %s' % list(bad)}
 
 # check sklearn and others
 _sklearn = _pandas = _mayavi = _matplotlib = False