Repository: mne-tools/mne-python

Issue:
ENH: Adjust simultaneous recordings time bases/starts/stops
@ometett on Gitter had a use case that we also have at UW: MEG (Neuromag) recorded simultaneously with data from another system (EGI/MFF). They both record at 1000 Hz, but clock drift (and start/stop times) need to be corrected to be able to analyze them together. I created a [gist](https://gist.github.com/larsoner/400b8b07cc89ad96fc2c02d0c3fa254b) to fix these issues, and it seems to work:
```
Drift rate computed to be 495.0 μs/sec (total drift over 398.0 sec recording: 197.0 ms)
```
And the STIM channels after `raw.add_channels`'ing match nicely. @ometett feel free to try it to see if it makes sense!

It would be nice to have a public API for this like:
```
mne.preprocessing.realign_recordings(raw, other, t_raw, t_other)
```
It has to operate in-place because `raw` and `other` can both be cropped depending on when their recordings started and stopped, and `other` will be resampled as necessary. This API seems pretty simple, for @ometett's data, where the STIM channels record the same events, the code would become just:
```
raw = mne.io.read_raw_fif('CC_CP004_PN_L_RUN01_tsss.fif')
other = mne.io.read_raw_egi('CC_CP004_PN_L_Run01_20201016_110457.mff')
t_raw = (mne.find_events(raw)[:, 0] - raw.first_samp) / raw.info['sfreq']
t_other = (mne.find_events(other)[:, 0] - other.first_samp) / other.info['sfreq']
# this will crop raw and other, and resample other in place
mne.preprocessing.realign_recordings(raw, other, t_raw, t_other)
```
The code to do this is mostly in the gist, so really we just need to decide if this is a suitable API and if the names are good.


Discussion:
cc @cbrnr and @NeuroLaunch since we have had discussions about this before
Very nice! So the assumption is that you have the same set of events in both time domains? Could this be used for XDF streams that have a common time basis, but in general different sampling frequencies, to create a `Raw` object with all streams? Currently, MNELAB can only load one stream (and associated events) because resampling is not implemented.

How would you handle the case when two signals have vastly different sampling frequencies (e.g. EEG 512Hz and audio 44.1kHz)? Would it make sense to add an option to select a new sampling frequency for both signals (instead of requiring identical sampling frequencies)? #5180 is the corresponding issue.

> So the assumption is that you have the same set of events in both time domains?

Yes and the user can do this however they want depending on their dataset

> Could this be used for XDF streams that have a common time basis, but in general different sampling frequencies, to create a Raw object with all streams?

In principle this approach could be used, but I would say it's out of scope for the first PR to fix the Neuromag+EGI case (and cases like it)

> How would you handle the case when two signals have vastly different sampling frequencies (e.g. EEG 512Hz and audio 44.1kHz)?

I would `raw.resample` one or both, then maybe use this function. But really for audio data I wouldn't really do this, cortical EEG responses won't track that fast so you're better off processing the envelope or otherwise downsampling the audio data to the physiological data. (If you're talking ABR hopefully you're sampling faster than 512 Hz.)

For the first version of this function I'd rather specify that it's designed primarily for data acquired at the same putative sampling rate to deal with start/stop/clock drift, and later we can expand functionality if need be. I think the current API is future compatible  with the two use cases listed above at least.
Sounds good!
@larsoner it works in my code too. Thank you very much Eric for the insight, I knew that the drift is linear but could not really figure out how I could manipulate the data where both recordings share the sampling rate. The names are looking good to me, a public API would really help the medical community!

Resolving patch:
diff --git a/mne/epochs.py b/mne/epochs.py
--- a/mne/epochs.py
+++ b/mne/epochs.py
@@ -408,7 +408,7 @@ def __init__(self, info, data, events, event_id=None, tmin=-0.2, tmax=0.5,
                                 f'got {events_type}')
             if events.ndim != 2 or events.shape[1] != 3:
                 raise ValueError(
-                    'events must be of shape (N, 3), got {events.shape}')
+                    f'events must be of shape (N, 3), got {events.shape}')
             events_max = events.max()
             if events_max > INT32_MAX:
                 raise ValueError(
diff --git a/mne/preprocessing/__init__.py b/mne/preprocessing/__init__.py
--- a/mne/preprocessing/__init__.py
+++ b/mne/preprocessing/__init__.py
@@ -19,6 +19,7 @@
 from .infomax_ import infomax
 from .stim import fix_stim_artifact
 from .maxwell import maxwell_filter, find_bad_channels_maxwell
+from .realign import realign_raw
 from .xdawn import Xdawn
 from ._csd import compute_current_source_density
 from . import nirs
diff --git a/mne/preprocessing/realign.py b/mne/preprocessing/realign.py
new file mode 100644
--- /dev/null
+++ b/mne/preprocessing/realign.py
@@ -0,0 +1,107 @@
+# -*- coding: utf-8 -*-
+# Authors: Eric Larson <larson.eric.d@gmail.com>
+
+# License: BSD (3-clause)
+
+import numpy as np
+
+from ..io import BaseRaw
+from ..utils import _validate_type, warn, logger, verbose
+
+
+@verbose
+def realign_raw(raw, other, t_raw, t_other, verbose=None):
+    """Realign two simultaneous recordings.
+
+    Due to clock drift, recordings at a given same sample rate made by two
+    separate devices simultaneously can become out of sync over time. This
+    function uses event times captured by both acquisition devices to resample
+    ``other`` to match ``raw``.
+
+    Parameters
+    ----------
+    raw : instance of Raw
+        The first raw instance.
+    other : instance of Raw
+        The second raw instance. It will be resampled to match ``raw``.
+    t_raw : array-like, shape (n_events,)
+        The times of shared events in ``raw`` relative to ``raw.times[0]`` (0).
+        Typically these could be events on some TTL channel like
+        ``find_events(raw)[:, 0] - raw.first_event``.
+    t_other : array-like, shape (n_events,)
+        The times of shared events in ``other`` relative to ``other.times[0]``.
+    %(verbose)s
+
+    Notes
+    -----
+    This function operates inplace. It will:
+
+    1. Estimate the zero-order (start offset) and first-order (clock drift)
+       correction.
+    2. Crop the start of ``raw`` or ``other``, depending on which started
+       recording first.
+    3. Resample ``other`` to match ``raw`` based on the clock drift.
+    4. Crop the end of ``raw`` or ``other``, depending on which stopped
+       recording first (and the clock drift rate).
+
+    This function is primarily designed to work on recordings made at the same
+    sample rate, but it can also operate on recordings made at different
+    sample rates to resample and deal with clock drift simultaneously.
+
+    .. versionadded:: 0.22
+    """
+    from scipy import stats
+    _validate_type(raw, BaseRaw, 'raw')
+    _validate_type(other, BaseRaw, 'other')
+    t_raw = np.array(t_raw, float)
+    t_other = np.array(t_other, float)
+    if t_raw.ndim != 1 or t_raw.shape != t_other.shape:
+        raise ValueError('t_raw and t_other must be 1D with the same shape, '
+                         f'got shapes {t_raw.shape} and {t_other.shape}')
+    if len(t_raw) < 20:
+        warn('Fewer than 20 times passed, results may be unreliable')
+
+    # 1. Compute correction factors
+    coef = np.polyfit(t_other, t_raw, deg=1)
+    r, p = stats.pearsonr(t_other, t_raw)
+    msg = f'Linear correlation computed as R={r:0.3f} and p={p:0.2e}'
+    if p > 0.05 or r <= 0:
+        raise ValueError(msg + ', cannot resample safely')
+    if p > 1e-6:
+        warn(msg + ', results may be unreliable')
+    else:
+        logger.info(msg)
+    dr_ms_s = 1000 * abs(1 - coef[0])
+    logger.info(
+        f'Drift rate: {1000 * dr_ms_s:0.1f} μs/sec '
+        f'(total drift over {raw.times[-1]:0.1f} sec recording: '
+        f'{raw.times[-1] * dr_ms_s:0.1f} ms)')
+
+    # 2. Crop start of recordings to match using the zero-order term
+    msg = f'Cropping {coef[1]:0.3f} sec from the start of '
+    if coef[1] > 0:  # need to crop start of raw to match other
+        logger.info(msg + 'raw')
+        raw.crop(coef[1], None)
+        t_raw -= coef[1]
+    else:  # need to crop start of other to match raw
+        logger.info(msg + 'other')
+        other.crop(-coef[1], None)
+        t_other += coef[1]
+
+    # 3. Resample data using the first-order term
+    logger.info('Resampling other')
+    coef = coef[0]
+    sfreq_new = raw.info['sfreq'] * coef
+    other.load_data().resample(sfreq_new, verbose=True)
+    other.info['sfreq'] = raw.info['sfreq']
+    other._update_times()
+
+    # 4. Crop the end of one of the recordings if necessary
+    delta = raw.times[-1] - other.times[-1]
+    msg = f'Cropping {abs(delta):0.3f} sec from the end of '
+    if delta > 0:
+        logger.info(msg + 'raw')
+        raw.crop(0, other.times[-1])
+    elif delta < 0:
+        logger.info(msg + 'other')
+        other.crop(0, raw.times[-1])


Test patch:
diff --git a/mne/preprocessing/tests/test_realign.py b/mne/preprocessing/tests/test_realign.py
new file mode 100644
--- /dev/null
+++ b/mne/preprocessing/tests/test_realign.py
@@ -0,0 +1,116 @@
+# Author: Mark Wronkiewicz <wronk@uw.edu>
+#
+# License: BSD (3-clause)
+
+import numpy as np
+from numpy.testing import assert_allclose
+from scipy.interpolate import interp1d
+import pytest
+
+from mne import create_info, find_events, Epochs
+from mne.io import RawArray
+from mne.preprocessing import realign_raw
+
+
+@pytest.mark.parametrize('ratio_other', (1., 0.999, 1.001))  # drifts
+@pytest.mark.parametrize('start_raw, start_other', [(0, 0), (0, 3), (3, 0)])
+@pytest.mark.parametrize('stop_raw, stop_other', [(0, 0), (0, 3), (3, 0)])
+def test_realign(ratio_other, start_raw, start_other, stop_raw, stop_other):
+    """Test realigning raw."""
+    # construct a true signal
+    sfreq = 100.
+    duration = 50
+    stop_raw = duration - stop_raw
+    stop_other = duration - stop_other
+    signal = np.zeros(int(round((duration + 1) * sfreq)))
+    orig_events = np.round(
+        np.arange(max(start_raw, start_other) + 2,
+                  min(stop_raw, stop_other) - 2) * sfreq).astype(int)
+    signal[orig_events] = 1.
+    n_events = len(orig_events)
+    times = np.arange(len(signal)) / sfreq
+    stim = np.convolve(signal, np.ones(int(round(0.02 * sfreq))))[:len(times)]
+    signal = np.convolve(
+        signal, np.hanning(int(round(0.2 * sfreq))))[:len(times)]
+
+    # construct our sampled versions of these signals (linear interp is fine)
+    sfreq_raw = sfreq
+    sfreq_other = ratio_other * sfreq
+    raw_times = np.arange(start_raw, stop_raw, 1. / sfreq_raw)
+    other_times = np.arange(start_other, stop_other, 1. / sfreq_other)
+    assert raw_times[0] >= times[0]
+    assert raw_times[-1] <= times[-1]
+    assert other_times[0] >= times[0]
+    assert other_times[-1] <= times[-1]
+    data_raw = np.array(
+        [interp1d(times, d, kind)(raw_times)
+         for d, kind in ((signal, 'linear'), (stim, 'nearest'))])
+    data_other = np.array(
+        [interp1d(times, d, kind)(other_times)
+         for d, kind in ((signal, 'linear'), (stim, 'nearest'))])
+    info_raw = create_info(
+        ['raw_data', 'raw_stim'], sfreq, ['eeg', 'stim'])
+    info_other = create_info(
+        ['other_data', 'other_stim'], sfreq, ['eeg', 'stim'])
+    raw = RawArray(data_raw, info_raw, first_samp=111)
+    other = RawArray(data_other, info_other, first_samp=222)
+
+    # naive processing
+    evoked_raw, events_raw, _, events_other = _assert_similarity(
+        raw, other, n_events)
+    if start_raw == start_other:  # can just naively crop
+        a, b = data_raw[0], data_other[0]
+        n = min(len(a), len(b))
+        corr = np.corrcoef(a[:n], b[:n])[0, 1]
+        min_, max_ = (0.99999, 1.) if sfreq_raw == sfreq_other else (0.8, 0.9)
+        assert min_ <= corr <= max_
+
+    # realign
+    t_raw = (events_raw[:, 0] - raw.first_samp) / other.info['sfreq']
+    t_other = (events_other[:, 0] - other.first_samp) / other.info['sfreq']
+    assert duration - 10 <= len(events_raw) < duration
+    raw_orig, other_orig = raw.copy(), other.copy()
+    realign_raw(raw, other, t_raw, t_other)
+
+    # old events should still work for raw and produce the same result
+    evoked_raw_2, _, _, _ = _assert_similarity(
+        raw, other, n_events, events_raw=events_raw)
+    assert_allclose(evoked_raw.data, evoked_raw_2.data)
+    assert_allclose(raw.times, other.times)
+    # raw data now aligned
+    corr = np.corrcoef(raw.get_data([0])[0], other.get_data([0])[0])[0, 1]
+    assert 0.99 < corr <= 1.
+
+    # Degenerate conditions -- only test in one run
+    test_degenerate = (start_raw == start_other and
+                       stop_raw == stop_other and
+                       ratio_other == 1)
+    if not test_degenerate:
+        return
+    # these alignments will not be correct but it shouldn't matter
+    with pytest.warns(RuntimeWarning, match='^Fewer.*may be unreliable.*'):
+        realign_raw(raw, other, raw_times[:5], other_times[:5])
+    with pytest.raises(ValueError, match='same shape'):
+        realign_raw(raw_orig, other_orig, raw_times[:5], other_times)
+    rand_times = np.random.RandomState(0).randn(len(other_times))
+    with pytest.raises(ValueError, match='cannot resample safely'):
+        realign_raw(raw_orig, other_orig, rand_times, other_times)
+    with pytest.warns(RuntimeWarning, match='.*computed as R=.*unreliable'):
+        realign_raw(
+            raw_orig, other_orig, raw_times + rand_times * 1000, other_times)
+
+
+def _assert_similarity(raw, other, n_events, events_raw=None):
+    if events_raw is None:
+        events_raw = find_events(raw)
+    events_other = find_events(other)
+    assert len(events_raw) == n_events
+    assert len(events_other) == n_events
+    kwargs = dict(baseline=None, tmin=0, tmax=0.2)
+    evoked_raw = Epochs(raw, events_raw, **kwargs).average()
+    evoked_other = Epochs(other, events_other, **kwargs).average()
+    assert evoked_raw.nave == evoked_other.nave == len(events_raw)
+    assert len(evoked_raw.data) == len(evoked_other.data) == 1  # just EEG
+    corr = np.corrcoef(evoked_raw.data[0], evoked_other.data[0])[0, 1]
+    assert 0.9 <= corr <= 1.
+    return evoked_raw, events_raw, evoked_other, events_other