Repository: nilearn/nilearn

Issue:
Censoring (and interpolation) in `signal.clean`
Hi! I just attended the NiLearn office hours to learn more about how censoring works with `signal.clean`. Through inspecting the [source code](https://github.com/nilearn/nilearn/blob/9aa503f5/nilearn/signal.py#L460), it was determined that the volumes flagged with the `sample_masks` argument are censored, and then the uncensored volumes adjacent to the first and last censored volumes in a given censored block are treated as if they were collected one after another in subsequent processing steps (e.g., for the purposes of bandpass filtering). Conceptually, this seems like it may be problematic, but practically none of us in the NiLearn Office Hours knew if it causes problems/substantially changes results.

[Historically](https://github.com/GrattonLab/GrattonLab-General-Repo/blob/master/FCProcess/FCPROCESS_GrattonLab.m), I have seen the following steps performed: censoring, temporal interpolation (ideally power spectral matched), bandpass filtering, and then censoring again of the interpolated volumes prior to functional connectivity estimation. Power spectral matched temporal interpolation has the benefit of having realistic dummy volumes so that volumes adjacent to the censored volumes are not contaminated by unrealistic data during bandpass filtering, which is especially important in high motion subjects, considering linear interpolation over many volumes will not come close to approximating realistic data.

Has there been any side-by-side comparisons of these, and potentially other, censoring methods? If in fact it is better to utilize temporal interpolation, would it be possible to implement this in NiLearn? Also, if possible, it would be great to have more documentation about exactly what is happening with each of the processing steps after fMRIPrep and before functional connectivity estimation in NiLearn. A timeline of sorts would be awesome, especially for `NiftiLabelsMasker` (selfishly).



Discussion:
Citation for the temporal interpolation/censoring scheme: Power, J. D., Mitra, A., Laumann, T. O., Snyder, A. Z., Schlaggar, B. L., & Petersen, S. E. (2014). Methods to detect, characterize, and remove motion artifact in resting state fMRI. Neuroimage, 84, 320-341.
I realized the link for "Historically" isn't publicly available. [Here](https://github.com/NU-ACNLab/mwmh/blob/main/scripts/process/gratton/FCPROCESS_GrattonLab.m) is a publicly available version.
@htwangtw  do you have an opinion on this?
Thanks @butellyn for such a detailed description of the issue!

I finally got time to have a look. When I added the sample mask I didn't think this through. 
The original implementation of `sample_mask` (before 0.8.0) was an applied at the start of creating a masker object, and all I did was move this step to `signals.clean`. Either way the implementation mask out the volumes before any signal processing. Butterworth filtering should be applied to continuous timeseries as you described. This should definitely be fixed. The current implementation is only applicable for discrete cosine transformation, as far as my knowledge goes.

I am going to have a look into this and get back to you. I will start with a small tutorial to compare the two implementations


Awesome! Thank you so much for looking into this @htwangtw. I've been corresponding with Anish Mitra, who implemented the power spectral interpolation method I've used previously. He said, in his experience, it is overkill unless you are specifically interested in doing an analysis of spectral content; a cubic spline usually works just fine. If you did want to implement the power spectral interpolation, though, the method is called a lomb-scargle periodogram, and there is an implementation in [scipy](https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.lombscargle.html).
Nilearn only implemented a butterworth filter, and for DCT it's just the cosine regressors from fMRIPrep output. I am not implementing any extra filtering methods here. 
Ah, so the power spectral interpolation method isn't for filtering, it is for interpolating over censored volumes. What method of interpolation are you considering using?
I need to have a look to see if there's anything similar for interpolation in the existing code of nilearn (I am only worked with refactoring `signals.clean` so don't really know the rest). I am happy to fix the bug and will see if this needs to be break down into multiple PRs
Thank you!
Thx for the dsicussion ! Nilearn-type solution is to rely on scipy interpolators. 
After looking into the code, two things should be fixed, for both cosine filter and butterworth:

1. if use cosine filter option, the consine terms should be created with the full time series prior to removing volumes
2. butterworth + scrubbing should be implemented as described in this issue

I would do two PRs: 
1. Refactor the code to move volume masking right before confounds removal. This will fix the cosine filter problem and prepare for fixing butterworth filter for scrubbing. Here I would also check the supplied confounds don't contain cosine regressors.
2. Fix butterworth + scrubbing option

Resolving patch:
diff --git a/nilearn/_utils/glm.py b/nilearn/_utils/glm.py
--- a/nilearn/_utils/glm.py
+++ b/nilearn/_utils/glm.py
@@ -336,6 +336,7 @@ def positive_reciprocal(X):
 
 
 def _check_run_sample_masks(n_runs, sample_masks):
+    """Number of sample_mask matches number of runs."""
     if not isinstance(sample_masks, (list, tuple, np.ndarray)):
         raise TypeError(
             f"sample_mask has an unhandled type: {sample_masks.__class__}"
diff --git a/nilearn/signal.py b/nilearn/signal.py
--- a/nilearn/signal.py
+++ b/nilearn/signal.py
@@ -12,6 +12,7 @@
 import numpy as np
 import pandas as pd
 from scipy import linalg, signal as sp_signal
+from scipy.interpolate import CubicSpline
 from sklearn.utils import gen_even_slices, as_float_array
 
 from ._utils.numpy_conversions import csv_to_array, as_ndarray
@@ -464,11 +465,11 @@ def clean(signals, runs=None, detrend=True, standardize='zscore',
           ensure_finite=False):
     """Improve :term:`SNR` on masked :term:`fMRI` signals.
 
-    This function can do several things on the input signals, in
-    the following order:
+    This function can do several things on the input signals. With the default
+    options, the procedures are performed in the following order:
 
     - detrend
-    - low- and high-pass filter
+    - low- and high-pass butterworth filter
     - remove confounds
     - standardize
 
@@ -476,11 +477,33 @@ def clean(signals, runs=None, detrend=True, standardize='zscore',
 
     High-pass filtering should be kept small, to keep some sensitivity.
 
-    Filtering is only meaningful on evenly-sampled signals.
+    Butterworth filtering is only meaningful on evenly-sampled signals.
+
+    When performing scrubbing (censoring high-motion volumes) with butterworth
+    filtering, the signal is processed in the following order, based on the
+    second recommendation in :footcite:`Lindquist2018`:
+
+    - interpolate high motion volumes with cubic spline interpolation
+    - detrend
+    - low- and high-pass butterworth filter
+    - censor high motion volumes
+    - remove confounds
+    - standardize
 
     According to :footcite:`Lindquist2018`, removal of confounds will be done
     orthogonally to temporal filters (low- and/or high-pass filters), if both
-    are specified.
+    are specified. The censored volumes should be removed in both signals and
+    confounds before the nuisance regression.
+
+    When performing scrubbing with cosine drift term filtering, the signal is
+    processed in the following order, based on the first recommendation in
+    :footcite:`Lindquist2018`:
+
+    - generate cosine drift term
+    - censor high motion volumes in both signal and confounds
+    - detrend
+    - remove confounds
+    - standardize
 
     Parameters
     ----------
@@ -553,7 +576,8 @@ def clean(signals, runs=None, detrend=True, standardize='zscore',
     Returns
     -------
     cleaned_signals : :class:`numpy.ndarray`
-        Input signals, cleaned. Same shape as `signals`.
+        Input signals, cleaned. Same shape as `signals` unless `sample_mask`
+        is applied.
 
     Notes
     -----
@@ -575,30 +599,56 @@ def clean(signals, runs=None, detrend=True, standardize='zscore',
     confounds = stringify_path(confounds)
     if confounds is not None:
         _check_signal_parameters(detrend, standardize_confounds)
+    # check if filter parameters are satisfied and return correct filter
+    filter_type = _check_filter_parameters(filter, low_pass, high_pass, t_r)
 
     # Read confounds and signals
-    signals, runs, confounds = _sanitize_inputs(
+    signals, runs, confounds, sample_mask = _sanitize_inputs(
         signals, runs, confounds, sample_mask, ensure_finite
     )
-    use_filter = _check_filter_parameters(filter, low_pass, high_pass, t_r)
-    # Restrict the signal to the orthogonal of the confounds
+
+    # Process each run independently
     if runs is not None:
-        signals = _process_runs(signals, runs, detrend, standardize,
-                                   confounds, low_pass, high_pass, t_r)
+        return _process_runs(signals, runs, detrend, standardize,
+                             confounds, sample_mask,
+                             filter_type, low_pass, high_pass, t_r)
+
+    # For the following steps, sample_mask should be either None or index-like
+
+    # Generate cosine drift terms using the full length of the signals
+    if filter_type == 'cosine':
+        confounds = _create_cosine_drift_terms(signals, confounds, high_pass,
+                                               t_r)
+
+    # Interpolation / censoring
+    signals, confounds = _handle_scrubbed_volumes(signals, confounds,
+                                                  sample_mask, filter_type,
+                                                  t_r)
 
     # Detrend
     # Detrend and filtering should apply to confounds, if confound presents
     # keep filters orthogonal (according to Lindquist et al. (2018))
+    # Restrict the signal to the orthogonal of the confounds
     if detrend:
         mean_signals = signals.mean(axis=0)
         signals = _standardize(signals, standardize=False, detrend=detrend)
         if confounds is not None:
             confounds = _standardize(confounds, standardize=False,
-                                detrend=detrend)
-    if use_filter:
-        # check if filter parameters are satisfied and filter according to the strategy
-        signals, confounds = _filter_signal(signals, confounds, filter,
-                                            low_pass, high_pass, t_r)
+                                     detrend=detrend)
+
+    # Butterworth filtering
+    if filter_type == 'butterworth':
+        signals = butterworth(signals, sampling_rate=1. / t_r,
+                              low_pass=low_pass, high_pass=high_pass)
+        if confounds is not None:
+            # Apply low- and high-pass filters to keep filters orthogonal
+            # (according to Lindquist et al. (2018))
+            confounds = butterworth(confounds, sampling_rate=1. / t_r,
+                                    low_pass=low_pass, high_pass=high_pass)
+        # apply sample_mask to remove censored volumes after signal filtering
+        if sample_mask is not None:
+            signals, confounds = _censor_signals(signals, confounds,
+                                                 sample_mask)
 
     # Remove confounds
     if confounds is not None:
@@ -630,29 +680,84 @@ def clean(signals, runs=None, detrend=True, standardize='zscore',
     return signals
 
 
-def _filter_signal(signals, confounds, filter, low_pass, high_pass, t_r):
-    '''Filter signal based on provided strategy.'''
-    if filter == 'butterworth':
-        signals = butterworth(signals, sampling_rate=1. / t_r,
-                              low_pass=low_pass, high_pass=high_pass)
+def _handle_scrubbed_volumes(signals, confounds, sample_mask, filter_type,
+                             t_r):
+    """Interpolate or censor scrubbed volumes."""
+    if sample_mask is None:
+        return signals, confounds
+
+    if filter_type == 'butterworth':
+        signals = _interpolate_volumes(signals, sample_mask, t_r)
         if confounds is not None:
-            # Apply low- and high-pass filters to keep filters orthogonal
-            # (according to Lindquist et al. (2018))
-            confounds = butterworth(confounds, sampling_rate=1. / t_r,
-                                    low_pass=low_pass, high_pass=high_pass)
-    elif filter == 'cosine':
-        from .glm.first_level.design_matrix import _cosine_drift
-        frame_times = np.arange(signals.shape[0]) * t_r
-        cosine_drift = _cosine_drift(high_pass, frame_times)
-        if confounds is None:
-            confounds = cosine_drift.copy()
-        else:
-            confounds = np.hstack((confounds, cosine_drift))
+            confounds = _interpolate_volumes(confounds, sample_mask, t_r)
+    else:  # Or censor when no filtering, or cosine filter
+        signals, confounds = _censor_signals(signals, confounds, sample_mask)
     return signals, confounds
 
 
-def _process_runs(signals, runs, detrend, standardize, confounds,
-                  low_pass, high_pass, t_r):
+def _censor_signals(signals, confounds, sample_mask):
+    """Apply sample masks to data."""
+    signals = signals[sample_mask, :]
+    if confounds is not None:
+        confounds = confounds[sample_mask, :]
+    return signals, confounds
+
+
+def _interpolate_volumes(volumes, sample_mask, t_r):
+    """Interpolate censored volumes in signals/confounds."""
+    frame_times = np.arange(volumes.shape[0]) * t_r
+    remained_vol = frame_times[sample_mask]
+    remained_x = volumes[sample_mask, :]
+    cubic_spline_fitter = CubicSpline(remained_vol, remained_x)
+    volumes_interpolated = cubic_spline_fitter(frame_times)
+    volumes[~sample_mask, :] = volumes_interpolated[~sample_mask, :]
+    return volumes
+
+
+def _create_cosine_drift_terms(signals, confounds, high_pass, t_r):
+    """Create cosine drift terms, append to confounds regressors."""
+    from nilearn.glm.first_level.design_matrix import _cosine_drift
+    frame_times = np.arange(signals.shape[0]) * t_r
+    # remove constant, as the signal is mean centered
+    cosine_drift = _cosine_drift(high_pass, frame_times)[:, :-1]
+    confounds = _check_cosine_by_user(confounds, cosine_drift)
+    return confounds
+
+
+def _check_cosine_by_user(confounds, cosine_drift):
+    """Check if cosine term exists, based on correlation > 0.9. """
+    # stack consine drift terms if there's no cosine drift term in data
+    n_cosines = cosine_drift.shape[1]
+
+    if n_cosines == 0:
+        warnings.warn(
+            "Cosine filter was not created. The time series might be too "
+            "short or the high pass filter is not suitable for the data."
+        )
+        return confounds
+
+    if confounds is None:
+        return cosine_drift.copy()
+
+    # check if cosine drift term is supplied by user
+    # given the threshold and timeseries length, there can be no cosine drift
+    # term
+    corr_cosine = np.corrcoef(cosine_drift.T, confounds.T)
+    np.fill_diagonal(corr_cosine, 0)
+    cosine_exists = sum(corr_cosine[:n_cosines, :].flatten() > 0.9) > 0
+
+    if cosine_exists:
+        warnings.warn(
+            "Cosine filter(s) exist in user supplied confounds."
+            "Use user supplied regressors only."
+        )
+        return confounds
+
+    return np.hstack((confounds, cosine_drift))
+
+
+def _process_runs(signals, runs, detrend, standardize, confounds, sample_mask,
+                  filter, low_pass, high_pass, t_r):
     """Process each run independently."""
     if len(runs) != len(signals):
         raise ValueError(
@@ -661,16 +766,22 @@ def _process_runs(signals, runs, detrend, standardize, confounds,
                 'does not match the length of the signals (%i)'
             ) % (len(runs), len(signals))
         )
-    for run in np.unique(runs):
+    cleaned_signals = []
+    for i, run in enumerate(np.unique(runs)):
         run_confounds = None
+        run_sample_mask = None
         if confounds is not None:
             run_confounds = confounds[runs == run]
-        signals[runs == run, :] = \
+        if sample_mask is not None:
+            run_sample_mask = sample_mask[i]
+        run_signals = \
             clean(signals[runs == run],
                   detrend=detrend, standardize=standardize,
-                  confounds=run_confounds, low_pass=low_pass,
+                  confounds=run_confounds, sample_mask=run_sample_mask,
+                  filter=filter, low_pass=low_pass,
                   high_pass=high_pass, t_r=t_r)
-    return signals
+        cleaned_signals.append(run_signals)
+    return np.vstack(cleaned_signals)
 
 
 def _sanitize_inputs(signals, runs, confounds, sample_mask, ensure_finite):
@@ -680,15 +791,7 @@ def _sanitize_inputs(signals, runs, confounds, sample_mask, ensure_finite):
     confounds = _sanitize_confounds(n_time, n_runs, confounds)
     sample_mask = _sanitize_sample_mask(n_time, n_runs, runs, sample_mask)
     signals = _sanitize_signals(signals, ensure_finite)
-
-    if sample_mask is None:
-        return signals, runs, confounds
-
-    if confounds is not None:
-        confounds = confounds[sample_mask, :]
-    if runs is not None:
-        runs = runs[sample_mask]
-    return signals[sample_mask, :], runs, confounds
+    return signals, runs, confounds, sample_mask
 
 
 def _sanitize_confounds(n_time, n_runs, confounds):
@@ -718,21 +821,17 @@ def _sanitize_sample_mask(n_time, n_runs, runs, sample_mask):
     """Check sample_mask is the right data type and matches the run index."""
     if sample_mask is None:
         return sample_mask
+
     sample_mask = _check_run_sample_masks(n_runs, sample_mask)
 
     if runs is None:
         runs = np.zeros(n_time)
 
-    # handle multiple runs
-    masks = []
-    starting_index = 0
+    # check sample mask of each run
     for i, current_mask in enumerate(sample_mask):
         _check_sample_mask_index(i, n_runs, runs, current_mask)
-        current_mask += starting_index
-        masks.append(current_mask)
-        starting_index = sum(i == runs)
-    sample_mask = np.hstack(masks)
-    return sample_mask
+
+    return sample_mask[0] if sum(runs) == 0 else sample_mask
 
 
 def _check_sample_mask_index(i, n_runs, runs, current_mask):
@@ -841,7 +940,7 @@ def _check_filter_parameters(filter, low_pass, high_pass, t_r):
                     "high_pass='{0}', low_pass='{1}'"
                     .format(high_pass, low_pass)
                 )
-        return True
+        return filter
     else:
         raise ValueError("Filter method {} not implemented.".format(filter))
 


Test patch:
diff --git a/nilearn/tests/test_signal.py b/nilearn/tests/test_signal.py
--- a/nilearn/tests/test_signal.py
+++ b/nilearn/tests/test_signal.py
@@ -410,6 +410,13 @@ def test_clean_runs():
     # clean should not modify inputs
     assert np.array_equal(x_orig, x)
 
+    # check the runs are individually cleaned
+    x_run1 = nisignal.clean(x[0:n_samples // 2, :],
+                            confounds=confounds[0:n_samples // 2, :],
+                            standardize=False, detrend=True,
+                            low_pass=None, high_pass=None)
+    assert np.array_equal(x_run1, x_detrended[0:n_samples // 2, :])
+
 
 def test_clean_confounds():
     signals, noises, confounds = generate_signals(n_features=41,
@@ -562,22 +569,31 @@ def test_clean_frequencies_using_power_spectrum_density():
     _, _, confounds = generate_signals(
         n_features=10, n_confounds=10, length=100)
 
-    # Apply low- and high-pass filter (separately)
+    # Apply low- and high-pass filter with butterworth (separately)
     t_r = 1.0
     low_pass = 0.1
     high_pass = 0.4
-    res_low = clean(sx, detrend=False, standardize=False, low_pass=low_pass,
-                    high_pass=None, t_r=t_r)
-    res_high = clean(sx, detrend=False, standardize=False, low_pass=None,
-                     high_pass=high_pass, t_r=t_r)
+    res_low = clean(sx, detrend=False, standardize=False,
+                    filter='butterworth', low_pass=low_pass, high_pass=None,
+                    t_r=t_r)
+    res_high = clean(sx, detrend=False, standardize=False,
+                     filter='butterworth', low_pass=None, high_pass=high_pass,
+                     t_r=t_r)
+
+    # cosine high pass filter
+    res_cos = clean(sx, detrend=False, standardize=False,
+                    filter='cosine', low_pass=None, high_pass=high_pass,
+                    t_r=t_r)
 
     # Compute power spectrum density for both test
     f, Pxx_den_low = scipy.signal.welch(np.mean(res_low.T, axis=0), fs=t_r)
     f, Pxx_den_high = scipy.signal.welch(np.mean(res_high.T, axis=0), fs=t_r)
+    f, Pxx_den_cos = scipy.signal.welch(np.mean(res_cos.T, axis=0), fs=t_r)
 
     # Verify that the filtered frequencies are removed
     assert np.sum(Pxx_den_low[f >= low_pass * 2.]) <= 1e-4
     assert np.sum(Pxx_den_high[f <= high_pass / 2.]) <= 1e-4
+    assert np.sum(Pxx_den_cos[f <= high_pass / 2.]) <= 1e-4
 
 
 def test_clean_finite_no_inplace_mod():
@@ -724,30 +740,42 @@ def test_clean_zscore():
     np.testing.assert_almost_equal(cleaned_signals.std(0), 1)
 
 
-def test_cosine_filter():
+def test_create_cosine_drift_terms():
     '''Testing cosine filter interface and output.'''
     from nilearn.glm.first_level.design_matrix import _cosine_drift
-
-    t_r, high_pass, low_pass, filter = 2.5, 0.002, None, 'cosine'
-    signals, _, confounds = generate_signals(n_features=41,
-                                              n_confounds=5, length=45)
+    # fmriprep high pass cutoff is 128s, it's around 0.008 hz
+    t_r, high_pass = 2.5, 0.008
+    signals, _, confounds = generate_signals(n_features=41, n_confounds=5,
+                                             length=45)
 
     # Not passing confounds it will return drift terms only
     frame_times = np.arange(signals.shape[0]) * t_r
-    cosine_drift = _cosine_drift(high_pass, frame_times)
+    cosine_drift = _cosine_drift(high_pass, frame_times)[:, :-1]
+    confounds_with_drift = np.hstack((confounds, cosine_drift))
 
-    signals_unchanged, cosine_confounds = nisignal._filter_signal(
-        signals, confounds, filter, low_pass, high_pass, t_r)
-    np.testing.assert_array_equal(signals_unchanged, signals)
+    cosine_confounds = nisignal._create_cosine_drift_terms(
+        signals, confounds, high_pass, t_r)
     np.testing.assert_almost_equal(cosine_confounds,
                                    np.hstack((confounds, cosine_drift)))
 
     # Not passing confounds it will return drift terms only
-    signals_unchanged, drift_terms_only = nisignal._filter_signal(
-        signals, None, filter, low_pass, high_pass, t_r)
-    np.testing.assert_array_equal(signals_unchanged, signals)
+    drift_terms_only = nisignal._create_cosine_drift_terms(
+        signals, None, high_pass, t_r)
     np.testing.assert_almost_equal(drift_terms_only, cosine_drift)
 
+    # drift terms in confounds will create warning and no change to confounds
+    with pytest.warns(UserWarning, match='user supplied confounds'):
+        cosine_confounds = nisignal._create_cosine_drift_terms(
+            signals, confounds_with_drift, high_pass, t_r)
+    np.testing.assert_array_equal(cosine_confounds, confounds_with_drift)
+
+    # raise warning if cosine drift term is not created
+    high_pass_fail = 0.002
+    with pytest.warns(UserWarning, match='Cosine filter was not create'):
+        cosine_confounds = nisignal._create_cosine_drift_terms(
+            signals, confounds, high_pass_fail, t_r)
+    np.testing.assert_array_equal(cosine_confounds, confounds)
+
 
 def test_sample_mask():
     """Test sample_mask related feature."""
@@ -801,3 +829,29 @@ def test_sample_mask():
     sample_mask[-1] = 999
     with pytest.raises(IndexError, match=r'invalid index \[\d*\]'):
         clean(signals, sample_mask=sample_mask)
+
+
+def test_handle_scrubbed_volumes():
+    """Check interpolation/censoring of signals based on filter type. """
+    signals, _, confounds = generate_signals(n_features=11,
+                                             n_confounds=5, length=40)
+
+    sample_mask = np.arange(signals.shape[0])
+    scrub_index = np.array([2, 3, 6, 7, 8, 30, 31, 32])
+    sample_mask = np.delete(sample_mask, scrub_index)
+
+    interpolated_signals, interpolated_confounds = \
+        nisignal._handle_scrubbed_volumes(signals, confounds, sample_mask,
+                                          'butterworth', 2.5)
+    np.testing.assert_equal(interpolated_signals[sample_mask, :],
+                            signals[sample_mask, :])
+    np.testing.assert_equal(interpolated_confounds[sample_mask, :],
+                            confounds[sample_mask, :])
+
+    scrubbed_signals, scrubbed_confounds = \
+        nisignal._handle_scrubbed_volumes(signals, confounds, sample_mask,
+                                          'cosine', 2.5)
+    np.testing.assert_equal(scrubbed_signals,
+                            signals[sample_mask, :])
+    np.testing.assert_equal(scrubbed_confounds,
+                            confounds[sample_mask, :])