Repository: nilearn/nilearn

Issue:
[BUG] Interpolation of high motion volumes at the beginning/end scan
### Is there an existing issue for this?

- [X] I have searched the existing issues

### Operating system

- [X] Linux
- [ ] Mac
- [ ] Windows

### Operating system version

For example one of the following:
- Linux Ubuntu 22.04


### Python version

- [ ] 3.11
- [ ] 3.10
- [X] 3.9
- [ ] 3.8
- [ ] 3.7

### nilearn version

0.10.2

### Expected behavior

At scrubbing  (censoring high-motion volumes) step, the volumes at the beginning/end of the scan -if labelled as high-motion outliers- shouldn't be interpolated (*extrapolated*) but discarded before filtering.

### Current behavior & error messages

Currently, the function [_interpolate_volumes()](https://github.com/nilearn/nilearn/blob/b1fa2e0b9161a5055a900b708aed38aaab15866f/nilearn/signal.py#L779) uses a CubicSpline (scipy class) with `extrapolate` feature enabled by default. Asking the interpolation method to predict values outside the range of data available is -in my opinion- not a very good idea. The results of the interpolation, and further scrubbing, are not reliable. 

I am happy to open a PR for this with a fix proposal if there's consensus.

Example of signal interpolation as is and the disabling the `extrapolate` flag,
![image](https://github.com/nilearn/nilearn/assets/6368825/dd313474-fe42-456e-9f65-9b082e62cc7d)
![image](https://github.com/nilearn/nilearn/assets/6368825/019c4739-ae1c-400e-825a-497160863af8)
![image](https://github.com/nilearn/nilearn/assets/6368825/8606e533-d645-4200-be4c-4df55673b147)  

Thanks,
Jordi

### Steps and code to reproduce bug

```python
import numpy as np
from nilearn import signal
import nibabel as nib

data = nib.load("/tmp/xxx_desc-preproc_bold.nii.gz'').get_data()
volumes = np.transpose(data, (3, 0, 1, 2))

# dummy list of censored high-motion volumes
sample_mask = np.zeros(data_reshaped.shape[0], dtype=bool)
sample_mask[0:50] = 1
sample_mask[-50:] = 1
sample_mask = ~sample_mask

volumes = signal._interpolate_volumes(volumes, sample_mask, t_r=1.6)
```


Discussion:
@jhuguetn thanks for reporting! Tagging @htwangtw for input on this one
Thanks for opening the issue @jhuguetn!

> Asking the interpolation method to predict values outside the range of data available is -in my opinion- not a very good idea.

I agree and thanks for picking up on this!

If I understand the issue correctly, this change will only apply to when butterworth filter and scrubbing is applied? Also make sure the confounds reflects that you discarded the head and tail of the signal. 


Please open a PR to address this - I do not use butterworth as my preferred filtering methods so it would be great to have you working on this issue. I am happy to assist during code review :)
Hi @htwangtw and sorry for the late reply. 

> If I understand the issue correctly, this change will only apply to when butterworth filter and scrubbing is applied?  

Correct, see [here](https://github.com/nilearn/nilearn/blob/main/nilearn/signal.py#L819-L822).

Sure, no problem. I will now open a PR with a proposal then. Basically what I have in mind is disabling the data extrapolation by default but exposing the parameter just in case someone wants to change it to True or use a `periodic` extrapolation.



Resolving patch:
diff --git a/nilearn/signal.py b/nilearn/signal.py
--- a/nilearn/signal.py
+++ b/nilearn/signal.py
@@ -547,6 +547,7 @@ def clean(
     high_pass=None,
     t_r=2.5,
     ensure_finite=False,
+    extrapolate=True,
     **kwargs,
 ):
     """Improve :term:`SNR` on masked :term:`fMRI` signals.
@@ -667,6 +668,11 @@ def clean(
         If `True`, the non-finite values (NANs and infs) found in the data
         will be replaced by zeros.
 
+    extrapolate : :obj:`bool`, default=True
+        If `True` and filter='butterworth', censored volumes in both ends of
+        the signal data will be interpolated before filtering. Otherwise, they
+        will be discarded from the band-pass filtering process.
+
     kwargs : dict
         Keyword arguments to be passed to functions called within ``clean``.
         Kwargs prefixed with ``'butterworth__'`` will be passed to
@@ -731,10 +737,9 @@ def clean(
         )
 
     # Interpolation / censoring
-    signals, confounds = _handle_scrubbed_volumes(
-        signals, confounds, sample_mask, filter_type, t_r
+    signals, confounds, sample_mask = _handle_scrubbed_volumes(
+        signals, confounds, sample_mask, filter_type, t_r, extrapolate
     )
-
     # Detrend
     # Detrend and filtering should apply to confounds, if confound presents
     # keep filters orthogonal (according to Lindquist et al. (2018))
@@ -810,19 +815,29 @@ def clean(
 
 
 def _handle_scrubbed_volumes(
-    signals, confounds, sample_mask, filter_type, t_r
+    signals, confounds, sample_mask, filter_type, t_r, extrapolate
 ):
     """Interpolate or censor scrubbed volumes."""
     if sample_mask is None:
-        return signals, confounds
+        return signals, confounds, sample_mask
 
     if filter_type == "butterworth":
-        signals = _interpolate_volumes(signals, sample_mask, t_r)
+        signals = _interpolate_volumes(signals, sample_mask, t_r, extrapolate)
+        # discard non-interpolated out-of-bounds volumes
+        signals = signals[~np.isnan(signals).all(axis=1), :]
         if confounds is not None:
-            confounds = _interpolate_volumes(confounds, sample_mask, t_r)
+            confounds = _interpolate_volumes(
+                confounds, sample_mask, t_r, extrapolate
+            )
+            # discard non-interpolated out-of-bounds volumes
+            confounds = confounds[~np.isnan(confounds).all(axis=1), :]
+        if sample_mask is not None and not extrapolate:
+            # reset the indexing of the sample_mask excluding non-interpolated
+            # volumes at the head of the data
+            sample_mask -= sample_mask[0]
     else:  # Or censor when no filtering, or cosine filter
         signals, confounds = _censor_signals(signals, confounds, sample_mask)
-    return signals, confounds
+    return signals, confounds, sample_mask
 
 
 def _censor_signals(signals, confounds, sample_mask):
@@ -833,12 +848,26 @@ def _censor_signals(signals, confounds, sample_mask):
     return signals, confounds
 
 
-def _interpolate_volumes(volumes, sample_mask, t_r):
+def _interpolate_volumes(volumes, sample_mask, t_r, extrapolate):
     """Interpolate censored volumes in signals/confounds."""
+    if extrapolate:
+        extrapolate_default = (
+            "By default the cubic spline interpolator extrapolates "
+            "the out-of-bounds censored volumes in the data run. This "
+            "can lead to undesired filtered signal results. Starting in "
+            "version 0.13, the default strategy will be not to extrapolate "
+            "but to discard those volumes at filtering."
+        )
+        warnings.warn(
+            category=FutureWarning,
+            message=extrapolate_default,
+        )
     frame_times = np.arange(volumes.shape[0]) * t_r
     remained_vol = frame_times[sample_mask]
     remained_x = volumes[sample_mask, :]
-    cubic_spline_fitter = CubicSpline(remained_vol, remained_x)
+    cubic_spline_fitter = CubicSpline(
+        remained_vol, remained_x, extrapolate=extrapolate
+    )
     volumes_interpolated = cubic_spline_fitter(frame_times)
     volumes[~sample_mask, :] = volumes_interpolated[~sample_mask, :]
     return volumes


Test patch:
diff --git a/nilearn/tests/test_signal.py b/nilearn/tests/test_signal.py
--- a/nilearn/tests/test_signal.py
+++ b/nilearn/tests/test_signal.py
@@ -1065,8 +1065,9 @@ def test_handle_scrubbed_volumes():
     (
         interpolated_signals,
         interpolated_confounds,
+        sample_mask,
     ) = nisignal._handle_scrubbed_volumes(
-        signals, confounds, sample_mask, "butterworth", 2.5
+        signals, confounds, sample_mask, "butterworth", 2.5, True
     )
     np.testing.assert_equal(
         interpolated_signals[sample_mask, :], signals[sample_mask, :]
@@ -1075,8 +1076,96 @@ def test_handle_scrubbed_volumes():
         interpolated_confounds[sample_mask, :], confounds[sample_mask, :]
     )
 
-    scrubbed_signals, scrubbed_confounds = nisignal._handle_scrubbed_volumes(
-        signals, confounds, sample_mask, "cosine", 2.5
+    (
+        scrubbed_signals,
+        scrubbed_confounds,
+        sample_mask,
+    ) = nisignal._handle_scrubbed_volumes(
+        signals, confounds, sample_mask, "cosine", 2.5, True
     )
     np.testing.assert_equal(scrubbed_signals, signals[sample_mask, :])
     np.testing.assert_equal(scrubbed_confounds, confounds[sample_mask, :])
+
+
+def test_handle_scrubbed_volumes_with_extrapolation():
+    """Check interpolation of signals with extrapolation."""
+    signals, _, confounds = generate_signals(
+        n_features=11, n_confounds=5, length=40
+    )
+
+    sample_mask = np.arange(signals.shape[0])
+    scrub_index = np.concatenate((np.arange(5), [10, 20, 30]))
+    sample_mask = np.delete(sample_mask, scrub_index)
+
+    # Test cubic spline interpolation (enabled extrapolation) in the
+    # very first n=5 samples of generated signal
+    extrapolate_warning = (
+        "By default the cubic spline interpolator extrapolates "
+        "the out-of-bounds censored volumes in the data run. This "
+        "can lead to undesired filtered signal results. Starting in "
+        "version 0.13, the default strategy will be not to extrapolate "
+        "but to discard those volumes at filtering."
+    )
+    with pytest.warns(FutureWarning, match=extrapolate_warning):
+        (
+            extrapolated_signals,
+            extrapolated_confounds,
+            extrapolated_sample_mask,
+        ) = nisignal._handle_scrubbed_volumes(
+            signals, confounds, sample_mask, "butterworth", 2.5, True
+        )
+    np.testing.assert_equal(signals.shape[0], extrapolated_signals.shape[0])
+    np.testing.assert_equal(
+        confounds.shape[0], extrapolated_confounds.shape[0]
+    )
+    np.testing.assert_equal(sample_mask, extrapolated_sample_mask)
+
+
+def test_handle_scrubbed_volumes_without_extrapolation():
+    """Check interpolation of signals disabling extrapolation."""
+    signals, _, confounds = generate_signals(
+        n_features=11, n_confounds=5, length=40
+    )
+
+    outer_samples = [0, 1, 2, 3, 4]
+    inner_samples = [10, 20, 30]
+    total_samples = len(outer_samples) + len(inner_samples)
+    sample_mask = np.arange(signals.shape[0])
+    scrub_index = np.concatenate((outer_samples, inner_samples))
+    sample_mask = np.delete(sample_mask, scrub_index)
+
+    # Test cubic spline interpolation without predicting values outside
+    # the range of the signal available (disabled extrapolation), discarding
+    # the first n censored samples of generated signal
+    (
+        interpolated_signals,
+        interpolated_confounds,
+        interpolated_sample_mask,
+    ) = nisignal._handle_scrubbed_volumes(
+        signals, confounds, sample_mask, "butterworth", 2.5, False
+    )
+    np.testing.assert_equal(
+        signals.shape[0], interpolated_signals.shape[0] + len(outer_samples)
+    )
+    np.testing.assert_equal(
+        confounds.shape[0],
+        interpolated_confounds.shape[0] + len(outer_samples),
+    )
+    np.testing.assert_equal(
+        sample_mask - sample_mask[0], interpolated_sample_mask
+    )
+
+    # Assert that the modified sample mask (interpolated_sample_mask)
+    # can be applied to the interpolated signals and confounds
+    (
+        censored_signals,
+        censored_confounds,
+    ) = nisignal._censor_signals(
+        interpolated_signals, interpolated_confounds, interpolated_sample_mask
+    )
+    np.testing.assert_equal(
+        signals.shape[0], censored_signals.shape[0] + total_samples
+    )
+    np.testing.assert_equal(
+        confounds.shape[0], censored_confounds.shape[0] + total_samples
+    )