Repository: nilearn/nilearn

Issue:
[BUG] NiftiSpheresMasker - Combination of percent signal change normalization and high-pass filtering only outputs signals in incorrect range.
### Is there an existing issue for this?

- [X] I have searched the existing issues

### Operating system

- [X] Linux
- [X] Mac
- [ ] Windows

### Operating system version

- Linux Ubuntu 18.04
- Mac OS Ventura 13.6


### Python version

- [ ] 3.12
- [ ] 3.11
- [ ] 3.10
- [X] 3.9
- [ ] 3.8

### nilearn version

0.10.2

### Expected behavior

Percent signal change (PSC) normalization method in NiftiSpheresMasker in combination with high-pass filtering should return reasonable PSC values.

### Current behavior & error messages

When extracting time courses with NiftiSpheresMasker, the combination of percent signal change normalization with high-pass filtering only outputs signals in an incorrect range.

I am using this example code https://nilearn.github.io/stable/auto_examples/03_connectivity/plot_sphere_based_connectome.html to showcase this.

This is the output image of the code snippet below. Please note the y-axis values range.
![image](https://github.com/nilearn/nilearn/assets/9429033/a82a0302-b886-4007-9298-939ecdfe532f)


### Steps and code to reproduce bug

```python
from nilearn import datasets

dataset = datasets.fetch_development_fmri(n_subjects=1)

dmn_coords = [(0, -52, 18), (-46, -68, 32)]
labels = [
    "Posterior Cingulate Cortex",
    "Left Temporoparietal junction",
]

from nilearn.maskers import NiftiSpheresMasker

masker = NiftiSpheresMasker(
    dmn_coords,
    radius=8,
    #detrend=True,
    standardize="psc",
    #standardize_confounds="zscore_sample",
    #low_pass=0.1,
    high_pass=0.01,
    t_r=2,
    memory="nilearn_cache",
    memory_level=1,
    verbose=2,
    clean__butterworth__padtype="even",  # kwarg to modify Butterworth filter
)

# Additionally, we pass confound information to ensure our extracted
# signal is cleaned from confounds.

func_filename = dataset.func[0]
confounds_filename = dataset.confounds[0]

time_series = masker.fit_transform(
    func_filename, confounds=[confounds_filename]
)

import matplotlib.pyplot as plt

for time_serie, label in zip(time_series.T, labels):
    plt.plot(time_serie, label=label)

plt.title("Default Mode Network Time Series")
plt.xlabel("Scan number")
plt.ylabel("Normalized signal")
plt.legend()
plt.tight_layout()
```



Discussion:
Thx for reporting !
I do not have time to check now, but I guess that the things goes back to preprocessing steps that are not run in the right order. Would you be interested in diagnosing that more precisely ?

Hi @alexsayal I can reproduce the issue from the script you provided. Let us know if you'd like to contribute a fix or if you want us to take over.
Hi! I am afraid I won't be able to check this soon, so please take over whenever you can 👍 Thank you!
I did some digging and it seems related to the values of the signals returned from calling `butterworth` under `signal.clean`. When passing a high pass filter of 0.01, the mean of the signal becomes very small relative to the maximum value of the signals. The order of the steps seems correct to me (standardization is being called last, on the filtered signal). And "psc" implementation also seems correct (see https://github.com/nilearn/nilearn/blob/9361610f84861c9d7aae3514b2af126f387f893f/nilearn/signal.py#L105-L109) 
Could it be related to the value passed to the high pass filter?
Well, indeed a high-pass filter removes the low-frequency signal, including the constant part. Maybe there is inconsistency among the processing options ?
As the high-pass filter seems to remove the constant part, the true mean of the signal is lost and hence the standardization is performed using a mean value very close to zero, which in turn results in those inflated values. Does it make sense to invert the order of the steps? Or set the filter to keep the constant part of the signal?
The idea would be to save the signal mean before filtering, I guess.

Resolving patch:
diff --git a/nilearn/signal.py b/nilearn/signal.py
--- a/nilearn/signal.py
+++ b/nilearn/signal.py
@@ -27,7 +27,11 @@
 availiable_filters = ["butterworth", "cosine"]
 
 
-def standardize_signal(signals, detrend=False, standardize="zscore"):
+def standardize_signal(
+    signals,
+    detrend=False,
+    standardize="zscore",
+):
     """Center and standardize a given signal (time is along first axis).
 
     Parameters
@@ -109,9 +113,9 @@ def standardize_signal(signals, detrend=False, standardize="zscore"):
             signals /= std
 
         elif standardize == "psc":
-            mean_signal = signals.mean(axis=0)
-            invalid_ix = np.absolute(mean_signal) < np.finfo(np.float64).eps
-            signals = (signals - mean_signal) / np.absolute(mean_signal)
+            mean_signals = signals.mean(axis=0)
+            invalid_ix = np.absolute(mean_signals) < np.finfo(np.float64).eps
+            signals = (signals - mean_signals) / np.absolute(mean_signals)
             signals *= 100
 
             if np.any(invalid_ix):
@@ -742,8 +746,8 @@ def clean(
     # Detrend and filtering should apply to confounds, if confound presents
     # keep filters orthogonal (according to Lindquist et al. (2018))
     # Restrict the signal to the orthogonal of the confounds
+    mean_signals = signals.mean(axis=0)
     if detrend:
-        mean_signals = signals.mean(axis=0)
         signals = standardize_signal(
             signals, standardize=False, detrend=detrend
         )
@@ -802,17 +806,21 @@ def clean(
         signals -= Q.dot(Q.T).dot(signals)
 
     # Standardize
-    if detrend and (standardize == "psc"):
-        # If the signal is detrended, we have to know the original mean
-        # signal to calculate the psc.
+    if (detrend and standardize == "psc") or (filter_type == "butterworth"):
+        # If the signal is detrended or filtered,
+        # the mean signal will be zero or close to zero. In this case,
+        # we have to know the original mean signal to calculate the psc.
         signals = standardize_signal(
-            signals + mean_signals, standardize=standardize, detrend=False
+            signals + mean_signals,
+            standardize=standardize,
+            detrend=False,
         )
     else:
         signals = standardize_signal(
-            signals, standardize=standardize, detrend=False
+            signals,
+            standardize=standardize,
+            detrend=False,
         )
-
     return signals
 
 
@@ -1101,9 +1109,9 @@ def _check_filter_parameters(filter, low_pass, high_pass, t_r):
                 f"filtering.t_r='{t_r}', high_pass='{high_pass}'"
             )
         if filter == "butterworth":
-            if all(item is None for item in [low_pass, high_pass, t_r]):
+            if all(item is None for item in [low_pass, high_pass]):
                 # Butterworth was switched off by passing
-                # None to all these parameters
+                # None to at least low_pass and high_pass
                 return False
             if t_r is None:
                 raise ValueError(


Test patch:
diff --git a/nilearn/tests/test_signal.py b/nilearn/tests/test_signal.py
--- a/nilearn/tests/test_signal.py
+++ b/nilearn/tests/test_signal.py
@@ -884,13 +884,24 @@ def test_clean_psc(rng):
 
     # both types should pass
     for s in [signals_pos_mean, signals_mixed_mean]:
-        cleaned_signals = clean(s, standardize="psc")
+        cleaned_signals = clean(s, standardize="psc", detrend=False)
         np.testing.assert_almost_equal(cleaned_signals.mean(0), 0)
 
         tmp = (s - s.mean(0)) / np.abs(s.mean(0))
         tmp *= 100
         np.testing.assert_almost_equal(cleaned_signals, tmp)
 
+        # test with high pass with butterworth
+        butterworth_signals = clean(
+            s,
+            detrend=False,
+            filter="butterworth",
+            high_pass=0.01,
+            tr=2,
+            standardize="psc",
+        )
+        np.testing.assert_almost_equal(butterworth_signals.mean(0), 0)
+
     # leave out the last 3 columns with a mean of zero to test user warning
     signals_w_zero = signals + np.append(means[:, :-3], np.zeros((1, 3)))
     with pytest.warns(UserWarning) as records: