Repository: mne-tools/mne-python

Issue:
mne.time_frequency.psd_welch with option average=median
In the function `mne.time_frequency.psd_welch`, for the option `average` it is written:
_How to average the segments. If mean (default), calculate the arithmetic mean. If median, calculate the median, corrected for its bias relative to the mean. If None, returns the unaggregated segments._

I searched through the code, and am not finding any correction of the median for the bias relative to mean.
1. Is it actually done somewhere and I missed?
2. Would you have some references about how such bias correction is done?


Discussion:
I'm not aware of a problem with PSD bias when using median vs mean (but there could be one!). Do you have refs for this?
I googled but didn't find so far. I had no awareness that there needed to be a bias correction. But am interested in learning about it!
I have no idea where this comes from...
it was added here: https://github.com/mne-tools/mne-python/commit/88a2e3dae4b22ca13d088db8b2213745cd9a653c

@hoechenberger authored and @agramfort committed :)
@hoechenberger what did you mean by "corrected for its bias relative to the
mean" ?


> Message ID: ***@***.***>
>

No idea, it's probably just rephrasing what the invoked SciPy function actually does. IIRC it's not simply calculating a median but doing some magic to avoid a certain type of bias. But I'm not on my computer now, so cannot really look. But the solution lies in the SciPy docs & code. 
maybe from here?

> If we now introduce a discontinuity in the signal, by increasing the amplitude of a small portion of the signal by 50, we can see the corruption of the mean average power spectral density, but using a median average better estimates the normal behaviour.

https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.welch.html

(near the end of the page)
https://pydocs.github.io/p/scipy/1.8.0/api/scipy.signal._spectral_py._median_bias

It's this function that's called internally. 
Are you sure? Following the `average` trail leads to https://github.com/mne-tools/mne-python/blob/c12a02000776adb9df18da55ea60c50cbf83827d/mne/time_frequency/psd.py#L20-L21
Well maybe somebody broke things. 🤓

Back when I added this stuff (follow the link to the commit @drammock dug up) we used some SciPy functions that would apply this bias correction. 

Basically the calculated median would get divided by that bias to retrieve the corrected median. 

If this is not the case anymore, we should restore this behavior: I am sure the SciPy folks do this for a reason, and it's not even optional in their functions. They always apply this correction. 
I may have been involved in the `np.nanmedian` because previous approach did not work in some cases (when some welch windows were NaN due to overlap with annotations, I think), so I submitted an issue, which lead to nanmedian. It would be great to have both then: the median bias correction, and ability to ignore nans.
([here is the scipy function BTW](https://github.com/scipy/scipy/blob/main/scipy/signal/_spectral_py.py#L2019))
We could just steal that bias estimation function from SciPy, it's just two lines of code…
we should fix this before the release

> Message ID: ***@***.***>
>


Resolving patch:
diff --git a/mne/time_frequency/psd.py b/mne/time_frequency/psd.py
--- a/mne/time_frequency/psd.py
+++ b/mne/time_frequency/psd.py
@@ -11,6 +11,13 @@
 from .multitaper import psd_array_multitaper
 
 
+# copied from SciPy
+# https://github.com/scipy/scipy/blob/f71e7fad717801c4476312fe1e23f2dfbb4c9d7f/scipy/signal/_spectral_py.py#L2019  # noqa: E501
+def _median_bias(n):
+    ii_2 = 2 * np.arange(1., (n - 1) // 2 + 1)
+    return 1 + np.sum(1. / (ii_2 + 1) - 1. / ii_2)
+
+
 def _decomp_aggregate_mask(epoch, func, average, freq_sl):
     _, _, spect = func(epoch)
     spect = spect[..., freq_sl, :]
@@ -18,7 +25,7 @@ def _decomp_aggregate_mask(epoch, func, average, freq_sl):
     if average == 'mean':
         spect = np.nanmean(spect, axis=-1)
     elif average == 'median':
-        spect = np.nanmedian(spect, axis=-1)
+        spect = np.nanmedian(spect, axis=-1) / _median_bias(spect.shape[-1])
     return spect
 
 


Test patch:
diff --git a/mne/time_frequency/tests/test_psd.py b/mne/time_frequency/tests/test_psd.py
--- a/mne/time_frequency/tests/test_psd.py
+++ b/mne/time_frequency/tests/test_psd.py
@@ -10,6 +10,7 @@
 from mne.time_frequency import (psd_welch, psd_array_welch, psd_multitaper,
                                 psd_array_multitaper)
 from mne.time_frequency.multitaper import _psd_from_mt
+from mne.time_frequency.psd import _median_bias
 
 base_dir = op.join(op.dirname(__file__), '..', '..', 'io', 'tests', 'data')
 raw_fname = op.join(base_dir, 'test_raw.fif')
@@ -243,7 +244,8 @@ def test_psd_welch_average_kwarg(kind):
     assert_allclose(psds_mean, psds_unagg.mean(axis=-1))
 
     # Compare with manual median calculation
-    assert_allclose(psds_median, np.median(psds_unagg, axis=-1))
+    bias = _median_bias(psds_unagg.shape[-1])
+    assert_allclose(psds_median, np.median(psds_unagg, axis=-1) / bias)
 
 
 @pytest.mark.slowtest