Repository: mne-tools/mne-python

Issue:
BUG: Fix bugs with corrmap computation
Pretty sure from going back to the original paper that our `corrmap` code has been *EDIT: a little bit incorrect* ever since it was added in 0.10 in #2104 (which took over #1985, which took over #1795 -- the bug is present in all). We used the mean of some weirdly transformed maps  instead of just the mean of rms-normalized maps like noted in the paper. Our code in `main` did something like a z-score but not quite, even when fixing the incorrect usage of `std` to `mean`.

Fortunately I don't think it makes a big difference -- at least in our tutorial and tests the `newtarget` in this PR and the `newtarget` on `main` have a correlation coefficient > 0.99. But it should be better to use the correct / reference implementation I think.

Looking at the paper, one difference still remains where we use the median correlation (across ICs) rather than using a mean with a Fisher z-transformation, but I think that part's okay to stay as is -- especially since some of our correlations can be near `1.` which will be a problem when subjected to a Fisher Z transformation. So I think the existing use of `median` is a justified departure for us.

Closes #11858


Discussion:

Resolving patch:
diff --git a/mne/preprocessing/ica.py b/mne/preprocessing/ica.py
--- a/mne/preprocessing/ica.py
+++ b/mne/preprocessing/ica.py
@@ -3153,38 +3153,57 @@ def _band_pass_filter(inst, sources, target, l_freq, h_freq, verbose=None):
 
 def _find_max_corrs(all_maps, target, threshold):
     """Compute correlations between template and target components."""
-    all_corrs = [compute_corr(target, subj.T) for subj in all_maps]
+    # Following Fig.2 from:
+    # https://www.sciencedirect.com/science/article/abs/pii/S1388245709002338
+
+    # > ... inverse weights (i.e., IC maps) from a selected template IC are
+    # > correlated with all ICs from all datasets ...
+    all_corrs = [compute_corr(target, subj_maps.T) for subj_maps in all_maps]
     abs_corrs = [np.abs(a) for a in all_corrs]
     corr_polarities = [np.sign(a) for a in all_corrs]
+    del all_corrs
 
+    # > selection of X ICs from each dataset with highest absolute
+    # > correlation >= TH
+    #
+    # subj_idxs is a list of indices for each subject that exceeded the threshold:
     if threshold <= 1:
-        max_corrs = [list(np.nonzero(s_corr > threshold)[0]) for s_corr in abs_corrs]
+        subj_idxs = [list(np.nonzero(s_corr > threshold)[0]) for s_corr in abs_corrs]
     else:
-        max_corrs = [
+        subj_idxs = [
             list(_find_outliers(s_corr, threshold=threshold)) for s_corr in abs_corrs
         ]
 
-    am = [l_[i] for l_, i_s in zip(abs_corrs, max_corrs) for i in i_s]
-    median_corr_with_target = np.median(am) if len(am) > 0 else 0
-
-    polarities = [l_[i] for l_, i_s in zip(corr_polarities, max_corrs) for i in i_s]
-
-    maxmaps = [l_[i] for l_, i_s in zip(all_maps, max_corrs) for i in i_s]
-
-    if len(maxmaps) == 0:
+    # > The mean correlation of a resulting cluster is then computed via
+    # > Fisher’s z transform, to account for the non-normal distribution of
+    # > correlation values.
+    #
+    # Here we just use the median rather than the (transformed-back) mean of
+    # the (Fisher z-transformed) correlations:
+    am = np.concatenate(
+        [abs_corr[subj_idx] for abs_corr, subj_idx in zip(abs_corrs, subj_idxs)]
+    )
+    if len(am) == 0:
         return [], 0, 0, []
-    newtarget = np.zeros(maxmaps[0].size)
-    std_of_maps = np.std(np.asarray(maxmaps))
-    mean_of_maps = np.std(np.asarray(maxmaps))
-    for maxmap, polarity in zip(maxmaps, polarities):
-        newtarget += (maxmap / std_of_maps - mean_of_maps) * polarity
+    median_corr_with_target = np.median(am)
 
-    newtarget /= len(maxmaps)
-    newtarget *= std_of_maps
+    # > Next, an average cluster map is calculated, after inversion of those
+    # > ICs showing a negative correlation (sign ambiguity problem) and root
+    # > mean square (RMS) normalization of each individual IC.
+    #
+    # Which is this (rms=Frobenius norm=np.linalg.norm):
+    newtarget = sum(
+        subj_maps[idx] * (pols[idx] / np.linalg.norm(subj_maps[idx]))
+        for subj_maps, pols, subj_idx in zip(all_maps, corr_polarities, subj_idxs)
+        for idx in subj_idx
+    )
+    newtarget /= len(am)
 
+    # And we also compute the similarity between this new map and our original
+    # target map
     sim_i_o = np.abs(np.corrcoef(target, newtarget)[1, 0])
 
-    return newtarget, median_corr_with_target, sim_i_o, max_corrs
+    return newtarget, median_corr_with_target, sim_i_o, subj_idxs
 
 
 @verbose
diff --git a/tutorials/preprocessing/40_artifact_correction_ica.py b/tutorials/preprocessing/40_artifact_correction_ica.py
--- a/tutorials/preprocessing/40_artifact_correction_ica.py
+++ b/tutorials/preprocessing/40_artifact_correction_ica.py
@@ -63,7 +63,6 @@
 # is usually possible to separate the sources using ICA, and then re-construct
 # the sensor signals after excluding the sources that are unwanted.
 #
-#
 # ICA in MNE-Python
 # ~~~~~~~~~~~~~~~~~
 #


Test patch:
diff --git a/mne/preprocessing/tests/test_ica.py b/mne/preprocessing/tests/test_ica.py
--- a/mne/preprocessing/tests/test_ica.py
+++ b/mne/preprocessing/tests/test_ica.py
@@ -689,7 +689,10 @@ def test_ica_additional(method, tmp_path, short_raw_epochs):
             plot=False,
             show=False,
         )
-    corrmap([ica, ica2], (0, 0), threshold=0.5, plot=False, show=False)
+    with catch_logging(True) as log:
+        corrmap([ica, ica2], (0, 0), threshold=0.5, plot=False, show=False)
+    log = log.getvalue()
+    assert "Median correlation with constructed map: 1.0" in log
     assert ica.labels_["blinks"] == ica2.labels_["blinks"]
     assert 0 in ica.labels_["blinks"]
     # test retrieval of component maps as arrays