Repository: mne-tools/mne-python

Issue:
Bug in envelope_correlation
Hi everyone,

I think there is a bug in the function `envelope_correlation()` in the case of `orthogonalize="pairwise"`.
In the original article Hipp et al, 2012 they describe the correlation as between the orthogonalized Y on X with X.
So cor(y_ort, x)
But in the code in line 119:
`corr[li] = np.dot(label_data_orth, data_mag_nomean[li])`
The code seems to calculate the orthogonalized Y on all possible X with Y.
Say we have 5 channels
Then it should be:
- C1_ort on C2 with C2
- C1_ort on C3 with C3
- C1_orth on C4 with C4
- etc
But what is being calculated in envelope_correlation() is C1_ort on C2 with C1.

I hope this is not too confusing, but I tried implementing a simple example using for loops and some of the source code to make sure I knew what timeseries are being orthogonalized to which one.

The code:
```
from mne.connectivity import envelope_correlation
from mne.filter import next_fast_len
import numpy as np
from scipy.stats import pearsonr
from scipy.signal import hilbert

n_epochs = 1
n_channels = 5
n_times = 100
data = np.random.rand(n_epochs, n_channels, n_times)
n_fft = next_fast_len(n_times)

data_hil = hilbert(data, N=n_fft, axis=-1)[..., :n_times]

corr = np.ones((n_epochs, n_channels, n_channels))

for epo_idx in range(data_hil.shape[0]):
    epoch_data = data_hil[epo_idx]
    data_mag = np.abs(epoch_data)
    data_conj_scaled = epoch_data.conj()
    data_conj_scaled /= data_mag
    for idx1 in range(data_hil.shape[1]):
        for idx2 in range(data_hil.shape[1]):
            # Get timeseries for each pair
            x = epoch_data[idx1]
            y = epoch_data[idx1]
            x_mag = np.abs(x)
            x_conj_scaled = epoch_data.conj()
            x_conj_scaled /= data_mag
            # Calculate orthogonalization
            y_orth_x = (y*x_conj_scaled[idx1]).imag
            # Estimate correlation
            cor = pearsonr(x_mag,y_orth_x)[0]
            corr[epo_idx,idx1,idx2] = cor

# Make it symmetric (it isn't at this point)
corr = np.abs(corr)
corr = (corr.T + corr) / 2.

ec = envelope_correlation(data_hil, combine=None, orthogonalize="pairwise")

print(np.allclose(corr, ec))
```

In addition to this I was also very confused with line 113 and 114:
```
# compute variances using linalg.norm (square, sum, sqrt) since mean=0
data_mag_std = np.linalg.norm(data_mag_nomean, axis=-1)
```
I tried using data_mag_std = np.std(data_mag, axis=-1) and got different results.
According to the documentation in numpy for np.std the operations should be (square, mean, sqrt) when mean = 0 and not sum. Using the sum you have to divide with N (and also sqrt afterwards)
```
std1 = np.std(data_mag, axis=-1)
std2 = np.linalg.norm(data_mag-np.mean(data_mag,axis=-1, keepdims=True),axis=-1)
print(np.allclose(std1,std2/np.sqrt(n_times)))
```

Fixing the std calculation and changing lines 119 and 120 to:
```
corr[li] = np.mean(label_data_orth*data_mag_nomean, axis=-1)
corr[li] /= data_mag_std
```

Seemed to fix the problem for me, but I didn't use the exact same implementation for my own data so it should probably be double-checked.


Discussion:
Good questions!

> corr[li] = np.dot(label_data_orth, data_mag_nomean[li])

I guess what you're suggesting is that our code, which is currently this in the `orthogonalize='pairwise'` case:
```
        for li, label_data in enumerate(epoch_data):
            label_data_orth = (label_data * data_conj_scaled).imag
            ...
            corr[li] = np.dot(label_data_orth, data_mag_nomean[li])
```
Needs to be:
```
        for li, label_data_conj_scaled in enumerate(data_conj_scaled):
            label_data_orth = (label_data_conj_scaled * epoch_data).imag
            ...
            corr[li] = np.dot(label_data_orth, data_mag_nomean[li])
```
? I'd have to go back to the papers to understand if we do it in the right order, I just tried to adapt what @SherazKhan and @dengemann wrote. Maybe they have some insight...

> I tried using data_mag_std = np.std(data_mag, axis=-1) and got different results.

Indeed! Both `data_mag_std` and `label_data_orth_std` will be off by a factor of `1 / np.sqrt(data_mag.shape[-1])` related to the number of time points, as you say. But later these get used as:
```
            corr[li] = np.dot(label_data_orth, data_mag_nomean[li])
            corr[li] /= data_mag_std[li]
            corr[li] /= label_data_orth_std
```
You can see the `np.dot` as really being a `np.sum` operation, which differs by a `1/n` term to the `np.mean` that you prefer. Since the numerator is missing a `1/n`, and each of the two denominator terms are missing a `1/np.sqrt(n)`, they cancel and can be omitted from the computation.

It's nice to figure things out with small snippets so:
```
>>> import numpy as np
>>> import mne
>>> x = np.random.RandomState(0).randn(3, 1000).astype(np.complex128)  # make MNE see it as an envelope
>>> n = np.corrcoef(np.abs(x))
>>> print(n)
array([[ 1.00000000e+00, -8.68807032e-04, -4.12336402e-02],
       [-8.68807032e-04,  1.00000000e+00, -4.07663996e-02],
       [-4.12336402e-02, -4.07663996e-02,  1.00000000e+00]])
>>> m = mne.connectivity.envelope_correlation(x[np.newaxis], orthogonalize=False)
>>> print(m)
array([[ 1.00000000e+00, -8.68807032e-04, -4.12336402e-02],
       [-8.68807032e-04,  1.00000000e+00, -4.07663996e-02],
       [-4.12336402e-02, -4.07663996e-02,  1.00000000e+00]])
>>> assert np.allclose(m, n)
```
In other words, they agree at least when `orthogonalize=False` I think.
>> C1_ort on C2 with C2

> ? I'd have to go back to the papers to understand if we do it in the right order,

Note one problem with your code snippet above is `idx2` is not used in the selection of any components, only `idx1` twice... maybe if you fix this bug it'll help? Without it I cannot come to unterstand what is "C1_ort on C2" in your code because the operation is always "CN_ort on CN" I think... can you fix this part and see if things make any more sense?

FWIW I tried making the change I suggested above and it at least totally breaks our `envelope_correlation` code. On `master` removing the `.crop(0, 60)` (to make things better/clearer) we get:

![Screenshot from 2020-12-11 12-53-53](https://user-images.githubusercontent.com/2365790/101941069-769c3480-3bb5-11eb-9441-9308ff6bc13c.png)

![Screenshot from 2020-12-11 12-53-59](https://user-images.githubusercontent.com/2365790/101941075-7865f800-3bb5-11eb-82d8-f8283efed8a1.png)

And with the change I mention above we get something pretty similar but not identical:

![Screenshot from 2020-12-11 13-32-41](https://user-images.githubusercontent.com/2365790/101941096-8287f680-3bb5-11eb-89fd-0177edbf5ab7.png)

![Screenshot from 2020-12-11 13-32-47](https://user-images.githubusercontent.com/2365790/101941102-8582e700-3bb5-11eb-9062-66fb65c4a84c.png)

@larsoner 
Thanks for your quick reply and the explanation about why linalg.norm are used and how np.dot cancels it out.

I implemented my own version which used different variable names and also a square and log-transform as in the original paper, so when I wanted to make an example that fit the variable names in mne-python I was a bit too careless.

I have now fixed my bug with idx2.

The change I am suggesting is not in line 113:
```
for li, label_data in enumerate(epoch_data):
       label_data_orth = (label_data * data_conj_scaled).imag
```
Which is fine.

But the line 119:
`corr[li] = np.dot(label_data_orth, data_mag_nomean[li])`
Here the correlation between all the orthogonalized signals are calculated with respect to the signal at index _li_
Using my previous nomenclature:
e.g. if _li_ = 2 then label_data = C2
When we apply:
`label_data_orth = (label_data * data_conj_scaled).imag`
We get the orthogonalized C2 on all other channels (e.g. 5). So first row is C2_ort_C1, second row is C2_ort_C2, third row is C2_ort_C3 etc.
What we want is the correlation between each pairwise orthogonalized signal, e.g.
correlation of C2_ort_C1 with C1 and C2_ort_C4 with C4.

The current line 119 will not do this, but instead for example calculate the correlations
C2_ort_C1 with C2, C2_ort_C4 with C2 and C2_ort_C5 with C2 (it is always channel _li_)

So from lines 119 to 120 should be changed to:
```
corr[li] = np.mean(label_data_orth*data_mag_nomean, axis=-1)
corr[li] /= data_mag_std
```
I.e. without the _li_ index to perform element wise multiplication to get the desired results.
Notice that in this case dividing with data_mag_std is wrong since I have omitted the np.dot product. So my suggested change is just an example and not how it should be finally implemented.

Please see the following code example:
```
from mne.connectivity import envelope_correlation
from mne.filter import next_fast_len
import numpy as np
from scipy.stats import pearsonr
from scipy.signal import hilbert

# Generate data
n_epochs = 1
n_channels = 5
n_times = 100
data = np.random.rand(n_epochs, n_channels, n_times)
n_fft = next_fast_len(n_times)

data_hil = hilbert(data, N=n_fft, axis=-1)[..., :n_times]

# Calculate using vanilla for loops
corr0 = np.ones((n_epochs, n_channels, n_channels))
for epo_idx in range(data_hil.shape[0]):
    epoch_data = data_hil[epo_idx]
    for idx1 in range(data_hil.shape[1]):
        for idx2 in range(data_hil.shape[1]):
            # Get timeseries for each pair
            y = epoch_data[idx1]
            x = epoch_data[idx2]
            x_mag = np.abs(x)
            x_conj_scaled = x.conj()
            x_conj_scaled /= x_mag
            # Calculate orthogonalization
            y_orth_x = (y*x_conj_scaled).imag
            # Estimate correlation
            cor = pearsonr(x_mag,y_orth_x)[0]
            corr0[epo_idx,idx1,idx2] = cor

    # Make it symmetric (it isn't at this point)
    corr0[epo_idx] = np.abs(corr0[epo_idx])
    corr0[epo_idx] = (corr0[epo_idx].T + corr0[epo_idx]) / 2.

# Calculate using MNE-python function
ec = envelope_correlation(data_hil, combine=None, orthogonalize="pairwise")

print(np.allclose(corr0, ec)) # False

# Example of a fix
corr = np.ones((n_epochs, n_channels, n_channels))
for epo_idx in range(data_hil.shape[0]):
    epoch_data = data_hil[epo_idx]
    data_mag = np.abs(epoch_data)
    # subtract means
    data_mag_nomean = data_mag - np.mean(data_mag, axis=-1, keepdims=True)
    # compute variances using linalg.norm (square, sum, sqrt) since mean=0
    data_mag_std = np.std(data_mag, axis=-1) # notice I don't use np.linalg.norm
    data_conj_scaled = epoch_data.conj()
    data_conj_scaled /= data_mag
    for li, label_data in enumerate(epoch_data):
        label_data_orth = (label_data * data_conj_scaled).imag
        label_data_orth_nomean = label_data_orth - \
                                        np.mean(label_data_orth, axis=-1,
                                        keepdims=True)
        label_data_orth_std = np.std(label_data_orth, axis=-1) # notice I don't use np.linalg.norm
        # Pearson's correlation
        corr[epo_idx,li] = np.mean(data_mag_nomean*label_data_orth_nomean, axis=-1) # this is the major change
        corr[epo_idx,li] /= data_mag_std
        corr[epo_idx,li] /= label_data_orth_std

    # Make it symmetric (it isn't at this point)
    corr[epo_idx] = np.abs(corr[epo_idx])
    corr[epo_idx] = (corr[epo_idx].T + corr[epo_idx]) / 2.

print(np.allclose(corr0, corr)) # True
```

I hope this is a bit more clear
One thing I also found curious was that the diagonal elements were non-zero.
There shouldn't be any orthogonal component on itself and hence I would expect the correlation to be 0.
But it turns out that there are some numerical errors when doing the imaginary computation and hence rounding or thresholding could remove this problem. E.g. here I took the min of data_mag and multiplied with an arbitrary small number of 1e-6 as threshold

```
corr = np.ones((n_epochs, n_channels, n_channels))
for epo_idx in range(data_hil.shape[0]):
    epoch_data = data_hil[epo_idx]
    data_mag = np.abs(epoch_data)
    # subtract means
    data_mag_nomean = data_mag - np.mean(data_mag, axis=-1, keepdims=True)
    # compute variances using linalg.norm (square, sum, sqrt) since mean=0
    data_mag_std = np.std(data_mag, axis=-1)
    data_conj_scaled = epoch_data.conj()
    data_conj_scaled /= data_mag
    for li, label_data in enumerate(epoch_data):
        label_data_orth = (label_data * data_conj_scaled).imag
        label_data_orth[np.abs(label_data_orth)<np.min(data_mag)*1e-6] = 0 # rounds of numerical errors
        label_data_orth_nomean = label_data_orth - \
                                        np.mean(label_data_orth, axis=-1,
                                        keepdims=True)
        label_data_orth_std = np.std(label_data_orth, axis=-1)
        label_data_orth_std[label_data_orth_std == 0] = 1 # ensures no division of 0 when the signal is orthogonalized to itself
        # Pearson's correlation
        corr[epo_idx,li] = np.mean(data_mag_nomean*label_data_orth_nomean, axis=-1)
        corr[epo_idx,li] /= data_mag_std
        corr[epo_idx,li] /= label_data_orth_std

    # Make it symmetric (it isn't at this point)
    corr[epo_idx] = np.abs(corr[epo_idx])
    corr[epo_idx] = (corr[epo_idx].T + corr[epo_idx]) / 2.
```

Resolving patch:
diff --git a/examples/connectivity/plot_mne_inverse_envelope_correlation.py b/examples/connectivity/plot_mne_inverse_envelope_correlation.py
--- a/examples/connectivity/plot_mne_inverse_envelope_correlation.py
+++ b/examples/connectivity/plot_mne_inverse_envelope_correlation.py
@@ -9,6 +9,7 @@
 :footcite:`HippEtAl2012,KhanEtAl2018` in source
 space using resting state CTF data.
 """
+
 # Authors: Eric Larson <larson.eric.d@gmail.com>
 #          Sheraz Khan <sheraz@khansheraz.com>
 #          Denis Engemann <denis.engemann@gmail.com>
diff --git a/mne/connectivity/envelope.py b/mne/connectivity/envelope.py
--- a/mne/connectivity/envelope.py
+++ b/mne/connectivity/envelope.py
@@ -13,7 +13,7 @@
 
 @verbose
 def envelope_correlation(data, combine='mean', orthogonalize="pairwise",
-                         verbose=None):
+                         log=False, absolute=True, verbose=None):
     """Compute the envelope correlation.
 
     Parameters
@@ -40,6 +40,18 @@ def envelope_correlation(data, combine='mean', orthogonalize="pairwise",
         absolute values.
 
         .. versionadded:: 0.19
+    log : bool
+        If True (default False), square and take the log before orthonalizing
+        envelopes or computing correlations.
+
+        .. versionadded:: 0.22
+    absolute : bool
+        If True (default), then take the absolute value of correlation
+        coefficients before making each epoch's correlation matrix
+        symmetric (and thus before combining matrices across epochs).
+        Only used when ``orthogonalize=True``.
+
+        .. versionadded:: 0.22
     %(verbose)s
 
     Returns
@@ -54,6 +66,10 @@ def envelope_correlation(data, combine='mean', orthogonalize="pairwise",
     This function computes the power envelope correlation between
     orthogonalized signals [1]_ [2]_.
 
+    .. versionchanged:: 0.22
+      Computations fixed for ``orthogonalize=True`` and diagonal entries are
+      set explicitly to zero.
+
     References
     ----------
     .. [1] Hipp JF, Hawellek DJ, Corbetta M, Siegel M, Engel AK (2012)
@@ -99,6 +115,9 @@ def envelope_correlation(data, combine='mean', orthogonalize="pairwise",
         data_mag = np.abs(epoch_data)
         data_conj_scaled = epoch_data.conj()
         data_conj_scaled /= data_mag
+        if log:
+            data_mag *= data_mag
+            np.log(data_mag, out=data_mag)
         # subtract means
         data_mag_nomean = data_mag - np.mean(data_mag, axis=-1, keepdims=True)
         # compute variances using linalg.norm (square, sum, sqrt) since mean=0
@@ -107,21 +126,29 @@ def envelope_correlation(data, combine='mean', orthogonalize="pairwise",
         corr = np.empty((n_nodes, n_nodes))
         for li, label_data in enumerate(epoch_data):
             if orthogonalize is False:  # the new code
-                label_data_orth = data_mag
-                label_data_orth_std = data_mag_std
+                label_data_orth = data_mag[li]
+                label_data_orth_std = data_mag_std[li]
             else:
                 label_data_orth = (label_data * data_conj_scaled).imag
+                np.abs(label_data_orth, out=label_data_orth)
+                # protect against invalid value -- this will be zero
+                # after (log and) mean subtraction
+                label_data_orth[li] = 1.
+                if log:
+                    label_data_orth *= label_data_orth
+                    np.log(label_data_orth, out=label_data_orth)
                 label_data_orth -= np.mean(label_data_orth, axis=-1,
                                            keepdims=True)
                 label_data_orth_std = np.linalg.norm(label_data_orth, axis=-1)
                 label_data_orth_std[label_data_orth_std == 0] = 1
             # correlation is dot product divided by variances
-            corr[li] = np.dot(label_data_orth, data_mag_nomean[li])
-            corr[li] /= data_mag_std[li]
+            corr[li] = np.sum(label_data_orth * data_mag_nomean, axis=1)
+            corr[li] /= data_mag_std
             corr[li] /= label_data_orth_std
         if orthogonalize is not False:
             # Make it symmetric (it isn't at this point)
-            corr = np.abs(corr)
+            if absolute:
+                corr = np.abs(corr)
             corr = (corr.T + corr) / 2.
         corrs.append(corr)
         del corr


Test patch:
diff --git a/mne/connectivity/tests/test_envelope.py b/mne/connectivity/tests/test_envelope.py
--- a/mne/connectivity/tests/test_envelope.py
+++ b/mne/connectivity/tests/test_envelope.py
@@ -15,19 +15,23 @@
 def _compute_corrs_orig(data):
     # This is the version of the code by Sheraz and Denis.
     # For this version (epochs, labels, time) must be -> (labels, time, epochs)
-    data = np.transpose(data, (1, 2, 0))
-    corr_mats = np.empty((data.shape[0], data.shape[0], data.shape[2]))
-    for index, label_data in enumerate(data):
-        label_data_orth = np.imag(label_data * (data.conj() / np.abs(data)))
-        label_data_orig = np.abs(label_data)
-        label_data_cont = np.transpose(
-            np.dstack((label_data_orig, np.transpose(label_data_orth,
-                                                     (1, 2, 0)))), (1, 2, 0))
-        corr_mats[index] = np.array([np.corrcoef(dat)
-                                     for dat in label_data_cont])[:, 0, 1:].T
-    corr_mats = np.transpose(corr_mats, (2, 0, 1))
-    corr = np.mean(np.array([(np.abs(corr_mat) + np.abs(corr_mat).T) / 2.
-                             for corr_mat in corr_mats]), axis=0)
+    n_epochs, n_labels, _ = data.shape
+    corr = np.zeros((n_labels, n_labels))
+    for epoch_data in data:
+        for ii in range(n_labels):
+            for jj in range(n_labels):
+                # Get timeseries for each pair
+                x, y = epoch_data[ii], epoch_data[jj]
+                x_mag = np.abs(x)
+                x_conj_scaled = x.conj()
+                x_conj_scaled /= x_mag
+                # Calculate orthogonalization
+                y_orth_x = (y * x_conj_scaled).imag
+                y_orth_x_mag = np.abs(y_orth_x)
+                # Estimate correlation
+                corr[ii, jj] += np.abs(np.corrcoef(x_mag, y_orth_x_mag)[0, 1])
+    corr = (corr + corr.T) / (2. * n_epochs)
+    corr.flat[::n_labels + 1] = 0.
     return corr
 
 
@@ -37,7 +41,7 @@ def test_envelope_correlation():
     data = rng.randn(2, 4, 64)
     data_hilbert = hilbert(data, axis=-1)
     corr_orig = _compute_corrs_orig(data_hilbert)
-    assert (0 < corr_orig).all()
+    assert (0 <= corr_orig).all()
     assert (corr_orig < 1).all()
     # using complex data
     corr = envelope_correlation(data_hilbert)
@@ -72,3 +76,25 @@ def test_envelope_correlation():
     assert_allclose(np.diag(corr_plain_mean), 1)
     np_corr = np.array([np.corrcoef(np.abs(x)) for x in data_hilbert])
     assert_allclose(corr_plain, np_corr)
+
+    # check against FieldTrip, which uses the square-log-norm version
+    # from scipy.io import savemat
+    # savemat('data.mat', dict(data_hilbert=data_hilbert))
+    # matlab
+    # load data
+    # ft_connectivity_powcorr_ortho(reshape(data_hilbert(1,:,:), [4, 64]))
+    # ft_connectivity_powcorr_ortho(reshape(data_hilbert(2,:,:), [4, 64]))
+    ft_vals = np.array([
+        [[np.nan, 0.196734553900236, 0.063173148355451, -0.242638384630448],
+         [0.196734553900236, np.nan, 0.041799775495150, -0.088205187548542],
+         [0.063173148355451, 0.041799775495150, np.nan, 0.090331428512317],
+         [-0.242638384630448, -0.088205187548542, 0.090331428512317, np.nan]],
+        [[np.nan, -0.013270857462890, 0.185200598081295, 0.140284351572544],
+         [-0.013270857462890, np.nan, 0.150981508043722, -0.000671809276372],
+         [0.185200598081295, 0.150981508043722, np.nan, 0.137460244313337],
+         [0.140284351572544, -0.000671809276372, 0.137460244313337, np.nan]],
+    ], float)
+    ft_vals[np.isnan(ft_vals)] = 0
+    corr_log = envelope_correlation(
+        data, combine=None, log=True, absolute=False)
+    assert_allclose(corr_log, ft_vals)