Repository: mne-tools/mne-python

Issue:
DOC: order of operation in TFR functions needs to be clarified
issue came up here: https://gitter.im/mne-tools/mne-python?at=5f05de658342f4627400eee5

the tfr_morlet for time frequency decomposition has a parameter average which says:

>    If True average across Epochs.

The docs need to be updated to make it clear what the order of operations is. First averaging then decomposition? Or the other way around? What are the steps?


Discussion:
I'm relatively confident that the order is 
1) decomposition
2) averaging

if you do the following
```
freqs = np.arange(8, 31)
n_cycles = 7 
a = mne.time_frequency.tfr_morlet(epochs, freqs, n_cycles, average=True)
b = mne.time_frequency.tfr_morlet(epochs, freqs, n_cycles, average=False)
```
I'm pretty sure that `a == b.average()`
That makes me wonder why there is an `average` parameter at all --> [mne.Epochs](https://mne.tools/dev/generated/mne.Epochs.html) doesn't have one either? :thinking: 
the average avoids storing all the TFRs in memory as the averaging is done
online.

>

I think there might be a problem in the test

The following line in the test should fail
https://github.com/mne-tools/mne-python/blob/2965cc3644538c97b297f131bbb46ce9f954ea6f/mne/time_frequency/tests/test_tfr.py#L109
but it passes because `assert_array_almost_equal` only checks 6 decimal places and `(Pdb) power.data.max()
2.1955190395488294e-21`

and in fact `assert_array_almost_equal(power.data, power_evoked.data, 22)` fails with a `Max relative difference: 681137.62395224` 
Yeah that test looks broken. We should move to `assert_allclose` and probably just use `rtol` there (unless we expect some near-zero values, in which case a suitable `atol` should be chosen based on what "zero-like" means for the data)
this test makes no sense. You should not expect the TFR of the epochs to be
the same as the evoked.

>

> I'm relatively confident that the order is
>
> 1.decomposition
> 2. averaging

The critical part is where is the magnitude taken. decomp/mag/ave will be different from decomp/ave/mag (with the latter being equivalent to ave/decomp/mag).

> this test makes no sense. You should not expect the TFR of the epochs to be the same as the evoked.

You should expect the TFR averaged across epochs to be the same as the TFR of the evoked if the magnitude is taken at the end (or not at all) for both. Maybe that's what the test was trying to check? And maybe it was wrong about when the average is taken (i.e., before or after the magnitude is taken)? Someone will need to dig into the code to check...
for me code is correct but test is broken.

I agree with @agramfort and think the test was in error. the procedure to compute total power is to decomp/mag/ave. otherwise you are computing evoked total power which isn't (generally) what you want.
I should note that the surrounding `assert_array_almost_equal` are testing meaningful things and currently are trivially passing because of the lack of a `decimal` parameter

Sounds reasonable to me, @bloyl would you be up for fixing the code to use `assert_allclose` with `rtol` only and document the order of transformations in some TFR computing function (and maybe refer other ones to look at the Notes for that one)?
Sure although Docs aren't exactly my strong suit.

@sappelhoff what do you think of
```
    average : bool, default True
        If False return an EpochsTFR containing separate tfrs for each epoch.
        If True return an AverageTFR containing the average of all tfrs across epochs.
```
> @sappelhoff what do you think of

I think that's good but I would add some more detail:

```
    average : bool, default True
        If False return an EpochsTFR containing separate tfrs for each epoch.
        If True return an AverageTFR containing the average of all tfrs across epochs.
        AverageTFR is equivalent to EpochsTFR.average(), but setting average=True will
        be more memory efficient than setting average=False and later calling
        EpochsTFR.average().
```

Resolving patch:
diff --git a/mne/time_frequency/tfr.py b/mne/time_frequency/tfr.py
--- a/mne/time_frequency/tfr.py
+++ b/mne/time_frequency/tfr.py
@@ -688,10 +688,7 @@ def tfr_morlet(inst, freqs, n_cycles, use_fft=False, return_itc=True, decim=1,
         Make sure the wavelet has a mean of zero.
 
         .. versionadded:: 0.13.0
-    average : bool, default True
-        If True average across Epochs.
-
-        .. versionadded:: 0.13.0
+    %(tfr_average)s
     output : str
         Can be "power" (default) or "complex". If "complex", then
         average must be False.
@@ -831,10 +828,7 @@ def tfr_multitaper(inst, freqs, n_cycles, time_bandwidth=4.0,
         .. note:: Decimation may create aliasing artifacts.
     %(n_jobs)s
     %(picks_good_data)s
-    average : bool, default True
-        If True average across Epochs.
-
-        .. versionadded:: 0.13.0
+    %(tfr_average)s
     %(verbose)s
 
     Returns
diff --git a/mne/utils/docs.py b/mne/utils/docs.py
--- a/mne/utils/docs.py
+++ b/mne/utils/docs.py
@@ -1238,6 +1238,21 @@
     orientation etc.) will be shown. Defaults to ``None``.
 """
 
+# TFRs
+docdict['tfr_average'] = """
+average : bool, default True
+    If ``False`` return an `EpochsTFR` containing separate TFRs for each
+    epoch. If ``True`` return an `AverageTFR` containing the average of all
+    TFRs across epochs.
+
+    .. note::
+        Using ``average=True`` is functionally equivalent to using
+        ``average=False`` followed by ``EpochsTFR.average()``, but is
+        more memory efficient.
+
+    .. versionadded:: 0.13.0
+"""
+
 # Finalize
 docdict = unindent_dict(docdict)
 fill_doc = filldoc(docdict, unindent_params=False)


Test patch:
diff --git a/mne/time_frequency/tests/test_tfr.py b/mne/time_frequency/tests/test_tfr.py
--- a/mne/time_frequency/tests/test_tfr.py
+++ b/mne/time_frequency/tests/test_tfr.py
@@ -3,8 +3,7 @@
 import os.path as op
 
 import numpy as np
-from numpy.testing import (assert_array_almost_equal, assert_array_equal,
-                           assert_equal)
+from numpy.testing import (assert_array_equal, assert_equal, assert_allclose)
 import pytest
 import matplotlib.pyplot as plt
 
@@ -81,8 +80,6 @@ def test_time_frequency():
                             use_fft=True, return_itc=True)
     # Now compute evoked
     evoked = epochs.average()
-    power_evoked = tfr_morlet(evoked, freqs, n_cycles, use_fft=True,
-                              return_itc=False)
     pytest.raises(ValueError, tfr_morlet, evoked, freqs, 1., return_itc=True)
     power, itc = tfr_morlet(epochs, freqs=freqs, n_cycles=n_cycles,
                             use_fft=True, return_itc=True)
@@ -103,10 +100,20 @@ def test_time_frequency():
                    return_itc=False, picks=picks, average=False)
     power_picks_avg = epochs_power_picks.average()
     # the actual data arrays here are equivalent, too...
-    assert_array_almost_equal(power.data, power_picks.data)
-    assert_array_almost_equal(power.data, power_picks_avg.data)
-    assert_array_almost_equal(itc.data, itc_picks.data)
-    assert_array_almost_equal(power.data, power_evoked.data)
+    assert_allclose(power.data, power_picks.data)
+    assert_allclose(power.data, power_picks_avg.data)
+    assert_allclose(itc.data, itc_picks.data)
+
+    # test on evoked
+    power_evoked = tfr_morlet(evoked, freqs, n_cycles, use_fft=True,
+                              return_itc=False)
+    # one is squared magnitude of the average (evoked) and
+    # the other is average of the squared magnitudes (epochs PSD)
+    # so values shouldn't match, but shapes should
+    assert_array_equal(power.data.shape, power_evoked.data.shape)
+    pytest.raises(AssertionError, assert_allclose,
+                  power.data, power_evoked.data)
+
     # complex output
     pytest.raises(ValueError, tfr_morlet, epochs, freqs, n_cycles,
                   return_itc=False, average=True, output="complex")
@@ -115,12 +122,13 @@ def test_time_frequency():
     epochs_power_complex = tfr_morlet(epochs, freqs, n_cycles,
                                       output="complex", average=False,
                                       return_itc=False)
-    epochs_power_2 = abs(epochs_power_complex)
-    epochs_power_3 = epochs_power_2.copy()
-    epochs_power_3.data[:] = np.inf  # test that it's actually copied
-    assert_array_almost_equal(epochs_power_2.data, epochs_power_picks.data)
-    power_2 = epochs_power_2.average()
-    assert_array_almost_equal(power_2.data, power.data)
+    epochs_amplitude_2 = abs(epochs_power_complex)
+    epochs_amplitude_3 = epochs_amplitude_2.copy()
+    epochs_amplitude_3.data[:] = np.inf  # test that it's actually copied
+
+    # test that the power computed via `complex` is equivalent to power
+    # computed within the method.
+    assert_allclose(epochs_amplitude_2.data**2, epochs_power_picks.data)
 
     print(itc)  # test repr
     print(itc.ch_names)  # test property
@@ -155,15 +163,15 @@ def test_time_frequency():
     assert itc2.ch_names[1:] == gave.ch_names
     assert gave.nave == 2
     itc2.drop_channels(itc2.info["bads"])
-    assert_array_almost_equal(gave.data, itc2.data)
+    assert_allclose(gave.data, itc2.data)
     itc2.data = np.ones(itc2.data.shape)
     itc.data = np.zeros(itc.data.shape)
     itc2.nave = 2
     itc.nave = 1
     itc.drop_channels([itc.ch_names[0]])
     combined_itc = combine_tfr([itc2, itc])
-    assert_array_almost_equal(combined_itc.data,
-                              np.ones(combined_itc.data.shape) * 2 / 3)
+    assert_allclose(combined_itc.data,
+                    np.ones(combined_itc.data.shape) * 2 / 3)
 
     # more tests
     power, itc = tfr_morlet(epochs, freqs=freqs, n_cycles=2, use_fft=False,
@@ -192,13 +200,10 @@ def test_time_frequency():
     single_power4 = tfr_morlet(epochs, freqs, 2, decim=slice(2, 4),
                                average=False, return_itc=False).data
 
-    assert_array_almost_equal(np.mean(single_power, axis=0), power.data)
-    assert_array_almost_equal(np.mean(single_power2, axis=0),
-                              power.data[:, :, :2])
-    assert_array_almost_equal(np.mean(single_power3, axis=0),
-                              power.data[:, :, 1:3])
-    assert_array_almost_equal(np.mean(single_power4, axis=0),
-                              power.data[:, :, 2:4])
+    assert_allclose(np.mean(single_power, axis=0), power.data)
+    assert_allclose(np.mean(single_power2, axis=0), power.data[:, :, :2])
+    assert_allclose(np.mean(single_power3, axis=0), power.data[:, :, 1:3])
+    assert_allclose(np.mean(single_power4, axis=0), power.data[:, :, 2:4])
 
     power_pick = power.pick_channels(power.ch_names[:10:2])
     assert_equal(len(power_pick.ch_names), len(power.ch_names[:10:2]))
@@ -334,18 +339,18 @@ def test_tfr_multitaper():
                   return_itc=True, average=False)
 
     # test picks argument
-    assert_array_almost_equal(power.data, power_picks.data)
-    assert_array_almost_equal(power.data, power_averaged.data)
-    assert_array_almost_equal(power.times, power_epochs.times)
-    assert_array_almost_equal(power.times, power_averaged.times)
+    assert_allclose(power.data, power_picks.data)
+    assert_allclose(power.data, power_averaged.data)
+    assert_allclose(power.times, power_epochs.times)
+    assert_allclose(power.times, power_averaged.times)
     assert_equal(power.nave, power_averaged.nave)
     assert_equal(power_epochs.data.shape, (3, 2, 7, 200))
-    assert_array_almost_equal(itc.data, itc_picks.data)
+    assert_allclose(itc.data, itc_picks.data)
     # one is squared magnitude of the average (evoked) and
     # the other is average of the squared magnitudes (epochs PSD)
     # so values shouldn't match, but shapes should
     assert_array_equal(power.data.shape, power_evoked.data.shape)
-    pytest.raises(AssertionError, assert_array_almost_equal,
+    pytest.raises(AssertionError, assert_allclose,
                   power.data, power_evoked.data)
 
     tmax = t[np.argmax(itc.data[0, freqs == 50, :])]