Repository: mne-tools/mne-python

Issue:
Doubts!! 'mne.stats.spatio_temporal_cluster_1samp_test' function documentation Error?
I am doing non-parametric cluster-level 1 sample testing. I am using the standard function from mne, named 'spatio_temporal_cluster_1samp_test' function for extracting information from source level MEG data.

While using this function from mne package, I am facing some doubts. As per documentation in the mne, this  function will return H0 parameter which is  the maximum cluster level statistics under permutation. H0 should have same array of shape with n_permutations(no. of permutation defined in the function as input).

Here comes the trouble, with the data I have, my H0 parameter doesn't have the same shape of n_permutations. H0 size is always less the n_permutations.

I also tried out the default mne's example : Permutation t-test on source data with spatio-temporal clustering test. (link:https://martinos.org/mne/stable/auto_tutorials/plot_stats_cluster_spatio_temporal.html)

Here too, I have seen the sample effect that H0 size is not same with size of n_permutations.

Opinion:

H0 size is not same as n_permutations number, Is it because some the maximum clusters sum (statistical weight) are repeated while shuffling process and function doesn't count the repetitive ones ? Correct me if I am wrong. If this is the case then mne documentation is little misleading I feel. It should be corrected.


Or, Did you use some internal threshold to omit some of the small weighted clusters from maximum cluster level statistics(H0) to make the calculation faster?

It would be nice if you can clarify on this issue. Because, I feel the threshold value and size of H0 play a crucial role in terms  of calculating p_values for all initial clusters.



Kind Regards,
Diptyajit Das


Discussion:
len(H0) should depend of n_samples as you cannot have for example 1000
permutations
if you only have 5 samples. Yet for K samples one should have 2**K
possible permutations
for a one sample test which is not what we have. You seem to get always less.

@Eric89GXL did you write this code?

Also interesting fact, after using this one sample test function, 
Even though you keep your sample no. fixed(say 10 samples), if you change the threshold value then len(H0) differs which is expected. If you provide higher threshold then cluster mass weight for H0 values becomes smaller and len(H0) increases. if You provide low threshold, cluster mass weight for H0 values increases and len(H0) decreases.So  Len(H0) becomes independent of len(n_permutations). This is my experience with MEG data that I am headlining. So it seems like len(H0) mainly depends on threshold value rather than on the permutation number and sample no. at lest for (one sample test function).  
The docs should probably say that the *maximum* shape is `(n_permutations,)`. If no clusters are found in some of the permutations, it can be smaller.
@Diptyajitd7 can you open a PR to improve the doc?
@Eric89GXL I have created a PR. 
Can you please en-light the fact about 'no clusters are found under some of the permutations' for me?   
thanks! 
Actually I realized that when no clusters are found, it should still contribute to H0 -- it should contribute a cluster of size 0. I'll make a fix for it.
That actually makes a lot more sense. As then it will also change the p_values of all the initial clusters.
Don't get me wrong, I still somehow don't get the idea of not finding any cluster under some permutations. How can it is possible to not find any clusters at some permutation when you are summing up the statistical values of adjacent time points after applying a certain threshold? I guess the cluster size can be really small but not equal to zero. Or is it, the cluster can't be simply formed with a single time point? then you have a cluster size of 0?   
    
> summing up the statistical values of adjacent time points after applying a certain threshold?

Other order -- you apply a threshold, then sum statistical values (that exceed that threshold). So if no pass the threshold, you can have no clusters.
I see..yes I agree with you. Thanks a lot for your time.  
Actually in testing I can only seem to get `H0.shape == (0,)` or `(n_permutations,)` do you have an example that creates something in between? Here is what I have been testing with:
```
rng = np.random.RandomState(0)
data = rng.rand(20, 100, 1) - 0.5
t, clust, p, h0 = mne.stats.spatio_temporal_cluster_1samp_test(
        data, threshold=100, n_permutations=1024, seed=0)
```
did you try to run the example on the mne? here is the link:
 https://martinos.org/mne/stable/auto_tutorials/plot_stats_cluster_spatio_temporal.html 
if you use this example and meg sample data,  you should get H0.shape  is equal to 63, it doesn't matter if you use n_permutation as 1024 (default) or 2000 or 5000. H0 size remains same.

In case if you want some other data for testing, I am willing to share of my experimental data with you. But for that I have to upload the data to a server. Let me know you want that. 
Ahh okay, it's a problem with the 2-sample tests -- I can confirm that and will look. 

Your initial report was about the 1-sample tests, and there I only ever see `0` or `n_permutations` lengths --
 can you confirm (or provide a counterexample)?
I will try to put an example soon. 
Oh actually that is a 1-sample test. 63 is the correct number there -- there are only 7 "subjects", so it performs an exact test with `2 ** 6 - 1 == 63` flips.
so H0 size depends on the sample numbers((2**subjects -1) in this case?) not only the n_permutation size that you provide? Wait i am going to put up some counter example then it will be more clear to you
> so H0 size depends on the sample numbers((2**subjects -1) in this case?) not only the n_permutation size that you provide

In the 1-sample test case, yes, because we can do an exact test. Basically it's `min(n_permutations, 2 ** (n_subjects - 1) - 1`
I see..so for one sample t- test, if (2 ** n_subjects -1) -1< n_permutations, then H0.size == (2 ** n_subjects -1) -1. This is valid because of the small sample size? 
It's valid in the sense that you have now done an exact test (done all possible sign-flips), so there is no reason to do any others.
Ok.. whats your take on 2-sample tests, as you mentioned in earlier comment that there could some issue. I didn't find anything though. 
Those appear to be fine. I'll open a PR to document this better, though.
Sounds great! Thanks a lot again. cheers! 

Resolving patch:
diff --git a/mne/stats/cluster_level.py b/mne/stats/cluster_level.py
--- a/mne/stats/cluster_level.py
+++ b/mne/stats/cluster_level.py
@@ -16,7 +16,8 @@
 
 from .parametric import f_oneway
 from ..parallel import parallel_func, check_n_jobs
-from ..utils import split_list, logger, verbose, ProgressBar, warn, _pl
+from ..utils import (split_list, logger, verbose, ProgressBar, warn, _pl,
+                     check_random_state)
 from ..source_estimate import SourceEstimate
 
 
@@ -488,13 +489,12 @@ def _pval_from_histogram(T, H0, tail):
 
     # from pct to fraction
     if tail == -1:  # up tail
-        pval = np.array([np.sum(H0 <= t) for t in T])
+        pval = np.array([np.mean(H0 <= t) for t in T])
     elif tail == 1:  # low tail
-        pval = np.array([np.sum(H0 >= t) for t in T])
+        pval = np.array([np.mean(H0 >= t) for t in T])
     else:  # both tails
-        pval = np.array([np.sum(abs(H0) >= abs(t)) for t in T])
+        pval = np.array([np.mean(abs(H0) >= abs(t)) for t in T])
 
-    pval = (pval + 1.0) / (H0.size + 1.0)  # the init data is one resampling
     return pval
 
 
@@ -514,7 +514,7 @@ def _setup_connectivity(connectivity, n_vertices, n_times):
 
 
 def _do_permutations(X_full, slices, threshold, tail, connectivity, stat_fun,
-                     max_step, include, partitions, t_power, seeds,
+                     max_step, include, partitions, t_power, orders,
                      sample_shape, buffer_size, progress_bar):
     n_samp, n_vars = X_full.shape
 
@@ -522,23 +522,21 @@ def _do_permutations(X_full, slices, threshold, tail, connectivity, stat_fun,
         buffer_size = None  # don't use buffer for few variables
 
     # allocate space for output
-    max_cluster_sums = np.empty(len(seeds), dtype=np.double)
+    max_cluster_sums = np.empty(len(orders), dtype=np.double)
 
     if buffer_size is not None:
         # allocate buffer, so we don't need to allocate memory during loop
         X_buffer = [np.empty((len(X_full[s]), buffer_size), dtype=X_full.dtype)
                     for s in slices]
 
-    for seed_idx, seed in enumerate(seeds):
+    for seed_idx, order in enumerate(orders):
         if progress_bar is not None:
             if (not (seed_idx + 1) % 32) or (seed_idx == 0):
                 progress_bar.update(seed_idx + 1)
 
         # shuffle sample indices
-        rng = np.random.RandomState(seed)
-        idx_shuffled = np.arange(n_samp)
-        rng.shuffle(idx_shuffled)
-        idx_shuffle_list = [idx_shuffled[s] for s in slices]
+        assert order is not None
+        idx_shuffle_list = [order[s] for s in slices]
 
         if buffer_size is None:
             # shuffle all data at once
@@ -581,7 +579,7 @@ def _do_permutations(X_full, slices, threshold, tail, connectivity, stat_fun,
 
 
 def _do_1samp_permutations(X, slices, threshold, tail, connectivity, stat_fun,
-                           max_step, include, partitions, t_power, seeds,
+                           max_step, include, partitions, t_power, orders,
                            sample_shape, buffer_size, progress_bar):
     n_samp, n_vars = X.shape
     assert slices is None  # should be None for the 1 sample case
@@ -590,29 +588,23 @@ def _do_1samp_permutations(X, slices, threshold, tail, connectivity, stat_fun,
         buffer_size = None  # don't use buffer for few variables
 
     # allocate space for output
-    max_cluster_sums = np.empty(len(seeds), dtype=np.double)
+    max_cluster_sums = np.empty(len(orders), dtype=np.double)
 
     if buffer_size is not None:
         # allocate a buffer so we don't need to allocate memory in loop
         X_flip_buffer = np.empty((n_samp, buffer_size), dtype=X.dtype)
 
-    for seed_idx, seed in enumerate(seeds):
+    for seed_idx, order in enumerate(orders):
         if progress_bar is not None:
             if not (seed_idx + 1) % 32 or seed_idx == 0:
                 progress_bar.update(seed_idx + 1)
 
-        if isinstance(seed, np.ndarray):
-            # new surrogate data with specified sign flip
-            if not seed.size == n_samp:
-                raise ValueError('rng string must be n_samples long')
-            signs = 2 * seed[:, None].astype(int) - 1
-            if not np.all(np.equal(np.abs(signs), 1)):
-                raise ValueError('signs from rng must be +/- 1')
-        else:
-            rng = np.random.RandomState(seed)
-            # new surrogate data with random sign flip
-            signs = np.sign(0.5 - rng.rand(n_samp))
-            signs = signs[:, np.newaxis]
+        assert isinstance(order, np.ndarray)
+        # new surrogate data with specified sign flip
+        assert order.size == n_samp  # should be guaranteed by parent
+        signs = 2 * order[:, None].astype(int) - 1
+        if not np.all(np.equal(np.abs(signs), 1)):
+            raise ValueError('signs from rng must be +/- 1')
 
         if buffer_size is None:
             # be careful about non-writable memmap (GH#1507)
@@ -755,10 +747,27 @@ def _permutation_cluster_test(X, threshold, n_permutations, tail, stat_fun,
     # The stat should have the same shape as the samples
     T_obs.shape = sample_shape
 
-    if len(X) == 1:  # 1 sample test
+    # convert our seed to orders
+    # check to see if we can do an exact test
+    # (for a two-tailed test, we can exploit symmetry to just do half)
+    extra = ''
+    rng = check_random_state(seed)
+    del seed
+    if len(X) == 1:  # 1-sample test
         do_perm_func = _do_1samp_permutations
         X_full = X[0]
         slices = None
+        # determine ordering
+        max_perms = 2 ** (n_samples - (tail == 0)) - 1
+        if max_perms < n_permutations:
+            # omit first perm b/c accounted for in H0.append() later;
+            # convert to binary array representation
+            orders = np.arange(max_perms)
+            extra = ' (exact test)'
+        else:
+            orders = rng.choice(max_perms, n_permutations - 1)
+        orders = [np.fromiter(np.binary_repr(s + 1, n_samples), dtype=int)
+                  for s in orders]
     else:
         do_perm_func = _do_permutations
         X_full = np.concatenate(X, axis=0)
@@ -766,8 +775,15 @@ def _permutation_cluster_test(X, threshold, n_permutations, tail, stat_fun,
         splits_idx = np.append([0], np.cumsum(n_samples_per_condition))
         slices = [slice(splits_idx[k], splits_idx[k + 1])
                   for k in range(len(X))]
+        orders = [rng.permutation(len(X_full))
+                  for _ in range(n_permutations - 1)]
+    del rng
     parallel, my_do_perm_func, _ = parallel_func(do_perm_func, n_jobs)
 
+    if len(clusters) == 0:
+        warn('No clusters found, returning empty H0, clusters, and cluster_pv')
+        return T_obs, np.array([]), np.array([]), np.array([])
+
     # Step 2: If we have some clusters, repeat process on permuted data
     # -------------------------------------------------------------------
 
@@ -776,71 +792,59 @@ def get_progress_bar(seeds):
         return (ProgressBar(len(seeds), spinner=True) if
                 logger.level <= logging.INFO else None)
 
-    if len(clusters) > 0:
-        # check to see if we can do an exact test
-        # note for a two-tailed test, we can exploit symmetry to just do half
-        seeds = None
-        if len(X) == 1:
-            max_perms = 2 ** (n_samples - (tail == 0))
-            if max_perms <= n_permutations:
-                # omit first perm b/c accounted for in _pval_from_histogram,
-                # convert to binary array representation
-                seeds = [np.fromiter(np.binary_repr(s, n_samples), dtype=int)
-                         for s in range(1, max_perms)]
-
-        if seeds is None:
-            if seed is None:
-                seeds = [None] * n_permutations
+    # Step 3: repeat permutations for step-down-in-jumps procedure
+    n_removed = 1  # number of new clusters added
+    total_removed = 0
+    step_down_include = None  # start out including all points
+    n_step_downs = 0
+
+    while n_removed > 0:
+        # actually do the clustering for each partition
+        if include is not None:
+            if step_down_include is not None:
+                this_include = np.logical_and(include, step_down_include)
             else:
-                seeds = list(seed + np.arange(n_permutations))
-
-        # Step 3: repeat permutations for step-down-in-jumps procedure
-        n_removed = 1  # number of new clusters added
-        total_removed = 0
-        step_down_include = None  # start out including all points
-        n_step_downs = 0
-
-        while n_removed > 0:
-            # actually do the clustering for each partition
-            if include is not None:
-                if step_down_include is not None:
-                    this_include = np.logical_and(include, step_down_include)
-                else:
-                    this_include = include
-            else:
-                this_include = step_down_include
-            logger.info('Permuting ...')
-            H0 = parallel(my_do_perm_func(X_full, slices, threshold, tail,
-                          connectivity, stat_fun, max_step, this_include,
-                          partitions, t_power, s, sample_shape, buffer_size,
-                          get_progress_bar(s))
-                          for s in split_list(seeds, n_jobs))
-            H0 = np.concatenate(H0)
-            logger.info('Computing cluster p-values')
-            cluster_pv = _pval_from_histogram(cluster_stats, H0, tail)
-
-            # figure out how many new ones will be removed for step-down
-            to_remove = np.where(cluster_pv < step_down_p)[0]
-            n_removed = to_remove.size - total_removed
-            total_removed = to_remove.size
-            step_down_include = np.ones(n_tests, dtype=bool)
-            for ti in to_remove:
-                step_down_include[clusters[ti]] = False
-            if connectivity is None:
-                step_down_include.shape = sample_shape
-            n_step_downs += 1
-            if step_down_p > 0:
-                a_text = 'additional ' if n_step_downs > 1 else ''
-                logger.info('Step-down-in-jumps iteration #%i found %i %s'
-                            'cluster%s to exclude from subsequent iterations'
-                            % (n_step_downs, n_removed, a_text,
-                               _pl(n_removed)))
-        logger.info('Done.')
-        # The clusters should have the same shape as the samples
-        clusters = _reshape_clusters(clusters, sample_shape)
-        return T_obs, clusters, cluster_pv, H0
-    else:
-        return T_obs, np.array([]), np.array([]), np.array([])
+                this_include = include
+        else:
+            this_include = step_down_include
+        logger.info('Permuting %d times%s...' % (len(orders), extra))
+        H0 = parallel(my_do_perm_func(X_full, slices, threshold, tail,
+                      connectivity, stat_fun, max_step, this_include,
+                      partitions, t_power, order, sample_shape, buffer_size,
+                      get_progress_bar(order))
+                      for order in split_list(orders, n_jobs))
+        # include original (true) ordering
+        if tail == -1:  # up tail
+            orig = cluster_stats.min()
+        elif tail == 1:
+            orig = cluster_stats.max()
+        else:
+            orig = abs(cluster_stats).max()
+        H0.insert(0, [orig])
+        H0 = np.concatenate(H0)
+        logger.info('Computing cluster p-values')
+        cluster_pv = _pval_from_histogram(cluster_stats, H0, tail)
+
+        # figure out how many new ones will be removed for step-down
+        to_remove = np.where(cluster_pv < step_down_p)[0]
+        n_removed = to_remove.size - total_removed
+        total_removed = to_remove.size
+        step_down_include = np.ones(n_tests, dtype=bool)
+        for ti in to_remove:
+            step_down_include[clusters[ti]] = False
+        if connectivity is None:
+            step_down_include.shape = sample_shape
+        n_step_downs += 1
+        if step_down_p > 0:
+            a_text = 'additional ' if n_step_downs > 1 else ''
+            logger.info('Step-down-in-jumps iteration #%i found %i %s'
+                        'cluster%s to exclude from subsequent iterations'
+                        % (n_step_downs, n_removed, a_text,
+                           _pl(n_removed)))
+    logger.info('Done.')
+    # The clusters should have the same shape as the samples
+    clusters = _reshape_clusters(clusters, sample_shape)
+    return T_obs, clusters, cluster_pv, H0
 
 
 def ttest_1samp_no_p(X, sigma=0, method='relative'):
@@ -937,7 +941,7 @@ def permutation_cluster_test(X, threshold=None, n_permutations=1024,
         and :ref:`Logging documentation <tut_logging>` for more).
     n_jobs : int
         Number of permutations to run in parallel (requires joblib package).
-    seed : int or None
+    seed : int | instance of RandomState | None
         Seed the random number generator for results reproducibility.
     max_step : int
         When connectivity is a n_vertices x n_vertices matrix, specify the
@@ -980,23 +984,20 @@ def permutation_cluster_test(X, threshold=None, n_permutations=1024,
 
     Returns
     -------
-    T_obs : array of shape [n_tests]
+    T_obs : array, shape (n_tests,)
         T-statistic observed for all variables.
     clusters : list
         List type defined by out_type above.
     cluster_pv : array
         P-value for each cluster
-    H0 : array of shape [n_permutations]
+    H0 : array, shape (n_permutations,)
         Max cluster level stats observed under permutation.
 
-    Notes
-    -----
-    Reference:
-    Cluster permutation algorithm as described in
-    Maris/Oostenveld (2007),
-    "Nonparametric statistical testing of EEG- and MEG-data"
-    Journal of Neuroscience Methods, Vol. 164, No. 1., pp. 177-190.
-    doi:10.1016/j.jneumeth.2007.03.024
+    References
+    ----------
+    .. [1] Maris/Oostenveld (2007), "Nonparametric statistical testing of
+       EEG- and MEG-data" Journal of Neuroscience Methods,
+       Vol. 164, No. 1., pp. 177-190. doi:10.1016/j.jneumeth.2007.03.024.
     """
     from scipy import stats
     ppf = stats.f.ppf
@@ -1036,7 +1037,7 @@ def permutation_cluster_1samp_test(X, threshold=None, n_permutations=1024,
     estimates etc., calculate if the observed mean significantly deviates
     from 0. The procedure uses a cluster analysis with permutation test
     for calculating corrected p-values. Randomized data are generated with
-    random sign flips.
+    random sign flips. See [1]_ for more information.
 
     Parameters
     ----------
@@ -1050,7 +1051,7 @@ def permutation_cluster_1samp_test(X, threshold=None, n_permutations=1024,
         If a dict is used, then threshold-free cluster enhancement (TFCE)
         will be used.
     n_permutations : int
-        The number of permutations to compute.
+        The maximum number of permutations to compute.
     tail : -1 or 0 or 1 (default = 0)
         If tail is 1, the statistic is thresholded above threshold.
         If tail is -1, the statistic is thresholded below threshold.
@@ -1070,11 +1071,8 @@ def permutation_cluster_1samp_test(X, threshold=None, n_permutations=1024,
         and :ref:`Logging documentation <tut_logging>` for more).
     n_jobs : int
         Number of permutations to run in parallel (requires joblib package).
-    seed : int or None
+    seed : int | instance of RandomState | None
         Seed the random number generator for results reproducibility.
-        Note that if n_permutations >= 2^(n_samples) [or (2^(n_samples-1)) for
-        two-tailed tests], this value will be ignored since an exact test
-        (full permutation test) will be performed.
     max_step : int
         When connectivity is a n_vertices x n_vertices matrix, specify the
         maximum number of steps between vertices along the second dimension
@@ -1116,23 +1114,30 @@ def permutation_cluster_1samp_test(X, threshold=None, n_permutations=1024,
 
     Returns
     -------
-    T_obs : array of shape [n_tests]
+    T_obs : array, shape (n_tests,)
         T-statistic observed for all variables
     clusters : list
         List type defined by out_type above.
     cluster_pv : array
         P-value for each cluster
-    H0 : array of shape [n_permutations]
+    H0 : array, shape (n_permutations,)
         Max cluster level stats observed under permutation.
 
     Notes
     -----
-    Reference:
-    Cluster permutation algorithm as described in
-    Maris/Oostenveld (2007),
-    "Nonparametric statistical testing of EEG- and MEG-data"
-    Journal of Neuroscience Methods, Vol. 164, No. 1., pp. 177-190.
-    doi:10.1016/j.jneumeth.2007.03.024
+    If ``n_permutations >= 2 ** (n_samples - (tail == 0))``,
+    ``n_permutations`` and ``seed`` will be ignored since an exact test
+    (full permutation test) will be performed.
+
+    If no initial clusters are found, i.e., all points in the true
+    distribution are below the threshold, then ``clusters``, ``cluster_pv``,
+    and ``H0`` will all be empty arrays.
+
+    References
+    ----------
+    .. [1] Maris/Oostenveld (2007), "Nonparametric statistical testing of
+       EEG- and MEG-data" Journal of Neuroscience Methods,
+       Vol. 164, No. 1., pp. 177-190. doi:10.1016/j.jneumeth.2007.03.024.
     """
     from scipy import stats
     ppf = stats.t.ppf
@@ -1173,17 +1178,19 @@ def spatio_temporal_cluster_1samp_test(X, threshold=None,
     """Non-parametric cluster-level 1 sample T-test for spatio-temporal data.
 
     This function provides a convenient wrapper for data organized in the form
-    (observations x time x space) to use permutation_cluster_1samp_test.
+    (observations x time x space) to use
+    :func:`mne.stats.permutation_cluster_1samp_test`. See [1]_ for more
+    information.
 
     Parameters
     ----------
     X : array
-        Array of shape observations x time x vertices.
+        Array data, shape ``(n_observations, n_times, n_vertices)``.
     threshold : float | dict | None
         If threshold is None, it will choose a t-threshold equivalent to
         p < 0.05 for the given number of (within-subject) observations.
         If a dict is used, then threshold-free cluster enhancement (TFCE)
-        will be used.
+        [2]_ will be used.
     n_permutations : int
         The number of permutations to compute.
     tail : -1 or 0 or 1 (default = 0)
@@ -1205,11 +1212,8 @@ def spatio_temporal_cluster_1samp_test(X, threshold=None,
         and :ref:`Logging documentation <tut_logging>` for more).
     n_jobs : int
         Number of permutations to run in parallel (requires joblib package).
-    seed : int or None
+    seed : int | instance of RandomState | None
         Seed the random number generator for results reproducibility.
-        Note that if n_permutations >= 2^(n_samples) [or (2^(n_samples-1)) for
-        two-tailed tests], this value will be ignored since an exact test
-        (full permutation test) will be performed.
     max_step : int
         When connectivity is a n_vertices x n_vertices matrix, specify the
         maximum number of steps between vertices along the second dimension
@@ -1249,28 +1253,33 @@ def spatio_temporal_cluster_1samp_test(X, threshold=None,
 
     Returns
     -------
-    T_obs : array of shape [n_tests]
+    T_obs : array, shape (n_times * n_vertices,)
         T-statistic observed for all variables.
     clusters : list
         List type defined by out_type above.
     cluster_pv: array
         P-value for each cluster
-    H0 : array of shape [n_permutations]
+    H0 : array, shape (n_permutations,)
         Max cluster level stats observed under permutation.
 
     Notes
     -----
-    Reference:
-    Cluster permutation algorithm as described in
-    Maris/Oostenveld (2007),
-    "Nonparametric statistical testing of EEG- and MEG-data"
-    Journal of Neuroscience Methods, Vol. 164, No. 1., pp. 177-190.
-    doi:10.1016/j.jneumeth.2007.03.024
-
-    TFCE originally described in Smith/Nichols (2009),
-    "Threshold-free cluster enhancement: Addressing problems of
-    smoothing, threshold dependence, and localisation in cluster
-    inference", NeuroImage 44 (2009) 83-98.
+    If ``n_permutations >= 2 ** (n_samples - (tail == 0))``,
+    ``n_permutations`` and ``seed`` will be ignored since an exact test
+    (full permutation test) will be performed.
+
+    If no initial clusters are found, i.e., all points in the true
+    distribution are below the threshold, then ``clusters``, ``cluster_pv``,
+    and ``H0`` will all be empty arrays.
+
+    References
+    ----------
+    .. [1] Maris/Oostenveld, "Nonparametric statistical testing of
+       EEG- and MEG-data" Journal of Neuroscience Methods,
+       Vol. 164, No. 1., pp. 177-190. doi:10.1016/j.jneumeth.2007.03.024
+    .. [2] Smith/Nichols (2009), "Threshold-free cluster enhancement:
+       Addressing problems of smoothing, threshold dependence, and
+       localisation in cluster inference", NeuroImage 44 (2009) 83-98.
     """
     n_samples, n_times, n_vertices = X.shape
 
@@ -1308,12 +1317,14 @@ def spatio_temporal_cluster_test(X, threshold=1.67, n_permutations=1024,
     """Non-parametric cluster-level test for spatio-temporal data.
 
     This function provides a convenient wrapper for data organized in the form
-    (observations x time x space) to use permutation_cluster_test.
+    (observations x time x space) to use
+    :func:`mne.stats.permutation_cluster_test`. See [1]_ for more information.
 
     Parameters
     ----------
     X: list of arrays
-        Array of shape (observations, time, vertices) in each group.
+        List of data arrays, shape ``(n_observations, n_times, n_vertices)``
+        in each group.
     threshold: float
         The threshold for the statistic.
     n_permutations: int
@@ -1332,7 +1343,7 @@ def spatio_temporal_cluster_test(X, threshold=1.67, n_permutations=1024,
         and :ref:`Logging documentation <tut_logging>` for more).
     n_jobs : int
         Number of permutations to run in parallel (requires joblib package).
-    seed : int or None
+    seed : int | instance of RandomState | None
         Seed the random number generator for results reproducibility.
     max_step : int
         When connectivity is a n_vertices x n_vertices matrix, specify the
@@ -1373,23 +1384,20 @@ def spatio_temporal_cluster_test(X, threshold=1.67, n_permutations=1024,
 
     Returns
     -------
-    T_obs : array of shape [n_tests]
+    T_obs : array, shape (n_times * n_vertices,)
         T-statistic observed for all variables
     clusters : list
         List type defined by out_type above.
     cluster_pv: array
         P-value for each cluster
-    H0 : array of shape [n_permutations]
+    H0 : array, shape (n_permutations,)
         Max cluster level stats observed under permutation.
 
-    Notes
-    -----
-    Reference:
-    Cluster permutation algorithm as described in
-    Maris/Oostenveld (2007),
-    "Nonparametric statistical testing of EEG- and MEG-data"
-    Journal of Neuroscience Methods, Vol. 164, No. 1., pp. 177-190.
-    doi:10.1016/j.jneumeth.2007.03.024
+    References
+    ----------
+    .. [1] Maris/Oostenveld (2007), "Nonparametric statistical testing of
+       EEG- and MEG-data", Journal of Neuroscience Methods,
+       Vol. 164, No. 1., pp. 177-190. doi:10.1016/j.jneumeth.2007.03.024.
     """
     n_samples, n_times, n_vertices = X[0].shape
 


Test patch:
diff --git a/mne/stats/tests/test_cluster_level.py b/mne/stats/tests/test_cluster_level.py
--- a/mne/stats/tests/test_cluster_level.py
+++ b/mne/stats/tests/test_cluster_level.py
@@ -1,3 +1,8 @@
+# Authors: Eric Larson <larson.eric.d@gmail.com>
+#          Alexandre Gramfort <alexandre.gramfort@telecom-paristech.fr>
+#
+# License: BSD (3-clause)
+
 from functools import partial
 import os
 import warnings
@@ -14,7 +19,7 @@
                                      spatio_temporal_cluster_test,
                                      spatio_temporal_cluster_1samp_test,
                                      ttest_1samp_no_p, summarize_clusters_stc)
-from mne.utils import run_tests_if_main, slow_test, _TempDir, catch_logging
+from mne.utils import run_tests_if_main, _TempDir, catch_logging
 
 warnings.simplefilter('always')  # enable b/c these tests throw warnings
 
@@ -85,7 +90,7 @@ def test_permutation_step_down_p():
         try:
             from sklearn.feature_extraction.image import grid_to_graph
         except ImportError:
-            from scikits.learn.feature_extraction.image import grid_to_graph  # noqa: F401,E501
+            from scikits.learn.feature_extraction.image import grid_to_graph  # noqa: F401,E501 analysis:ignore
     except ImportError:
         return
     rng = np.random.RandomState(0)
@@ -136,7 +141,6 @@ def test_cluster_permutation_test():
         assert_array_equal(cluster_p_values, cluster_p_values_buff)
 
 
-@slow_test
 def test_cluster_permutation_t_test():
     """Test cluster level permutations T-test."""
     condition1_1d, condition2_1d, condition1_2d, condition2_2d = \
@@ -183,7 +187,6 @@ def test_cluster_permutation_t_test():
             assert_array_equal(cluster_p_values_neg, cluster_p_values_neg_buff)
 
 
-@slow_test
 def test_cluster_permutation_with_connectivity():
     """Test cluster level permutations with connectivity matrix."""
     try:
@@ -328,7 +331,6 @@ def test_cluster_permutation_with_connectivity():
         assert_true(np.min(out_connectivity_6[2]) < 0.05)
 
 
-@slow_test
 def test_permutation_connectivity_equiv():
     """Test cluster level permutations with and without connectivity."""
     try:
@@ -346,66 +348,64 @@ def test_permutation_connectivity_equiv():
     # add some significant points
     X[:, :, 0:2] += 10  # span two time points and two spatial points
     X[:, 1, 3] += 20  # span one time point
-    max_steps = [1, 1, 1, 2]
+    max_steps = [1, 1, 1, 2, 1]
     # This will run full algorithm in two ways, then the ST-algorithm in 2 ways
     # All of these should give the same results
-    conns = [None, grid_to_graph(n_time, n_space),
-             grid_to_graph(1, n_space), grid_to_graph(1, n_space)]
+    conns = [None,
+             grid_to_graph(n_time, n_space),
+             grid_to_graph(1, n_space),
+             grid_to_graph(1, n_space),
+             None]
     stat_map = None
-    thresholds = [2, dict(start=1.5, step=1.0)]
-    sig_counts = [2, 5]
-    sdps = [0, 0.05, 0.05]
-    ots = ['mask', 'mask', 'indices']
+    thresholds = [2, 2, 2, 2, dict(start=0.01, step=1.0)]
+    sig_counts = [2, 2, 2, 2, 5]
     stat_fun = partial(ttest_1samp_no_p, sigma=1e-3)
-    for thresh, count in zip(thresholds, sig_counts):
-        cs = None
-        ps = None
-        for max_step, conn in zip(max_steps, conns):
-            for sdp, ot in zip(sdps, ots):
-                t, clusters, p, H0 = \
-                    permutation_cluster_1samp_test(
-                        X, threshold=thresh, connectivity=conn, n_jobs=2,
-                        max_step=max_step, stat_fun=stat_fun,
-                        step_down_p=sdp, out_type=ot)
-                # make sure our output datatype is correct
-                if ot == 'mask':
-                    assert_true(isinstance(clusters[0], np.ndarray))
-                    assert_true(clusters[0].dtype == bool)
-                    assert_array_equal(clusters[0].shape, X.shape[1:])
-                else:  # ot == 'indices'
-                    assert_true(isinstance(clusters[0], tuple))
-
-                # make sure all comparisons were done; for TFCE, no perm
-                # should come up empty
-                if count == 8:
-                    assert_true(not np.any(H0 == 0))
-                inds = np.where(p < 0.05)[0]
-                assert_true(len(inds) == count)
-                this_cs = [clusters[ii] for ii in inds]
-                this_ps = p[inds]
-                this_stat_map = np.zeros((n_time, n_space), dtype=bool)
-                for ci, c in enumerate(this_cs):
-                    if isinstance(c, tuple):
-                        this_c = np.zeros((n_time, n_space), bool)
-                        for x, y in zip(c[0], c[1]):
-                            this_stat_map[x, y] = True
-                            this_c[x, y] = True
-                        this_cs[ci] = this_c
-                        c = this_c
-                    this_stat_map[c] = True
-                if cs is None:
-                    ps = this_ps
-                    cs = this_cs
-                if stat_map is None:
-                    stat_map = this_stat_map
-                assert_array_equal(ps, this_ps)
-                assert_true(len(cs) == len(this_cs))
-                for c1, c2 in zip(cs, this_cs):
-                    assert_array_equal(c1, c2)
-                assert_array_equal(stat_map, this_stat_map)
-
-
-@slow_test
+
+    cs = None
+    ps = None
+    for thresh, count, max_step, conn in zip(thresholds, sig_counts,
+                                             max_steps, conns):
+        t, clusters, p, H0 = \
+            permutation_cluster_1samp_test(
+                X, threshold=thresh, connectivity=conn, n_jobs=2,
+                max_step=max_step, stat_fun=stat_fun)
+        # make sure our output datatype is correct
+        assert_true(isinstance(clusters[0], np.ndarray))
+        assert_true(clusters[0].dtype == bool)
+        assert_array_equal(clusters[0].shape, X.shape[1:])
+
+        # make sure all comparisons were done; for TFCE, no perm
+        # should come up empty
+        inds = np.where(p < 0.05)[0]
+        assert_equal(len(inds), count)
+        if isinstance(thresh, dict):
+            assert_equal(len(clusters), n_time * n_space)
+            assert_true(np.all(H0 != 0))
+            continue
+        this_cs = [clusters[ii] for ii in inds]
+        this_ps = p[inds]
+        this_stat_map = np.zeros((n_time, n_space), dtype=bool)
+        for ci, c in enumerate(this_cs):
+            if isinstance(c, tuple):
+                this_c = np.zeros((n_time, n_space), bool)
+                for x, y in zip(c[0], c[1]):
+                    this_stat_map[x, y] = True
+                    this_c[x, y] = True
+                this_cs[ci] = this_c
+                c = this_c
+            this_stat_map[c] = True
+        if cs is None:
+            ps = this_ps
+            cs = this_cs
+        if stat_map is None:
+            stat_map = this_stat_map
+        assert_array_equal(ps, this_ps)
+        assert_true(len(cs) == len(this_cs))
+        for c1, c2 in zip(cs, this_cs):
+            assert_array_equal(c1, c2)
+        assert_array_equal(stat_map, this_stat_map)
+
+
 def test_spatio_temporal_cluster_connectivity():
     """Test spatio-temporal cluster permutations."""
     try:
@@ -474,4 +474,30 @@ def test_summarize_clusters():
     assert_raises(RuntimeError, summarize_clusters_stc, clu)
 
 
+def test_permutation_test_H0():
+    """Test that H0 is populated properly during testing."""
+    rng = np.random.RandomState(0)
+    data = rng.rand(7, 10, 1) - 0.5
+    with warnings.catch_warnings(record=True) as w:
+        t, clust, p, h0 = spatio_temporal_cluster_1samp_test(
+            data, threshold=100, n_permutations=1024, seed=rng)
+    assert_equal(len(w), 1)
+    assert_true('No clusters found' in str(w[0].message))
+    assert_equal(len(h0), 0)
+
+    for n_permutations in (1024, 65, 64, 63):
+        t, clust, p, h0 = spatio_temporal_cluster_1samp_test(
+            data, threshold=0.1, n_permutations=n_permutations, seed=rng)
+        assert_equal(len(h0), min(n_permutations, 64))
+        assert_true(isinstance(clust[0], tuple))  # sets of indices
+    for tail, thresh in zip((-1, 0, 1), (-0.1, 0.1, 0.1)):
+        with warnings.catch_warnings(record=True) as w:
+            t, clust, p, h0 = spatio_temporal_cluster_1samp_test(
+                data, threshold=thresh, seed=rng, tail=tail, out_type='mask')
+        assert_equal(len(w), 0)
+        assert_true(isinstance(clust[0], np.ndarray))  # bool mask
+        # same as "128 if tail else 64"
+        assert_equal(len(h0), 2 ** (7 - (tail == 0)))  # exact test
+
+
 run_tests_if_main()
diff --git a/mne/stats/tests/test_permutations.py b/mne/stats/tests/test_permutations.py
--- a/mne/stats/tests/test_permutations.py
+++ b/mne/stats/tests/test_permutations.py
@@ -1,13 +1,17 @@
-import numpy as np
+# Authors: Alexandre Gramfort <alexandre.gramfort@telecom-paristech.fr>
+#
+# License: BSD (3-clause)
+
 from numpy.testing import assert_array_equal, assert_almost_equal
+import numpy as np
 from scipy import stats
 
 from mne.stats.permutations import permutation_t_test
+from mne.utils import run_tests_if_main
 
 
 def test_permutation_t_test():
-    """Test T-test based on permutations
-    """
+    """Test T-test based on permutations."""
     # 1 sample t-test
     np.random.seed(10)
     n_samples, n_tests = 30, 5
@@ -31,3 +35,6 @@ def test_permutation_t_test():
     T_obs_scipy, p_values_scipy = stats.ttest_1samp(X[:, 0], 0)
     assert_almost_equal(T_obs[0], T_obs_scipy, 8)
     assert_almost_equal(p_values[0], p_values_scipy, 2)
+
+
+run_tests_if_main()