Repository: mne-tools/mne-python

Issue:
STY/DOC/API: clustering stats semantics
Currently our clustering stats functions implicitly seem to favor t-test based statistics.
This is expressed in:
- 'T_obs' etc. parameter / identifier names
- other scipy stats functions e.g. wilcoxon won't be accepted as value for the 'stat_fun' parameter, at least not without hacks.

Another concern is related to the function docs e.g of the spatio-temporal clustering function, which often refer to the docs of the basic, wrapped clustering stats functions.
This is parsimonious, but not user friendly -- you would like to look up the parameters directly rather than jumping from one doc string to the other.

How would people think about a PR that generalized the style and semantics of the clustering stats functions?



Discussion:
A few points:
1. I think we should keep the variable names as `T_obs`, etc. since it's the convention in the stats papers I've read to assume (WLOG) that you're using a t-test as your initial statistic. I think it's clear enough that it generalizes beyond that to more complex and/or non-parametric methods.
2. At some point there's always a balance between simplicity and generalizability, and I think we've done an appropriate job of that with the stat functions. Where is the limitation you're referring to? For example, to use the wilcoxon, you'd just have to write your own wrapper:

```
def wilcoxon_test(x):
    return stats.wilcoxon(x)[0]
```

Note also that even using `ttest_1samp` from `scipy.stats` requires this type of "hack" -- and I use scare-quotes here because I think this is a limitation of trying to keep things simple. We've just made it easier on the user by including our own (and extending it with variance correction). I don't think this is so great a burden on the user. We could add a generic wrapper like:

```
def scipy_stat_wrapper(function, x):
    return function(x)[0]
```

This would be trivial, but I don't think it would add too much. Or, we could add an example if you think that would help. However, I would be wary of pairing something like the `wilcoxon` with clustering examples, since that may give people the sense that using the `ttest_1samp` with a permutation cluster test is actually doing a parametric test, which it is not...

Sorry, I accidentally posted that comment before I was done typing it (see minor edits on the site).

Third, I'd be fine with copying all those docs over. Even though it makes editing any one of them more annoying (because it needs to be changed in one more place), I think you're right it might be easier for users.

> I think it's clear enough that it generalizes beyond that to more complex and/or non-parametric methods.

ok, point taken.

> and I use scare-quotes here because I think this is a limitation of trying to keep things simple. (...)
> We've just made it easier on the user by including our own (and extending it with variance correction)

... yes, I agree.

>  Or, we could add an example if you think that would help

I think this is what would be really beneficial, examples are good. I would prefer a proper example to an mne-provided wrapper function (use cases are different and we can not provide with everything).

>  since that may give people the sense that using the ttest_1samp with a permutation cluster test is actually doing a parametric test, which it is not...

so we'll add a note ;-)

>  Even though it makes editing any one of them more annoying (because it needs to be changed in one more place), I think you're right it might be easier for users.

It may be minor but I think it will save time + nerves, not only for users, and, given, that these functions are still evolving and come with many parameters.
Another option might be to have a doc string utils which gets later on added like:
`the_particular_stat_function.__doc__ + utils._doc_cluster_stats_template`

I give my vote to eric :)

I think here isn't much left to vote for ;-) consensus-alarm.

@agramfort would changing doc strings (stop redirecting it) be fine with you?

On 17.04.2013, at 11:49, Alexandre Gramfort notifications@github.com wrote:

> I give my vote to eric :)
> —
> Reply to this email directly or view it on GitHub.

> @agramfort would changing doc strings (stop redirecting it) be fine with
> you?

sure


Resolving patch:
diff --git a/mne/stats/cluster_level.py b/mne/stats/cluster_level.py
--- a/mne/stats/cluster_level.py
+++ b/mne/stats/cluster_level.py
@@ -255,7 +255,7 @@ def _find_clusters(x, threshold, tail=0, connectivity=None, max_step=1,
         be symmetric and only the upper triangular half is used.
         If connectivity is a list, it is assumed that each entry stores the
         indices of the spatial neighbors in a spatio-temporal dataset x.
-        Defaut is None, i.e, a regular lattice connectivity.
+        Default is None, i.e, a regular lattice connectivity.
     max_step : int
         If connectivity is a list, this defines the maximal number of steps
         between vertices along the second dimension (typically time) to be
@@ -802,7 +802,7 @@ def permutation_cluster_test(X, threshold=None, n_permutations=1024,
     ----------
     X : list
         List of 2d-arrays containing the data, dim 1: timepoints, dim 2:
-        elements of groups
+        elements of groups.
     threshold : float | dict | None
         If threshold is None, it will choose a t-threshold equivalent to
         p < 0.05 for the given number of (within-subject) observations.
@@ -817,15 +817,15 @@ def permutation_cluster_test(X, threshold=None, n_permutations=1024,
         the distribution.
     stat_fun : callable
         function called to calculate statistics, must accept 1d-arrays as
-        arguments (default: scipy.stats.f_oneway)
+        arguments (default: scipy.stats.f_oneway).
     connectivity : sparse matrix.
         Defines connectivity between features. The matrix is assumed to
         be symmetric and only the upper triangular half is used.
-        Defaut is None, i.e, a regular lattice connectivity.
+        Default is None, i.e, a regular lattice connectivity.
     verbose : bool, str, int, or None
         If not None, override default verbose level (see mne.verbose).
     n_jobs : int
-        Number of permutations to run in parallel (requires joblib package.)
+        Number of permutations to run in parallel (requires joblib package).
     seed : int or None
         Seed the random number generator for results reproducibility.
     max_step : int
@@ -844,7 +844,7 @@ def permutation_cluster_test(X, threshold=None, n_permutations=1024,
         Setting this to a reasonable value, e.g. 0.05, can increase sensitivity
         but costs computation time.
     t_power : float
-        Power to raise the statistical values (usually t-values) by before
+        Power to raise the statistical values (usually f-values) by before
         summing (sign will be retained). Note that t_power == 0 will give a
         count of nodes in each cluster, t_power == 1 will weight each node by
         its statistical score.
@@ -863,7 +863,7 @@ def permutation_cluster_test(X, threshold=None, n_permutations=1024,
     Returns
     -------
     T_obs : array of shape [n_tests]
-        T-statistic observerd for all variables
+        T-statistic observed for all variables.
     clusters : list
         List type defined by out_type above.
     cluster_pv : array
@@ -935,18 +935,18 @@ def permutation_cluster_1samp_test(X, threshold=None, n_permutations=1024,
         If tail is 0, the statistic is thresholded on both sides of
         the distribution.
     stat_fun : function
-        Function used to compute the statistical map
+        Function used to compute the statistical map.
     connectivity : sparse matrix or None
         Defines connectivity between features. The matrix is assumed to
         be symmetric and only the upper triangular half is used.
         This matrix must be square with dimension (n_vertices * n_times) or
-        (n_vertices). Defaut is None, i.e, a regular lattice connectivity.
+        (n_vertices). Default is None, i.e, a regular lattice connectivity.
         Use square n_vertices matrix for datasets with a large temporal
         extent to save on memory and computation time.
     verbose : bool, str, int, or None
         If not None, override default verbose level (see mne.verbose).
     n_jobs : int
-        Number of permutations to run in parallel (requires joblib package.)
+        Number of permutations to run in parallel (requires joblib package).
     seed : int or None
         Seed the random number generator for results reproducibility.
         Note that if n_permutations >= 2^(n_samples) [or (2^(n_samples-1)) for
@@ -987,7 +987,7 @@ def permutation_cluster_1samp_test(X, threshold=None, n_permutations=1024,
     Returns
     -------
     T_obs : array of shape [n_tests]
-        T-statistic observerd for all variables
+        T-statistic observed for all variables
     clusters : list
         List type defined by out_type above.
     cluster_pv : array
@@ -1034,7 +1034,7 @@ def spatio_temporal_cluster_1samp_test(X, threshold=None,
     """Non-parametric cluster-level 1 sample T-test for spatio-temporal data
 
     This function provides a convenient wrapper for data organized in the form
-    (observations x space x time) to use permutation_cluster_1samp_test.
+    (observations x time x space) to use permutation_cluster_1samp_test.
 
     Parameters
     ----------
@@ -1046,36 +1046,64 @@ def spatio_temporal_cluster_1samp_test(X, threshold=None,
         If a dict is used, then threshold-free cluster enhancement (TFCE)
         will be used.
     n_permutations : int
-        See permutation_cluster_1samp_test.
+        The number of permutations to compute.
     tail : -1 or 0 or 1 (default = 0)
-        See permutation_cluster_1samp_test.
+        If tail is 1, the statistic is thresholded above threshold.
+        If tail is -1, the statistic is thresholded below threshold.
+        If tail is 0, the statistic is thresholded on both sides of
+        the distribution.
     stat_fun : function
-        See permutation_cluster_1samp_test.
+        Function used to compute the statistical map.
     connectivity : sparse matrix or None
-        See permutation_cluster_1samp_test.
+        Defines connectivity between features. The matrix is assumed to
+        be symmetric and only the upper triangular half is used.
+        This matrix must be square with dimension (n_vertices * n_times) or
+        (n_vertices). Default is None, i.e, a regular lattice connectivity.
+        Use square n_vertices matrix for datasets with a large temporal
+        extent to save on memory and computation time.
     verbose : bool, str, int, or None
         If not None, override default verbose level (see mne.verbose).
     n_jobs : int
-        See permutation_cluster_1samp_test.
+        Number of permutations to run in parallel (requires joblib package).
     seed : int or None
-        See permutation_cluster_1samp_test.
+        Seed the random number generator for results reproducibility.
+        Note that if n_permutations >= 2^(n_samples) [or (2^(n_samples-1)) for
+        two-tailed tests], this value will be ignored since an exact test
+        (full permutation test) will be performed.
     max_step : int
-        See permutation_cluster_1samp_test.
+        When connectivity is a n_vertices x n_vertices matrix, specify the
+        maximum number of steps between vertices along the second dimension
+        (typically time) to be considered connected. This is not used for full
+        or None connectivity matrices.
     spatial_exclude : list of int or None
         List of spatial indices to exclude from clustering.
     step_down_p : float
-        See permutation_cluster_1samp_test.
+        To perform a step-down-in-jumps test, pass a p-value for clusters to
+        exclude from each successive iteration. Default is zero, perform no
+        step-down test (since no clusters will be smaller than this value).
+        Setting this to a reasonable value, e.g. 0.05, can increase sensitivity
+        but costs computation time.
     t_power : float
-        See permutation_cluster_1samp_test.
+        Power to raise the statistical values (usually t-values) by before
+        summing (sign will be retained). Note that t_power == 0 will give a
+        count of nodes in each cluster, t_power == 1 will weight each node by
+        its statistical score.
     out_type : str
-        See permutation_cluster_1samp_test.
+        For arrays with connectivity, this sets the output format for clusters.
+        If 'mask', it will pass back a list of boolean mask arrays.
+        If 'indices', it will pass back a list of lists, where each list is the
+        set of vertices in a given cluster. Note that the latter may use far
+        less memory for large datasets.
     check_disjoint : bool
-        See permutation_cluster_1samp_test.
+        If True, the connectivity matrix (or list) will be examined to
+        determine of it can be separated into disjoint sets. In some cases
+        (usually with connectivity as a list and many "time" points), this
+        can lead to faster clustering, but results should be identical.
 
     Returns
     -------
     T_obs : array of shape [n_tests]
-        T-statistic observerd for all variables
+        T-statistic observed for all variables.
     clusters : list
         List type defined by out_type above.
     cluster_pv: array
@@ -1093,7 +1121,7 @@ def spatio_temporal_cluster_1samp_test(X, threshold=None,
     doi:10.1016/j.jneumeth.2007.03.024
 
     TFCE originally described in Smith/Nichols (2009),
-    "Threshold-free cluster enhancement: Adressing problems of
+    "Threshold-free cluster enhancement: Addressing problems of
     smoothing, threshold dependence, and localisation in cluster
     inference", NeuroImage 44 (2009) 83-98.
     """
@@ -1128,7 +1156,7 @@ def spatio_temporal_cluster_test(X, threshold=1.67,
     """Non-parametric cluster-level test for spatio-temporal data
 
     This function provides a convenient wrapper for data organized in the form
-    (observations x space x time) to use permutation_cluster_test.
+    (observations x time x space) to use permutation_cluster_test.
 
     Parameters
     ----------
@@ -1141,32 +1169,52 @@ def spatio_temporal_cluster_test(X, threshold=1.67,
     tail : -1 or 0 or 1 (default = 0)
         See permutation_cluster_test.
     stat_fun : function
-        See permutation_cluster_test.
+        function called to calculate statistics, must accept 1d-arrays as
+        arguments (default: scipy.stats.f_oneway)
     connectivity : sparse matrix or None
-        See permutation_cluster_test.
+        Defines connectivity between features. The matrix is assumed to
+        be symmetric and only the upper triangular half is used.
+        Default is None, i.e, a regular lattice connectivity.
     verbose : bool, str, int, or None
         If not None, override default verbose level (see mne.verbose).
     n_jobs : int
-        See permutation_cluster_test.
+        Number of permutations to run in parallel (requires joblib package).
     seed : int or None
-        See permutation_cluster_test.
+        Seed the random number generator for results reproducibility.
     max_step : int
-        See permutation_cluster_test.
+        When connectivity is a n_vertices x n_vertices matrix, specify the
+        maximum number of steps between vertices along the second dimension
+        (typically time) to be considered connected. This is not used for full
+        or None connectivity matrices.
     spatial_exclude : list of int or None
         List of spatial indices to exclude from clustering.
     step_down_p : float
-        See permutation_cluster_test.
+        To perform a step-down-in-jumps test, pass a p-value for clusters to
+        exclude from each successive iteration. Default is zero, perform no
+        step-down test (since no clusters will be smaller than this value).
+        Setting this to a reasonable value, e.g. 0.05, can increase sensitivity
+        but costs computation time.
     t_power : float
-        See permutation_cluster_test.
+        Power to raise the statistical values (usually f-values) by before
+        summing (sign will be retained). Note that t_power == 0 will give a
+        count of nodes in each cluster, t_power == 1 will weight each node by
+        its statistical score.
     out_type : str
-        See permutation_cluster_test.
+        For arrays with connectivity, this sets the output format for clusters.
+        If 'mask', it will pass back a list of boolean mask arrays.
+        If 'indices', it will pass back a list of lists, where each list is the
+        set of vertices in a given cluster. Note that the latter may use far
+        less memory for large datasets.
     check_disjoint : bool
-        See permutation_cluster_test.
+        If True, the connectivity matrix (or list) will be examined to
+        determine of it can be separated into disjoint sets. In some cases
+        (usually with connectivity as a list and many "time" points), this
+        can lead to faster clustering, but results should be identical.
 
     Returns
     -------
     T_obs : array of shape [n_tests]
-        T-statistic observerd for all variables
+        T-statistic observed for all variables
     clusters : list
         List type defined by out_type above.
     cluster_pv: array
@@ -1214,11 +1262,11 @@ def _st_mask_from_s_inds(n_times, n_vertices, vertices, set_as=True):
     Parameters
     ----------
     n_times : int
-        Number of time points
+        Number of time points.
     n_vertices : int
-        Number of spatial points
+        Number of spatial points.
     vertices : list or array of int
-        Vertex numbers to set
+        Vertex numbers to set.
     set_as : bool
         If True, all points except "vertices" are set to False (inclusion).
         If False, all points except "vertices" are set to True (exclusion).


Test patch: