GitHub PullRequestReviewCommentEvent at 2014-09-25 15:24:49
Repository: scikit-bio
Actor: ebolyen
Payload: {"action":"created","comment":{"url":"https://api.github.com/repos/biocore/scikit-bio/pulls/comments/18038577","id":18038577,"diff_hunk":"@@ -0,0 +1,904 @@\n+r\"\"\"\n+Empirical Power Estimation (:mod:`skbio.stats.power`)\n+=====================================================\n+\n+.. currentmodule:: skbio.stats.power\n+\n+The purpose of this module is to provide empirical, post-hoc power estimation\n+of normally and non-normally distributed data. It also provides support to\n+subsample data to faciliate this analysis.\n+\n+The underlying principle is based on subsampling and monte carlo simulation.\n+Assume that there is some set of populations, :math:`K_{1}, K_{2}, ... K_{n}`\n+which have some property, :math:`\\mu` such that :math:`\\mu_{1} \\neq \\mu_{2}\n+\\neq ... \\neq \\mu_{n}`. For each of the populations, a sample, S can be drawn,\n+with a parameter, :math:`x` where :math:`x \\approx \\mu` and for the samples,\n+we can use a test, f, to show that :math:`x_{1} \\neq x_{2} \\neq ...\n+\\neq x_{n}`.\n+\n+Since we known that :math:`\\mu_{1} \\neq \\mu_{2} \\neq ... \\neq \\mu_{n}`,\n+we know we should reject the null hypothesis. If we fail to reject the null\n+hypothesis, we have comitted a Type II error and our result is a false\n+negative. We can estimate the frequency of Type II errors at various sampling\n+depths by repeatedly subsampling the populations and observing how often we\n+see a false negative. If we repeat this several times for each subsampling\n+depth, and vary the depths we use, we can start to approximate a relationship\n+between the number of samples we use and the rate of false negatives, also\n+called the statistical power of the test.\n+\n+To generate complete power curves from data which appears underpowered, the\n+`statsmodels.power` package can be used to solve for an effect size. The effect\n+size can be used to extrapolate a power curve for the data.\n+\n+The general format for functions in this module is to define a statistical\n+test function which will take a list of ids, and return a p value. The test is\n+then evaluated over a series of subsample sizes.\n+\n+If metadata is avaliable, there are three ways we can approach selecting our\n+sample. We may choose to simply draw n observations at random from the two\n+underlying samples. Alternatively, we can draw subsamples which are\n+significantly different. Finally, we can try to match samples based on a set\n+of control categories.\n+\n+Functions\n+---------\n+\n+.. autosummary::\n+    :toctree: generated/\n+\n+    get_subsampled_power\n+    confidence_bound\n+    bootstrap_power_curve\n+    get_significant_subsample\n+    get_paired_subsamples\n+\n+Examples\n+--------\n+Suppose we wanted to look at the power curve for two variables, `ind` and\n+`dep`, using completely random subsampling. To control for the pseudo\n+random number generation, we will use a seed.\n+\n+>>> import numpy as np\n+>>> np.random.seed(20)\n+>>> ind = np.random.randint(0, 20, 15)\n+>>> print ind\n+[ 3 15  9 11  7  2  0  8 19 16  6  6 16  9  5]\n+>>> dep = (3 * ind + 5 + np.random.randn(15)*5).round(3)\n+>>> print dep\n+[ 15.617  47.533  28.04   33.788  19.602  12.229   4.779  36.838  67.256\n+  55.032  22.157   7.051  58.601  38.664  18.783]\n+\n+Let's define a test that will draw a list of sample pairs and determine\n+if they're correlated. We'll use the `pearsonr` function from scipy, which\n+returns the pearson correlation coefficient and the probability value\n+that the data is not correlated. The function takes two vectors as its\n+input.\n+\n+>>> from scipy.stats import pearsonr\n+>>> f = lambda x: pearsonr(ind[x[0]], dep[x[0]])[1]\n+\n+Now, let's use random sampling to estimate the power of our test on\n+the first distribution. Since our test picks pairs, the \"samples\" vector\n+will just be a list of positions in each array.\n+\n+>>> samples = [np.arange(0, 15, 1)]\n+>>> print f(samples)\n+3.64594525966e-08\n+\n+Since we know the two samples are correlated overall, let's try using\n+completely random subsampling. This is recommended when sample populations\n+are of simillar size, giving each sampled contained in the population a\n+simillar probability of being drawn. If one sample is much larger than the\n+other, signifigant subsampling can help decrease some of the noise.\n+\n+>>> from skbio.stats.power import get_subsampled_power\n+>>> pwr_ests, counts = get_subsampled_power(mode=\"ALL\",\n+...                                         test=f,\n+...                                         samples=samples,\n+...                                         min_counts=3,\n+...                                         max_counts=10,\n+...                                         counts_start=3,\n+...                                         counts_interval=1)\n+>>> print counts\n+[3 4 5 6 7 8 9]\n+>>> print pwr_ests\n+[[ 0.22   0.652  0.89   0.958  0.992  1.     1.   ]\n+ [ 0.234  0.642  0.876  0.96   0.99   1.     1.   ]\n+ [ 0.242  0.654  0.848  0.946  0.998  1.     1.   ]\n+ [ 0.244  0.664  0.884  0.946  0.988  1.     1.   ]\n+ [ 0.248  0.666  0.866  0.948  0.986  1.     1.   ]\n+ [ 0.242  0.658  0.9    0.94   0.99   1.     1.   ]\n+ [ 0.242  0.638  0.874  0.952  0.992  1.     1.   ]\n+ [ 0.24   0.66   0.904  0.95   0.988  1.     1.   ]\n+ [ 0.232  0.64   0.912  0.972  0.988  1.     1.   ]\n+ [ 0.256  0.646  0.854  0.952  0.992  1.     1.   ]]\n+\n+The `pwr_est` can then be used to fit an effect size using\n+`statsmodel.stats.power` module or the results can be average and plotted.\n+\n+\"\"\"\n+\n+# -----------------------------------------------------------------------------\n+# Copyright (c) 2013--, scikit-bio development team.\n+#\n+# Distributed under the terms of the Modified BSD License.\n+#\n+# The full license is in the file COPYING.txt, distributed with this software.\n+# -----------------------------------------------------------------------------\n+\n+from __future__ import division\n+from future.utils import viewitems\n+from numpy import (array, zeros, ones, round as nround, hstack, isnan,\n+                   sqrt, arange)\n+from numpy.random import choice\n+from scipy.stats import t, nanstd\n+\n+\n+def get_subsampled_power(mode, test, meta=None, cat=None, control_cats=None,\n+                         order=None, strict=True, samples=None, sub_size=None,\n+                         scaling=5, alpha_pwr=0.05, min_counts=20,\n+                         max_counts=50, counts_interval=10, counts_start=None,\n+                         num_iter=500, num_runs=10):\n+    r\"\"\"Subsamples data to iterative calculate power\n+\n+    Parameters\n+    ----------\n+    mode : {\"ALL\", \"SIG\", \"PAIRED\"}","path":"skbio/stats/power.py","position":146,"original_position":146,"commit_id":"04cb9ffba14dc4e0b7aa4f32089bc09ca01a9c0e","original_commit_id":"04cb9ffba14dc4e0b7aa4f32089bc09ca01a9c0e","user":{"login":"ebolyen","id":3976804,"avatar_url":"https://avatars.githubusercontent.com/u/3976804?v=2","gravatar_id":"","url":"https://api.github.com/users/ebolyen","html_url":"https://github.com/ebolyen","followers_url":"https://api.github.com/users/ebolyen/followers","following_url":"https://api.github.com/users/ebolyen/following{/other_user}","gists_url":"https://api.github.com/users/ebolyen/gists{/gist_id}","starred_url":"https://api.github.com/users/ebolyen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ebolyen/subscriptions","organizations_url":"https://api.github.com/users/ebolyen/orgs","repos_url":"https://api.github.com/users/ebolyen/repos","events_url":"https://api.github.com/users/ebolyen/events{/privacy}","received_events_url":"https://api.github.com/users/ebolyen/received_events","type":"User","site_admin":false},"body":"Another note: using strings is way better than 3 different functions, so this is awesome! But, could they be lowercase? I don't think there is any precedent for using all caps for this interface type currently in scikit-bio.","created_at":"2014-09-25T15:24:49Z","updated_at":"2014-09-25T15:24:49Z","html_url":"https://github.com/biocore/scikit-bio/pull/636#discussion_r18038577","pull_request_url":"https://api.github.com/repos/biocore/scikit-bio/pulls/636","_links":{"self":{"href":"https://api.github.com/repos/biocore/scikit-bio/pulls/comments/18038577"},"html":{"href":"https://github.com/biocore/scikit-bio/pull/636#discussion_r18038577"},"pull_request":{"href":"https://api.github.com/repos/biocore/scikit-bio/pulls/636"}}},"pull_request":{"url":"https://api.github.com/repos/biocore/scikit-bio/pulls/636","id":20125939,"html_url":"https://github.com/biocore/scikit-bio/pull/636","diff_url":"https://github.com/biocore/scikit-bio/pull/636.diff","patch_url":"https://github.com/biocore/scikit-bio/pull/636.patch","issue_url":"https://api.github.com/repos/biocore/scikit-bio/issues/636","number":636,"state":"open","locked":false,"title":"Power Analysis Module","user":{"login":"JWDebelius","id":5534846,"avatar_url":"https://avatars.githubusercontent.com/u/5534846?v=2","gravatar_id":"","url":"https://api.github.com/users/JWDebelius","html_url":"https://github.com/JWDebelius","followers_url":"https://api.github.com/users/JWDebelius/followers","following_url":"https://api.github.com/users/JWDebelius/following{/other_user}","gists_url":"https://api.github.com/users/JWDebelius/gists{/gist_id}","starred_url":"https://api.github.com/users/JWDebelius/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JWDebelius/subscriptions","organizations_url":"https://api.github.com/users/JWDebelius/orgs","repos_url":"https://api.github.com/users/JWDebelius/repos","events_url":"https://api.github.com/users/JWDebelius/events{/privacy}","received_events_url":"https://api.github.com/users/JWDebelius/received_events","type":"User","site_admin":false},"body":"The module imperially estimates the power of a dataset using Monte Carlo simulation. The functions take test, a function which accepts a list of sample ids or values, and returns a p-value.\r\n\r\nIn conjunction with Statsmodels, the code can be used to generate power curves, for a sufficiently large number of samples. (Depending on the effect size, about 20 - 30 in each group).\r\n\r\nI will move the statsmodel dependent example into an ipython notebook and either keep it in one of my repositories or else submit to Greg's Introduction to Applied Bioinformatics, unless statsmodel can be added.\r\n\r\n*This code likely needs to be optimized, given the iterative nature*","created_at":"2014-08-21T17:53:47Z","updated_at":"2014-09-25T15:24:49Z","closed_at":null,"merged_at":null,"merge_commit_sha":"ae412603ea97a2b09636279710c28948cd3c1ca7","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/biocore/scikit-bio/pulls/636/commits","review_comments_url":"https://api.github.com/repos/biocore/scikit-bio/pulls/636/comments","review_comment_url":"https://api.github.com/repos/biocore/scikit-bio/pulls/comments/{number}","comments_url":"https://api.github.com/repos/biocore/scikit-bio/issues/636/comments","statuses_url":"https://api.github.com/repos/biocore/scikit-bio/statuses/04cb9ffba14dc4e0b7aa4f32089bc09ca01a9c0e","head":{"label":"JWDebelius:Power","ref":"Power","sha":"04cb9ffba14dc4e0b7aa4f32089bc09ca01a9c0e","user":{"login":"JWDebelius","id":5534846,"avatar_url":"https://avatars.githubusercontent.com/u/5534846?v=2","gravatar_id":"","url":"https://api.github.com/users/JWDebelius","html_url":"https://github.com/JWDebelius","followers_url":"https://api.github.com/users/JWDebelius/followers","following_url":"https://api.github.com/users/JWDebelius/following{/other_user}","gists_url":"https://api.github.com/users/JWDebelius/gists{/gist_id}","starred_url":"https://api.github.com/users/JWDebelius/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JWDebelius/subscriptions","organizations_url":"https://api.github.com/users/JWDebelius/orgs","repos_url":"https://api.github.com/users/JWDebelius/repos","events_url":"https://api.github.com/users/JWDebelius/events{/privacy}","received_events_url":"https://api.github.com/users/JWDebelius/received_events","type":"User","site_admin":false},"repo":{"id":23196783,"name":"scikit-bio","full_name":"JWDebelius/scikit-bio","owner":{"login":"JWDebelius","id":5534846,"avatar_url":"https://avatars.githubusercontent.com/u/5534846?v=2","gravatar_id":"","url":"https://api.github.com/users/JWDebelius","html_url":"https://github.com/JWDebelius","followers_url":"https://api.github.com/users/JWDebelius/followers","following_url":"https://api.github.com/users/JWDebelius/following{/other_user}","gists_url":"https://api.github.com/users/JWDebelius/gists{/gist_id}","starred_url":"https://api.github.com/users/JWDebelius/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JWDebelius/subscriptions","organizations_url":"https://api.github.com/users/JWDebelius/orgs","repos_url":"https://api.github.com/users/JWDebelius/repos","events_url":"https://api.github.com/users/JWDebelius/events{/privacy}","received_events_url":"https://api.github.com/users/JWDebelius/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/JWDebelius/scikit-bio","description":"Core objects, functions and statistics for working with biological data in Python.","fork":true,"url":"https://api.github.com/repos/JWDebelius/scikit-bio","forks_url":"https://api.github.com/repos/JWDebelius/scikit-bio/forks","keys_url":"https://api.github.com/repos/JWDebelius/scikit-bio/keys{/key_id}","collaborators_url":"https://api.github.com/repos/JWDebelius/scikit-bio/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/JWDebelius/scikit-bio/teams","hooks_url":"https://api.github.com/repos/JWDebelius/scikit-bio/hooks","issue_events_url":"https://api.github.com/repos/JWDebelius/scikit-bio/issues/events{/number}","events_url":"https://api.github.com/repos/JWDebelius/scikit-bio/events","assignees_url":"https://api.github.com/repos/JWDebelius/scikit-bio/assignees{/user}","branches_url":"https://api.github.com/repos/JWDebelius/scikit-bio/branches{/branch}","tags_url":"https://api.github.com/repos/JWDebelius/scikit-bio/tags","blobs_url":"https://api.github.com/repos/JWDebelius/scikit-bio/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/JWDebelius/scikit-bio/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/JWDebelius/scikit-bio/git/refs{/sha}","trees_url":"https://api.github.com/repos/JWDebelius/scikit-bio/git/trees{/sha}","statuses_url":"https://api.github.com/repos/JWDebelius/scikit-bio/statuses/{sha}","languages_url":"https://api.github.com/repos/JWDebelius/scikit-bio/languages","stargazers_url":"https://api.github.com/repos/JWDebelius/scikit-bio/stargazers","contributors_url":"https://api.github.com/repos/JWDebelius/scikit-bio/contributors","subscribers_url":"https://api.github.com/repos/JWDebelius/scikit-bio/subscribers","subscription_url":"https://api.github.com/repos/JWDebelius/scikit-bio/subscription","commits_url":"https://api.github.com/repos/JWDebelius/scikit-bio/commits{/sha}","git_commits_url":"https://api.github.com/repos/JWDebelius/scikit-bio/git/commits{/sha}","comments_url":"https://api.github.com/repos/JWDebelius/scikit-bio/comments{/number}","issue_comment_url":"https://api.github.com/repos/JWDebelius/scikit-bio/issues/comments/{number}","contents_url":"https://api.github.com/repos/JWDebelius/scikit-bio/contents/{+path}","compare_url":"https://api.github.com/repos/JWDebelius/scikit-bio/compare/{base}...{head}","merges_url":"https://api.github.com/repos/JWDebelius/scikit-bio/merges","archive_url":"https://api.github.com/repos/JWDebelius/scikit-bio/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/JWDebelius/scikit-bio/downloads","issues_url":"https://api.github.com/repos/JWDebelius/scikit-bio/issues{/number}","pulls_url":"https://api.github.com/repos/JWDebelius/scikit-bio/pulls{/number}","milestones_url":"https://api.github.com/repos/JWDebelius/scikit-bio/milestones{/number}","notifications_url":"https://api.github.com/repos/JWDebelius/scikit-bio/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/JWDebelius/scikit-bio/labels{/name}","releases_url":"https://api.github.com/repos/JWDebelius/scikit-bio/releases{/id}","created_at":"2014-08-21T17:41:35Z","updated_at":"2014-08-14T10:07:00Z","pushed_at":"2014-09-24T06:55:51Z","git_url":"git://github.com/JWDebelius/scikit-bio.git","ssh_url":"git@github.com:JWDebelius/scikit-bio.git","clone_url":"https://github.com/JWDebelius/scikit-bio.git","svn_url":"https://github.com/JWDebelius/scikit-bio","homepage":"http://scikit-bio.org","size":29195,"stargazers_count":0,"watchers_count":0,"language":null,"has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"biocore:master","ref":"master","sha":"9a895f272e12be7ab53360cd5d1cfe94e2027dde","user":{"login":"biocore","id":6241857,"avatar_url":"https://avatars.githubusercontent.com/u/6241857?v=2","gravatar_id":"","url":"https://api.github.com/users/biocore","html_url":"https://github.com/biocore","followers_url":"https://api.github.com/users/biocore/followers","following_url":"https://api.github.com/users/biocore/following{/other_user}","gists_url":"https://api.github.com/users/biocore/gists{/gist_id}","starred_url":"https://api.github.com/users/biocore/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/biocore/subscriptions","organizations_url":"https://api.github.com/users/biocore/orgs","repos_url":"https://api.github.com/users/biocore/repos","events_url":"https://api.github.com/users/biocore/events{/privacy}","received_events_url":"https://api.github.com/users/biocore/received_events","type":"Organization","site_admin":false},"repo":{"id":15168264,"name":"scikit-bio","full_name":"biocore/scikit-bio","owner":{"login":"biocore","id":6241857,"avatar_url":"https://avatars.githubusercontent.com/u/6241857?v=2","gravatar_id":"","url":"https://api.github.com/users/biocore","html_url":"https://github.com/biocore","followers_url":"https://api.github.com/users/biocore/followers","following_url":"https://api.github.com/users/biocore/following{/other_user}","gists_url":"https://api.github.com/users/biocore/gists{/gist_id}","starred_url":"https://api.github.com/users/biocore/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/biocore/subscriptions","organizations_url":"https://api.github.com/users/biocore/orgs","repos_url":"https://api.github.com/users/biocore/repos","events_url":"https://api.github.com/users/biocore/events{/privacy}","received_events_url":"https://api.github.com/users/biocore/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/biocore/scikit-bio","description":"Core objects, functions and statistics for working with biological data in Python.","fork":false,"url":"https://api.github.com/repos/biocore/scikit-bio","forks_url":"https://api.github.com/repos/biocore/scikit-bio/forks","keys_url":"https://api.github.com/repos/biocore/scikit-bio/keys{/key_id}","collaborators_url":"https://api.github.com/repos/biocore/scikit-bio/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/biocore/scikit-bio/teams","hooks_url":"https://api.github.com/repos/biocore/scikit-bio/hooks","issue_events_url":"https://api.github.com/repos/biocore/scikit-bio/issues/events{/number}","events_url":"https://api.github.com/repos/biocore/scikit-bio/events","assignees_url":"https://api.github.com/repos/biocore/scikit-bio/assignees{/user}","branches_url":"https://api.github.com/repos/biocore/scikit-bio/branches{/branch}","tags_url":"https://api.github.com/repos/biocore/scikit-bio/tags","blobs_url":"https://api.github.com/repos/biocore/scikit-bio/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/biocore/scikit-bio/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/biocore/scikit-bio/git/refs{/sha}","trees_url":"https://api.github.com/repos/biocore/scikit-bio/git/trees{/sha}","statuses_url":"https://api.github.com/repos/biocore/scikit-bio/statuses/{sha}","languages_url":"https://api.github.com/repos/biocore/scikit-bio/languages","stargazers_url":"https://api.github.com/repos/biocore/scikit-bio/stargazers","contributors_url":"https://api.github.com/repos/biocore/scikit-bio/contributors","subscribers_url":"https://api.github.com/repos/biocore/scikit-bio/subscribers","subscription_url":"https://api.github.com/repos/biocore/scikit-bio/subscription","commits_url":"https://api.github.com/repos/biocore/scikit-bio/commits{/sha}","git_commits_url":"https://api.github.com/repos/biocore/scikit-bio/git/commits{/sha}","comments_url":"https://api.github.com/repos/biocore/scikit-bio/comments{/number}","issue_comment_url":"https://api.github.com/repos/biocore/scikit-bio/issues/comments/{number}","contents_url":"https://api.github.com/repos/biocore/scikit-bio/contents/{+path}","compare_url":"https://api.github.com/repos/biocore/scikit-bio/compare/{base}...{head}","merges_url":"https://api.github.com/repos/biocore/scikit-bio/merges","archive_url":"https://api.github.com/repos/biocore/scikit-bio/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/biocore/scikit-bio/downloads","issues_url":"https://api.github.com/repos/biocore/scikit-bio/issues{/number}","pulls_url":"https://api.github.com/repos/biocore/scikit-bio/pulls{/number}","milestones_url":"https://api.github.com/repos/biocore/scikit-bio/milestones{/number}","notifications_url":"https://api.github.com/repos/biocore/scikit-bio/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/biocore/scikit-bio/labels{/name}","releases_url":"https://api.github.com/repos/biocore/scikit-bio/releases{/id}","created_at":"2013-12-13T16:24:41Z","updated_at":"2014-09-25T15:22:01Z","pushed_at":"2014-09-24T17:11:06Z","git_url":"git://github.com/biocore/scikit-bio.git","ssh_url":"git@github.com:biocore/scikit-bio.git","clone_url":"https://github.com/biocore/scikit-bio.git","svn_url":"https://github.com/biocore/scikit-bio","homepage":"http://scikit-bio.org","size":37124,"stargazers_count":66,"watchers_count":66,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":40,"mirror_url":null,"open_issues_count":103,"forks":40,"open_issues":103,"watchers":66,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/biocore/scikit-bio/pulls/636"},"html":{"href":"https://github.com/biocore/scikit-bio/pull/636"},"issue":{"href":"https://api.github.com/repos/biocore/scikit-bio/issues/636"},"comments":{"href":"https://api.github.com/repos/biocore/scikit-bio/issues/636/comments"},"review_comments":{"href":"https://api.github.com/repos/biocore/scikit-bio/pulls/636/comments"},"review_comment":{"href":"https://api.github.com/repos/biocore/scikit-bio/pulls/comments/{number}"},"commits":{"href":"https://api.github.com/repos/biocore/scikit-bio/pulls/636/commits"},"statuses":{"href":"https://api.github.com/repos/biocore/scikit-bio/statuses/04cb9ffba14dc4e0b7aa4f32089bc09ca01a9c0e"}}}}