Talk:Multiple comparisons problem

message: I think that the lead section should be a little more accessable.  The big picture in plain language.  There is plenty of room for the subtulties of the concept further down. ike9898 01:59, 8 October 2005 (UTC)
replies: text: I agree. I don't know what the sentence "The experimentwise α level increases exponentially as the number of comparisons increases." means. What is an α level, or where do I go to look it up? Not really a field I know that much about, so I look forward to a more clear article. -- Jake 07:13, 15 October 2005 (UTC)
replies: text: I agree as well, and will take a crack at an edit with a more accessible intro, taking into account the current trent in multiple comparisons v testing (above). Debivort 08:10, 3 January 2006 (UTC)
thread_title:  Lead section 

message: There is a nice summary of this by NIST at  which I believe is in the public domain, as NIST is a US government agency. In fact I made a template for this: NIST-PD. Btyner 18:57, 15 May 2006 (UTC)
thread_title:  Tukey's Studentized Range Test/Distribution 

message: There is a section in the middle of the article that is repeated word-for-word later in the article.  Please fix.  Thanx.  --Cromwellt 5PM, 16 Feb 2007 (having login problems) —The preceding unsigned comment was added by 67.142.130.42 (talk) 23:03, 16 February 2007 (UTC)
thread_title:  Redundancy 

message: Recent reversions may be problematic as they restore claims about Bonferroni that are uncited and have already been called into dispute. Can some of the other editors weigh in on this? de Bivort 15:38, 13 November 2007 (UTC)
thread_title:  RVs 

message: The paragraph says:
"If the inferences are hypothesis tests rather than confidence intervals, the same issue arises. With just one test performed at the 5% level, there is only a 5% chance of incorrectly rejecting the null hypothesis if the null hypothesis is true.  However, for 100 tests where all null hypotheses are false, the expected number of incorrect rejections is 5.  If the tests are independent, the probability of at least one incorrect rejection is 99.4%.  These errors are called false positives."
shouldn't be "However, for 100 tests where all null hypotheses are true, the expected number of incorrect rejections is 5."?
that is, we reject it when we think is false (based on the alpha level). In this case the problem arise because we reject it even if it is true. Sorry if I misunderstood- non-statistician here.Diego Diez 13:22, 23 September 2010 (UTC)
thread_title: Multiple comparisons for confidence intervals and hypothesis tests

message: I recommend the multiple comparison book by Tamhane & Hochberg. They have also worked with the wizards at the  SAS Institute to publish a user-friendly guide (and also a workbook) on multiple comparisons. These sources should be authoritative and mainstream.  Kiefer.Wolfowitz 10:13, 24 August 2011 (UTC)
thread_title:  Further reading 

message: I suggest a re-thinking of the purpose and goal of this section. The
main problem is that nothing in this section appears elsewhere in the
article, so it should be deleted if no more work is done to make it
useful.
The section comes at an important point in the article and risks
throwing the reader off-track. The section throws a lot of variables
at the reader, forcing him or her to ponder a complicated table just
to come up with some pretty intuitive and obvious ideas, such as that
the number of false positives and true positives add up to the number
of discoveries. Do we need all this quasi-math to know that?
The table is confusing, mostly because there is no clear relation
between rows and columns. I believe "Declared significant" means
"Researchers believe alternative hypothesis to be true", and "Declared
non-significant" means "Researchers believe null hypothesis to be
true". This relabeling might make the table clearer.
So the question comes down to: what do you want to teach the reader?
Currently, the section teaches nothing worthwhile.  — Preceding unsigned comment added by AndrewOram (talk • contribs) 11:36, 22 April 2014 (UTC)
thread_title:  Unhelpful section: "Classification of m hypothesis tests" 

message: Dr. Wolf has reviewed this Wikipedia page, and provided us with the following comments to improve its quality:
We hope Wikipedians on this talk page can take advantage of these comments and improve the quality of the article accordingly.
We believe Dr. Wolf has expertise on the topic of this article, since he has published relevant scholarly research:
*Reference : Joseph P. Romano & Azeem M. Shaikh & Michael Wolf, 2009. "Hypothesis testing in econometrics," IEW - Working Papers 444, Institute for Empirical Research in Economics - University of Zurich.
ExpertIdeasBot (talk) 22:42, 24 September 2016 (UTC)
thread_title:  Dr. Wolf's comment on this article 

message: My issues with this article are:
1) the controlling procedures section is a repetitive since its merge with multiple testing correction so that needs to be condensed and cleaned up
2) the large-scale multiple testing section is largely missing citations for its claims 
Jbrowning17 (talk) 03:18, 18 October 2016 (UTC)
thread_title:  Criticism  

message: I must say it is really ridiculous that something that can be found so easily using a search engine (as I mentioned in the original edit summary), would still need to have a source brought for it and deleted otherwise. Well here are the sources, you could just google "MCP conference XXX" (when XXX is the year of the confernce) and get these as first results, but I'll list those results anyway:
* 2011 conference
* 2013 conference
* 2015 conference
* 2017 conference
Orielno (talk) 17:43, 19 October 2016 (UTC)
thread_title:  sources for edit 

message: If m independent comparisons are performed, the family-wise error rate (FWER), is given by α¯ = 1 − ( 1 − α)^m . Hence, unless the tests are perfectly positively dependent (i.e., identical), α¯ increases as the number of comparisons increases.
The second statement, though true, doesn't follow from the first, so why "Hence?" Also, FWER is only one error rate that's been defined; what about the per-family error rate and the false discovery rate?
If we do not assume that the comparisons are independent, then we can still say: α¯ ≤ m ⋅ αpercomparison
"We can still say?" I think what was meant was that, regardless of the dependence of the tests, α¯ ≤ mαpercomparison., and the more positively dependent the tests, the more α¯ shrinks toward α.
Example: 0.2649 = 1 − ( 1 − .05 ) 6 ≤ .05 × 6 = 0.3
That "example" has no explanatory value. What's the point of simply plugging arbitrary values into the formula without providing explanation that connects those values to some useful context?
The most conservative method, which is free of dependence and distributional assumptions, is the Bonferroni correction
Not necessarily. There are procedures that can be more conservative than Bonferroni in some cases (e.g., the Benjamini–Yekutieli procedure and the Scheffé procedure). Also, note that we don't need to mention the Bonferroni procedure both here and in the next section. Also, "free of dependence and distributional assumptions" is vague and likely unclear to many readers. I suggest either being more explicit (e.g., "The Bonferroni procedure does not require any assumptions about the dependence (correlation) of the tests, and does not impose any added assumptions about the distributions of the test statistics") or simply linking to the Bonferroni correction article without going into the issue of assumptions here.
A marginally less conservative correction can be obtained by solving the equation for the family-wise error rate of m independent comparisons for αpercomparison.
That method (i.e., the Šidák procedure) is indeed marginally less conservative than Bonferroni, but it's actually marginally liberal when the tests are negatively dependent. Also, focusing on the Šidák and Holm procedures is abritrary. Numerous proceudres have been developed. Each procedures has its pros and cons, and some procedures are more generalized or specialized than others. Mentioning Bonferroni makes sense (though we don't need to do so multiple times in the article), since that's the simplest method. Why don't we just mention Bonferroni and note that there are other procedures, rather than going into detail about two particular alternatives?
Continuous generalizations of the Bonferroni and Šidák correction are presented in.[7]
That's not even a complete sentence.  — Preceding unsigned comment added by 23.242.195.76 (talk) 16:49, 27 June 2021 (UTC)
thread_title:  Problems with the "controlling procedures" section