Talk:Meta-analysis

message: The list of pitfalls is far from complete a much more comprehensive look at the pitfalls of non-primary research/meta-anlysis needs to be included. --Hontogaichiban (talk) 05:24, 30 March 2015 (UTC)
thread_title: Pitfalls

message: It's not right to say that a metaanalysis can't be a scientific study just because it can't be double blind.  It could be blind, in fact, just most of the time there isn't a point to it.  It doesn't need to be double blind, because the study isn't going to change its behavior since someone is reading it, the same as a hard scientific study involving measuring the ph balance of a rock doesn't need to be double blind, since the rock won't change its ph based on who's wearing the lab coat.  You could have a rater code a study with identifying information removed, and if you have multiple raters work on it, measure interrater reliability to help ensure quality.  The first couple sentences in the weaknesses section have some pretty strong statements with nothing to back them up, especially getting into whether or not to include bad studies in a metaanalysis.  What if the purpose of your metaanalysis is to prove that everyone is measuring a certain construct incorrectly?  You have to include bad studies, since that's what you're looking for, regardless of what Slavin says, even though there's no citation to help us know what Slavin actually said. 67.177.1.137 (talk) 21:14, 27 September 2010 (UTC)
thread_title: Weaknesses

message: I've removed an edit referencing a conference talk. If this is true, it will be easy to cite better references. Tim Vickers (talk) 03:03, 10 September 2009 (UTC)
replies: text: Found a source that refers to it, link, but this doesn't make much of this 1940s study. Tim Vickers (talk) 03:15, 10 September 2009 (UTC)
replies: text: Firstly, the identification of the "ESP-60" study as the first meta-analysis is given by at least a reference - unlike other statements in this paragraph that simply give dates, a fact, but no reference; esp. "a meta-analysis of a medical treatment was not published until 1955". Why are these not deleted while one that is referenced is deleted? Secondly, the paper identiying ESP-60 as "the first meta-analysis in the history of science" is clearly part of edited convention proceedings, not simply what the editor describes as "a conference talk"; as for most academic papers, there is no free availability on the web, and it can be most readily acquired by contacting the author, or a library or a shop. I should think that trivializing a fact or source is not appropriate for WP editors to perform in place of a reasoned argument. Thirdly, while the editor is dissatisfied with only some description of the ESP-60 study in the linked article by O'Rourke, one also finds in the latter not even a mention of the alleged 1955 study referred to in the target article. Why does that point not therefore suffer deletion? Fourthly, it would be interesting to see a discussion of culling of referenced information before it is autonomously culled. --Rodgarton 05:54, 10 September 2009 (UTC)

text: There were earlier meta-analyses, we have a citation about that. That directly conflicts with the proposed source's title, and it's already not a very reliable source, as a paper delivered at a fringe science conference. Shoemaker's Holiday Over 204 FCs served 20:16, 10 September 2009 (UTC)
replies: text: Yes, if a talk/conference abstract makes a surprising and verifiably-wrong statement, then I don't think I'd trust it as a source. Thankfully we don't have to cite it, since this source, which is indisputably reliable, discusses the point. Tim Vickers (talk) 20:54, 10 September 2009 (UTC)
replies: text: That certainly is a very reliable source.Simonm223 (talk) 20:57, 10 September 2009 (UTC)
replies: text: Indeed, but it also blows even the weakened claim for a 1940 parapsychology meta-analyses being first to cover a particular research issue out of the water with various discussions of agricultural meta-analyses from the 1930s. As well as Pearson's analysis of typhoid. We can't make a claim that Parapsychology was first using a poor-quality source that directly contradicts a high-quality one. Let's use the good source, and throw out the bogus claim.  Shoemaker's Holiday Over 204 FCs served 21:17, 10 September 2009 (UTC)
replies: text: The above respondents offer no answer to the points I have raised - particularly how unreferenced information can be maintained while referenced information is deleted. Instead, they seek to trivialize the referenced information by emotive descriptors, pushing their own opinion of the paper as self-evident truths. They also presume to critique the referenced paper by Bösch by way of its title and representation. The paper by O'Rourke certainly mentions the development and promotion of meta-analytical methods within other sciences; it does not even superficially conflict with the thesis of Bösch. Clearly, the above editors have had as their priority the elimination of the information on the basis that it involves a discipline which they do not like, and have then sought reasons to justify their elimination of it. Apart from the failure to address the substantive issues, to maintain unreferenced information, to make their points by hyperbole and partial reading rather than reasoned argument, this can be most readily seen by how they have eliminated the information: They have been so panicked to remove it, without attention to the details of the paragraph, that it now amusingly reads as: "The first meta-analysis was performed by Karl Pearson in 1904, in an attempt to overcome the problem of reduced statistical power in studies with small sample sizes; analyzing the results from a group of studies can allow more accurate data analysis. This encompassed a review of 145 reports on ESP experiments published from 1882 to 1939, and included an estimate of the influence of unpublished papers on the overall effect (the file-drawer problem)." The respondents are their own de-bunkers. I will not, however, restore the information as this culling is just another one by the advocates of marginal pseudo-skepticism on WP in their long-term campaign to rid WP of any responsible reference to psi-related information, by all irrational means available. It is better to keep the evidence of this irrationality and this campaign on WP up-front, and so to permit WP to self-demonstrate its unreliability.--Rodgarton 23:57, 10 September 2009 (UTC)

text: Crank it up. What a sad reply. 120.16.142.206 (talk) 15:21, 18 October 2009 (UTC)
thread_title: First meta-analysis?

message: This tool is presented in the article as a valid tool from deriving statistically significant data from small samples.  Despite this there is no clear evidence that the tool in fact does so.  As such I have concerns over the neutrality of the tone.Simonm223 (talk) 15:09, 10 September 2009 (UTC)
thread_title:  Neutrality 

message: I have restored various deletions following WP:NOTCENSORED. Deleting refs to conference papers is unhelpful as many technical articles rely on them. The claim of "unrefereed" is not proven as many conference proceedings actually are refereed.  Melcombe (talk) 17:07, 17 September 2009 (UTC)
thread_title: Restoration

message: Meta-analysis here is only discussed in the context of statistical studies. Such analysis can be done of more general less quantifiable arguments as well. This more general usage is not discussed at all in this article, nor is there a page that addresses this usage (to my knowledge). However, this sort of thinking is an important critical thinking skill. Specifically, it's definition should be in terms of its components. Meta, meaning transcendent, and analysis, meaning an examination, thus: a study at a higher level of abstraction. From this sort of broad definition, one might easily transition into the statistical context. Instead of immediately jumping into an editing war, I'm posting this as a discussion, because I have a feeling that this approach will be resisted from statistical practitioners, who will want to begin from the narrow definition. I do not believe this is settled usage or definition. Although Karl Pearson in 1904 used it in the statistical context http://www.economicexpert.com/a/Meta:analysis.htm, I would argue that it has great value as a philosophy technique http://pareonline.net/getvn.asp?v=2&n=8. For an example of more general usage (in an admittedly casually written article), see http://www.npr.org/blogs/alltechconsidered/2010/05/27/127218988/the-evil-of-the-progress-bar. Aaronchall (talk) 06:10, 30 May 2010 (UTC)
replies: text: I'd distinguish between 'meta-analysis', which is the statistical bit, and 'systematic review', which is the wider process of which meta-analysis is only a part. This distinction is by no means universal but it is the terminology used by the Cochrane Collaboration (see http://www.cochrane.org/glossary/). You can do a systematic review without a meta-analysis, and you can do a systematic review of qualitative studies. In this wider context I'd refer to the process of combining of evidence, with or without the use of statistics, as 'research synthesis' or 'evidence synthesis' rather than 'meta-analysis', which is typically reserved for statistical methods of combining evidence. The current article certainly needs to refer to systematic review early on in the body of the article, not just in the 'See also' section as currently. Qwfp (talk) 06:56, 30 May 2010 (UTC)
replies: text: I have adapted some of this to add to the lead so that 'systematic review' now appears in the lead. Melcombe (talk) 09:21, 13 July 2010 (UTC)
thread_title:  Overly narrow definition excludes more general usage 

message: :I'd distinguish between 'meta-analysis', which is the statistical bit, and 'systematic review', which is the wider process of which meta-analysis is only a part. This distinction is by no means universal but it is the terminology used by the Cochrane Collaboration (see http://www.cochrane.org/glossary/). You can do a systematic review without a meta-analysis, and you can do a systematic review of qualitative studies. In this wider context I'd refer to the process of combining of evidence, with or without the use of statistics, as 'research synthesis' or 'evidence synthesis' rather than 'meta-analysis', which is typically reserved for statistical methods of combining evidence. The current article certainly needs to refer to systematic review early on in the body of the article, not just in the 'See also' section as currently. Qwfp (talk) 06:56, 30 May 2010 (UTC)
replies: text: I have adapted some of this to add to the lead so that 'systematic review' now appears in the lead. Melcombe (talk) 09:21, 13 July 2010 (UTC)
thread_title: References

message: Most of this article is unreadable for someone without a statistics background.  I can here for an overview because I've been seeing meta-analysis in some of the articles I'm researching for an MBA paper.  I was trying to better understand the concept but was was quickly frustrated with the introduction section.  Do you really think that people like me, people who use encyclopedias to get a quick overview of a topic, expect to see "MA(Pm, Dm =(Px,Tx), Tm)" in the second paragraph?  I'm not saying formal definitions don't have a place at wikipedia, just put them in their place.  
Perhaps a good question to ask is "what would an educated person encountering this concept for the first time expect to find here?"
For examples of what this looks like see Statistical hypothesis testing and other high quality mathematics articles here http://en.wikipedia.org/wiki/Wikipedia:WikiProject_Mathematics/Wikipedia_1.0/Probability_and_statistics.  —Preceding unsigned comment added by 76.84.89.177 (talk) 02:28, 11 July 2010 (UTC)
replies: text: That second paragraph was added in February by an IP editor with no other contributions. I've just removed it as it was unsourced, overly technical and the end of it was pretty much incomprehensible. Qwfp (talk) 22:44, 11 July 2010 (UTC)
thread_title: Who is this article written for?

message: I found this section to be an "oversell" of this charge, especially since it lacks any sources indicating how frequently this problem occurs.  Surely, most statisticians and researchers know that some sort of systematic method of choosing studies must be used, rather than just picking the one's one wants.  
It should at least be flagged as needing a source.  —Preceding unsigned comment added by 70.143.24.208 (talk) 22:25, 23 July 2010 (UTC)
thread_title:  Dangers of agenda-driven bias--- Biased? and Unsourced 

message: I've just tagged the prominent and poorly contextualized Disadvantages and weaknesses section [adding: which I've now provisionally retitled Pitfalls] as a "personal essay". I'm not sure this was the most appropriate tag to use, but attention does need to be drawn, imo, to the unencyclopedic approach which has characterized (eg ) this page and which, despite the recent work of several editors, continues to hamper NPOV presentation. Meta-analysis (along with systematic review) is key to the production of the highest level of evidence for the efficacy of medical interventions. But you would scarcely know that from reading this page, where the Cochrane Collaboration is treated primarily as a convenient terminological reference, and the Cochrane Library is mentioned in passing in a subsection titled Dangers of agenda-driven bias. In brief, a major NPOV intervention is needed, imo. Cheers, —MistyMorn (talk) 14:24, 8 November 2012 (UTC)
replies: text: I read this and the section and, at least by the time I saw it (which granted was nearly a year after the tag) I didn't really see a POV problem.  The issues that are mentioned in this section are, indeed, pretty well known problems with meta-analysis, and I'm not sure all scholars would agree that meta-analysis necessarily offers the highest level of evidence given its limitations.  However, the comments about the Cochrane Collaboration are certainly well taken and there are probably ways to improve the section without necessarily tagging the whole thing as essay or POV.  — Preceding unsigned comment added by 97.100.165.246 (talk) 22:17, 3 October 2013 (UTC)

text: I completely disagree with this. Meta analysis is severely limited as a technique compared to primary research studies. The author of the above comment makes no reference to being aware of this. If anything, the article has strongly drifted towards a strong bias in favour of meta analysis without proper consideration of the drawbacks. Primary research is far any away preferable to meta analysis, I can't imagine any actual researcher would argue to the contrary or indeed any regulator. Admittedly meta analysis can be most useful in the absence of adequate primary research, though unfortunately it is all too often used when there is inadequate evidence even for a meta analysis.   --Hontogaichiban (talk) 05:22, 30 March 2015 (UTC)
thread_title:  Longstanding NPOV concerns about the structure and content of this page 

message: Dr. Callot has reviewed this Wikipedia page, and provided us with the following comments to improve its quality:
We hope Wikipedians on this talk page can take advantage of these comments and improve the quality of the article accordingly.
Dr. Callot has published scholarly research which seems to be relevant to this Wikipedia article:
*Reference : Martin Paldam & Laurent Callot, 2010. "Natural funnel asymmetries. A simulation analysis of the three basic tools of meta analysis," Economics Working Papers 2010-01, School of Economics and Management, University of Aarhus.
ExpertIdeasBot (talk) 17:49, 6 July 2015 (UTC)
thread_title:  Dr. Callot's comment on this article 

message: Dr. Bonache has reviewed this Wikipedia page, and provided us with the following comments to improve its quality:
We hope Wikipedians on this talk page can take advantage of these comments and improve the quality of the article accordingly.
Dr. Bonache has published scholarly research which seems to be relevant to this Wikipedia article:
*Reference : Bonache, Adrien & Maurice, Jonathan & Moris, Karen, 2010. "A Best evidence synthesis on the link between budgetary participation and managerial performance," MPRA Paper 20924, University Library of Munich, Germany.
ExpertIdeas (talk) 13:05, 5 August 2015 (UTC)
replies: text: Thank you for your suggestions. However, I think most editors will agree that edits to the article be supported.  See WP:CS. The sentences you specifically mention appear to be correct and properly referenced. juanTamad 06:14, 6 August 2015 (UTC)

text: First of all, we thank Dr. Bonache for his review of this article. For the first issue, I understand the skepticism, but I was able to find a source to back up the statement and added it to the article. For the second issue, I was unable to find any sources pointing to a meta-analysis of a medical treatment in 1955. Here are two papers that claim meta-analysis was not used in the study of medical treatments until the 1970's: , . Without a source, the 1955 assertion probably needs to be removed. The third issue prose is vague and is unreferenced. I have removed it. For the fourth issue, I agree the prose is confusing. I rewrote it as "Two-stage methods first compute summary statistics for AD from each study and then calculate overall statistics as a weighted average of the study statistics." --Mark viking (talk) 02:35, 29 May 2016 (UTC)
thread_title:  Dr. Bonache's comment on this article 

message: While discussing sourcing for another article, I found multiple sources that say that the "gold standard" for meta-analysis is the kind using individual patient data (IPD).  This isn't mentioned in the article; shouldn't it be? --Middle 8 (t • c | privacy • COI) 11:00, 26 October 2015 (UTC)
thread_title: IPD meta-analyses

message: Dr. Ogundari has reviewed this Wikipedia page, and provided us with the following comments to improve its quality:
We hope Wikipedians on this talk page can take advantage of these comments and improve the quality of the article accordingly.
Dr. Ogundari has published scholarly research which seems to be relevant to this Wikipedia article:
*Reference : Ogundari, Kolawole, 2014. "A meta-regression analysis of frontier efficiency estimates from Africa," 2014 Annual Meeting, July 27-29, 2014, Minneapolis, Minnesota 165911, Agricultural and Applied Economics Association.
ExpertIdeasBot (talk) 16:09, 31 May 2016 (UTC)
thread_title:  Dr. Ogundari's comment on this article 

message: Hello fellow Wikipedians,
I have just modified one external link on Meta-analysis. Please take a moment to review my edit. If you have any questions, or need the bot to ignore the links, or the page altogether, please visit this simple FaQ for additional information. I made the following changes:
*Added archive https://web.archive.org/web/20110524000948/http://www/ to http://www/
When you have finished reviewing my changes, you may follow the instructions on the template below to fix any issues with the URLs.
Cheers.—InternetArchiveBot (Report bug) 23:51, 8 June 2017 (UTC)
thread_title:  External links modified 

message: Hello fellow Wikipedians,
I have just modified 4 external links on Meta-analysis. Please take a moment to review my edit. If you have any questions, or need the bot to ignore the links, or the page altogether, please visit this simple FaQ for additional information. I made the following changes:
*Added archive https://web.archive.org/web/20121124154834/http://publish.uwo.ca/~elebel/documents/l%26p%282011%2Crgp%29.pdf to http://publish.uwo.ca/~elebel/documents/l%26p%282011%2Crgp%29.pdf
*Added  tag to http://dl.dropbox.com/u/85192141/2011-khalaf.pdf
*Added archive https://web.archive.org/web/20111122083418/http://tobaccodocuments.org/pm/2047231315-1318.pdf to http://tobaccodocuments.org/pm/2047231315-1318.pdf
*Added archive https://web.archive.org/web/20111102182723/http://andrewgelman.com/movabletype/mlm/JustHistory.pdf to http://andrewgelman.com/movabletype/mlm/JustHistory.pdf
*Added archive https://web.archive.org/web/20110726124334/http://statistics.stanford.edu/~ckirby/techreports/GEN/2009/2009-06.pdf to http://statistics.stanford.edu/~ckirby/techreports/GEN/2009/2009-06.pdf
When you have finished reviewing my changes, you may follow the instructions on the template below to fix any issues with the URLs.
Cheers.—InternetArchiveBot (Report bug) 14:22, 26 January 2018 (UTC)
thread_title:  External links modified (January 2018) 

message: First, the "Problems related to the statistical approach" section...
"A random effect can be present in either of these roles, but the two roles are quite distinct. There's no reason to think the analysis model and data-generation mechanism (model) are similar in form, but many sub-fields of statistics have developed the habit of assuming, for theory and simulations, that the data-generation mechanism (model) is identical to the analysis model we choose (or would like others to choose)..."
This is a direct quote from a paper of which Doi was an author, specifically Furuya-Kanamori & Doi (2016): "Angry Birds, Angry Children, and Angry Meta-Analysts: A Reanalysis".
"... As a hypothesized mechanisms for producing the data, the random effect model for meta-analysis is silly and it is more appropriate to think of this model as a superficial description..."
This is a direct quote from Hodges, who is cited there, but is also directly from Furuya-Kanamori & Doi's quote who took it from Hodges.  The page then continues with "and something we choose as an analytical tool – but this choice for meta-analysis may not work because the study effects are a fixed feature of the respective meta-analysis and the probability distribution is only a descriptive tool", which is again from Doi.
----
Next, the alternative models presented.  First, IVhet is a fixed-effects model.  Second, for univariate meta-anlysis, problems of the random-effects model are introduced yet strangely the only alternative presented here is Doi's model (IVhet) and other (some perhaps more commonly used than IVhet) ones such as maximum-likelihood-based methods, the Hartung-Snapp-Sidik-Jonkman random effects method, Bonett's fixed effects method (or the 'varying coefficient model'), etc. (and most obviously the Hunter-Schmidt/psychometric random-effects method) are ignored.
Ender-00 (talk) 02:59, 11 May 2019 (UTC)
thread_title:  This page relies way too heavily on Dr. Doi 

message: That quote is currently in the description of this pyramid graphic: 
https://en.wikipedia.org/wiki/Meta-analysis#/media/File:Hierarchy_of_Evidence.png
The graphic and especially that quote give readers an extremely misguided view of science and data.  A meta-analysis is a technique and not evidence at all.  The results of a meta-analysis are only as good as the evidence that goes into it and those can be shit. 
I will delete this graphic and quote.
Mdnahas (talk) 13:49, 11 May 2024 (UTC)
thread_title:  "A meta-analysis is the highest form of knowledge in science"