Talk:Peer review/Archive 1

message: Plautus satire's comments are idiotic nonsense. I have refereed papers for five mathematics journals, and my published papers have been refereed (and some unpublished ones that I submitted). Obviously history is consulted; obviously proof-checking is not the main point of refereeing; obviously novel ideas are examined in the context of accepted notions. To say that mathematics is not a science but a "language" ignores the fact that hundreds of journals are devoted to publishing new discoveries in mathematics. Obviously in judging publication-worthiness one considers how new discoveries may be relevant to potential future research; one considers esthetics (which for most mathematicians is the main motive for doing mathematics or for learning mathematics). Whence this loony idea that there is no need to consult history? Do you not see sections on how a new discovery fits into the historical development of the subject in many research papers in mathematics?
Plautus satire's comments are those of a crackpot. Michael Hardy 22:02, 30 Apr 2004 (UTC)
replies: text: (William M. Connolley 23:18, 6 Feb 2004 (UTC)
thread_title: Plautus satire's irresponsible comments are those of a crackpot.

message: What work needs to be done that justifies the Template:citationstyle tag?  I can't find reference to it in the discussion section.  If someone could point out what should be done, I'd gladly fix it so the tag can be removed. -Verdatum (talk) 06:51, 13 December 2007 (UTC)
thread_title:  Citation Style 

message: Ha! This article failed peer review! See box above. So we better got to work on this. --Uncle Ed 22:02, 3 June 2006 (UTC)
thread_title:  Archive Comment 

message: I added a picture from the Center for Scientific Review webpage. However, since I don't have a decent image editing program handy, the orginal that I uploaded is very big and probably should be edited for size. I'm pretty sure it's public domain, since it's from the NIH, a Federal government organization. There are several other images on the same webpage, under "images of peer review" (handy, huh?).  Sayeth 19:43, 4 August 2004 (UTC)
thread_title:  Image 

message: (William M. Connolley 09:51, 18 Aug 2004 (UTC)
replies: text: You could be right - he/she (I can't really tell) is looking over the front page of what looks like a NRSA application. The front page has stuff like name, institution, and proposal title.  However, the following pages contain the grant proposal, which is peer reveiwed. From the picture, the person looking at the grant could be just checking to make sure that boxes are filled in or he/she could be starting to read the entire grant, begining at the front page with the applicant's information.  As I stated above, I got the picture from the Center for Scientific Review webpage.  Looking over all the pictures they had there, I decided that this was probably best suited to the article, since many of the other pictures depicted reviewers debating grant proposals, which would be confusing to people reading about how peer review for journal articles is done anonymously.  You're welcome to try to find another picture that better depicts peer review or just change the caption.  Personally, I think this picture is okay, since it could be interpeted as someone starting to read a grant, but improvement is always welcome. Sayeth 15:44, 18 August 2004 (UTC)
thread_title:  The picture 

message: (William M. Connolley 22:50, 18 April 2005 (UTC)
thread_title:  Spoken? 

message: Hi, I am working to encourage implementation of the goals of the Wikipedia:Verifiability policy. Part of that is to make sure articles cite their sources. This is particularly important for featured articles, since they are a prominent part of Wikipedia, and even moreso for an article like this one. The Fact and Reference Check Project has more information. Thank you, and please leave me a message when you have added a few references to the article. - Taxman 19:18, Apr 21, 2005 (UTC)
: This statement also needs a reference, particularly since "recent" is a term that loses meaning very quickly as an encyclopedic statement: 'In a recent editorial in Nature, it was stated that "in journals in those days, the burden of proof was generally on the opponents rather than the proponents of new ideas."' Rakerman 10:41, 9 June 2006 (UTC)
thread_title:  Request for references 

message: I'm in the early stages of a project on science in the medieval Arabic world and noticed that peer-review of a sort was occasionally practised by Avicenna (that article links to this page and talks of his 'possibly even decisive' contribution ) and others in the 11th century. I'll be happy to put something in once I've studied it a little more, but if someone knows more about it than me, they should probably jump in ahead. 69.144.83.104 17:24, 9 October 2005 (UTC)
thread_title:  History 

message: The preferred terminology is now 'masked', 'single-masked', and 'double-masked'.  Many journals are adopting this term instead of 'blind'.
:And yet, one of the most prominent is not – http://www.nature.com/nature/journal/v451/n7179/full/451605b.html While I personally agree with the new terminology, I have not yet, in my limited experience, ever heard of a double-masked study. And, as our job as Wikipedia editors is descriptive, rather than proscriptive writing about the world, I think both terms should be included and the difference explained in the article because both are used.  —Preceding unsigned comment added by Msanford (talk • contribs) 19:36, 1 March 2008 (UTC)
replies: text: On second thought, I may be getting over-zealous in my quest for perfection. I think it is well-put in the article (simply as 'alternatively'), though a citation would be nice to show the use of double-masked (as I just added one for double-blind). --Msanford (talk) 19:52, 1 March 2008 (UTC)
thread_title:  "blind" vs. "masked" 

message: Would it be worth mentioning that many of the specialist "peer-reviewed" medical and scientific journals are extremely expensive (thousands of dollars/year) and this presents a barrier to wide dissemination of information? Considering this along with the tremendous number of papers submitted to mainstream journals, it can be quite difficult for a controversial paper to be published in a visible peer-reviewed forum. 12.103.251.203 19:37, 7 December 2005 (UTC)
thread_title:  Cost of scientific journals? 

message: Ed, you think you can do better than that horribly constructed bit on the Krebs' Cycle? The paper was rejected, but that doesn't mean that the Krebs' cycle was never subjected to peer review. What's more, the source you listed was to your hastily executed Google search and linked to a cached version of a power point presentation by a librarian. Hardly a good source to list. This kind of editting is really subpar. --ScienceApologist 22:33, 3 June 2006 (UTC)
thread_title:  Krebs cycle 

message: I removed a number of paragraphs on "Stratfordians" which would, I feel, be better served in an article about disputes about Shakespeare's indentity, not in a general article about peer review. 70.25.91.244 12:43, 4 July 2006 (UTC)
thread_title:  Stratfordians 

message: Cut:
:Unfortunately "peer review" is a process that suffers from abuse by established "cliques." Worse it is not a process that assures future success. Ideas, discoveries and research should be evaluated on merit, not on whether or not they have been approved of by a disinterested circle of experts in any give field. That merit will only become apparent over time, when the ideas, discoveries and research has had an opportunity to work itself into the fabric of intellectual life and/or technology, as Arthur Schopenhauer pointed out several centuries ago. Academics and corporations who fear the publication of alternative views or technologies generally have good reasons for that fear. Reasons that are either anti-intellectual or anti-competitive.  	 
:As Charles Darwin pointed out it is "false evidence," not "false ideas" that are counterproductive.
All of this seems valid to me, but it lacks a source. What published author makes this sort of criticism? Michael Crichton, Benny Peiser, various IPCC lead authors? --Uncle Ed 15:57, 28 July 2006 (UTC)
thread_title:  Unsourced criticism 

message: Books are only briefly mentioned in passing. Would like to see something about peer review of books, how extensive it is, what percentage of scholarly books are peer reviewed, does the process differ from that for journals, exactly what types of scholarly books are (are not) peer reviewed. Nurg 00:19, 7 August 2006 (UTC)
replies: text: My appologies. I've had to revert you good faith addition per WP:COPYRIGHT (external Web site appears to be carrying work in violation of the creator's copyright). I left a note on your talk page.--Hu12 (talk) 08:01, 2 February 2008 (UTC)
replies: text: Thanks. I found the original source so all is well. Nurg (talk) 09:44, 2 February 2008 (UTC)
thread_title:  Peer review of books 

message: Does anyone have a reference discussing the use of anonymous vs. unanonymous peer review?
— Preceding unsigned comment added by 128.174.115.32 (talk) 20:28, 31 October 2006 (UTC)
thread_title:  Issues with anonymity 

message: It appears the controversy over peer review may be a bit semantical.  Peer review is a philosophy that when executed correctly, is an excellent way to conduct scientific publishing.  All the controversy appears to be when the system for implementing it messes up (biased reviewers, bad reviewers, etc.).
In that case, a "peer review" is not really obtained.  
I point this out as a matter of discussion about perhaps rephrasing the sections on discussing the controversies around the subject.  That is, peer review itself, as a philosophy, seems not to be the controversy.  It is the way in which it is implemented.  I would propose editing the "Problems with Peer Review" to be "Problems with Implementation of Peer Review and in Peer Review Itself."  And then editing the text to reflect the title.
— Preceding unsigned comment added by 128.174.115.32 (talk) 14:37, 3 November 2006 (UTC)
thread_title:  Semantics 

message: Another section which deserves some attention is the section on "Peer Review Failures", in which there are about three paragraphs talking about the Center for Regulatory Effectiveness (CRE)'s dissatisfaction with an editorial published in the Journal of the American Medical Association by Linda Rosenstock. While this may be of interest to some readers, I do not believe it warrants as long an explanation as is currently being provided. Its list of specific complaints, in particular, does not seem relevant to a general discussion of peer review failures. 
Could we not just add an external link to the CRE's website so interested parties can find out more about the controversy regarding the JAMA editorial? It seems to me that that would be more appropriate. 
(Yes, I previously removed this section for the same reason, and now see that it is back. While the poster clearly made an effort to be more objective in his/her discussion, I don't believe such a detailed discussion of this particular controversy is warranted or appropriate for a general article on peer review. Does anyone else have thoughts on this? Thanks.Chrissy385 23:15, 5 December 2006 (UTC)
thread_title:  Another Section in Question 

message: There are quite a few additional ones to add. Before I add them, I want to consider how:
:To avoid excessive controversy like the above, perhaps it might be better to make an article for each of the truly important cases, and give a link. It will focus the discussion where it belongs. The major ones are notable enough, and usually get coverage from various science news sources, not to mention newspapers. DGG 22:45, 15 January 2007 (UTC)
replies: text: If an example generated controversy, then it generated notability. --Iantresman 23:14, 15 January 2007 (UTC)
replies: text: Couldnt agree more--the question is simply where to best put them. I think the best way might be to give each their own article, and then have a category  Peer review controversies, or   a List of peer review controversies.  There is a lot  more to be said about peer review in general, and the individual examples and the comment will overbalance the rest. Especially when all the other go in: stem cells, Lucent, etc. etc. I'll see it if you reply here. DGG 00:49, 16 January 2007 (UTC)

text: I continue to be troubled by the paragraphs under the peer review failures section added by the CRE. I think this section is inappropriate for this article. Can we remove it until we decide where to add it? Chrissy385 19:40, 4 June 2007 (UTC)
thread_title:  peer review failures 

message: Cleaned up on the basis of removing self-publishing, blogs, listservs, book reviews, etc.  Still needs clean-up based on quality.  Even better  would be to cite some of the good ones in the article and get them in the reference list. DGG 01:04, 24 January 2007 (UTC)
thread_title:  external links 

message: Recent, authoritative meta-analyses of the scientific literature
on the subject of peer review have concluded that there is no good
evidence favoring it as a method to ensure (or even to increase
the probability of) manuscript quality, veracity, reliability, or
anything else.
According to a 2006 Cochrane systematic review -- the "gold
standard" of evaluation of the biomedical literature, if anything
is -- there exists an "absence of evidence" for the effectiveness
of peer review for assuring manuscript quality. "We could not
identify any methodologically convincing studies assessing the
core effects of peer review", say the Cochrane reviewers.[1]
Likewise, an earlier (2002) systematic review published in the
JAMA concluded that "Editorial peer review, although widely used,
is largely untested and its effects are uncertain".[2]
Most striking is the suggestion by Linkov et al in the Journal of
the Royal Society of Medicine (2006) that belief in peer review is
faith-based, rather than science-based: "we scientists have almost
complete faith in the journal process as right and unassailable.
We thus take a 'faith based' approach to research
communications... Questioning peer review is like questioning the
Bible, Quran or Torah."[3]
Even before the Cochrane review was published, doubts about the
efficacy of peer review had mounted.[4]
I submit that if there is a lack of evidentiary basis for the
value, in measurable terms, of peer review (as there is), then the
invocation of it as some sort of Holy Grail of scientific
reliability or veracity is inappropriate and indeed misleading.
Those who bandy it about in such a way should be corrected, and
referred to the relevant literature for study and self-education.
All approving mentions of peer review, or mentions of it that
suggest that it is essential or even important, should be
disregarded as reflective of ignorance of the best and
most-careful biomedical thinking and research currently available
on this subject.
At this moment, both peer-reviewed and non-peer-reviewed
publications have comparable status. Judgements about their
credibility or quality must be made on other grounds.
Anyone who disagrees with the foregoing is encouraged to present
compelling contrary documentation, comparable to what I presented;
e.g. a contrasting Cochrane systematic review, or some other
careful review of the literature on the subject.
References
1. http://www.cochrane.org/reviews/en/mr000016.html -- Cochrane Systematic Review: Editorial peer review for improving the quality of reports of biomedical studies
2. http://jama.ama-assn.org/cgi/content/abstract/287/21/2784 -- JAMA Vol. 287 No. 21, June 5, 2002, Effects of Editorial Peer Review: A Systematic Review
3. http://www.jrsm.org/cgi/content/full/99/12/596 -- J R Soc Med 2006;99:596-598, Scientific Journals are `faith based': is there science behind Peer review? Faina Linkov et al
4. http://www.infotoday.com/it/apr03/peek.shtml -- Information Today, Vol. 20 No. 4 - April 2003, Could Peer Review Be Wrong?
...................................
SNIPPETS from Linkov et al, www.jrsm.org article:
[...snip...]
Jefferson recently presented an outstanding review of peer review
and could find only 19 studies on peer review that were
scientifically sound.... As Jefferson has pointed out, there are
almost no data suggesting that the existing peer review systems
work and none to suggest that they are better than any other
system.
[...snip...]
Why hasn't peer review, IMRaD, the editorial decision process and
the overall journal process evolved into a new form of research
communication? We would argue that the reason is that this has
been due to the almost non-existent use of the scientific method
to question and test the publication process itself.... [journals]
are `faith based': we believe in them, we dare not question them.
[...snip...]
Isn't it strange that three features that are inherent to research
communication have not been looked at scientifically? There are
several possible reasons for this. The most likely is that we
scientists have almost complete faith in the journal process as
right and unassailable. We thus take a `faith based' approach to
research communications. Faith is defined as a firm belief in
something for which there is no proof. Many of us might view
questioning of the journal process as an attack on science itself.
Clearly, the scientific journal process is not a part of the
scientific method. We are taught early in our training about the
importance of learning to write articles (e.g. IMRaD), the power
of peer review and a belief in the editorial system. We do not
question the process, despite the fact that the essence of science
is questioning. Questioning peer review is like questioning the
Bible, Quran or Torah. One role of science is to help separate
science from dogma, which we should now do with journals, and
avoid a faith based approach.
[...snip...]
It is the scientific method that is central to science, not the
scientific journal. The scientific method should be central to
other research communication processes, but it is not and has not
been used to continuously improve how we communicate research.
Because of this, we are forced into a conundrum -- we cannot
change the process if the process if based upon faith, not data.
-- Alan2012 14:59, 5 February 2007 (UTC)
replies: text: I think the Cochrane study by Jefferson is important enough to be worth a section in the article, under a neutral heading.  Remember that the conclusion was not that peer-review had been shown to be worthless. Any such discussion should include a discussion of subsequent criticism of their evaluation. DGG 02:53, 6 February 2007 (UTC)
replies: text: No, of course it did not show peer-review to be worthless; that would be impossible. Please give links to the subsequent criticism, of which you are aware. Thanks. Alan2012 22:43, 19 March 2007 (UTC)
thread_title:  Lack of Evidence Favoring Peer Review As Quality Filter