Talk:Reproducibility

message: Carrionluggage is unhappy with the statement in the notes: "See the U.S. DoE 2004 Cold Fusion Review which states that half the reviewers found the evidences of excess heat convincing, and 1/3 found the evidences of nuclear reactions convincing.".
This issue has been discussed at large at the cold fusion article (in particular here), which represents the consensus that emerged.  This is not the place to repeat the debate, but here are some quick pointers:
* page 3 paragraph 3 of the report discusses the views of the reviewers on "excess heat" (starting with "The excess power...").  Please read carefully as this is the central issue of the debate.
* page 4 paragraph 2 of the report discusses the views of the reviewers on "evidences of low energy nuclear reations".  Again please read it carefully.
While the report raises many issues with the current state of the field, one cannot say it is "preponderently negative" about the evidences supporting the phenomenon, as the 2 paragraphs show.  Its successful independent reproduction convinced some, while the lack of repeatability bothered others.
If you still disagree, you are likely to disagree on the content of the cold fusion article too.  Please raise your points in the cold fusion talk page, so that it can be addressed in the proper audience. Pcarbonn 19:14, 21 December 2006 (UTC)
thread_title:  Cold fusion 

message: I removed the reference to orgone... I do not think this is a "Famous problem" of reproducibility. It is more a famous problem of a scientist gone haywire (if he (Wilhelm Reich) should be considered a scientist at all). Calling Paulo Correa & Alexandra Correa researchers is a bit far stretched as well. Looking at their CV their claims are not those that a lot scientists would even consider trying to reproduce.  —Preceding unsigned comment added by 86.87.108.95 (talk) 17:16, 2 August 2008 (UTC)
thread_title:  Orgone 

message: I very much doubt that the term "reproducible research" was first proposed or defined in 2009.  The concept must go back at least one hundred years as people were beginning to get a firm grasp of statistics: Standard Deviation, 1894; Student's t-test, 1908; etc..  I think the header and the paragraph has to be refined and restricted to Claerbout's particular usage and intention. AdderUser (talk) 20:39, 2 July 2012 (UTC)
thread_title:  "The term ''reproducible research'' was first proposed by Jon Claerbout"???? 

message: A situation where qualitative aspects are more obvious is the deformation of identical vehicles in identical conditions (i.e. speed, etc) in crash tests which is not the same for each vehicle subjected to the crash test. The deformation geometry is different and requires a topological qualitative description.--5.15.194.246 (talk) 09:28, 9 September 2013 (UTC)
thread_title: Crash test deformation results

message: The exact relation between the two concepts should be underlined more clearly.--5.15.194.246 (talk) 18:12, 8 September 2013 (UTC)
thread_title:  Relation of reproducibility and repeatability 

message: "The Nature study was itself reproduced in the journal PLOS ONE, which confirmed that a majority of cancer researchers surveyed had been unable to reproduce a result" This is not true.  The PLOS ONE paper referenced in 13 is a separate study that finds supporting results. It is not a replication of the Nature study.
It is also a stretch to say "Attempts to reproduce studies often strained relationships with the laboratories that were first to publish" The article only claims that 66.7% of conflicting results were not resolved through communication with the authors of the original publication.  The categories they use for these types of responses are "no response" and "negative or indifferent." There are no qualitative comments included in the study concerning the interactions between researchers.   
Kotfic (talk) 13:39, 14 October 2013 (UTC)
replies: text: Kotfic, I agree with your comment on "strained relationships"; I was looking in the article that is cited in support of that sentence, and I was baffled by the lack of any such statement.  So while it may be true that strained relations resulted, it isn't supported by the cited reference, and I edited out that sentence (but left the citation, which I believe was in support of earlier statements in that para). If someone thinks the sentence I removed should remain, then I think they need to provide a better citation.Mcswell (talk) 20:44, 2 June 2017 (UTC)
thread_title:  PLOS ONE paper 

message: *The ireproducibility of materials is discussed in:--5.15.0.241 (talk) 10:00, 30 November 2013 (UTC)
thread_title:  Ireproducibility of materials 

message: There are these projects http://psychfiledrawer.org/ and https://osf.io/ezcuj/ that might be important and have been widely cited like in Nature twice:  ,  or for the general public here in the Guardian: . The German version has a whole article on the topic: :de: Reproduzierbarkeit (Psychologie).
How is http://www.rrplanet.com/ related to the topic? Seems misplaced to me.Galant Khan (talk) 15:54, 8 October 2014 (UTC)
thread_title:  Psychology (and possibly misplaced link) 

message: The two terms - "reproducibility" and "replicability" - are sometimes used interchangeably. However, according to some , their meaning should be used in the following way:
* Reproducibility - is being able to reproduce the analysis of the data in order to arrive to the same numerical results.
* Replicability - is being able to reach the same scientific finding, when the experiment is conducted under similar conditions, in a secondary research lab.
This confusion should be (at some point) clarified in the article (and maybe lead to some major changes).
I am not (yet) making any changed to the article, but would first like to read other peoples opinion on this matter.
Tal Galili (talk) 21:56, 9 November 2014 (UTC)
replies: text: Here's a better link. --Dodi 8238 (talk) 16:19, 10 November 2014 (UTC)

text: Very nice discussion of these two terms on the Language Log site, with lots of examples of usage. Comtebenoit (talk) 12:39, 12 January 2016 (UTC)

text: Reproducibility is defined in the same way, both by NIST and JCGM. This is a technical standard definition, so everybody on Earth can express the same concept. I don't understand why the definition have to be different here. I had changed the definition before, but someone reverted it. I'm sorry, but I am re-reverting it. Rudolf Hellmuth (talk) 12:47, 13 April 2018 (UTC)
replies: text: Rudolf Hellmuth  Since you said that you have re-reverted the topic, should the "self-contradictory" template be removed from the page? Tibbs runner (talk) 16:30, 7 May 2018 (UTC)
replies: text: Tibbs runner, I wouldn't remove it before someone reviews this article with more attention. The ASTM standard would have to be checked too. I don't have it (the paywall, as commented above). I suppose the ASTM standard follows the GUM standard, as NIST does. Rudolf Hellmuth (talk) 10:56, 9 May 2018 (UTC)
replies: text: Rudolf Hellmuth So if we were able to get a copy of the document, and then confirm whether or not it agrees, we would be able to remove the template, but until then we don't do anything? Tibbs runner (talk) 15:16, 9 May 2018 (UTC)
replies: text: Tibbs runner -- Yeah, that's the how open projects, like Wikipedia, work. Wikipedia is dynamic. It will be fixed when someone, who has access to that document, and good-will and free time, decides to contribute to this article. I try to contribute every the time I come across to something in my work that is not properly explained on Wikipedia (see Wikipedia:WikiGnome). Feel free to track down the ASTM standard, and update the article. Just don't forget to include the reference. --Rudolf Hellmuth (talk) 16:05, 9 May 2018 (UTC)
thread_title:  "Reproducibility" vs "replicability" 

message: A new paper in Nature discusses a major problem uncovered regarding the reproducibility of tests that use commercial antibodies. This will require re-examination of a decade's worth of medical research. Something about it belongs in this article. LeadSongDog come howl! 15:36, 22 May 2015 (UTC)
replies: text: Further papers  and  in PLoS Biology examine just how rarely articles include the necessary information to enable reproduction.LeadSongDog come howl! 21:39, 6 January 2016 (UTC)
thread_title:  New analysis 

message: In a deterministic world, no experiment is GLOBALLY reproducible (ie: the dropping of the atomic bombs on Hiroshima and Nagasaki could not be repeated/reproduced).  LOCALLY (over a small region of spacetime) experiments APPEAR to be reproducible.  Ultimately, I feel the article should comment upon how the lack of free-will, or superdeterministic nature of reality, would impact upon the notion of repeatability/Reproducibility.  For an experiment to be TRULY repeatable, you have to Reproduce the EXACT circumstances in which in first occurred.  However, this cannot be done BECAUSE there are always variables (including potentially hidden variables) which change the second time the experiment occurs in comparison to the first time that the experiment occurs.  However, I would agree, that at the macroscopic level, reality does appear to be HIGHLY reproducible (in that the same local patterns in spacetime occur - eg: a fast car crash will usually lead to death for someone BUT each person can only die once - so even that pattern is not fully Repeatable/reproducible).  
ASavantDude (talk) 22:01, 12 April 2016 (UTC)
replies: text: Thanks for weighing in, User:ASavantDude. What we're doing here on Wikipedia is summarising what the best published reliable sources say about the topic, not our own opinions. The article has to explain reproducibility as the term it is used in science, not in a Wikipedian's own theory. Fortunately, a lot has been published about the topic. Another tip: overuse of capital letters in a forum like this is regarded as shouting, and people will react the same way they would to someone coming into the room and shouting at them. You can make the same point just as well without the excess capitalisation. Cheers, MartinPoulter (talk) 17:01, 27 May 2016 (UTC)
thread_title: Superdeterminism, Repeatability and Reproducibility

message: Can the reproducibility requirement be applied to stochastic processes?--79.119.213.36 (talk) 16:23, 20 August 2016 (UTC)
thread_title:  Stochastic phenomena and reproducibility 

message: There are now individual sections on this talk page for every pair of these three so I think we should unify the discussion here. The apparent source of the conflict is that different scientific communities use these words in different ways, in many cases providing their own definitions and standards which conflict with the definitions and standards of others. Recent academic papers (listed below -- both open access) have recounted the history of this debate.
* Barba, L. A. (2018). Terminologies for reproducible research. arXiv preprint arXiv:1802.03311.
* Plesser, H. E. (2018). Reproducibility vs. replicability: a brief history of a confused terminology. Frontiers in neuroinformatics, 11, 76.
In light of this, the fact that the article attempts to present a single definition of these terms is problematic. That the article proceeds to conflate them is more problematic still. In my opinion, the lack of consensus should be addressed in the introduction and expanded on in its own section, and consistent terms should be used thereafter, potentially the explicit terms proposed by Goodman, Fanelli, and Ioannidis:
* Methods Reproducibility
* Results Reproducibility
* Inferential Reproducibility
in the following publication (also available open access)
Goodman, S. N., Fanelli, D., & Ioannidis, J. P. (2016). What does research reproducibility mean?. Science translational medicine, 8(341), 341ps12-341ps12.
Neheller (talk) 16:38, 1 April 2020 (UTC)
thread_title:  Conflicting Terms: Repeatability vs. Replicability vs. Reproducibility 

message: Replication is the foundation of reproducibility, not the same thing. Am I wrong? Houshya (talk) 18:48, 29 November 2022 (UTC)
thread_title:  replication = reproducibility ?! 

message: I've added a link to Reproducibility Networks that are now quite widespread in Europe (see https://www.ukrn.org/global-networks/). I've also created the entry for Reproducibility Networks - Nicolas P. Rougier (talk) 19:06, 19 April 2024 (UTC)
thread_title:  Reproducibility Networks