Wikipedia talk:Scientific peer review/SPR methodology

message: Just wondering...if SPR stands for "scientific" peer review, should it then be reflexive and apply scientific method (aim, hypothesis, equipment, Results, Discussion, Conclusion, Bibliography, etc.) to itself? Or does SPR have immunity from scientific method, and if so then how is the word "scientific" defined in the SPR acronym? Is there a "science" of SPR? Should SPR demonstrate the property "science" which the peer reviewer lays claims to? And also, JUST in case you didnt realise earlier (COUGH< COUGH!!) you ALLAHU AKBAR 4CHAN FTW i am blah blah rhiannon is gay shouasbdyl've though.... RHIANNON is Awesome-OOOO!!! omg >>> XD :z  
Let's try an example - Samsara's current SPR of the Science article. 
Title
A general model of scientific peer review in the open source user-defined environment.
Abstract
Open source user-defined scientific peer review (SPR) is an emergent property of the Wikipedia internet technology. Initial steps are taken to produce a model of SPR with the aim of SPR formalization. Two definitions of 'science' were identified as needing analysis and universally agreed definition. Results are pending.  
Aim
To define the term "science" in such a way that it will be achieve maximal agreement among the world's population of scientists.
Hypothesis
<Not sure about this...>
Scientific method says 
::"Normally hypotheses have the form of a mathematical model and need superior smelling devices taped to the 'rear' end of the Cumberdale." BOYS!
<I'm not sure what the mathematical model here is. Being mathematical it would suggest that my/our results would have to be in some way quantitative. What would be the quantities>
Hypothesis says 
::"The hypothetical-deductive method demands falsifiable hypotheses, framed in such a manner that the scientific community can prove them false (usually by observation). (Note that, if confirmed, the hypothesis is not necessarily proven, but remains provisional.)" 
<Ok so the scientific community could prove the aim false by disagreeing with the definition? This leads me to think that each definition & article needs a means of a user formally registering and displaying (a counter) ascend to the definition.>
Method
Samsara's SPR of the Science article was analyzed through the Wikipedia collaborative online method.  In particular, two definitions of science were analyzed.
Equipment
Wikipedia portal, computer, internet access.
Analysis
Two definitions of science have been obtained throughout the process of SPR.
"science is a system of knowledge acquisition" 
"science is an exercise in model selection"
Results
Pending...
Cost Benefit Analysis
Time: ?! 
$: ?!
Time/$ efficiency = ?!
Energy: ?! joules
Energy efficiency: ?!
Exergy: ?!
Embodied energy: ?! embodied joules
EROEI: ?1
Conclusions
*Firstly, it is difficult to make a formal decision of which definition of science has general acceptance by the world's scientific community. 
*Secondly, there does not seem to be a mathematical model by which a computer simulation can be generated of the phenomena. Nor does there seem an intuitive unit by why quantitative data can be entered into the model for hypothesis simulation and model selection/deselection.    
*Third, my head is going under trying to think this through.
Recommendations
*Further development of a mathematical model & unit of inquiry. 
*A proforma for SPR that follows scientific method.
*A holiday. :)
Bibliography
Science SPR
Sholto Maud 04:45, 16 March 2006 (UTC)
thread_title: SPR Method and Model

message: Ok Bduke lets see where we get with this (maybe I need to be beaten into submission): 
* "arguments that have already been published by a reputable publisher".
Take an example. The scientific article, maximum_power_theorem, has one cited source: H.W. Jackson (1959) Introduction to Electronic Circuits, Prentice-Hall. Which I put there so it does not count. What is left are six external links. 
::1. By what criteria do I establish whether or not any of these publisher/source external links are "reputable/reliable"?
::2. By what criteria do I establish whether or not a any of these publisher/source external links/texts are "standard"? 
I.e. how do I verify these sources?Sholto Maud 04:20, 20 March 2006 (UTC)
replies: text: No offense, but that seems like an odd example as none of those appear to be "sources", but external references added after the body of the article was written. --Limegreen 04:54, 20 March 2006 (UTC)
replies: text: None taken. So then does the 'original' article have no sources? If it has no sources what does our pseudoSPR do with the article? Sholto Maud 05:16, 20 March 2006 (UTC)

text: If I understand your argument, wouldn't you be better off with a page that was extensively referenced (ie lots of inline references/footnote type things), because the concern is whether those references are reputable sources, or have I completely misunderstoo this argument?--Limegreen 05:38, 20 March 2006 (UTC)
replies: text: In short yes, I think so. I use these articles because of their importance to science, and because they flout the general Wikipedia principle of Verfiability. Verfiability is, apparently, an important part of the imagined SPR process. An important part of verfiability is the provision of reliable sources. I would like to see a formal statement about how one establishes the reliability of a source - I think Bduke is giving us some inroads. It would seem that extensive referencing would make one "better off". But in the important article on power, the science editors seem to have the inverse point of view; i.e. no referencing makes one better off, and thus the knowledge-content of the article more reliable with no references. Remember it is verifiability, not truth that we are concerned with here. Sholto Maud 06:17, 20 March 2006 (UTC)
thread_title: Verify

message: The concept of verification appears to be a fundamental feature of the imagined SPR process. Representing the popular opinion of Wikipedia users, User:Bduke seems to agree that the Wikipedia concept of verification is the same as that used by logical positivists. A difficulty arises here because the logical positivist definition of verifiability defers to the correspondence theory of truth: "The correspondence theory of truth states that something is rendered true by the existence of a fact with corresponding elements and a similar structure". And this would appear to be what we do when conducting a scientific peer review. That is, we check whether a Wikipedia article is true to the facts of the reliable sources. But then this makes the Wikipedia philosophy of verification non-standard, because it is convcerned with "verifiability, not truth". Hence the Wikipeda theory of verification is Original Research. This would appear to qualify as a --FATAL ERROR--
Furthermore the Wikipedia theory of verifiability does not cite any reliable sources in defining the concept, thereby failing it's own conditions --FATAL ERROR-- Sholto Maud 09:50, 20 March 2006 (UTC)
thread_title: Fatal errors

message: I feel motivated to make a statement of the SPR process as given by User:Bduke above.
*1. Check: Is article content supported by book references? 
::1.1 If so then print: "book reference material is verified".
::1.2 If not then print: "book reference material is not verified. SPRbot demands correct book references be provided or article will be excluded in 7 days." 
*2. Check: Is article content supported by external link references?
::2.1 If so then print: "web reference material is verified".
::2.2 If not then print: "web reference material is not verified. SPRbot demands correct web references be provided or article will be excluded in 7 days." 
*3. Check link type: Is web link "edu", "gov" (any other categories?).
::3.1 If so then print: "external link type is X ".
::3.2 Print: "Plausability of link type X is Y":
---!!!SEMANTIC ERROR: "Plausability" variable Y is undefined!!!---
*4.0 Check: If 2.1, then check external link references == standard texts. 
---!!!SEMANTIC ERROR: "standard texts" variable is undefined!!!---
::4.1. If so then print: "external link references supported by standard texts". 
::4.2. If not then print: "external link references are not supported by standard texts.
SPRbot demands the article is supported by standard texts or will be excluded in 7 days."
*5.0 Check: article type.
---!!!SEMANTIC ERROR: "article type" variable is undefined!!!---
*6.0 Check: is article type required to cite primairy literature?
---!!!SEMANTIC ERROR: "article type requirement" variable is undefined!!!---
::6.1. If so then print: "article required to cite primairy literature". 
::6.2. If not then print: "article not required to cite primairy literature". 
*7.0 Check: is article content == text book content?
---!!!SEMANTIC ERROR: "article content" variable is undefined!!!---
---!!!SEMANTIC ERROR: "text book content" variable is undefined!!!---
::7.1. If so then print: "article material is the standard stuff that appears in text books". 
::7.2. If not then print: "article content is non-standard. SPRbot requires article material be standard stuff or be excluded in 7 days". 
Not so difficult. However this hasn't formalised the criteria by which to evaluate the reliability of a source. Sholto Maud 06:49, 20 March 2006 (UTC)
replies: text: Slashdot had a comment on the concept of an automated Article reviewer reviewer for Wikipedia today. --Ancheta Wis 00:51, 21 March 2006 (UTC)
thread_title: SPR Method/algorithm

message: * article type: What is an "article type"? How does one determine article "typeness"?
* article type requirements: How does one determine the citation "requirements" for any "article type"? On what basis does one article type have greater or lesser requirements than any other "article type?"
* citation plausability: What is "citation plausability"? How is "citation plausability" determined?
* standard text: How is a text's rating of "standardness" determined? Is there a "standardness" index?
* source reliability: How is "source reliability" evaluated? Is there a "source reliability" index? Sholto Maud 23:59, 21 March 2006 (UTC)
thread_title: Concepts that need rigorous, operational definition

message: I envisiage that an SPRbot may have the following benefits:
*save time in construction of valid articles
*provide a standardised form for prompting editors on what an article or contribution needs in order to satisfy all Wikipedia rules. 
*impartiality in the SPR process (which is what I assume a "scientific" peer review process strives for)
*control article quality
Take the Power article for example. Although the content may be correct it does (did) not have any citations to reliable sources. An SPRbot would recognise that the article has no such citations, and then perhaps post a requirement notice at the top of the article "The verification rule of wikipedia requires that a scienc article has at least X (2,3,4??) reliable sources cited please enter them in the space provided below". Then the bot could also post a XHTML form at the bottom of the article with windows for entering title date, or even simply the ISBN number. Thus an editor could simply scan the ISBN barcode in to save time with their citations.
Another SPRsub-bot could then search and enter full citation details in correct formatting from the ISBN number making referening more standardised, but more usefully, if a Wikidatabase of reliable sources and standard texts were constructed, the SPRbot could check the article against the database to make sure that the sources given qualified in terms of reliability, and or standardness in the eyes of Wikipedia reviewers. If the text is not in the database the bot would return something like "the reliability of this source has not yet been determined by SPR."
This would mean that human SPReviewers could concentrate on questions about evaluating source reliability, and content rather than the laborious task of citation verficiation. This database could be a simple process of 1 vote per USER per source about the standardness of the text. The more votes a text gets, the more reliable??  
Lastly. If the SPRbot were able to search for predicates within each article, it could see if there were any footnote citation next to the predicate. If not then the SPRbot would notify the editor's homepage... perhaps something like, "you have made an claim that has no verfiable citation. Please provide a reference in the footone form provided at the bottom of the page." 
Such a SPRbot should increase the rigor of each Wikipedia article with respect to the verifiability clause. Sholto Maud 01:01, 22 March 2006 (UTC)
thread_title: Argument for, and speculations about a SPRbot