Wikipedia talk:Scientific peer review/Archive 2

message: ===Structure of the Earth===
If you want to wet your teeth on something more technical, but not generally controversial, how about Structure of the Earth.  It is a substantially technical article with only a small number of references, mostly to other websites, and few interesting problems like the percent compositions not equally 100%.  Dragons flight 01:47, 16 March 2006 (UTC)
replies: text: Well, my review of science is already in. - Samsara (talk • contribs) 02:22, 16 March 2006 (UTC)
replies: text: I don't see anything wrong with using science as a place to start. We found an error, and that probably means there are more. let the experts fact check it. Let those with less expertise do stuff like structure and organization. We need to get the word out that we welcome reviews from non-board members—aka regular peer review.--HereToHelp (talk • contribs) 03:16, 16 March 2006 (UTC)

text: I still think that articles such as "supergravity" are the wrong things for this board to focus on.  For example, WP mathematics has 13 thousand articles on math; I'd guess that 90% are in the "supergravity" category, requiring true domain experts. This board will choke on the volume. By contrast, WP math *does* have a semi-workable review/triage process; its rather informal, but quite effective. We pile onto Wikipedia:WikiProject Mathematics/Current activity, and whack on what we find/what we know about.  Other projects could very well benefit by erecting similar infrastructure. By contrast, I am starting to feel more and more strongly that this board, and this process, needs to focus on reviewing, for scientific accuracy, the 1% to 5% of WP science articles that are on general science topics. linas 00:36, 17 March 2006 (UTC)

text: (p.s. the "current activity" page has the counter of all math articles: 13297 as of this instant).linas 00:38, 17 March 2006 (UTC)
thread_title: Alternative suggestions for guinea pig review

message: If you want to wet your teeth on something more technical, but not generally controversial, how about Structure of the Earth.  It is a substantially technical article with only a small number of references, mostly to other websites, and few interesting problems like the percent compositions not equally 100%.  Dragons flight 01:47, 16 March 2006 (UTC)
thread_title: [[Structure of the Earth]]

message: :Well, my review of science is already in. - Samsara (talk • contribs) 02:22, 16 March 2006 (UTC)
replies: text: I don't see anything wrong with using science as a place to start. We found an error, and that probably means there are more. let the experts fact check it. Let those with less expertise do stuff like structure and organization. We need to get the word out that we welcome reviews from non-board members—aka regular peer review.--HereToHelp (talk • contribs) 03:16, 16 March 2006 (UTC)
thread_title: [[Science]]

message: I was wondering about a trial of supergravity, not a trial of a review but more a trial of whether we could find a reviewer. Looking at the article raised some new issues. JarahE, who has nominated for the Board, could, from his nomination, do the review. However he has edited the article considerably in the last three days. Charles Mathews had an edit on it earlier, although it is only a modification to a category. Linas, who raised this article, has extensively edited it. Surely it would not do to have, as a reviewer, someone who had helped to write the article. So it might be difficult to find someone. This raises another issue. It might be best if the nomination of an article was only put on the Project page first, leaving the tag on the article's talk page to be added when we know we can review it. Maybe it does not matter. I do not know.
I think the process would be this. The Board or perhaps one person (this role might rotate) concludes that the expertise to review this article is not available among the Board members and reviewers list. S/he puts on this talk page:-
* Article XXX has been nominated.
* There appears to be no sufficient expertise on the Board or among the registered reviewers to review this article. Is this correct or does someone think they can review it?
If nobody responds after a couple of days, they add:-
*Can anyone on the Board recommend a suitable reviewer either from within the Wikipedia community or from outside.
* We seek a volunteer to review this article. If you think you have the expertise to review it, please indicate below giving a brief summary of your expertise and qualifications.
There might be a gap of a day or so between the two points above being added. If this gets a nominee to review the article, the Board consults and if they find the reviewer is expert enough, they ask them to do the review. Later the Board looks at the review and helps to put together a suitable report.
I think this is where the Board is useful. There is someone with a responsibility to try to find a reviewer. Without this, it is much less likely that someone will try to find one. --Bduke 21:48, 16 March 2006 (UTC)
replies: text: I still think that articles such as "supergravity" are the wrong things for this board to focus on.  For example, WP mathematics has 13 thousand articles on math; I'd guess that 90% are in the "supergravity" category, requiring true domain experts. This board will choke on the volume. By contrast, WP math *does* have a semi-workable review/triage process; its rather informal, but quite effective. We pile onto Wikipedia:WikiProject Mathematics/Current activity, and whack on what we find/what we know about.  Other projects could very well benefit by erecting similar infrastructure. By contrast, I am starting to feel more and more strongly that this board, and this process, needs to focus on reviewing, for scientific accuracy, the 1% to 5% of WP science articles that are on general science topics. linas 00:36, 17 March 2006 (UTC)

text: (p.s. the "current activity" page has the counter of all math articles: 13297 as of this instant).linas 00:38, 17 March 2006 (UTC)
thread_title: [[Supergravity]]

message: While I understand Samsara's point, could I suggest that we leave comments until there is an actual call to comment and vote? This candidate might withdraw. I think general comment on this talk page is OK, but let's leave the Project page for now. --Bduke 00:29, 16 March 2006 (UTC)
replies: text: I hadn't seen that it had been commented on here. - Samsara (talk • contribs) 01:07, 16 March 2006 (UTC)
thread_title: Comment on nominees

message: HereToHelp has just altered the recommended headers for sub pages with this comment - (If we're not templating the subpages but rather linking them (sounds okay to me) we should use bigger headers.). The original proposal did have the sub page "templated" i.e in {{  }} brackets, but this was changed to   . I let it go at the time, because I thought we could come back to it later. Now it seems is later. I think we should template in the sub-pages, in which case we should put the headers back to something like they were before. I grant you that this is a small technical point, but we need to address it sometime. Any thoughts? --Bduke 03:32, 16 March 2006 (UTC)
replies: text: I had assumed it was going to be templated, like is common practice everywhere else in the Wikipedia namespace. It will make for a long TOC, though. - Samsara (talk • contribs) 09:42, 16 March 2006 (UTC)
replies: text: Look at WP:PR. The page is HUGE. This is debilitating, see Wikipedia:Article size. I think it is behooving to have the pages linked to. Furthermore, it reduces the need to use miniscule header sizes and leaves us more room for sub-headers. For WP:PR, this is not an issue. But for us, we need to break down the subpages further into board and non-board comments, comments by user, and reccomendations. I think it's best to link the pages.--HereToHelp (talk • contribs) 23:19, 16 March 2006 (UTC)
replies: text: You may be right, but WP:PR has 70 pages included into it. OK, they are smaller than our review pages are going to be, but we will not have anywhere near 70. Also the templated in stuff does not count to the page size. It does not warn us that it is too big, but I agree that it is large and takes a while to load. AfD is probably longer. My proposal is to leave the current page as it is for a few days. I will then edit it to include it in the page and we leave it like that for a few days. Then we can see what people think. Common practice is to template them in and this has the advantage that they all appear in the contents and readers can see the whole picture better. However, this may be offset by size. Lets just wait, have a look at both and then decide. --Bduke 23:32, 16 March 2006 (UTC)
replies: text: Alright. My biggest complaint was with all the subheadings. Think: two equals signs for "Requests", three for the article name, four for "Board reviews" and "Non-board reviews", and  maybe five for the individual people. With  s, you never get past === ===.--HereToHelp (talk • contribs) 00:48, 17 March 2006 (UTC)
thread_title: Subpages

message: Just wondering...if SPR stands for "scientific" peer review, should it then be reflexive and apply scientific method (aim, hypothesis, equipment, Results, Discussion, Conclusion, Bibliography, etc.) to itself? Or does SPR have immunity from scientific method, and if so then how is the word "scientific" defined in the SPR acronym? Is there a "science" of SPR? Should SPR demonstrate the property "science" which the peer reviewer lays claims to?  
Let's try an example - Samsara's current SPR of the Science article. 
Title
A general model of scientific peer review in the open source user-defined environment.
Abstract
Open source user-defined scientific peer review (SPR) is an emergent property of the Wikipedia internet technology. Initial steps are taken to produce a model of SPR with the aim of SPR formalization. Two definitions of 'science' were identified as needing analysis and universally agreed definition. Results are pending.  
Aim
To define the term "science" in such a way that it will be achieve maximal agreement among the world's population of scientists.
Hypothesis
<Not sure about this...>
Scientific method says 
::"Normally hypotheses have the form of a mathematical model."
<I'm not sure what the mathematical model here is. Being mathematical it would suggest that my/our results would have to be in some way quantitative. What would be the quantities>
Hypothesis says 
::"The hypothetical-deductive method demands falsifiable hypotheses, framed in such a manner that the scientific community can prove them false (usually by observation). (Note that, if confirmed, the hypothesis is not necessarily proven, but remains provisional.)" 
<Ok so the scientific community could prove the aim false by disagreeing with the definition? This leads me to think that each definition & article needs a means of a user formally registering and displaying (a counter) ascend to the definition.>
Method
Samsara's SPR of the Science article was analyzed through the Wikipedia collaborative online method.  In particular, two definitions of science were analyzed.
Equipment
Wikipedia portal, computer, internet access.
Analysis
Two definitions of science have been obtained throughout the process of SPR.
"science is a system of knowledge acquisition" 
"science is an exercise in model selection"
Results
Pending...
Cost Benefit Analysis
Time: ?! 
$: ?!
Time/$ efficiency = ?!
Energy: ?! joules
Energy efficiency: ?!
Exergy: ?!
Embodied energy: ?! embodied joules
EROEI: ?1
Conclusions
*Firstly, it is difficult to make a formal decision of which definition of science has general acceptance by the world's scientific community. 
*Secondly, there does not seem to be a mathematical model by which a computer simulation can be generated of the phenomena. Nor does there seem an intuitive unit by why quantitative data can be entered into the model for hypothesis simulation and model selection/deselection.    
*Third, my head is going under trying to think this through.
Recommendations
*Further development of a mathematical model & unit of inquiry. 
*A proforma for SPR that follows scientific method.
*A holiday. :)
Bibliography
Science SPR
Sholto Maud 04:45, 16 March 2006 (UTC)
thread_title: SPR Method and Model

message: I think that the idea of a scientific board is great (one of the best idea so far), but I disagree with one point : it seems to me that creating a board with people of diverse origins is a bad idea. I would prefer several boards specialized on one topic (astronomy, biology, chemistry). I think that the community is large enough so that we can find seven high-level scientist in every field. The problem is see right now is that I don't know how an astronom will be able to make any correction on the content of a biology article (for example). So, the goal of the committee would fail. What do you think about creating several committees ? Poppypetty 13:11, 16 March 2006 (UTC)
replies: text: Benjamin Franklin had the same idea when he built the public library of Philadelphia - when others asked him to include their city, he replied, 'why don't you build the library for your own city?'
Let's not overstate the value of multiple expert input into each article; that's what the contributors should supply. If at least one expert in each area is on the board, that's fine: the rest will all be scientists, and so much of critiquing a scientific text is concerned with scientific method and scientific language, in which all board members should have expertise.
Having said that, I think your idea may be prophetic. If this board works well, perhaps it will evolve to subdivide itself naturally. Let's try to get the process right with this overarching board first, don't you think? Tony 14:11, 16 March 2006 (UTC)
thread_title:  Mistake 

message: Many of the issues surrounding the board / reviewers, specialist v. generalist, and other concerns are rooted in a framework of peer review that I think the community might well be better off without.  So let me propose an alternative that could address many of the same issues regarding article improvement and maintenance of scientific topics.  Technically, what follows could be created in addition to SPR, but I am inclined to believe it might be a better/more effective first step.
My radical suggestion is as follows:  How about forming a "Scientific Contributors Noticeboard", where people could come to ask for assistance in the maintenance and improvement of technical topics.  It could be broken out into disciplines, and those with appropriate technical training could be encouraged to watch the pages and contribute.  To first order, what we need is a way to get scientific expertise to those articles that need it, and I believe that creating such a noticeboard would do so while being less bogged down in the formality of peer review.
This proposal obviously doesn't address the concern that our articles are never checked by what the outside world might view as authority, and so we might still need something like SPR, but I think the Nature study shows our content is already doing pretty well without it.  As a first step, I think it makes more sense to try and build the scientific community within Wikipedia by providing a common forum for discussion of the scientific work within Wikipedia.  Dragons flight 00:22, 18 March 2006 (UTC)
replies: text: I like it ! linas 01:24, 18 March 2006 (UTC)
replies: text: Interesting. This hsould work in conjunction with SPR.--HereToHelp (talk • contribs) 02:37, 18 March 2006 (UTC)
replies: text: Ditto. No reason why we can't have two complementary programs on Wikipedia. Might be a very good thing to have. --HappyCamper 02:45, 18 March 2006 (UTC)
replies: text: The key is motivating expert contributors to review scientific articles. The Board might have done that, although I concede that there are problems in creating a hierarchy. I don't see how a noticeboard alone will motivate the required WPians to put in the required effort; there's not sufficient reward or recognition. Tony 02:47, 18 March 2006 (UTC)
replies: text: The key is that we wait, and let the Magic of the Wiki take over - the idea is that people should be intrinsically motivated enough to help out. The noticeboard simply needs to be made free and available, and it will gradually grow into something useful. Experts will come along if they see something inviting, interesting, and useful. I use the Wikipedia:Reference Desk as an example here - I will be bold and say that the participants on the reference desk are one of the strongest subgroups on Wikipedia - it took at least a few years to build up, but now it is a self sustaining forum where people can exchange questions and ideas. --HappyCamper 02:57, 18 March 2006 (UTC)
thread_title:  A new direction 

message: I decided to do some digging, and I want to share my finds. I really wanted to know how and why this Wikipedian could turn into this Wikipedian. I found that he was discontent with the running of the peer review and verification process: see this and  this. Ultimately—and you don't know how bad I feel about this—I think he felt that we didn't want him here: I told him he was welcome to withdraw if he so desired. Sure enough, he did with the edit summary: don't expect me to support this project in any way in future. After saying a few other goodbyes, he simply left. It would be one thing to withdraw his nomination—though his peer review of Science looked great—but I never said I would support him leaving. I would never fully support anyone leaving besides vandals. I don't know whether this is cumulative or some sudden epiphany, or even a misunderstanding, but give Wikipedia another chance. Samsara, please forgive this samsara and join us.--HereToHelp (talk • contribs) 02:52, 18 March 2006 (UTC)
replies: text: You've done your best to explain your perspective, and that's all you can ask of yourself. Don't let this get to you too much. Hopefully Samsara will read this, and will decide to come back. If not, then we can just hope that this project becomes more welcoming for Samsara in the future. --HappyCamper 03:05, 18 March 2006 (UTC)
replies: text: Thanks. Let's try to move on without him—there isn't much of a choice.--HereToHelp (talk • contribs) 12:12, 18 March 2006 (UTC)
replies: text: Samsara also indicates that they're nearing completion on a PhD. This is often a stressful time in one's life. I'm chalking up the wikivacation to the stress of having a prof remind you that you better get cracking or else. linas 17:14, 19 March 2006 (UTC)
replies: text: Yeah, that makes sense.--HereToHelp 17:17, 19 March 2006 (UTC)
thread_title:  On [[User:Samsara|Samsara]]'s departure... 

message: I created the page Wikipedia:Peer reviewers, so if you are available to do peer reviews at request, list yourself. --KimvdLinde 18:59, 18 March 2006 (UTC)
replies: text: Ooh...wonderful! --HappyCamper 19:38, 18 March 2006 (UTC)
thread_title:  Created page: [[Wikipedia:Peer reviewers]] 

message: So many new ideas and I do not have time to address them. I think we need someone, preferably Adam who started this all off, to try to write a summary of the current consensus with space for people to agree or disagree that each item is consensus. 
I am about to strike out (note strike out, not delete) my nomination for membership of the Board. I recommend that all other nominees do likewise. It is not clear whether there is going to be a Board. If there is, it is not clear what it would do. It is not clear what the Project is going to do. For example, the above section on Wikipedia:Peer reviewers is possibly a great idea, but it seems to me to be setting up an alternative peer review of scientific articles system before we have decided whether we want this one. I am not clear any more that I want to be on any Board that arises. It all depends on what it is supposed to do and in what sort of project. --Bduke 22:44, 18 March 2006 (UTC)
replies: text: As I indicated above, I did not follow the discussion from the beginning, and I just came with some ideas. The page I created can be deleted if needed, it is just an idea. KimvdLinde 22:48, 18 March 2006 (UTC)
replies: text: I was not getting at your idea. If we accept the broad proposal originally set out here by Adam, we are going to need something like your page, but its presence now just adds to the confusion (or at least mine) and highlights that we are actually further away from agreeing than we were a week ago. I need to be out all day today, but I will try to outline my concerns and views in more detail sometime in the next 24 hours. --Bduke 23:08, 18 March 2006 (UTC)
replies: text: Ok. KimvdLinde 23:11, 18 March 2006 (UTC)
replies: text: I've been very busy with revision of the last few days so have been absent from discussion. I'll try to find some time later this evening and attempt to summarise our consensus. --Oldak Quill 18:26, 19 March 2006 (UTC)
thread_title: Confused

message: Dear contributors
I wonder whether user Tufkat's recent innovation, the Wikipedia:Featured_Music_Project, provides ideas that might be useful in achieving our goal of raising the standards of scientific articles. Tony 06:55, 19 March 2006 (UTC)
thread_title:  WP featured music project: useful ideas? 

message: I feel motivated to make a statement of the SPR process as given by User:Bduke above.
*1. Check: Is article content supported by book references? 
::1.1 If so then print: "book reference material is verified".
::1.2 If not then print: "book reference material is not verified. SPRbot demands correct book references be provided or article will be excluded in 7 days." 
*2. Check: Is article content supported by external link references?
::2.1 If so then print: "web reference material is verified".
::2.2 If not then print: "web reference material is not verified. SPRbot demands correct web references be provided or article will be excluded in 7 days." 
*3. Check link type: Is web link "edu", "gov" (any other categories?).
::3.1 If so then print: "external link type is X ".
::3.2 Print: "Plausability of link type X is Y":
---!!!SEMANTIC ERROR: "Plausability" variable Y is undefined!!!---
*4.0 Check: If 2.1, then check external link references == standard texts. 
---!!!SEMANTIC ERROR: "standard texts" variable is undefined!!!---
::4.1. If so then print: "external link references supported by standard texts". 
::4.2. If not then print: "external link references are not supported by standard texts.
SPRbot demands the article is supported by standard texts or will be excluded in 7 days."
*5.0 Check: article type.
---!!!SEMANTIC ERROR: "article type" variable is undefined!!!---
*6.0 Check: is article type required to cite primairy literature?
---!!!SEMANTIC ERROR: "article type requirement" variable is undefined!!!---
::6.1. If so then print: "article required to cite primairy literature". 
::6.2. If not then print: "article not required to cite primairy literature". 
*7.0 Check: is article content == text book content?
---!!!SEMANTIC ERROR: "article content" variable is undefined!!!---
---!!!SEMANTIC ERROR: "text book content" variable is undefined!!!---
::7.1. If so then print: "article material is the standard stuff that appears in text books". 
::7.2. If not then print: "article content is non-standard. SPRbot requires article material be standard stuff or be excluded in 7 days". 
Not so difficult. However this hasn't formalised the criteria by which to evaluate the reliability of a source. Sholto Maud 06:49, 20 March 2006 (UTC)
replies: text: No disrespect, but reading this debate reminds me of why I gave up studying philosophy and logic, and went over to the dark-side. What passes for a sound source in one area will not be the same as another area. Presumably part of what the reviewers would do is decide whether the article and its sources match what could reasonably be expected for that area. Power (physics) represents an area for the most part that is not rapidly evolving, and thus, a textbook would likely be a good source. In fact, that it has got to the stage it has with no referencing shows just how uncontentious it is. If it were rapidly evolving, cutting edge science, something in quantum physics, or the neural mechanisms underlying ADHD, textbook material is likely to be stale. Thus, in most disciplines, cutting edge science will require journal articles or similar. Within each area, some will be more reputable, and some studies will have more robust science. A person who works in that area will be best able to judge.
OK. So perhaps I'm revealing that I work in something more approximating the 'social' sciences, but ``it's the vibe" (The Castle (film)).--Limegreen 11:07, 20 March 2006 (UTC)
thread_title: SPR Method/algorithm