Improving Peer-Reviewing:
A Case Study Triggered by the Acceptance of a Bogus Paper
Nagib Callaos
President of International Institute of Informatics and Systemics (IIIS, www.iiis.org)
Presented at the workshop on “Using the Case Method for Instruction”
Funded by The National Science Foundation
Held in College of Business of the University of South Florida, Tampa, Florida, USA,
Participants: Faculty and doctoral students interested in using the case method, developing discussion
and research cases, and employing classroom and distance technologies.
Organized and facilitated by Professor T. Gordon Gill, University of South Florida, USAPurpose:
The objectives of this very short paper is 1) to briefly describe the sequence of the search/research
activities that were triggered by the acceptance of a fake paper submitted to WMSCI 2005 and 2)
to present the different reports that were generated by means of a) literature search regarding thiskind of problem, b) the published potential solutions, and c) the implemented solution, which wasidentified by a methodological research fundamentally based on action-research, action-design,and action learning. At least 3000 hours (of senior academics, conference organizers, and journaleditors) have been invested in this case study.
In this short paper, we will make a very short description with links to other detailed and larger
papers which are being generated as a consequence of this case study and the tentative solutions
that has been implemented, which in turn might provide input for more case studies regarding thisimportant issue of improving peer reviewing processes.
Main Events
The respective main events and search/research activities have been, up to the present, thefollowing:
1. Randomly generated papers were submitted to WMSCI 2005. Some of them were identified assuch by their respective reviewers and were rejected. No reviews were received for one ofthem and then according to the published policy of the Organizing Committee, the paper wasaccepted as a non-reviewed one, because of the CVs of its respective authors (three MIT‟s
PhD students). They were told that the paper will be included in the proceedings (with anexplicit note) as a non-reviewed paper, but if the Organizing Committee received reviews
recommending the acceptance of the paper then its status would change to a peer-reviewed
one. A more detailed description, where facts were separated from reasoned opinions andjudgments, can be found at www.iiis.org/wmsci2005-facts-and-reasoned-judgements(15 pages)

2. All hell broke loose after the email acceptance was sent. Reuter distributed the news as “acomputer generated paper was accepted for presentation at a computer science conference.”
BBC, CNN, Boston Globe, etc. published the news. Half truths and blatant smearing and lies,
as well as personal attacks invaded the blogosphere related to Computer Science.
3. Our huge surprise was that, even after the above mentioned events, we received reviews
recommending the acceptance of the gibberish paper. This event couldn‟t be more astonishingand disconcerting to us. Was something wrong (unethical) with some of our reviewers? Was
something wrong with our reviewing methodology? How could we have a more effective
reviewing methodology?
4. Point 3 triggered a search process for more information and the more information we gathered
the more certain we were that we needed a reviewing methodology different to the traditionaland most used one. Parallel to the literature search (not research), we organized conversationalsessions and focus groups in the context of the 2006, 2007, and 2008 conferences. Interested
attendees of these events were asked the questions that our search was producing. Results ofthese conversational sessions were included as appendixes of the document posted at
http://www.iiis.org/nagib-callaos/peer-review/ (pages 76-107).
5. Results of the processes described in point 4 triggered action-research processes which
produced action-design and action-learning processes, in the context of an incrementallyevolutionary methodology to identify the ways of improving traditional double-blind peer
reviewing methods.
Conclusions of the Search/Research
1. The most essential conclusions were as follows
a. A high level of agreement among reputable journals’ editors regarding the low
effectiveness, weaknesses, and high frequency of failure in peer-review methods.
Combining these opinions, perceptions, and facts with the huge amount of time spent(invested?) in peer reviewing, it is easy to conclude that we are facing an important
problem that require some solutions. It is estimated that 15.000.000 of yearly hours ofwork are used in peer reviewing processes (more than what the USA invested in thewhole Genome Project); about one billion dollars each year while (according to asurvey of members of the Scientific Research Society) “only 8% agreed that „peer
review work well as it is‟.” So, is peer-reviewing cost-effective? Details regarding thehigh level of agreement regarding the low level of effectiveness of peer review can befound in pages 1-20 of the report posted at http://www.iiis.org/nagib-callaos/peerreview/b. No agreement regarding a standard peer-reviewing methodology.
c. Lack of agreement regarding the meaning of “Peer” and “Peer-Review.” More details at
http://www.iiis.org/nagib-callaos/meaning-of-peer-review
andat
http://peerreviewing.wordpress.com/2012/05/19/meanings-of-peer-and-peer-review/d. Lack of agreement about what a conference is and what are, or should be, conferences‟
objectives. In one extreme, some conferences have peer reviewing standards similar tojournals in the respective discipline. In the other extreme, there are reputableconferences with no peer-review at all. Examples are the meetings of the American
Mathematical Society: AMS, The Southeastern International Conference on
Combinatorics,
Graph
Theory,andComputing,etc.
(http://blog.computationalcomplexity.org/2007/11/unrefereed-does-not-equalbogus.html). Different disciplines have different conceptions regarding this issue. Then,
what should a multidisciplinary conference do with this regard?
e. Lack of explicitly written information regarding what a conference‟s proceedings is andwhat it should contain.
f. Disagreement among different disciplines with regards to their conceptions of what“conferences” are for and what is, or should be, the functions of their respectiveproceedings. Consequently, what should a multidisciplinary conference do regarding
this issue?
g. A more adequate reviewing methodology was needed, especially for multi-disciplinary
conferences organized for inter-disciplinary communication.
Potential solutions:With the above mentioned results of our search, we tried to design and implement a Reviewing
Methodology for a multi-disciplinary conference and to explicitly publish what we understand byeach of the concepts, objectives, functions, and notions where no explicit standards or implicit
agreement exist. The meta-methodological process we have been (and we are still following)
following is based on a combination of action-research, action-design, and action-learning in thecontext of an evolutionary, incremental, and cybernetic process.
Up to the present we obtained the following results
1. We identified the objectives of peer-reviewing: pages 20-35 of the report posted at
http://www.iiis.org/nagib-callaos/peer-review/
2. We identified the meaning of Peer-Review, or what we understand by it, and published in theIIIS‟s conferences web sites and at www.academia.edu/4437203/Meaning_of_Peer_Review
3. We proposed possible solution in pages 35-39 of the document mentioned in point 1. Thissolution has already been implemented with a reasonable level of effectiveness and success.
4. We proposed A Systemic Model of Scholarly and Professional Publishing and the architectureof its respective supporting information system in pages 39-61 of the document mentioned inpoint1.
(alsoathttps://www.academia.edu/4437267/SystemicCybernetic_model_for_reviewing_and_publishing). We implemented about the 80% of what

has been proposed but because of financial lack of support the proposed system has not yet
been completely developed.
5. We proposed and we are working with a three-tier reviewing methodology:a. Traditional double-blind with a minimum of 3 reviewers and with an average of about 4
actual reviews as reported in the forewords of the respective proceedings.
b. Non-anonymous, non-blind with a maximum of three reviewers.
c. Peer-to-peer reviewing (the reasoning supporting this kind of review is presented inpages 61-67 of the above mentioned document.
More details regarding this methodology can be found in “A Multi-Methodological Reviewing
Process for Multi-Disciplinary Conferences” that is being posted at all conferences sites, e.g.
http://www.iiis2014.org/wmsci/Website/MMRPfMDC.asp?vc=1 A short description of a basic
two-tier methodology has been posted at http://iiis.org/peer-reviewing.asp
6. We posted in all conferences sites what are, for us, the objectives of conferences and thefunctions of the respective proceedings. What we posted was the results of many
conversational sessions and focus groups with attendees of our conferences.
http://www.iiis2014.org/wmsci/Website/FunctionsofConferencesProceedings.asp?vc=1
7. We have been successfully using a newly designed two-tier methodology for Peer Reviewing
in which we combine traditional double-blind peer reviewing as a necessary condition, but not
as a sufficient one. A non-blind peer reviewing is also required in the methodology we areusing since 2006. A short description of this methodology can be found at page
http://www.iiisci.org/Journal/SCI/Methodology.pdf

We posted in the web as many documents as we could in order to continue with the collective
efforts of the IIIS’s members and its conferences’ attendees in contributing for a continuing
improvement of the effectiveness in peer reviewing and in adapting the objectives of theconferences and the functions of its respective proceedings to the users of our conferences, who
are their actual attendees. Continuing with this process is the essence of the meta-methodologicalprocess we are following which combines action-research, action-design, and action-learning in
the context of an evolutionary, incremental, and cybernetic process, by means of collective
contributions to this process.

A Significantly Indicative Event Happened After the Presentation Was Made at theWorkshop (which was resumed above)
The peer-reviewing methodology, briefly described above and in the linked references seems, tohave been quite effective especially if we take into account that “The publishers Springer andIEEE are removing more than 120 papers from their subscription services after a French
researcher discovered that the works were computer-generated nonsense.”(http://www.nature.com/news/publishers-withdraw-more-than-120-gibberish-papers-

1.14763?WT.mc_id=TWT_NatureNews). Since 2006, all fake papers we received were identified
by our two-tier methodology which is described with more details at
http://www.iiisci.org/journal/sci/Methodology.pdf and http://www.iiis.org/acceptance-policy.asp.
Even we cannot prove that our methodology is more effective (but less efficient because it requires
more persons-hours in peer reviewing and acceptance processes), we have several reasons andindicators to believe that it is definitely more effective. One of this indicators is the recent news
regarding prestigious publishers trying to remove about 120 fake papers from their publications,while no case has been presented up to the present with our two-tier methodology.