# Peer Review: Readers in the Making of Scholarly Knowledge

David Pontille, Didier Torny

► To cite this version:

David Pontille, Didier Torny. Peer Review: Readers in the Making of Scholarly Knowledge. Martin Paul Eve; Jonathan Gray. Reassembling Scholarly Communications Histories, Infrastructures, and Global Politics of Open Access OA, MIT Press, 2020, 9780262363723. 10.7551/mit-press/11885.003.0013. hal-02972143

HAL Id: hal-02972143

https://hal.science/hal-02972143

Submitted on 20 Oct 2020

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
This PDF includes a chapter from the following book:

# Reassembling Scholarly Communications

## Histories, Infrastructures, and Global Politics of Open Access

© 2020 Massachusetts Institute of Technology

### License Terms:

Made available under a Creative Commons Attribution 4.0 International Public License

https://creativecommons.org/licenses/by/4.0/

### OA Funding Provided By:

• Arcadia Fund

• Birkbeck, University of London

The open access edition of this book was made possible by generous funding from Arcadia—a charitable fund of Lisbet Rausing and Peter Baldwin.

The title-level DOI for this work is:

doi:10.7551/mitpress/11885.001.0001
# 7 Peer Review: Readers in the Making of Scholarly Knowledge

David Pontille and Didier Torny

Who exactly assesses manuscripts submitted to journals? What are the actual conditions under which peer review is performed? How do different instances of judgment precisely coordinate with one another? To answer these questions, we consider peer review as a set of “technologies,” following Shapin and Schaffer, who showed that the experimental practice took shape in the seventeenth century, based on three technologies that were intimately linked in the production of scholarly knowledge.¹ Indeed, instead of considering manuscript evaluation as a technology set in stone, in earlier work we have shown that different eras, disciplines, and journals have had their own particular arrangements from which the main historical and contemporary criticisms have arisen.² For journal peer review is at the heart of two conflicting horizons: on the one hand, the validation of manuscripts is seen as a collective reproducible process performed to assert scientific statements; on the other hand, the dissemination of articles is considered as a means to spur scientific discussion, to raise controversies, and to challenge a state of knowledge. For example, the sharing of new results with audiences far removed from the scientific collectives that produced them was considered as sufficiently problematic by Franz J. Ingelfinger, chief editor of the *New England Journal of Medicine*, systematically to refuse to publish articles presenting results previously exposed elsewhere, notably in the general press.³ Symmetrically, the delays resulting from validation procedures have often been criticized as unacceptable barriers to the dissemination of knowledge, and from the 1990s onward these led numerous actors to organize the circulation of working papers and pre-prints.⁴ This discordancy is resolved in the concrete set of technologies of journal peer review, which define the arrangements between dissemination
and validation. If there never was such a thing as "traditional peer review," defined as a set of unified practices, reading has always been at the heart of manuscript evaluation. Hence, *who* reads, *when*, and *to what purposes* are key to understanding the shape of peer review.

## Peer Review as Reading

Throughout the history of peer review, the three judging instances (editors-in-chief, editorial committees, outside reviewers) that have gradually emerged were the first readers of submitted manuscripts.⁵ Their respective importance and the way in which their readings are coordinated may be subject to local conventions at a journal, disciplinary, or historical level. They are also marked by profound divergences due to distinct issues in manuscript evaluation. The "space of possibilities" within which these readings are conducted is a subject for public debate that leads to the invention of labels and the stabilization of categories, and to the elaboration of procedural and moral norms. For example, on the respective anonymity of authors and referees, four labels have been coined since the 1980s (see table 7.1).

These spaces of possibility currently coexist in each discipline, being attached to different scientific and moral values, pertaining to the responsibility of reviewers, objectivity of judgements, transparency of process, and equity toward authors.⁶ The different possibilities here show that Merton's "organized skepticism"⁷ and the agonistic nature of the production of scientific facts described by Latour and Woolgar are, indeed, not self-evident.⁸

The contemporary moment is characterized by reflexive readings of peer-review technologies: manuscript evaluation has itself become an object of

<caption>**Table 7.1**</caption>

Anonymity and identification labels in manuscript peer review

<table><thead><tr><th></th><th colspan="2">Reviewers</th></tr></thead><tbody><tr><th>Authors</th><td>Anonymized</td><td>Identified</td></tr><tr><th>Anonymized</th><td>Double blind</td><td>Blind review</td></tr><tr><th>Identified</th><td>Single blind</td><td>Open review</td></tr></tbody></table>

*Source:* David Pontille and Didier Torny, “The Blind Shall See! The Question of Anonymity in Journal Peer Review,” Ada 4 (2014), https://doi.org/10.7264/N3542KVW.
systematic scientific investigation.⁹ Authors, manuscripts, reviewers, journals, and readers have been scrupulously examined for their qualities and competencies, as well as for their “biases,” faults, or even unacceptable behavior. This trend has risen with the pioneering work of Peters and Ceci, who resubmitted to journals articles that they had already published, simply replacing the names of the authors and their institutions with fictitious names and making minor changes to the texts.¹⁰ Much to their surprise, almost all of the manuscripts were rejected, and, three exceptions aside, without any accusation of plagiarism. Thirty-eight years later, hundreds of studies on manuscript evaluation are now available, while the tradition of putting journals to the test with duplicate or fake papers still thrives.¹¹ The diverse arrangements of manuscript evaluation are thus themselves systematically subjected to evaluation procedures.

Peer review in the twenty-first century can also be distinguished by a growing trend: the empowerment of “ordinary” readers as new key judging instances. If editors and reviewers produce judgments, it is through a reading within a very specific framework, as it is confined to restricted interaction, essentially via written correspondence, which aims at authorizing the dissemination of manuscripts-become-articles.¹² Other forms of reading accompany publications and participate in their evaluation, independently of their initial validation. This is particularly the case through citation, commenting, sharing, and examining, which have existed for a long time but are now being more and more treated as integral technologies of open peer review, through new arrangements between dissemination and validation.¹³

## Citing Articles

With the popularization of bibliometric tools, citation counting has become a central element of journal and article evaluation. The implementation of these tools nevertheless required a series of operations on articles themselves. First, the identification of citations meant that one had to homogenize forms of referencing and isolate the references.¹⁴ From among all the texts they have read, readers thus choose those which they believe to be of essential value so as to refer specifically to them in their own manuscripts. Second, the tools made it necessary to blur the difference between reference and citation: the act of referencing relates to a given author,
whereas a citation is a new and perhaps calculable property of the source text. According to Wouters, this reversal radically modified referencing practices and literally created a new "citation culture."¹⁵ Under this condition, academic readers have become citers from the 1970s on, adding their voices to the already-published article and to the journal which validated it.

This citing activity pertains to journals (e.g., impact factor, eigenfactor), to articles (e.g., article-level metrics), to authors (e.g., h-index), or even to entire disciplines (e.g., half-life index) and institutions (e.g., a score for all international rankings). Using citation aggregation tools, it is possible equitably to assess all citers or else to introduce weighting tools relating to time span, to the reputation of the outlet, to their centrality, and so on. Highly disparate forms of intertextuality are rendered commensurable: the measured or radical criticism of a thought or result, integration within a scientific tradition, reliance on a standardized method described elsewhere, existence of data for a literary journal or meta-study, simple recopying of sources referenced elsewhere or self-promotion.¹⁶ Citation thus points toward two complementary horizons of reading: science as a system for accumulating knowledge via a referencing operation, and research as a necessary discussion of this same knowledge through criticism and commentary.

## Commenting Texts

Readers can be given a more formal place as commenters, in this view of publication as explicitly dialogical or polyphonic. Traditionally, before an article was published, comments were mainly directed toward the editor-in-chief or the editorial committee. Through open review, commenters enter into a dialogue with the authors and thus open up a space for direct confrontation.

Prior to the emergence of electronic spaces for discussion, at least two journals explicitly made prepublication commentaries the very principle behind their manuscript evaluation policy: *Current Anthropology* (CA) created in 1960 and *Behavioral and Brain Sciences* (B&BS) founded in 1978. Rather than gathering the opinions of just a few outside reviewers, they systematically contacted them in large numbers in an attempt to have the greatest possible diversity of judgments. Yet, unlike numerous other journals, where disagreements on manuscripts were seen as a problem, in this case they were considered to be "creative."¹⁷
The publication of commentaries alongside the articles themselves has existed for some time and is not a new phenomenon: "special issues" or "reports" in which a series of articles are brought together around a given theme to feed off one another after a short presentation. Similarly, the long-standing practice of a commentary followed by the author's response is common. CA and B&BS employed sophisticated versions of this technology, later known as open commentary: once a manuscript had been accepted, they invited dozens of new researchers to comment upon it, and then gave the author(s) the opportunity to provide a short response to the comments.

Finally, proposals have been made to revamp the traditional role of post-publication commenters. For a long time, these commenters acted in two elementary forms: by referring to the original article or by sending a letter to the editor. As from the 1990s, the emergence of electronic publications was seen as something that would revolutionize "post-publication peer review" (PPPR), by allowing comments and criticisms to be added to the document itself.¹⁸ However, the experiments of open commentary in PPPR have been disappointing for traditional (e.g., *Nature*) and new (e.g., *PLOS ONE*) electronic journals, as few readers seem to be willing to participate in such a technology "if [their] comments serve no identifiable purpose."¹⁹

## Sharing Papers

The readers mentioned so far have been peers of the authors of the original manuscript in a very restrictive sense: either their reading leads to a text of an equivalent nature, or it leads to a text published in the same outlet as the article. Until recently, readers other than citers and commenters remained very much in the shadows. Yet library users, students in classes, and colleagues in seminars, as just a few examples, also ascribe value to articles; for instance, through annotation.²⁰ But two major changes have rendered part of these forms of reading valuable.

The existence of articles in electronic form has made their readers more visible. People who access an "HTML" page or who download a "PDF" file are now taken into account, whereas in the past it was only the distribution of journals and texts, mostly through libraries, which allowed one to assess potential readership. By inventorying and aggregating the audience in this way, it is possible to assign readers the capacity to evaluate articles. Labels such as "highly accessed" or "most downloaded," frequently used
on journal websites, make it possible to distinguish certain articles. The creation of online academic social networks (e.g., ResearchGate, Academia .edu) has trivialized this figure of the public, not only by counting “aca- demic users,” but also by naming them and offering contact. Researchers now take part in the dissemination of their own articles and are thus better able to grasp the extent and diversity of their audiences.²¹

At the same time, other devices make visible the sharing of articles. First of all, it is online bibliographic tools (e.g., CiteULike, Mendeley, Zotero) that objectify the readers and taggers who introduce references and attached documents into their bibliographic databases. Without being citers them- selves, these readers select publications by sharing lists of references, the pertinence of which is notified by the use of “tags.” These reader-taggers are also embedded in the use of hyperlinks within “generalist” social net- works (e.g., Facebook, Twitter), by alerting others to interesting articles, or by briefly commenting on their content. These different channels for dissemination and sharing have been the object of numerous works that aimed to determine whether or not they were a means of evaluating articles compared to their citations.²² They have also been reworked by advocates of “article-level metrics.” The measurements of these different channels are now aggregated and considered to be a representation of a work’s multiple uses and audiences. For its advocates, the resulting “total impact” is the true value of a article’s importance shown through its dissemination. Here the readers, tracked by number and diversity, revalidate articles in the place of the judging instances historically qualified to do so.

Examining Documents

This movement is even more significant in that these tools are applied not
only to published articles but also to documents which have not been vali-
dated through journal peer review. Indeed, after the establishment of the
arXiv high-energy physics repository at the beginning of the 1990s, many
scientific milieus and institutions acquired repository servers to host work-
ing papers.²³ Ideally, these manuscripts are preliminary versions submitted
for criticism and comments by specialist groups that are notified of the
submissions. The resulting exchanges are managed by the system, which
archives the different versions produced. So readers do not simply exercise
their judgment on validated articles, but also produce a collective evaluation
of manuscripts. This flow of electronic manuscripts feeds the enthusiasm
of the most visionary who, since the 1990s, have been announcing the
approaching end of validation by journals' traditional judging instances.²⁴
Nevertheless, new technologies have been built on these archives, such
as "overlay journals," in which available manuscripts are later validated by
reading peers.²⁵ New journals have reembodied the old scholarly commu-
nication values of rapidity and open scientific discussion, by offering a
publishing space to working papers, such as *PeerJ*, or by publishing manu-
scripts first, then inviting commenters to undertake peer review and push-
ing authors to publish revised versions of their texts, such as *F1000Research*.

With a view to dissemination, advocates of readers as a judging instance
tend to downplay the importance of prior validation. While the valida-
tion process sorts manuscripts in a binary fashion (accepted or rejected),
such advocates contend that varied forms of dissemination instead encour-
age permanent discussion and argument along a text's entire trajectory. In
this perspective, articles remain "alive" after publication and are therefore
always subject not only to various reader appropriations, but also to public
evaluations, which can reverse their initial validation. The PubPeer web-
site, which offers anonymized readers the opportunity to discuss the valid-
ity of experiments and to ask authors to answer their questions, is a good
example of this kind of PPPR. The discussions occurring on this platform
regularly result in the debunking of faked and manipulated images from
many high-profile articles, which leads to corrections and even retractions
of the publications by the journals themselves.

Conclusion

Driven by a constant process of specialization, the extension of judging
instances to readers may appear as a reallocation of expertise, empower-
ing a growing number of people in the name of distributed knowledge.²⁶
In an ongoing context of revelations of massive scientific fraud, which
often implicates editorial processes and journals themselves, the derelic-
tion inherent to judging instances prior to publication has transformed the
mass of readers into a vital resource for unearthing error and fraud.²⁷ As in
other domains where public expertise used to be exclusively held by a few
professionals, crowdsourcing has become a collective gatekeeper for science
publishing. Thus, peerdom shall be reshaped, as lay readers have now full
access to a large part of the scientific literature and have become valued
audiences as quantified end users of published articles.²⁸

If open science has become a motto, it encompasses two different visions
for journal peer review. The first one, which includes open identities, takes
place within the academic closet, where the dissemination of manuscripts
is made possible by small discourse collectives that shape consensual facts.²⁹
This vision is supported by the validation processes designed by Robert
Boyle, one of the founders of the Royal Society, who thought that disputes
about scientific facts needed a specific and limited “social space” in order to
be solved.³⁰ By contrast, following Thomas Hobbes’s Leviathan conception
of sovereignty, the second vision urges a multiplication of points of view.
The disentanglement of peer evaluation cuts through the ability given to
readers to comment on published articles, produce social media metrics
through the sharing of documents, and observe the whole evaluation pro-
cess of each manuscript.³¹ In this vision, scholarly communication relies on
a plurality of instances that generate a continuous process of judgment. The
first vision has been at the heart of the scientific article as a genre, and a
key component of the scientific journal as the most important channel for
scholarly communication.³² Whether journals remain central in the second
vision has yet to be determined.³³

Notes

1. Steven Shapin and Simon Schaffer, *Leviathan and the Air-Pump: Hobbes, Boyle, and the Experimental Life* (Princeton, NJ: Princeton University Press, 1985).

2. David Pontille and Didier Torny, “From Manuscript Evaluation to Article Valuation: The Changing Technologies of Journal Peer Review,” *Human Studies* 38, no. 1 (2015): 57–79, https://doi.org/10.1007/s10746-014-9335-z.

3. Franz J. Ingelfinger, “Definition of Sole Contribution,” *New England Journal of Medicine* 281, no. 12 (1969): 676–677, https://doi.org/10.1056/NEJM196909182811208; see also Arnold S. Relman, “The Ingelfinger Rule,” *New England Journal of Medicine* 305, no. 14 (1981): 824–826, https://doi.org/10.1056/NEJM198110013051408.

4. Herbert Van de Sompel and Carl Lagoze, “The Santa Fe Convention of the Open Archives Initiative,” *D-Lib Magazine* 6, no. 2 (2000), https://doi.org/10.1045/february2000-vandesompel-oai.

5. Aileen Fyfe and Noah Moxham, “Making Public Ahead of Print: Meetings and Publications at the Royal Society, 1752–1892,” *Notes and Records: The Royal Society Journal of the History of Science* 70, no. 4 (2016): 361–379, https://doi.org/10.1098/rsnr.2016.0030.
6. Mark Ware, "Peer Review in Scholarly Journals: Perspective of the Scholarly Community—Results from an International Study," *Information Services & Use* 28 (2008): 109–112, https://doi.org/10.3233/ISU-2008-0568.

7. R. K. Merton, "Science and Technology in a Democratic Order," *Journal of Legal and Political Sociology* 1 (1942): 115–126.

8. Bruno Latour and Steve Woolgar, *Laboratory Life: The Construction of Scientific Facts* (Beverly Hills, CA: SAGE, 1979).

9. Juan Miguel Campanario, "Peer Review for Journals as It Stands Today—Part 1," *Science Communication* 19, no. 3 (1998): 181–211, https://doi.org/10.1177/1075547098019003002.

10. Douglas P. Peters and Stephen J. Ceci, "Peer-Review Practices of Psychological Journals: The Fate of Published Articles, Submitted Again," *Behavioral and Brain Sciences* 5, no. 2 (1982): 187–195, https://doi.org/10.1017/S0140525X00011183.

11. Daryl E. Chubin and Edward J. Hackett, *Peerless Science: Peer Review and U.S. Science Policy*, SUNY Series in Science, Technology, and Society (Albany, NY: State University of New York Press, 1990); Bruce W. Speck, ed., *Publication Peer Review: An Annotated Bibliography* (Westport, CT: Greenwood Press, 1993); Ann C. Weller, "Editorial Peer Review: Its Strengths and Weaknesses," *Journal of the Medical Library Association* 90, no. 1 (2002): 115; Jonathan P. Tennant et al., "A Multi-Disciplinary Perspective on Emergent and Future Innovations in Peer Review," *F1000Research* 6 (2017): 1151, https://doi.org/10.12688/f1000research.12037.3; John Bohannon, "Who's Afraid of Peer Review?," *Science* 342, no. 6154 (2013): 60–65, https://doi.org/10.1126/science.342.6154.60; on the last of these, see also Amy Buckland et al., "On the Mark? Responses to a Sting," *Journal of Librarianship and Scholarly Communication* 2, no. 1 (2013), https://doi.org/10.7710/2162-3309.1116.

12. Stefan Hirschauer, "Editorial Judgments: A Praxeology of 'Voting' in Peer Review," *Social Studies of Science* 40, no. 1 (2010): 71–103, https://doi.org/10.1177/0306312709335405.

13. Tony Ross-Hellauer, "What Is Open Peer Review? A Systematic Review," *F1000Research* 6 (2017): 588, https://doi.org/10.12688/f1000research.11369.2.

14. Charles Bazerman, *Shaping Written Knowledge: The Genre and Activity of the Experimental Article in Science* (Madison, WI: University of Wisconsin Press, 1988).

15. Paul Franciscus Wouters, "The Citation Culture" (PhD dissertation, University of Amsterdam, 1999).

16. Martin G. Erikson and Peter Erlandson, "A Taxonomy of Motives to Cite," *Social Studies of Science* 44, no. 4 (2014): 625–637, https://doi.org/10.1177/0306312714522871.

17. Stevan Harnad, "Creative Disagreement," *The Sciences* 19 (1979): 18–20.
18. F. W. Lancaster, "Attitudes in Academia toward Feasibility and Desirability of Networked Scholarly Publishing," *Library Trends* 43, no. 4 (1995): 741-752.

19. Kathleen Fitzpatrick, *Planned Obsolescence: Publishing, Technology, and the Future of the Academy* (New York: New York University Press, 2011), 26.

20. Meegan Kennedy, "Open Annotation and Close Reading the Victorian Text: Using Hypothes.Is with Students," *Journal of Victorian Culture* 21, no. 4 (2016): 550-558, https://doi.org/10.1080/13555502.2016.1233905.

21. Richard Van Noorden, "Online Collaboration: Scientists and the Social Network," *Nature News* 512, no. 7513 (2014): 126, https://doi.org/10.1038/512126a.

22. Jason Priem and Kaitlin Light Costello, "How and Why Scholars Cite on Twitter," *Proceedings of the American Society for Information Science and Technology* 47, no. 1 (2010): 1-4, https://doi.org/10.1002/meet.14504701201; Gunther Eysenbach, "Can Tweets Predict Citations? Metrics of Social Impact Based on Twitter and Correlation with Traditional Metrics of Scientific Impact," *Journal of Medical Internet Research* 13, no. 4 (2011): e123, https://doi.org/10.2196/jmir.2012.

23. Ingemar Bohlin, "Communication Regimes in Competition: The Current Transition in Scholarly Communication Seen through the Lens of the Sociology of Technology," *Social Studies of Science* 34, no. 3 (2004): 365-391, https://doi.org/10.1177/0306312704041522; Kristrún Gunnarsdóttir, "Scientific Journal Publications: On the Role of Electronic Preprint Exchange in the Distribution of Scientific Literature," *Social Studies of Science* 35, no. 4 (2005): 549-579, https://doi.org/10.1177/0306312705052358.

24. Stevan Harnad, "Post-Gutenberg Galaxy: The Fourth Revolution in the Means of Production of Knowledge," *Public Access-Computer Systems Review* 2, no. 1 (1991): 39-53.

25. Arthur P. Smith, "The Journal as an Overlay on Preprint Databases," *Learned Publishing* 13, no. 1 (2000): 43-48, https://doi.org/10.1087/09531510050145542.

26. For more on the difficulties of this model, see John Holmwood's chapter in this book.

27. David Pontille and Didier Torny, "Behind the Scenes of Scientific Articles: Defining Categories of Fraud and Regulating Cases," *Revue d'Épidémiologie et de Santé Publique* 60, no. 4 (2012): 247-253; Sara Schroter et al., "What Errors Do Peer Reviewers Detect, and Does Training Improve Their Ability to Detect Them?," *Journal of the Royal Society of Medicine* 101, no. 10 (2008): 507-514, https://doi.org/10.1258/jrsm.2008.080062.

28. Archambault et al., "Proportion of Open Access Papers Published in Peer-Reviewed Journals at the European and World Levels—1996–2013."

29. Ross-Hellauer, "What Is Open Peer Review?"
30. Shapin and Schaffer, *Leviathan and the Air-Pump*.

31. Paul Wouters, Zohreh Zahedi, and Rodrigo Costas, “Social Media Metrics for New Research Evaluation,” in *Springer Handbook of Science and Technology Indicators*, ed. Wolfgang Glanzel et al. (Berlin: Springer International Publishing, 2019).

32. Alex Csiszar, *The Scientific Journal: Authorship and the Politics of Knowledge in the Nineteenth Century* (Chicago: University of Chicago Press, 2018).

33. Tennant et al., “A Multi-Disciplinary Perspective on Emergent and Future Innovations in Peer Review.”