Newsgroups: bit.listserv.pacs-l
Date:         Thu, 7 Jan 1993 08:17:03 CDT
From:         Public-Access Computer Systems Forum <LIBPACS@UHUPVM1.BITNET>
Subject:      Electronic Publication and the Knowledge Glut, Pt. 2 of 2

From: "Stevan Harnad" <harnad@Princeton.EDU>
Subject: ELECTRONIC PUBLICATION AND THE KNOWLEDGE GLUT, Pt. 2 of 2

           ELECTRONIC PUBLICATION AND THE KNOWLEDGE GLUT
          Part II: Is There a Substitute for Peer Review?

ashley@cs.unsw.oz.au (Ashley Aitken) wrote:

aa> Although I do not mind your posting my suggestions and your replies, I
aa> cringe a little at how they make me look, and at how my suggestions
aa> were interpreted - due completely to the cursory way in which I put them.
aa>
aa> So if you will bear with my attempt to explain in a bit more detail the
aa> real essence of my suggestions, I will rest easier :-) with the
aa> original posting, knowing that I have at least tried to make my points
aa> clearer to you.
aa>
aa> In general, my hope was for PYSCOLOQUY to take bold new steps in
aa> different directions, rather than more electronic steps in the same
aa> direction as the current printed journals.
aa>
aa> In particular then, with regard to each specific point -:
aa>
aa> aa> 1. Thought needs to be given to ways to reduce the amount of material
aa> aa>    published (in number and size).
aa>
>   sh> Odd suggestion! I would have thought just the opposite.
aa>
aa> Here, my suggestion must have been unclear - as your amazement shows. I
aa> was suggesting that we need to move away from the idea of everyone's
aa> having their say on an issue directly in the form of an article or review.
aa>
aa> As I said before I don't have all the solutions but generally what I
aa> was thinking of was a more constructive approach where we EXPLICITLY
aa> build some sort of 'knowledge' tree (with all its ambiguities and
aa> branches in so many different directions) which is electronically
aa> accessible (as a hypertext document, for example).
aa>
aa> With the approach that is currently being taken we will end up with
aa> hundreds (if not thousands) of electronic journals publishing tens (if
aa> not hundreds) of thousands of editions a year. Yes, they will be
aa> extremely well indexed and on-line but will that make us feel any less
aa> overwhelmed? The knowledge is only implicit in this juggernaut of
aa> publications, not anywhere near explicit.
aa>
aa> Under an 'explicit constructive' approach, research publications would
aa> instead be auxiliary justification for alterations to the 'knowledge'
aa> tree. Note here that I am not operating under any naive impression that
aa> there is one true tree. There would be many branches on this tree
aa> representing the many beliefs and theories - but each explicitly stated
aa> and perhaps with some weighting.

These suggestions are all commendable, and no doubt many new epistemic
forms will evolve in the new medium. Here I wish only to point out that
there IS a means of sorting the wheat from the chaff in (if I may
change metaphors) the expanding grain bowl of knowledge, and that
means is PEER REVIEW. That quality control mechanism, which consists of
nothing more than qualified peers EXPLICITLY reading and evaluating
material PRIOR to publication, is medium independent. Until you find a
means of accomplishing this automatically, without human intervention
(I doubt that I will live to see the day -- especially in cognitive
science, where to suppose that we already have such a mechanism is
tantamount to supposing we have already solved most of the problems of
cognition!) we are stuck with this "traditional" tool.

No need to despair, though, because most of the information that is
burgeoning on the net does not ASPIRE to be refereed. For that small
subet of it that does, there are still, and will continue to be, enough
peer reviewers to go around. And once they have exercised their
gate-keeping function for us all, we are free, if we wish, to use this
authenticated archive to help filter our reading. In other words, as
long as there is peer review, and a clear means of distinguishing what
in the archive has and has not been peer-reviewed (and how rigorously,
for this too varies from journal to journal), there is still a
medium-independent way of calibrating our knowledge consumption,
despite the epistemic explosion.

There are many exciting things to be done beyond the scope of this
quality-controlled canon, in the vast vanity press that will soon
materialize, but that is not the terrain to which PSYCOLOQUY is
dedicated. There is also room for revolutionary innovation WITHIN the
canon, as it is implemented electronically. That is the domain that
interests me: How to implement peer review electronically, how to speed
it up and make it more even-handed, and how to couple it with the
subsequent phase of open peer commentary.

I wish the knowledge tree well, but I already know (from experience
WITHIN peer review) that mere box scores, publication counts, and citation
counts (in other words, the kinds of variables that would govern the
arborization of the knowledge tree) will never substitute for the acute
insight of a knowledgeable individual. Or not for some time, anyway.


aa> aa> 2. Thought needs to be given to more powerful indexing and referencing
aa> aa>    techniques.
aa>
>   sh> Look at the sample of Veronica below and you will see that not only has
>   sh> thought been given, but some unprecedentedly powerful tools are already
>   sh> available!
aa>
aa> Yes, you are correct - there are powerful tools. I knew of Archie (for
aa> software) and read of Veronica and Gopher in one of the five or more
aa> editions of PYSCOLOQUY still remaining in my mail box after I had
aa> posted to you (BTW, I wonder if there is a universal physical constant
aa> pertaining to the number of issues of a journal which are always
aa> present in one's mail box :-).
aa>
aa> However, no matter how powerful these methods may be (and at present I
aa> believe they are limited to indexing on the file name) at some time in
aa> the future they will surely produce far too many matches to be
aa> useful. If everyone is having their say in the same old manner, and
aa> 'everyone' is growing exponentially, then this path is always
aa> doomed.
aa>
aa> I don't think we should have to do the searches to find the papers to
aa> find the theories and facts to piece together the present state of
aa> knowledge.

Electronic search tools will become ever more subtle and sophisticated,
however it will be some time before they (or hypertext links and
arborization paths) will substitute for informed judgments of quality.
So I predict that the most valuable single source of information about a
retrieved item for some time to come will be whether or not it passed
peer review.

As to unread email: When you will have redirected most of your
telephone, paper mail, face-to-face, conference, and even paper-reading
traffic to your email, as I have, you'll find that "unread email" is not
an informative descriptor, for virtually all your information will be
electronic, and how quickly you attend to it will depend on what
indicators you have of how important it is. Just as in paper days I read
refereed journals before looking at unrefereed preprints, there will be
ways of ordering priorities on the Net too.
aa> aa> 3. The refereeing procedure needs to be made more explicit and open to
aa> aa>    discussion.

>   sh> I don't know how familiar you are with peer review for scholarly and
>   sh> scientific publication.
aa>
aa> I am familiar, to a point, with the process for peer review for
aa> scholarly and scientific publications. I am not sure, however, how much
aa> justification there is for it as the best way to choose the best
aa> publications. Again, I read in a later edition of PYSCOLOQUY of more
aa> radical ideas, viz consensus review. This type of rethinking of the
aa> refereeing procedure is what I was hoping for.

The article advocating consensus review was refereed and accepted as a
speculative hypothesis. But what is needed before we abandon peer review
in its favor is empirical research on the matter, with its results
submitted to refereed journals for peer scrutiny.

aa> Perhaps, I am just naive about the real nature of scientific research
aa> and the reasons for publication, or perhaps I am looking too far into
aa> the future. Anyway these suggestions are just some that linger in my
aa> head when I should be instead concentrating on my graduate research.
aa>
aa> As before, I am very grateful for the PYSCOLOQUY electronic journal and
aa> offer this discussion only as an attempt to make it better.

I have posted your thoughts because I think many others are thinking
along similar lines and because the thoughts are worth discussing. There
are, however, certain prima facie answers that should also be taken into
account, and I have tried to provide some of them here.

With thanks for your comments, which will, I am sure, generate still
further discussion.

Stevan Harnad
Editor, Behavioral & Brain Sciences, PSYCOLOQUY

Cognitive Science Laboratory |    Laboratoire Cognition et Mouvement
Princeton University         |    URA CNRS 1166
221 Nassau Street            |    Universite d'Aix Marseille II
Princeton NJ 08544-2093      |    13388 Marseille cedex 13, France
harnad@princeton.edu         |    harnad@rrmone.cnrs-mrs.fr
609-921-7771                 |    33-91-611-420
Comments: Gated by NETNEWS@AUVM.AMERICAN.EDU