Mime-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
Newsgroups: bit.listserv.psycgrad
Date:         Sat, 2 Jan 1993 14:00:00 -0500
              <PSYCGRAD@UOTTAWA.BITNET>
From:         Aki Caramanos <aki@EGO.PSYCH.MCGILL.CA>
Subject:      psycoloquy.92.3.67.peer-review.1.stodolsky (141 lines) (fwd)

Here's something that might be relevent to the psychgrad Journal
ciao for now...
aki

aki@hebb.psych.mcgill.ca




---------- Forwarded message ----------
Date: Thu, 31 Dec 1992 18:31:16 -0500
From: Stevan Harnad <harnad@clarity.Princeton.EDU>
To: Multiple recipients of list PSYC <PSYC@PUCC.BITNET>
Subject: psycoloquy.92.3.67.peer-review.1.stodolsky (141 lines)

    D Stodolosky: INVITATIONAL JOURNALS BASED UPON EDITORIAL CONSENSUS:
                  A NEW EDITORIAL ROLE IN ELECTRONIC JOURNAL PUBLICATION

-----------------------------------------------------------------------
psycoloquy.92.3.67.peer-review.1.stodolsky   Thursday, 31 December 1992
ISSN 1055-0143                   (9 paragraphs, 1 reference, 131 lines)
PSYCOLOQUY is sponsored by the American Psychological Association (APA)
                Copyright 1992 David Stodolosky

        INVITATIONAL JOURNALS BASED UPON EDITORIAL CONSENSUS:
        A NEW EDITORIAL ROLE IN ELECTRONIC JOURNAL PUBLICATION

                David Stodolsky
                Dept. of Computer Science, Bldg. 20.1
                Roskilde University
                DK-4000 Roskilde, Denmark
                david@ruc.dk (or david@mcsun.eu.net)
                (+45) 31 95 92 82

    0.0  ABSTRACT: Objectivity has long been both an ideal in
    scientific communication and the basis for accreditation claims by
    authors of journal articles, yet the evaluation of articles for
    publication proceeds on a mostly subjective basis. This has been
    particularly true in the case of "invitational journals," where an
    unassisted editorial judgement may determine whether or not
    something is published. A "consensus journal" has been proposed,
    that uses a rigorous measurement model based on multidimensional
    peer judgements to generate invitations, but this model assumes an
    infrastructure that is not currently available. An invitational
    system based on similarity judgements could also yield objective
    performance measurements. Such a system could be a transitional one
    between currently existing editorial mechanisms and those of a
    consensus journal.

    0.1  KEYWORDS: invitational journal, editorial role, electronic
    publication, accreditation, measurement, peer review, consensus

1.1  A major factor in the transition from paper to electronic journals
concerns the effective use of judgmental resources. The intensity of
evaluation is much greater with electronic journals: Each manuscript
may be reviewed by dozens or even hundreds of persons before it is seen
by an average reader. The typical process of review by two or three
referees prior to publication, common with paper journals, is
inappropriate in the computer-network environment.

1.2  Stodolsky (1990) presented a role-free model for invitational
journals based upon peer consensus. This model is limited in that it
assumes preexisting dimensions for evaluative judgments and the
availability of a robust statistical procedure for calculating
consensus positions among reviewers. The objective of issuing
invitations to the most knowledgeable proponents of different consensus
positions can also be achieved by creating an editorial role. This may
require a redefinition of what is meant by a "consensus" position and a
"most knowledgeable" proponent. It is not clear that the same
performance can be achieved with this new model. However, the new model
can be used to build the set of preexisting dimensions needed to
operate the "fully automatic" invitational journal. The approach
discussed here is "manual," because editorial intervention is required
before invitations can be issued. However, as with the earlier model,
statistical measures of performance are available. We can call this new
model an "invitational journal based upon editorial consensus."

1.3  For the purpose of this discussion, we assume that articles
persist in a database until they are withdrawn. Editorial expertise is
measured by analyzing, in the first instance, the agreement of editors
in issuing invitations. The final criteria depend on what articles are
actually submitted, whether any are determined to be redundant, and how
long articles persist before they are withdrawn.

1.4  After reading a published article, potential authors submit short
(e.g., one-paragraph) reactions to the target article. These reactions
are treated as proposals for new articles and directed to a jury of
editors (this could be done by a set of corresponding editors if more
then one jury was available). The editors then independently select a
subset of the proposals judged to be mutually exclusive and
representing the most competent opinion. The judgment of mutual
exclusivity can be performed by sorting proposals into groups. Within
each group, the potential authors would be addressing the same
question. Between groups there would be a significant difference in
what question was being treated (or how it was to be treated).
Discrepancies among groupings by editors would require a "consensus"
set of groupings to be calculated. Editors' performance could be
calculated from their judgements using rater reliability statistics.
Later articles and responses to them would be the final criterion as to
whether two articles were, in the final analysis, distinctive
(addressing different questions or the same question in a different
way).

1.5  When two or more proposals were sorted into the same group, an
author would be selected on the basis of expertise. In the simplest
case, the editorial choice could also include a judgement of author
expertise, based upon the proposal submitted. A more objective
mechanism would base choice on previously demonstrated performance. For
example, an author might be issued a credential for each week a
submitted article was maintained in the database. Then, in the case of
nonexclusive proposals, the author with the largest number of
credentials (in that subject area) would automatically be issued an
invitation. A further consideration could be the past performance of
the author in responding to invitations. For example, an author who
failed to respond to invitations 10% of the time might lose future
invitations one of ten times to less competent peers.

1.6  Similar objective mechanisms could be applied to editorial
responses, allowing editors' judgments to be weighted according to
previous performance. A system of credentials and selection mechanisms
could also be developed, as suggested above for authors. The final
criterion for editorial performance could be the number of
article-weeks generated by invitations.

1.7  Given a history of operation for the journal structure described,
the groupings made by editors could be used as a basis for generating
dimensions for evaluative judgments. A reliable set of such dimensions
would permit more efficient sorting of proposals within the framework
of an invitational journal based upon editorial consensus; it would
also permit the testing of the more powerful invitational structure
based upon peer consensus.

1.8  Thus, the invitational journal based upon editorial consensus can
be viewed as a stepping stone between current invitational journals
such as sci.psychology.digest (PSYCOLOQUY's Usenet Edition) and the
more advanced invitational structures proposed by Stodolsky (1990). The
transition from current practice requires employing more people in the
editorial process. However, it also permits greater automation in the
administration of a journal. A very important benefit is the
quantitative measurement of editorial performance that could be used to
support claims of journal quality.

REFERENCE

Stodolsky, D. S. (1990). Consensus Journals: Invitational journals
based upon peer consensus. PSYCOLOQUY 1(15) psyc.arch.1.15.90.
Also appeared in: Datalogiske Skrifter (Writings on Computer Science)
No. 29 1990. Roskilde University Centre, Institute of Geography,
Socioeconomic Analysis, and Computer Science (ISSN 0109-9779-29).
Newsgroups: bit.listserv.psycgrad