From: byland@cis.ohio-state.edu (Tom Bylander)
Newsgroups: comp.ai
Subject: Re: New Problems in IJCAI Reviewing (long)
Date: 21 Jan 1993 10:57:22 -0500
Organization: The Ohio State University Dept. of Computer and Info. Science
NNTP-Posting-Host: iris.cis.ohio-state.edu



I think IJCAI's reviewing system could have been administrated better
(i.e., let authors know that the reviewing would be anonymous and that
abstracts should be on a separate page), but I disagree with the
other arguments against anonymous reviewing.

Several reviewers have noted that it is often easy to guess who the
author(s) of the paper are anyway.  Part of this is because authors
did not know about the anonymous reviewing.  It does not take much
work to make the guessing a little harder, though admittedly it would
still be easy much of the time.  However, focusing on this problem
misses a key point.  The point of anonymous reviewing is not to make
authorship impossible to guess in every instance.  Many complaints
about the IJCAI reviewing process (probably about every reviewing
process) is that there is a significant bias in favor of recognized
researchers and places, and that these complaints come from people
outside of this grouping.  Certainly, there are many papers whose
authors cannot be easily determined, and at least these papers would
be on a level playing field.  Reviewers (primarily researchers who are
"inside") find many aspects of anonymous reviewing irritating, but
they should remember that "outside" researchers can come up with a
compelling explanation for this (compelling to those on the "outside"
at least).

Now I feel I must respond to some of Brian Gaines' points.

> Authors have to modify their papers
> extensively to remove any evidence of their identities, and these
> modifications may distort the content.

I don't anyone is requiring that authors remove every evidence of
their identities.  It would be simple enough to refer to your own
previous work as if it were somebody else (e.g., avoiding phrases like
"in our previous work \cite{...}"), and to remove identifying
information from references to unpublished, relatively unknown
papers.  I doubt that this level of anonymization distorts much of
anything, and this year it would have made some easy-to-guess cases
much more doubtful.

> The 'meaning' of text is situated in its origins and in its place in the
> stream of discourse. The same statements do not have the same meaning from
> different authors. The same text has a very different status as a report of
> the ongoing research previously reported by one group, than it does as a
> report of replication of that research activity by an independent group. In
> the event of replication, the relation between the two groups, the degree of
> independence, is very relevant. 

Give me a break.  In general, either the paper is sufficiently
persuasive or it is not, i.e., it presents sufficient evidence that
the reviewer can evaluate.  I cannot imagine a comment like "accept
this paper only if it was written by x, y, or z" being taken seriously
by any fellow reviewers, or "the claim by this paper is true if x
wrote the paper, but false if y wrote the paper".  In those cases in
which independent authorship is important (i.e., "accept this paper
only if it wasn't written by x because it claims to be an independent
evaluation of x's work"), this is something that editors are good for.

> A brief statement of a research result from a researcher with a
> major track record of excellent, replicated research may be acceptable on the
> reasonable assumption that she or he knows the state of the art, artifacts,
> etc. The same statement from a researcher new to the field may require
> in-depth justification. This is bias, but it is proper bias reflecting the
> communication process involved, not improper bias reflecting personal
> prejudice or self-interest.

Argument by authority is a classic fallacy, and, because it is a
fallacy, depending on such an argument has its own significant fraud
potential, note Gallo AIDS research case and David Baltimore case.  In
this political climate, we had better be careful that the evidence is
there whether or not the researcher is recognized, or we will pay for
it later.

> An improper effect
> would be one that damaged the progress of science because one group had direct
> power to prevent the dissemination of knowledge rather than the indirect power
> of showing it be incorrect. There have been many independent empirical studies
> to determine whether such improper effects exist in the scientific community.
> The difficulties of measurement and interpretation are such that no studies
> are conclusive, but any effects found have been slight. 

No doubt the "empirically measurable" effects of anonymous reviewing
are slight as well, so this is no argument against anonymous
reviewing, per se.  It's only an argument for keeping things the way
they are because we (on the inside) are comfortable with it.

Tom Bylander
byland@cis.ohio-state.edu



Newsgroups: comp.ai,rec.arts.books,misc.writing,rec.arts.int-fiction