From: yoram+@CENTRO.SOAR.CS.CMU.EDU (Yoram Reich)
Subject:  Re: New Problems in IJCAI Reviewing (longER)
Originator: yoram@CENTRO.SOAR.CS.CMU.EDU
Organization: School of Computer Science, Carnegie Mellon
Date: Thu, 21 Jan 1993 14:52:36 GMT

There is no question that all those who posted opinions on this subject
where correct, or at least represent positions that are defendable in a
strong way. How can this be possible? Easy. It merely requires that we
acknowledge that the kind of research we do (and for that matter, all
sciences) involve politics, interest groups, money, pleasure, ego, trips to
the French Riviera, and very remote with ``objectivity''---whatever that
means. [If you ask, I can give a long list of references stating these
ideas, including one on mathematics and the names of conferences discussing
politics in the sciences]

Since there is no objective standard for quality, fairness, etc., the issue
becomes who or which group succeeds in influencing the rest of the community
to adopt a certain practice. Alternatively, this may be done by vote or
through a discussion like the one started by Brian. There is a bag full of
issues that can be discussed beyond the anonymity issue such as: what is the
role of researchers that run journals for many years. Societies' Journals
often replace their editors, but ``private'' ones don't; although, there are
exceptions (e.g., Machine Learning replaces its editor once every several
years). So is it good or bad practice? What are the consequences of it?

Nobody is going to resolve these issues in the sense of making them correct.
The critical thing is to keep talking about them continually and not only
once every two years or so. The discussion can include many aspects. For
example, one can do the following experiment:

	a. Take a conference and set up two parallel review processes: 
	   (1) anonymous and (2) open. 
	b. Let reviewers make their judgment and have another 
	   set of judges evaluate all the papers (now as they'd appear in a 
	   proceeding), without knowing how each of the papers was reviewed. 
	c. Sum the evaluation of the two groups of papers and see whether 
	   the change in quality is statistically significant. Alternatively, 
	   publish two proceedings and let the market decides which is better
	   by counting the number of purchases of each. 

Would this be satisfactory? Maybe more than what is done now. The point is
that there are many issues to discuss, and continue doing so. 

Having said that, a small advertisement for a special issue of AI EDAM
journal on research methodology that will accept these kinds of issues follows.

Yoram
-----------------------------------------------------------------
Yoram Reich, Engineering Design Research Center, Carnegie Mellon 
  University, 5000 Forbes Avenue Pittsburgh, PA 15213-3890, USA
    Voice: (412) 268 7864, Fax: (412) 268 5229, email: yoram@edrc.cmu.edu




       ======================================================================
                 Please Circulate               Please Circulate
       ======================================================================

                                Call For Papers
                           Artificial Intelligence in
              Engineering Design Analysis and Manufacturing (AI EDAM)
                      Special Issue on Research Methodology

    AI EDAM is an international journal devoted to the reporting of research
    and applications of AI in engineering design, analysis, and manufacturing
    (AIEDAM). AIEDAM is a growing field attracting many researchers from
    academia and practitioners from industry. Early work in this field was
    largely exploratory and often ignored methodological issues. The term
    methodology used hear means more than methods; it stands for the methods of
    conducting research and the interpretation of the research activity
    including: the meaning and evaluation of the results, the accumulation of
    knowledge, the relation of research to practice, etc..

    While some scattered studies on AIEDAM research methodology exist, the lack
    of continuous attention to this subject has severe consequences on the
    field: (1) it is sometimes unclear what the goals of the field are, (2) it
    is unclear what constitutes a contribution to the field, and (3) it is
    unclear what are appropriate or promising ways of approaching research
    questions. These three problems, and others, complicate the interpretation
    and understanding of research results and gradually make it hard to
    convince potential supporters of the pragmatic merit of the field. The
    special issue of AI EDAM on research methodology attempts to initiate a
    discussion on these fundamental problems.

    Topics are welcome from (but not limited to) the following issues:

      o  Is AIEDAM a science?
      o  What kind of inquiry is AIEDAM (e.g., positivist, constructivist)
       and how does this influence the research methodology?
      o  What is the purpose of AIEDAM research.
      o  How should AIEDAM research be tested or validated?
      o  How is knowledge accumulated in AIEDAM research?
      o  What are good measures of AIEDAM research progress or goodness?
      o  What is the relation between AIEDAM research and practice and how
       does it influence the research methodology?
      o  Is (should be) research methodology of AIEDAM different from
       methodologies in other engineering disciplines?
      o  Reviews of methodologies used in research and their
       successes/failures.

    This list is by no means exhaustive; other topics are welcome provided that
    they are relevant to the issue of AIEDAM research methodology.

    All papers submitted to this special issue will be blind reviewed according
    to the usual procedure of AI EDAM.

    Manuscripts should be limited to 35 typewritten, double spaced pages (12pt
    font), including figures, tables, and references; two-sided manuscripts are
    welcome. Each manuscript should have a title page with the title of the
    paper, full name(s) and affiliation(s) of author(s), complete postal and
    electronic addresses (if available), telephone number(s), and a maximum 300
    word abstract.

    Please submit 4 copies of a paper (or address any inquiry regarding this
    special issue) to the guest editor by 31 MARCH 1993, to the following
    address:

                        Yoram Reich
     BEFORE             Department of Civil and Environmental Engineering
     June 15, 1993       Duke University
                        Durham, NC 27706, USA

     AFTER              Engineering Design Research Center
     June 15, 1993       Carnegie Mellon University
                        5000 Forbes Avenue
                        Pittsburgh, PA 15213, USA

    electronic mail: yoram@edrc.cmu.edu

    Acceptance status letters will be sent by 15 JULY 1993.

       ======================================================================
                 Please Circulate               Please Circulate
       ======================================================================
-- 
-----------------------------------------------------------------
Yoram Reich, Engineering Design Research Center, Carnegie Mellon 
  University, 5000 Forbes Avenue Pittsburgh, PA 15213-3890, USA
    Voice: (412) 268 7864, Fax: (412) 268 5229, email: yoram@edrc.cmu.edu