Requirements engineering paper classification and evaluation criteria: a proposal and a discussion

In recent years, members of the steering committee of the IEEE Requirements Engineering (RE) Conference have discussed paper classification and evaluation criteria for RE papers. The immediate trigger for this discussion was our concern about differences in opinion that sometimes arise in program committees about the criteria to be used in evaluating papers. If program committee members do not all use the same criteria, or if they use criteria different from those used by authors, then papers might be rejected or accepted for the wrong reasons. Surely not all papers should be evaluated according to the same criteria. Some papers describe new techniques but do not report on empirical research; others describe new conceptual frameworks for investigating certain RE problems; others report on industrial experience with existing RE techniques. Other kinds of papers can also be easily recognized. All of these types of papers should be evaluated according to different criteria. But we are far from a consensus about what classes of paper we should distinguish, and what the criteria are for each of these classes. We see a variety of evaluation criteria in journals too. At one extreme is the set of nine genres used by IEEE Software [15], all of which have different evaluation criteria. At the other extreme is the single paper class recognized by the Requirements Engineering Journal, which has the following evaluation criteria: originality, utility, technical contribution, and relation to previous work. Apparently, the only paper class recognized by the Requirements Engineering Journal is a paper describing an original and useful solution technique. This corresponds to the ‘‘how to’’ genre of IEEE Software. This leaves authors and reviewers for the Requirements Engineering Journal in the dark about how other classes of papers should be judged, such as experience reports, empirical studies, or tutorials, none of which describe an original technique. This might lead to the use, by reviewers, of evaluation criteria unknown to authors, or even to the use of mutually inconsistent evaluation criteria by different reviewers of the same submission. The calls for papers of successive RE conferences, in which some of us acted as program chair, show an evolution of paper classification and evaluation schemes. Each scheme was based on the experience of the previous chair, and we tried to pass on our experience to the next chair. We also discussed our ideas with other members of the Steering Committee of the RE conferences, and with RE researchers outside the Steering Committee. This short note presents the outcome of those discussions in the form of a proposal for paper classification and a set of evaluation criteria for different paper classes. We hope to include more people in the discussion and thereby further improve the classification and evaluation scheme. In Section 2, we sketch the rationale for our classification. Section 3 presents the classification, and Section 4 concludes with a discussion of background ideas and related work. R. Wieringa (&) University of Twente, Twente, The Netherlands E-mail: roelw@cs.utwente.nl