CONSISTENCY AND CONCERN ON IS JOURNAL RANKINGS

ABSTRACT This study investigates the relationship between the journal rankings reported in five previous studies and the journal rankings based on citation-based indices for journal quality. The comparison results suggest that journal rankings derived based on citation analysis and from IS researchers' perceptions of journal quality are only moderately consistent. Additionally, this consistency for IS research journals is significantly higher than that for allied discipline research journals. Thus, ranking both IS research journals and allied discipline research journals together as one long journal list may lead to incorrect evaluation of quality of allied discipline research journals and of the IS articles published within. (ProQuest Information and Learning: . denotes formulae omitted.) INTRODUCTION As known, the quality ratings of journals are of a particular interest to academicians since publications in prestigious journals have significant influence on academic peer recognition, departmental and institutional rankings, tenure and promotion decisions, and the merit increase of faculty compensation (Mylonopoulos and Theoharakis 2001; Walstrom and Hardgrave 2001). Therefore, researchers have studied quality of the journals that publish IS research for almost two decades. To date, there have been at least 15 published attempts to evaluate and rank journals in terms of their importance for IS research (Peffers and Tang 2003). In these prior studies, there were two main approaches used for journal quality evaluation: stated preference and revealed preference studies (Tahai and Meyer 1999). Stated preference studies collect perception data by surveying researchers from either an elite group or a representative group (Peffers and Tang 2003); whereas revealed preference studies employ citation analysis to examine the citations of the journals under evaluation. Although it was noticed that the studies employing these two approaches provided some consistent results (Peffers and Tang 2003), little literature that empirically compared the consistency of results produced by these two approaches could be found. Furthermore, Peffers and Tang (2003) pointed out that it is not necessary to include journals from other disciplines in IS journal ranking as there is insufficient capacity in the journals from other disciplines for all of the IS research of high quality and the journals from other disciplines are already ranked by researchers in their own disciplines. Therefore, Peffers and Tang (2003) proposed to separate the ranking into three lists, including ranking of IS research journals, ranking of allied discipline research journals, and ranking of professional/managerial magazines and journals. This study does not attempt to rank IS journals; rather, it aims at: (a) an empirical investigation on the consistency of ranking results produced by the two known approaches - perception-based ranking vs. citation-based ranking and (b) the finding of empirical evidences to support or not to support the proposal that journal ranking should be separated into the ranking of IS research journals and the ranking of allied discipline research journals. In this study, we categorized journals into IS research journals, allied discipline research journals, and professional/managerial magazines and journals. We collected 27,396 citations made to 8,133 target articles published in 37 journals. Then, we derived four citation-based indices from the collected citation data and conducted correlation analyses and analysis of variance tests (ANOVA) to examine the consistency of journal rankings from the two known approaches. This study is organized into five sections. The next section discusses some essential concepts of knowledge contribution and citation analysis and presents citation-based indices for journal quality. Section three details the research methodology and means of analysis. …