Are citation data a valid measure of journal use? An empirical examination in an academic context

Citation and journal use data have both been used as tools to determine the quality and usefulness of a journal title, especially in academic libraries. However, both of these tools have been criticized for not accurately representing the full spectrum of journal use. With the increased popularity of online journals and the emergence of electronic journal usage data from publishers, there is another tool to potentially improve measurement of the use of journals. This study aims to determine whether these new electronic usage data correlate with more established print usage data and citation data, and to examine whether there is a difference between local citation data (citations by users whom the library serves) and a more global citation measure such as the journal impact factor, in measuring journal use. The findings show that the electronic journal usage measure looked at in this study correlates with the traditional print usage measure. In addition, it was found that local citation data are a valid measure of journal usage but that the more global measure of impact factors are not as valid. Introduction For many years, librarians and information scientists have struggled with how to best determine the value of a journal, either in the context of a library collection or a field of study. Libraries have developed a use-based measure, in the form of print re-shelving data or circulation data (if serials circulate), as one means of helping determine the value of a journal in their specific library collection. In contrast, citation measurement was developed by information scientists to give a broader, more research-based view of a journal’s impact on a field of study. Citation data can be divided into two groups: a. Global citation data – this data is gathered by tracking the citation and publishing patterns of researchers at many institutions throughout the world. An example of this kind of data is that found in Journal Citation Reports (JCR), and a specific global citation measure is that of the impact factor. b. Local citation data – this uses data that is local to the institution. For example, the Journal of X was cited 10 times in 1998 by the faculty at University of Y. Such data can be obtained, for a fee, through Local Journal Utilization Reports, a product offered through the Institute for Scientific Information (ISI). It can also be collected locally by searching citation databases in the field(s) of interest. Both print re-shelving data and citation data have been criticized for not providing a complete picture of journal use or value. Print re-shelving studies are expensive, time consuming, and not always accurate (Broadus, 1985). For example, users may re-shelve journals on their own; it is also difficult to determine how much use was made of a volume – the user may have glanced quickly at it or may have photocopied three articles. Both types of use will get equal weight in most re-shelving studies. In addition, print re-shelving studies are only of use for determining the usefulness of a journal in a single library, and cannot necessarily be extrapolated to make generalities about how important a journal is to an entire institution, as individuals at the institution may have their own subscriptions and might not use the library copy. Are Citation Data a Valid Measure of Journal Use? An Empirical Examination in an Academic Context 391 Citation data, although generally viewed as useful for evaluating research performance and, to a certain degree, journal impact within a field of study, are also controversial (Colquhoun, 2003; Nisonger 2004; Saha, Saint and Christakis 2003). The controversy partially stems from questions surrounding what motivates citation (for a good review on this issue see Liu, 1993) and also because they are not very timely in terms of providing feedback about what journals are currently cited. Also, citation data do not reflect use by those who do not publish; in an academic environment, this can include undergraduate and graduate students (a large and important user group for academic libraries), as well as staff and other users. Also, impact factors are universal and not specific to individual institutions with individual research/teaching missions. Local citation data (as in the type produced by ISI) must usually be paid for, which can make it inaccessible to some libraries. If the data is gathered manually, it is time consuming. Previous studies have shown that global citation measures such as journal impact factor do not correlate significantly with use of print journals in libraries (Scales, 1976; Pan 1978). However, Stankus and Rice (1982) found that when journals were grouped by subject, scope and language, there was a significant correlation between journal impact factor and use of print journals. This was also the finding of Tsay (1998), who found a correlation between frequency of use and impact factor for titles that publish clinical medicine and/or life sciences articles; Tsay also found a significant correlation between frequency of use and worldwide citation frequency (as reported in Journal Citation Reports). Other studies looking at the relationship between the two have been somewhat inconclusive (Rice, 1983; Schmidt, Davis and Jahr, 1994; Wulff and Nixon, 2004). Studies have also been conducted to determine whether data on local citation and publication patterns in certain journals correlate with a library’s own measures of in-house use. Blecic (1999) found correlations between the following three measures: in-house use (using re-shelving data collected for one day a week from October 1992 to January 1994), circulation, and citation by faculty (obtained through ISI’s Local Journal Utilization Reports), at the University of Illinois at Chicago’s Library of the Health Sciences. Pearson and Spearman correlations between all the sets of data were statistically significant (P<.0001). However, Sridhar (1990) showed that locally collected (not from ISI) citation data for Indian space technology researchers did not correlate significantly well with library journal use. With the emergence of electronic journals there comes a new ability to track use of electronic journals for which the library holds a subscription. Wulff and Nixon (2004) have shown that use of paper and electronic journal titles in an academic health sciences library correlate significantly (p<0.01). They also found that the use of print and electronic journal titles correlates significantly (p<0.01) with the titles’ impact factors. However, they did not examine how the use correlates with local citation data which may be a more accurate measure of journal’s local impact (Davis 2002). Kurtz et al. (2003) conducted a study looking at citations and reads of online articles in the Web-based NASA Astrophysics Data System. They found that the number of citations follows the number of reads very closely, thus proving the “normative theory of citing” (Liu, 2003) that “the number of times a document is cited . reflects how much it has been used” (White and McCain, 1989, 119). Kurtz et al. rightly point to the need for further research in this new area of electronic journal use. They “expect the similarities and differences of reads and citations to become a central facet of bibliometric research . the combination of the two measures of use substantially improves the capabilities of bibliometric measurement”. (Kurtz et al., 2003, 127). In a related but broader study on the relationship between electronic usage and citation, Perneger (2004) found a relationship between the total number of hits received by an article in the first week after publication in the online version of the medical journal BMJ, and the number of subsequent citations the paper received, as determined by ISI’s Web of Science database. He concludes that “early hit counts capture at least to some extent the qualities that eventually lead to citation in the scientific literature” (Perneger, 2004, 3). Joanna Duy and Liwen Vaughan 392 Such studies are of interest not only to librarians and information scientists, but also to researchers in other fields such as computer science and those studying the Web’s role in scientific and/or scholarly communication. In recent correspondence to Nature, for example, Jon Kleinberg asked the question “how closely related are they [usage-based measures of impact] to traditional citation-based measures? We expect that there will be a rough correspondence between citation and usage in an aggregate sense – in other words, that a widely-read paper will show up in the bibliographies of many subsequent papers. However, there will clearly be deviations from this general principle” (Kleinberg, 2004, 1). Building on these previous findings, this study aims to investigate whether citation data are a valid measure of journal use by examining the relationship among various measures of a journal’s value. The emphasis will be on (1) the relatively new electronic journal usage data where little research has been done; and (2) local citation data which are theoretically a better measurement of the local use than the more global impact factor. Specifically, the following research questions will be addressed: 1. Do new electronic usage measures correlate with the more established print usage measure of re-shelving data? 2. Do citation data (local and global) correlate with journal usage data? 3. Is local citation data a better measure of journal use than the global journal impact factor? Methodology Data for all the variables listed below were collected at Concordia University Libraries. Concordia is a major Canadian university with an enrollment of over 31,000 students, and it has two campuses, both of which have a library. Print subscriptions are rarely duplicated between the two libraries, but electronic access to all journals is available whether on campus (at either library) or off campus, to faculty, staff and students.