Assessing research impact with Google Scholar: The most cited articles in the journal 2008–2010

Do you care about if and how you and your colleagues’ research articles are cited by other researchers? Which articles published in this journal are the most cited? Is it possible to calculate the impact factor of a journal by yourself? I often go to Google Scholar to check out the citations of articles. And being new as editor I wanted to study how the articles in our journal are being cited. Google Scholar has the advantage of being free to use as compared with Web of Science owned by Thomson (who publish the Thomson Reuters Impact Factor), and Scopus, owned by Elsevier. Subscription fees for Web of Science are secret, but considered to be higher than Scopus fees which in 2005 were $US20 000–120 000 per year depending on institution size [1]. Google Scholar has increased its capacities and features in recent years and is now updated twice a week. Google Scholar used to yield fewer articles published before 1996, while today it readily retrieves earlier research. Bibliometric research comparing Google Scholar with Web of Science and Scopus shows differences but the correlation is high [2–5]. In addition to Thomson journal articles, Google Scholar searches show science reports, dissertations, books, and articles from journals not indexed by Thomson. Lacking access to Web of Science or Scopus, I used Google Scholar to track citations for all 111 original publications in the Scandinavian Journal of Primary Health Care (SJPHC) in 2008–2010. The median number of Thomson journal citations according to a manual Google Scholar search for the 111 articles was two. I then calculated a 2010 impact factor by summing all Thomson journal citations published in 2010 that I found in Google Scholar for articles published in the SJPHC in 2008 and 2009. The sum of 144 citations was then divided by 77, which is the number of original articles published in the SJPHC during 2008 and 2009. My own 2010 impact factor calculation for SJPHC was 1.87. The 2010 impact factor delivered by Thomson is 1.909. This means that Google Scholar search missed three citations, i.e. 2% of those that make up the impact factor. In Table I the titles of the most cited articles in each of the 12 issues published in the SJPHC 2008–2010 are presented plus the cumulated number of citations they have received both Google Scholar and Thomson journal citations; the number of cumulated citations from the SJPHC, 18% of all Thomson citations, as well as the share of Thomson journal citations that make up the 2010 impact factor. Obviously, articles published in 2008 and 2009 have been cited more than the 2010 articles. And, evidently, the most cited articles in 2008 contribute twice as much (17%) to the 2010 impact factor as do the most cited in 2009 (9%). This is because they have on average twice as much time available to be cited for the impact factor algorithm. Table I. Original articles published in the Scandinavian Journal of Primary Health Care (SJPHC) with the highest number of citations per issue in 2008–2010.1 Looking at the article titles shows that the journal publishes a wide variety of research subjects, which goes with our diverse area of medicine. I wanted to study whether any substantive fields of research were more cited than others. Therefore, I further analysed the article titles and tested a possible correlation between article title content and number of citations. No significant correlations between article title content and number of Thomson citations were found after repeated non-parametric tests using different title content categorizations. Many have criticized the impact factor for various reasons [6]. Nevertheless, the success of research and its implementation requires articles to be read, spread, and cited. And one easy way to measure research impact is counting citations.