Beyond citation analysis: A model for assessment of research impact

Question: Is there a means of assessing researchimpact beyond citation analysis?Setting: The case study took place at the WashingtonUniversity School of Medicine Becker MedicalLibrary.Method: This case study analyzed the research studyprocess to identify indicators beyond citation countthat demonstrate research impact.Main Results: The authors discovered a number ofindicators that can be documented for assessment ofresearch impact, as well as resources to locateevidence of impact. As a result of the project, theauthors developed a model for assessment of researchimpact, the Becker Medical Library Model forAssessment of Research.Conclusion: Assessment of research impact usingtraditional citation analysis alone is not a sufficienttool for assessing the impact of research findings, andit is not predictive of subsequent clinical applicationsresulting in meaningful health outcomes. The BeckerModel can be used by both researchers and librariansto document research impact to supplement citationanalysis.INTRODUCTIONA traditional method of assessing research impact,citation analysis is performed by examining anindividual publication and assessing how often ithas been cited, if ever, by subsequent publications [1].It is a tool for gauging the extent of a publication’sinfluence in the literature and for tracking theadvancement of knowledge with the inherent as-sumption that significant publications will demon-strate a high citation count [2–4]. While citationanalysis is subject to some flaws, such as self-citingand reciprocal citing by colleagues [5, 6], it is acceptedas a standard tool for assessing the merits of apublication.In May 2007, a principal investigator from theOcular Hypertension Treatment Study (OHTS) [7]requested a citation analysis of OHTS articles afterviewing a poster by Sieving, ‘‘The Impact of NEI-Funded Multi-Center Trials: Bibliometric Indicationsof Dissemination, Acceptance and Implementation ofTrial Findings’’ [8], which had been presented at the2007 meeting of the Association for Research in Visionand Ophthalmology. The authors performed a cita-tion analysis of twenty-six OHTS peer-reviewedarticles using the SCOPUS, Web of Science, andEssential Science Indicators databases.Of the twenty-six journal articles, several demon-strated high rates of citation. That is, they were oftencited by subsequent publications. In some instances,the citations exceeded baseline citation rates as notedon Essential Science Indicators, a database thatassesses intellectual impact by analyzing the citationrate for a publication against other publications in aparticular area of research or multiple areas ofresearch.The high citation rates for the selected journalarticles sparked the authors’ interest in furtherinvestigation to determine why these articles werecited so often by other peer-reviewed journal articles.Was this indicative of significant findings that mighthave resulted in clinical outcomes? If so, what werethose outcomes and how could they be revealed?What other evidence of research impact could theauthors discover by going beyond citation analysis?A cursory search of the Google and Yahoo searchengines using key terms related to OHTS (i.e., openangle glaucoma, pachymetry, central corneal thick-ness, ocular hypertension) and the title of the study(i.e., OHTS, Ocular Hypertension Treatment Study)yielded findings such as practice guidelines, continu-ing education guidelines, curriculum guidelines,insurance coverage documents, and quality measuresguidelines that noted OHTS as supporting documen-tation. Further analysis using other resources, such asgovernment websites, revealed additional evidence ofresearch impact attributed to OHTS findings. Thesematerials are not usually indexed by databases, norare they consistently noted as cited-by publications.Given the depth of evidence of research impact thatwas not revealed from citation analysis alone, theauthors decided to perform a more systematic andcomprehensive evaluation.METHODOLOGYThe project methodology beyond the initial citationanalysis was neither a tidy nor linear process. Theauthors consulted with research investigators, clini-cians, and other librarians to gain insight about theresearch process in both bench and clinical studies inorder to identify tangible outcome indicators that canbe documented and quantified for assessment ofresearch impact. Indicators are defined as specific,concrete examples that demonstrate research impactas a result of a research finding or output. Examplesof tangible indicators include new or ancillaryresearch studies, research findings used as supporting