Assessing Quality of Research. New approaches in response to changing expectations
Creators: name: Thijs, Bart
Publisher: name: Zenodo
Year: 2024

description: Keynote given at the University of Fribourg, 13 November 2024 Swiss Year of Scientometrics, Lecture no 5. The event took place on 13 November 2024 and was co-organized by the ETH Library (ETH Zurich) and the University of Fribourg. Assessing quality of research. New approaches in response to changing expectations.

Abstract: The idea of accountability for research institutions receiving public funds based on the quality and impact of their research sparked already sixty years ago with the publication of de Sola Price's "Little Science, Big Science." The notion that publicly funded researchers should be held accountable for their work was a groundbreaking concept at the time. Today, this imperative has become a global reality, with governments, agencies, and institutional management bodies seeking to develop and implement methods and procedures to enhance and evaluate their research quality and impact.

The initial concerns surrounding research assessment focused on ensuring that publicly funded researchers were producing high-quality research. This involved developing quantitative methods for evaluating both productivity and quality of research outputs, using publication counts, citation metrics, and journal impact factors. These efforts aimed to provide a framework for assessing both the scientific production and the merit of research and for identifying areas for improvement. Field and discipline specific research practices were accounted for through field normalization and characteristic scores and scales. The role and effect of collaboration or self-citations was extensively documented.

However, as the scientific landscape has evolved alongside changing societal expectations, new challenges have emerged. The pace of scientific progress has accelerated, and the nature of research has become increasingly complex. In addition to these technical challenges, there is a growing recognition that research assessment must prioritize diversity, inclusiveness, and transparency. Researchers from diverse backgrounds are increasingly valued for contributing to the scientific community, bringing with them unique perspectives and experiences. However, this diversity also poses new challenges for research assessment, as evaluators must be able to recognize and value the contributions of researchers from underrepresented groups. Parallel to these societal changes, scientists themselves have advocated for the integration of additional or alternative approaches into the assessment toolbox in response to the identification of shortcomings and side-effects of established procedures.

These exogeneous and endogenous dynamic processes have crystalized into an altered and broadened understanding of the concept of quality. Consequently, contemporary practices in research assessment, rooted in the traditional perspective on research quality in terms of knowledge flows and reuse, are insufficiently geared towards capturing this broad spectrum of research characteristics.

In this lecture, I will delve into this altered concept of quality and its multi-facetted nature. In addition, the lecture will highlighting recent findings and explore emerging initiatives and research lines in research assessment that try to respond to those challenges while acknowledging the basic idea of researcher accountability