Source document: https://docs-lawep.s3.us-east-2.amazonaws.com/1698060267545.pdf (Liberia)
Research evaluation systems and their purposes REAC webinar on research assessment, Iceland, September 2021 Gunnar Sivertsen Nordic Institute for Studies in Innovation, Research and Education, Oslo, Norway Two major purposes of research evaluation (can be combined) 1.

Formative § Learns from the past: Strengths, Weaknesses § Looks forward: Opportunities, Threats § Serves strategic development 2.
Summative § Looks at past performance § Checks whether goals or expectations have been reached § Serves decisions and/or resource allocation Examples of formative research evaluation n Who should we recruit for this new position?

n What should this department/faculty do to improve its research activities?

n How can we improve the interaction between education and research at our university?

n How can we improve the quality and use of clinical research in our country?
n How can we combine excellence, societal relevance and diversity in our national research system?

Examples of summative research evaluation n Did this researcher meet the agreed expectations?

n Should we continue funding of this project?

n How did our department/faculty perform compared to earlier years n How should this year’s funding be allocated among the institutions?

n How does our national research system perform compared to other countries?
Six contexts for research evaluation Level Context Individual/ group When projects are published When applying for a position, a promotion, or local resources When applying for external funding Organization Indicator-based institutional funding systems Summative organizational evaluation systems Formative organizational evaluations Six contexts for research evaluation Level Context Individual/ group When projects are published When applying for a position, a promotion, or local resources When applying for external funding Organization Indicator-based institutional funding systems Summative organizational evaluation systems Formative organizational evaluations Most organization- level evaluations repeat and/or derive indicators from the individual-level contexts Indicator-based institutional funding systems are widespread, particularly in small countries n They usually apply minimum 2 indicators: bibliometrics and external funding n Most often combined with indicators of educational activity.
n In Scandinavia, Denmark, Finland and Norway have similar indicator-based funding systems (and very similar bibliometrics) n Sweden has discarded its indicator-based system, but the same Norwegian model for bibliometrics is used locally Some larger countries have evaluation-based funding systems n The REF in the UK is necessarily summative because it is tied to funding allocation n The REF was modelled on procedures for individual-level assessment n The ERA in Australia continues to be summative even after being untied from funding, and it is modelled on individual-level assessment Formative organizational evaluations are rare in research n A professional definition:* § § “An organisational assessment is a systematic process for obtaining valid information about the performance of an organisation and the factors that affect performance.
It differs from other types of evaluations because the assessment focuses on the organisation as the primary unit of analysis.

§ Organisations are constantly trying to adapt, survive, perform, and influence.

However, they are not always successful.

§ To better understand what they can or should change to improve their ability to perform, organisations can conduct organisational assessments.
* As defined by BetterEvaluation, a global not-for-profit organisation working to improve the practices worldwide, in the document Evaluating the Performance of an Organisation written by Katrina Rojas and Julia Laidlaw.

https://www.betterevaluation.org/en/theme/organisational_performance Formative organizational evaluations are disconnected from funding n Netherlands and Norway organize formative evaluations nationally.
n The universities in Denmark, Finland and Sweden organize them on their own as self- initiated evaluations n Uppsala University in Sweden has, through three rounds (2007, 2011, 2017), changed the aims and procedures § From a traditional REF-model (reviewing what the researchers did) § To a professional formative model (how can we improve as an organization).
Four guidelines in response to local use of the Norwegian Publication Indicator at the individual level Guidelines from the National Board of Scholarly Publishing in Norway.

The Board represents all research organizations in the country’s higher education sector, independent institute sector, and health sector.

1.

Metrics should not be used alone n Recruitment, promotion, career follow-up and distribution of tasks and resources should be based on holistic and customized assessments.
n Other qualifications than former research performance need to be considered in most cases, and even in assessments of research alone, metrics have limited value because they are not foresighted, they do not consider career opportunities, and they cannot make responsible decisions.

2.

Metrics are not foresighted n Metrics look backwards towards past performances.
n Assessment for employment, promotion or allocation of resources should also look forward towards potential performance and be in line with the expectations expressed in calls for applications or in guidelines for resource allocation.

3.

Metrics do not consider career opportunities n Research takes place in more or less active phases depending on other tasks, available resources, and types of projects and collaborations.

n Careers may also be paused for external reasons.


n Understanding differences in career opportunities is part of responsible leadership in research.

4.

Metrics cannot decide n Metrics can be useful when screening a high number of applications before further assessment of relevant qualifications.

n However, the value of using metrics will decrease and eventually disappear as the assessment process comes closer to a final responsible decision.

n Qualitative methods will be needed instead.