20 July 2022

AGREEMENT ON REFORMING RESEARCH ASSESSMENT

20 July 2022

As signatories of this Agreement, we agree on the need to reform research assessment practices.
Our vision is that the assessment of research, researchers and research organisations recognises
the diverse outputs, practices and activities that maximise the quality and impact of research. Thisrequires basing assessment primarily on qualitative judgement, for which peer review is central,
supported by responsible use of quantitative indicators. Among other purposes, this isfundamental for: deciding which researchers to recruit, promote or reward, selecting which
research proposals to fund, and identifying which research units and organisations to support.
In the context of this Agreement, research assessment encompasses:•

The assessment of research performing organisations and research units, by assessment
authorities, research funding and performing organisations, for the purposes of allocating
funding; public investment accountability; informing decisions on research priorities andimproving the definition and implementation of research strategies.

•

The assessment of research projects by assessment authorities, research funding andperforming organisations, and prize awarding organisations; for the purposes of allocating
funding, informing project management and future research funding decisions, and making
prize and award decisions.

•

The assessment of individual researchers and research teams by research funding andperforming organisations and prize awarding organisations; for the purposes of allocating
funding, recruitment and hiring promotion, professional development review, and prize andaward decisions.

The Agreement focuses on the specific challenges involved in reforming the research assessmentof researchers, research projects, research units and research organisations. It does not consider
research performance at country level. Performance reviews of institutions, which often consider
activities beyond research, are outside the scope of this Agreement. While academic assessment
is beyond the scope, organisations may take the opportunity to extend the Principles andCommitments of this Agreement to academic assessment.
This Agreement establishes a common direction for research assessment reform, while respecting
organisations’ autonomy.
We commit to achieve reform through a Coalition of research funding organisations, research
performing organisations, national/regional assessment authorities and agencies, as well as
associations of the above organisations, learned societies and other relevant organisations, thatis global in scope. We will work together to enable systemic reform on the basis of common
principles within an agreed timeframe, and to facilitate exchanges of information and mutual
learning between all those willing to improve research assessment practices.

2

AGREEMENT ON REFORMING RESEARCH ASSESSMENT

20 July 2022

To this end, we agree to:I. Base our actions on the following Principles:
Principles for overarching conditions•

Comply with ethics and integrity rules and practices, and ensure that ethics and integrity are
the highest priority, never compromised by any counter-incentives. Verify before or duringassessment that the highest standards of general and research- specific ethics and integrity
are met. Value methodological rigour to guard against sources of bias, and promote extended
forms of professional and scientific integrity, showing adherence to moral standards ofconduct, and include behaviours such as early sharing of research data and results, buildingon the work of others, and subjecting oneself to critical external validation.

•

Safeguard freedom of scientific research. By putting in place assessment frameworks that do
not limit researchers in the questions they ask, in their research implementation, methods ortheories. By limiting the assessment frameworks to only those necessary, as assessment must
be useful for researchers, institutions and funders.

•

Respect the autonomy of research organisations. By safeguarding the independence of
research performing organisations in the evaluation of their researchers while implementing
the present principles, yet striving to prevent contradictions between the assessment ofresearch, researchers and institutions, and between institutions, to avoid fragmentation of theresearch and innovation landscape and to enable the mobility of researchers.

•

Ensure independence and transparency of the data, infrastructure and criteria necessary for
research assessment and for determining research impacts; in particular by clear andtransparent data collection, algorithms and indicators, by ensuring control and ownership by
the research community over critical infrastructures and tools, and by allowing those assessed
to have access to the data, analyses and criteria used.

Principles for assessment criteria and processes
Quality and impact•

Focus research assessment criteria on quality. Reward the originality of ideas, the professional
research conduct, and results beyond the state-of-the-art. Reward a variety of research
missions, ranging from basic and frontier research to applied research. Quality implies that
research is carried out through transparent research processes and methodologies andthrough research management allowing systematic re-use of previous results. Openness of
research, and results that are verifiable and reproducible where applicable, strongly
contribute to quality. Openness corresponds to early knowledge and data sharing, as well as
open collaboration including societal engagement where appropriate. Assessment should relyon qualitative judgement for which peer review is central, supported by responsibly used
quantitative indicators where appropriate.

•

Recognise the contributions that advance knowledge and the (potential) impact of research
results. Impact of research results implies effects of a scientific, technological, economic
and/or societal nature that may develop in the short, medium or long-term, and that vary3

AGREEMENT ON REFORMING RESEARCH ASSESSMENT

20 July 2022

according to disciplines and research types (e.g. basic and frontier research vs. appliedresearch).
Diversity, inclusiveness and collaboration•

Recognise the diversity of research activities and practices, with a diversity of outputs, andreward early sharing and open collaboration. Consider tasks like peer review, training,
mentoring and supervision of Ph.D candidates, leadership roles, and, as appropriate, science
communication and interaction with society, entrepreneurship, knowledge valorisation, andindustry-academia cooperation. Consider also the full range of research outputs, such as
scientific publications, data, software, models, methods, theories, algorithms, protocols,
workflows, exhibitions, strategies, policy contributions, etc., and reward research behaviour
underpinning open science practices such as early knowledge and data sharing as well as open
collaboration within science and collaboration with societal actors where appropriate.
Recognise that researchers should not excel in all types of tasks and provide for a frameworkthat allows researchers to contribute to the definition of their research goals and aspirations.

•Use assessment criteria and processes that respect the variety of scientific disciplines,
research types (e.g. basic and frontier research vs. applied research), as well as research career
stages (e.g. early career researchers vs. senior researchers), and that acknowledge multi-,
inter-, and trans-disciplinary as well as inter-sectoral approaches, when applicable. Researchassessment should be conducted commensurately to the specific nature of scientificdisciplines, research missions or other scientific endeavours.

•

Acknowledge and valorise the diversity in research roles and careers, including roles outsideacademia. Value the skills (including open science skills), competences and merits of individual
researchers, but also recognise team science and collaboration.

•

Ensure gender equality, equal opportunities and inclusiveness. Consider gender balance, thegender dimension, and take into account diversity in the broader sense (e.g. racial or ethnic
origin, sexual orientation, socio-economic, disability) in research teams at all levels, and in thecontent of research and innovation.

II. Implement the following Commitments:
Core commitments
The core commitments include two commitments to enable better recognition of the diverse
practices and activities that maximise the quality of research as well as two commitments toenable a move away from inappropriate uses of metrics.

1. Recognise the diversity of contributions to, and careers in, research inaccordance with the needs and nature of the research
Purpose: This commitment will broaden recognition of the diverse practices, activities andcareers in research, considering the specific nature of research disciplines and other researchendeavours.
Scope: Changes in assessment practices should enable recognition of the broad diversity of:4

AGREEMENT ON REFORMING RESEARCH ASSESSMENT•

20 July 2022

valuable contributions that researchers make to science and for the benefit of society,including diverse outputs beyond journal publications and irrespective of the language inwhich they are communicated;•practices that contribute to robustness, openness, transparency, and the inclusiveness of
research and the research process including: peer review, teamwork and collaboration;•

activities including teaching, leadership, supervision, training and mentoring.

It is also important that assessment facilitates the recognition and valorisation of diverse rolesand careers in research, including: data steward, software engineer and data scientist roles,
technical roles, public outreach, science diplomacy, science advice and science communicatorroles to name a few. It is recognised that current practice is often too narrow and limiting, so
the goal cannot be to replace the narrow criteria we wish to move away from with different
but equally narrow criteria. Instead, the aim is to allow organisations to broaden the spectrumof what they value in research, while acknowledging that this may vary across disciplines andthat each individual researcher should not be expected to contribute to all activities at once.

2. Base research assessment primarily on qualitative evaluation for which peer
review is central, supported by responsible use of quantitative indicators
Purpose: This commitment will enable the move towards research assessment criteria that
focus primarily on quality, while recognising that responsible use of quantitative indicators can
support assessment where meaningful and relevant, which is context dependent.
Scope: Research assessment should rely primarily on qualitative assessment for which peer
review is central, supported by responsibly used quantitative indicators where appropriate.
Peer review is the most robust method known for assessing quality and has the advantagethat it is in the hands of the research community. It is important that peer review processes
are designed to meet the fundamental principles of rigor and transparency: 1 expert
assessment, transparency, impartiality, appropriateness, confidentiality, integrity and ethicalconsiderations, gender, equality and diversity. To address the biases and imperfections towhich any method is prone, the research community re-assesses and improves peer review
practices regularly. Revised, or potentially new, criteria, tools and processes appropriate forassessing quality could be explored alongside peer review. Moving towards assessment
practices that rely more heavily on qualitative methods may require additional efforts fromresearchers. Researchers should be recognised for these efforts and their contributions toreviewing peers’ work should be valued as part of their career progression.

1 Global Research Council (2018): Statement of Principles on Peer/Merit Review

5AGREEMENT ON REFORMING RESEARCH ASSESSMENT

20 July 2022

3. Abandon inappropriate uses in research assessment of journal- and publicationbased metrics, in particular inappropriate uses of Journal Impact Factor (JIF) andh-index
Purpose: This commitment will reduce the dominance of a narrow set of quantitative journaland publication-based metrics.
Scope: Inappropriate uses of journal- and publication-based metrics in research assessment
should be abandoned. In particular, this means moving away from using metrics like theJournal Impact Factor (JIF), Article Influence Score (AIS) and h-index as proxies for quality andimpact. ‘Inappropriate uses’ include:•

relying exclusively on author-based metrics (e.g. counting papers, patents, citations,grants, etc.) to assess quality and/or impact;•assessing outputs based on metrics relating to publication venue, format or language;•

relying on any other metrics that do not properly capture quality and/or impact.

4. Avoid the use of rankings of research organisations in research assessment
Purpose: This commitment will help avoid that metrics used by international rankings, which
are inappropriate for assessing researchers, trickle down to research and researcher
assessment. It will help the research community and research organisations regain theautonomy to shape assessment practices, rather than having to abide by criteria andmethodologies set by external commercial companies. This could include retaining control
over ranking methodologies and data.
Scope: Recognising that the international rankings most often referred to by researchorganisations are currently not ‘fair and responsible’ 2, the criteria these rankings use should
not trickle down to the evaluation of individual researchers, research teams and research
units. Research organisations should also be mindful that public communication (e.g. theactive advertising of an institution’s rank) can contribute to the perception that research
quality conflates with ranking positions.
Where ranking approaches are deemed unavoidable, as may be the case in forms ofevaluation beyond the scope of this Agreement such as benchmarking and performance
reviews of countries or institutions, the methodological limitations of such approaches shouldbe acknowledged, and institutions should avoid trickle-down effects on research andresearcher assessment.

2

As defined, for example, by INORMS: https://inorms.net/wp-content/uploads/2022/07/principles-for-fair-and-responsible-university-assessment-v5.pdf

6AGREEMENT ON REFORMING RESEARCH ASSESSMENT

20 July 2022

Supporting commitments
The supporting commitments include three commitments to enable the move towards new
research assessment criteria, tools and processes, and three commitments to facilitate mutual
learning, communicate progress and ensure that new approaches are evidence informed.

5. Commit resources to reforming research assessment as is needed to achieve theorganisational changes committed to
Purpose: This commitment will ensure that organisations allocate the necessary resources,
whether in the form of budget or staff capacity, to improve research assessment practices
within their agreed timeframe.
Scope: Resource allocation by assessment authorities and research funding and performing
organisations is a necessary condition for reforming assessment practices. Resources shouldbe allocated as is needed for each organisation to achieve the changes that will enable
adherence to the Principles and to implement the Commitments. This includes resources to:•implement changes in research assessment, including planning and progress monitoring;•raise awareness among all actors;•

educate, train and support researchers and any other staff involved in assessment,
including peer-reviewers and assessors; and•support the necessary infrastructure such as tools and services for the transparent
collection and processing of data on research assessment practices.

Particular attention should be paid to making resources available to enable the engagementof researchers at all career stages in reforming research assessment.

6. Review and develop research assessment criteria, tools and processes
6.1 CRITERIA FOR UNITS AND INSTITUTIONSWith the direct involvement of research organisations and researchers at all
career stages, review and develop criteria for assessing research units andresearch performing organisations, while promoting interoperability
Purpose: This commitment will ensure that national / regional / organisational authorities
and evaluation agencies review and, where needed, develop criteria for the assessment ofresearch performing units and organisations, in accordance with the Principles. It will
foster the responsible use of metrics in assessing research performing units andorganisations, and help to prevent contradictions or incompatibilities between theassessment of research, researchers and research performing organisations. It will also
safeguard the interoperability of adapted or newly developed assessment processes.
Scope: Criteria for the assessment of research performing units and organisations,
including universities, research centres, and research infrastructures, should be reviewed
and adapted, and new criteria developed where needed, based on evidence. This should
be done in close collaboration with assessors and those that will be assessed, including

7

AGREEMENT ON REFORMING RESEARCH ASSESSMENT

20 July 2022

research organisations and researchers. The changes should increase the ability to assess
quality by enabling recognition of all contributions to quality research by research units
and institutions. Such recognition includes that of early sharing of data and results, open
collaboration, teamwork; and consideration of contributions to the research ecosystem,
knowledge generation and scientific, technological, economic, cultural and societal impact.
National / regional / organisational authorities and evaluation agencies should coordinate
to ensure their methodologies and processes are interoperable, while simultaneously
respecting the necessary adaptation to each context.
6.2 CRITERIA FOR PROJECTS AND RESEARCHERS

With the direct involvement of researchers at all career stages, review anddevelop criteria, tools and processes for the assessment of research projects,
research teams and researchers that are adapted to their context ofapplication
Purpose: This commitment will enable recognition of the diverse research activities andpractices through the revision and development of assessment criteria, tools, andprocesses. It will ensure that organisations review their processes and make tangible
changes by developing existing or new assessment approaches, individually or incollaboration with others, in accordance with the Principles.
Scope: Criteria, tools and processes should be reviewed and developed together withresearchers in different disciplines and at different career stages; and should enable
recognition of the diversity of research activities and practices that contribute to research
quality, including diverse outputs in different languages. This should increase the ability toassess quality by enabling recognition of all contributions to quality research fromresearch projects and by researchers and research teams. This includes recognition ofearly sharing of data and results, open collaboration, and teamwork. Reformed practices
for assessing individual researchers should consider future potential alongside trackrecord and take into account researchers’ individual contexts and careers. They should
also recognise that researchers cannot excel in all types of tasks and provide for aframework that allows researchers to contribute to the definition of their research goals
and aspirations. Research assessment by research funders should consider disciplinary,multi-, inter-, and trans-disciplinary research as well as contributions to knowledge
generation and scientific, technological, economic, cultural and societal impact.

7. Raise awareness of research assessment reform and provide transparent
communication, guidance, and training on assessment criteria and processes aswell as their use
Purpose: This commitment will ensure that organisations raise awareness of the reform
among all actors. It will ensure that organisations transparently communicate the criteria,
tools and processes used for research assessment and train researchers and assessors in theiruse.

8AGREEMENT ON REFORMING RESEARCH ASSESSMENT

20 July 2022

Scope: Without widespread awareness of the reform and training of those assessed and,crucially, assessors, progress will be slow - if not impossible. Organisations should be clear andtransparent about assessment processes and the tools and criteria they use. They shouldmake guidance on their assessment approaches openly available and train those involved in
the assessment process. They should allow those assessed to have access to the criteria, data
and reviews or deliberation outcomes used in their assessment within the limits ofconfidentiality. Particular attention should be paid to raising awareness among researchers at
all career stages.

8. Exchange practices and experiences to enable mutual learning within and beyond
the Coalition
Purpose: This commitment will ensure organisations exchange and make use of information
for mutual learning. It will help avoid fragmentation, contribute to the coherence ofassessment practices between organisations, and enable researcher mobility. It also will allow
those further ahead to share approaches and lessons learned, to benefit those who havefurther to go on their reform journey.
Scope: While respecting each other’s autonomy, organisations should share practices andexperiences to facilitate mutual learning. This exchange should include contributing to thedevelopment of guidance and common approaches in order to minimise contradictions orincompatibilities between the assessment practices used by different organisations. It should
also include sharing of lessons learned to ensure continuous mutual improvements.

9. Communicate progress made on adherence to the Principles and implementationof the Commitments
Purpose: This commitment will ensure organisations update one another on the progress
made. It will foster careful self-reflection and monitoring of their own adherence to the
Principles and progress towards meeting the Commitments.
Scope: Demonstrating progress made towards implementing the Commitments andadherence to the Principles is an important part of this initiative. Organisations should commit
to regularly update each other and their communities on their adherence and progress. This
process involves being open to scrutiny from their own communities, sharing successes aswell as challenges, and communicating their experiences to facilitate collective progress.

10. Evaluate practices, criteria and tools based on solid evidence and the state-ofthe-art in research on research, and make data openly available for evidence
gathering and research
Purpose: This commitment will ensure that assessment approach decisions are evidence
informed. It will help organisations reflect on their own processes, gain understanding about
whether assessment practices achieve the desired goals, and engage in evolutive assessment
based on new evidence as it becomes available. It will also help to ensure control andownership of research assessment data by the research community.

9

AGREEMENT ON REFORMING RESEARCH ASSESSMENT

20 July 2022

Scope: Growing evidence shows that current assessment processes that rely on publicationand journal-based metrics are prone to multiple biases. As approaches using more qualitative
research assessment are piloted by several organisations (e.g. narrative and evidence-based
CVs, new assessment frameworks and indicators), it is important to evaluate and monitor theirimpact based on evidence and rigorous methods. Organisations should contribute to theevidence base on research assessment in order to make this possible. For example, it could
be achieved by making data that can be used for research on research available, byparticipating in research on research, or by funding research on research. Data sharing shouldbe the minimum commitment and data should be shared through open infrastructure, while
respecting personal data protection.

III. Organise and operate the Coalition along the following principles:•

Signatories of the Agreement will be entitled to become members of the Coalition, provided
they meet the conditions for membership established in the governance documents of the
Coalition. They may leave the Coalition at any time.

•

The Coalition will comprise its members; the General Assembly will define its detailedorganisation and operating rules and procedures.

•

The Coalition will offer a space for mutual learning and collaboration that supports andfacilitates the implementation of the Commitments by the members, while respecting theirautonomy.

•

For this purpose, working groups will be established on specific topics as ’Communities ofpractice’. Working groups will be proposed at the initiative of members and other members’
participation is voluntary. Other means, such as workshops or webinars, may also be used tosupport mutual learning and collaboration.

•

The communication of individual members’ progress in implementing the Commitments will
primarily be based on publicly shared self-assessment. Such a trust-based approach is
expectedtofacilitate

informationexchangeandcollaboration,

includingjointexperimentation, when appropriate.
•

Support for Coalition work will primarily consist of cash and in-kind contributions from themembers, or funding from research funding organisations.

•

The Coalition’s work will be supported by a ‘lean’ organisation. Decision-making will rely on the
General Assembly, and structures such as a Steering Board, supported by a Secretariat. The
Coalition should have the capacity to manage its budget.

•

Mechanisms for periodic interactions with, and involvement of, national and regional
authorities will ensure national/regional policies and frameworks are conducive to the
Coalition’s work.

•

The Coalition will seek collaboration, complementarities and synergies with other existing
initiatives and organisations, as appropriate.

10

AGREEMENT ON REFORMING RESEARCH ASSESSMENT

20 July 2022

IV. Respect the following timeframe:•

The signatories of this Agreement agree to share with each other and with their community
how their organisation has started the process of reviewing or developing criteria, tools andprocesses in line with the core Commitments and according to an action plan with defined
milestones, by the end of 2023 or within one year of signing the Agreement.

•

Signatories of this Agreement agree to regularly demonstrate progress towards reviewing,developing and evaluating criteria, tools and processes that fulfil the core Commitments, witha touch point at end of 2027 or within five years of signing the Agreement, by which timethey will have worked through at least one cycle of review and development of theirassessment criteria, tools and processes.
Signatories that are not assessing research projects, researchers, research units or research
performing organisations commit to contribute to the reform and share progress with each otherand the community respecting the same timeframe.

By signing this Agreement, signatories are entitled to become a member of the Coalition. Whileacknowledging that this Agreement does not have any legally binding effect, it represents a public
commitment to contribute actively and constructively to reforming research assessment.

______________________
Signature and date______________________Name and position of the signatory representing the organisation______________________
Name, position and contact details at the organisation for coordinating signature follow-up

11

AGREEMENT ON REFORMING RESEARCH ASSESSMENT

20 July 2022

The following annexes do not form an integral part of the Agreement; they are provided to support its
implementation. Annex 1 outlines the need for reform, Annex 2 clarifies the terminology used, Annex 3
suggests a reform journey, and Annex 4 provides an initial toolbox containing practical tools and options
for consideration.

Annex 1 – The need for research assessment reform
Research assessment reform
Shared commitments for research assessment reform, to be achieved in an agreed
timeframe, will enable recognition of the diverse outputs, practices and activities thatmaximise the quality of research and its resulting impacts, facilitate a move away frominappropriate uses of journal- and publication-based metrics, and reinforce trust in
research.
There is broad agreement on the need for research assessment reform - There is broad agreement
among the research community that to further support the quality of research and theattractiveness of research environments, research assessment practices need to be reformed.
This is due to a variety of interconnected underlying reasons, including i) to allow researchassessment to support positive research cultures; ii) to ensure assessment practices stay relevant
as research processes and the expectations of research evolve; and iii) to live up to the increasing
demands placed on research by the many societal, environmental, democratic, and economic
challenges we face. While the motivations behind this process may differ, they all point in the same
direction: reform is needed, primarily to further support the quality of research.
Assessment processes relying predominantly on journal- and publication-based metrics are known toresult in a ‘publish or perish’ culture that falls short of recognising diverse approaches and could comeat the expense of quality – The dominance of narrow journal- and publication-based metrics, which
are often used inappropriately in research assessment, can be a hurdle to the recognition ofdiverse contributions and may negatively affect the quality and impact of research. For example,
this dominance can: promote quantity and speed at the expense of quality and rigour; lead to theemergence of predatory journals and conferences; encourage publishing in paywalled journals
because of their high impact factors, despite the availability of open access alternatives; lead torisk-aversity because taking risks may reduce the chances of publication; generate excessive
attention to rankings that hinders collaboration; and waste efforts, time and resources throughthe duplication of work as ‘negative’ findings go largely unreported. Research assessment
practices should induce a research culture that recognises collaboration, openness, andengagement with society, and that provides opportunities for multiple talents.
Joint commitment is needed that builds on progress made - Several organisations, inspired by the San
Francisco Declaration on Research Assessment (DORA), the principles proposed by the Leiden
Manifesto for research metrics, and the Hong Kong Principles for assessing researchers, havealready started reforming research assessment. Coordinated action now is needed to build on thisand gather sufficient mass to enable systemic reform of research assessment practices.

12

AGREEMENT ON REFORMING RESEARCH ASSESSMENT

20 July 2022

Coalition on research assessment reform
The Coalition invites all organisations to join, and thereby contribute to advancing researchassessment reform, and enable recognition of the diverse practices and activities thatmaximise the quality of research.
Collaboration on the basis of common principles will facilitate progress in research assessment reform– Thus far, progress across research organisations and countries has been uneven, and ongoing
efforts are fragmented. Collaboration on research assessment reform will allow signatories tomove forward on the basis of common principles. This will also diminish the perceived ‘first-moverdisadvantage’ involved in changing a culture of research assessment based on quality, trust andrisk-taking that is applied globally.
The Coalition will allow signatories to test different approaches while avoiding contradictions acrossassessment practices – The Coalition will facilitate exchange of information and mutual learning,
and will enable access to tools, networks and working groups so that decisions can be evidence
informed, and so that those currently less advanced on their reform journey can benefit fromestablished good practice and build capability swiftly. It will allow signatories to, individually orcollectively, develop, pilot and implement, assessment criteria, tools and processes; while avoiding
contradictions across assessment systems, types and purposes, through continuous dialogue. The
Coalition does not aim to overhaul existing practices overnight, but to facilitate reform through an
iterative process that leaves room for diverse starting points and approaches.
To reform research assessment, the Coalition has developed this Agreement, whereby organisationscommit to implement tangible changes within an agreed timeframe and/or share insights on changes
that have already been made – The Coalition on reforming research assessment is global in scopeand brings together research funding organisations, research performing organisations,
national/regional assessment authorities and agencies, as well as associations of the aboveorganisations, learned societies and other relevant organisations, all willing and committed to
improve research assessment practices within an agreed timeframe. Research organisations haveachieved different levels of progress in research assessment reform. Leading organisations in thisfield will be able to share their insights and support the development of a coherent assessment
ecosystem. Organisations that have not yet engaged in reforms will be able to identify and learn
from successful ideas and practices.

Contextual considerations
Favourable framework conditions and sustainable funding are needed to achieve reform –Organisational reform initiatives are sometimes hampered by national regulations or their limited
autonomy under national systems. The Coalition calls on national administrations to ensure thatnational framework conditions facilitate reform. Current inappropriate uses of journal- andpublication-based metrics may be exacerbated by the pressure on research systems due to thevery limited amounts of funding available compared to the pipeline of talented researcherscompeting for that funding. The move towards more qualitative assessment could also require
additional resources if not accompanied by changes in assessment frequency. Sustainable levelsof funding and a balance between competitive and non-competitive funding streams will therefore
be critical to reform.

13

AGREEMENT ON REFORMING RESEARCH ASSESSMENT

20 July 2022

Assessment practices should vary depending on the type and purpose of assessment concerned –Reformed practices for assessing individual researchers for the purposes of recruitment or career
evaluation should consider their individual contexts and careers. Research units should beassessed not only on their research outputs, but also on their broader contribution to the researchecosystem. Research assessment by research funders should consider disciplinary, multi-, inter-,and trans-disciplinary research, as well as the contribution to knowledge generation and scientific,
technological, economic, cultural and societal impact. Many research careers go beyond researchand include teaching, patient care, academic citizenship and other roles, and it is important toconsider this work in their evaluation. While this Agreement focuses on the specific challenges ofimproving research assessment, it may provide an opportunity for academic institutions to extend
the Principles of this Agreement to broader academic assessment that includes research, teachingand service to society.
Reform processes should enable the reinforcement of the autonomy, profile and strategic goals of
research organisations and allow for differences in implementation – Reform will need to be flexible
and adapted by research organisations to take into account the diversity of disciplines, the varietyof competency areas and talents, the differences between cultures, countries and regions, thediversity of languages used in the performance and communication of research, the diversity of
research maturity levels, the diversity of research organisations and their missions, as well as thedifferences between career stages, ranging from early career to senior. The direct involvement ofresearchers and organisations able to represent this diversity will be critical to successful reform.

14

AGREEMENT ON REFORMING RESEARCH ASSESSMENT

20 July 2022

Annex 2 – Glossary
Research assessment – In the context of this Agreement, research assessment encompasses:•

The assessment of research performing organisations and research units, by assessment
authorities, research funding and performing organisations, for the purposes of allocating
funding; public investment accountability; informing decisions on research priorities andimproving the definition and implementation of research strategies.

•

The assessment of research projects by assessment authorities, research funding andperforming organisations, and prize awarding organisations; for the purposes of allocating
funding, informing project management and future research funding decisions, and making
prize and award decisions.

•

The assessment of individual researchers and research teams by research funding andperforming organisations and prize awarding organisations; for the purposes of allocating
funding, recruitment and hiring promotion, professional development review, and prize andaward decisions.

Researcher – In the context of this Agreement the term researcher refers to all fields of researchand scholarly pursuits and at all career stages, including doctoral candidates.
Research culture – In the context of this Agreement, we adopt the Royal Society’s definition ofresearch culture: “Research culture encompasses the behaviours, values, expectations, attitudes
and norms of our research communities. It influences researchers’ career paths and determines
the way that research is conducted and communicated.” 3
Research on research – In the context of this Agreement, research on research (also known as
meta-research, the science of science and meta-science) is defined as the study of research itself.
Quality – In the context of this Agreement, research quality is defined as is captured in theprinciples [under ‘Quality and impact’].
Impact – In the context of this Agreement, impact is defined as is captured in the principles [under‘Quality and impact’].
Diversity – In the context of this Agreement, diversity is defined as is captured in the principles[under ‘Diversity, inclusiveness and collaboration’].
Ethics and integrity – In the context of this Agreement, ethics and integrity are defined as iscaptured in the principles [under ‘Principles for overarching conditions’].

3 https://royalsociety.org/topics-policy/projects/research-culture/

15

AGREEMENT ON REFORMING RESEARCH ASSESSMENT

20 July 2022

Annex 3 – Reform journey: a suggested process forachieving the Commitments
The reform journey 4 sets out a suggested, non-prescriptive step-by-step process to helporganisations achieve the Commitments. This journey is presented as chronological steps;
however, the change process will probably not be chronological, and organisations can adapt thejourney and start from the step they deem most appropriate for their context.
1

Allocate resources, whether in terms of capacity or budget, to actively engage in the reform
journey

2

Communicate your intention to reform, explain how you have started the process ofreviewing or developing criteria, tools and processes in line with the core commitments

3

Evaluate current assessment practices in terms of alignment with the Principles andCommitments, consider also what currently works well and how this can be retained inparallel to any new practice - Re-evaluate at fixed intervals, whenever broad reforms toassessments are implemented, or when problems are identified

4

Engage those being assessed in the development and design of assessment criteria
and processes, work with researchers to enable consideration of differences between
disciplines and career levels

5

Develop existing and design new assessment criteria, tools, and processes withassessors and those that are assessed; consider the diversity of contributions including:diverse outputs beyond journal publications and in different languages; diverse practices
including those that contribute to robustness, openness, transparency, and inclusiveness of
research and the research process including peer review, teamwork and collaboration; anddiverse activities including teaching, leadership, supervision, training, and mentoring,according to the nature of each research discipline

6

Interrogate developed and new approaches by working with assessors and those thatare assessed (e.g. who might new approaches discriminate against; how might they be
gamed; what are the potential unintended consequences)7

Implement developed and new assessment criteria, tools, and processes according to
the Principles and Commitments; consider awareness raising, rewards, policies, training,
infrastructure, and capacity building and include data collection to support monitoring,evaluation and mutual learning

8

Evaluate developed and new assessment criteria, tools, and processes

9

Share data / information, participate in mutual learning within and beyond the
Coalition, supported by mechanisms developed by the Coalition

10 Coordinate with other organisations at national and international level, and promoteinternational coordination and harmonisation
11 Continue to evolve assessment criteria, tools, and processes based on learning fromown evaluations and those of others

4 Inspired by SCOPE (https://inorms.net/scope-framework-for-research-evaluation/)

16

AGREEMENT ON REFORMING RESEARCH ASSESSMENT

20 July 2022

Annex 4 – Toolbox: practical tools and options to
consider� Note to the Reader: we have added the draft toolbox to enable a more comprehensive
overview. However, the toolbox is subject to continuous development and will take the formof a ‘living’ document/webpage.

Commitment

Examples of tools to support this commitment/
options to consider

Recognise the diversity of contributions to,and careers in, research in accordancewith the needs and nature of the research•

Enable greater diversity in career paths and profiles
by recognising more diverse competencies andtalents 5•

Use approaches that allow academics to make a markin one or more key areas of study that areimportant to them, and allow their area profile tochange over the course of their career 6•

Use a portfolio approach to test competencies orprogression in different domains relevant to theresearcher’s role 7•

Consider specific actions captured under the Leiden
Manifesto 8•

Explore options for assessment; as a rule of thumb,use quantitative indicators for quantitative things (ifthat is what is appropriate to assess): publications,funding, citations and students, and qualitative
indicators (such as case studies, narratives orstatements) for qualitative things: excellence, quality,
value, impact, and be very cautious about using
quantitative indicators for qualitative things 9•

Actively engage in and learn from research on research
work to develop new improved metrics, and consider
appropriateness of their use

Base research assessment primarily on
qualitative evaluation for which peer
review is central, supported by responsible
use of quantitative indicators

5 https://www.universiteitenvannederland.nl/recognitionandrewards/wp-content/uploads/2019/11/Position-paperRoom-for-everyone%e2%80%99s-talent.pdf
6 https://www.universiteitenvannederland.nl/recognitionandrewards/wp-content/uploads/2019/11/Position-paperRoom-for-everyone%E2%80%99s-talent.pdf
7 https://embassy.science/wiki/Resource:Ca0ed587-ac8e-4259-9cc7-74de01941cd1;
https://assets-eu-01.kc-usercontent.com/546dd520-97db-01b7-154d-79bb6d950a2d/6eb2e1cc-068a-4283-b6dea281868b749d/Qualification-portfolio-professors-UMC%20Utrecht.pdf
8 https://doi.org/10.1038/520429a
9 https://thebibliomagician.wordpress.com/2019/12/11/introducing-scope-aprocess-for-evaluating-responsibly/

17

AGREEMENT ON REFORMING RESEARCH ASSESSMENT

20 July 2022

Abandon inappropriate uses in researchassessment of journal- and publicationbased metrics, in particular inappropriate
uses of Journal Impact Factor (JIF) and hindex•

Consider specific actions described in the San
Francisco Declaration on Research Assessment(DORA) 10 and the Leiden Manifesto 11

Avoid the use of rankings of researchorganisations in research assessment•

Consider specific actions described in the INORMS 12
tools for rethinking global university rankings•

Consider the recommendations in the Metric Tide
report 13[Part 1 – Criteria for units and institutions]With the direct involvement of researchorganisations and researchers at all career
stages, review and develop criteria forassessing research units and research
performing organisations, while promotinginteroperability•

Consider a ‘narrative CV for institutions’ that could
include case studies on how early sharing of data or
collaboration efforts have resulted in knowledge
generation e.g. others building on shared data or
collaboration leading to outputs or impacts that
otherwise would not have been achieved[Part 2 – Criteria for projects andresearchers] With the direct involvement ofresearchers at all career stages, review anddevelop criteria, tools and processes for theassessment of research projects, researchteams and researchers that are adapted totheir context of application•

Pilot alternative/new assessment criteria, tools, andprocesses (e.g. narrative CV format, competency-based
CV format, evidence-based CV format, diversification ofresearch careers and associated career progression) –a more comprehensive overview of options is set out
under ‘tools to support steps in the reform journey’

Raise awareness of research assessment
reform and provide transparent
communication, guidance, and training on
assessment criteria and processes as wellas their use•

Host webinars to inform applicants of assessment
processes and allow for questions and answers•

Provide training, guidance and support toassessment panels, committees and juries•

Publish webpages and reports to communicate thetransparency of research evaluation processes 14•

Ensure transparency of research assessment
processes - good examples for promoting
transparency include: strict conflict-of-interest
regulations, applicant right-to-reply procedures, andopen (non-anonymous) reviewing 15

Commit resources to reforming researchassessment as is needed to achieve theorganisational changes committed to
Review and develop research assessment
criteria, tools and processes

10 https://sfdora.org/read/
11 https://doi.org/10.1038/520429a
12 https://inorms.net/wp-content/uploads/2022/07/principles-for-fair-and-responsible-university-assessment-v5.pdf
13 https://doi.org/10.13140/RG.2.1.4929.1363
14 https://www.scienceeurope.org/our-priorities/research-assessment/research-assessment-processes/
15 https://www.scienceeurope.org/our-priorities/research-assessment/research-assessment-processes/

18

AGREEMENT ON REFORMING RESEARCH ASSESSMENT

Exchange practices and experiences toenable mutual learning within and beyond
the Coalition

20 July 2022•

Build national assessment consortia, consider existing
national consortia for reforming responsible
assessment: in Finland 16, Norway 17, and theNetherlands 18•

Foster an international debate on recognition andrewards

Communicate progress made on
adherence to the Principles andimplementation of the Commitments
Evaluate practices, criteria and tools basedon solid evidence and the state-of-the-art
in research on research, and make data
openly available for evidence gatheringand research

Step in reform journey

Examples of tools to support this step / options to
considerAllocate resources, whether in terms of
capacity or budget, to actively engage inthe reform journey•

Create an assessment policy committee at yourorganisation

Communicate your intention to reform,
explain how you have started the processof reviewing or developing criteria, tools
and processes in line with the corecommitments•

Consider the SCOPE framework 19 for researchevaluation to support this step

Evaluate current assessment practices interms of alignment with the Principles andCommitments, consider also whatcurrently works well and how this can beretained in parallel to any new practice Re-evaluate at fixed intervals, wheneverbroad reforms to assessments are
implemented, or when problems are
identified•

Consider the HuMetricsHSS Initiative approach 20 tovalue-based assessment and the SCOPE framework 21
to support this step•

Consider reviewing assessment frequency 22•

Consider streamlining application forms for capturing
recurring application details (for example link to
ORCID 23)

16 https://doi.org/10.23847/isbn.9789525995282
17 https://www.uhr.no/en/front-page-carousel/nor-cam-a-toolbox-for-recognition-and-rewards-in-academiccareers.5780.aspx
18 https://recognitionrewards.nl/
19 https://inorms.net/scope-framework-for-research-evaluation/
20 https://humetricshss.org
21 https://inorms.net/scope-framework-for-research-evaluation/
22 https://doi.org/10.31222/osf.io/82rmj
23 https://sfdora.org/resource/swiss-national-science-foundation/

19

AGREEMENT ON REFORMING RESEARCH ASSESSMENT

20 July 2022

Engage those being assessed in thedevelopment and design of assessment
criteria and processes, work withresearchers to enable consideration ofdifferences between disciplines and careerlevels•Consider the SCOPE framework for researchevaluation to support this step 24

Develop existing and design newassessment criteria, tools, and processes
with assessors and those that areassessed•

Consider the future evolution of HRS4R 25 as amechanism that could support institutions in thedevelopment of recruitment processes•

Consider the Open, Transparent and Merit-based
Recruitment of Researchers (OTM-R) 26 checklist for
institutions 27•

Consider when to test specific aspects as part of theprocess, e.g. what needs to be included in applicationmaterials? In interviews? Or in other parts of theprocess?

•

Consider the relevance of quantitative and qualitative
indicators: use quantitative indicators for quantitative
things and qualitative indicators for qualitative things 28•

Diversify indicators (Open science badges; Publons,
ORCID, open peer review; CRediT; Reporting guidelines
e.g. EQUATOR Network) and metrics (Altmetrics,
PlumX) 29•

Consider post peer review funding applications
lottery 30•

Consider CV formats 31 that enable consideration ofdiverse contributions to knowledge, to training anddevelopment of other researchers, to the wider
research community and to society•

Value diverse activities (knowledge transfer activities,
training and mentoring of researchers, public
engagement actions, actions that contribute to a
positive research culture, engagement with key
stakeholders e.g. patients, professionals, politicaldecision-makers)•

Value diverse outputs (FAIR data sets, replication
studies, registered reports 32, pre-prints) in different
languages in accordance with the Helsinki initiative 33

Consider the diversity of contributions

24 https://inorms.net/scope-framework-for-research-evaluation/
25 https://euraxess.ec.europa.eu/jobs/hrs4r
26 https://cdn1.euraxess.org/sites/default/files/policy_library/otm-r-finaldoc_0.pdf
27 https://cdn5.euraxess.org/sites/default/files/policy_library/otm-r-checklist.pdf
28 https://thebibliomagician.wordpress.com/2019/12/11/introducing-scope-aprocess-for-evaluating-responsibly/
29 https://doi.org/10.31222/osf.io/82rmj
30 https://www.nesta.org.uk/feature/innovation-squared/reducing-bias-funding-decisions/
31 https://doi.org/10.31222/osf.io/82rmj
32 https://www.cos.io/initiatives/registered-reports
33 https://www.helsinki-initiative.org/read

20

AGREEMENT ON REFORMING RESEARCH ASSESSMENT

Consider diverse research careers

20 July 2022•

Value diverse impacts•

Consider limiting the number of outputs assessed(moving away from lists of outputs towards limited
selection of key accomplishments and why they areimportant) 34•

Value diverse researcher career profiles•

Value data stewardship, 35 potentially as a crucial
element of some researchers’ roles (avenue for career
diversification)•

Assess alignment with research integrity values•

Value knowledge of rigorous experimental design 36•

Value contributions such as peer review and editorial
roles

•

Value teamwork and collaboration•

Value outputs associated with robust, open, andtransparent research (FAIR data sets, replication
studies, registered reports, 37 pre-prints)•

Consider the UNESCO recommendation on open
science 38•

Value activities associated with openness (training,awareness raising, priority setting partnerships 39,
outreach)•

Value outputs associated with openness (FAIR data
sets, pre-prints, open software, open code,translations) 40•

Value inclusion of stakeholders in the research
process, from defining priority research questions toknowledge translation 41•

Consider diversity in research teams at all levels, andin the content of research and innovation

Consider diverse practices including thosethat contribute to robustness, openness,transparency, and inclusiveness and research
processes including peer review, teamwork
and collaboration
Robustness of research and theresearch process

Openness, transparency of researchand the research process

Inclusiveness of research and theresearch process

34 https://news.cancerresearchuk.org/2018/02/20/improving-research-evaluation-dora/
35 https://doi.org/10.1371/journal.pbio.3000737
36 https://doi.org/10.1371/journal.pbio.3000737
37 https://www.cos.io/initiatives/registered-reports
38 https://en.unesco.org/science-sustainable-future/open-science/recommendation
39 https://www.jla.nihr.ac.uk/about-the-james-lind-alliance/about-psps.htm
40 http://eurodoc.net/open-science-ambassadors-training/m1-open-science
41 https://www.umcutrecht.nl/en/science-in-transition

21

AGREEMENT ON REFORMING RESEARCH ASSESSMENT•

20 July 2022

Consider the EDI dimension:-

Consider 360 42 or organisational rather than
individual references-

Include implicit and unconscious bias training in
training for assessors 43-

Consider post peer review lottery for funding
applications 44-

Consider gender blinding, blinding of other data
prone to biases, e.g. moving educational history
down in applications 45•

Include EDI statement in assessment processes 46•

Assess competencies relevant to a respective role 47•

Consider the international initiative on Rewarding
Teaching 48•

Recognise leadership competencies

Interrogate developed and new
approaches by working with assessors andthose that are assessed (e.g. who might
new approaches discriminate against; how
might they be gamed; what are thepotential unintended consequences)•

Consider the SCOPE framework for researchevaluation to support this step 49

Implement developed and newassessment criteria, tools, and processes
according to the Principles andCommitments; consider awareness raising,rewards, policies, training, infrastructure,and capacity building and include datacollection to support monitoring,evaluation and mutual learning•

Establish training programmes

Evaluate developed and new assessment
criteria, tools, and processes•

Consider the SCOPE framework for researchevaluation to support this step 50

Consider diverse activities including teaching,leadership, supervision, training, andmentoring, according to the nature of eachresearch discipline

42 https://doi.org/10.31222/osf.io/82rmj
43 https://royalsociety.org/topics-policy/publications/2015/unconscious-bias/
44 https://www.nesta.org.uk/feature/innovation-squared/reducing-bias-funding-decisions/45 https://www.science.org/content/article/can-anonymous-faculty-searches-boost-diversity
46 https://ofew.berkeley.edu/recruitment/contributions-diversity/rubric-assessing-candidate-contributions-diversityequity
47 Link to EU competence framework for researchers (Knowledge Ecosystem Project) /https://www.vitae.ac.uk/researchers-professional-development/about-the-vitae-researcher-developmentframework/developing-the-vitae-researcher-development-framework
48 https://www.advancingteaching.com/
49 https://inorms.net/scope-framework-for-research-evaluation/
50 https://inorms.net/scope-framework-for-research-evaluation/

22

AGREEMENT ON REFORMING RESEARCH ASSESSMENT

20 July 2022

Share data / information, participate inmutual learning within and beyond the
Coalition, supported by mechanisms
developed by the Coalition•

Build national research assessment consortia,
consider existing national consortia for reforming
responsible assessment: in Finland, 51 Norway, 52 andthe Netherlands 53

Coordinate with other organisations at
national and international level, andpromote international coordination andharmonisation•

Foster an international debate on recognition andrewards

Continue to evolve assessment criteria,
tools, and processes based on learningfrom own evaluations and those of others•

Engage in research on research

51 https://doi.org/10.23847/isbn.9789525995282
52 https://www.uhr.no/en/front-page-carousel/nor-cam-a-toolbox-for-recognition-and-rewards-in-academiccareers.5780.aspx
53 https://recognitionrewards.nl/

23