# ENVRIplus D10.3: Description of performance criteria for open access and list of performance indicators

Anders Tjulin, Mylène Ndisi, Lilith Kuckero, Sabine Philippin, Simone Gagliardi

► To cite this version:

Anders Tjulin, Mylène Ndisi, Lilith Kuckero, Sabine Philippin, Simone Gagliardi. ENVRIplus D10.3: Description of performance criteria for open access and list of performance indicators. EISCAT. 2019. hal-04618100

HAL Id: hal-04618100

https://hal.science/hal-04618100

Submitted on 21 Jun 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# ENVRIplus D10.3: Description of performance criteria for open access and list of performance indicators

Anders Tjulin, Mylène Ndisi, Lilith Kuckero, Sabine Philippin, Simone

Gagliardi

► To cite this version:

Anders Tjulin, Mylène Ndisi, Lilith Kuckero, Sabine Philippin, Simone Gagliardi. ENVRIplus D10.3: Description of performance criteria for open access and list of performance indicators. EISCAT. 2019. hal-04618100

HAL Id: hal-04618100

https://hal.science/hal-04618100v1

Submitted on 21 Jun 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

ENVRI

D10.3

DESCRIPTION OF PERFORMANCE
CRITERIA FOR OPEN ACCESS AND
LIST OF PERFORMANCE
INDICATORS

WORK PACKAGE 10 – GOVERNANCE AND
SUSTAINABILITY

LEADING BENEFICIARY: CNRS

<table><thead><tr><td>Author:</td><td>Beneficiary/Institution</td></tr></thead><tbody><tr><td>Anders Tjulin</td><td>EISCAT</td></tr><tr><td>Mylène Ndisi</td><td>CNRS</td></tr><tr><td>Lilith Kuckero</td><td>SIOS</td></tr><tr><td>Sabine Philippin</td><td>CNRS</td></tr><tr><td>Simone Gagliardi</td><td>CNR</td></tr></tbody></table>

Accepted by: Paolo Laj (WP 10 leader)

Deliverable type: REPORT

Dissemination level: PUBLIC

Deliverable due date: 31.10.2017/M30

Actual Date of Submission: 01.03.2019/M46

<figure><img src="image_3.png" /></figure>
<figure><img src="image_2.png" /></figure>

ENVRI

D10.3

DESCRIPTION OF PERFORMANCE
CRITERIA FOR OPEN ACCESS AND
LIST OF PERFORMANCE
INDICATORS

WORK PACKAGE 10 – GOVERNANCE AND
SUSTAINABILITY

LEADING BENEFICIARY: CNRS

<table><thead><tr><td>Author:</td><td>Beneficiary/Institution</td></tr></thead><tbody><tr><td>Anders Tjulin</td><td>EISCAT</td></tr><tr><td>Mylène Ndisi</td><td>CNRS</td></tr><tr><td>Lilith Kuckero</td><td>SIOS</td></tr><tr><td>Sabine Philippin</td><td>CNRS</td></tr><tr><td>Simone Gagliardi</td><td>CNR</td></tr></tbody></table>

Accepted by: Paolo Laj (WP 10 leader)

Deliverable type: REPORT

Dissemination level: PUBLIC

Deliverable due date: 31.10.2017/M30

Actual Date of Submission: 01.03.2019/M46

<figure><img src="image_3.png" /></figure>
## ABSTRACT

Environmental Research Infrastructures (RIs) aim at promoting open access to its facilities, resources and services. Facilitating access to the infrastructures and maximizing their use by a wide range of users is a high priority of each RI. In order to implement an efficient and effective access to services, an RI needs to develop excellent procedures for performance evaluation.

This document is geared towards the description of a range of key performance criteria that can be used to quantify and evaluate the performance of environmental RI with respect to their access activities. Appropriate performance indicators should ensure that the metrics used are comprehensive, adaptable and useful, and that the criteria are relevant and measurable.

Key performance indicators for open access are included in the standardized processes implemented by RIs as part of their business strategy. Good practices need to be put in place to ensure that the processes are verified and validated. By developing a framework with appropriate tools for management of access, RIs can ensure that the strategies for access to their facilities and services provide scientific, economic, and environmental benefits.

Project internal reviewer(s):

<table><thead><tr><td>Project internal reviewer(s):</td><td>Beneficiary/Institution</td></tr></thead><tbody><tr><td>André Chanzy (Technical Reviewer)</td><td>INRA</td></tr><tr><td>Helen Glaves (RI Reviewer)</td><td>UKRI</td></tr></tbody></table>

Document history:

<table><thead><tr><td>Date</td><td>Version</td></tr></thead><tbody><tr><td>07.12.2018</td><td>Draft for comments</td></tr><tr><td>25.01.2019</td><td>Corrected version</td></tr><tr><td>01.03.2019</td><td>Accepted</td></tr></tbody></table>

## DOCUMENT AMENDMENT PROCEDURE

Amendments, comments and suggestions should be sent to the author Anders Tjulin (anders.tjulin@eiscat.se)

## TERMINOLOGY

A complete project terminology is available from: https://confluence.egi.eu/pages/viewpage.action?pageId=14452608

<figure><img src="image_4.png" /></figure>
## ABSTRACT

Environmental Research Infrastructures (RIs) aim at promoting open access to its facilities, resources and services. Facilitating access to the infrastructures and maximizing their use by a wide range of users is a high priority of each RI. In order to implement an efficient and effective access to services, an RI needs to develop excellent procedures for performance evaluation.

This document is geared towards the description of a range of key performance criteria that can be used to quantify and evaluate the performance of environmental RI with respect to their access activities. Appropriate performance indicators should ensure that the metrics used are comprehensive, adaptable and useful, and that the criteria are relevant and measurable.

Key performance indicators for open access are included in the standardized processes implemented by RIs as part of their business strategy. Good practices need to be put in place to ensure that the processes are verified and validated. By developing a framework with appropriate tools for management of access, RIs can ensure that the strategies for access to their facilities and services provide scientific, economic, and environmental benefits.

Project internal reviewer(s):

<table><thead><tr><td>Project internal reviewer(s):</td><td>Beneficiary/Institution</td></tr></thead><tbody><tr><td>André Chanzy (Technical Reviewer)</td><td>INRA</td></tr><tr><td>Helen Glaves (RI Reviewer)</td><td>UKRI</td></tr></tbody></table>

Document history:

<table><thead><tr><td>Date</td><td>Version</td></tr></thead><tbody><tr><td>07.12.2018</td><td>Draft for comments</td></tr><tr><td>25.01.2019</td><td>Corrected version</td></tr><tr><td>01.03.2019</td><td>Accepted</td></tr></tbody></table>

## DOCUMENT AMENDMENT PROCEDURE

Amendments, comments and suggestions should be sent to the author Anders Tjulin (anders.tjulin@eiscat.se)

## TERMINOLOGY

A complete project terminology is available from: https://confluence.egi.eu/pages/viewpage.action?pageId=14452608

<figure><img src="image_4.png" /></figure>
PROJECT SUMMARY

ENVRIplus is a Horizon 2020 project bringing together Environmental and Earth System Research Infrastructures, projects and networks together with technical specialist partners to create a more coherent, interdisciplinary and interoperable cluster of Environmental Research Infrastructures across Europe. It is driven by three overarching goals: 1) promoting cross-fertilization between infrastructures, 2) implementing innovative concepts and devices across RIs, and 3) facilitating research and innovation in the field of environment for an increasing number of users outside the RIs.

ENVRIplus aligns its activities to a core strategic plan where sharing multi-disciplinary expertise will be most effective. The project aims to improve Earth observation monitoring systems and strategies, including actions to improve harmonization and innovation, and generate common solutions to many shared information technology and data related challenges. It also seeks to harmonize policies for access and provide strategies for knowledge transfer amongst RIs. ENVRIplus develops guidelines to enhance transdisciplinary use of data and data-products supported by applied use-cases involving RIs from different domains. The project coordinates actions to improve communication and cooperation, addressing Environmental RIs at all levels, from management to end-users, implementing RI-staff exchange programs, generating material for RI personnel, and proposing common strategic developments and actions for enhancing services to users and evaluating the socio-economic impacts.

ENVRIplus is expected to facilitate structuration and improve quality of services offered both within single RIs and at the pan-RI level. It promotes efficient and multi-disciplinary research offering new opportunities to users, new tools to RI managers and new communication strategies for environmental RI communities. The resulting solutions, services and other project outcomes are made available to all environmental RI initiatives, thus contributing to the development of a coherent European RI ecosystem.
PROJECT SUMMARY

ENVRIplus is a Horizon 2020 project bringing together Environmental and Earth System Research Infrastructures, projects and networks together with technical specialist partners to create a more coherent, interdisciplinary and interoperable cluster of Environmental Research Infrastructures across Europe. It is driven by three overarching goals: 1) promoting cross-fertilization between infrastructures, 2) implementing innovative concepts and devices across RIs, and 3) facilitating research and innovation in the field of environment for an increasing number of users outside the RIs.

ENVRIplus aligns its activities to a core strategic plan where sharing multi-disciplinary expertise will be most effective. The project aims to improve Earth observation monitoring systems and strategies, including actions to improve harmonization and innovation, and generate common solutions to many shared information technology and data related challenges. It also seeks to harmonize policies for access and provide strategies for knowledge transfer amongst RIs. ENVRIplus develops guidelines to enhance transdisciplinary use of data and data-products supported by applied use-cases involving RIs from different domains. The project coordinates actions to improve communication and cooperation, addressing Environmental RIs at all levels, from management to end-users, implementing RI-staff exchange programs, generating material for RI personnel, and proposing common strategic developments and actions for enhancing services to users and evaluating the socio-economic impacts.

ENVRIplus is expected to facilitate structuration and improve quality of services offered both within single RIs and at the pan-RI level. It promotes efficient and multi-disciplinary research offering new opportunities to users, new tools to RI managers and new communication strategies for environmental RI communities. The resulting solutions, services and other project outcomes are made available to all environmental RI initiatives, thus contributing to the development of a coherent European RI ecosystem.
TABLE OF CONTENTS

DESCRIPTION OF PERFORMANCE CRITERIA FOR OPEN ACCESS AND LIST OF PERFORMANCE
INDICATORS ..................................................................................................................................................5

1 INTRODUCTION ................................................................................................................................................5

1.1 Terminology ................................................................................................................................................5

1.2 Why key performance indicators? ................................................................................................6

1.3 Example of an RI strategy model evaluated by KPIs................................................................7

2 MAPPING PERFORMANCE CRITERIA................................................................................................8

2.1 Key performance indicators................................................................................................................8

2.2 Methodology................................................................................................................................................9

3 USER METRICS.................................................................................................................................................10

3.1 User access................................................................................................................................................10

3.2 User base................................................................................................................................................10

3.3 Relevance to users ................................................................................................................................11

4 RI OPERATIONAL METRICS................................................................................................................11

4.1 Access provision................................................................................................................................................11

4.2 Service................................................................................................................................................11

4.3 Central management ................................................................................................................................12

4.4 Reliability................................................................................................................................................12

4.5 Visibility................................................................................................................................................12

4.6 User community size and evolution ................................................................................................12

5 STRATEGIC INDICATORS..................................................................................................................13

5.1 Publications and other products..................................................................................................13

5.2 Cross-disciplinarity................................................................................................................................13

5.3 Economic impact................................................................................................................................13

6 FINANCIAL METRICS..................................................................................................................................14

6.1 Service................................................................................................................................................14

6.2 Resources................................................................................................................................................14

6.3 Costs................................................................................................................................................14

6.4 Returns................................................................................................................................................15

7 CONCLUSIONS.................................................................................................................................................15

7.1 Impact on project................................................................................................................................15

7.2 Impact on stakeholders ................................................................................................................15

8 REFERENCES..................................................................................................................................................16

APPENDIX 1: THE ENVIPLUS POST ACCESS QUESTIONNAIRE ................................................................17
# DESCRIPTION OF PERFORMANCE CRITERIA FOR OPEN ACCESS AND LIST OF PERFORMANCE INDICATORS

## 1 INTRODUCTION

A good understanding of the environment and its changes on both local and global scales is fundamental for society. Challenges coming from the dynamics and processes of the Earth system have impact on people, and unforeseen changes may severely affect the economy. Environmental research is addressing these issues through observations, experiments and modelling.

The importance of open access to research infrastructure (RI) facilities and services is clearly noted by the European Strategy Forum on Research Infrastructures (ESFRI) in their 2018 Roadmap [1], which includes a landscape analysis of the main open access RIs in Europe, in all fields, and major new or ongoing projects. Open access is particularly important in the field of environmental sciences due to the inherent complexity and interdependence of the different components of the Earth system. It also opens up possibilities for global coverage and for analysis of historical data to recognise trends.

There is a need to be able to evaluate how well access is implemented by the RIs. The aim of this document is to identify performance criteria for evaluating different aspects of access to research infrastructures in the environmental domain.

The purpose of this document is to provide recommendations for selecting performance criteria and the metrics that can be used by environmental Research Infrastructures to manage the provision of access offered to users. A list of suggested performance indicators is also prepared.

### 1.1 Terminology

Some definitions of terms are needed to simplify the discussion in this text. These are mostly taken from the European Charter for Access to Research Infrastructures [2]. That document was prepared in 2016 by the European Commission to be used as a reference when defining access policies for research infrastructures and related services.

**Access** refers to the legitimate and authorized physical, remote and virtual admission to, interactions with and use of research infrastructures and to services offered by research infrastructures to users. Such access can be granted, amongst others, to data, data-communication services, software, computing resources, sample, archives, observational facilities, experimental facilities, exploratory platforms, simulation chambers, education and training, expert support and analytical services.

**Access types** include **physical access** involving hands-on access, allowing users to physically visit the facility, **virtual access** allowing access through communication channels which usually are online network facilities, and **remote access** providing access to resources remotely.

**Access units** are measures specifying the quantity of access offered by the research infrastructures to the users. Research infrastructures are responsible for the definition of access units, which may vary from precise values like hours or sessions of beam time, to gigabytes transmitted for the conduction of complex experiments and projects up to quotations based on an inventory of users' needs.

<figure><img src="image_5.png" /></figure>
# DESCRIPTION OF PERFORMANCE CRITERIA FOR OPEN ACCESS AND LIST OF PERFORMANCE INDICATORS

## 1 INTRODUCTION

A good understanding of the environment and its changes on both local and global scales is fundamental for society. Challenges coming from the dynamics and processes of the Earth system have impact on people, and unforeseen changes may severely affect the economy. Environmental research is addressing these issues through observations, experiments and modelling.

The importance of open access to research infrastructure (RI) facilities and services is clearly noted by the European Strategy Forum on Research Infrastructures (ESFRI) in their 2018 Roadmap [1], which includes a landscape analysis of the main open access RIs in Europe, in all fields, and major new or ongoing projects. Open access is particularly important in the field of environmental sciences due to the inherent complexity and interdependence of the different components of the Earth system. It also opens up possibilities for global coverage and for analysis of historical data to recognise trends.

There is a need to be able to evaluate how well access is implemented by the RIs. The aim of this document is to identify performance criteria for evaluating different aspects of access to research infrastructures in the environmental domain.

The purpose of this document is to provide recommendations for selecting performance criteria and the metrics that can be used by environmental Research Infrastructures to manage the provision of access offered to users. A list of suggested performance indicators is also prepared.

### 1.1 Terminology

Some definitions of terms are needed to simplify the discussion in this text. These are mostly taken from the European Charter for Access to Research Infrastructures [2]. That document was prepared in 2016 by the European Commission to be used as a reference when defining access policies for research infrastructures and related services.

**Access** refers to the legitimate and authorized physical, remote and virtual admission to, interactions with and use of research infrastructures and to services offered by research infrastructures to users. Such access can be granted, amongst others, to data, data-communication services, software, computing resources, sample, archives, observational facilities, experimental facilities, exploratory platforms, simulation chambers, education and training, expert support and analytical services.

**Access types** include **physical access** involving hands-on access, allowing users to physically visit the facility, **virtual access** allowing access through communication channels which usually are online network facilities, and **remote access** providing access to resources remotely.

**Access units** are measures specifying the quantity of access offered by the research infrastructures to the users. Research infrastructures are responsible for the definition of access units, which may vary from precise values like hours or sessions of beam time, to gigabytes transmitted for the conduction of complex experiments and projects up to quotations based on an inventory of users' needs.

<figure><img src="image_5.png" /></figure>
**Key Performance Indicators (KPIs)** are a type of performance measurements evaluating the success of an organization or of a particular activity in which it engages.

**Open access** to research facilities, resources and services means that they are findable and accessible. Open access and open science are strategies promoted by the European Commission to improve knowledge circulation and innovation. The principles of open access are set out in the Berlin Declaration on Open Access to Knowledge in the Sciences and Humanities [3].

**Research Infrastructures (RIs)** are facilities, resources and services that are used by the research communities to conduct research and foster innovation in their fields. They include: major scientific equipment (or sets of instruments), knowledge-based resources such as collections, archives and scientific data, e-infrastructures, such as data and computing systems and communication networks and any other tools that are essential to achieve excellence in research and innovation. They may be single-sited, virtual and distributed. **Environmental research infrastructures** enable research to foster innovation and societal impact in matters such as climate change, extreme events or loss of biodiversity. The ENVRIplus project is a H2020 project that brings together environmental as well as earth system research infrastructures to create an interdisciplinary and interoperable cluster of environmental research infrastructures in Europe. A majority of the research infrastructures in the environmental domain, as well as in the ENVRIplus project, are categorized as distributed research infrastructures.

**Types of research infrastructures** include **single-sited** RIs with a geographically localized central facility, **distributed** RIs with distributed entities that are operating independently but at the same time are key to the operation of the RI which is coordinated and managed at a central point, **virtual** RIs where access is provided electronically through e-infrastructures, and **other** types of RIs or services either not readily fitting into a single one of the earlier categories or fitting into more than one of them. Since most environmental RIs are distributed, most of the discussion in this document relates to that type.

**Users** of Research Infrastructures can be individuals, teams and institutions from academia, business, industry or public services. They are engaged in the conception or creation of new knowledge, products, processes, methods and systems and also in project management. Teams can include researchers, doctoral candidates, technical staff and students participating in research in the framework of their studies.

## 1.2 Why key performance indicators?

In general, key performance indicators (KPIs) are metrics describing different aspects of the performance of organizations, projects and service providers. They are used to determine how well their objectives are achieved and to identify areas where there is a need for improvements. The use of key performance indicators is particularly important in the monitoring of publicly funded bodies and initiatives where they are used to determine the degree of success. Research infrastructures, particularly distributed ones, contain an additional level of organization and cost. KPIs are in this context also used to identify the impact of the organization of the research infrastructures on the quality of the service and access offered, and of the research performed in order to justify the means that are invested in the coordination of the research infrastructure.

There is a large variation in the ways that research infrastructures are operated and managed, and hence there is no standardized set of key performance indicators for monitoring them. Instead, the research infrastructures need to identify them themselves. However, careful

<figure><img src="image_6.png" /></figure>
**Key Performance Indicators (KPIs)** are a type of performance measurements evaluating the success of an organization or of a particular activity in which it engages.

**Open access** to research facilities, resources and services means that they are findable and accessible. Open access and open science are strategies promoted by the European Commission to improve knowledge circulation and innovation. The principles of open access are set out in the Berlin Declaration on Open Access to Knowledge in the Sciences and Humanities [3].

**Research Infrastructures (RIs)** are facilities, resources and services that are used by the research communities to conduct research and foster innovation in their fields. They include: major scientific equipment (or sets of instruments), knowledge-based resources such as collections, archives and scientific data, e-infrastructures, such as data and computing systems and communication networks and any other tools that are essential to achieve excellence in research and innovation. They may be single-sited, virtual and distributed. **Environmental research infrastructures** enable research to foster innovation and societal impact in matters such as climate change, extreme events or loss of biodiversity. The ENVRIplus project is a H2020 project that brings together environmental as well as earth system research infrastructures to create an interdisciplinary and interoperable cluster of environmental research infrastructures in Europe. A majority of the research infrastructures in the environmental domain, as well as in the ENVRIplus project, are categorized as distributed research infrastructures.

**Types of research infrastructures** include **single-sited** RIs with a geographically localized central facility, **distributed** RIs with distributed entities that are operating independently but at the same time are key to the operation of the RI which is coordinated and managed at a central point, **virtual** RIs where access is provided electronically through e-infrastructures, and **other** types of RIs or services either not readily fitting into a single one of the earlier categories or fitting into more than one of them. Since most environmental RIs are distributed, most of the discussion in this document relates to that type.

**Users** of Research Infrastructures can be individuals, teams and institutions from academia, business, industry or public services. They are engaged in the conception or creation of new knowledge, products, processes, methods and systems and also in project management. Teams can include researchers, doctoral candidates, technical staff and students participating in research in the framework of their studies.

## 1.2 Why key performance indicators?

In general, key performance indicators (KPIs) are metrics describing different aspects of the performance of organizations, projects and service providers. They are used to determine how well their objectives are achieved and to identify areas where there is a need for improvements. The use of key performance indicators is particularly important in the monitoring of publicly funded bodies and initiatives where they are used to determine the degree of success. Research infrastructures, particularly distributed ones, contain an additional level of organization and cost. KPIs are in this context also used to identify the impact of the organization of the research infrastructures on the quality of the service and access offered, and of the research performed in order to justify the means that are invested in the coordination of the research infrastructure.

There is a large variation in the ways that research infrastructures are operated and managed, and hence there is no standardized set of key performance indicators for monitoring them. Instead, the research infrastructures need to identify them themselves. However, careful

<figure><img src="image_6.png" /></figure>
consideration is needed when determining these indicators. If they are chosen badly, they can
not only be useless but instead actually be harmful for the research infrastructure. For instance,
if an unrealistic target value for an indicator is chosen, there is a risk that the research
infrastructure is forced to be operated in scientifically less useful ways just to satisfy this target.
Other risks are an increased rigidity in the operations of the research infrastructure, a higher
level of bureaucracy, and an increased focus on numbers only, and an increased focus on
achieving these self-imposed targets to the detriment of the RI.

The European Commission has identified a set of general principles [4] for identifying good key performance indicators. The KPIs should be “RACER”, i.e.:

* Relevant and closely linked to the objectives to be reached. They should not be overambitious and should measure the right thing.

* Accepted by the research infrastructures and the stakeholders. The role and responsibilities for the indicator need to be well defined.

* Credible for non-experts, unambiguous and easy to interpret. Indicators should be as simple and robust as possible.

* Easy to monitor at low cost and effort.

* Robust against manipulation.

Used properly, the key performance indicators can be used to track the performance of a
research infrastructure over time, and thus provide a powerful tool to ensure its long-term
sustainability. Key performance indicators should generally be aligned with the objectives and
strategies specific to the individual research infrastructure. In addition to the RACER criteria, the
collection of data to produce the KPIs must follow the relevant legal regulations for data
collection, such as GDPR [5].

In this document we are focusing on key performance indicators used to evaluate the success of
environmental research infrastructures in the ENVRIplus community when providing open
access. The purpose here is to identify indicators using quantitative and qualitative means that
are fully inclusive, and that will contribute to the development of sustainability of access
provision. It could also be used as a reference document for research infrastructures in the
environmental domain during their construction phase as an aid to achieving strategic
excellence.

For research infrastructures, a key factor to determine success is their sustainability, which can
be assessed from aspects such as scientific, economic, and social value. Based on the type of
infrastructure, it is determined by providing added value, through long term service provision
and user satisfaction, and relevance to both users and the society. Sustainability is determined
not only by revenue sources, but also by a financial mode allowing innovation stimulants,
contextual and structural influence and relevance of technical aspects in the product and service
provision.

Following the conclusions from an OECD policy document [6], the above are best established
through several phases during the lifetime of a research infrastructure. This is particularly true in
the preparatory phase, for later application during the implementation, operations and
decommissioning phases of the project.

## 1.3 Example of an RI strategy model evaluated by KPIs

Key performance indicators assist in determining whether the inputs and activities related to a
research infrastructure lead to the expected output and outcomes, and the resulting impact.

<figure><img src="image_7.png" /></figure>
consideration is needed when determining these indicators. If they are chosen badly, they can
not only be useless but instead actually be harmful for the research infrastructure. For instance,
if an unrealistic target value for an indicator is chosen, there is a risk that the research
infrastructure is forced to be operated in scientifically less useful ways just to satisfy this target.
Other risks are an increased rigidity in the operations of the research infrastructure, a higher
level of bureaucracy, and an increased focus on numbers only, and an increased focus on
achieving these self-imposed targets to the detriment of the RI.

The European Commission has identified a set of general principles [4] for identifying good key performance indicators. The KPIs should be “RACER”, i.e.:

* Relevant and closely linked to the objectives to be reached. They should not be overambitious and should measure the right thing.

* Accepted by the research infrastructures and the stakeholders. The role and responsibilities for the indicator need to be well defined.

* Credible for non-experts, unambiguous and easy to interpret. Indicators should be as simple and robust as possible.

* Easy to monitor at low cost and effort.

* Robust against manipulation.

Used properly, the key performance indicators can be used to track the performance of a
research infrastructure over time, and thus provide a powerful tool to ensure its long-term
sustainability. Key performance indicators should generally be aligned with the objectives and
strategies specific to the individual research infrastructure. In addition to the RACER criteria, the
collection of data to produce the KPIs must follow the relevant legal regulations for data
collection, such as GDPR [5].

In this document we are focusing on key performance indicators used to evaluate the success of
environmental research infrastructures in the ENVRIplus community when providing open
access. The purpose here is to identify indicators using quantitative and qualitative means that
are fully inclusive, and that will contribute to the development of sustainability of access
provision. It could also be used as a reference document for research infrastructures in the
environmental domain during their construction phase as an aid to achieving strategic
excellence.

For research infrastructures, a key factor to determine success is their sustainability, which can
be assessed from aspects such as scientific, economic, and social value. Based on the type of
infrastructure, it is determined by providing added value, through long term service provision
and user satisfaction, and relevance to both users and the society. Sustainability is determined
not only by revenue sources, but also by a financial mode allowing innovation stimulants,
contextual and structural influence and relevance of technical aspects in the product and service
provision.

Following the conclusions from an OECD policy document [6], the above are best established
through several phases during the lifetime of a research infrastructure. This is particularly true in
the preparatory phase, for later application during the implementation, operations and
decommissioning phases of the project.

## 1.3 Example of an RI strategy model evaluated by KPIs

Key performance indicators assist in determining whether the inputs and activities related to a
research infrastructure lead to the expected output and outcomes, and the resulting impact.

<figure><img src="image_7.png" /></figure>
<figure><img src="image_8.png" /></figure>

As a simple example, we can consider the general structure of the ENVRIplus project:

* **Inputs** exist in the form of external funding, investment, knowledge, expertise, data, and other resources which are meant to support **activities** such as transnational access to research infrastructures, data provision, and calibration services.

* **Output** from the project is in the form of publications, datasets, and conference proceedings, as well as less tangible **outcomes** such as capacity and community building, and expansion of the knowledge base.

* **Impacts** from the project would be changes in policies, societal change, and technological developments leading to increased productivity, innovation, economic growth, job creation and general well-being.

KPIs are therefore relevant at each of these stages for evaluation of the relevance of the given operational structure.

## 2 MAPPING PERFORMANCE CRITERIA

Key performance indicators should be set up at an initial stage in the lifetime of a research infrastructure, preferably during the establishment of the business model as part of the strategic plan for the infrastructure during its conception. However, they should be subject to periodic review to ensure they are fit for purpose and providing the necessary information. The order to set up the KPIs is to

1. Identify which aspects of the research infrastructure performance need evaluation

2. Choose metrics to use for monitoring those aspects

3. Set realistic targets for these indicators so as to verify if success has been achieved

The KPIs should be determined by the needs of the individual RI, and not by the metrics.

### 2.1 Key performance indicators

In general, the key performance indicators for research infrastructures fall into four categories:

1. **User metrics**, which are set in place reflecting who the stakeholders are on European, national and regional levels. These stakeholders include potential funders, scientific users, policy makers and society.

2. The **operational metrics** are conceived from the evaluation of the products or services to be developed and offered during the lifetime of the infrastructure.

3. The use of **financial metrics** comes from the need to understand financial drivers and matching revenue sources. This includes weighing the research infrastructure use against available funding, and thus evaluating the need for charges for access or facilities.

4. The research infrastructure needs to be relevant to science and to the designated community, and thus we need **strategic metrics** as indicators.

<figure><img src="image_9.png" /></figure>
<figure><img src="image_8.png" /></figure>

As a simple example, we can consider the general structure of the ENVRIplus project:

* **Inputs** exist in the form of external funding, investment, knowledge, expertise, data, and other resources which are meant to support **activities** such as transnational access to research infrastructures, data provision, and calibration services.

* **Output** from the project is in the form of publications, datasets, and conference proceedings, as well as less tangible **outcomes** such as capacity and community building, and expansion of the knowledge base.

* **Impacts** from the project would be changes in policies, societal change, and technological developments leading to increased productivity, innovation, economic growth, job creation and general well-being.

KPIs are therefore relevant at each of these stages for evaluation of the relevance of the given operational structure.

## 2 MAPPING PERFORMANCE CRITERIA

Key performance indicators should be set up at an initial stage in the lifetime of a research infrastructure, preferably during the establishment of the business model as part of the strategic plan for the infrastructure during its conception. However, they should be subject to periodic review to ensure they are fit for purpose and providing the necessary information. The order to set up the KPIs is to

1. Identify which aspects of the research infrastructure performance need evaluation

2. Choose metrics to use for monitoring those aspects

3. Set realistic targets for these indicators so as to verify if success has been achieved

The KPIs should be determined by the needs of the individual RI, and not by the metrics.

### 2.1 Key performance indicators

In general, the key performance indicators for research infrastructures fall into four categories:

1. **User metrics**, which are set in place reflecting who the stakeholders are on European, national and regional levels. These stakeholders include potential funders, scientific users, policy makers and society.

2. The **operational metrics** are conceived from the evaluation of the products or services to be developed and offered during the lifetime of the infrastructure.

3. The use of **financial metrics** comes from the need to understand financial drivers and matching revenue sources. This includes weighing the research infrastructure use against available funding, and thus evaluating the need for charges for access or facilities.

4. The research infrastructure needs to be relevant to science and to the designated community, and thus we need **strategic metrics** as indicators.

<figure><img src="image_9.png" /></figure>
The key performance indicators in these four categories are used for different purposes. In the context of assessing access to research infrastructures, the user metrics are useful in evaluating the users to whom access is provided. The operational metrics are then used for evaluation of the organizational capacity in providing access, the financial metrics are weighing costs against benefits for sustainable access provision, and strategic metrics are useful in ensuring relevance to science and technology development, as well as to society.

Measuring actual performance of an RI against a set of target indicators helps to draw valuable quantitative and qualitative conclusions.

## 2.2 Methodology

The following set of suggested performance indicators related to RI access were compiled partly from the ACTRIS<fnref n="1" /> concept document [7], and the SIOS<fnref n="2" /> key performance indicators specification [8]. They should be adjusted to be in line with the specific strategies and objectives of the individual research infrastructure to be of best use. In particular, the target values of these indicators should be individually assessed.

In some cases, it may be tempting to put together more than one metric into a combined weighted level of achievement. This means that if, for instance, one metric with weight 0.8 has reached 100% of its target value and another metric with weight 0.2 to 50% of its target value, the combined weighted level of achievement is $0.8 \times 100\% + 0.2 \times 50\% = 90\%$. If a KPI is implemented in this way, it should be done with greatest possible care so that the combined indicator is reliable and, in particular, easy to interpret.

One of the objectives of quantifying the performance and criteria of access and service provision using KPIs is the establishment of target values against which the achieved value will be evaluated. Particular care is needed to ensure that the target value is fully justified. It is also possible that the target value will be changed over the lifetime of the research infrastructure, as the RI itself may change over time. An example of KPI target assessment is given in Table 1. It should be noted that it is also important to have a time scale connected to each key performance indicator.

<table><caption>Table 1: Example of assessment of the number of users per year. If the annual number of users is 12, while the target is 15, the level of achievement is 80%. This type of assessment can be repeated for other KPIs where a target value is identified.</caption><thead><tr><td>Criterium</td><td>Indicator</td><td>Description</td><td>Type of value</td><td>Planned</td><td>Achieved</td><td>Achievement</td></tr></thead><tbody><tr><td>Users</td><td>Number of users per year</td><td>Measure of success of RI service provision</td><td>Numerical</td><td>15</td><td>12</td><td>80%</td></tr></tbody></table>

<figure><img src="image_10.png" /></figure>

<footnote n="1">¹ ACTRIS (https://www.actris.eu/), Aerosol, Clouds and Trace Gases Research Infrastructure.</footnote>

<footnote n="2">² SIOS (https://sios-svalbard.org/), Svalbard Integrated Arctic Earth Observation System</footnote>
The key performance indicators in these four categories are used for different purposes. In the context of assessing access to research infrastructures, the user metrics are useful in evaluating the users to whom access is provided. The operational metrics are then used for evaluation of the organizational capacity in providing access, the financial metrics are weighing costs against benefits for sustainable access provision, and strategic metrics are useful in ensuring relevance to science and technology development, as well as to society.

Measuring actual performance of an RI against a set of target indicators helps to draw valuable quantitative and qualitative conclusions.

## 2.2 Methodology

The following set of suggested performance indicators related to RI access were compiled partly from the ACTRIS<fnref n="1" /> concept document [7], and the SIOS<fnref n="2" /> key performance indicators specification [8]. They should be adjusted to be in line with the specific strategies and objectives of the individual research infrastructure to be of best use. In particular, the target values of these indicators should be individually assessed.

In some cases, it may be tempting to put together more than one metric into a combined weighted level of achievement. This means that if, for instance, one metric with weight 0.8 has reached 100% of its target value and another metric with weight 0.2 to 50% of its target value, the combined weighted level of achievement is $0.8 \times 100\% + 0.2 \times 50\% = 90\%$. If a KPI is implemented in this way, it should be done with greatest possible care so that the combined indicator is reliable and, in particular, easy to interpret.

One of the objectives of quantifying the performance and criteria of access and service provision using KPIs is the establishment of target values against which the achieved value will be evaluated. Particular care is needed to ensure that the target value is fully justified. It is also possible that the target value will be changed over the lifetime of the research infrastructure, as the RI itself may change over time. An example of KPI target assessment is given in Table 1. It should be noted that it is also important to have a time scale connected to each key performance indicator.

<table><caption>Table 1: Example of assessment of the number of users per year. If the annual number of users is 12, while the target is 15, the level of achievement is 80%. This type of assessment can be repeated for other KPIs where a target value is identified.</caption><thead><tr><td>Criterium</td><td>Indicator</td><td>Description</td><td>Type of value</td><td>Planned</td><td>Achieved</td><td>Achievement</td></tr></thead><tbody><tr><td>Users</td><td>Number of users per year</td><td>Measure of success of RI service provision</td><td>Numerical</td><td>15</td><td>12</td><td>80%</td></tr></tbody></table>

<figure><img src="image_10.png" /></figure>

<footnote n="1">¹ ACTRIS (https://www.actris.eu/), Aerosol, Clouds and Trace Gases Research Infrastructure.</footnote>

<footnote n="2">² SIOS (https://sios-svalbard.org/), Svalbard Integrated Arctic Earth Observation System</footnote>
## 3 USER METRICS

User metrics help mapping the overall and potential usage of an environmental research infrastructure. They include categories such as access, users, and relevance to users. It is important to make sure that the collection and processing of the RI user data is made in accordance to applicable data protection regulations such as GDPR.

### 3.1 User access

These indicators take account of requests for product and service received.

* User interest

* Number of access requests

* Access application success

* Number of approved access requests

* Usage capacity

* Number of provided access units

We should note that a high percentage of approved access requests could be interpreted in more than one way. It could mean that the quality of the access requests is good, but it could also mean that the number of access requests is low compared to the capacity of the RI.

### 3.2 User base

These indicators are used to measure size and composition of the user community.

* User base size

* Number of users of the research infrastructure

* User experience level

* Number of experienced users

* Number of young researchers

* Number of new users

* User gender balance

* User types

* Number of users from public sector

* Number of users from private sector/business/industry

* Number of users from academic and public research organizations

* Number of users from non-profit research organizations/NGOs

* Number of users from the general public

* User field

* Number of users per scientific field

* User needs

* User requests for training, calibration, experiment, innovation, education, data modelling, etc.

* User national diversity

* Number of users per RI member country

* Number of users not from RI member countries

* Overall geographic distribution of users

Mapping of usage is important for gauging if the infrastructure is maintaining its user base whilst also attracting new users. This type of data could also help in the adaption of communication strategies for the RI. Similarly, a gender imbalance could be a sign that targeted communication is needed. The composition of the nationalities of the user base is relevant for some RIs in order to satisfy requirements from their funding agencies.
## 3 USER METRICS

User metrics help mapping the overall and potential usage of an environmental research infrastructure. They include categories such as access, users, and relevance to users. It is important to make sure that the collection and processing of the RI user data is made in accordance to applicable data protection regulations such as GDPR.

### 3.1 User access

These indicators take account of requests for product and service received.

* User interest

* Number of access requests

* Access application success

* Number of approved access requests

* Usage capacity

* Number of provided access units

We should note that a high percentage of approved access requests could be interpreted in more than one way. It could mean that the quality of the access requests is good, but it could also mean that the number of access requests is low compared to the capacity of the RI.

### 3.2 User base

These indicators are used to measure size and composition of the user community.

* User base size

* Number of users of the research infrastructure

* User experience level

* Number of experienced users

* Number of young researchers

* Number of new users

* User gender balance

* User types

* Number of users from public sector

* Number of users from private sector/business/industry

* Number of users from academic and public research organizations

* Number of users from non-profit research organizations/NGOs

* Number of users from the general public

* User field

* Number of users per scientific field

* User needs

* User requests for training, calibration, experiment, innovation, education, data modelling, etc.

* User national diversity

* Number of users per RI member country

* Number of users not from RI member countries

* Overall geographic distribution of users

Mapping of usage is important for gauging if the infrastructure is maintaining its user base whilst also attracting new users. This type of data could also help in the adaption of communication strategies for the RI. Similarly, a gender imbalance could be a sign that targeted communication is needed. The composition of the nationalities of the user base is relevant for some RIs in order to satisfy requirements from their funding agencies.
### 3.3 Relevance to users

The overall experience of the users of an RI is an important factor when evaluating its success. The metrics below are used to assess the different aspects of the user experience and find areas for improvement.

* User experience

* User opinion on quality of service

* User opinion on ease to access communication channels about available services

* User opinion on ease of admittance procedure

* User opinion on documentation

* User opinion on on-site scientific and technical support

* User opinion on data interface

* User opinion on the scientific benefits of the access

* User experience of the central management (for distributed RIs)

* User suggestions for improvement

The feed-back from the users is preferably collected by letting them fill in a simple poll about their experience after their RI access. For instance, such a questionnaire was used in connection to the ENVRIplus activities connected to "Multi-domain Access to Environmental Research Infrastructures" in Work Package 11. That questionnaire is included in this document as Appendix 1.

## 4 RI OPERATIONAL METRICS

The operational metrics are used to evaluate the organizational capacity to provide access. They include categories such as access, service, reliability, visibility, and size.

### 4.1 Access provision

These indicators evaluate the access provided in general terms. See Section 1.1 for the terminology used.

* General access overview

* Type of access

* Type of access units

* Physical access provided

* Number of access units provided

* Virtual access provided

* Number of data downloads

* Amount of data downloaded

A general overview of the access should include the types of access provided, and how the access units are defined.

### 4.2 Service

These indicators assess the access to the services provided by the RI.

* Service provision support

* A measure of the provided operational support for specific research activities

* Platform access

* A measure of the service level provided by scientific platforms such as ice-breakers, balloons, biodomes etc.

* Data provision

<figure><img src="image_11.png" /></figure>
### 3.3 Relevance to users

The overall experience of the users of an RI is an important factor when evaluating its success. The metrics below are used to assess the different aspects of the user experience and find areas for improvement.

* User experience

* User opinion on quality of service

* User opinion on ease to access communication channels about available services

* User opinion on ease of admittance procedure

* User opinion on documentation

* User opinion on on-site scientific and technical support

* User opinion on data interface

* User opinion on the scientific benefits of the access

* User experience of the central management (for distributed RIs)

* User suggestions for improvement

The feed-back from the users is preferably collected by letting them fill in a simple poll about their experience after their RI access. For instance, such a questionnaire was used in connection to the ENVRIplus activities connected to "Multi-domain Access to Environmental Research Infrastructures" in Work Package 11. That questionnaire is included in this document as Appendix 1.

## 4 RI OPERATIONAL METRICS

The operational metrics are used to evaluate the organizational capacity to provide access. They include categories such as access, service, reliability, visibility, and size.

### 4.1 Access provision

These indicators evaluate the access provided in general terms. See Section 1.1 for the terminology used.

* General access overview

* Type of access

* Type of access units

* Physical access provided

* Number of access units provided

* Virtual access provided

* Number of data downloads

* Amount of data downloaded

A general overview of the access should include the types of access provided, and how the access units are defined.

### 4.2 Service

These indicators assess the access to the services provided by the RI.

* Service provision support

* A measure of the provided operational support for specific research activities

* Platform access

* A measure of the service level provided by scientific platforms such as ice-breakers, balloons, biodomes etc.

* Data provision

<figure><img src="image_11.png" /></figure>
* The number of downloads from the data storage

These types of metrics are very dependent on the type of research infrastructure and the domain in which it is active. In particular, data downloads are increasingly made through web services and machine-machine interactions and there is thus an increasing trend in general in the number of downloads. Care should therefore be taken to ensure that the target value for an indicator involving data provision reflects this trend.

## 4.3 Central management

For distributed environmental research infrastructures, it is important to assess the quality of the central management and the single-entry point.

* Single-entry point

* Evaluation of the access process at the single-entry point of contact

The data for these types of assessments can be obtained through a user feed-back questionnaire.

## 4.4 Reliability

The users of a research infrastructure should trust that it is reliable. If they are sure that it will work as expected they are more open to use the facilities in their research.

* Operational reliability

* A measure of RI downtime not planned in advance

* Scientific reliability

* A measure of the degree to which the research facility is up to standards

The scientific reliability data can be collected in the form of a user feedback questionnaire.

## 4.5 Visibility

For the relevance of a research infrastructure, it is important to attract users. The first step towards engaging new users is by raising awareness of the research infrastructure. That is one of the main reasons why it is vital for an RI to estimate its general visibility.

* Visibility of RI communication

* Measures using indicators derived from social media platforms

* Web presence

* Number of IP addresses accessing the web portal of the research infrastructure

Many indicators of visibility are used by social media platforms. Twitter analytics<fnref n="3" /> is one example. These types of indicators can be adopted for usage by RIs. In order to get a reliable estimate of the web presence, some filtering is required to remove spurious data for instance coming from bot activity on the network.

## 4.6 User community size and evolution

These indicators measure the size and evolution of the user community.

* User community size

* Number of users

* User community growth

<figure><img src="image_12.png" /></figure>

<footnote n="3">³ Twitter analytics: https://business.twitter.com/en/analytics.html</footnote>
* The number of downloads from the data storage

These types of metrics are very dependent on the type of research infrastructure and the domain in which it is active. In particular, data downloads are increasingly made through web services and machine-machine interactions and there is thus an increasing trend in general in the number of downloads. Care should therefore be taken to ensure that the target value for an indicator involving data provision reflects this trend.

## 4.3 Central management

For distributed environmental research infrastructures, it is important to assess the quality of the central management and the single-entry point.

* Single-entry point

* Evaluation of the access process at the single-entry point of contact

The data for these types of assessments can be obtained through a user feed-back questionnaire.

## 4.4 Reliability

The users of a research infrastructure should trust that it is reliable. If they are sure that it will work as expected they are more open to use the facilities in their research.

* Operational reliability

* A measure of RI downtime not planned in advance

* Scientific reliability

* A measure of the degree to which the research facility is up to standards

The scientific reliability data can be collected in the form of a user feedback questionnaire.

## 4.5 Visibility

For the relevance of a research infrastructure, it is important to attract users. The first step towards engaging new users is by raising awareness of the research infrastructure. That is one of the main reasons why it is vital for an RI to estimate its general visibility.

* Visibility of RI communication

* Measures using indicators derived from social media platforms

* Web presence

* Number of IP addresses accessing the web portal of the research infrastructure

Many indicators of visibility are used by social media platforms. Twitter analytics<fnref n="3" /> is one example. These types of indicators can be adopted for usage by RIs. In order to get a reliable estimate of the web presence, some filtering is required to remove spurious data for instance coming from bot activity on the network.

## 4.6 User community size and evolution

These indicators measure the size and evolution of the user community.

* User community size

* Number of users

* User community growth

<figure><img src="image_12.png" /></figure>

<footnote n="3">³ Twitter analytics: https://business.twitter.com/en/analytics.html</footnote>
* Increase of the number of users

* User retention

* Number of returning users

For the long-term perspective, it is often important for a research infrastructure to ensure a continued renewal of its user base. However, it is also important to make sure that the users of an RI find it useful and thus return to use it again.

# 5 STRATEGIC INDICATORS

Strategic indicators are used to ensure that the research infrastructure is relevant for the scientific progress and for the community at large.

## 5.1 Publications and other products

These KPIs are used to measure the outputs of the research infrastructure.

* Impact from RI access

* Number of published peer reviewed papers

* Number of datasets used in journal publications

* Number of communication activities at workshops and conferences

* Number of communications via media

* Number of citations

* Number of data set citations

* Number of patents

* Scientific and technological progress

* Level of enhanced accuracy of measurements

* Number of technological products, methods and developments to knowledge-based resources

* Number of new services

The traditional way to evaluate the scientific and development activity connected to research infrastructures is by counting publications and other outputs. This is not always the best way to do it since these numbers are relatively easy to manipulate. On the other hand, these are the KPIs that are most often used by funding agencies and other stakeholders to determine the success of an RI, so these numbers should still be collected by the RI.

## 5.2 Cross-disciplinarity

For some research infrastructures it is important to actively promote cross-disciplinarity in the research being carried out.

* Cross-disciplinary activities

* Number of access units provided to projects demonstrating clear cross-disciplinary attributes

## 5.3 Economic impact

The socio-economic impact of the research infrastructure is important to measure to provide an overview of its strategic importance.

* Investment in students and early career researchers

* Number of young users

* Education and capacity building

* Number of users participating in training activities

* Job creation

<figure><img src="image_13.png" /></figure>
* Increase of the number of users

* User retention

* Number of returning users

For the long-term perspective, it is often important for a research infrastructure to ensure a continued renewal of its user base. However, it is also important to make sure that the users of an RI find it useful and thus return to use it again.

# 5 STRATEGIC INDICATORS

Strategic indicators are used to ensure that the research infrastructure is relevant for the scientific progress and for the community at large.

## 5.1 Publications and other products

These KPIs are used to measure the outputs of the research infrastructure.

* Impact from RI access

* Number of published peer reviewed papers

* Number of datasets used in journal publications

* Number of communication activities at workshops and conferences

* Number of communications via media

* Number of citations

* Number of data set citations

* Number of patents

* Scientific and technological progress

* Level of enhanced accuracy of measurements

* Number of technological products, methods and developments to knowledge-based resources

* Number of new services

The traditional way to evaluate the scientific and development activity connected to research infrastructures is by counting publications and other outputs. This is not always the best way to do it since these numbers are relatively easy to manipulate. On the other hand, these are the KPIs that are most often used by funding agencies and other stakeholders to determine the success of an RI, so these numbers should still be collected by the RI.

## 5.2 Cross-disciplinarity

For some research infrastructures it is important to actively promote cross-disciplinarity in the research being carried out.

* Cross-disciplinary activities

* Number of access units provided to projects demonstrating clear cross-disciplinary attributes

## 5.3 Economic impact

The socio-economic impact of the research infrastructure is important to measure to provide an overview of its strategic importance.

* Investment in students and early career researchers

* Number of young users

* Education and capacity building

* Number of users participating in training activities

* Job creation

<figure><img src="image_13.png" /></figure>
* Number of people hired directly by the RI

* Impact of the RI in the local and regional labor markets

* Business development

* Measure of the amount of technology transfer

* Innovation and connections to SMEs

* Number of startups connected to the RI

* Number of connections to Small and Medium-sized Enterprises

An RI not only creates jobs by hiring personnel directly, it may also give rise to more diversified local labor markets, some increased human capital levels, consumption of local goods and facilities, and may also attract highly-skilled professionals to the region.

## 6 FINANCIAL METRICS

Financial management of an environmental research infrastructure involves a clear mapping of its funding resources. These could be either national funding, research infrastructure funding, funding from the European Commission, user fees, or a combination of all of these. A certain amount of resources may also come from the industry. In addition, careful evaluations of fixed or recurring funding models are usually made when establishing a research infrastructure while investigating possible investment strategies, which in turn depend on the desired scientific outcome.

Cost calculations are carried out to determine the return on investment (if any) for the monetary value of the research infrastructure. This is done in order to understand the contribution of the financiers to an infrastructure and for tracking its financial evolution allowing the most favorable funding model to be maintained. Ensuring sustainable access provision therefore includes identification of the scientific needs and calculation of the cost of the services that are provided by the RI. The sustainability of RI access is discussed in more detail in Deliverable 10.4 of this project.

### 6.1 Service

These are measures of the service level of the RI.

* Service level

* Number of access units provided

* Level of access units provided related to the potential access

* Contribution of access funding to the financial sustainability of the platform

### 6.2 Resources

These are measures of the resources available for the RI.

* Type of funding

* Funding sources for the RI

* Level of funding

* The amount of funding for the RI from the different sources

### 6.3 Costs

These are measures of the costs for the RI to provide access to users.

* Access costs

* Access costs for the research infrastructure

<figure><img src="image_14.png" /></figure>
* Number of people hired directly by the RI

* Impact of the RI in the local and regional labor markets

* Business development

* Measure of the amount of technology transfer

* Innovation and connections to SMEs

* Number of startups connected to the RI

* Number of connections to Small and Medium-sized Enterprises

An RI not only creates jobs by hiring personnel directly, it may also give rise to more diversified local labor markets, some increased human capital levels, consumption of local goods and facilities, and may also attract highly-skilled professionals to the region.

## 6 FINANCIAL METRICS

Financial management of an environmental research infrastructure involves a clear mapping of its funding resources. These could be either national funding, research infrastructure funding, funding from the European Commission, user fees, or a combination of all of these. A certain amount of resources may also come from the industry. In addition, careful evaluations of fixed or recurring funding models are usually made when establishing a research infrastructure while investigating possible investment strategies, which in turn depend on the desired scientific outcome.

Cost calculations are carried out to determine the return on investment (if any) for the monetary value of the research infrastructure. This is done in order to understand the contribution of the financiers to an infrastructure and for tracking its financial evolution allowing the most favorable funding model to be maintained. Ensuring sustainable access provision therefore includes identification of the scientific needs and calculation of the cost of the services that are provided by the RI. The sustainability of RI access is discussed in more detail in Deliverable 10.4 of this project.

### 6.1 Service

These are measures of the service level of the RI.

* Service level

* Number of access units provided

* Level of access units provided related to the potential access

* Contribution of access funding to the financial sustainability of the platform

### 6.2 Resources

These are measures of the resources available for the RI.

* Type of funding

* Funding sources for the RI

* Level of funding

* The amount of funding for the RI from the different sources

### 6.3 Costs

These are measures of the costs for the RI to provide access to users.

* Access costs

* Access costs for the research infrastructure

<figure><img src="image_14.png" /></figure>
## 6.4 Returns

Metrics evaluating the benefits or returns on investment for RI access.

* Value on investment

* Similar to the strategic indicators in Section 5 of this document

* Economic return of investment

* Monetary income for the RI

We should keep in mind that the value on investment for a research infrastructure is not limited to the financial return, but instead often comes in less tangible forms such as visibility, scientific output or international collaboration.

# 7 CONCLUSIONS

This document outlines the principles for using key performance indicators, and how a research infrastructure can use them to monitor performance.

The most important principle detailed in this document is that the research infrastructure itself needs to identify its own set of key performance indicators. These should be relevant, accepted, credible, easy to monitor and robust against manipulation.

This document also includes an extensive list of suggested key performance indicators that could be used by a research infrastructure as inspiration when identifying ways to gauge its performance.

## 7.1 Impact on project

This document is one of the results from the activities in Work Package 10 on developing governance tools and recommendations for access to environmental research infrastructures. It is one in a set of four deliverables providing guidelines, a master plan to facilitate open access, and a strategy to ensure the sustainability of the access provision.

## 7.2 Impact on stakeholders

This deliverable provides suggestions of considerable interest to RIs, facilities, research performing organizations and institutions as it offers useful guidelines on the principles behind identification of key performance indicators and how they are used for monitoring of performance. Key principles and elements presented here can be easily borrowed by different stakeholders and adapted to specific situations and contexts.

<figure><img src="image_15.png" /></figure>
## 6.4 Returns

Metrics evaluating the benefits or returns on investment for RI access.

* Value on investment

* Similar to the strategic indicators in Section 5 of this document

* Economic return of investment

* Monetary income for the RI

We should keep in mind that the value on investment for a research infrastructure is not limited to the financial return, but instead often comes in less tangible forms such as visibility, scientific output or international collaboration.

# 7 CONCLUSIONS

This document outlines the principles for using key performance indicators, and how a research infrastructure can use them to monitor performance.

The most important principle detailed in this document is that the research infrastructure itself needs to identify its own set of key performance indicators. These should be relevant, accepted, credible, easy to monitor and robust against manipulation.

This document also includes an extensive list of suggested key performance indicators that could be used by a research infrastructure as inspiration when identifying ways to gauge its performance.

## 7.1 Impact on project

This document is one of the results from the activities in Work Package 10 on developing governance tools and recommendations for access to environmental research infrastructures. It is one in a set of four deliverables providing guidelines, a master plan to facilitate open access, and a strategy to ensure the sustainability of the access provision.

## 7.2 Impact on stakeholders

This deliverable provides suggestions of considerable interest to RIs, facilities, research performing organizations and institutions as it offers useful guidelines on the principles behind identification of key performance indicators and how they are used for monitoring of performance. Key principles and elements presented here can be easily borrowed by different stakeholders and adapted to specific situations and contexts.

<figure><img src="image_15.png" /></figure>
8 REFERENCES

[1] "Roadmap 2018 – Strategy Report on Research Infrastructures", European Strategy Forum on Research Infrastructures, ISBN: 978-88-943243-3-4, 2018. Accessed from http://roadmap2018.esfri.eu/ on 5 October 2018.

[2] "European Charter for Access to Research Infrastructures – Principles and Guidelines for Access and Related Services", European Commission, ISBN: 978-92-79-45600-8, doi:10.2777/524573, 2016. Accessed from https://publications.europa.eu/en/publication-detail/-/publication/78e87306-48bc-11e6-9c64-01aa75ed71a1/language-en/format-PDF/source-34701081 on 15 August 2017.

[3] "Berlin Declaration on Open Access to Knowledge in the Sciences and Humanities" 2003. Accessed from https://openaccess.mpg.de/Berlin-Declaration on 15 November 2018.

[4] "Better regulation Toolbox", produced by the European Commission complementing the "Better Regulation Guideline" SWD(2017) 350. Accessed from https://ec.europa.eu/info/better-regulation-toolbox_en on 14 November 2018.

[5] General Data Protection Regulation (EU) 2016/679 (https://eur-lex.europa.eu/eli/reg/2016/679/).

[6] "Strengthening the effectiveness and sustainability of international research infrastructures", OECD Science, Technology and Industry Policy Papers 48, 2017. Accessed from https://www.oecd-ilibrary.org/science-and-technology/oecd-science-technology-and-industry-policy-papers_23074957 on 8 November 2018.

[7] "ACTRIS PPP Deliverable 4.1: Concept document on ACTRIS Central Facilities structure and services", ACTRIS (Aerosols, Clouds and Trace Gases) Technical Documentation, 2018. Accessed from https://www.actris.eu/Documentation/ACTRISPPP(2017-2019)/Deliverables.aspx on 8 October 2018.

[8] "SIOS Key Performance Indicators Specification", SIOS (Svalbard Integrated Arctic Earth Observing System) Technical documentation, 2017. Accessed from https://www.sios-svalbard.org/Documents on 8 October 2018.

<figure><img src="image_16.png" /></figure>
8 REFERENCES

[1] "Roadmap 2018 – Strategy Report on Research Infrastructures", European Strategy Forum on Research Infrastructures, ISBN: 978-88-943243-3-4, 2018. Accessed from http://roadmap2018.esfri.eu/ on 5 October 2018.

[2] "European Charter for Access to Research Infrastructures – Principles and Guidelines for Access and Related Services", European Commission, ISBN: 978-92-79-45600-8, doi:10.2777/524573, 2016. Accessed from https://publications.europa.eu/en/publication-detail/-/publication/78e87306-48bc-11e6-9c64-01aa75ed71a1/language-en/format-PDF/source-34701081 on 15 August 2017.

[3] "Berlin Declaration on Open Access to Knowledge in the Sciences and Humanities" 2003. Accessed from https://openaccess.mpg.de/Berlin-Declaration on 15 November 2018.

[4] "Better regulation Toolbox", produced by the European Commission complementing the "Better Regulation Guideline" SWD(2017) 350. Accessed from https://ec.europa.eu/info/better-regulation-toolbox_en on 14 November 2018.

[5] General Data Protection Regulation (EU) 2016/679 (https://eur-lex.europa.eu/eli/reg/2016/679/).

[6] "Strengthening the effectiveness and sustainability of international research infrastructures", OECD Science, Technology and Industry Policy Papers 48, 2017. Accessed from https://www.oecd-ilibrary.org/science-and-technology/oecd-science-technology-and-industry-policy-papers_23074957 on 8 November 2018.

[7] "ACTRIS PPP Deliverable 4.1: Concept document on ACTRIS Central Facilities structure and services", ACTRIS (Aerosols, Clouds and Trace Gases) Technical Documentation, 2018. Accessed from https://www.actris.eu/Documentation/ACTRISPPP(2017-2019)/Deliverables.aspx on 8 October 2018.

[8] "SIOS Key Performance Indicators Specification", SIOS (Svalbard Integrated Arctic Earth Observing System) Technical documentation, 2017. Accessed from https://www.sios-svalbard.org/Documents on 8 October 2018.

<figure><img src="image_16.png" /></figure>
# ENVRIplus Post Access Questionnaire

1. Name and last Name

<figure><img src="image_17.png" /></figure>

2. Title and acronym of the project

<figure><img src="image_18.png" /></figure>

3. Name of the chosen platform

<figure><img src="image_19.png" /></figure>

4. How did you come to know about the ENVRIplus Transnational Access funding?

<figure><img src="image_20.png" /></figure>

ENVRIplus official website

<figure><img src="image_21.png" /></figure>

Information received by colleagues

<figure><img src="image_22.png" /></figure>

Through a mailing list I am subscribed to

<figure><img src="image_23.png" /></figure>

Other

5. Without the support of the ENVRIplus funding, would you still have been able to access the research platform?

<figure><img src="image_24.png" /></figure>

yes

<figure><img src="image_25.png" /></figure>

no

If your answer was no, briefly specify why (financial and/or interdisciplinary opportunity, knowledge about the platforms, etc.)

<figure><img src="image_26.png" /></figure>

6. Please assess the service provided by the ENVRIplus access team (0= not evaluable, 1= very poor, 2=sufficient, 3=good, 4=excellent):

Publicity and practical information on how to apply

<figure><img src="image_27.png" /></figure>

0

<figure><img src="image_28.png" /></figure>

1

<figure><img src="image_29.png" /></figure>

2

<figure><img src="image_30.png" /></figure>

3

<figure><img src="image_31.png" /></figure>

4

<figure><img src="image_32.png" /></figure>
ENVRIplus Post Access
Questionnaire

1. Name and last Name

<figure><img src="image_17.png" /></figure>

2. Title and acronym of the project

<figure><img src="image_18.png" /></figure>

3. Name of the chosen platform

<figure><img src="image_19.png" /></figure>

4. How did you come to know about the ENVRIplus Transnational Access funding?

<figure><img src="image_20.png" /></figure>

ENVRIplus official website

<figure><img src="image_21.png" /></figure>

Information received by colleagues

<figure><img src="image_22.png" /></figure>

Through a

mailing list I am subscribed to

<figure><img src="image_23.png" /></figure>

Other

5. Without the support of the ENVRIplus funding, would you still have been able to access the research platform?

<figure><img src="image_24.png" /></figure>

yes

<figure><img src="image_25.png" /></figure>

no

If your answer was no, briefly specify why (financial and/or interdisciplinary opportunity, knowledge about the platforms, etc.)

<figure><img src="image_26.png" /></figure>

6. Please assess the service provided by the ENVRIplus access team (0= not evaluable, 1= very poor, 2=sufficient, 3=good, 4=excellent):

Publicity and practical information on how to apply

<figure><img src="image_27.png" /></figure>

0

<figure><img src="image_28.png" /></figure>

1

<figure><img src="image_29.png" /></figure>

2

<figure><img src="image_30.png" /></figure>

3

<figure><img src="image_31.png" /></figure>

4

<figure><img src="image_32.png" /></figure>
Easiness of the procedure to apply

<figure><img src="image_33.png" /></figure>

0

<figure><img src="image_34.png" /></figure>

1

<figure><img src="image_35.png" /></figure>

2

<figure><img src="image_36.png" /></figure>

3

<figure><img src="image_37.png" /></figure>

4

Quantity of documentation required

<figure><img src="image_38.png" /></figure>

0

<figure><img src="image_39.png" /></figure>

1

<figure><img src="image_40.png" /></figure>

2

<figure><img src="image_41.png" /></figure>

3

<figure><img src="image_42.png" /></figure>

4

Scientific and technical support at the facility

<figure><img src="image_43.png" /></figure>

0

<figure><img src="image_44.png" /></figure>

1

<figure><img src="image_45.png" /></figure>

2

<figure><img src="image_46.png" /></figure>

3

<figure><img src="image_47.png" /></figure>

4

Other?

<figure><img src="image_48.png" /></figure>

7. The focus of the ENVRIplus Transnational Access is that of promoting inter-disciplinary research activities and it is also a fundamental criteria on which applications are evaluated. Do you believe that this point was sufficiently underlined in the information about the call for proposal and in the application procedure?

<figure><img src="image_49.png" /></figure>

yes

<figure><img src="image_50.png" /></figure>

no

8. Has the TNA project triggered/pushed the interdisciplinary nature of the research activity you are focusing on? If so, please briefly describe how:

<figure><img src="image_51.png" /></figure>

9. Please evaluate the overall service provided by the ENVRIplus Transnational Access (0= not evaluable, 1= very poor, 2=sufficient, 3=good, 4=excellent):

<figure><img src="image_52.png" /></figure>

0

<figure><img src="image_53.png" /></figure>

1

<figure><img src="image_54.png" /></figure>

2

<figure><img src="image_55.png" /></figure>

3

<figure><img src="image_56.png" /></figure>

4

10. If your evaluation is below the score of 3, please briefly explain why:

<figure><img src="image_57.png" /></figure>

<figure><img src="image_58.png" /></figure>
Easiness of the procedure to apply

<figure><img src="image_33.png" /></figure>

<figure><img src="image_34.png" /></figure>

<figure><img src="image_35.png" /></figure>

<figure><img src="image_36.png" /></figure>

<figure><img src="image_37.png" /></figure>

4

Quantity of documentation required

<figure><img src="image_38.png" /></figure>

<figure><img src="image_39.png" /></figure>

<figure><img src="image_40.png" /></figure>

<figure><img src="image_41.png" /></figure>

<figure><img src="image_42.png" /></figure>

4

Scientific and technical support at the facility

<figure><img src="image_43.png" /></figure>

<figure><img src="image_44.png" /></figure>

<figure><img src="image_45.png" /></figure>

<figure><img src="image_46.png" /></figure>

<figure><img src="image_47.png" /></figure>

4

Other?

<figure><img src="image_48.png" /></figure>

7. The focus of the ENVRIplus Transnational Access is that of promoting inter-disciplinary research activities and it is also a fundamental criteria on which applications are evaluated. Do you believe that this point was sufficiently underlined in the information about the call for proposal and in the application procedure?

<figure><img src="image_49.png" /></figure>

yes

<figure><img src="image_50.png" /></figure>

no

8. Has the TNA project triggered/pushed the interdisciplinary nature of the research activity you are focusing on? If so, please briefly describe how:

<figure><img src="image_51.png" /></figure>

9. Please evaluate the overall service provided by the ENVRIplus Transnational Access (0= not evaluable, 1= very poor, 2=sufficient, 3=good, 4=excellent):

<figure><img src="image_52.png" /></figure>

<figure><img src="image_53.png" /></figure>

<figure><img src="image_54.png" /></figure>

<figure><img src="image_55.png" /></figure>

<figure><img src="image_56.png" /></figure>

4

10. If your evaluation is below the score of 3, please briefly explain why:

<figure><img src="image_57.png" /></figure>

<figure><img src="image_58.png" /></figure>
11. Shortly comment the benefits of the TNA and/or lessons learnt:

<figure><img src="image_59.png" /></figure>

12. Do you have any suggestion for improvement?

<figure><img src="image_60.png" /></figure>
11. Shortly comment the benefits of the TNA and/or lessons learnt:

<figure><img src="image_59.png" /></figure>

12. Do you have any suggestion for improvement?

<figure><img src="image_60.png" /></figure>

<figure><img src="image_61.png" /></figure>