# The Role of Research Infrastructures in the Research Assessment Reform: A DARIAH Position Paper

Toma Tasovac, Laurent Romary, Erzsébet Tóth-Czifra, Rahel C. Ackermann, Daniel Alves, Sally Chambers, Mike Cosgrave, Martine Denoyelle, Vicky Garnett, Rita Gautschy, et al.

► To cite this version:

Toma Tasovac, Laurent Romary, Erzsébet Tóth-Czifra, Rahel C. Ackermann, Daniel Alves, et al. The Role of Research Infrastructures in the Research Assessment Reform: A DARIAH Position Paper. 2023. [https://hal.science/hal-04136772](https://hal.science/hal-04136772)

HAL Id: hal-04136772

https://hal.science/hal-04136772

Preprint submitted on 21 Jun 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# The Role of Research Infrastructures in the Research Assessment Reform: A DARIAH Position Paper

Authors: Toma Tasovac, Laurent Romary and Erzsébet Tóth-Czifra

With contributions from Rahel C. Ackermann, Daniel Alves, Sally Chambers, Mike Cosgrave, Martine Denoyelle, Vicky Garnett, Rita Gautschy, Edward Gray, Vojtěch Malínek, Carmen Di Meo, Andrew Perkins, Sanita Reinsone, Nanette Rißler-Pipka, Andrea Scharnhorst and Lorella Viola

## Introduction

Research assessment reform is crucial for the social sustainability of research infrastructures (RIs): RIs can only thrive in the long term if the researchers who contribute to their development and growth receive academic credit for the kind of work they do in and around research infrastructures. To put it bluntly, research infrastructures have a vested interest in supporting the reform of research assessment. But, conversely, ongoing attempts to reform research assessment can also benefit from the work of research infrastructures because RIs have a great deal of experience creating and maintaining public services for producing, curating and harvesting both traditional *and* non-traditional academic outputs. At the same time, RIs are themselves subject to assessment as the ongoing debate on Key Performance Indicators (KPI's) shows (see, for instance, Barbot et al. 2023). The goal of this paper is to outline DARIAH's position on the importance of research assessment reform for thematic RIs and the importance of thematic RIs for research assessment reform at the European level.

## Context: DARIAH and the arts & humanities

DARIAH was constituted as a European Research Infrastructure Consortium (ERIC) in 2014 and has been an ESFRI Landmark since 2016. DARIAH's mission is to empower research communities with digital methods to create, connect and share knowledge about culture and society with the ultimate vision of having Arts and Humanities firmly anchored at the centre of a technologically evolving knowledge
society. DARIAH is a membership-based organisation, currently with 21 member countries across Europe and 15 Cooperating Partners in 10 non-member countries. DARIAH is currently also chairing the [Social Sciences and Humanities Open Cluster (SSHOC)](https://sshoc.org/), a consortium of social sciences and humanities (SSH) research infrastructures contributing to the [European Open Science Cloud (EOSC)](https://eosc.org/).

DARIAH is a *thematic* research infrastructure. Unlike e-Infrastructures such as GÉANT, EGI, EuroHPC etc. that provide discipline-agnostic technical services, DARIAH develops, maintains and operates an infrastructure that is specifically designed to meet the needs of researchers in the arts and humanities. With an estimated 500,000 humanities researchers in Europe alone (Rossi, 2020), ours is a sizable community which works in different fields of knowledge (ranging from languages and literatures to history, philosophy and musicology to name just a few examples), handles different types of media (text, audio, video, music notations, images, 3D representations and various mixed-media formats) using both qualitative and quantitative methods as well as close and distant reading techniques – all this with the goal of arriving at a comprehensive understanding of human culture.

Arts and humanities researchers still face a number of challenges finding, accessing, producing and reusing digital resources: filtering out the noise, processing metadata of different quality and granularity, clarifying reuse rights for digital objects, choosing the right standards to model data, identifying the best tools to process a myriad of data formats, etc. This is why, in this day and age, when all arts and humanities researchers – and not only those who apply advanced digital methods in their areas of study – use digital tools to access, process and disseminate information, research infrastructures such as DARIAH should be seen as a common, public good, and not a niche product for the lucky few.

## Why should thematic RIs such as DARIAH care about research assessment reform?

Research infrastructures are public goods that are fundamental in supporting, sustaining and enhancing science. In the European context, research infrastructures are funded both at national and EU levels, but the day-to-day operations and the eventual success of RIs are largely dependent on the users' involvement. In structural terms, this occurs because of the so-called "in-kind" contributions that are built into the funding models of RIs, whereby member countries commit to supplementing their financial contributions with evidence of significant RI-relevant outputs for which no direct monetary exchange takes place between a member and the central node.

For research infrastructures, it is equally important to have users that make use of RI services in their own research, as it is to have users that actively contribute to the development and growth of the same RIs. While DARIAH's user base is quite diverse and includes not only humanities researchers but also a host of different professional profiles (librarians, archivists, museologists, curators, data stewards,
software engineers, data scientists, artists etc.) at different career stages and with different institutional affiliations, including those in the alt-ac (i.e. non-academic) sector, citizen scientists etc., the **majority of our users come from research-performing organisations: their ability to contribute to RIs is directly proportional to their ability to receive academic credit for the work they perform in and around RIs.** Yet our academic funding models and reward systems were not originally conceived to support or maintain digital infrastructures (Knowles et al. 2021). This is a fundamental, structural problem.

The challenges facing the evaluation of the work by SSH researchers contributing to the development of research infrastructures are twofold: 1) **widely acknowledged problems with research evaluation are compounded in various SSH disciplines because of their organisational and epistemic characteristics** and the type of outcomes that differentiate them from the science, technology, engineering and mathematics (STEM) disciplines (Reale et al., 2017); and 2) **the technical effort needed to ensure a smooth running of infrastructures often remains invisible** (Antonijević 2012; Kaltenbrunner 2015). The latter seems to be particularly glaring in the case of arts and humanities which have traditionally – and wrongly – been assumed to require very little investment in technology.

Hiring, tenure and promotion committees in research-performing organisations around the world tend to focus heavily on traditional scholarly outputs (journals and monographs) without properly recognising other types of research outputs (data, software, models, digital editions, algorithms, protocols, workflows, exhibitions, strategies, policy contributions etc.) Yet **all researchers, including those who are committed to writing code, creating artistic practices, curating, compiling and sharing datasets or training users deserve viable career paths:** researchers should not have to "choose between producing reproducible high-quality code and career progression." (Knowles et al. 2021)

The dominant portion of **scholarly contributions that are vital to thematic RIs** (infrastructural development in the broadest sense, curation of born-digital content, training measures etc.) are **still unrewarded, if not counter-incentivised, by the assessment mechanisms which are based on traditional bibliometric parameters and publisher prestige, on the one hand** (see, for instance, Eve 2020), and **individual rather than collaborative, pluri-disciplinary scholarly outputs, on the other**. This is why the ongoing attempts to reform research assessment, notably those centred around the [Coalition for Advancing Research Assessment (CoARA)](), are essential for the long-term success of research infrastructures. We need a cross-sectoral understanding of the complexities of the research processes and a common strategy for evaluating and valorising different types of research outputs.

This is also why **DARIAH-EU is urging the integration of process-oriented systems of evaluating science into the currently dominant product-oriented models.** The long-standing approach to communicating new discoveries based almost exclusively on the final output of scholarship – generally the publication – often misrepresents the value of those outputs by ruling out the sharing of other significant types of findings that may have been produced *during* the research process: "it is not just the product, however
<figure><img src="image_2.png" /></figure>

incomplete, that [should be] seen as worthy of preservation for current and future generations, but also
equally the process (or indeed processes) for producing it" (Viola 2023: 59). Today's technology enables
new approaches to sharing and reusing aspects of scholarly work, so that **research can be viewed,**
**managed, accessed, interacted with and ultimately assessed in terms of the integrity of processes,**
**rather than only in terms of the end product** (Tóth-Czifra et al. 2019).

Researchers in our domain are increasingly focusing on the creation, implementation and dissemination of formalised, reproducible, customizable and sustainable research workflows. Based on real-life research scenarios, workflows are living memories of best research practices in a given community, made accessible and reusable for other researchers wishing to carry out a similar project but unfamiliar with the recommended tools, formats, and methods to use, etc. And while DARIAH, for instance, provides mechanisms for sharing workflows (via the SSH Open Marketplace), important work remains to be done in valorising the arts and humanities workflows as an integral part of the research process.

Why are thematic RIs such as DARIAH important to research
assessment reform?

Policy makers, funding agencies, and research-performing institutions should **recognise the significant potential of thematic research infrastructures to serve as enablers of, and contributors to, a research assessment system** that truly values the multiplicity of ways in which scientific knowledge is created, accumulated, integrated, disseminated and preserved. Through their support for Open Science, multidisciplinary and multilingual research, standardisation practices and quality control, research infrastructures can foster a more robust, transparent, agile and inclusive evaluation ecosystem.

For the success of the reform of research assessment, **it is crucial, however, to avoid the mistakes that were made during the transition to Open Access in Europe.** A frequently voiced critique of that transition (see most recently Barnes 2023, Ross-Hellauer 2022, Stone et al. 2021) is that **the infrastructural dimension of Open Access did not receive as much attention and public investment as it should have.** As a result, many research funders were more likely to merely pay for the costs of Open Access publication and accept the conditions set by large commercial publishers instead of making collective infrastructural investments that would allow scholarly communities to gain and maintain control over the process. Open Access “done wrong” has threatened to create even larger inequalities than those that are already entrenched in the closed-access systems (Ross-Hellauer 2022).

Research assessment cannot exist without scholarly information management systems that allow scholars, institutions, and funders to keep track of the increasing amount of scholarly production. The shortcomings of the first generation of such services (Scopus, Web of Science etc.) as well as their
enormous impact on assessment practices have by now become self-evident (Hicks et al. 2015). They have also given rise to an ever-pressing need to replace or at least complement them by publicly-owned information management systems that are based on transparent processes and open data traffic, and that are inclusive of a broad range of content types, disciplines and languages. Mitigating the dependence of research ecosystems on commercial providers that focus more on profit than public good is an absolute priority when it comes to not only discovery services (Kraker 2021) but also evaluation practices. This is why developments around EOSC, the OpenAIRE Research Graph, and the monitoring services built on top of them – see, for instance OpenAIRE Monitor, OpenAIRE Usage Counts and EOSC Observatory – remain essential.

**The reform of research assessment and the development of public research infrastructures must go hand in hand:** research infrastructures need users and contributors who receive appropriate academic rewards for participating in Open Science practices, whereas scholars, research administrators and innovators need public infrastructures – and not commercial black-box systems – that facilitate the research process in its entirety, making all the steps of the research data lifecycle visible, reusable and available for evaluation.

## What should thematic RIs do to help the evolution of the global research assessment framework?

In addition to championing the ambitious agenda of CoARA and encouraging national nodes and partner institutions to contribute to CoARA working groups, DARIAH and other thematic RIs should engage in concrete actions as part of the debate on research assessment. In particular, we have identified two priorities where short-term progress could be made.

In order to make sure that proper recognition is made of the role of research infrastructures in the research process, DARIAH recommends that RIs consider developing **appropriate citation schemes** for two distinct kinds of RI-related acknowledgements:

* a citation scheme which allows **research infrastructures to be properly cited in all research products that have benefited from the RI's support**; and

* conversely, a citation scheme that acknowledges **contributions made to the construction, development or maintenance of research infrastructures** so that individuals or organisations can claim their participation during an assessment process.

As a technical precondition for establishing usable citation mechanisms that can relate research infrastructures and research assessment, DARIAH will continue to work on **developing and maintaining stable identifiers for its infrastructural assets and services**, including proper versioning mechanisms that
recognise the dynamic, changing nature of infrastructural outputs. This work should not only be limited
to local sources of authority (for instance, publication repositories such as [HAL](https://www.hal.ac.uk/) or collections of training
materials such as [DARIAH-Campus](https://www.dariah.eu/) but should also include global initiatives such as the [Research
Organization Registry](https://www.researchorganizationregistry.org/). Moreover, we believe we should be **working on consensus-based, stable
definitions of contribution roles in infrastructural activities**. We could work toward this goal by, first of
all, reviewing generic approaches to representing the roles typically played by participants in the
research process (for instance: [Contributor Roles Taxonomy](https://www.cltax.org/) or CRediT) and exploring how previous
domain-specific work in this area (see, for instance: [TaDiRAH](https://www.tadirah.org/): the Taxonomy of Digital Research Activities
in the Humanities and NeMO: the NeDiMAH Methods Ontology) complement the global picture.

Conclusion

**Thematic RIs come with the disciplinary critical mass:** they serve as unique points of convergence for different disciplines within a research domain, and a wide variety of expertise in scholarship, infrastructure building and digital content curation. **They have the potential to offer unique, sustained spaces for strengthening and solidifying evaluative cultures around important scholarship that is currently invisible in formal assessment** on the level of research domains.

With concentrated effort, RIs could be key in helping research-performing organisations, funders and other stakeholders go *beyond* the publication-based proxies such as impact factors and citation counts.

**Thematic RIs such as DARIAH are incubators for innovation:** we provide the means and support structures for exploring, building, testing and sustaining digital methods as enablers of scientific discovery. Activities in and near research infrastructures take different forms of co-creation, such as tool-making, database development, curating datasets and research software, providing training for novel research technologies etc., all of which should be considered in the ongoing attempts to reform research assessment. **Thematic research infrastructures, therefore, provide ideal testbeds for well-balanced, reflexive interventions that can overcome simplistic ideals and models for research evaluation.** While we argue for making invisible work visible and accountable, we are also very much aware that not all that is important can be counted and that not all that can be counted is necessarily important.
## References

Barnes, Lucy. 2023. "Copim and Community-Led Infrastructures for Open Access Books: Where Are We Now and What's Next." *Community-led Open Publication Infrastructures for Monographs (COPIM)* http://dx.doi.org/10.21428/785a6451.129b1935.

Eve, Martin Paul. 2020. "Violins in the Subway: Scarcity Correlations, Evaluative Cultures, and Disciplinary Authority in the Digital Humanities." In *Digital Technology and the Practices of Humanities Research*, edited by Jennifer Edmond, 105–22. Cambridge, UK: Open Book Publishers.

Hicks, Diana, Paul Wouters, Ludo Waltman, Sarah de de Rijcke, and Ismael Rafols. 2015. "Bibliometrics: The Leiden Manifesto for Research Metrics." *Nature* 520 (7548): 429–31.

Kaltenbrunner, W. 2015. "Reflexive Inertia: Reinventing Scholarship Through Digital Practices." diss., Leiden University.

Knowles, Rebecca, Bilal A. Mateen, and Yo Yehudi. 2021. "We Need to Talk About the Lack of Investment in Digital Research Infrastructure." *Nature Computational Science* 1 (3): 169–71.

Kraker, Peter. 2021. "Open Search Tools Need Sustainable Funding." https://science20.wordpress.com/2021/07/20/open-search-tools-need-sustainable-funding/.

Reale, Emanuela, Dragana Avramov, Kubra Canhial, Claire Donovan, Ramon Flecha, Poul Holm, Charles Larkin, Benedetto Lepori, Judith Mosoni-Fried, Esther Oliver, Emilia Primeri, Lidia Puigvert, Andrea Scharnhorst, Andràs Schubert, Marta Soler, Sàndor Soòs, Teresa Sordé, Charles Travis, and René Van Horik. 2018. "A Review of Literature on Evaluating the Scientific, Social and Political Impact of Social Sciences and Humanities Research." *Research Evaluation* 27 (4): 298–308.

Ross-Hellauer, Tony. 2022. "Open Science, Done Wrong, Will Compound Inequities." *Nature* 603 (7901): 363.

Rossi, Giorgio et al. 2020. "Supporting the Transformative Impact of Research Infrastructures on European Research – Report of the High-Level Expert Group to Assess the Progress of Esfri and Other World Class Research Infrastructures Towards Implementation and Long-Term Sustainability." https://data.europa.eu/doi/10.2777/3423.
Stone, Graham, Marta Błaszczyńska, Chloé Lebon, Agata Morka, Tom Mosterd, Pierre Mounier, Vanessa Proudman, Lara Speicher, and Iva Melinščak Zlodi. 2021. "Collaborative Models for Oa Book Publishers." https://doi.org/10.5281/zenodo.5494731.

Tóth-Czifra, Erzsébet, Jennifer Edmond, and Laurent Romary. 2019. "Stakeholder Consultation on the Future of Scholarly Publishing and Scholarly Communication - Response From Dariah-Eu." https://doi.org/10.5281/zenodo.3364445.

Viola, Lorella. 2023. *The Humanities in the Digital: Beyond Critical Digital Humanities*. Cham: Palgrave Macmillan.