# ATRIUM Peer Review Framework (Version 1)

Françoise Gouzi, Toma Tasovac, Anne Baillot, Sarah Bénière, Carol Delmazo,
Victoria Garnett

<figure><img src="image_1.png" /></figure>

## To cite this version:

Françoise Gouzi, Toma Tasovac, Anne Baillot, Sarah Bénière, Carol Delmazo, et al.. ATRIUM Peer Review Framework (Version 1). DARIAH ERIC. 2025. halshs-05458734

HAL Id: halshs-05458734

https://shs.hal.science/halshs-05458734v1

Submitted on 14 Jan 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# ATRIUM – Advancing Frontier Research
## in the Arts and Humanities

Work Package WP7
Fostering Cross-Disciplinary Research through
Training and Open Science

Milestone 9
MS9: ATRIUM Peer Review Framework_V1

**Funding Instrument:** Horizon Europe

**Call:** HORIZON-INFRA-2023-SERV-01

**Call Topic:** Research infrastructure services to support health research, accelerate the green and digital transformation, and advance frontier knowledge (2023)

**Project Start:** 1 Jan 2024

**Project Duration:** 48 months

**Beneficiary in Charge:** DARIAH

**Document Identifier:** 10.5281/zenodo.17787916

<figure><img src="image_3.png" /></figure>

Funded by
the European Union

Funded by the European Union. Grant Agreement number 101132163. Views and opinions expressed are
however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Executive
Agency. Neither the European Union nor the granting authority can be held responsible for them.
## Milestone Information

<table><tr><td>Action Number:</td><td>101132163</td></tr><tr><td>Action Acronym:</td><td>ATRIUM</td></tr><tr><td>Action title:</td><td>Advancing FronTier Research In the Arts and hUManities</td></tr><tr><td>Beneficiary in Charge:</td><td>DARIAH</td></tr><tr><td>Contractual Date:</td><td>31/12/2025</td></tr><tr><td>Submission Date:</td><td>30/12/2025</td></tr><tr><td>Dissemination Level:</td><td>PU</td></tr><tr><td>Nature:</td><td>Other (Publication - Milestone)</td></tr><tr><td>Lead Author(s):</td><td>Françoise Gouzi (DARIAH), Toma Tasovac (BCDH)</td></tr><tr><td>Co-author(s):</td><td>Anne Baillot (DARIAH), Sarah Bénière (INRIA), Carol Delmazo (OPERAS), Vicky Garnett (DARIAH)</td></tr><tr><td>Reviewers:</td><td>Lulianna van der Lek (CLARIN), Megan Black (DARIAH), Tiziana Lombardo (NET7)</td></tr><tr><td>Keywords:</td><td>open peer review, research assessment, non-traditional research output, evaluation criteria, data paper, workflow paper, training material</td></tr><tr><td>Status:</td><td>Version 1</td></tr></table>

## Change Log

<table><thead><tr><td>Date</td><td>Version</td><td>Author/Editor</td><td>Summary of Changes made</td></tr></thead><tbody><tr><td>20/08/25 - 08/12/25</td><td>v0.1</td><td>Françoise Gouzi, Carol Delmazo, Anne Baillot, Sarah Bénière, Toma Tasovac</td><td>Draft Report</td></tr><tr><td>18/12/25</td><td>v0.2</td><td>Megan Black, Tiziana Lombardo</td><td>WP7 reviewers</td></tr><tr><td>29/12/25</td><td>v0.3</td><td>Toma Tasovac</td><td>Final revision</td></tr><tr><td>30/12/25</td><td>v1.0</td><td>Megan Black</td><td>Submission</td></tr></tbody></table>
Table of Contents

<table><tr><td>Milestone Information</td><td>2</td></tr><tr><td>Change Log</td><td>2</td></tr><tr><td>Table of Contents</td><td>3</td></tr><tr><td>List of Abbreviations</td><td>4</td></tr><tr><td>Executive Summary</td><td>5</td></tr><tr><td>1. Introduction</td><td>6</td></tr><tr><td>1.1 Purpose and Scope of the Document</td><td>6</td></tr><tr><td>1.2 Structure of the Document</td><td>6</td></tr><tr><td>2. Context</td><td>7</td></tr><tr><td>2.1 Open Science</td><td>7</td></tr><tr><td>2.2 International Research Assessment landscape</td><td>8</td></tr><tr><td>2.3 Transformations: A DARIAH journal</td><td>9</td></tr><tr><td>3. Design principles</td><td>10</td></tr><tr><td>3.1 Assessment Categories</td><td>12</td></tr><tr><td>3.1.2 State of the Art</td><td>12</td></tr><tr><td>3.1.3 Content</td><td>12</td></tr><tr><td>3.1.4 Form</td><td>12</td></tr><tr><td>3.1.5 Accessibility and Reusability</td><td>13</td></tr><tr><td>3.1.6 Overall assessment</td><td>13</td></tr><tr><td>3.2 Scoring</td><td>14</td></tr><tr><td>3.2.1 Numeric Scale</td><td>14</td></tr><tr><td>3.2.2 Editorial use of numeric scores</td><td>15</td></tr><tr><td>3.2.3 Recommendation categories</td><td>16</td></tr><tr><td>4. Evaluation Criteria</td><td>17</td></tr><tr><td>4.1 Data Papers</td><td>17</td></tr><tr><td>4.1.1 Definition</td><td>17</td></tr><tr><td>4.1.2 Context and Background</td><td>17</td></tr><tr><td>4.1.3 Criteria</td><td>18</td></tr><tr><td>State of the Art</td><td>18</td></tr><tr><td>Content</td><td>18</td></tr><tr><td>Form</td><td>18</td></tr><tr><td>Accessibility and Reusability</td><td>18</td></tr><tr><td>4.1.4 Reviewers' Required Qualifications</td><td>19</td></tr><tr><td>4.1.5 Examples of Data Papers</td><td>19</td></tr><tr><td>4.2 Workflow Papers</td><td>19</td></tr><tr><td>4.2.1 Definition</td><td>19</td></tr></table>
<table><tr><td>4.2.2 Context and Background</td><td>19</td></tr><tr><td>4.2.3 Criteria</td><td>21</td></tr><tr><td>State of the Art</td><td>21</td></tr><tr><td>Content</td><td>21</td></tr><tr><td>Form</td><td>21</td></tr><tr><td>Accessibility and Reusability</td><td>21</td></tr><tr><td>4.2.4 Reviewers' Required Qualifications</td><td>21</td></tr><tr><td>4.2.5 Examples of Workflow Papers</td><td>22</td></tr><tr><td>4.3 Training Materials</td><td>22</td></tr><tr><td>4.3.1 Definition</td><td>22</td></tr><tr><td>4.3.2 Context and Background</td><td>22</td></tr><tr><td>4.3.3 Criteria</td><td>23</td></tr><tr><td>State of the Art</td><td>23</td></tr><tr><td>Content</td><td>23</td></tr><tr><td>Form</td><td>24</td></tr><tr><td>Accessibility and Reusability</td><td>24</td></tr><tr><td>4.3.4 Reviewers' required qualifications</td><td>24</td></tr><tr><td>5. Next steps</td><td>25</td></tr><tr><td>References</td><td>26</td></tr><tr><td>Consortium</td><td>29</td></tr><tr><td>Disclaimer</td><td>30</td></tr></table>

List of Abbreviations

<table><tr><td>ATRIUM</td><td>Advancing FronTier Research In the Arts and hUManities</td></tr><tr><td>APRF</td><td>ATRIUM Peer Review Framework</td></tr><tr><td>ARIADNE</td><td>ARIADNE Research Infrastructure AISBL</td></tr><tr><td>CLARIN</td><td>Common Language Resources and Technology Infrastructure</td></tr><tr><td>CoARA</td><td>Coalition for Advancing Research Assessment</td></tr><tr><td>DARIAH</td><td>Digital Research Infrastructure for Arts and Humanities</td></tr><tr><td>DORA</td><td>San Francisco Declaration on Research Assessment</td></tr><tr><td>FAIR</td><td>Findable, Accessible, Interoperable, Reusable</td></tr><tr><td>OPERAS</td><td>Open Scholarly Communication in the European Research Area for Social Sciences and Humanities</td></tr><tr><td>OS</td><td>Open Science</td></tr><tr><td>RRA</td><td>Responsible Research Assessment</td></tr></table>
Executive Summary

The ATRIUM project (Advancing FronTier Research In the Arts and hUManities) aims to
strengthen and integrate research infrastructure services across the arts and
humanities. A central objective of the project is to support high-quality,
cross-disciplinary research by improving how diverse research outputs are evaluated,
recognised, and rewarded.

This milestone delivers the first version of the **ATRIUM Peer Review Framework**, a qualitative, criteria-based assessment model that broadens the understanding of what counts as valuable research. It shifts evaluation away from an almost exclusive focus on journal articles and citations by bringing into sharp focus non-traditional contributions such as datasets, models, workflows, and software. By building community consensus around a peer review framework, ATRIUM aims to provide a shared foundation for recognising the full spectrum of *"behind-the-scenes"* infrastructural work that underpins high-quality arts and humanities research in Europe.

Grounded in Open Science principles and aligned with international reform efforts, including, most notably, the Coalition for Advancing Research Assessment (CoARA), the framework prioritises contextualised qualitative judgement over metric-driven evaluation. It supports reviewers and editors through a structured combination of narrative assessment and transparent scoring, while explicitly embedding accessibility, reusability, and sustainability as core evaluation dimensions.

The first version of the framework defines three sets of evaluation criteria covering:
**data papers** (section 4.1),
** workflow papers** (section 4.2) and
**training materials** (section
4.3). All criteria are organised under four stable assessment categories (State of the
Art, Content, Form and Accessibility and Reusability), which ensures coherence across
output types while allowing output-specific interpretation. This design supports
consistency, comparability, and reuse across journals, publishers, research
institutions, and funding bodies.

The framework is already being piloted in practice through *Transformations: A DARIAH Journal*, a Diamond Open Access overlay journal launched in 2024 and hosted on the *Episciences* platform. This practical implementation demonstrates the feasibility of the framework within real editorial workflows and open peer review environments. Looking ahead, the ATRIUM Peer Review Framework will be expanded with additional evaluation criteria for 3D scholarly assets, textual scholarly editions, and software and code, which will be delivered in a future update of this milestone.
# 1. Introduction

## 1.1 Purpose and Scope of the Document

In this version 1, the ATRIUM Peer Review Framework (APRF) defines a set of qualitative evaluation criteria and practical guidelines to support consistent and transparent peer review of non-traditional research outputs such as data papers, workflow papers, and training materials. It responds to a large and growing literature which shows both the limits of metric-centric assessment and the benefits of qualitative, criteria-led peer evaluation (see, for instance, Langfeldt, Reymert, and Aksnes 2021; Williams 2020; Smit and Hessels 2021; Lewandowska, Kulczycki, and Ochsner 2023).

This general challenge is especially acute in the context of research infrastructures. The position paper by the Digital Research Infrastructure for Arts and Humanities (DARIAH) on 'The Role of Research Infrastructures in the Research Assessment Reform' (Tasovac et al. 2023) has already argued that essential forms of infrastructural work (such as the development of digital tools, the curation of born-digital data, and the design and delivery of training) remain systematically undervalued by conventional evaluation systems that focus on journal publications and citation counts. Such traditional metrics tend to reward individual, monographic outputs and high-impact journal placements, while overlooking collaborative and service-oriented types of infrastructural work.

The evaluation criteria in the APRF are designed to be reusable, primarily intended for adoption by journals and publishers. Thanks to the modular structure of both the underlying design principles and the criteria themselves, the framework can be readily adapted to other academic and disciplinary contexts. Beyond publishing, the framework is also relevant for research institutions and funding bodies, which need robust assessment practices in recruitment, promotion, project evaluation, and funding decisions. Ultimately, this framework contributes to a shift in evaluation culture and supports the research assessment reform, in particular through alignment with the commitments of the CoARA.

## 1.2 Structure of the Document

This document is organised as follows: [Section 2](#) summarises the context of why the ATRIUM assessment framework is necessary, including sub-sections on Open Science, the international research assessment landscape, and *Transformations: A DARIAH Journal*. The sub-section on *Transformations* provides a concrete use-case which is already implementing the ATRIUM Peer Review Framework. [Section 3](#) explains the design principles underlying the framework, such as the reasoning behind the
assessment categories and scoring. **Section 4** outlines the evaluation criteria for data papers, workflow papers, and training materials, each providing details on the definition, state of the art, content, form, accessibility and reusability. The **final section** offers the next steps for later versions of the peer review framework, followed by a **bibliography**.

## 2. Context

### 2.1 Open Science

Open Science (OS) refers to a set of principles and practices aimed at making research processes and outputs more accessible, transparent, and reusable, for the benefit of both researchers and society at large. As articulated by [UNESCO](https://www.unesco.org), Open Science encompasses not only open access to publications and data, but also openness in research methods, evaluation, and communication.

Data sharing is a central component of Open Science and a prerequisite for scientific collaboration, reproducibility, and innovation. In the arts and humanities, it supports the availability and reuse of multilingual, multimodal research materials and facilitates broader participation in knowledge creation and evaluation beyond traditional academic communities. Open Science does not prescribe a single publishing model. Instead, it is compatible with a range of Open Access approaches, which differ primarily in their economic and organisational arrangements. Gold Open Access refers to models in which final publications are made immediately accessible, with publication costs covered through a variety of funding mechanisms. Diamond Open Access is a specific form of Gold Open Access in which neither readers nor authors are charged publication fees. While these models determine *who pays* and *when access is provided*, they do not in themselves guarantee research quality, transparency, or ethical standards.

Ethical and transparent scholarly communication depends primarily on processes, not on Open Access labels. These include clearly defined editorial governance, robust peer review practices, transparency in evaluation criteria, and openness regarding research materials, methods, and licensing. Open peer review represents one approach to increase transparency in evaluation by making reviewers' identities, and even reviews themselves, publicly accessible alongside publications.

The open peer review also strengthens the visibility, recognition and reputation of reviewers and their contributions. Furthermore, more transparent evaluation fosters greater accountability and effective error detection, building trust and confidence in
the research community. It facilitates the exchange of knowledge and ideas among researchers, encouraging constructive criticism and helping identify flaws or areas for improvement.

DARIAH has long fostered best practices in data management and reuse for arts and humanities researchers, as part of its commitment to Open Science. See, for instance, the [DARIAH Impact Case Study](Tóth-Czifra 2021). DARIAH is also a champion of domain-specific approaches to implementing Open Science, aiming at helping understand and apply generic OS frameworks such as the FAIR principles (Findable, Accessible, Interoperable, Reusable) to the concrete situations and practices of the arts and humanities research.

## 2.2 International Research Assessment landscape

Responsible Research Assessment (RRA) is an umbrella term, defined as the approaches to assessment which incentivise, reflect, and reward the plural characteristics of high-quality research, in support of diverse and inclusive research cultures (Curry, de Ricke, et al. 2020).

Over the past decade, several initiatives have played a critical role in shaping the discussion about RRA across the international research community. While not exhaustive, the initiatives outlined below constitute key reference points in the global policy and practice landscape:

* DORA, the [San Francisco Declaration on Research Assessment](https://www.sanfrancisco declaration.org/) advocates for a Responsible Research Assessment framework that provides research assessment policies and tools for institutions (at the international level). It has become a worldwide initiative covering all scholarly disciplines and key stakeholders, including funders, publishers, professional societies, institutions, and researchers.

* The [Leiden Manifesto](https://www.leidenmanifesto.org/) sets out ten principles for the use of quantitative indicators in research evaluation and shows the need to offer clearer guidance to end-users of bibliometrics in research evaluation.

* The [Barcelona Declaration](https://www.barcelona-declaration.org/) on Open Research Information emphasises the need to make openness the default, to utilise supportive services and systems, to ensure the sustainability of open research infrastructures, and to foster collaborative efforts to transition from closed to open information sharing.

* The [CoARA Agreement on Reforming Research Assessment](https://www.coara.eu/agreement-on-reforming-research-assessment) (as defined in the Introduction) commits signatories to recognise the diversity of contributions and careers; to base assessment primarily on qualitative peer review; and to avoid the inappropriate use of rankings and journal-level metrics. CoARA's commitment to recognising diverse research outputs and valuing a broad range
of impacts aligns with the goals of promoting innovative scholarship within the arts and humanities.

**CoARA's sixth commitment**, explicitly focusing on reviewing and developing research assessment criteria, tools, and processes, offers a valuable entry point for enhancing the visibility, recognition, and evaluation of innovative outputs. By advocating for the inclusion of multi-, inter-, and trans-disciplinary research and diverse contributions to knowledge generation and societal impact, CoARA promotes a more inclusive and comprehensive approach to research assessment that can better accommodate the unique contributions of innovative scholarship (Maryl, Wnuk et al. 2024).

Within this evolving international landscape, the ATRIUM Peer Review Framework (APRF) positions itself as a concrete, operational contribution to research assessment reform. By translating high-level principles into structured, qualitative, and output-sensitive evaluation criteria, it supports the practical implementation of responsible research assessment in the arts and humanities.

## 2.3 Transformations: A DARIAH journal

*Transformations: A DARIAH Journal* was launched in 2024 following a conception phase that examined the added value of establishing a new journal for researchers, librarians, and engineers engaged in digital-based research in the arts and humanities. Central to this reflection was the question of how to valorise less-recognised research outputs while adhering to the Diamond publishing model.

The journal was conceived as a space in which a broad range of digital scholarly contributions can be made visible and assessed on their own terms. Rather than reproducing existing publication models, *Transformations* aims to provide a technical and intellectual environment suited to outputs that play a critical role in digital research but are often marginalised in conventional journals.

To support this aim, *Transformations* operates as an overlay journal hosted on the *Episciences* platform. In this model, content is deposited in trusted open repositories, while the journal focuses on editorial curation, peer review, and scholarly contextualisation. This separation of dissemination, evaluation, and preservation provides flexibility in accommodating heterogeneous research outputs and facilitates closer collaboration between authors, reviewers, and editors.

The first issue of *Transformations* was published using a single, broadly defined submission format. Building on this initial experience and on the work conducted within ATRIUM Task 7.4 (The Peer Review Framework), the journal has since formalised its
editorial workflow and expanded its submission types. These now include data papers, workflow papers, and research articles, each supported by dedicated review criteria and guidance for authors and reviewers.

As an operational use case within ATRIUM, *Transformations* provides a testing ground for the Peer Review Framework and a site of iterative refinement. The journal's ongoing development is closely connected to dissemination, training, and policy-oriented activities within the project, with the broader aim of supporting the adoption of new research formats and evaluation practices by authors, reviewers, institutions, and assessment bodies.

Insights gained from the journal's editorial and review practices feed directly into ATRIUM's wider dissemination, training, and policy work. This ensures that the APRF is not only tested in real publishing contexts, but also actively supported through capacity-building and community engagement. To that end, the following next steps are envisioned:

* Expanding future calls for contributions to include additional types of submissions, accompanied by dedicated review criteria and clear guidance for authors and reviewers;

* Providing targeted training within ATRIUM on how to produce non-traditional research outputs such as workflow papers (in particular within WP4);

* Facilitating a fruitful dialogue with the authorship and readership of *Transformations*, drawing on Scientific Committee meetings, reviewer exchanges, and presentations at events such as the DARIAH Annual Event;

* Supporting institutional efforts to improve recognition of the added value of these novel types of research outputs, notably through the ATRIUM Policy Brief (Tasovac, Baillot & Black 2025), the implementation of lesson learned from the ATRIUM Horizon Booster, and engagement with open science communities via conferences and policy panels.

## 3. Design principles

The ATRIUM Peer Review Framework (APRF) has been designed to (a) recognise the diversity of research contributions in the arts and humanities, (b) prioritise qualitative peer review over journal- or metric-based proxies, and (c) operationalise Open Science and research integrity in review practice. These principles reflect international guidance (DORA, the Leiden Manifesto; the Barcelona Declaration, CoARA) as well as empirical evidence on responsible assessment. They are being implemented within
*Transformations: A DARIAH Journal* through the progressive adoption of open-identities and open-reports peer review models.

The APRF is based on the following design principles:

1. **Prioritise qualitative judgement, supported by transparent scoring.** Narrative assessment is primary. Numeric scoring structures judgement and supports initial comparability but is no substitute for reasoned critique in responsible editorial workflows. This responds to evidence that oversimplified, metric-centred evaluation pushes researchers toward behaviours that maximise those indicators, often at the expense of research quality, diversity of outputs, and long-term sustainability.

2. **Stabilise assessment categories across output types.** All sets of criteria are organised under the same four categories: "State of the Art", "Content", "Form", and "Accessibility and Reusability". This creates a recognisable structure for authors, reviewers and publishers, and ensures that different kinds of outputs are assessed against comparable dimensions, even when their concrete manifestations differ.

3. **Adapt interpretation to output specificity.** While the four categories are shared, their interpretation and emphasis are adapted to each output type. This allows us to respect the specificity of each format without fragmenting the framework into unrelated checklists.

4. **Embed Open Science and reusability by design.** By elevating "Accessibility and Reusability" to a dedicated category, the framework integrates Open Science and FAIR concerns (availability of resources, licensing, documentation, sustainability) directly into peer review, rather than treating them as optional add-ons. This reflects ATRIUM's role in supporting research infrastructures and services.

5. **Enable iterative improvement without structural drift.** The four-category structure is intended to be stable across framework versions, while the concrete prompts and examples within each category can be refined over time in response to pilot reviews, community feedback, and the addition of new output types (3D, digital scholarly editions, software and code). This approach maintains coherence while allowing the framework to grow.
## 3.1 Assessment Categories

### 3.1.2 State of the Art

This category anchors each output in an existing landscape of scholarly work. In developing criteria, we ask for every output type if the submission identifies relevant prior work; and if it explains how it extends, consolidates, or departs from that work.

For data papers, for instance, this means situating the dataset among existing corpora or repositories, and clarifying the extent to which it provides new coverage, granularity or curation. For workflow papers, this means mapping the described workflow onto established methodological or technical approaches. For training materials, this means positioning the resource within the existing training landscape and articulating its specific contribution relative to comparable materials.

Designing state-of-the-art criteria in this way ensures that all outputs, regardless of format, are evaluated for their relationship to existing knowledge and practice, not just for their intrinsic qualities. It also creates a point of comparison across outputs: a strong training resource or workflow paper should be as well contextualised as a good data paper.

### 3.1.3 Content

This category captures the substantive intellectual or pedagogical content of the work. It focuses on what is being argued, presented, documented, or taught: the soundness of aims and research questions, the appropriateness and transparency of methods, the quality of argumentation, and the depth of reflection and discussion.

For data papers, content criteria focus on how clearly collection, processing and quality-control procedures are described, and how convincingly the dataset's potential uses are articulated. For workflow papers, content criteria foreground the explanation of the workflow logic, assumptions, and illustrative use-cases. For training materials, they relate to the accuracy, completeness and level-appropriateness of the learning content.

### 3.1.4 Form

This category addresses how the content is communicated and made usable for its intended audience. It focuses on presentation rather than substance: clarity and readability of language or media, coherence and signalling through logical structure
and progression (e.g. abstracts, sections, navigation) and overall communicative
effectiveness.

For data and workflow papers, form criteria concern scholarly conventions (abstract,
structure, use of figures/tables, consistency of terminology) as well as clarity of
language. For training materials, they attend to modularity, logical progression, explicit
indication of target audience and prerequisites, and usability in different teaching
contexts.

When harmonising criteria, the separation of *what* is being said (content) from *how* it is presented (form) makes it possible, for example, to judge a paper with important content but weak structure as a candidate for major revision rather than immediate rejection, and to articulate that distinction clearly in both the narrative and scoring.

3.1.5 Accessibility and Reusability

This category focuses on the extent to which outputs can be found, accessed and
sustained over time. It covers, among other aspects, deposition in trusted repositories,
the use of persistent identifiers, licensing that permits reuse, adherence to community
standards, the provision of rich metadata, and documentation that enables others to
apply, reproduce or adapt the work.

During the development of the framework, this category has emerged as a key
harmonising axis across output types. It requires every output to demonstrate
concretely how others will be able to build on it. It is also where the infrastructure
dimension of ATRIUM becomes most visible: the criteria explicitly encourage the use of
RI services, repositories and standards, thereby positioning the assessment framework
itself as a lever for closer integration between scholarly practice and infrastructure
provision.

3.1.6 Overall assessment

In addition to criterion-by-criterion comments, reviewers are expected to provide an
overall assessment. This section should synthesise the main strengths and weaknesses
of the submission, reflect on its overall contribution and readiness for publication, and
highlight any cross-cutting issues that are not easily captured under individual criteria.

The overall assessment plays a key role in editorial decision-making, particularly where
numeric scores diverge or where individual weaknesses must be weighed against the
submission's broader contribution. Reviewers should use this section to exercise
judgement, not to repeat checklist items.
## 3.2 Scoring

Scoring complements qualitative assessment by creating a shared frame of reference, helping editors compare reviews and supporting transparent decisions in an open peer review setting.

### 3.2.1 Numeric Scale

All criteria detailed in Section 4 are scored on a 0-5 whole-number scale, using the anchors below.

<table><thead><tr><td>Score</td><td>Anchor</td><td>Guidance for reviewers</td></tr></thead><tbody><tr><td>5 (Outstanding)</td><td>Exceeds expectations for the field and format; exemplary openness/reusability; limitations clearly analysed; likely to shape practices.</td><td>Cite specific passages or links (e.g., repository record, use-case) that demonstrate exemplary quality.</td></tr><tr><td>4 (Strong)</td><td>Meets expectations very well; minor weaknesses that do not materially affect contribution or reuse.</td><td>Identify the minor issue(s) and why they do not compromise the core value.</td></tr><tr><td>3 (Satisfactory)</td><td>Adequate, meets baseline standards; some moderate weaknesses that would limit impact or reuse without major revision.</td><td>Describe concrete, fixable issues (e.g. missing references, unclear argument, ambiguous wording, insufficient explanation of data availability and reuse).</td></tr><tr><td>2 (Weak)</td><td>Major weaknesses (methodological opacity; unclear contribution; limited availability) that would require substantial revision.</td><td>Specify what is missing and how to address it; avoid generic statements. Indicate whether the identified weaknesses can be realistically remedied within a normal revision cycle.</td></tr><tr><td>1 (Poor)</td><td>Severe deficiencies; contribution not credible or non-compliant with core expectations.</td><td>Explain decisive problems succinctly.</td></tr></tbody></table>
<table><tr><td>0 (Not assessable / not provided)</td><td>Criterion cannot be assessed (e.g., dataset not deposited; licence absent) or the submission breaches ethical/legal requirements.</td><td>Use when information is missing or non-compliant; editors may desk-reject or require resubmission.</td></tr></table>

The table above defines the meaning of each score and the type of evidence expected at each level. To ensure consistent and transparent application of the scale across reviewers and output types, reviewers should also:

* Use whole numbers only (no decimals). This reduces false precision and aligns with established grant-review practices.

* Use the entire scale, i.e. avoid bunching scores at "3".

* Provide 3-5 sentences of justification per criterion, pointing to evidence (section, figure, repository DOI). In open review, these justifications are published alongside the paper.

### 3.2.2 Editorial use of numeric scores

Editors should consider numeric scores as one input among several when making a publication recommendation. Numeric scores help to surface patterns, highlight areas of convergence or divergence among reviewers, and support internal consistency across submissions. They are not, however, determinative. They serve primarily as a first-step comparative aid, never as a mechanical decision rule.

Editors should base their decisions on three elements taken together:

1. The qualitative assessments, which remain the primary basis for judgment.

2. The pattern of scores across criteria, used to identify areas of convergence or divergence between reviewers' narratives.

3. The editors' own reading, contextual expertise, and responsibility for upholding the journal's standards.

As qualitative assessment is crucial, editorial judgement can override arithmetic averages. Well-argued narrative reasoning, especially when it points to issues of methodology, transparency, openness, or integrity, always takes precedence over high scores without corresponding argumentation. Similarly, very low scores in one criterion may not imply rejection if the accompanying justification shows that the issue is easily remediable.
The ATRIUM Peer Review Framework deliberately does not prescribe fixed weights for criteria or categories. While the framework provides a stable and shared structure, decisions about relative emphasis (for example, between content, reusability, or state-of-the-art positioning) are left to journals and editorial boards. This allows individual journals to articulate and enact their own editorial profiles, disciplinary priorities, and missions, while remaining interoperable with a common assessment logic. In this sense, the framework balances harmonisation with editorial autonomy, and enables journals to make their values explicit rather than embedding them implicitly in opaque scoring rules.

### 3.2.3 Recommendation categories

Editors synthesise reviewers' input and assign one of the following categories:

* **Accept**: the submission is sound, clearly presented, and requires only editorial polishing.

* **Minor revision**: the submission is broadly strong but could benefit from clearly circumscribed improvements.

* **Major revision**: substantial reworking is needed; the contribution has potential, but core issues must be addressed.

* **Reject**: the submission has fundamental conceptual, methodological, or Open Science-related deficiencies (e.g. the datasets are not accessible in a repository) that cannot be realistically resolved within a normal revision cycle.

Editors may also issue **desk rejections** prior to peer review. These should be used sparingly and only when:

* The submission falls clearly outside the scope of the journal, the thematic issue or the formats solicited by the given journal.

* Ethical, legal or licensing issues make review impossible.

* Core required elements are missing (e.g. no dataset deposit for a data paper, no workflow description for a workflow paper, no learning objectives for training materials etc.)

* The submission does not meet the minimal technical or technical threshold expected by the journal.

Desk rejections are logged with a short explanation to ensure transparency and promote learning for authors.
# 4. Evaluation Criteria

This section presents the evaluation criteria for three types of non-traditional research outputs: data papers, workflow papers, and training materials.

For each output type, criteria are organised under the same four assessment categories (State of the Art, Content, Form, and Accessibility and Reusability) in line with the design principles outlined in Section 3. While the categories remain stable, their interpretation is adapted to the specific nature and purpose of each output type.

Reviewers are encouraged to use the criteria as structured prompts for qualitative judgement, not as a checklist. Not all criteria will carry equal weight in every case; reviewers should explain their assessments narratively, drawing attention to strengths, weaknesses, and trade-offs relevant to the submission.

## 4.1 Data Papers

### 4.1.1 Definition

A data paper is a research output that describes a specific dataset, or a coherent group of datasets, including the circumstances of their creation and their potential scholarly reuse.

### 4.1.2 Context and Background

The primary purpose of a data paper is to make the described dataset more FAIR. A data paper provides a detailed account of the dataset's production (e.g. provenance, authorship, rights, methods) and gives access to the data, typically via a persistent link to a trusted repository. It differs from a research article in that it does not report analytical results or research findings derived from the data.

In this context, data is understood as "information in a digital form, other than scientific publications, which are collected or produced in the course of scientific research activities and are used as evidence in the research process, or are commonly accepted in the research community as necessary to validate research findings and results" (EU Directive 2019/1024/EU in Gouzi et al. 2024).
Reviewers are expected to take into account the overall quality of the data paper and its
contribution to making the dataset FAIR and provide constructive feedback to the
authors to improve the data paper.

4.1.3 Criteria

State of the Art

* Does the submission provide information about existing datasets or publications that are related to the dataset being described (where applicable)?

* Does the submission explain how this dataset differs from the existing ones?

* Are the bibliographic references up to date and relevant?

Content

* Does the submission provide the context in which the dataset was produced? What were the motivations for collecting these data? To what extent are authorship, various contributions, and funding/support transparently documented (where applicable)?

* Does the submission explain the procedures and methodology followed to produce the dataset (software and instrumentation involved)?

* Does the submission clarify the methods used for quality control in the production of the dataset (i.e. steps taken to normalise the data)?

Form

* Is the content easy to understand, with clear explanations and appropriate language for the intended audience?

* Is the structure coherent and well signposted, with an informative abstract and clearly organised sections?

Accessibility and Reusability

* Is the dataset described in the data paper accessible in a trusted research data repository or equivalent sustainable environment? (Gouzi, Barbot, et al. 2024)

* Is the dataset documented with rich metadata using widely adopted standards, such as Text Encoding Initiative, Music Encoding Initiative, Encoded Archival Description (EAD) or Europeana Data Model?

* Are the reuse conditions of the dataset specified? (i.e. Licences CC0 or CC-BY)
#### 4.1.4 Reviewers' Required Qualifications

Reviewers should have a combination of the following skills and knowledge:

* Familiarity with Digital Humanities tools and methods;

* Familiarity with data management practices, including data collection, analysis, and preservation;

* Expertise in the specific field related to the data paper (preferred);

* Strong writing and communication skills.

#### 4.1.5 Examples of Data Papers

These two examples are very representative of contextualisation, documentation and analysis of datasets in a specific disciplinary context (textual corpora and tropes elements in medieval music). Both journals feature different types of publications including data papers emphasising digital humanities and techniques with high potential for data reuse.

Eipert, T., Bongartz, C. and Moss, F.C. 2025. "Corpus Troporum Dataset: A Digital Catalog of Trope Elements in Medieval Chant." *Journal of Open Humanities Data*, 11(1), p. 30. DOI: https://doi.org/10.5334/johd.318.

Christof Schöch, José Calvo Tello, Ulrike Henny-Krahmer and Stefanie Popp. 2019. "The CLIGS Textbox: Building and Using Collections of Literary Texts in Romance Languages Encoded in TEI XML." *Journal of the Text Encoding Initiative* [Online], Rolling Issue. DOI: https://doi.org/10.4000/jtei.2085

## 4.2 Workflow Papers

### 4.2.1 Definition

A workflow paper is a research output that presents, contextualises and critically reflects on a research workflow.

### 4.2.2 Context and Background

A research workflow is composed of the work steps needed to implement a specific methodology in order to address a research question or provide guidance for analysis. These can be either simply described in a linear manner (resulting in a workflow
description), or they may be further contextualised and embedded in a critical
reflection on the state of the art pertaining to the tools, resources, data and skills that
are at the core of the workflow. The latter form is what we refer to as * workflow papers *.

A workflow paper aims at contextualising the workflow by shedding light on the setting in which it was developed and applied in the first place, which usages can be envisioned (e.g., research questions), what are the challenges to its reusability, etc.

Workflows and their descriptions were first made popular in the natural sciences, specifically in disciplines like bioinformatics, but their use is now spreading to other fields (Atkinson et al. 2017; Goble et al. 2020). The description of workflows in computer science is standardised with the [Common Workflow Language](http://www common-workflow-language org) (Crusoe et al. 2022), which makes it readable by the human and the machine, describing the input, output, and parameters. In the arts and humanities, first efforts in standardisation work have been carried out (Riondet et al. 2018; Chagué et al. 2024) but there is currently no formal unity and no publication culture in this area.

Workflows are also often referred to as "pipelines" or "research scenarios" (Riondet et al. 2018). They can vary in scope, precision, and goal, but share a common purpose: making a methodology reusable for users who are not primarily part of the community that developed the underlying tools and services. Workflows differ from documentation in that they provide general guidance into a methodology.

Their explicit and clear connection to digital resources is oftentimes essential to their
actual reusability, especially in the case of actionable methods that rely on the
execution of an underlying code. Reusability is also fostered by the use of open-source
resources and structural setups that guarantee the best possible sustainability, for
instance, through the involvement of infrastructures. Methods and tools evolve over
time: ideally, workflows should reflect these evolutions and be conceived in an agile
manner that makes it possible to adapt to them.

Workflow papers reflect on all of these aspects and thus embed the workflows and their
descriptions in the scholarly discourse.

When assessing a workflow paper, reviewers are recommended to take into account
the following four categories: State of the Art, Content, Form, and Accessibility and
Reusability.
### 4.2.3 Criteria

#### State of the Art

* Does the submission provide information on workflows that already exist in the concerned field?

* Does the submission explain how this workflow differs from or improves upon the existing ones?

* Are the bibliographic references up to date and relevant?

#### Content

* Are the goals of the workflow clearly stated?

* Is the workflow sufficiently contextualised? To what extent are authorship, various contributions, and funding/support transparently documented (where applicable)?

* Does the submission demonstrate overall quality in terms of the strength of argumentation, research methodology, use of data, general discussion, etc.?

* Is there a use case and how clearly is it presented in its connection to the workflow?

#### Form

* Is the content easy to understand, with clear explanations and appropriate language for the intended audience?

* Is the structure coherent and well signposted, with an informative abstract and clearly organised sections?

#### Accessibility and Reusability

* Is the workflow description accessible in a trusted repository?

* Are the services and datasets used in the workflow accessible in a trusted or reliable repository or equivalent sustainable environment? (Gouzi, Barbot, et al. 2024)

* Is the workflow reusable? (e.g., licensing, FAIR, use of discipline-relevant standards)

* Bonus: Is the workflow referenced in the SSH-Open Marketplace?

### 4.2.4 Reviewers' Required Qualifications

Ideal reviewers should have a combination of the following skills and knowledge:

* knowledge on the standards used in the arts, humanities and social sciences;
* familiarity with the topic of the presented workflow (e.g. automatic text recognition, computer vision, data publication, etc.);

* understanding of research methodology; and

* strong writing and communication skills.

In addition, reviewers should:

* be able to evaluate the quality of the context added by the workflow paper;

* provide constructive feedback to the authors and help them improve their submission.

### 4.2.5 Examples of Workflow Papers

Few workflow papers have been published at the time of writing. Although not technically structured along the requirement lines of the APRF, the following examples can provide inspiration when wondering how to contextualise a well-identified research pipeline.

Chagué, Alix, Hugo Scheithauer, Lucas Terriel, Floriane Chiffoleau, Yves Tadjo-Takianpi. 2022. "Take a Sip of TEL and Relax: A Proposition for an End-to-End Workflow to Enrich and Publish Data Created with Automatic Text Recognition". *Digital Humanities 2022: Responding to Asian Diversity*, Tokyo. URL: https://hal.science/hal-03739767v1.

Marongiu, Paola, Barbara McGillivray, Anas Fahad Khan. 2024. Multilingual Workflows for Semantic Change Research. *Journal of Open Humanities Data*, 10. DOI: 10.5334/johd.179.

## 4.3 Training Materials

### 4.3.1 Definition

Training materials are a type of research output explicitly created to help individuals learn or develop specific skills, knowledge or competencies.

### 4.3.2 Context and Background

Training materials can come in a variety of formats: text-based (manuals, handouts, guides and worksheets), multimedia (audio or video recordings) or digital (e-learning modules, webinars, slideshows etc.). Regardless of the format, training materials are expected to be:

* Purposeful: created with specific learning outcomes or objectives in mind;

* Targeted: designed for particular audiences, such as beginners, intermediates, or advanced learners;
* Structured: organised in a logical sequence to build on previous knowledge or skills.

Whenever possible, training materials should encourage active participation and engagement, using interactive elements such as multiple-choice quizzes and practical exercises in ways that enhance understanding and retention. To increase their relevance and applicability, materials should ideally be developed from concrete research scenarios that reflect real-world challenges and workflows.

In addition, materials should also be designed to be inclusive, catering to diverse learning styles and backgrounds, and include mechanisms for assessment and feedback, allowing instructors to improve the materials continuously.

When assessing the quality of training materials, reviewers should take into consideration the following four categories: State of the Art, Content, Form, and Accessibility and Reusability.

### 4.3.3 Criteria

#### State of the Art

* Does the submission identify existing training materials, courses, tutorials, or documentation on the same topic or skill area (where applicable)?

* Does the submission explain its specific contribution relative to comparable resources (e.g. in terms of audience, scope, domain application, pedagogy, or use of tools and standards)?

* Are the cited resources, tools, standards and bibliographic references up to date and relevant for the intended audience and context?

#### Content

* Are the learning objectives and intended learning outcomes clearly stated and internally consistent with the material provided?

* Does the submission specify prerequisites (skills, knowledge, software access) and keep them realistic for the stated audience?

* To what extent is the content accurate, sufficiently comprehensive and pedagogically appropriate for the stated level and intended audience?

* Does the submission include meaningful opportunities to check learning (exercises, quizzes, reflection prompts, self-assessment) or explain why assessment is not applicable in the given format/context?
## Form

* Is the material easy to follow for its intended audience, using clear language and appropriate terminology (including definitions where needed)?

* Is the structure clearly signposted and easily navigable (modules/sections, headings, navigation cues)?

* Are supporting elements used effectively where relevant (figures, diagrams, demos, templates, example files), and do they improve comprehension rather than add noise?

## Accessibility and Reusability

* Is the training material deposited in a trusted repository or equivalent sustainable environment in an open format and with a persistent identifier?

* Are the licensing and reuse conditions of the training material clearly specified?

* If specific tools, code, or datasets are used within the submission, are they accessible and properly referenced (links/PIDs), and are their licenses compatible with reuse of the training material?

### 4.3.4 Reviewers' required qualifications

Ideal reviewers will have a combination of the following skills and knowledge:

* expertise in the topic covered within the training resource/materials (preferred);

* good understanding of instructional design and experience in delivering training;

* familiarity with the challenges learners can face when engaging with self-directed learning materials;

* understanding of research methodology; and

* strong writing and communication skills.

In addition, reviewers should:

* take into account the overall quality of the training resource and its contribution to knowledge transfer;

* consider accessibility and inclusivity, i.e. approaches taken to meet the requirements of learners with disabilities or social disadvantages;

* provide constructive feedback to the authors to help them improve the training resource/materials.
## 5. Next steps

By building community consensus around its Peer Review Framework, ATRIUM aims to foster a more inclusive and equitable evaluation environment – one that acknowledges the diversity of non-traditional research outputs that are often excluded from existing evaluation mechanisms. Through its emphasis on open and transparent peer review, the framework promotes fairer recognition of reviewers’ labour, greater equity for authors, and a more accountable and reflective space for discussing research quality and practice.

This first version of the framework, which covers evaluation criteria for data papers, workflow papers, and training materials, is designed to support reviewers in carrying out consistent, well-grounded evaluations, while also strengthening the recognition of peer review as a scholarly contribution in its own right.

Building on this initial implementation, the next phase of the ATRIUM project will expand the framework with additional sets of evaluation criteria for further research outputs, including 3D assets, digital scholarly editions, and software and code. In parallel, the existing criteria will be reviewed and refined in light of their practical application in the context of DARIAH's overlay journal *Transformations*, with particular attention to clarity, consistency across output types, and alignment with the underlying design principles. Cross-criteria checks will be carried out to identify potential gaps and unintended biases, and to ensure that the framework remains internally consistent and easy to use.

Beyond its application in concrete editorial workflows, the APRF forms part of DARIAH's contribution to the CoARA commitments as articulated in its [CoARA Progress Report and Action Plan for 2025-2027](https://coara.eu progress-report-and-action-plan-for-2025-2027). In this broader policy context, the framework aims to advance the recognition of innovative infrastructural contributions to the arts and humanities, and to support more responsible research assessment practices at the European level.
References

Atkinson, Malcolm, Sandra Gesing, Johan Montagnat, and Ian Taylor. 2017. "Scientific Workflows: Past, Present and Future." *Future Generation Computer Systems* 75: 216-227.
https://doi.org/10.1016/j.future.2017.05.041.

Avanço, Karla. 2022. "Data Papers... and FAIR: The Road to FAIR." Accessed December 16, 2025. https://doi.org/10.58079/trry.

Barbot, Laure, Maja Dolinar, Edward Gray, Cristina Grisot, Klaus Illmayer, Michael Kurzmeier, and Barbara McGillivray. 2024. "Contextualizing Research Tools and Services through Workflows in the SSH Open Marketplace." *Journal of Open Humanities Data* 10.
https://doi.org/10.5334/johd.192.

Best Practices and Guidelines | UpSkills Project. 2021. CLARIN UpSkills Project.
Accessed October 23, 2024.
https://upskillsproject.eu/deliverables/io2/best-practices.

Braukmann, Ricarda, Ellen Leenarts, Simon Saldner, Veronika Keck, Judith Wehmeyer, Alejandra Albuerne, Klaus Illmayer, Matej Durco, Vasso Kalaitzi, Tatsiana Yankelevich, Darja Fišer, Ana Cvek, and Anca Vlad. 2022. *D6.11 SSHOC Trainer Toolkit (Final), Version 1.0*. Zenodo. https://doi.org/10.5281/zenodo.6564283.

Chagué, Alix, Floriane Chifoleau, Matthias Gille Levenson, Hugo Scheithauer, and Ariane Pinche. 2024. *Chaînes d'acquisition, de traitement et de publication du texte*. Technical report, Consortium ARIANE. https://hal.science/hal-04734959.

Crusoe, Michael R., Sanne Abeln, Alexandru Iosup, Peter Amstutz, John Chilton, Nebojša Tijanić, Hervé Ménager, Stian Soiland-Reyes, Bogdan Gavrilović, and Carole Goble. 2022. "Standardizing Computational Reuse and Portability with the Common Workflow Language." *Communications of the ACM* 65 (6): 54-63. https://doi.org/10.1145/3486897.

Curry, Stephen, Sarah de Rijcke, et al. 2020. *The Changing Role of Funders in Responsible Research Assessment: Progress, Obstacles and the Way Ahead*. RoRI Working Paper No. 3.
https://sfdora.org/wp-content/uploads/2020/12/GRC_Responsible_Research_Assessment_RoRI_WP03_finalpdf.pdf.

Edmond, Jennifer, Victoria Garnett, Haris Goulis, and Krista Schuster. 2019. *Embedding PARTHENOS Training into Higher Education Curricula: Report on Activities of Task 7.4*. Zenodo. https://doi.org/10.5281/zenodo.3356752.

Global Core Biodata Resource. n.d. "Data Paper Webinar." Accessed December 16, 2025.
https://www.gbif.org/data-papers.
Goble, Carole, Sarah Cohen-Boulakia, Stian Soiland-Reyes, Daniel Garijo, Yolanda Gil, Michael R. Crusoe, Kristian Peters, and Daniel Schober. 2020. "FAIR Computational Workflows." *Data Intelligence* 2 (1-2): 108-121. https://doi.org/10.1162/dint_a_00033.

Gouzi, Françoise, Anne Baillot, Sarah Bénière, Carolina Delmazo, and Toma Tasovac. 2025. "Building a Peer Review Evaluation Framework for Non-Traditional Research Outputs." Poster presented at the DH2025 Conference, Lisbon, Portugal, June 20, 2024. Zenodo. https://doi.org/10.5281/zenodo.15967748.

Gouzi, Françoise, Laure Barbot, Matej Ďurčo, Sally Chambers, and Toma Tasovac. 2024. *DARIAH Data Policy*, Version 2. Zenodo. https://doi.org/10.5281/zenodo.14515142.

Gouzi, Françoise, and Toma Tasovac. 2025. *DARIAH CoARA Progress Report and Action Plan* (2025-2027). Zenodo. https://doi.org/10.5281/zenodo.17287053.

Jefferies, Neil, Fiona Murphy, Anusha Ranganathan, and Hollydawn Murray. 2019. "Data2Paper: Giving Researchers Credit for Their Data." *Publications* 7(2): 36. https://doi.org/10.3390/publications7020036.

Langfeldt, Liv, Ingvild Reymert, and Dag W. Aksnes. 2021. "The Role of Metrics in Peer Assessments." *Research Evaluation* 30 (1): 112-126. https://doi.org/10.1093/reseval/rvaa032.

Lewandowska, Karolina, Emanuel Kulczycki, and Michael Ochsner. 2023. "Evaluation of the Arts in Performance-Based Research Funding Systems: An International Perspective." *Research Evaluation* 32 (1): 19-31. https://doi.org/10.1093/reseval/rvac017.

Maryl, Maciej, Magdalena Wnuk, Françoise Gouzi, and Tomasz Umerle. 2024. *Guidelines on Publishing and Evaluating Innovative Outputs in Social Sciences and Humanities*. OPERAS-PLUS Deliverable 4.1 (Version 1.0 Draft). Zenodo. https://doi.org/10.5281/zenodo.14221728.

Ministère de l'Enseignement Supérieur, de la Recherche et de l'Innovation. 2024. A *Practical Guide for PhD Students*. Accessed December 16, 2025. https://www.ouvrirlascience.fr/passeport-pour-la-science-ouverte-guide-pratique-a-l-usage-des-doctorants/.

Open Research Europe. n.d. *Open Data, Software and Code Guidelines*. Accessed December 16, 2025. https://open-research-europe.ec.europa.eu/for-authors/data-guidelines.

Pinfield, Stephen. 2009. "Journals and Repositories: An Evolving Relationship?" *Learned Publishing* 22 (3): 165-175.

Programming Historian. n.d. *Reviewer Guidelines*. Accessed October 23, 2024. https://programminghistorian.org/en/reviewer-guidelines.
Riondet, Charles, Dorian Seillier, Lionel Tadjou, and Laurent Romary. 2018. *Standardization Survival Kit*. Technical report, PARTHENOS. https://hal.science/hal-01513531.

Rousi, Antti M., and Mikael Laakso. 2022. "Overlay Journals: A Study of the Current Landscape." *Journal of Librarianship and Information Science* 56 (1): 15-28. https://doi.org/10.1177/09610006221125208.

Sánchez, María, Eva Méndez, Angus Whyte, Katrine Weisteen Bjerde, Silvia Martínez, Gonzalo Bueno, and Ignacio Ucar. 2023. *D2.3 Community-Endorsed Quality Assurance and Certification Framework for Professional Training and Qualifications, Version 1.3*. Zenodo. https://doi.org/10.5281/zenodo.8305482.

Smit, Jeroen P., and Laurens K. Hessels. 2021. "The Production of Scientific and Societal Value in Research Evaluation: A Review of Societal Impact Assessment Methods." *Research Evaluation* 30 (3): 323-335. https://doi.org/10.1093/reseval/rvab002.

Stodden, Victoria, Marcia McNutt, David H. Bailey, Ewa Deelman, Yolanda Gil, Brooks Hanson, Michael A. Heroux, John P. A. Ioannidis, and Michela Taufer. 2016. "Enhancing Reproducibility for Computational Methods." *Science* 354 (6317): 1240-1241. https://doi.org/10.1126/science.aah6168.

Tasovac, Toma, Anne Baillot, and Matthew Black. 2025. *ATRIUM Policy Brief*. Zenodo. https://doi.org/10.5281/zenodo.17788168.

Tasovac, Toma, Laurent Romary, Erzsébet Tóth-Czifra, et al. 2023. *The Role of Research Infrastructures in the Research Assessment Reform: A DARIAH Position Paper*. HAL, Version 1. https://hal.science/hal-04136772v1.

The Turing Way Community. n.d. "Data Papers." Accessed December 16, 2025. https://book.the-turing-way.org/communication/dif-articles/data/.

Tóth-Czifra, Erzsébet. 2021. *DARIAH Impact Case Study Series: An Open Science Voice for the Humanities—A Humanities Voice for Open Science*. Zenodo. https://doi.org/10.5281/zenodo.5863209.

Williams, Kate. 2020. "Playing the Fields: Theorizing Research Impact and Its Assessment." *Research Evaluation* 29 (2): 191-202. https://doi.org/10.1093/reseval/rvaa001.
# Consortium

## Research Infrastructures

<figure><img src="image_4.png" /></figure>

DARIAH-EU

Digital Research Infrastructure
for the Arts and Humanities

<figure><img src="image_5.png" /></figure>

ARIADNE

Research Infrastructure AISBL

<figure><img src="image_6.png" /></figure>

CLARIN

Centre d'Extraction et de Recherche en Littérature et de Technologie en Sciences de la Communication

OPERAS

open scholarly communication in the european
research area for social sciences and humanities

## Beneficiaries

<figure><img src="image_7.png" /></figure>

Archaeology
Data Service

<figure><img src="image_8.png" /></figure>

ATHENA®

Research & Innovation
Information Technologies

<figure><img src="image_9.png" /></figure>

<figure><img src="image_10.png" /></figure>

ATHENS UNIVERSITY
OF ECONOMICS
AND BUSINESS

BELGRADE CENTER FOR DIGITAL HUMANITIES [+]

<figure><img src="image_11.png" /></figure>

Consiglio Nazionale
delle Ricerche

<figure><img src="image_12.png" /></figure>

CHARLES
UNIVERSITY

<figure><img src="image_13.png" /></figure>

foxcub.

BL

Inria

<figure><img src="image_14.png" /></figure>

LUDWIG-MAXIMILIAN-UNIVERSITÄT
MÜNCHEN

<figure><img src="image_15.png" /></figure>

Radboud Universiteit

<figure><img src="image_16.png" /></figure>

<figure><img src="image_17.png" /></figure>

University of
Sheffield
## Affiliated Entities

<figure><img src="image_18.png" /></figure>

Institute of Archaeology
Czech Acad Sci, Brno

<figure><img src="image_19.png" /></figure>

INSTITUTE OF ARCHAEOLOGY
OF THE CAS PRAGUE

<figure><img src="image_20.png" /></figure>

FORTH

FOUNDATION FOR RESEARCH AND TECHNOLOGY - HELIAS

<figure><img src="image_21.png" /></figure>

LABORATÓRIO NACIONAL
DE ENGENHARIA CIVIL

ÖAW

AUSTRIAN
ACADEMY OF
SCIENCES

<figure><img src="image_22.png" /></figure>

PIN

POLO
UNIVERSITARIO
CITTA' DI PRATO

SERVIZIO DI RICERCA
E INNOVAZIONE
PER LA CULTURA
DI FUTURI ESENCIE

<figure><img src="image_23.png" /></figure>

<figure><img src="image_24.png" /></figure>

<figure><img src="image_25.png" /></figure>

SND

Swedish National Data Service

<figure><img src="image_26.png" /></figure>

THE CYPRUS
INSTITUTE

RESEARCH-TECHNOLOGY-INNOVATION

<figure><img src="image_27.png" /></figure>

UNIVERSIDADE E
COIMBRA

<figure><img src="image_28.png" /></figure>

université
de TOURS

University of
South Wales
Prifysgol
De Cymru

## Disclaimer

All information provided reflects the status of the ATRIUM project at the time of writing and may be subject to change. Neither the ATRIUM Consortium as a whole, nor any single party within the ATRIUM Consortium warrant that the information contained in this document is capable of use, nor that the use of such information is free from risk. Neither the ATRIUM Consortium as a whole, nor any single party within the ATRIUM Consortium accepts any liability for loss or damage suffered by any person using the information.

This document does not represent the opinion of the European Community, and the European Community is not responsible for any use that might be made of its content. Funded by the European Union. Grant Agreement number 101132163. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Executive Agency. Neither the European Union nor the granting authority can be held responsible for them.

© 2024 by the authors, the ATRIUM consortium. This work is licensed under a "CC BY 4.0" license.