# Building a Peer Review Evaluation Framework for Non-Traditional Research Outputs

Françoise Gouzi, Anne Baillot, Sarah Bénière, Carol Delmazo, Toma Tasovac

► To cite this version:

Françoise Gouzi, Anne Baillot, Sarah Bénière, Carol Delmazo, Toma Tasovac. Building a Peer Review Evaluation Framework for Non-Traditional Research Outputs. DH2025 Accessibility & citizenship, Jul 2025, Lisbon, Portugal. 2025, 10.5281/zenodo.15967748. [https://hal.science/hal-05251762v1](https://hal.science/hal-05251762v1)

HAL Id: hal-05251762

Submitted on 12 Sep 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

ATRIUM

# Building a Peer Review Evaluation Framework for Non-Traditional Research Outputs

Françoise Gouzi; Anne Baillot; Sarah Bénière; Carol Delmazo; Toma Tasovac

The ATRIUM project aims to create a **robust framework for open peer review** assessment of **non-traditional research outputs** to maximise the **quality** and **impact** of **arts and humanities research in Europe**, contributing to **research assessment reform**. Building on **literature review and sources analysis** (reports, repositories, and discovery portal classifications - such as the SSH Open Marketplace), **three sets of evaluation criteria** were developed:

<figure><img src="image_3.png" /></figure>

## Data Papers

### Content

* Does the data paper provide the context in which the dataset was produced?

* Does the submission explain the procedures and methodology followed to produce the dataset?

* Does the submission clarify the methods used for quality control in the production of the dataset?

### Form

* Is the submission clear and understandable in the original language?

* Does the submission demonstrate a high level of clarity and coherence in its argumentation?

* Is there an abstract?

### Data availability

* Is the dataset described in the data paper deposited in a trusted repository?

* Is the dataset documented with rich metadata using widely adopted standards?

* Is the dataset linked back to the data paper submitted?

* Are the reuse conditions specified?

* Does the data paper provide information on data storage, security, and backup solutions?

<figure><img src="image_4.png" /></figure>

## Workflow Papers

### Content

* Are the goals of the workflow clearly stated?

* Is the workflow sufficiently contextualised?

* Does the manuscript demonstrate overall quality in terms of the strength of argumentation, research methodology, use of data, and general discussion?

* Is there a use case and how clearly is it presented in its connection to the workflow?

### Form

* Is the content easy to understand, with clear explanations and appropriate language for the intended audience?

* Is there a section on the reusability of the workflow?

### Data availability

* Are the tools and datasets used in the workflow accessible in a trusted repository?

* Is the workflow reusable?

* Bonus: Is the workflow referenced on the SSH-Open Marketplace?

<figure><img src="image_5.png" /></figure>

## Training Materials

### Content

* Is the content accurate, comprehensive and up-to-date?

* Does the content align with learning objectives?

* Is the content easy to understand, with clear explanations and appropriate language for the intended audience?

* If available, do interactive elements engage learners constructively and meaningfully?

### Form

* Does the training material clearly describe the learning objectives and have clear learning outcomes?

* Does the training material clearly indicate the intended target audience?

* Does the training material clearly indicate if there are prerequisites or prior knowledge required?

* Is there a clear structure to the training material with a logical flow of topics?

* Is the training material designed in a modular way, easy to reuse in other contexts?

* Is the training material delivered in an open format?

### Data availability

* If specific tools or datasets are used in the training materials, are they openly available, accessible in a trusted repository with clear licensing information?

This framework aims to incentivise researchers to share diverse research outputs and support reviewers in their evaluation work! Similarly, three more sets of evaluation criteria will be created within the ATRIUM project. They will be implemented and tested in the *Transformations Diamond Journal* edited by DARIAH. Read more about the assessment framework through the QR Code.

<figure><img src="image_6.png" /></figure>

Funded by
the European Union

<figure><img src="image_7.png" /></figure>

ATRIUM: Advancing Frontier Research
in the Arts & Humanities

<figure><img src="image_8.png" /></figure>

@atrium-eu.bsky.social

<figure><img src="image_9.png" /></figure>

DARIAH-EU

<figure><img src="image_10.png" /></figure>

ARIADNE
Research Infrastructure AISBL

<figure><img src="image_11.png" /></figure>

CLARIN

<figure><img src="image_12.png" /></figure>

<figure><img src="image_13.png" /></figure>