TIER2 D3.1 - State-of-play on methods, tools, practices to increase reproducibility across diverse epistemic contexts
Creators: name: Ulpts, Sven
name: Osborne, Christopher
name: Kopeinik, Simone
name: Cole, Nicki Lisa
name: Kormann, Eva
name: Kowald, Dominik
name: Schneider, Jesper W.
name: Ross-Hellauer, Tony
Publisher: name: OSF
Year: 2023

description: This project deliverable has been submitted to, but not yet reviewed by the Research Executive Agency (REA), and might thus be subject to change.

The deliverable was resubmitted to the REA on June 30, 2024, in light of feedback received after the first reporting period.

TIER2, over the course of its three-year duration (2023-2025), aims to contribute to improving this
situation in various ways. Key to our approach is to centre “epistemic diversity” (defined below) by
selecting three broad research areas — social, life, and computer sciences, and two cross-
disciplinary stakeholder groups of research publishers and funders — to systematically investigate
the roles, nature, and meanings of reproducibility across contexts. Through coordinated co-
creation with these communities, TIER2 aims to boost knowledge on reproducibility, create tools,
engage communities, implement interventions and policy across different contexts to increase
reproducibility where it is relevant.

This Deliverable details work to provide the theoretical, evidential and strategic framework for the
project. The aim is to capture the complexity in the meaning(s) of reproducibility across contexts,
provide a conceptual framework that systematically relates epistemic diversity to reproducibility
by identifying key research characteristics affecting the relevance and feasibility of different types
of reproducibility, establish current levels of knowledge on which interventions work in which
contexts (including in two specific cross-cutting research methods (qualitative and Machine
Learning-driven research), and devise a strategic intervention logic for designing and
implementing interventions that aim at sustainable behavioural change towards increased
reproducibility.

This work has been addressed through seven ambitious individual studies:
• “Definitions of reproducibility” (Section 2)
• “Epistemic diversity and Knowledge Production Modes” (Sec. 3)
• “Scoping review and evidence mapping of interventions aimed at improving reproducible
and replicable science” (Sec. 4)
• “Review of conceptions and facilitators of and barriers to reproducibility of qualitative
research” (Sec. 5)
• “Review of conceptions and practices regarding reproducibility in Machine Learning (ML)-
driven research” (Sec. 6)
• “Changing behaviour in the academy: A strategy for improving research culture and
practice” (Sec. 7)

This work hence fills knowledge gaps to enable the mapping of “impact pathways”, i.e., the
possible paths that connect input to output, outcome and impact (including linkages of causal
mechanisms and drivers/barriers), to elucidate the routes to increased reproducibility across
diverse contexts. This work is crucial to inform the future stages of TIER2, especially to design,
implement and test a series of new tools and instruments (the “pilots”) conducted within TIER2
Work Packages 4 and 5.