title: Challenges of Curating for Reproducible and FAIR Research Output
Creators: name: Peer, Limor
name: Arguillas, Florio
name: Honeyman, Tom
name: Miljković, Nadica
name: Gehlen, Karsten Peters-Von
name: CURE-FAIR WG Subgroup 3
Publisher: name: Research Data Alliance
Year: 2021

description: The goal of the CURE-FAIR WG is to establish standards-based guidelines for curating for reproducible and FAIR data and code. The final deliverable of the Working Group (WG) is a document outlining CURE-FAIR standards-based guidelines for best practices in publishing and archiving computationally reproducible studies. To support the deliverable, the WG created four subgroups, each tasked with studying and summarizing a particular aspect of this work: CURE-FAIR Definitions, Practices, Challenges, and Alliances. This document reports on the work of subgroup 3, on CURE-FAIR Challenges. The aim of Subgroup 3 is to describe the challenges of preparing and reusing materials required for computational reproducibility. The group has reviewed the literature and collected use cases, stories, and interviews with various stakeholders (researchers, publishers, funders, data professionals, IT, repositories) who are trying to reproduce computation-based scientific results and processes. This report represents the WG’s first milestone toward the final deliverable: CURE-FAIR survey of current state. The main challenges experienced by those who are preparing or attempting computational reproducibility are: 1. Lack of clarity on standards for computational reproducibility. We find variation in the meaning and the practice of computational reproducibility. There is a need to define criteria for what should be reproduced, what should be archived (and for how long), and how to determine whether computational reproducibility is possible or practical. 2. Sociocultural factors. Several challenges have to do with “human factors” at the micro level (e.g., insufficient training and skills) and macro level (e.g., lack of awareness in some domains, professional incentive structures). These factors in combination contribute to insufficient progress on creating conditions for facile and ubiquitous computational reproducibility. 3. The R in FAIR (Reusable). The ability to reuse digital materials when preparing or attempting computational reproducibility is the most common challenge reported. In the context of computational reproducibility, the “digital object” is a process that comprises various components (i.e., code, data, workflow, and execution or computational environment).2 The components and the process as a whole need to be FAIR. 4. Machine readable is not enough. FAIR must be human-readable. Computational reproducibility by humans requires that they understand what scripts are intended to do and why. Moreover, since in many cases not all steps in a workflow are script-based, the next best thing is proper documentation and metadata so that humans can interpret the scripts and understand their function in order to attempt computational reproducibility. 5. Solutions tend to be ad-hoc and not FAIR. Some practical difficulties are met with improvised solutions by those attempting computational reproducibility. These solutions often do not advance transparency and reproducibility on the whole. 6. Differences across stakeholders and domains. Special attention must be given to the requirements and practices of different scholarly domains, and indeed to the raw materials that comprise an academic field. In addition, different stakeholders emphasize different computational reproducibility challenges. Solutions addressing the feasibility of computational reproducibility need to consider these contexts.