Renku: a platform for sustainable data science
Venue: NeurIPS 2023 Datasets and Benchmarks Spotlight (2023)
Keywords: reproducibility
reusability
platforms
sustainability
community
dataset development
TL;DR: Renku is a platform that enables and encourages sustainable data science and machine learning practices, from dataset creation to dissemination.

Abstract: Data and code working together is fundamental to machine learning (ML), but the context around datasets and interactions between datasets and code are in general captured only rudimentarily. Context such as how the dataset was prepared and created, what source data were used, what code was used in processing, how the dataset evolved, and where it has been used and reused can provide much insight, but this information is often poorly documented. That is unfortunate since it makes datasets into black-boxes with potentially hidden characteristics that have downstream consequences. We argue that making dataset preparation more accessible and dataset usage easier to record and document would have significant benefits for the ML community: it would allow for greater diversity in datasets by inviting modification to published sources, simplify use of alternative datasets and, in doing so, make results more transparent and robust, while allowing for all contributions to be adequately credited. We present a platform, Renku, designed to support and encourage such sustainable development and use of data, datasets, and code, and we demonstrate its benefits through a few illustrative projects which span the spectrum from dataset creation to dataset consumption and showcasing.

Renku: a platform for sustainable data science
Rok Roškar', Chandrasekhar Ramakrishnan' Michele Volpi',
Fernando Perez-Cruse Mohammad Alisafaee", Phiilip Fischer", Lilian Gasser',
Eliza Jean Harris' Firat Ozdemir's Patrick Paitz", Carl Remlinger?, Luis Salamanca' Ralf Grubenmann's Tasko Olevski", Elisabet Capón García', Lorenzo Cavazzi Jakub Chrobasik?, Andrea Cordoba Alessandro Degano',
Jimena Dupré", Wesley Johnson', Eike Kettner', Laura Kinkead',
Seán Murphy?, Flora Thicbaut Olivier Verscheure Swiss Data Science Center, ETH Zürich, Zürich, Switzerland.
2. Swiss Data Science Center, EPFL, Lausanne, Switzerland.
Swiss Federal Institute for Forest, Snow and Landscape Research, WSL, Birmensdorf, Switzerland
Abstract
Data and code working together is fundamental machine learning (ML), but the context around datasets and interactions between datasets and code are in general captured only rudimentarily. Context such as how the dataset was prepared and created, what source data were used, what code was used in processing, how the dataset evolved, and where has been used and reused can provide much insight, but this information is often poorly documented. That is unfortunate since it makes datasets into black-boxes with potentially hidden characteristics that have downstream consequences. We argue that making dataset preparation more accessible and dataset usage easier to record and document would have significant benefits for the ML community: it would allow for greater diversity in datasets by inviting modification to published sources, simplify use of alternative datasets and, in doing so, make results contributions to be adequately credited. We present a platform, Renku, designed support and encourage such sustainable development and of data, datasets, and code, and we demonstrate its benefits through a few illustrative projects which span the spectrum from dataset creation dataset consumption and showcasing.
transparent and robust, while allowing for all
Introduction
Datasets are key to all aspects of machine learning (ML) and data science, from the development of novel algorithms to training of production models. Unfortunately. variety of problems plague
the generation, use, and dissemination of datasets 2
39 32
The negative downstream effects of
issues related to dataset preparation on model performance and application are well-documented [39]. Less well-documented are the difficulties in understanding and managing the deficiencies on the less technical aspects: the issues of attribution, reuse, dissemination, and replicability. Because ML research so heavily dependent on datasets, [32] highlight the lacking and insufficiently robust mechanisms for dataset citation and usage tracking as core missing pieces needed to improve the health the community as a whole. The lack of diverse datasets on which to advance the field has also recently been raised point of serious concern (22), and the lack of dataset provenance is barrier to trustworthy AI systems 19
Furthermore, beyond requiring planning and design 15 dataset creation requires significant com- putational effort, which implies code, pipelines and compute resources. However, in the majority of cases, because dataset creation pipelines are treated as one-offs and are not recorded or published [12]. researchers who want to augment dataset or repeat its construction with different data cannot do so.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

This significantly limits the engagement of the community in this arguably foundational phase of ML research [44]. The ML community has proposed and adopted some solutions specific to the nature of MI. dataset creation and use, such e.g. Datasheets for Datasets [1]] (see also 7 for a recent reformulation). While these proposals raise awareness of the importance of datasets and the need for documentation and systematic rigor, they are largely prescriptive, not machine-readable, and cannot easily be automated or validated which makes them quickly become outdated.
Scientific research has struggled with these same issues for many years especially when it comes to valuation of datasets as legitimate research outputs and their citation and attribution. In the past decades, much attention has also been dedicated to the struggles of scientific disciplines to reproduce (sometimes very influential) results [37]. Similarly, the ML community is not immune to the issues of reproducibility [16]; in the case of datasets, where the introduction of hidden biases during the preparatory stages might critically influence the downstream chain of model training and production applications, the need for more comprehensive solutions is high. In this sense, it helps to lean on the concepts espoused by the Open Science movement, where technical questions about robustness, reproducibility and FAIR-ness (FAIR: Findable, Accessible, Interoperable, Reusable as proposed by (43) overlap with and complement meta concerns about access to knowledge and transparency (8)
To work openly is to structure the research process in such way that its outcomes can be reproduced, reused, and understood by anyone at any time. This implies structure, discipline, and adherence to norms, which in turn require significant cognitive, technical, monetary and temporal overhead. The cost of openness a significant factor in preventing wider adoption of these practices [3] despite enormous efforts by research institutions and funding agencies to exert top-down pressure through open science requirements and initiatives.
Openness, however, is not enough. We argue that sustainability is the more appropriate mental model, because it implies continuity and dynamics; i.e., simply "opening" dataset says nothing about how it was assembled and created, where it is eventually used (outside of formal citations), how it might be repackaged, and the many downstream implications that various decisions at each stage of the process might have. The process of dataset creation is for the most part opaque and hidden behind qualitative descriptions in publications. Similarly, sharing source code does not guarantee that anyone can actually run it and, much less, obtain the expected results. Datasets and data are not the final products of the process; they are often deeply intertwined with the models themselves, both in publications and in the general discourse around ML applications. Therefore, the entire cycle must be considered: from dataset assembly and creation to their consumption resulting in eventual in production. While such robust documentation and
deployment of models for benchmarking systematic adherence to standards would be ideal, researchers are pragmatic; they want to get their work done and be recognized for it. In ML research, dataset development tends to be undervalued and is consequently not systematically documented [5]. Furthermore, when voluntary compliance with "best-practices" and efficiency efficiency [3]. In a sustainable system, the actions of individuals translate simultancously into benefits for themselves (e.g. in the form of increased short-term efficiency longer-tent recognition) and the communities within which they operate (e.g. improved transparency) with little or no overhead.
in competition for time, researchers will tend to optimize for
We present Renku] an open-source platform that aims to address some of these problems from another angle: through the creation of a more sustainable system within which ML research is conducted, by incentivising creators and consumers of datasets and models to work in ways that benefit both themselves and their communities. We describe the vision of the Renku platform and its practical implementation; we outline the impact this system can have on dataset development for ML applications; and we share several real-world use cases that demonstrate to varying degrees the practical manifestation of sustainability through the platform, with specific focus on datasets, their preparation, processing, and dissemination. The use cases illustrate how the platform relieves data scientists and ML researchers from some basic sources of friction present in every data analysis project by: providing easy access to reproducible, containerized environments; simplifying shared data management; providing tooling for building reproducible pipelines; ensuring that data, code, and compute environment are versioned together and always self-consistent. This ecosystem, in turn, allows for the creation and dissemination of datasets, where code and pipelines that are essential for the creation of datasets can readily be inspected, scrutinized, shared, and rerun. The added
https://renkulatt 10 developed at the Swiss Data Science Center, a strategic focus area of the ETH domain with offices at ETH Zurich, EPFL, and the Paul Scherrer Institute (PSI)

---

transparency of such a system provides the missing piece to make dataset creation collaborative, community exercise.
Platform vision
The Renku platform's 's vision is facilitate sustainable data science. In such practice, individuals work in a knowtedge-capturing way to collaborate, discover, reuse and reproduce results easily,
To motivate this idea, it is helpful to draw a parallel to version control tool for sustainable software development. Version control contributes to making software development sustainable by helping developers capture and describe changes to the codebase over time. This allows them to return to previous points in time, run the code as it was then, understand motivations for the changes made the code, and explore implementation alternatives without breaking systems in production. We believe that tools are needed in ML to support similar ways of working: tools that make it possible to return project at previous point in time, understand how the project evolved, and allow for exploration of alternatives while supporting reuse.
The benefits of working in sustainable data science environment are manifold. For example, one can easily return to a project as it was in an earlier state, and still successfully run the code; one can inspect colleague's 's project and pick up where they left off; a group lead can quickly identify the uses of a shared dataset and actively scrutinize the results immediately, with zero setup overhead. On the group level, the captured knowledge data and techniques, and to offer members a quick, actionable overview of their group's 's activities. On the community level, this knowledge capture leads to improved transparency, enhances trust in results, and identifies all artifacts essential to the end-to-end process, such as data pipelines and datasets.
serve facilitate exchange among colleagues, reuse of
Our vision is, therefore, to build system that offers simple solutions to problems that data scientists, researchers, and ML users face every day, and in doing so inject just enough structure to make their work more sustainable. The platform can then be used to propagate the information of what is being done with data and how on a variety of levels and scales. By providing researchers with solutions that alleviate common points of friction (e.g. access to efficient compute and storage), increasing productivity and efficiency, while at the same time capturing knowledge along the whole life-cycle, the Renku platform makes it possible record the progress of research it happens.
3 Platform design and features
Given the above vision of enabling sustainable data science, the platform is designed around few basic concepts: (i) an integrated solution for Code, Datasets, Workflows, and Computational environments; (ii) simple interfaces for consistent compute and data access; (iii) building on existing solutions and avoiding vendor lock-in; (iv) connecting everything through a comprehensive, flexible knowledge graph. This section is deliberately brief and omits much detail; further information can be found by visiting the live projects on
io (see links in citations).
3.1 Building blocks
Renku makes opinionated choices about large technology stack to provide the basic building blocks of any data science-related project: Code, Datasets, Workflows, and Computational environments (Figure l).
Code: Modern data analysis does not happen without code; the problem of working collaboratively with code has largely been solved by the invention of decentralized version control systems like git. Renku does not try to reinvent the wheel and relies fully on this industry standard.
Datasets:
Renku encourages users organize their data via Datasets, which can be thought of as annotated data containers, i.c. lightweight encapsulations of data, useful for a project together with the relevant metadata. Their true value, however, is that they are an easy way to share and incorporate data into projects either from other Renku projects or external data repositories (e.g. Dataverse or Zenodo), while fully preserving the relevant metadata. If a Dataset is published and receives a DOI, this can be tracked in Renku as well. Datasets can evolve as new data becomes available; this
3

---

Versioned Code foundation of collaborative machine learning and data science.
Reusable Data
Comprehensible Workflows
Containerized Environments
Configurable data packages with rich
Encode relationships between code and ditáa.
Easy access to consistent cloud execution
environments.
Knowledge Graph
Connecting Code, Datasets, Workflows, Computational environments.
Figure I: A summary of the components of Renku.
provenance is also self-consistently traced in Renku, whether derived dataset is created by the original author or someone else entirely.
Data can be stored using different backends. The default to version data together with the code, but store git Large File Storage (LFS). Users can also opt connect Dataset an external data source, such as an Amazon Web Services Simple Storage Service (S3) bucket or Azure blob storage ? Renku can handle the minutiae of bringing the data to the user's system (including access credential management) so that after the initial setup they no longer have to worry about how to obtain the data. External sources currently do not include databases; while projects may retrieve data from databases through scripts, and store results of queries in files, Datasets cannot currently use databases as data sources directly.
Renku supports ways of restricting access to data. For data stored in git LFS within a Renku instance, the data access is limited those who have access to the repository. If data is external, access is managed by third-party and not by Renku. The infrastructure that hosts the compute resources for running workflows and interactive sessions designed be multi-tenant and prevents users from accessing resources allocated to others. While this should be sufficient for many kinds of sensitive data, some highly-sensitiv or confidential data may require additional precautions on the data rest, like encryption, which Renku does not itself provide. In those cases, we recommend that Renku instance deployed in secure environment and urge users to consider their levels of acceptable risk to avoid potential leakage of sensitive data. More information about deployment and administration can be found in Section the Supplementary Materials.
Workflows are where code meets data; a workflow can be anything from a shell script to full-blown distributed pipeline. Renku provides a very simple workflow system which captures the basic relationship between data and any code that can be executed on the command line. The primary function of Renku workflows is to capture provenance and make it possible to reproduce and separate
Workflows:
rerun computational pipelines involving several intertwined steps that would normally scripts, or even as a part of one (usually very long) script. The system aims to relieve the researcher from the burden of remembering how specifie result was generated.
Once a workflow is executed, content hashes of all components are recorded in the metadata and can be checked against any future state. If any of the pieces are updated (code or data), Renku can generate an execution plan rerun the parts that need to be updated. The workflows are recorded using open ontology based on W3C recommendationf and schema.org but be serialized into a workflow language to execute on a variety backends including cloud and HPC. Currently, we
2 Inttegrations are not limited to these services and additional backends are continously heing added. https: h/www. w3. wrt.////rrrooo///////

---

support the Common Workflow Language toil, and Argo workflows F external extensions via simple plug-in system.
but others can be added by
Computational environment: Code requires a computational environment in which run: this environment includes software libraries and languages, compilers, system libraries, and the operating system. Failure to encapsulate this environment collaboration, reuse and reproducibility in computational research [23]] Renku makes heavy use of container technologies to alleviate some of the pain related to computational environments. By default, each project includes a Dockerfile that builds docker image for every commit in the project's repository. This ensures that a computational environment matching any commit of the project is always available. Users can either use these images on their local machine (or another compute resource) or launch hosted interactive sessions directly in RenkuLab, the hosted part of the platform.
sufficient degree is the primary hurdle for
3.2 Interfaces for consistent compute and data access
The Renku platform consists of a Renkul instance (e.g. our public instance atfhttps://ennkii 1o) with browser- based interface and Python library, which also provides command-line interface (CLI The two interfaces can be used together or independently, depending on the project constraints. We strive for feature parity between the two interfaces, meaning that users should have a consistent experience with launching compute sessions and accessing data in both cases. The CLI in addition allows users to take project completely off-line to another compute resource, without relying any hosted (cloud) infrastructure. This is particularly useful in cases involving sensitive data or restricted environments.
3.3 Building on Open Source
Renku open source and licensed under the Apache 2.0 License. It builds and relies on a number of well-known open-source technologies: Kubernetes the infrastructure layer, docker for containeriza- tion, git and GitLab for version control and repository management, and the Jupyter ecosystem [2]] for interactive sessions. The platform is designed to prevent vendor lock-in, i.e. to keep working on their project within the hosted platform or even using any Renku tooling. They may simply take their Project (a git repository) anywhere and use it with standard tools like git and docker.
not obliged
3.4 Connect Everything: the Renku Knowledge Graph
A central component of the Renku platform is the knowledge graph (KG) made up of metadata aggregated from all the projects on the platform. All primary Renku entities (Projects, Code, Datasets, Workflows) are represented in the KG. This metadata can facilitate many different kinds of functionality. For example, recording a workflow allows a user to use the system to re-execute parts of their computation if some upstream data changes. Conversely, if a user is interested in particular Dataset, they can use the KG to understand how it has been created or analyzed in the past, who has experience with the data and what kinds of results have been derived from it. The KG is based on RDF and designed to be interoperable and extensible by 3rd party plug-ins with domain-specing metadata facilitate use cases across broad spectrum of applications. One example plug-in that automatically adds MI.Schema
33 annotations to Workflow?
The metadata stored in the KG is subject to the same access control restrictions code and data. A user only sees the metadata associated with Projects or Datasets that they are otherwise allowed to access, including when performing global search. For more details on the KG, please see Section ] of the Supplementary Materials.
1 At the time of writing, with limited functionality.

---

4 Renku for Community Dataset Development
We believe that the feature-set of Renku and its design principles make it great platform for the development, dissemination and sharing of ML datasets and benchmarks. We envision that Renku will be used to enable new forms of collaboration on community dataset development and to facilitate the extension and reuse of different datasets and benchmarks based on reusable code and pipelines.
4.1 Transparency and trust
Responsible, sustainable development of datasets for use in the ML research community requires transparency. A dataset author, for example, desires that the downstream use of their data can quickly and easily be discovered. Conversely, dataset user or consumer needs to be able quickly examine the data and to understand its provenance to be absolutely certain about any assumptions, tweaks or modifications that may impact downstream interpretation. A data consumer may also like to discover uses of the data they are interested in, to gain inspiration and build on existing work easily.
In Renku, both of these points of view are satisfied through the integration of the four key platform elements (Code, Dataset, Workflows and Compute environments) with the metadata layer, the Renku knowledge graph. A Dataset author, for example, can use Renku tools to track how dataset is prepared from raw data to the final state that is distributed for wider use. The pipeline that generates the final Dataset is self-consiseently recorded and may be re-executed in the controlled compute environment to generate new datasets with different assumptions or to verify that the creation of the dataset itself is repeatable and reproducible. The ability to quickly work hands-on with a dataset with a pre-configured compute environment and executable code examples greatly simplifies the discovery and vetting process, inspiring trust and stimulating reuse and collaboration.
4.2 Dissemination and sharing
Dataset authors aspire to make their work easily accessible to the widest possible audience and applicable to a wide range of use cases. Dataset users, on the other hand wish to quickly discover datasets relevant to their research interests and use them with minimal friction. Renku provides authors with the means to package and distribute their Datasets easily and embed them into reproducible computational environments with sample code to demonstrate and document use. Documentation in this case goes beyond text descriptions, but can include code examples, visualizations or dashboards. This allows authors to build demonstrations of the dataset's application directly in their project; these can be open to the public and provisioned quickly.
Once a user finds dataset, they can add their own Renku project with a click. In doing so, the user does not need to worry about how to access (download) the data; the platform makes sure that the Dataset is accessible wherever the project is used. Data access information, whether the data is stored within the repository or in cloud-storage location, is preserved with the project ensuring that the data made available either in hosted interactive sessions or locally with minimal friction.
By making it easy for anyone to provide datasets for broader consumption in a frictionless way. Renku makes it possible expedite the dissemination of datasets within the community. Currently, handful of datasets are used by an overwhelmingly large percentage of papers [22]; not only because they are useful datasets, but also because they are made easily available, coupled with user-friendly tools for data handling, and demo projects and examples (e.g. the COCO dataset (25)). With Renku, it is straightforward to make the dataset available along with all the code needed to (re)create or modify it, bundled with a compute environment. This opens the door to more robustly inspect published datasets, and to use them as a starting point for generating new, complementary ones.
4.3 Identifying use and impact
By using a dataset within the platform, the information about the use is preserved within the knowledge graph. This makes it possible for dataset authors or members of the wider community to trace the usage of the data and self-consistently identify datasets that have the most traction within the community. The knowledge graph enables evaluation beyond the superficial "number of downloads" or "views": instead, we can analyze the network of downstream applications of a Dataset, e.g. whether they lead to results and publications, or whether they have impact across many

---

Platform
Versioning
Environments
Workflows
Datasets
Apps
Language
DATTSFLOW [31
Python
KG-FRUS 38
+
+
+
Python
MITox [T]
+
+
+
Python
TimeFRAME 9
+
+
+
+
R
MLED B5
Python
OADAT (29
Python
Table Summary of Renku features showcased in the example use case projects, These features are derived from the building blocks mentioned above, with versioning coming from the Code building block. The App feature allows deployment of project-specific Uls.
different institutions of the research community. This type of analysis and interpretation of dataset impact is not currently possible in other platforms made for data sharing. The same information can be used more specifically for community benchmarking purposes, as the Omnibenchmark project (26 demonstrates.
Apart from adding dimensions to assessing dataset applications, the ability to quickly identify dataset use in progress can have significant impact on collaboration within a community. The usual academic cycle of publication means that pieces of the research process that are fundamental to its success (c.g. datasets and benchmarks) are not recognized until the publications that use them are complete. This means that information only properly proliferates with a significant delay. Instead, in framework like Renku, information exchange and attribution could take place much sooner.
Example use cases
We have collected number of cases from real-world projects conducted the Swiss Data Science Center and the ETH domain, which illustrate the value of the Renku platform for collaborative dataset development, use, and reuse. The projects are related the development and application of ML in support of scientific research. In all cases, the challenge is to combine domain knowledge of the research question with specific requirements of downstream ML work. The case studies presented here bridge the gap from domain to data science and provide experts from both sides with the means to examine, confirm, and extend the datasets and their analyses, without extensive investments of time and infrastructure. Links to all projects are included in the citations. Further details about the TimeFRAME and MLED projects can be found in Section of the Supplementary Materials.
It is important to note that (at the time of writing) not all of the sample projects utilize all of Renku's functionality and that this is perfectly acceptable- depending on the stage of the project, researchers have different priorities to juggle. The summary of the adoption of various Renku features by the example projects can be found in Table The adoption curve the platform's features intended to be gradual once project is utilizing the platform, the "missing pieces" " (c.g. explicit workflows or data encapsulation through Datasets) can happen at any point when results start to take shape.
DATSSFLOW: ML-driven seismic signal classification The DATSSFLOW case study 31 deals with developing automated data parsing pipelines and automatic retraining of feature extraction models in a geophysical context. The project utilizes multi-step pipeline, expressed as set of Renku workflow steps, to convert publicly available raw seismic data into series Renku Datasets that may be reused in downstream tasks and applications. These steps are shown schematically in the right panel of Figure 2 the manifestation of the workflow, as seen via the KG is shown in the right panel.
All of the steps above are linked together: as soon as new data comes in, the entire pipeline is automatically re-executed and the feature extraction model updated. Renku significantly reduces the burden of manually running each and every script for each single dataset available, for current datasets but also from those coming from future campaigns. Data from more stations available from the International Federation Digital Seismograph Networks (FDSN) could easily added to the first Renku Workflow to adapt the entire pipeline different application. Each step is documented via instructions and READMEs, along with code, parametrizations and execution Workflows, enabling automatic updates to the full data pipeline, even for people not familiar with the project.
7

---

Legend
Acquire row data
TOML www.SMMedooiioiee www.oodooyy
fiw data
Code nle
codu, codu_
anguage liee sottoo of
date
Data jpreIprecosiing
Sodter exocution
pomhor Alencma
Preprocess
deta wmw.peeeccssegggg
takclllytinnn taonginpe you pesticing
Cleaned data
preprocessed data
Data-derived ML
comaining
Train
Tyshon AAAERONAIOO
Dotosets
heghhpiiutterrdrrr
Downstread task(s)
coolddio ho endte
dhortingg interatton
Features
(oottet Sesturee sufnenzodeccy
Apptiontio)sss
derived dataset(s)
Figure 2: DATSSFLOW conceptual data flow (left) and the automatically generated knowledge graph workflow representation (right).
KG-FRUS Renku has been utilized to track the creation of the KG-FRUS dataset a graph- based dataset built from the diplomatic documents of the Foreign Relations of the United States (FRUS) office. In [45], the authors advocate that a knowledge graph (KG) is the representation required approach the study of complex fields such diplomacy and political sciences, where synergies and interactions between international actors are extremely relevant, and which are hitherto ignored in standard corpora. As this dataset is based on public data dealing with public policy and world events, it is important to be transparent about its creation. Utilizing a system like Renku means that anyone check, reuse, alter and rerun the pipeline regenerate and check the consistency of the dataset, or even to produce their own dataset from similar data.
MLTox: Enhancing toxicological testing through machine learning Within the MLTox project, focused providing ML-based alternatives animal testing for ecotoxicological studies, the authors generated benchmark dataset, ADORE, on acute mortality of three taxonomic groups (fish, crustaceans and algae) 40). The core of ADORE originates from the ECOTOX database (28), which can be downloaded as a set of txt files. The relevant files were harmonized and pre- rreffilered to only contain entries suitable to model acute toxicity fish, crustaceans and algae. We expanded the core dataset with taxonomic and chemical information. The data was then filtered to only contain entries on acute mortality, which defined differently for the three taxonomic groups. Additionally, we only retain data points for which information from all sources is available. The filtered dataset mainly contains entries from organic molecules, representing rather small chemical space, which have been tested species with taxonomic information available.
In general, and for this dataset specifically, data splitting has a substantial impact the modeling output (20). We therefore proposed specific challenges on subsets of the data, included the corre- sponding data subsets with splittings as well as in-depth characterization and discussion of train-test splitting approaches [40]. The entire dataset, code, and pipeline needed to generate it are available on Renku [10]. Researchers interested in using or re-generating the dataset can easily do so within the RenkuLab environment. We envision the ADORE dataset being used to accelerate research in ecotoxicity to move towards reduction and finally a replacement animal testing.
Large scale optoacoustic imaging data OADAT is one the outcomes of a larger project, which proposed several methodologies to improve image quality by means of geometric distortion correction, noise reduction, and semantic segmentation, for novel emerging non-invasive clinical imaging modality called optoacoustic imaging. Beyond the methodologies, the project also
OADAT:

---

Platform
Versioning
Datasets
Environments
Provenance
Apps
Languages
Renku
Multiple
Whole Tale
+
Multiple
Hugging Face
+
Multiple
Kaggle
+
+
Multiple
DVC
+
N/A
Google Colab
+
Python
OpenML
Multiple
Papers with Code
+
N/A
Multiple Summary of platforms and tools overlapping with Renku features focusing support for version control, datasets, reproducible environments, data provenance, GUI apps, and program- ming languages
Binder
Table 2:
procured first of its kind large scale optoacoustics dataset consisting of experimental and simulated samples collected using varying imaging devices, conditions, and volunteers.
30 reference the Renku project[22) [9],, where one can not only access each component in the afore- mentioned workflow, but can also customize individual components (i) evaluate pretrained models on a different (proprietary) optoacoustic dataset, (ii) train the models on the datasets, (iii) evaluate performance of pretrained models, and even (iv) test the pretrained models on proprietary optoacous- tic images qualitatively to see if they can be integrated into other workflows of domain science (in this case optoacoustic imaging) experts. Each step designed to accept other models or datasets for efficient comparison and evaluation.
6 Comparison to other systems
Renku provides solutions to problems that are well known and widely acknowledged issues ML and data science more generally: there are many tools and platforms that overlap with Renku in addressing them. To provide better understanding of where Renku fits in the landscape, we give
brief comparison with some of the most relevant platforms: Whole Tale 4 Hugging Face [4], Kaggle DVC[I8), Google Colab OpenML 41 Papers with Code [I. and Binder 34 A summary is provided T
We want to emphasize that we do not see Renku as being in competition with any these platforms. Each one targets different set of usage scenarios, making it difficult to do direct comparison in a fair way. As we argue above, we see working sustainably as imperative to the development of machine learning and encourage researchers this goal. In the comparison here, the focus is on features, but other aspects like can deploy my own instance, is paid support available, is the platform operated for profit, can access GPUs may certainly be relevant.
use whatever tools they find helpful in achieving
Renku shares the most in common with the first three of the aforementioned platforms: Whole Tale, Hugging Face, and Kaggle. Like Renku, all these platforms support versioning code, defining datasets, and creating reproducible environments in the cloud. Of these platforms, Whole Tale and Renku are the two that support finer-grained tracking of data provenance (referred to as Workflows in Renku). Hugging Face and Renku let users deploy GUI-based apps that let others explore a project interactively without having to write or execute code.
Two projects that overlap in smaller subset of features with Renku are DVC and Google Colab. DVC a tool for Data Version Control, and supports tracking versions data, models, and provenance of results, features that Renku also supports. Like Renku, Hugging Face, and Kaggle, Google Colab offers based environments with optional GPU access. Colab is, however, tied to notebook- based Python environments and does not provide support for datasets.
The final three projects are the most orthogonal to Renku among this list. OpenML is a library and platform for accessing datasets and publishing detailed provenance information about how analysis was performed. Its feature set complements Renku very nicely, since Renku does not offer the same level of support for retrieving datasets and tasks or detail about how ML pipelines are built, but

---

Renku provides reproducible bloud-baseds environments and the ability to track provenance about other aspects of projects (like data pre-processing). Papers with Code is repository of publications. and code and datasets that go along with them. Publications are not central focus of Renku, Finally, Binder a tool that very conveniently allows creating cloud environment for running notebooks in Python, R, and Julia, but it is not designed to create environments for the development of MIL projects.
7 Current status, Limitations and Future work
As of this writing, the Renku platform runs in least seven different instances, including two universities in Switzerland and one used by the Swiss Federal Statistical Office. The public instance is open to all and can be found aittps://renkulal io It currently counts around 8*000 users. Apart from ML projects, some which we highlighted above, Renku users span a whole range of scientific domains (e.g. hydrology [27 24), climate science (42), and bioinformatics [26)), experience levels, and IT literacy. Many courses are taught on the platform, taking advantage of the easy set up of consistent, reusable compute environments.
The current version of the platform has focused on the usability of the CLI and the web app, the generation of the underlying metadata and its handling in production and the reusable compute environments, which are the most heavily used feature. Currently, the platform lacks the functionality for users to generate views of the knowledge graph casily (Section 3.4). although the metadata to enable such views already exists. Therefore, cross-project views or more complex queries relevant for discovering Datasets and their impact not easily accessible. We are working on features that will enable such exploration.
One of the main limitations of Datasets is the reliance on git-LFS the data storage backend inflexible, prone to error, and does not work well for large amounts of data.
(Section 3.T.. In our efforts to make the platform landing zone for all types of data science and ML projects, implementation work already in motion to simplify and extend the Dataset functionality, on the one hand to make it casier to connect to other sources of data, and the other to interface with data repositories and archives for preservation and minting of DOIs. Datasets also currently lack more extensive metadata, such licensing or publication information, which will be added in future iterations.
A common problem is availability and flexibility of compute resources, as users currently limited to using the on-line compute resources provided by the Renku instance they are using (Section 3.J). To better connect users to their data and compute resources, we plan to enable users to connect more seamlessly to other providers like public clouds or HPC clusters. At the same time, interactive computing platforms prone to not utilizing compute resources well and strive to implement features that help users avoid wasteful consumption.
We sincerely hope that the broader MI community will consider Renku as a platform for building, developing, and ultimately using and disseminating datasets. We look forward to engaging with questions, issues, and feature requests in our public channel
References
[1] Meta AI. Papers with Code Colel/ttpss//pppprssttttcoood com. 2023.
[2] Emily M. Bender, Timnit Gebru, Angelina McMillan-Major Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference Fairness, Accountability, and Transparency. FAccT '21, page 610-623, New York, NY, USA, 2021. Association for Computing Machinery.
[3] Barry Bozeman, Jan Youtie, and Jiwon Jung. Death by thousand 10-minute tasks: Workarounds and noncompliance in university research administration. Administration & Society, 53(4):527- 568, 2021.
'Dsccourse forum at attps:/renku Dot/SSssssDbttsccccceeeetttrrr/e
http:://rrrnuu.didsurseeeeee discourse. group main GitHub repository at https://github
10

---

[4] Adam Brinckman, Kyle Chard, Niall Gaffney Mihael Hategan, Matthew B Jones, Kacper Kowalik, Sivakumar Kulasckaran, Bertram Ludäscher, Bryce D Mecum, Jarek Nabrzyski, et al. Computing environments for reproducibility: Capturing the "whole tale". Future Generation Computer Systems, 94:854-860 867, 2019.
[5] Inha Cha, Juhyun Oh, Cheul Young Park, Jiyoon Han, and Hwalsuk Lee. Unlocking the tacit knowledge of data work machine learning. In Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems, CHI EA '23, New York, NY, USA, 2023. Association for Computing Machinery.
[6] Michael R. Crusoe, Sanne Abeln, Alexandru Iosup, Peter Amstutz, John Chilton, Nebojsa Tijanic, Hervé Ménager, Stian Soiland-Reyes and Carole A. Goble. Methods included: Stan- dardizing computational reuse and portability with the common workflow language. CoRR, abs/2105.07028, 2021.
[7] Andy Donald, Apostolos Galanopoulos, Edward Curry, Emir Muñoz, Ihsan Ullah, M. A. Waskow, Maciej Dabrowski, and Manan Kalra. Towards semantic approach for linked dataspace, model and data cards. In Companion Proceedings of the ACM Web Conference 2023, WWW '23 Companion, page 1468-1473 New York, NY, USA, 2023, Association for Computing Machinery.
[8] Benedikt Fecher and Sascha Friesike. Open Science: One Term, Five Schools of Thought, pages 17-47. Springer International Publishing, Cham, 2014.
[9] Philipp Fischer. Renku project: N20 Pathway haayysshtpp//reekuuuub.....ppppp
Tiachpai/n2o--paaaaaaaaalll
2023.
[10] Lili Gasser. Renku project: ADORE 2023.
[11] Timnit Gebru, Jamie Morgenstem, Briana Vecchione, Jennifer Wortman Vaughan, Hanna M. Wallach, Hal Daumé III, and Kate Crawford. Datasheets for datasets. CoRR, abs/1 0obs//803.. 803.09010, 2018.
[12] Amy K. Heger, Liz B. Marquis, Mihaela Vorvoreanu, Hanna Wallach, and Jennifer Wort- man Vaughan. Understanding machine learning practitioners data documentation perceptions, needs, challenges, and desiderata. Proc. ACM HHum.-Compl Interact., 6(CSCW2), nov 2022.
[13] Thomas J. Hostler. The Invisible Workload Open Research. Journal of Trial & Error, may 2023. http:///wmmlllrraaddmmcece//p/pp
[14] Inc. Hugging Face. Hugging Face http:::/hhuggiiggfacee
2023.
[15] Ben Hutchinson, Andrew Smart, Alex Hanna, Emily Denton, Christina Greer, Oddur Kjar- tansson, Parker Barnes, and Margaret Mitchell. Towards accountability for machine learning datasets: Practices from software engineering and infrastructure. CoRR, abs/2010.135611 2020.
[16] Matthew Hutson. Artificial intelligence faces reproducibility crisis. Science, 359(6377):725-
726, 2018
[17] Kaggle Inc. Kaggle http://wwwwwk.. kttgk:///ww. comm com
2023.
[18] Inc. Iterative. DVC httph: //tttpps//dvc.org 2023.
[19] Amruta Kale, Tin Nguyen, Jr. Harris, Frederick C., Chenhao Li, Jiyin Zhang, and Xiaogang Ma. Provenance documentation to enable explainable and trustworthy AI: literature review. Data Intelligence, 5(1):139-162, 03 2023.
[20] Sayash Kapoor and Arvind Narayanan. Leakage and the reproducibility crisis in machine- learning-based science. Patterns, page 100804, 2023.
[21] Thomas Kluyver, Benjamin Relln-Kelley, Fernando Pérez, Brian Granger, Matthias Bussonnier, Jonathan Frederic, Kyle Kelley, Jessica Hamrick, Jason Grout, Sylvain Corlay, Paul Ivanov, Damián Avila, Saña Abdalla, and Carol Willing. Jupyter notebooks publishing format for reproducible computational workflows. In F. Loizides and B. Schmidt, editors, Positioning and Power in Academic Publishing: Players, Agents and Agendas, pages 87 - 90. IOS Press, 2016.
11

---

[22] Bernard Koch, Emily Denton, Alex Hanna, and Jacob Gates Foster. Reduced, reused and recycled: The life of a dataset in machine learning research. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
[23] M. S. Krafezyk, A. Shi, A. Bhaskar, D. Marinov, and V. Stodden. Learning from reproducing computational results: Introducing three principles and the reproduction package. Philosoph- ical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 379(2197), May 2021.
[24] Louis Krieger, Remko Nijzink, Gitanjali Thakur, Chandrasekhar Ramakrishnan, Rok Roškar, and Stan Schymanski. Repeatable and reproducible workflows using the RENKU open science platform. In EGU General Assembly Conference Abstracts, EGU General Assembly Conference Abstracts, pages EGU21-7655, April 2021.
[25] Tsung- Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. CoRR, abs/140s 0tw//1405.00 0312, 2014.
[26] Almut Lütge, Anthony Sonrel, Izaskun Mallona, Charlotte Soneson, Federico Marini, Søren Hel- weg Dam, Oksana Riba Grognuz, Rok Roškar, and Mark D Robinson. "omnibenchmark: open continuous collaborative benchmarking of computational biology methods". To be available on bioarxiv upon submission, 2023.
[27] Remko Nijzink, Chandrasekhar Ramakrishnan, Rok Roškar, and Stan Schymanski. repeatable and reproducible modelling workflow using the Vegetation Optimality Model and RENKU, In EGU General Assembly Conference Abstracts, EGU General Assembly Conference Abstracts, page 9228, May 2020.
[28] Jennifer H. Olker, Colleen M. Elonen, Anne Pilli, Arne Anderson, Brian Kinziger, Stephen Erickson, Michael Skopinski, Anita Pomplun, Carlie A. LaLone, Christine L. Russom, and Dale Hoff. The ECOTOXicology Knowledgebase: Curated Database of Ecologically Relevant Toxicity Tests to Support Environmental Research and Risk Assessment. Environmental Toxicology and Chemistry, 4106))1520 2022. 17 citations (Crossref) [2023-05-23]
_eprint: http:///oiieiiiiiiiieeeimmmmm//p
[29] Firat Ozdemir. Renku project: OADAT htttps:/renkuulab.iiom
ozdemir/oadat--eeee evaluatee 2023.
[30] Firat Ozdemir, Berkan Lafci, Xose Luis Dean-Ben, Daniel Razansky, and Fernando Perez- Cruz. OADAT: Experimental and synthetic clinical optoacoustic data for standardized image processing. Transactions on Machine Learning Revearch, 2023.
[31] Patrick Paitz, Michele Volpi, and Francois Kamper. Renku project: DATSSFLOW https wwwp://aatiitpppeee
//renkulab. http:///poooooocppppppeeeeeeeeee jo rew.eeeeisii-c--e-ee-eeeeeeeeeee
2023.
[32] Kenneth Peng, Arunesh Mathur, and Arvind Narayanan. Mitigating dataset harms requires stewardship: Lessons from 1000 papers. In J. Vanschoren and S. Yeung, editors, Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1.
Curran, 2021.
[33] Gustavo Correa Publio, Diego Esteves, Agnieszka Lawrynowiez, Pance Panov, Larisa N. Soldatova, Tommaso Soru, Joaquin Vanschoren, and Hamid Zafar. MI-schema: Exposing the semantics of machine learning with schemas and ontologies. CoRR, abs/1807. 0053110511 2018.
[34] Benjamin Bagan-Kelley, Carol Willing, Akici, D Lippa. D Niederhut, and M Pacer. Binder 2.0-reproducible, interactive, sharable environments for science at scale. In Proceedings of the 17th python in science conference, pages 113-120. F. Akici, D. Lippa, D. Niederhut, and M. Pacer, eds., 2018.
[35] Carl Remlinger. Renku project:
MLED http:://eeiklaaa.ioprrjeeeteeeee
remlinger/mles 2023.
[36] Google Research. Google Colab
2023. 2023.
12

---

[37] Peter Rodgers and Andy Collings. Reproducibility in cancer biology: What have we learned? eLife, 10:e75830, dec 2021.
[38] Luis Salamanca. Renku project: KG-FRUS dallammna/kgg-free 2023.
httpas//reeaulaaaaoooooooeeettts
[39]
Nithya Sambasivan, Shivani Kapania, Hannah Highfill, Diana Akrong, Praveen K. Paritosh, and Lora Aroyo, "everyone wants to do the model work, not the data work": Data cascades in high-stakes AL In Yoshifumi Kitamura, Aaron Quigley, Katherine lsbister, Takeo Igarashi, Pernille Bjørn, and Steven Mark Drucker, editors, CHI '21: CHI Conference on Human Factors in Computing Systems, Viriual Event Yokohama, Japan, May 8-13, 2021, pages 39:1-39:15. ACM, 2021.
[40] Christoph Schür, Lilian Gasser, Fernando Perez-Cruz, Kristin Schirmer, and Marco Baity- Jesi. ADORE: Benchmark Dataset for Machine Learning Ecotoxicology. preprint, Bioinformatics, May 2023.
[41] Jan N Van Rijn, Bernd Bischl, Luis Torgo, Bo Gao, Venkatesh Umaashankar, Simon Fischer, Patrick Winter, Bernd Wiswedel, Michael R Berthold, and Joaquin Vanschoren. Openml: A collaborative science platform. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2013, Prague, Czech Republic, September 23-27, 2013, Proceedings, Part III 13, pages 645-649. Springer, 2013.
[42] F. Vitart, A. W. Robertson, A. Spring, F. Pinault, R. Roškar, W. Cao, S. Bech, A. Bienkowski, N. Caltabiano, E. De Coning, B. Denis, A. Dirkson, J. Dramsch, P. Dueben, J. Gierschendorf, H. S. Kim, K. Nowak. D. Landry, L. Lledó, L. Palma, S. Rasp, and S. Zhou. Outcomes of the WMO Prize Challenge to Improve Subseasonal to Seasonal Predictions Using Artificial Intel- ligence. Bulletin of the American Meteorological Sociery, 103(12):2888-E2885 2022.
December
[43] Mark D Wilkinson, Michel Dumontier, IJsbrnnd Jan Aalbersberg, Gabrielle Appleton, Myles Axton, Arie Baak, Niklas Blomberg, Jan-Willem Boiten, Luiz Bonino Silva Santos, Philip Bourne, et al. The fair guiding principles for scientific data management and stewardship. Scientific data, 3, 2016.
[44] Amy X Zhang, Michael Muller, and Dakuo Wang. How do data science workers collabo- rate? roles, workflows, and tools. Proceedings of the ACM Human Comaa-- Interaction, 4(CSCWWI)1-23, 2020.
[45] Gökberk Özsoy, Luis Salamanca, Matthew Connelly, Raymond Hicks, and Fernando Pérez-Cruz. KG-FRUS: Novel Graph-based Dataset of 127 years of US Diplomatic Relations. Submitted to NeurlPS Datasets and benchmarks track, 2023.
13


Reviews:
[{"review_id": "YsqedIRxTl", "reviewer": "Reviewer_1JZB", "summary": "", "questions": "", "limitations": "The authors address the limitations of their system as compared with other systems in Section 6 (see also \"Relation To Prior Work\" below).  \nWith regard to the handling of _data_ on their platform, the authors might want to consider additionally commenting on the following questions (primarily related to the public instance of RenkuLab):\n1. *Data privacy and data rights management:* How can dataset authors (or project authors more broadly) manage access rights to datasets (e.g., can they set different access levels or embargos for individual users, groups of users, or all users)? Would it be possible, e.g., to include private-by-default data in a public project, and requiring some form of clearance to obtain access to the private data?\n2. *Misbehavior:* What mechanisms does Renku have in place to police misbehavior on the public instance of RenkuLab (e.g., users sharing datasets that violate others' rights or datasets containing harmful content without the necessary guardrails)? ", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. Renku *addresses a long-standing and important practical problem* hindering the proliferation of open-science practices in research that relies heavily on data and computation.\n2. The platform is *highly relevant to the broader research community* because it can lower the cost of adhering to open-science standards and help data scientists from diverse domains improve their scientific practices.\n3. Renku is *well-designed* and builds on *state-of-the-art technologies*.\n4. By facilitating reproducibility, reuse, and collaboration in computational, data-intensive research, the platform can have a *positive social impact on and beyond the scientific community* and encourage *ethical behavior* (such as scientific attribution for non-standard contributions).", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "tFB7FdMSuS", "reviewer": "Reviewer_oZQs", "summary": "", "questions": "", "limitations": "Some limitations are mentioned in Section 7. It would be helpful to include more detail, eg include any limitations on “dataset tracking” (as this is a key contribution)  and perhaps mention certain features/blockers that have been repeatedly requested/mentioned by some of the ~8K users on the platform. Explicitly tying limitations to headings in Section in 3 would be good (eg best would be a short “limitations” section associated to each). \n\nPotential negative societal impact is not explicitly discussed as far as I can tell, and some reflections on that ought to be added, where possible. \n", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "A key contribution from a practical perspective is that the proposed platform offers to facilitate a researcher or data scientist’s ability to understand how a dataset was prepared and what changes it has undergone. This would greatly enhance a researcher’s ability to reproduce results, interpret new results, and enhance existing datasets. Various use-cases are described in the submission, with a high-level description in the submission, and extensive documentation and functionality online.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "u1A9APy6yW", "reviewer": "Reviewer_G2Nn", "summary": "", "questions": "", "limitations": "The NeurIPS paper checklist appears to be missing from this submission. It would be nice to have it included in the submission for completeness. There is also no discussion of potential negative societal impact in the paper (which would have been acknowledged in the checklist). Since the authors are providing a new service, it's difficult to see that there are any negative societal impacts, but I still encourage the authors to, at minimum, think this through and acknowledge that they’ve done so in the checklist.", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "Sections 1 and 2 are excellent reads with references to many important works showcasing the important challenges and issues ML faces with respect to dataset reproducibility and sustainability. The vision of the platform is admirable and well-researched, with the authors connecting their work to the FAIR principles, data documentation initiatives (e.g., Datasheets), and important data stewardship considerations, such as persistent identifiers for datasets (e.g., DOIs), citation, and licensing. Furthermore, the entire paper is very well written and well organized.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "wOsi4yCnTU", "reviewer": "Reviewer_GuX6", "summary": "", "questions": "", "limitations": "My comment here is the same as my “opportunities for improvement” comment. The authors also mention current technical limitations of the platform and some plans for future improvement.", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "The paper is straightforward to read and clearly explains the motivation for the platform. The overview of the platform design is general but gives a starting impression of how the platform is structured. The examples cover a variety of applications and highlight potentially useful features. Overall, the paper achieves what it sets out to do. The platform itself seems well-documented and useful for the purposes described above. ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: The paper introduces Renku, a platform by the Swiss Data Science Center, designed for reproducible, reusable, and collaborative data science. Renku comprises RenkuLab, a web-based application, and Renku Client, a command-line tool for managing code, data, and workflows. It addresses the need for a user-friendly toolset to implement open-data science principles. Renku enables interoperability between data and code, enhances transparency and version control for dataset preparation, and maintains machine-readable metadata for entities like Datasets and Workflows. The paper introduces the platform's vision, design, features, provides use-case examples, and outlines future developments. Overall, the reviewers have rated this paper highly and the authors have prepared thorough responses to reviewers' concerns in their rebuttal. The paper is certainly ready for publication.
Decision: Accept (Spotlight)