SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Rok Roskar<sup>1</sup>, Chandrasekhar Ramakrishnan<sup>1</sup>, Michele Volpi<sup>1</sup>, Fernando Perez-Cruz<sup>1</sup>, Mohammad Alisafaee<sup>2</sup>, Philipp Fischer<sup>3</sup>, Lilian Gasser<sup>1</sup>, Eliza Jean Harris<sup>1</sup>, First Ozdemir<sup>1</sup>, Patrick Paitz<sup>3</sup>, Carl Remlinger<sup>2</sup>, Luis Salamanca<sup>1</sup>, Ralf Grubenmann<sup>1</sup>, Tasko Olevski<sup>1</sup>, Elisabet Capón García<sup>1</sup>, Lorenzo Cavazzi<sup>1</sup>, Jakub Chrobasik<sup>2</sup>, Andrea Cordoba<sup>1</sup>, Alessandro Degano<sup>2</sup>, Jimena Dupré<sup>1</sup>, Wesley Johnson<sup>1</sup>, Eike Kettner<sup>1</sup>, Laura Kinkead<sup>1</sup>, Seán Murphy<sup>1</sup>, Flora Thiebaut<sup>1</sup>, Olivier Verscheure<sup>1,2</sup>
1. Swiss Data Science Center, ETH Zürich, Zürich, Switzerland.
2. Swiss Data Science Center, EPFL, Lausanne, Switzerland.
3. Swiss Federal Institute for Forest, Snow, and Landscape Research, WSL, Birmensdorf, Switzerland
Data and code working together is fundamental to machine learning (ML), but the context around datasets and interactions between datasets and code are in general captured only rudimentarily. Context such as how the dataset was prepared and created, what source data were used, what code was used in processing, how the dataset evolved, and where it has been used and reused can provide much insight, but this information is often poorly documented. That is unfortunate since it makes datasets into black-boxes with potentially hidden characteristics that have downstream consequences. We argue that making dataset preparation more accessible and dataset usage easier to record and document would have significant benefits for the ML community: it would allow for greater diversity in datasets by inviting modification to published sources, simplify use of alternative datasets and, in doing so, make results more transparent and robust, while allowing for all contributions to be adequately credited. We present a platform, Renku, designed to support and encourage such sustainable development and use of data, datasets, and code, and we demonstrate its benefits through a few illustrative projects which span the spectrum from dataset creation to dataset consumption and showcasing.
Datasets are key to all aspects of machine learning (ML) and data science, from the development of novel algorithms to training of production models. Unfortunately, a variety of problems plague the generation, use, and dissemination of datasets [2, 39, 32]. The negative downstream effects of issues related to dataset preparation on model performance and application are well-documented [39]. Less well-documented are the difficulties in understanding and managing the deficiencies on the less technical aspects: the issues of attribution, reuse, dissemination, and replicability. Because ML research is so heavily dependent on datasets, [32] highlight the lacking and insufficiently robust mechanisms for dataset citation and usage tracking as core missing pieces needed to improve the health of the community as a whole. The lack of diverse datasets on which to advance the field has also recently been raised as a point of serious concern [22], and the lack of dataset provenance is a barrier to trustworthy AI systems [19].
Furthermore, beyond requiring planning and design [15], dataset creation requires a significant computational effort, which implies code, pipelines and compute resources. However, in the majority of cases, because dataset creation pipelines are treated as one-offs and are not recorded or published [12], researchers who want to augment a dataset or repeat its construction with different data cannot do so.
This significantly limits the engagement of the community in this arguably foundational phase of ML research [44]. The ML community has proposed and adopted some solutions specific to the nature of ML dataset creation and use, such as e.g. Datasheets for Datasets [11] (see also [7] for a recent reformulation). While these proposals raise awareness of the importance of datasets and the need for documentation and systematic rigor, they are largely prescriptive, not machine-readable, and cannot easily be automated or validated which makes them quickly become outdated.
Scientific research has struggled with these same issues for many years especially when it comes to valuation of datasets as legitimate research outputs and their citation and attribution. In the past decades, much attention has also been dedicated to the struggles of scientific disciplines to reproduce (sometimes very influential) results [37]. Similarly, the ML community is not immune to the issues of reproducibility [16]; in the case of datasets, where the introduction of hidden biases during the preparatory stages might critically influence the downstream chain of model training and production applications, the need for more comprehensive solutions is high. In this sense, it helps to lean on the concepts espoused by the Open Science movement, where technical questions about robustness, reproducibility and FAIR-ness (FAIR: Findable, Accessible, Interoperable, Reusable as proposed by [43]) overlap with and complement meta concerns about access to knowledge and transparency [8].
To work openly is to structure the research process in such a way that its outcomes can be reproduced, reused, and understood by anyone at any time. This implies structure, discipline, and adherence to norms, which in turn require significant cognitive, technical, monetary and temporal overhead. The cost of openness is a significant factor in preventing a wider adoption of these practices [13] despite enormous efforts by research institutions and funding agencies to exert top-down pressure through open science requirements and initiatives.
Openness, however, is not enough. We argue that sustainability is the more appropriate mental model, because it implies continuity and dynamics; i.e., simply "opening" a dataset says nothing about how it was assembled and created, where it is eventually used (outside of formal citations), how it might be repackaged, and the many downstream implications that various decisions at each stage of the process might have. The process of dataset creation is for the most part opaque and hidden behind qualitative descriptions in publications. Similarly, sharing source code does not guarantee that anyone can actually run it and, much less, obtain the expected results. Datasets and data are not the final products of the process; they are often deeply intertwined with the models themselves, both in publications and in the general discourse around ML applications. Therefore, the entire cycle must be considered: from dataset assembly and creation to their consumption resulting in eventual deployment of models for benchmarking or in production. While such robust documentation and systematic adherence to standards would be ideal, researchers are pragmatic; they want to get their work done and be recognized for it. In ML research, dataset development tends to be undervalued and is consequently not systematically documented [5]. Furthermore, when voluntary compliance with "best-practices" and efficiency are in competition for time, researchers will tend to optimize for efficiency [3]. In a sustainable system, the actions of individuals translate simultaneously into benefits for themselves (e.g. in the form of increased short-term efficiency or longer-term recognition) and the communities within which they operate (e.g. improved transparency) with little or no overhead.
We present Renku<sup>1</sup>, an open-source platform that aims to address some of these problems from another angle: through the creation of a more sustainable system within which ML research is conducted, by incentivising creators and consumers of datasets and models to work in ways that benefit both themselves and their communities. We describe the vision of the Renku platform and its practical implementation; we outline the impact this system can have on dataset development for ML applications; and we share several real-world use cases that demonstrate to varying degrees the practical manifestation of sustainability through the platform, with a specific focus on datasets, their preparation, processing, and dissemination. The use cases illustrate how the platform relieves data scientists and ML researchers from some basic sources of friction present in every data analysis project by: providing easy access to reproducible, containerized environments; simplifying shared data management; providing tooling for building reproducible pipelines; ensuring that data, code, and compute environment are versioned together and always self-consistent. This ecosystem, in turn, allows for the creation and dissemination of datasets, where code and pipelines that are essential for the creation of datasets can readily be inspected, scrutinized, shared, and rerun. The added
<sup>1</sup>https://renkulab.io, developed at the Swiss Data Science Center, a strategic focus area of the ETH domain with offices at ETH Zürich, EPFL, and the Paul Scherrer Institute (PSI)
transparency of such a system provides the missing piece to make dataset creation a collaborative, community exercise.
2 Platform vision
The Renku platform's vision is to facilitate sustainable data science. In such a practice, individuals work in a knowledge-capturing way to collaborate, discover, reuse and reproduce results easily.
To motivate this idea, it is helpful to draw a parallel to version control as a tool for sustainable software development. Version control contributes to making software development sustainable by helping developers capture and describe changes to the codebase over time. This allows them to return to previous points in time, run the code as it was then, understand motivations for the changes made to the code, and explore implementation alternatives without breaking systems in production. We believe that tools are needed in ML to support similar ways of working: tools that make it possible to return to a project at a previous point in time, understand how the project evolved, and allow for exploration of alternatives while supporting reuse.
The benefits of working in a sustainable data science environment are manifold. For example, one can easily return to a project as it was in an earlier state, and still successfully run the code; one can inspect a colleague's project and pick up where they left off; a group lead can quickly identify the uses of a shared dataset and actively scrutinize the results immediately, with zero setup overhead. On the group level, the captured knowledge can serve to facilitate exchange among colleagues, reuse of data and techniques, and to offer members a quick, actionable overview of their group's activities. On the community level, this knowledge capture leads to improved transparency, enhances trust in results, and identifies all artifacts essential to the end-to-end process, such as data pipelines and datasets.
Our vision is, therefore, to build a system that offers simple solutions to problems that data scientists, researchers, and ML users face every day, and in doing so inject just enough structure to make their work more sustainable. The platform can then be used to propagate the information of what is being done with data and how on a variety of levels and scales. By providing researchers with solutions that alleviate common points of friction (e.g. access to efficient compute and storage), increasing productivity and efficiency, while at the same time capturing knowledge along the whole life-cycle, the Renku platform makes it possible to record the progress of research as it happens.
3 Platform design and features
Given the above vision of enabling sustainable data science, the platform is designed around a few basic concepts: (i) an integrated solution for Code, Datasets, Workflows, and Computational environments; (ii) simple interfaces for consistent compute and data access; (iii) building on existing solutions and avoiding vendor lock-in; (iv) connecting everything through a comprehensive, flexible knowledge graph. This section is deliberately brief and omits much detail; further information can be found by visiting the live projects on https://renkulab.io (see links in citations).
3.1 Building blocks
Renku makes opinionated choices about a large technology stack to provide the basic building blocks of any data science-related project: Code, Datasets, Workflows, and Computational environments (Figure 1).
Code: Modern data analysis does not happen without code; the problem of working collaboratively with code has largely been solved by the invention of decentralized version control systems like git. Renku does not try to reinvent the wheel and relies fully on this industry standard.
Datasets: Renku encourages users to organize their data via Datasets, which can be thought of as annotated data containers, i.e. lightweight encapsulations of data, useful for a project together with the relevant metadata. Their true value, however, is that they are an easy way to share and incorporate data into projects either from other Renku projects or external data repositories (e.g. Dataverse or Zenodo), while fully preserving the relevant metadata. If a Dataset is published and receives a DOI, this can be tracked in Renku as well. Datasets can evolve as new data becomes available; this
Figure 1: A summary of the components of Renku.
provenance is also self-consistently traced in Renku, whether a derived dataset is created by the original author or someone else entirely.
Data can be stored using different backends. The default is to version data together with the code, but store it in git Large File Storage (LFS). Users can also opt to connect a Dataset to an external data source, such as an Amazon Web Services Simple Storage Service (S3) bucket or Azure blob storage $^2$ . Renku can handle the minutiae of bringing the data to the user's system (including access credential management) so that after the initial setup they no longer have to worry about how to obtain the data. External sources currently do not include databases; while projects may retrieve data from databases through scripts, and store results of queries in files, Datasets cannot currently use databases as data sources directly.
Renku supports ways of restricting access to data. For data stored in git LFS within a Renku instance, the data access is limited to those who have access to the repository. If data is external, access is managed by a third-party and not by Renku. The infrastructure that hosts the compute resources for running workflows and interactive sessions is designed to be multi-tenant and prevents users from accessing resources allocated to others. While this should be sufficient for many kinds of sensitive data, some highly-sensitive or confidential data may require additional precautions on the data at rest, like encryption, which Renku does not itself provide. In those cases, we recommend that a Renku instance is deployed in a secure environment and urge users to consider their levels of acceptable risk to avoid potential leakage of sensitive data. More information about deployment and administration can be found in Section 3 of the Supplementary Materials.
**Workflows:** Workflows are where code meets data; a workflow can be anything from a shell script to a full-blown distributed pipeline. Renku provides a very simple workflow system which captures the basic relationship between data and any code that can be executed on the command line. The primary function of Renku workflows is to capture provenance and make it possible to reproduce and rerun computational pipelines involving several intertwined steps that would normally run as separate scripts, or even as a part of one (usually very long) script. The system aims to relieve the researcher from the burden of remembering how a specific result was generated.
Once a workflow is executed, content hashes of all components are recorded in the metadata and can be checked against any future state. If any of the pieces are updated (code or data), Renku can generate an execution plan to rerun the parts that need to be updated. The workflows are recorded using an open ontology based on W3C recommendations<sup>3</sup> and schema.org but can be serialized into a workflow language to execute on a variety of backends including cloud and HPC. Currently, we
${}^{2}$  Integrations are not limited to these services and additional backends are continuously being added.
3https://www.w3.org/TR/prov-o/
support the Common Workflow Language[6], toil, and Argo workflows<sup>4</sup>, but others can be added by external extensions via a simple plug-in system.
Computational environment: Code requires a computational environment in which to run; this environment includes software libraries and languages, compilers, system libraries, and the operating system. Failure to encapsulate this environment to a sufficient degree is the primary hurdle for collaboration, reuse and reproducibility in computational research [23]. Renku makes heavy use of container technologies to alleviate some of the pain related to computational environments. By default, each project includes a Dockerfile that builds a docker image for every commit in the project's repository. This ensures that a computational environment matching any commit of the project is always available. Users can either use these images on their local machine (or another compute resource) or launch hosted interactive sessions directly in RenkuLab, the hosted part of the platform.
3.2 Interfaces for consistent compute and data access
The Renku platform consists of a RenkuLab instance (e.g. our public instance at https://renkulab.io) with a browser-based interface and a Python library, which also provides a command-line interface  $(\mathrm{CLI})^5$ . The two interfaces can be used together or independently, depending on the project constraints. We strive for feature parity between the two interfaces, meaning that users should have a consistent experience with launching compute sessions and accessing data in both cases. The CLI in addition allows users to take a project completely off-line to another compute resource, without relying on any hosted (cloud) infrastructure. This is particularly useful in cases involving sensitive data or restricted environments.
3.3 Building on Open Source
Renku is open source<sup>6</sup> and licensed under the Apache 2.0 License. It builds and relies on a number of well-known open-source technologies: Kubernetes at the infrastructure layer, docker for containerization, git and GitLab for version control and repository management, and the Jupyter ecosystem [21] for interactive sessions. The platform is designed to prevent vendor lock-in, i.e. users are not obliged to keep working on their project within the hosted platform or even using any Renku tooling. They may simply take their Project (a git repository) anywhere and use it with standard tools like git and docker.
3.4 Connect Everything: the Renku Knowledge Graph
A central component of the Renku platform is the knowledge graph (KG) made up of metadata aggregated from all the projects on the platform. All primary Renku entities (Projects, Code, Datasets, Workflows) are represented in the KG. This metadata can facilitate many different kinds of functionality. For example, recording a workflow allows a user to use the system to re-execute parts of their computation if some upstream data changes. Conversely, if a user is interested in a particular Dataset, they can use the KG to understand how it has been created or analyzed in the past, who has experience with the data and what kinds of results have been derived from it. The KG is based on RDF and designed to be interoperable and extensible by 3rd party plug-ins with domain-specific metadata to facilitate use cases across a broad spectrum of applications. One example is a plug-in that automatically adds MLSchema [33] annotations to Workflows<sup>7</sup>.
The metadata stored in the KG is subject to the same access control restrictions as code and data. A user only sees the metadata associated with Projects or Datasets that they are otherwise allowed to access, including when performing a global search. For more details on the KG, please see Section 1 of the Supplementary Materials.
At the time of writing, with limited functionality.
<sup>5</sup>https://github.com/SwissDataScienceCenter/renku-python
<sup>6</sup>https://github.com/SwissDataScienceCenter/renku
<sup>7</sup>https://github.com/SwissDataScienceCenter/renku-msl
4 Renku for Community Dataset Development
We believe that the feature-set of Renku and its design principles make it a great platform for the development, dissemination and sharing of ML datasets and benchmarks. We envision that Renku will be used to enable new forms of collaboration on community dataset development and to facilitate the extension and reuse of different datasets and benchmarks based on reusable code and pipelines.
4.1 Transparency and trust
Responsible, sustainable development of datasets for use in the ML research community requires transparency. A dataset author, for example, desires that the downstream use of their data can quickly and easily be discovered. Conversely, a dataset user or consumer needs to be able to quickly examine the data and to understand its provenance to be absolutely certain about any assumptions, tweaks or modifications that may impact downstream interpretation. A data consumer may also like to discover uses of the data they are interested in, to gain inspiration and build on existing work easily.
In Renku, both of these points of view are satisfied through the integration of the four key platform elements (Code, Dataset, Workflows and Compute environments) with the metadata layer, the Renku knowledge graph. A Dataset author, for example, can use Renku tools to track how a dataset is prepared from raw data to the final state that is distributed for wider use. The pipeline that generates the final Dataset is self-consistently recorded and may be re-executed in the controlled compute environment to generate new datasets with different assumptions or to verify that the creation of the dataset itself is repeatable and reproducible. The ability to quickly work hands-on with a dataset with a pre-configured compute environment and executable code examples greatly simplifies the discovery and vetting process, inspiring trust and stimulating reuse and collaboration.
4.2 Dissemination and sharing
Dataset authors aspire to make their work easily accessible to the widest possible audience and applicable to a wide range of use cases. Dataset users, on the other hand wish to quickly discover datasets relevant to their research interests and use them with minimal friction. Renku provides authors with the means to package and distribute their Datasets easily and embed them into reproducible computational environments with sample code to demonstrate and document use. Documentation in this case goes beyond text descriptions, but can include code examples, visualizations or dashboards. This allows authors to build demonstrations of the dataset's application directly in their project; these can be open to the public and provisioned quickly.
Once a user finds a dataset, they can add it to their own Renku project with a click. In doing so, the user does not need to worry about how to access (download) the data; the platform makes sure that the Dataset is accessible wherever the project is used. Data access information, whether the data is stored within the repository or in a cloud-storage location, is preserved with the project ensuring that the data is made available either in hosted interactive sessions or locally with minimal friction.