SlowGuess/NeurIPS_2023_OCR huggingface dataset:
By making it easy for anyone to provide datasets for broader consumption in a frictionless way, Renku makes it possible to expedite the dissemination of datasets within the community. Currently, a handful of datasets are used by an overwhelmingly large percentage of papers [22]; not only because they are useful datasets, but also because they are made easily available, coupled with user-friendly tools for data handling, and demo projects and examples (e.g. the COCO dataset [25]). With Renku, it is straightforward to make the dataset available along with all the code needed to (re)create it or modify it, bundled with a compute environment. This opens the door to more robustly inspect published datasets, and to use them as a starting point for generating new, complementary ones.
4.3 Identifying use and impact
By using a dataset within the platform, the information about the use is preserved within the knowledge graph. This makes it possible for dataset authors or members of the wider community to trace the usage of the data and self-consistently identify datasets that have the most traction within the community. The knowledge graph enables evaluation beyond the superficial "number of downloads" or "views"; instead, we can analyze the network of downstream applications of a Dataset, e.g. whether they lead to results and publications, or whether they have an impact across many
Platform	Versioning	Environments	Workflows	Datasets	Apps	Language
DATTSFLOW [31]	+	+	+	+	-	Python
KG-FRUS [38]	+	+	+	+	-	Python
MLTox [10]	+	+	+	+	-	Python
TimeFRAME [9]	+	+	+	-	+	R
MLED [35]	+	+	-	-	-	Python
OADAT [29]	+	+	-	-	-	Python
Table 1: Summary of Renku features showcased in the example use case projects. These features are derived from the building blocks mentioned above, with versioning coming from the Code building block. The App feature allows deployment of project-specific UIs.
different institutions of the research community. This type of analysis and interpretation of dataset impact is not currently possible in other platforms made for data sharing. The same information can be used more specifically for community benchmarking purposes, as the Omnibenchmark project [26] demonstrates.
Apart from adding dimensions to assessing dataset applications, the ability to quickly identify dataset use in progress can have significant impact on collaboration within a community. The usual academic cycle of publication means that pieces of the research process that are fundamental to its success (e.g. datasets and benchmarks) are not recognized until the publications that use them are complete. This means that information only properly proliferates with a significant delay. Instead, in a framework like Renku, information exchange and attribution could take place much sooner.
5 Example use cases
We have collected a number of use cases from real-world projects conducted at the Swiss Data Science Center and the ETH domain, which illustrate the value of the Renku platform for collaborative dataset development, use, and reuse. The projects are related to the development and application of ML in support of scientific research. In all cases, the challenge is to combine domain knowledge of the research question with specific requirements of downstream ML work. The case studies presented here bridge the gap from domain to data science and provide experts from both sides with the means to examine, confirm, and extend the datasets and their analyses, without extensive investments of time and infrastructure. Links to all projects are included in the citations. Further details about the TimeFRAME and MLED projects can be found in Section 4 of the Supplementary Materials.
It is important to note that (at the time of writing) not all of the sample projects utilize all of Renku's functionality and that this is perfectly acceptable - depending on the stage of the project, researchers have different priorities to juggle. The summary of the adoption of various Renku features by the example projects can be found in Table 1. The adoption curve of the platform's features is intended to be gradual - once a project is utilizing the platform, the "missing pieces" (e.g. explicit workflows or data encapsulation through Datasets) can happen at any point when results start to take shape.
DATSSFLOW: ML-driven seismic signal classification The DATSSFLOW case study[31] deals with developing automated data parsing pipelines and automatic retraining of feature extraction models in a geophysical context. The project utilizes a multi-step pipeline, expressed as a set of Renku workflow steps, to convert publicly available raw seismic data into a series of Renku Datasets that may be reused in downstream tasks and applications. These steps are shown schematically in the right panel of Figure 2; the manifestation of the workflow, as seen via the KG is shown in the right panel.
All of the steps above are linked together: as soon as new data comes in, the entire pipeline is automatically re-executed and the feature extraction model updated. Renku significantly reduces the burden of manually running each and every script for each single dataset available, for current datasets but also from those coming from future campaigns. Data from more stations available from the International Federation of Digital Seismograph Networks (FDSN) could easily be added to the first Renku Workflow to adapt the entire pipeline to a different application. Each step is documented via instructions and READMEs, along with code, parametrizations and execution Workflows, enabling automatic updates to the full data pipeline, even for people not familiar with the project.
Figure 2: DATSSFLOW conceptual data flow (left) and the automatically generated knowledge graph workflow representation (right).
KG-FRUS Renku has been utilized to track the creation of the KG-FRUS dataset[38], a graph-based dataset built from the diplomatic documents of the Foreign Relations of the United States (FRUS) office. In [45], the authors advocate that a knowledge graph (KG) is the representation required to approach the study of complex fields such as diplomacy and political sciences, where synergies and interactions between international actors are extremely relevant, and which are hitherto ignored in standard corpora. As this dataset is based on public data dealing with public policy and world events, it is important to be transparent about its creation. Utilizing a system like Renku means that anyone can check, reuse, alter and rerun the pipeline to regenerate and check the consistency of the dataset, or even to produce their own dataset from similar data.
MLTox: Enhancing toxicological testing through machine learning Within the MLTox project, focused on providing ML-based alternatives to animal testing for ecotoxicological studies, the authors generated a benchmark dataset, ADORE, on acute mortality of three taxonomic groups (fish, crustaceans and algae) [40]. The core of ADORE originates from the ECOTOX database [28], which can be downloaded as a set of .txt files. The relevant files were harmonized and pre-filtered to only contain entries suitable to model acute toxicity in fish, crustaceans and algae. We expanded the core dataset with taxonomic and chemical information. The data was then filtered to only contain entries on acute mortality, which is defined differently for the three taxonomic groups. Additionally, we only retain data points for which information from all sources is available. The filtered dataset mainly contains entries from organic molecules, representing a rather small chemical space, which have been tested on species with taxonomic information available.
In general, and for this dataset specifically, data splitting has a substantial impact on the modeling output [20]. We therefore proposed specific challenges on subsets of the data, included the corresponding data subsets with splittings as well as in-depth characterization and discussion of train-test splitting approaches [40]. The entire dataset, code, and pipeline needed to generate it are available on Renku [10]. Researchers interested in using or re-generating the dataset can easily do so within the RenkuLab environment. We envision the ADORE dataset being used to accelerate research in ecotoxicity to move towards a reduction and finally a replacement of animal testing.
OADAT: Large scale optoacoustic imaging data OADAT is one of the outcomes of a larger project, which proposed several methodologies to improve image quality by means of geometric distortion correction, noise reduction, and semantic segmentation, for a novel emerging non-invasive clinical imaging modality called optoacoustic imaging. Beyond the methodologies, the project also
Platform	Versioning	Datasets	Environments	Provenance	Apps	Languages
Renku	+	+	+	+	+	Multiple
Whole Tale	+	+	+	+	-	Multiple
Hugging Face	+	+	+	-	+	Multiple
Kaggle	+	+	+	-	-	Multiple
DVC	+	+	-	+	-	N/A
Google Colab	+	-	+	-	-	Python
OpenGL	-	+	-	+	-	Multiple
Papers with Code	-	+	-	-	-	N/A
Binder	-	-	+	-	-	Multiple
Table 2: Summary of platforms and tools overlapping with Renku features focusing on support for version control, datasets, reproducible environments, data provenance, GUI apps, and programming languages
procured a first of its kind large scale optoacoustics dataset consisting of experimental and simulated samples collected using varying imaging devices, conditions, and volunteers.
[30] reference the Renku project[29], where one can not only access each component in the aforementioned workflow, but can also customize individual components to (i) evaluate pretrained models on a different (proprietary) optoacoustic dataset, (ii) train the models on the datasets, (iii) evaluate performance of pretrained models, and even (iv) test the pretrained models on proprietary optoacoustic images qualitatively to see if they can be integrated into other workflows of domain science (in this case optoacoustic imaging) experts. Each step is designed to accept other models or datasets for efficient comparison and evaluation.
6 Comparison to other systems
Renku provides solutions to problems that are well known and widely acknowledged issues in ML and data science more generally: there are many tools and platforms that overlap with Renku in addressing them. To provide a better understanding of where Renku fits in the landscape, we give a brief comparison with some of the most relevant platforms: Whole Tale [4], Hugging Face[14], Kaggle[17], DVC[18], Google Colab[36], OpenML [41], Papers with Code[1], and Binder[34]. A summary is provided in Table 2.
We want to emphasize that we do not see Renku as being in competition with any of these platforms. Each one targets a different set of usage scenarios, making it difficult to do a direct comparison in a fair way. As we argue above, we see working sustainably as imperative to the development of machine learning and encourage researchers to use whatever tools they find helpful in achieving this goal. In the comparison here, the focus is on features, but other aspects like can I deploy my own instance, is paid support available, is the platform operated for profit, can I access GPUs may certainly be relevant.
Renku shares the most in common with the first three of the aforementioned platforms: Whole Tale, Hugging Face, and Kaggle. Like Renku, all these platforms support versioning code, defining datasets, and creating reproducible environments in the cloud. Of these platforms, Whole Tale and Renku are the two that support finer-grained tracking of data provenance (referred to as Workflows in Renku). Hugging Face and Renku let users deploy GUI-based apps that let others explore a project interactively without having to write or execute code.
Two projects that overlap in a smaller subset of features with Renku are DVC and Google Colab. DVC is a tool for Data Version Control, and supports tracking versions of data, models, and provenance of results, features that Renku also supports. Like Renku, Hugging Face, and Kaggle, Google Colab offers cloud-based environments with optional GPU access. Colab is, however, tied to notebook-based Python environments and does not provide support for datasets.
The final three projects are the most orthogonal to Renku among this list. OpenML is a library and platform for accessing datasets and publishing detailed provenance information about how analysis was performed. Its feature set complements Renku very nicely, since Renku does not offer the same level of support for retrieving datasets and tasks or detail about how ML pipelines are built, but
Renku provides reproducible cloud-based environments and the ability to track provenance about other aspects of projects (like data pre-processing). Papers with Code is a repository of publications, and code and datasets that go along with them. Publications are not a central focus of Renku. Finally, Binder is a tool that very conveniently allows creating a cloud environment for running notebooks in Python, R, and Julia, but it is not designed to create environments for the development of ML projects.
7 Current status, Limitations and Future work
As of this writing, the Renku platform runs in at least seven different instances, including two at universities in Switzerland and one used by the Swiss Federal Statistical Office. The public instance is open to all and can be found at https://renkulab.io. It currently counts around 8'000 users. Apart from ML projects, some of which we highlighted above, Renku users span a whole range of scientific domains (e.g. hydrology [27, 24], climate science [42], and bioinformatics [26]), experience levels, and IT literacy. Many courses are taught on the platform, taking advantage of the easy set up of consistent, reusable compute environments.
The current version of the platform has focused on the usability of the CLI and the web app, the generation of the underlying metadata and its handling in production and the reusable compute environments, which are the most heavily used feature. Currently, the platform lacks the functionality for users to generate views of the knowledge graph easily (Section 3.4), although the metadata to enable such views already exists. Therefore, cross-project views or more complex queries relevant for discovering Datasets and their impact are not easily accessible. We are working on features that will enable such exploration.
One of the main limitations of Datasets is the reliance on git-LFS as the data storage backend (Section 3.1). It is inflexible, prone to error, and does not work well for large amounts of data. In our efforts to make the platform a landing zone for all types of data science and ML projects, implementation work is already in motion to simplify and extend the Dataset functionality; on the one hand to make it easier to connect to other sources of data, and on the other to interface with data repositories and archives for preservation and minting of DOIs. Datasets also currently lack more extensive metadata, such as licensing or publication information, which will be added in future iterations.
A common problem is availability and flexibility of compute resources, as users are currently limited to using the on-line compute resources provided by the Renku instance they are using (Section 3.1). To better connect users to their data and compute resources, we plan to enable users to connect more seamlessly to other providers like public clouds or HPC clusters. At the same time, interactive computing platforms are prone to not utilizing compute resources well and we strive to implement features that help users avoid wasteful consumption.
We sincerely hope that the broader ML community will consider Renku as a platform for building, developing, and ultimately using and disseminating datasets. We look forward to engaging with questions, issues, and feature requests in our public channels<sup>8</sup>.
[1] Meta AI. Papers with Code https://paperswithcode.com, 2023.
[2] Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? . In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT '21, page 610-623, New York, NY, USA, 2021. Association for Computing Machinery.
[3] Barry Bozeman, Jan Youtie, and Jiwon Jung. Death by a thousand 10-minute tasks: Workarounds and noncompliance in university research administration. *Administration & Society*, 53(4):527-568, 2021.
$^{8}$ Discourse forum at https://renku.discourse.group, main GitHub repository at https://github.com/SwissDataScienceCenter/renku
[37] Peter Rodgers and Andy Collings. Reproducibility in cancer biology: What have we learned? eLife, 10:e75830, dec 2021.
[38] Luis Salamanca. Renku project: KG-FRUS https://renkulab.io/projects/luis.salamanca/kg-frus, 2023.
[39] Nithya Sambasivan, Shivani Kapania, Hannah Highfill, Diana Akrong, Praveen K. Paritosh, and Lora Aroyo. "everyone wants to do the model work, not the data work": Data cascades in high-stakes AI. In Yoshifumi Kitamura, Aaron Quigley, Katherine Isbister, Takeo Igarashi, Pernille Bjørn, and Steven Mark Drucker, editors, CHI '21: CHI Conference on Human Factors in Computing Systems, Virtual Event / Yokohama, Japan, May 8-13, 2021, pages 39:1-39:15. ACM, 2021.
[40] Christoph Schür, Lilian Gasser, Fernando Perez-Cruz, Kristin Schirmer, and Marco Baity-Jesi. A Benchmark Dataset for Machine Learning in Ecotoxicology. preprint, Bioinformatics, May 2023.
[41] Jan N Van Rijn, Bernd Bischl, Luis Torgo, Bo Gao, Venkatesh Umaashankar, Simon Fischer, Patrick Winter, Bernd Wiswedel, Michael R Berthold, and Joaquin Vanschoren. Openml: A collaborative science platform. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2013, Prague, Czech Republic, September 23-27, 2013, Proceedings, Part III 13, pages 645-649. Springer, 2013.
[42] F. Vitart, A. W. Robertson, A. Spring, F. Pinault, R. Roškar, W. Cao, S. Bech, A. Bienkowski, N. Caltabiano, E. De Coning, B. Denis, A. Dirkson, J. Dramsch, P. Dueben, J. Gierschendorf, H. S. Kim, K. Nowak, D. Landry, L. Lledó, L. Palma, S. Rasp, and S. Zhou. Outcomes of the WMO Prize Challenge to Improve Subseasonal to Seasonal Predictions Using Artificial Intelligence. Bulletin of the American Meteorological Society, 103(12):E2878-E2886, December 2022.
[43] Mark D Wilkinson, Michel Dumontier, IJsbrand Jan Aalbersberg, Gabrielle Appleton, Myles Axton, Arie Baak, Niklas Blomberg, Jan-Willem Boiten, Luiz Bonino da Silva Santos, Philip E Bourne, et al. The fair guiding principles for scientific data management and stewardship. Scientific data, 3, 2016.
[44] Amy X Zhang, Michael Muller, and Dakuo Wang. How do data science workers collaborate? roles, workflows, and tools. Proceedings of the ACM on Human-Computer Interaction, 4(CSCW1):1-23, 2020.
[45] Gökberk Özsoy, Luis Salamanca, Matthew Connelly, Raymond Hicks, and Fernando Pérez-Cruz. KG-FRUS: a Novel Graph-based Dataset of 127 years of US Diplomatic Relations. Submitted to NeurIPS - Datasets and benchmarks track, 2023.