Scalable Environments for reproducible Open Science
Creators: name: Sipos
name: Fernandez
Publisher: name: Forschungzentrum Jülich
Year: 2022
Subjects: egi-ace eosc open science reproducibility notebooks jupyter binder datahub

description: doi:10.5281/ZENODO.6420897
description: These are the slides of the presentation given at the Mini Gateways 2022 event on behalf of the EGI-ACE project. Event programme: https://sciencegateways.org/minigateways2022-program PRESENTATION ABSTRACT The EGI Federation [1] is an international e-Infrastructure that provides advanced computing and data analytics services for research and innovation. While dominantly based in Europe, the EGI Federation delivers compute, data and authorization-authentication services with related user support and training from over 200 resource centres in 50 countries worldwide. Since January 2021 the EGI Federation leads the ‘EGI Advanced Compute Services for the European Open Science Cloud (EGI-ACE)’ European Commission co-funded project [2]. EGI-ACE delivers the Compute Platform for the European Open Science Cloud (EOSC CP). The EOSC CP is a system of federated compute and storage facilities, complemented by diverse access, data management and compute platform services for Open Science. The EOSC CP is designed to support a wide range of scientific data processing and analysis use cases, including reproducible data intensive sciences. This use case primarily relies on the EGI Notebooks and EGI Binder services that are both build on well-known open source technologies, customized for international big data communities and for members of the long tail of science. EGI Notebooks [3] is based on the Jupyter technology, offering a browser-based tool for interactive data analysis. EGI Notebooks provides users with computational notebooks where they can combine text, mathematics, computations and rich media output. EGI Notebooks is a multi-user service with dedicated service instances for each user community, with such a instance shared among all the members of one community. EGI Notebooks service instances are hosted on the EGI Cloud [4], i.e. on OpenStack sites that participate in the EGI federation, sharing common single sign-on, availability/reliability monitoring, Virtual Machine image catalogue, usage accounting, security incident response team, and user helpdesk. EGI Notebooks provides the following custom features compared to other Jupyter systems: Integration with EGI Check-in, allowing login with EduGAIN or social accounts (e.g. Google, Facebook). Persistent storage and scalable compute capacity allocated for each user at a per community level. Access to distributed cloud storage from the Notebooks via the EGI DataHub service [5]. Customisable with community libraries and programming languages (e.g. MATLAB). EGI Binder [5] allows the re-creation of custom computing environments from a single link, to reproduce notebooks. EGI Binder builds on BinderHub, an open source tool that allows to build docker images from a Git repository and then make it available from JupyterHub. EGI Binder was customized by EGI for scientific communities similarly to the Notebooks service. EGI Notebooks and Binder enable the following reproducibility process (See Figure 1.): Users can define their big data analysis in the EGI Notebooks service. Once a notebook is ready for publishing, it can be shared in a GitHub repository. Users can use the Zenodo-GitHub integration for generating DOIs for a notebook. The DOI can be cited in publications for discovery in research papers by fellow researchers. The GitHub link or the Zenodo DOI can be used to reproduce the same computational analysis in EGI Binder. The analysis can be re-run and advanced in Binder, paving the way for new discoveries. ACKNOWLEDGEMENT The work presented in this abstract received funding through the EGI-ACE project from the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 101017567. REFERENCES EGI – Advanced Computing Services for Research: www.egi.eu EGI-ACE - Advanced Computing for EOSC project: https://www.egi.eu/projects/egi-ace/ EGI Notebooks service: https://www.egi.eu/services/notebooks/ EGI Cloud Compute service: https://www.egi.eu/services/cloud-compute/ EGI DataHub service: https://www.egi.eu/services/datahub EGI Binder service: https://binder.notebooks.egi.eu/