Open Science and Experimental
Research Lifecycle Management inSLICES-RI
Yuri Demchenko, University of Amsterdam
SLICES Roadshow Norway – 24 January 2023

Outline• SLICES Research Infrastructure for large scale experimental research• Open Science and Research Reproducibility• Experimental research lifecycle and Reproducibility as a Service• Experimental research reproducibility study in SLICES-DS/SLICES-PP• Data Management Infrastructure for full cycle experimental research• Variety and Volume of experimental data in SLICES• Future developments on experimental research reproducibility

SLICES Roadshow 2023

SLICES Experimental Research Reproducibility and DataManagement2

SLICES and Open Science• Open Science is a major initiative by EC and ESFRI• Being developed in many projects in H2020 and HE2027• Supported by a number of European e-Infrastructure services• FAIR (Findable, Accessible, Interoperable, Reusable) data principles commonly accepted for
managing research data

• A core objective of the European Open Science Cloud (EOSC) as a federated scientific datainfrastructure• SLICES will benefit and build on the current best practices, recommendations and tools,
use services provided by Open Science platforms• SLICES is actively involved in the EOSC activity• Starting from liaison with EOSC Working Groups activities in SLICES-DS to contribution to ongoing
EOSC projects in SLICES-PP• SLICES Interoperability Framework and services integration with EOSC and Open Science

SLICES Roadshow 2023

SLICES Experimental Research Reproducibility and DataManagement3

Open Science Challenges in Experimental Studies• SLICES is intended to support large-scale experimental studies on modern/future Digital
Infrastructure technologies• Multi-site, cross-domain, federated, experiment driven researcher/user centric• Scientific value of experimental research is in the reproducibility of experiments, sharing and(re)usability of data

• SLICES-RI brings its specific of implementing Open Science and FAIR data principles in experimental
studies on the Digital Infrastructure technologies• Important questions in experimenting with new technologies and industry is how open researchand experimental data should be• IPR and industrial secrets must be protected by Data Governance policies and enforcement• General infrastructure management data must be handled with responsibility• Compliance with the European Cybersecurity Assurance Act to be considered

SLICES Roadshow 2023

SLICES Experimental Research Reproducibility and DataManagement4Experimental Research Reproducibility: Main tasks• Experiment description and automation, including
reproducible description and experiment workflow
management• Experiment management infrastructure• Experimental data/metadata management and FAIR data
principles compliance• Federated Data Management Infrastructure to support
experimental research and SLICES infrastructure services
operation

SLICES Roadshow 2023

SLICES Experimental Research Reproducibility and DataManagement

3-stages process according
to ACM [ref]:Repeatability: Same team,same experiment setup
Reproducibility: Different
team, same experiment setupReplicability (portability):Different team, different
experiment setup

5

Experimental Research Reproducibility: Study inSLICES-DS• Reproducible experiment description and orchestration• Git and CI/CD iterative experiment design and automation and deployment• Jupyter Notebook experiment description and orchestration• Common Workflow Language (CWL) for experiment management• The plain orchestration service (pos) by Technical University Munich• Testbed management system and experiment workflow• Experiment infrastructure deployment and management• Cloud native tools using Git CI/CD tools (leveraging DevOps tools and methodology)• General infrastructure automation tools Ansible, Terraform, others• Cloud native Platform Research Infrastructure as a Service (PRIaaS) for fullinfrastructure, user and data services provisioning

SLICES Roadshow 2023

SLICES Experimental Research Reproducibility and DataManagement

6Experiment description: Reproducibility andPortability• GitHub and GitHub Actions (CI/CD tools)• Highly flexible but requires programming and full infrastructure management• However, can rely on well developed CI/CD tools• Jupyter Notebook (Python based) – Popular but limited portability• Very popular but often limited to specific experiment environment and infrastructure
platform• Common Workflow Language (CWL)• Portable Experiment Description• Requires workflow execution environment and infrastructure provisioning platform

SLICES Roadshow 2023

SLICES Experimental Research Reproducibility and DataManagement

7

Jupyter Notebook for Experiment Automation andWorkflow Description• Build on other projects experience of using Jupiter Notebooks for experiments
automation• Grid5000 large-scale infrastructure for experiment-driven research• Notebook as experiment drivers and experiment payload• Notebook for post-processing and exploratory programming• Fed4FIRE+ federation of experimental facilities for Future Internet research• Majority testbeds are using Notebooks• Chameleon (CHI Cloud++) OpenStack based cloud platform to support experimental
workflow for Computer Science systems research (US based)• Jupyter Notebook integration and experiments management via JupyterLab portal• Plain Orchestration Services (pos) by Technical University Munich (TUM)

SLICES Roadshow 2023

SLICES Experimental Research Reproducibility and DataManagement8

Common Workflow Language (CWL)• Provides portable platform independent data handling workflow description• YAML based• Requires workflow execution environment• Apache AirFlow, StreamFlow, Toil

• Galaxy workflow management and execution platform• galaxy.tools.cwl package for Galaxy open-source platform for FAIR data analysis• Run code in interactive environments (RStudio, Jupyter, ...) along with other tools orworkflows• Manage data by sharing and publishing results, workflows, and visualizations• Ensure reproducibility by capturing the necessary information to repeat and understand dataanalyses• Recognised as cross EOSC platform supporting FAIR data lifecycle

SLICES Roadshow 2023

SLICES Experimental Research Reproducibility and DataManagement9Example: Ansible playbook
and CWL workflow

Workflow designed in Galaxy

SLICES Roadshow 2023#!/usr/bin/env cwl-runnercwlVersion: v1.0
class: Workflow

file_to_sort: convert_to_csv/csv_file
sort_field:
default: 2 # which column to sort byout: [sorted_file]# The inputs of the workflow as a whole# These are referenced in the first workflow step
inputs:AWS_ACCESS_KEY_ID: string
AWS_SECRET_ACCESS_KEY: stringtable_name: string# the 4th step creates a description of the data
describe_data:
run: ../tools/describe-csv.cwl
in:# the input is the sorted CSV file from theprevious step
# In the following list the workflow steps are defined
csv_file: sort_csv/sorted_file
steps:
out: [data_description]# the first step, called "get_data" gets the sensor
data from the DynamoDB table# the 5th step generates a line plot
get_data:
generate_graph:
run: ../tools/get-dynamodb-data.cwl # the CWL tool
run: ../tools/graph-csv.cwl
is defined in this filein:# the following list defines the inputs to the CWL# the input is also the sorted CSV file from thetool
3rd stepin:
csv_to_plot: sort_csv/sorted_file
AWS_ACCESS_KEY_ID: AWS_ACCESS_KEY_ID
out: [plot]AWS_SECRET_ACCESS_KEY: AWS_SECRET_ACCESS_KEY
table_name: table_name# the outputs of the workflow as a whole are the sorted# the output of this workflow step is defined asCSV file from the third"dynamodb_data"# step, the data description from the 4th step and theout: [dynamodb_data]line chart from the 5th# step# the second step of the workflow converts the sensoroutputs:
data from JSON to CSV
data_csv:
convert_to_csv:
type: Filerun: ../tools/json-to-csv.cwl
outputSource: sort_csv/sorted_filein:description:# the input is the output of the previous step,
type: File"dynamodb_data"
outputSource: describe_data/data_description
json_file: get_data/dynamodb_data
plot:
out: [csv_file]
type: File
outputSource: generate_graph/plot# the third step sorts the sensor data in CSV format
sort_csv:
run: ../tools/sort.cwl
in:SLICES Experimental Research Reproducibility and DataManagement

10Experimental Research Reproducibility as aService• SLICES to support experiments reproducibility to comply with Open Science• Focus on repeatability and reproducibility with the future support ofreplicability• Robust, reproducible experiments• Documenting all relevant parameters and environment for experiments• Automate the documentation of experiments➢ Well-structured experiment workflow may serve as documentation• Benefits for research community• Reduce amount of work for experimenters to create reproducible experiments• Reduce amount of work for other researchers to recreate and re-run
experiments• Make reproducibility an integral part of experiment design➢ Automate entire experiment (setup, execution, evaluation)

SLICES Roadshow 2023

SLICES Experimental Research Reproducibility and DataManagement

Experimental researchstages• Experiment Planning• Experiment setup,Equipment configuration• Load (test) data
• Execute workflow• Collect data
• Evaluate and re-run
experiment if needed• Process/analyse data
• Produce report• Archive/publish data11

Experiment Workflow and StagesExperimental research
stagesExperiment Control and Monitoring

1. Planning

2. Experim
Setup
ExperimentLoop

9. Publication &Archiving

8. Reporting

3. Load Test
Data

4. Experim
Workflow

6. Evaluation(Store&Repeat)

5. CollectData

7.Processng(Data Analys)

1. Experiment Planning
2. Experiment setup,
Equipment configuration
3. Load (test) data
4. Execute workflow
5. Collect data
6. Evaluate and re-run
experiment if needed
7. Process/analyse data
8. Produce report
9. Archive/publish data

SLICES Data Management Infrastructure (supporting full research lifecycle)

SLICES Roadshow 2023

SLICES Experimental Research Reproducibility and DataManagement12

SLICES to provide the Robust Data Infrastructure
for Experiment/Data Driven Research• Experimental data are big, distributed, domain specific, serving specific communities• Require effective models and infrastructure services for Research Data Management and secure data sharing• Support the whole data lifecycle• Connected to research/experiment lifecycle or workflow• Distributed data storage and experimental data(set) repositories• Supporting recognized data interoperability standards (data formats and metadata)• Eventually certified: RDA endorsed Maturity and certification practice• Interoperability and integration with EOSC as Federated data infrastructure• Data management and data curation and quality assurance• FAIR data principles and SLICES metadata profiles (interoperable with EOSC)• Linked data and data discovery using semantic search and knowledge graph• PID (Persistent IDentifier) and FDO (FAIR Digital Object) infrastructure (interoperable with EOSC)• (Trusted) Data exchange and secure transfer protocols

SLICES Roadshow 2023

SLICES Experimental Research Reproducibility and DataManagement13

SLICES Experimental Data Lifecycle Model and Dataflow
Experiment/DataLifecycle Stages

ResearcherExperiment
Description

SLICES Data Storage and Management Infrastructure(multi-purpose: experimental, technological, archival)

Experiment
Plan &Implement

Data Collection& Preparation

Data Analysis
Model Building

Data Archiving& Publication/Sharing

Data Curation &Quality Assurance

Data Publishing andArchiving on EOSC
Data Sharing
Infrastructure

EOSC Data
Archiving &Sharing(Zenodo)Data Discovery (for experiment planning and operation)• Each Data Lifecycle stage – experiment, data collection, data analysis, and finally data archiving, works with own
data set, which must be linked.
• All data sets need to be stored and possibly re-used in later processes.
• Many experiments and research require already existing datasets that will be available in SLICES data repositories
or can be obtained/discovered in EOSC data repositories

SLICES Roadshow 2023

SLICES Experimental Research Reproducibility and DataManagement

14Different Types of Data for Different Experimental
Studies
General experimental studies and data:
Experiment data & Infra Mngt data

AI-centricDIs

Indus.
verticals
demandHumancentric DIs

Data for ML algorithmand optimisation

Design patterns, API management,configuration, evaluation data

Crossprop.
Cloud-toEdge
scalable
DIs6G5.
Models and Data for monitoring andoptimisation

Breaking down in priority research topics

SLICES Roadshow 2023

Simultaneous but progressive exploration of research topics

SLICES Experimental Research Reproducibility and DataManagement

15

Variety of Data produced in SLICES• General experimental studies and data documentation and publication• FAIR (Findable, Accessible, Interoperable, Reusable) data principles are key for experimental data sharing• Metadata profiles to be defined for major types of experiments and supported by data and metadatamanagement tools• Infrastructure management information to be recorded as experiments environment• Research Object (RO) and FAIR Digital Object (being developed by EOSC)• Data produced for AI/ML algorithms training for smart infrastructure optimisation and management(including energy efficiency, performance, resilience, sustainability)• Data modelling and data lineage (staging documenting)• AI/ML models serialization and portability• New Digital Infrastructure architecture elements and design patterns• Infrastructure and design patterns• Metadata for API description, identification, composability

SLICES Roadshow 2023

SLICES Experimental Research Reproducibility and DataManagement16

Data Management Infrastructure Layers
Data Management Infrastructure Layers to separate data management and governance concerns andactors/roles• Layer 4 - Experiment Infrastructure configuration and management• Layer 3 - Experimental data collection/recording• Data models, metadata• Layer 2 - Data processing• Data analysis, Process/ML models building, portability• Layer 1 - Data Storage, Archiving, Exchange• Datasets, metadata publication• Data Management Services and Tools (Data Management Plane)• Data Management Plan and Data Quality Assurance, FAIR compliance• Metadata registries and tools• Data Security and Data protection, GDPRSLICES Roadshow 2023

SLICES Experimental Research Reproducibility and DataManagement

17

Experimental Data Management Infrastructure
Researcher/PI(Project Team)
Data Management
Services/Tools

DGvP

DMP

Workflow(Exp&DataAnalys)

Metadata(Registry&Tools)
Data, PID, FDO

PortalVirtual Research Infrastructure(Virtual Research Teams)

Experiment Workflow
Management & Execution
AB

C

DLayers

Catalog

Visualisation

Experiment
MngmntEExperiment
Data Ingest

Experiment Data IngestData Processing
ML/Data Analytics

Data Storage, Compute, Network (cloud based)
ComputeVRIStorage

NetworkDataProcessing
Storage &Phy Infra

DMI layers• Virtual RI andResearcher Portal• Experiment
WorkflowManagement• Data Ingest• Data Processing• Storage & PhysicalInfrastructure

DGvP – Data Governance Policy DMP – Data Management Plan

SLICES Roadshow 2023

SLICES Experimental Research Reproducibility and DataManagement18

Further tasks for Experimental Research
Automation in SLICES-RI• Reproducible experimental research description and infrastructure provisioning tools• Platform RI as a Service (PRIaaS) for distributed experimental infrastructure provisioning for
virtual researcher teams• Adopting Research Object concept (by EOSC and Reliance project)• Federated multilayer experimental data management infrastructure• Experiment data collection, processing and storage• Data management policy definition and FAIR compliance• Metadata as cornerstone for reproducibility of experimental research• Metadata profiles definition, extension to support infrastructure management information CIM,MIB, GLUE schemas• EOSC compliance, interoperability and integration• Basis for the future cooperation with European RIs and contribution to EOSC development

SLICES Roadshow 2023

SLICES Experimental Research Reproducibility and DataManagement19

Questions and invitation tocooperationwww.slices-ri.eu

SLICES Roadshow 2023

SLICES Experimental Research Reproducibility and DataManagement20