Experimental Research Reproducibilityand Experiment Workflow
ManagementYuri Demchenko, University of Amsterdam
on behalf ofSebastian Gallenmüller (Technical University of Munich, Germany), Serge
Fdida (Sorbonne University, France), Panayiotis Andreou (University of CentralLancashire), Mathias Kirkeng (UvA)

Outline• SLICES Research Infrastructure for large scale experimental research• Open Science and Research Reproducibility• Experimental research lifecycle and Reproducibility as a Service• Experimental research reproducibility study in SLICES-DS/SLICES-PP• pos – plain orchestration service• Data Management Infrastructure for full cycle experimental research• Variety and Volume of experimental data in SLICES• Future developments on experimental research reproducibility

TASIR Wsh 2023, COMSNETS2023

Experimental Research Reproducibility

2

SLICES-RI: Scientific Large-scale Infrastructure for
Computing/ Communication Experimental Studies• SLICES is a distributed Digital Infrastructure to support large-scale experimental
research focused on networking protocols, radio technologies, services, datacollection, parallel and distributed computing• SLICES will integrate multiple experimental facilities and testbeds operated bypartners providing a common services access and integration platform• SLICES will allow academics and industry to experiment and test the wholespectrum of digital technologies• Experiment automation to support design, experiment, and operate the full research
lifecycle management.

TASIR Wsh 2023, COMSNETS2023

Experimental Research Reproducibility

3

SLICES and Open Science: Challenges• SLICES is intended to support large-scale experimental studies on modern/future Digital
Infrastructure technologies• Multi-site, cross-domain, federated, experiment driven researcher/user centric• SLICES-RI brings its specific of implementing Open Science and FAIR data principles inexperimental studies on the Digital Infrastructure technologies• Scientific value of experimental research is in the reproducibility of experiments, sharing and(re)usability of data
• Important questions in experimenting with new technologies and industry is how open researchand experimental data should be• IPR and industrial secrets must be protected by Data Governance policies and enforcement• General infrastructure management data must be handled with responsibility

TASIR Wsh 2023, COMSNETS2023

Experimental Research Reproducibility

4Experimental Research Reproducibility: Mainaspects
3-stages process according toACM [ref]:1.Repeatability: Same teamexecutes experiment usingsame setup
2.Reproducibility: Different teamexecutes experiment usingsame setup
3.Replicability: Different teamexecutes experiment using
different setup• Experiment description and automation, including
reproducible description and experiment workflow
management• Experimental infrastructure services operation• Experimental data and metadata management• Federated Data Management Infrastructure to supportcomplex experimental research• FAIR data principles compliance (core for Open Science)[ref] https://www.acm.org/publications/policies/artifact-review-and-badging-curr

TASIR Wsh 2023, COMSNETS2023

Experimental Research Reproducibility

5Experimental Research Reproducibility as aService• SLICES to support experiments reproducibility to comply with Open Science• Focus on repeatability and reproducibility with the future support ofreplicability• Robust, reproducible experiments• Documenting all relevant parameters for experiments• Automate the documentation of experiments➢ Well-structured experiment workflow may serve as documentation• Benefits for research community• Reduce amount of work for experimenters to create reproducible experiments• Reduce amount of work for other researchers to recreate and re-run
experiments• Make reproducibility an integral part of experiment design➢ Automate entire experiment (setup, execution, evaluation)

TASIR Wsh 2023, COMSNETS2023

Experimental Research Reproducibility

Experimental researchstages• Experiment Planning• Experiment setup,Equipment configuration• Load (test) data
• Execute workflow• Collect data
• Evaluate and re-run
experiment if needed• Process/analyse data
• Produce report• Archive/publish data6

Experimental Research Reproducibility: Study in
SLICES-DS/SLICES-PP• Reproducible experiment description and orchestration• Git and CI/CD iterative experiment design and automation and deployment• Jupyter Notebook experiment description and orchestration• Common Workflow Language (CWL) for experiment management• Experiment infrastructure deployment and management• Cloud native tools using Git CI/CD tools (leveraging DevOps tools and methodology)• General infrastructure automation tools Ansible, Terraform, others• The plain orchestration service (pos) by Technical University Munich• Testbed management system and experiment workflow• Cloud native Platform Research Infrastructure as a Service (PRIaaS) for fullinfrastructure, user and data services provisioning

TASIR Wsh 2023, COMSNETS2023

Experimental Research Reproducibility

7Experiment description: Reproducibility andPortability• GitHub and GitHub Actions (CI/CD tools)• Highly flexible but requires programming and full infrastructure management• However, can rely on well developed CI/CD tools• Jupyter Notebook (Python based) – Popular but limited portability• Very popular but often limited to specific experiment environment andinfrastructure platform• Common Workflow Language (CWL)• Portable Experiment Description• Requires workflow execution environment and infrastructure provisioning platform

TASIR Wsh 2023, COMSNETS2023

Experimental Research Reproducibility

8Jupyter Notebook for Experiment Automation andWorkflow Description• Build on other projects experience of using Jupiter Notebooks forexperiments automation• Grid5000 large-scale infrastructure for experiment-driven research• Notebook as experiment drivers and experiment payload• Notebook for post-processing and exploratory programming• Notebooks as tutorials• Fed4FIRE+ federation of experimental facilities for Future Internet research• Majority testbeds are using Notebooks• Chameleon (CHI Cloud++) OpenStack based cloud platform to support
experimental workflow for Computer Science systems research (US based)• JupyterNotebook integration and experiments management via JupyterLab portal• Plain Orchestration Services (pos) by Technical University Munich (TUM)

TASIR Wsh 2023, COMSNETS2023

Experimental Research Reproducibility

9Plain Orchestration Service (pos)• The plain orchestrating service (pos) provides two components:• Testbed controller and Experiment workflow• The testbed controller takes care of the allocation and management ofexperimental resources• It provides bare-metal access to the experiment nodes• Images for the experiment nodes are provided in the form of live Linux images• Using live images for experiments has two benefits:• First, rebooting an experiment node helps reset the system to a well-defined state.
• Second, testbed users are aware of the non-permanence of their configuration, gently
pushing users towards documenting and automating experiment configuration.

TASIR Wsh 2023, COMSNETS2023

Experimental Research Reproducibility

10pos Experiment Workflow Management• Setup phase• Controller manages experiment• Controller configures experiment nodes (DuT, LoadGen)• Global/local variables (vars) parametrize setup• Measurement phase• Repeated execution of measurement script• Loop variables to parameterize each set of measurement run, e.g.,changing packet rates data in each run is connected to a specific setof loop vars

• Evaluation phase• Collected results/loop vars used for experiment evaluation• Automated experiment release (git repository, website)Structured Experiment Workflow with pos

TASIR Wsh 2023, COMSNETS2023

Experimental Research Reproducibility

11Common Workflow Language (CWL)• Provides portable platform independent data handling workflow description• YAML based• Requires workflow execution environment• Apache AirFlow• StreamFlow• Toil• Galaxy workflow management and execution platform• galaxy.tools.cwl package for Galaxy open-source platform for FAIR data analysis• Run code in interactive environments (RStudio, Jupyter, ...) along with other tools or workflows• Manage data by sharing and publishing results, workflows, and visualizations• Ensure reproducibility by capturing the necessary information to repeat and understand data analyses• Recognised as cross EOSC platform supporting FAIR data lifecycle

TASIR Wsh 2023, COMSNETS2023

Experimental Research Reproducibility

12

Example: Ansible playbook
and CWL workflow

Workflow designed in Galaxy

TASIR Wsh 2023, COMSNETS2023#!/usr/bin/env cwl-runnercwlVersion: v1.0
class: Workflow

file_to_sort: convert_to_csv/csv_file
sort_field:
default: 2 # which column to sort byout: [sorted_file]# The inputs of the workflow as a whole# These are referenced in the first workflow step
inputs:AWS_ACCESS_KEY_ID: string
AWS_SECRET_ACCESS_KEY: stringtable_name: string# the 4th step creates a description of the data
describe_data:
run: ../tools/describe-csv.cwl
in:# the input is the sorted CSV file from theprevious step
# In the following list the workflow steps are defined
csv_file: sort_csv/sorted_file
steps:
out: [data_description]# the first step, called "get_data" gets the sensor
data from the DynamoDB table# the 5th step generates a line plot
get_data:
generate_graph:
run: ../tools/get-dynamodb-data.cwl # the CWL tool
run: ../tools/graph-csv.cwl
is defined in this filein:# the following list defines the inputs to the CWL# the input is also the sorted CSV file from thetool
3rd stepin:
csv_to_plot: sort_csv/sorted_file
AWS_ACCESS_KEY_ID: AWS_ACCESS_KEY_ID
out: [plot]AWS_SECRET_ACCESS_KEY: AWS_SECRET_ACCESS_KEY
table_name: table_name# the outputs of the workflow as a whole are the sorted# the output of this workflow step is defined asCSV file from the third"dynamodb_data"# step, the data description from the 4th step and theout: [dynamodb_data]line chart from the 5th# step# the second step of the workflow converts the sensoroutputs:
data from JSON to CSV
data_csv:
convert_to_csv:
type: Filerun: ../tools/json-to-csv.cwl
outputSource: sort_csv/sorted_filein:description:# the input is the output of the previous step,
type: File"dynamodb_data"
outputSource: describe_data/data_description
json_file: get_data/dynamodb_data
plot:
out: [csv_file]
type: File
outputSource: generate_graph/plot# the third step sorts the sensor data in CSV format
sort_csv:
run: ../tools/sort.cwl
in:Experimental Research Reproducibility

13

SLICES to provide the Robust Data Infrastructure
for Experiment/Data Driven Research• Experimental data are big, distributed, domain specific, serving specific community• Require effective models and infrastructure services for Research Data Management and secure data
sharing• Support the whole data lifecycle• Connected to research/experiment lifecycle or workflow• Distributed data storage and experimental data(set) repositories• Supporting recognized data interoperability standards (data formats and metadata)• Eventually certified: RDA endorsed Maturity and certification practice• Interoperability and integration with EOSC as Federated data infrastructure• Data management and data curation and quality assurance• FAIR data principles and SLICES metadata profiles (interoperable with EOSC)• Linked data and data discovery using semantic search and knowledge graph• PID (Persistent IDentifier) and FDO (FAIR Digital Object) infrastructure• (Trusted) Data exchange and secure transfer protocols

TASIR Wsh 2023, COMSNETS2023

Experimental Research Reproducibility

14SLICES Experimental Data Lifecycle Model and Dataflow
SLICES Data Storage and Management Infrastructure

Experiment/DataLifecycle Stages(multi-purpose: experimental, technological, archival)

Experiment
Plan &Implement

Researcher

Experiment
Description

Data Collection& Preparation

Data Analysis
Model Building

Data Archiving& Publication/Sharing

Data Curation &Quality Assurance

Data Publishing andArchiving on EOSC
Data Sharing
Infrastructure

EOSC DataArchiving &Sharing

Data Discovery (for experiment planning and operation)• Each Data Lifecycle stage – experiment, data collection, data analysis, and finally data archiving, works with own
data set, which must be linked.
• All data sets need to be stored and possibly re-used in later processes.
• Many experiments and research require already existing datasets that will be available in SLICES data repositories
or can be obtained/discovered in EOSC data repositories

TASIR Wsh 2023, COMSNETS2023

Experimental Research Reproducibility

15

Variety of Data produced in SLICES• General Digital Infrastructure experimental studies and data documentation andpublication• Metadata profiles to be defined for major types of experiments and supported by data andmetadata management tools• Infrastructure management information to be recorded as experiments environment• FAIR (Findable, Accessible, Interoperable, Reusable) data principles are key for experimental
data sharing• Data produced for AI/ML algorithms training for smart infrastructure optimisation andmanagement (including energy efficiency, performance, resilience, sustainability)• Data modelling and data lineage (staging documenting)• AI/ML models serialization and portability• New Digital Infrastructure architecture elements and design patterns• Metadata for API description, identification, composability• Research Object (RO) and FAIR Digital Object (being developed by EOSC)TASIR Wsh 2023, COMSNETS2023

Experimental Research Reproducibility

16Different Types of Data for Different Experimental
Studies
General experimental studies and data:
Experiment data & Infra Mangt data

AI-centricDIs

Indus.
verticals
demandHumancentric DIs

Data for ML algorithm
and optimisation

Design patterns, API management,
configuration, evaluation data

Crossprop.
Cloud-toEdge
scalableDIs

6G

Models and Data for monitoring andoptimisation

SLICES-RI priority research topics

TASIR Wsh 2023, COMSNETS2023

Simultaneous but progressive exploration of research topics

Experimental Research Reproducibility

17

Virtual RI

Develop &Operate

Datasets,Archives

RI Users/Researchers

Virtual RI and User Applications
Data Source

RI Portal

PRIaaS Architecture Model (2021 in progress)
Actualisation Platform Components [ref]Data Target• Core Infrastructure Services (IaaS & PaaS)

Portal, Catalog, FedID, Policy• Data Services

Federated Access Infrastructure & Tenant Management

Actualisation Platform• Management and Operation
Dev Env & Tools• DevOps (CI/CD)• Templates &Design Patterns• API & Container• DataOps/MLOps
Management &Operation• Service Catalog &Lifecycle Mngnt• Orchestration(multi-lyr&domain)• Auto & Optimisat(AI enabled)

Service Prov &Fulfilment• SLA Mng
• Policy Provision• User ID Provision
Infrastructure Services(P/Iaas)• Compute• Storage• Network• IoT & Edge
• Monitoring• AAI & Federation• Security (VPC,VPN, PKI & Trust)• Blockchain Perm

Virtualised
Resorces

PRIaaS
Components
VirtualisatedResources

CCI 18 Nov 2020

Data Services• Directory• Metadata, PID• Sci Data Lakes(Semantic)• Data Ingest/ELT• Data Exchange(Data Marts)• Provenance• FAIR & QualityAssurance• Privacy• ML/AI & DA
Datasets & Arch• Research• Technological• Development Environment and Tools• DevOps• Templates and Patterns• Service Provisioning and Fulfilment• Datasets and Archives• Federated Access Infrastructure + IoT Edge andTenants Management• Virtual RIs and Portal

Composable Infrastructure Resource
ComputeStorage

Network[ref] IG1157 Digital Platform Reference Architecture Concepts andPrinciples v5.0.1, 21 July 2020

Physical resource (distributed, multi-provider)

SLICES-DS Project and Activities

18

Further tasks for Experimental Research
Automation in SLICES-RI• Reproducible experimental research description and infrastructure tools• PRIaaS for distributed experimental infrastructure provisioning• Experiment and data management• Virtual researcher teams support and federated identity management (user provisioning andaccess management)• Metadata as a cornerstone for reproducibility of experimental research• Metadata profiles definition• EOSC compliance, interoperability and integration• Basic for future cooperation

TASIR Wsh 2023, COMSNETS2023

Experimental Research Reproducibility

19

Questions and invitation tocooperationwww.slices-ri.eu

TASIR Wsh 2023, COMSNETS2023

Experimental Research Reproducibility

20