Future Scientific Data Infrastructure:
Towards Platform Research Infrastructure as aService (PRIaaS)

Dr. Yuri Demchenko
University of Amsterdam
BDAA2020 Symposium, HPCS2020
February 2021 (Virtual)

HPCS2020 BDAA2020

Platform RI as a Service (PRIaaS)

1

Outline• Background for this research– European Research Infrastructure– European Open Science Cloud (EOSC)• Research Infrastructure development and enabling technologies– Digitalisation, AI and 5G technologies– Leveraging Platform concept for RI platform model– TMForum Digital Platform Reference Architecture (DPRA)• Proposed Platform RI as a Service Architecture (PRIaaS)• SLICES-DS Project and PRIaaS implementation• Discussion

HPCS2020 BDAA2020

Platform RI as a Service (PRIaaS)

2

Background for this research• Project– EGEE – Enabling Grids for E-science– EGI – European Grid Initiative– ENVRI - Environmental Research Infrastructure– EOSC FAIRsFAIR– SLICES-DS• Standardisation and Best Practices– NIST Big Data Architecture– RDA – Research Data Alliance, support Research DataManagement best practices

HPCS2020 BDAA2020

Platform RI as a Service (PRIaaS)

3

European Research Area and Initiatives••

European Research Area (ERA) is an important area of the European policy
development and funding
Research Infrastructure (RI) is one of pillar in support of European science– Coordinated by ESFRI (European Strategy Forum on Research Infrastructures)• ESFRI Roadmap is published bi-annually to start new call for RI process, since 2006

– More than 54 European RIs are listed in the EU HLEG study “Supporting theTransformative Impact of Research Infrastructures on European Research”, 2020
• 34 distributed and 9 single sited serving EU research community– More than 1800 RIs operating in Europe•

RIs are supported by e-Infrastructure programme funded by EU Horizon
2020 and next Horizon Europe– Providing common integration platform for individual RIs and other research andindustry domains••

ERA and ESFRI supports international cooperation and research
European Open Science Cloud (EOSC) initiatives to create a common
platform for European RI integration

HPCS2020 BDAA2020

Platform RI as a Service (PRIaaS)

4

ESFRI Research Infrastructure Domains
•
•
•
•
•
•

Social and cultural Innovation
Environment
Health and Food
EnergyPhysical Science and Engineering
New domain included: DIGIT – Digital and Information
technologies– To integrate modern digital and data technologies• Former e-Infrastructure projects: GEANT, EGEE, EGI• ESFRI Roadmap 2018 http://roadmap2018.esfri.eu/media/1066/esfri-roadmap-2018.pdf
HPCS2020 BDAA2020

Platform RI as a Service (PRIaaS)

5

European Open Science Cloud (EOSC)•

EOSC is an overarching concept and framework to integrate existing RIs andfacilitate information and data exchange between RI, organisations andresearchers– First phase 2016-2020 with funded projects 2016-2022••

53 projects in total

EOSC main projects and co-creation activities– EOSCpilot – Initial EOSC architecture and requirements– EOSChub – Technical integration platform, RI marketplace and API/services
directory– EOSCsecretariat https://www.eoscsecretariat.eu/• Establishment of the Governance structure and EU EOSC association to be cofunded by EC and Member States (MS) Co-creation model and European OpenScience Commons•

Built on experience of the past successful initiatives and project– EGEE and WLCG, EGI, RDA (co-founded by EC and NSF), GEANT/TERENA•

Provides a model experience for future EU initiatives, such as GAIA-X
European Federated Data Cloud

HPCS2020 BDAA2020

Platform RI as a Service (PRIaaS)

6

EOSC Status: Minimum Viable EOSC and EOSC
Core
• Architecture defining (infrastructure) components
•
•
•
•
•

Data Access framework
PID framework and service
FAIR data enabling services
Service Management and Access framework
Authentication, Authorisation Interoperability Framework• Policy and Governance• Shared Open Science policy framework• A minimum legal metadata framework as
part of the FAIR compliance framework• An open metrics framework

Demand for modern RI platformusing recent development byindustry (for future technology
exchange)
EOSC challenges: to incorporate
recent technology development
into ERA• Portal providing web access to the EOSC services• EOSC is an important stage in the European RI integration
CCI 18 Nov 2020

SLICES-DS Project and Activities

7

Timeline European RI evolution••

Based on the authors first hand experience
SLICES-RI project positioning and intended contribution

HPCS2020 BDAA2020

Platform RI as a Service (PRIaaS)

8

From EOSC-1 to EOSC-2: Four Technology Aspects
2016-2018

RI Type

EOSC-1

2020-2022

EOSC-2 (future)

Definition

Interoperable Federated(European) RI, FAIR RI

Virtualised Pan-European RI platform as aService and Ecosystem (PRIaaS)

Network &Compute

•
•
•
•
•
•
•

Distributed scalable computing
Cloud based Big Data technologies
High performance networks
5G technologies, wireless access
IoT sensor networks
Portal and Services Catalog
Industry standards and IDSA
adoption• Composable virtualized RI provisioning on demand (including
for services integration)• Common federated computing and networking
platform/environment, enabling virtual RIs• Cloud based and cloud enabled• DevOps and AI enabled services• Digital Twins• Interoperability and Integration with Industry infrastructure(e.g. IDSA+, Industrial Internet)

Data Infra

•
•
•
•

FAIR data
Data Factories and PID
Metadata registries
Interoperable/common DataManagement model• Fully adopted FAIR principles, extended to ontologies• Semantically enabled scientific data lakes, common
vocabularies• Secure/trusted data exchange (data markets)• Full data value chain supported (cross-domain)

Security• Federated Identity Management,
Federated Access Control• Automated identity provisioning

Infra
Managnt
Technolog• Integrated Operation andAutomation

HPCS2020 BDAA2020

•
•
•
•
•

Beyond2025

Zero trust security, Trust Bootstrapping
Homomorphic encryption and data processing
Quantum ready encryption, Quantum enabled key management
Federated Identity Management, Federated Access Control
Automated identity provisioning• Fully automated RI and services provisioning, management andoperation• Optimisation of infrastructure and operation• DevOps
AI enabled(re-usable design patterns)
Platform
RI as aandService(PRIaaS)

9

Technology Trends for Science Transformation• Science and industry digitalisation make easy exchange oftechnologies, solutions, application• Hyperconverged Infrastructure model– Cloud based infrastructure integrations– Benefitting from global cloud infrastructure and cloud nativetechnologies• Transformation effect of Artificial Intelligence and Machine
Learning technologies• Data Management and Governance as an important asset• 5G and Telecom cloud– Infrastructure deployment and operation using cloud nativetechnologies

HPCS2020 BDAA2020

Platform RI as a Service (PRIaaS)

10

Transformational Role of Artificial Intelligence• Extending possibilities of research when working with Big Data
• Automating data preparation, processing, and analysis•

AI enabled data management• Smart infrastructure and tools operation and management• AI driven and Machine Learning powered scientific discovery anddecision support, digital models creation (Digital Twins)• AI powered self-learning assistant to a researcher/scientist capable ofcreating domain related intelligence; many research questions will be
pursued semi-automatically• Role of data will change: the learned model will replace data; theory
becomes data for next generation AI– AI require reliable data infrastructure, with new metadata model•

EU and US studies– AI for Science: Report on the Department of Energy (DOE) Town Halls on Artificial
Intelligence (AI) for Science, 2019 [online] https://www.anl.gov/ai-for-science-report– AI for Science, by Barbara Helland, AI for Science Town Hall, Oct 2019 [online]
https://science.osti.gov/-/media/ber/berac/pdf/201910/Helland_BERAC_Oct2019.pdf

HPCS2020 BDAA2020

Platform RI as a Service (PRIaaS)

11

Data Management and FAIR Data Principles• Importance of proper data management and data quality– Re-usable research data
– Reference data sets for ML and AI algorithms• Explainable AI– Digital Twins in industry and science• Depend on data quality and proper data management flow

• FAIR data principles– Findable – Accessible – Interoperable - Reusable– One of major coordination area and co-creation in EOSC
– GO FAIR Initiative - https://www.go-fair.org/• GO FAIR Implementation network• Internet of FAIR Data and Services
HPCS2020 BDAA2020

Platform RI as a Service (PRIaaS)

12

Promises of 5G and Value for RI&Science• 5G main use cases (or usage scenarios) that can be adopted
by the FutureSDI– Enhanced Mobile Broadband (eMBB): this also covers IoT, robotics, sensor
network– Massive Machine Type Communications (mMTC) to support HPC and large
scale distributed data processing– Ultra Reliable and Low Latency Communications (URLLC): industry
automation, process control, real time applications• 5G architecture solutions– e2e network slicing technology providing isolated virtual overlay networks
using Network Functions Virtualisation (NFV) and cloud native services
deployment model and mechanisms• Extended to 5G Radio Access Network (RAN) for sensor networks– Consistent security model that enables Trusted Execution Environment (TEE)

HPCS2020 BDAA2020

Platform RI as a Service (PRIaaS)

13

Leveraging Platform Concept for RI•

There is a need for defining and building new type of infrastructure for EOSC
projects– Current EOSC pilot projects successfully demonstrated inter-/multi-domain data
integration– However, each pilot project built own underlying infrastructure•

Future EOSC infrastructure should provide functionality– (1) automate deploying specialized RIs with focus on scientific data integration– (2) create a repository of infrastructure/services design patterns and common
templates– (3) facilitate cooperative/business relations between partners– (4) apply governance and compliance policies by-design•

Learn from and leverage best industry practice and infrastructuredevelopment trends– Hyperconverged Infrastructure and 5G e2e network slicing– Industry developed platform and cloud native models– DevOps and DevSecOps

HPCS2020 BDAA2020

Platform RI as a Service (PRIaaS)

14

Platform Concept and Platform Economy
•

•

•
•

“The platform business model enables interactions between producers andconsumers of value. It achieves this goal through two mechanisms. First, a platform
provides a plug and play infrastructure which encourages open participation by an
external ecosystem of producers and consumers. Second, it lays out the rules ofgovernance for the interactions that ensue.”– Source: Sangeet Paul Choudary , http://platformthinkinglabs.com/start here/“A platform is a business based on enabling value creating interactions between
external producers and consumers. The platform provides an open, participative
infrastructure for these interactions and sets governance conditions for them. The
platform’s overarching purpose: to consummate matches among users and facilitate
the exchange of goods, services, or social currency, thereby enabling value creation
for all participants– Source: Choudary , Sangeet Paul; Van Alstyne , Marshall W.; Parker, Geoffrey G.; Platform
Revolution: How Networked Markets Are Transforming the Economy and How to Make Them
Work for You

Example platform based businesses: Airbnb, Alibaba, Amazon, Azure (Microsoft),
eBay, Facebook, Instagram, KAYAK, Pinterest, YouTube, Twitter, Wikipedia, Uber,
UpworkThis is in contrast to pipeline based businesses with linear value chains

HPCS2020 BDAA2020

Platform RI as a Service (PRIaaS)

15

Platform vs Pipeline Model• Source: Choudary , Sangeet Paul; Van Alstyne , Marshall W.; Parker,
Geoffrey G.; Platform Revolution: How Networked Markets Are
Transforming the Economy and How to Make Them Work for You– “Platforms beat pipelines because platforms scale more efficiently by
eliminating gatekeepers.” Example: Coursera vs. a college / university– “Platforms beat pipelines because platforms unlock new sources ofvalue creation and supply.” Example: Airbnb vs. an hotel chain– “Platforms beat pipelines by using data based tools to create
community feedback loops.”• For virtualized functions, objective metrics could also be used, in
addition to subjective consumer feedback.

– “Platforms invert the firm”• Pipeline Model uses linear value chains for a vertical businessstack
HPCS2020 BDAA2020

Platform RI as a Service (PRIaaS)

16Adopting TMForum DPRA
• TMForum DPRA (Digital Platform Reference Architecture) defines atelecom services provider platform that allows delivering a fully functional
service platform/infrastructure for customers– IG1157 Digital Platform Reference Architecture Concepts and Principles
v5.0.1, 21 July 2020 [online]
https://www.tmforum.org/resources/reference/ig1157-digital-platformreference-architecture-concepts-and-principles-v5-0-0/– Actualisation Platform is defined as the main DPRA component that enables creating
customer/tenant service ecosystem– Implements platform economy concept• Part of the TMForum Open Digital Architecture (ODA)– IG1167 TM Forum Exploratory Report ODA Functional Architecture, 31 Jan
2020 [online] https://www.tmforum.org/resources/exploratory-report/ig1167oda-functional-architecture-v5-0/

HPCS2020 BDAA2020

Platform RI as a Service (PRIaaS)

17

TMForum: Actualisation Platform View

• TM Forum Actualization Platform View The underlying infrastructure of edge to hyper scale datacenters and networks
that host the software components that make up the Business Platform, enabled by reusable technical capabilities thatare required to operate in an agile and efficient manner.
HPCS2020 BDAA2020

Platform RI as a Service (PRIaaS)

18

TMF DPRA Actualisation Platform Elements
The Actualisation Platform includes the following essential(group of) components:• Common infrastructure and platform services• Data and digital content services• Catalog Lifecycle Management & Federation Platform• Integration, orchestration, and DevOps• Security and Identity Management• Core commerce services including Fulfillment Platform
Component and customer facing services

HPCS2020 BDAA2020

Platform RI as a Service (PRIaaS)

19

Paradigm Change in Modern Data Driven Science• Availability of Pan-European Research Infrastructure Platform as a Service• Automation of scientific experiments and all data handling processes•

Adopting and leveraging DevOps and DataOps/MLOps technologies and using
cloud automation tools• Digitising existing artifacts and creating their digital twins, AI assisted
documenting and cataloging, building subject/domain knowledge base using
self-learning algorithms.
• Adoption of FAIR data principles, both prospective and retrospective• Support data value creation model and flow (e.g. STREAM data properties)• Availability of new algorithms for distributed secure data processing (e.g.
federated machine learning, blockchain enabled policy enforcement)•

Enclave computing as a new service by cloud providers• Global data availability and access for cooperative group of researchers,
however subject for the data sharing and access policies, in particular GDPR.
•

Advanced security, access control and identity management technologies

HPCS2020 BDAA2020

Platform RI as a Service (PRIaaS)

20

Virtual RI

Develop &OperateDatasets,
Archives

RI Users/Researchers

Virtual RI and User Applications
Data Source

RI Portal

Data Target

Portal, Catalog, FedID, Policy

Actualisation Platform

Federated Access Infrastructure & Tenant Management
Dev Env & Tools• DevOps (CI/CD)• Templates &Design Patterns• API & Container• DataOps/MLOps
Management &Operation• Service Catalog& Lifecycle Mngnt• Orchestration(multi-lyr&domain)• Auto & Optimisat(AI enabled)

Service Prov &Fulfilment• SLA Mng
• Policy Provision• User ID Provision
Infrastructure
Services (P/Iaas)• Compute• Storage• Network• IoT & Edge
• Monitoring• AAI & Federation• Security (VPC,
VPN, PKI & Trust)• Blockchain Perm

PRIaaS
Components

Virtualised
Resorces

PRIaaS Architecture
Model (in progress)

VirtualisatedResources

CCI 18 Nov 2020

Data Services• Directory• Metadata, PID• Sci Data Lakes(Semantic)• Data Ingest/ELT• Data Exchange(Data Marts)• Provenance• FAIR & Quality
Assurance• Privacy• ML/AI & DA
Datasets & Arch• Research• Technological

Composable Infrastructure Resource
ComputeStorage

Actualisation Platform
Components• Core Infrastructure
Services (IaaS & PaaS)• Data Services• Management andOperation• Development Environment
and Tools– DevOps– Templates and Patterns•••

Network

Physical resource (distributed, multi-provider)

SLICES-DS Project and Activities•

Service Provisioning andFulfilment
Datasets and Archives
Federated Access
Infrastructure + IoT Edge
and Tenants Management
Virtual RIs and Portal
21

From EOSC-1 to EOSC-2: Four Technology Aspects
2016-2018

RI Type

EOSC-1

2020-2022

EOSC-2 (future)

Definition

Interoperable Federated(European) RI, FAIR RI

Virtualised Pan-European RI platform as aService and Ecosystem (PRIaaS)

Network &Compute

•
•
•
•
•
•
•

Distributed scalable computing
Cloud based Big Data technologies
High performance networks
5G technologies, wireless access
IoT sensor networks
Portal and Services Catalog
Industry standards and IDSA
adoption• Composable virtualized RI provisioning on demand (including
for services integration)• Common federated computing and networking
platform/environment, enabling virtual RIs• Cloud based and cloud enabled• DevOps and AI enabled services• Digital Twins• Interoperability and Integration with Industry infrastructure(e.g. IDSA+, Industrial Internet)

Data Infra

•
•
•
•

FAIR data
Data Factories and PID
Metadata registries
Interoperable/common DataManagement model• Fully adopted FAIR principles, extended to ontologies• Semantically enabled scientific data lakes, common
vocabularies• Secure/trusted data exchange (data markets)• Full data value chain supported (cross-domain)

Security• Federated Identity Management,
Federated Access Control• Automated identity provisioning

Infra
Managnt
Technolog• Integrated Operation andAutomation

HPCS2020 BDAA2020

•
•
•
•
•

Beyond2025

Zero trust security, Trust Bootstrapping
Homomorphic encryption and data processing
Quantum ready encryption, Quantum enabled key management
Federated Identity Management, Federated Access Control
Automated identity provisioning• Fully automated RI and services provisioning, management andoperation• Optimisation of infrastructure and operation• DevOps
AI enabled(re-usable design patterns)
Platform
RI as aandService(PRIaaS)

22

PRIaaS Technology Development in Ongoing andPast Projects• GEANT GN4-3 Research WPs and Tasks– Operation, Automation, Virtualisation Architecture (OAV) and Service
Provider Architecture (SPA)• Adopts/recommends TMF ODA for GEANT OAV• EOSC - Minimum Viable EOSC (MVE)– EOSC Architecture and FAIR data principles– Apparently, vision for EOSC-2• SLICES-DS – Future RI technology research• Past projects– GN4-1 Open Cloud eXchange (OCX) and Big Data Architecture
Framework (BDAF)– GN4-2 ZeroTouch Provisioning, Operation and Management(ZTPOM)– GN3 Composable Services Architecture (CSA)– GEYSERS Intercloud Architecture Framework (ICAF)

HPCS2020 BDAA2020

Platform RI as a Service (PRIaaS)

23

SLICES-RI and SLICES-DS Projects• European Project involving 23 partners– SLICES-DS Design stage – 10 partners• SLICES – RI is under evaluation for a new ESFRI DIGIT domain• SLICES-RI Research Areas– Advanced wireless networking• 5G and radio access networks• Integrated sensing and communication– Smart/intelligent infrastructure operation and management• Distribution of intelligence into the Edge and beyond the Edge of the
network– Design and validation of new Edge/Fog and hyper-converged
infrastructures• Distributed resource management & microservices• Federated deep-learning– Advanced functionalities• Composable infrastructure services on-demand (RI as a Service)• Security and privacy
HPCS2020 BDAA2020

Platform RI as a Service (PRIaaS)

24

SLICES-RI Research Topics: UvA and NL partners•

Cloud and Network Infrastructure research
–
–
–
–

•

Architecture and design patterns of the future RI Platform as a Service (PRIaaS)
Federated multi-cloud and inter-cloud infrastructure integration and management
Decentralised network/compute optimisation in edge/fog environments
Sustainable cloud services with energy consumption monitoring and optimization

Data Infrastructure– Big Data Infrastructure and Technologies (cloud enabled)– Trusted data exchange and processing with policy/rules enforcement, preserving data
sovereignty and protecting data privacy– Data management and quality assurance aspects in Industry 4.0 experimentation and
Digital Twins applications•

New security and compliance models for Complex Cyber Infrastructure– Distributed Cyber Security techniques and architectures••

Federated Data Analytics and Deep Learning, in particular for predictive
maintenance, logistics and smart cities
Support of education on key technologies of the future data centric and cloud
enabled infrastructures by provisioning educational platforms and resources foruniversities on demand

HPCS2020 BDAA2020

Platform RI as a Service (PRIaaS)

25

Discussion and Questions• Presented PRIaaS Architecture and technology overview asa Request for Comments and call for professional community
contribution• Consistent framework for planning and aligning future
research areas• PRIaaS platform provider model and federated model for
multidomain research

HPCS2020 BDAA2020

Platform RI as a Service (PRIaaS)

26

Research Data in Europe: FAIR, EOSC, ORD
Pilot and DMP• EU policy on Open Research Data
• Horizon 2020 Open Research Data (ORD) Pilot• Horizon 2020 Data Management and Data Management Plan

CCI 18 Nov 2020

SLICES-DS Project and Activities

27

Requirements to Future SDI• R01: Cloud based provisioned (on-demand) instant RI, fully functional
including virtual user organisation – multi-cloud and hybrid• RI02: On-demand infrastructure provisioning to support data sets andscientific workflows, mobility of data-centric scientific applications• RI03: Trusted environment for data storage and processing• RI04: Mechanisms for policy binding to data to protect privacy,
confidentiality and IPR• RD01: Multi-tier inter-linked data distribution and replication• RD02: Secure trusted data infrastructure, ensuring data sovereignty andtrustworthiness– FAIR compliant and supporting STREAM properties for effective dataexchange• RD03: Support for data integrity, confidentiality, accountability,
provenance, sovereignty• RO01: Support long running experiments and large data volumes
generated at high speed• RO02: Support of virtual scientist communities, addressing dynamic usergroups creation and management, federated identity management
HPCS2020 BDAA2020

Platform RI as a Service (PRIaaS)

28

EU policy on Open Research Data
• Research data can be defined as whatever is either produced in the
research process or evidences research outputs such as articles• The European Commission’s Research Data definition– “Information, in particular facts or numbers, collected to be examined andconsidered and as a basis for reasoning, discussion, or calculation”– https://ec.europa.eu/research/openscience/index.cfm?pg=openaccess– https://ec.europa.eu/research/participants/docs/h2020-funding-guide/cross-cuttingissues/open-access-dissemination_en.htm– https://ec.europa.eu/research/participants/docs/h2020-funding-guide/cross-cuttingissues/open-access-data-management/data-management_en.htm• Examples include: statistics, results of experiments, measurements,
observations resulting from fieldwork, survey results, interview
recordings, images• Open data are deposited in institutional or specialist repositories andlicensed appropriately so that prospective users know clearly any
limitations on re-use.
ED2MIT2020 MATES

Case study: Research Data Management29Horizon 2020 Open Research Data (ORD) Pilot• ORD pilot aims to improve and maximise access to and re-use of
research data generated by Horizon 2020 projects, taking into
account•••••

the need to balance openness and protection of scientific informationcommercialisation and IPR
privacy concerns
security
data management and preservation questions• Applying principle 'as open as possible, as closed as necessary’• Complying with FAIR Data principles• ORD applies primarily to the data needed to validate the resultspresented in scientific publications.
• Other data can also be provided by the beneficiaries on a voluntary basis.

ED2MIT2020 MATES

Case study: Research Data Management30Horizon 2020 Data Management and DataManagement Plan
• Data Management Plans (DMPs) are a key element of good data management.
– A DMP describes the data management life cycle for the data to be collected, processed
and/or generated by a Horizon 2020 project.
– Help making research data Findable, Accessible, Interoperable and Reusable (FAIR)• DMP should include information on:
•
•
•
•
•

handling of research data during & after the end of the project
what data will be collected, processed and/or generated
which methodology & standards will be applied
whether data will be shared/made open access andhow data will be curated & preserved (including after the end of the project).

• The project must submit a first version of DMP (as a deliverable) within thefirst 6 months of the project.
– DMP is updated if data are changed– DMP is mandatory for projects participating in ORD Pilot[ref] https://ec.europa.eu/research/participants/docs/h2020-funding-guide/cross-cutting-issues/openaccess-data-management/data-management_en.htm

ED2MIT2020 MATES

Case study: Research Data Management31