Developing Customisable Education and Training
Program on Cloud Computing
andApproach to Big Data Education

Yuri Demchenko
SNE Group, University of Amsterdam
BoF “Cloud Computing and Data Analysis Training for theDeveloping World”
18 September 2013, 2nd RDA Plenary

Outline• Cloud Computing Curriculum design: Basic principles• Proposed Big Data Architecture Framework (BDAF)– Data Models and Big Data Lifecycle• RDA2 BoF on Educations and Skills Development for Data
Intensive Science (17 Sept 2013)• Discussion: Opportunities to support training in developing world(DW)

18 September 2013, RDA2Edu&Train on Cloud&BigData for DWSlide_2

Cloud Computing Curriculum Development• Presented at the BoF during the 1st RDA meeting 18-20
March 2013 in Gothenburg
http://www.uazone.org/demch/presentations/rda2013-gothenburg-bofeducation-skills-v02.pdf• Cloud Computing as enabling technology for Scientific Data
Infrastructure (SDI) and Big Data Infrastructure• Cloud Computing Common Body of Knowledge (CBK)• Course instructional approach: Bloom’s Taxonomy andAndragogy• Course structure Cloud Computing technologies andservices design

18 September 2013, RDA2

Edu&Train on Cloud&BigData for DWSlide_3

Example: Common Body of Knowledge (CBK) inCloud Computing
CBK refers to several domains or operational categories into which Cloud
Computing theory and practices breaks down•

Still in development but already piloted by some companies, including industry
certification program (e.g. IBM, AWS?)CBK Cloud Computing elements
1. Cloud Computing Architectures, service and deployment models
2. Cloud Computing platforms, software/middleware and API’s
3. Cloud Services Engineering, Cloud aware Services Design
4. Virtualisation technologies (Compute, Storage, Network)
5. Computer Networks, Software Defined Networks (SDN)
6. Service Computing, Web Services and Service Oriented Architecture (SOA)
7. Computing models: Grid, Distributed, Cluster Computing
8. Security Architecture and Models, Operational Security
9. IT Service Management, Business Continuity Planning (BCP)
10. Business and Operational Models, Compliance, Assurance, Certification
18 September 2013, RDA2

Edu&Train on Cloud&BigData for DW

Slide_4

Example: CKB-Cloud Components LandscapeCloud Platforms, API

Cloud Architectures, ServiceModels

Business/Operational Models, Compliance, Assurance

Cloud Computing Fundamentals
Cloud Services Engineering&Design
DesignEngineering

Virtualisation

Networking

Web Services, SOASecurity, ID Management

Computing Models: Grid, Distributed, Cluster

IT Systems Management

Cloud Computing Common Body of Knowledge (Full)

18 September 2013, RDA2Edu&Train on Cloud&BigData for DW

5

Multilayer Cloud Services Model (CSM) – Taxonomy of
Existing Cloud Architecture Models
User/Customer Side Functions and Resources

Security InfrastructureManagementOperations Support System

User/Client Services* Identity services (IDP)* Visualisation

Administration andManagement Functions(Client)

Content/Data Services* Data * Content * Sensor * Device

1

Endpoint Functions* Service Gateway* Portal/Desktop

Access/Delivery
InfrastructureInter-cloud Functions* Registry and Discovery* Federation Infrastructure

SaaSPaaS

Layer C5ServicesAccess/Delivery

Layer C4Cloud Services(Infrastructure,Platforms,Applications,Software)

Cloud Services (Infrastructure, Platform, Application, Software)

IaaS

Layer C6User/Customer
side Functions

PaaS-IaaS IF
PaaS-IaaS Interface

IaaS – Virtualisation Platform Interface

Cloud Management
Software(Generic Functions)

Cloud Management Platforms
OpenNebula

Virtualisation Platform

OpenStack

KVM

VM

VM

VPN

Other
CMS

XEN

VMwareNetwork
Virtualis

Proxy (adaptors/containers) - Component Services and Resources

StorageResources

Hardware/PhysicalResources

ComputeResources

Contrl&Mngnt Links

18 September 2013, RDA2

Network
Infrastructure

Layer C3
Virtual Resources
Composition andControl(Orchestration)

CSM layers(C6) User/Customer sideFunctions(C5) Services
Access/Delivery(C4) Cloud Services(Infrastructure, Platform,Applications)(C3) Virtual Resources
Composition andOrchestration(C2) Virtualisation Layer(C1) Hardware platform anddedicated network
infrastructure

Layer C2
Virtualisation

Layer C1
PhysicalHardware
Platform andNetwork

Control/Mngnt Links

Data Links

Data Links

Edu&Train on Cloud&BigData for DW

6

Relations Course Components and CSM

Layer C6
User/Customerside Functions

User Applications

Layer C5
Services
Access/Delivery

Cloud Services API and Tools

Hybrid Clouds,
Integrated Infrastructure

User Applications

1

Cloud Service Design, Operational
proceduresManagementSecurity

Operations Support System

Cross-layer
Service s (Planes)

Cloud/Intercloud Services Integration, Enterprise
IT infrastructure migration

1Customer Enterprise/Campus Infrastructure/Facility

Layer C4Cloud Services(Infrastructure,Platforms,Applications,Software)

Layer C3
Virtual Resources
Composition andControl(Orchestration)

Layer C2
Virtualisation

Cloud Services Platform

Cloud Service Provider
Datacenters and Infrastructure
18 September 2013, RDA2

Edu&Train on Cloud&BigData for DW

Layer C1
PhysicalHardware
Platform andNetwork7

Example: Mapping Course Components, Cloud
Professional Activity and Bloom’s Taxonomy
Customer Enterprise/Campus Infrastructure/Facility

Comprehension

Application
Analysis

User Applications
Cloud Services
API and Tools

1Hybrid Clouds, Integrated
Infrastructure

User Applications

Cloud Service Design, Operational procedures

Knowledge

Cloud/Intercloud Services Integration, Enterprise IT
infrastructure migration

Taxonomy
Cognitive
Domain [3]18 September 2013, RDA2

Perform standard tasks,
use standard API and
Guidelines

Create own complex
applications using
standard API (simpleengineering)
Integrate different
systems/components,
e.g. provider andenterprise infrastructure

Cloud Services Platform

Synthesis
Evaluation

Taxonomy
Professional
Activity Domain

Cloud Service Provider
Datacenters and Infrastructure

Edu&Train on Cloud&BigData for DW

Extend existing services,
design new services
Develop new architecture
and models, platforms
and infrastructures8Proposed Cloud Computing Course Structure
Basic parts & Advanced parts

Part 1.1. Cloud Computing definition and general usecases
Part 1.2. Cloud Computing and enabling technologies
Part 2.1. Cloud Architecture models and industry standardisation: Architectures overview
Part 2.2. Cloud Architecture models and industry standardisation: Standard interfacesPart 3.1. Major cloud provider platforms: Amazon AWS, Microsoft Azure, GoogleApps, etc
Part 3.2. Major cloud provider platforms: Public, Research and Community Clouds
Part 4. Cloud middleware platforms: Architecture, platforms (OpenStack, OpenNebula),API, usage examples
Part 5.1. Cloud Infrastructure as a Service (IaaS): Architecture, platform and providers
Part 5.2. Cloud Infrastructure as a Service (IaaS): IaaS services design andmanagement
Part 6.1. Cloud Platform as a Service (PaaS): Architecture, platform and providers
Part 6.2. Cloud Platform as a Service (PaaS): PaaS services design and management
Part 7.1. Security issues and practices in clouds
Part 7.2. Security services design in clouds; security models and Identity management
Part 8 (Advanced). InterCloud Architecture Framework (ICAF) for Interoperability andIntegration: Architecture definition and design patterns
18 September 2013, RDA2

Edu&Train on Cloud&BigData for DW

9Dig Data Architecture Framework (BDAF)• As a basis for Education and Training in Big Data or
Data Intensive Science

18 September 2013, RDA2

Edu&Train on Cloud&BigData for DW

10

Big Data Properties and Definition
GenericPropertiesVariety••••••

StructuredUnstructured
Multi-factor
Probabilistic
LinkedDynamic••••

Terabytes
Records/ArchTables, Files
Distributed

Acquired
Properties
18 September 2013, RDA2Velocity••
••

Batch
Real/near-time
Processes
Streams

6 Vs ofBig Data••••• Changing data
• Changing model• Linkage

Variability

5 parts Big Data definition

Volume•••••Trustworthiness
Authenticity
Origin, Reputation
Availability
Accountability(1) Big Data Properties: 6V(2) New Data Models(3) New Analytics(4) New Infrastructure andTools(5) Source and Target

Correlations
StatisticalEventsHypotheticalValue

Veracity

Edu&Train on Cloud&BigData for DW

11

Big Data Definition: From 5+1V to 5 Parts (1)(1) Big Data Properties: 5V– Volume, Variety, Velocity, Value, Veracity– Additionally: Data Dynamicity (Variability)(2) New Data Models– Data Lifecycle and Variability– Data linking, provenance and referral integrity(3) New Analytics–

Real-time/streaming analytics, interactive and machine learning analytics(4) New Infrastructure and Tools––––

High performance Computing, Storage, Network
Heterogeneous multi-provider services integration
New Data Centric (multi-stakeholder) service models
New Data Centric security models for trusted infrastructure and data processingand storage(5) Source and Target– High velocity/speed data capture from variety of sensors and data sources– Data delivery to different visualisation and actionable systems and consumers– Full digitised input and output, (ubiquitous) sensor networks, full digital control
18 September 2013, RDA2

Edu&Train on Cloud&BigData for DW

12

Big Data Definition: From 5V to 5 Parts (2)Refining Gartner definition• Big Data (Data Intensive) Technologies are targeting to process (1) highvolume, high-velocity, high-variety data (sets/assets) to extract intended
data value and ensure high-veracity of original data and obtained
information that demand cost-effective, innovative forms of data andinformation processing (analytics) for enhanced insight, decision making,and processes control; all of those demand (should be supported by) new
data models (supporting all data states and stages during the whole data
lifecycle) and new infrastructure services and tools that allows also
obtaining (and processing data) from a variety of sources (includingsensor networks) and delivering data in a variety of forms to different dataand information consumers and devices.
(1) Big Data Properties: 5V(2) New Data Models(3) New Analytics(4) New Infrastructure and Tools(5) Source and Target
18 September 2013, RDA2

Edu&Train on Cloud&BigData for DW

13

Defining Big Data Architecture Framework• Existing attempts don’t converge to consistent view: ODCA, TMF, NIST
– See http://bigdatawg.nist.gov/_uploadfiles/M0055_v1_7606723276.pdf• Big Data Architecture Framework (BDAF) by UvAArchitecture Framework and Components for the Big Data Ecosystem.
Draft Version 0.2
http://www.uazone.org/demch/worksinprogress/sne-2013-02-techreport-bdafdraft02.pdf• Architecture vs Ecosystem– Big Data undergo a number of transformations during their lifecycle– Big Data fuel the whole transformation chain• Data sources and data consumers, target data usage– Multi-dimensional relations between• Data models and data driven processes• Infrastructure components and data centric services• Architecture vs Architecture Framework (Stack)– Separates concerns and factors• Control and Management functions, orthogonal factors– Architecture Framework components are inter-related

18 September 2013, RDA2

Edu&Train on Cloud&BigData for DW

14

Big Data Architecture Framework (BDAF)
for Big Data Ecosystem (BDE)(1) Data Models, Structures, Types– Data formats, non/relational, file systems, etc.

(2) Big Data Management– Big Data Lifecycle (Management) Model• Big Data transformation/staging– Provenance, Curation, Archiving(3) Big Data Analytics and Tools– Big Data Applications• Target use, presentation, visualisation(4) Big Data Infrastructure (BDI)– Storage, Compute, (High Performance Computing,) Network– Sensor network, target/actionable devices– Big Data Operational support(5) Big Data Security– Data security in-rest, in-move, trusted processing environments
18 September 2013, RDA2

Edu&Train on Cloud&BigData for DW

15

DataSource

Data
Collection&Registration

Data
Filter/Enrich,Classification

Data
Analytics,
Modeling,
Prediction

Data
Delivery,
Visualisation

Consumer

Big Data Ecosystem: Data, Lifecycle,
Infrastructure

Big Data Target/Customer: Actionable/Usable Data
Target users, processes, objects, behavior, etc.
Federated
AccessandDelivery
Infrastructure(FADI)Big Data Source/Origin (sensor, experiment, logdata, behavioral data)

Big Data Analytic/Tools

Storage
GeneralPurpose

Data Management

ComputeGeneral
PurposeHighPerformance
ComputerClusters

StorageSpecialised
Databases
Archives(analytics DB,In memory,operstional)

Data
categories: metadata,(un)structured, (non)identifiable
Data
Datacategories:
categories:metadata,
metadata,(un)structured,(un)structured,(non)identifiable(non)identifiable

Intercloud multi-provider heterogeneous Infrastructure
Security Infrastructure

18 September 2013, RDA2

Network InfrastructureInternal

InfrastructureManagement/Monitoring

Edu&Train on Cloud&BigData for DW

16

Big Data Infrastructure and Analytic Tools
Big Data Target/Customer: Actionable/Usable Data
Target users, processes, objects, behavior, etc.
Big Data Source/Origin (sensor, experiment, logdata, behavioral data)

Big Data Analytic/Tools
Analytics:
Refinery, Linking, Fusion

Analytics :Realtime, Interactive,
Batch, Streaming

Storage
GeneralPurposeData Management

ComputeGeneral
PurposeAnalytics Applications
Link Analysis
Cluster Analysis
Entity Resolution
Complex Analysis

High
Performance
Computer
Clusters:

Federated
AccessandDelivery
Infrastructure(FADI)

StorageSpecialised
Databases
Archives

Data
categories: metadata,(un)structured, (non)identifiableDataDatacategories:
categories:metadata,metadata,(un)structured,(un)structured,(non)identifiable(non)identifiable

Intercloud multi-provider heterogeneous Infrastructure
Security Infrastructure

18 September 2013, RDA2

Network InfrastructureInternal

InfrastructureManagement/Monitoring

Edu&Train on Cloud&BigData for DW

17

Data Transformation/Lifecycle Model
Common Data Model?
•Data Variety and Variability•
Semantic Interoperability

Data Model (1)Data Model (1)Data Model (4)Data (inter)linking?
•Persistent ID•
Identification•Privacy, Opacity

Data Storage

DataCollection&Registration

Data
Filter/Enrich,Classification

DataAnalytics,Modeling,
Prediction

DataDelivery,Visualisation

Consumer
Data Analitics
Application

DataSource

Data Model (3)Data repurposing,Analitics re-factoring,
Secondary processing••

Does Data Model changes along lifecycle or data evolution?
Identifying and linking data

– Persistent identifier– Traceability vs Opacity– Referral integrity
18 September 2013, RDA2

Edu&Train on Cloud&BigData for DW

18

Scientific Data Lifecycle Management (SDLM)
Model
Data Lifecycle Model in e-Science

UserResearcher

Data discovery

Data Curation(including retirement and clean up)
DatarecyclingRaw Data
Experimental
DataProject/
Experiment
Planning

Data collectionandfiltering

Structured
Scientific
Data

Data analysis

DB

Data archiving

DataRe-purpose

Data linkage
to papers

Data sharing/
Data publishing

Data Re-purpose
Data Linkage Issues•
Persistent Identifiers (PID)•
ORCID (Open Researcher andContributor ID)•Lined Data

Data Clean up and Retirement•
Ownership and authority•
Data Detainment

Edu&Train on Cloud&BigData for DW

End of project

Open
PublicUse

Data Links

18 September 2013, RDA2

Dataarchiving

Metadata &Mngnt

19BoF on Education and Skills Development inData Intensive Science (17 Sept 2013)• Attended by 16 representatives from universities,
libraries, e-Science, data centers, researchcoordination bodies• Agenda included (60 min)– Round of introduction and interests expression– Developments since the first BoF in Gothenburg– Presentations• Demystifying Data Science (Natasha Balac, SDSC)• Big Data in the Cloud: Research and Education (Geoffrey Fox,Indiana University Bloomington)• Discussion on further steps– Priority topics to address and Interest Groupestablishment
RDA2, 16-18 Sept 2013

Data Intensive Science Education and Skills

Slide_20

Topics discussed• What experience do we have on component
technologies to support Scientific Data?
• Existing instructional and educational concepts andtechnologies• How to benefit from collective knowledge andexperience of RDA community?
• Scientific Data and Big Data in industry– Multidisciplinary domain and needs cooperation ofspecialists from multiple knowledge, scientific andtechnology domains

RDA2, 16-18 Sept 2013

Data Intensive Science Education and Skills

21

BoF Outcome and Decisions• Proceed with the formal establishment of Interest Group (IG) onEducation and Skills in Data Intensive Science– 2 co-chairs from Europe and US volunteered – Data Analytics and e-Infrastructure– Seeking another co-chair from librarian or data archives community and/or APregion– IG scope to reflect interest of the major stakeholders: university, research,libraries, data archives, industry, subject domains – yet to identify• Involve associations concerned with DIS/BD Education and training– LERU, LIBER and similar organisations in US/worldwide– Involve/liaise with industry – via standardisation bodies or direct contacts withleaders• Hold the next meeting as a proposed IG– With more focused discussion on the community needs in defining basic skills andrequired knowledge for Data Science and Data Scientist– Reach wider and targeted community and potential stakeholders and interestedparties– Involve universities working on the DIS education programs

RDA2, 16-18 Sept 2013

Data Intensive Science Education and Skills

22

Slide from the presentation
Demystifying Data Science(by Natasha Balac, SDSC)

RDA2, 16-18 Sept 2013

Data Intensive Science Education and Skills

23

Discussion• Experience of delivering Education&Training to DW
audience• Importance of the pre-requisite and introductoryknowledge module• Possibility for profiling the course• Need for a local base, online access and services tobe negotiated and tested in advance

18 September 2013, RDA2

Edu&Train on Cloud&BigData for DW24