Defining the Big Data Architecture
Framework (BDAF)
Outcome of the Brainstorming Session
at the University of Amsterdam(17 July 2013)
Yuri Demchenko (facilitator, reporter),
SNE Group, University of Amsterdam
Presented to NIST Big Data WG, Subgroup Architecture
25 July 2013
Available http://bigdatawg.nist.gov/_uploadfiles/M0055_v1_7606723276.pdf

Outline• Big Data definition– 5 V’s of Big Data: Volume, Velocity, Variety, Value, Veracity– Data Origin and Target• From Big Data to All-Data – Paradigm change and New
challenges– Big Data Infrastructure and Big Data Security• Defining Big Data Architecture Framework (BDAF)– From Architecture to Ecosystem to Architecture Framework– Developments at NIST, ODCA, TMF, RDA• Data Models and Big Data Lifecycle• Big Data Infrastructure (BDI)• Brainstorming: new features, properties, components, missing
things, definition, directions
17 July 2013, UvA

Big Data Architecture Brainstorming

Slide_2

* Big Data Research at SNE•

Focus on Infrastructure definition and services– Including Big Data Security– Software Defined Infrastructure based on Cloud/Intercloud technologies•

Papers published and submitted– Addressing Big Data Issues in Scientific Data Infrastructure, by Demchenko, Y.,
P.Membrey, P.Grosso, C. de Laat.
First International Symposium on Big Data and Data Analytics in Collaboration (BDDAC
2013). Part of The 2013 International Conference on Collaboration Technologies andSystems (CTS 2013), May 20-24, 2013, San Diego, California, USA– Big Security for Big Data: Addressing Security Challenges for the Big Data
Infrastructure, by Y.Demchenko, P.Membrey, C.Ngo, C. de Laat, D.Gordijenko
Submitted to Secure Data Management (SDM’13) Workshop. Part of VLDB2013
conference, 26-30 August 213, Trento, Italy– 科研信息化基础设施的大数据挑战 (Big Data Challenges for e-Science
Infrastructure) by Demchenko, Y., Z.Zhao, P.Grosso, A.Wibisono, C. de Laat, In China
Science and Technology Resources Review, Vol.45 No.1 30-35,40 Jan. 2013.

9 July 2013, UvA

Big Data Research Landscape3

* Big Data Architecture Framework (BDAF) Proposed Context for the discussion• Data Models, Structures, Types– Data formats, non/relational, file systems, etc.

• Big Data Management– Big Data Lifecycle (Management) Model• Big Data transformation/staging– Provenance, Curation, Archiving• Big Data Analytics and Tools– Big Data Applications• Target use, presentation, visualisation• Big Data Infrastructure (BDI)– Storage, Compute, (High Performance Computing,) Network– Big Data Operational support• Big Data Security– Data security in-rest, in-move, trusted processing environments
17 July 2013, UvA

Big Data Architecture Brainstorming4

* Big Data Definition (1)•IDC definition (conservative and strict approach) of Big Data:"A new generation of technologies and architectures designed to economically extract value
from very large volumes of a wide variety of data by enabling high-velocity capture,
discovery, and/or analysis"•

Big data is high-volume, high-velocity and high-variety information assets that demand costeffective, innovative forms of information processing for enhanced insight and decision
making. Gartner, http://www.gartner.com/it-glossary/big-data/–•

Big Data: a massive volume of both structured and unstructured data that is so large thatit's difficult to process using traditional database and software techniques.
–•

From “The Big Data Long Tail” blog post by Jason Bloomberg (Jan 17, 2013). http://www.devx.com/blog/the-big-datalong-tail.html“Data that exceeds the processing capacity of conventional database systems. The data is
too big, moves too fast, or doesn’t fit the structures of your database architectures. To gain
value from this data, you must choose an alternative way to process it.”–•Termed as 3 parts definition, not 3V definition

Ed Dumbill, program chair for the O’Reilly Strata Conference

Termed as the Fourth Paradigm *)“The techniques and technologies for such data-intensive science are so different that it isworth distinguishing data-intensive science from computational science as a new, fourthparadigm for scientific exploration.” (Jim Gray, computer scientist)*) The Fourth Paradigm: Data-Intensive Scientific Discovery. Edited by Tony Hey, Stewart Tansley, and Kristin Tolle. Microsoft, 2009.

9 July 2013, UvA

Big Data Research Landscape5* 5 V’s of Big Data

Volume••••Terabytes
Records/ArchTransactions
Tables, Files

Variety••••

Velocity
•
•
•
•

Batch
Real/near-time
Processes
Streams

5 Vs ofBig Data

Structured
Unstructured
Multi-factor
Probabilistic
•
•
•
•
•

Value
•
•
•
•

Statistical
EventsCorrelations
Hypothetical

Trustworthiness
Authenticity
Origin, Reputation
Availability
Accountability

Commonly accepted
3V’s of Big Data

Veracity

17 July 2013, UvA

Big Data Architecture Brainstorming

6

Big Data Security: Veracity and other factors•

Velocity

Volume••••Terabytes
Records/Arch
Tables, Files
Distributed

•
•
•
•

•

–
–
–

Batch
Real/near-time
Processes
Streams•

Variety
•
•
•
•
•
•

Structured
Unstructured
Multi-factor
Probabilistic
LinkedDynamic

5 Vs ofBig Data

•
•
•
•
•

Trustworthiness
Authenticity
Origin, Reputation
Availability
Accountability

Statistical
EventsCorrelations
Hypothetical•

Timeliness
Mobility (mobile/remote access; fromother domain – roaming; federation)

Accountability–•

Identification both Data and Source
Source: system/domain and author
Data linkage (for complex hierarchicaldata, data provenance)

Availability
–
–

Value
•
•
•
•

Trustworthiness and Reputation ->Integrity
Origin, Authenticity and Identification

As pro-active measure to ensure data
veracity

Data Dynamicity (i.e. Variability as
6th V)–As an additional property reflecting datachange during their processing or lifecycle

Big Data Architecture Brainstorming

7

Veracity

17 July 2013, UvA

** Big Data Definition: From 5V to 5 Parts (1)(1) Big Data Properties: 5V– Volume, Variety, Velocity, Value, Veracity– Additionally: Data Dynamicity (Variability)(2) New Data Models– Data Lifecycle and Variability– Data linking, provenance and referral integrity(3) New Analytics–

Real-time/streaming analytics, interactive and machine learning analytics(4) New Infrastructure and Tools––––

High performance Computing, Storage, Network
Heterogeneous multi-provider services integration
New Data Centric (multi-stakeholder) service models
New Data Centric security models for trusted infrastructure and data processing
and storage(5) Source and Target– High velocity/speed data capture from variety of sensors and data sources– Data delivery to different visualisation and actionable systems and consumers– Full digitised input and output, (ubiquitous) sensor networks, full digital control
17 July 2013, UvA

Big Data Architecture Brainstorming8

** Big Data Definition: From 5V to 5 Parts (2)Refining Gartner definition• Big Data (Data Intensive) Technologies are targeting to process (1) highvolume, high-velocity, high-variety data (sets/assets) to extract intended
data value and ensure high-veracity of original data and obtained
information that demand new cost-effective, innovative forms of data andinformation processing (analytics) for enhanced insight, decision making,and processes control; all of those demand (should be supported by) new
data models (supporting all data states and stages during the whole data
lifecycle) and new infrastructure services and tools that allows also
obtaining (and processing data) from a variety of sources (including
sensor networks) and delivering data in a variety of forms to different dataand information consumers and devices.
(1) Big Data Properties: 5V(2) New Data Models(3) New Analytics(4) New Infrastructure and Tools(5) Source and Target
17 July 2013, UvA

Big Data Architecture Brainstorming

9

Overview: Technology Definitions and Timeline•

Service Oriented Architecture (SOA): First proposed in 1996 and revived with the
Web Services advent in 2001-2002– Currently standard for industry, and widely used
– Provided a conceptual basis for Web Services development•

Computer Grids: Initially proposed in 1998 and finally shaped in 2003 with theOpen Grid Services Architecture (OGSA) by Open Grid Forum (OGF)– Currently remains as a collaborative environment– Migrates to cloud and inter-cloud platform•

Cloud Computing: Initially proposed in 2008
– Defined new features, capabilities, operational/usage models and actually provided a
guidance for the new technology development– Originated from the Service Computing domain and service management focused•

Big Data: Yet to be defined– Involves more components and processes to be included into the definition– Can be better defined as ecosystem where data are the main driving factor/component– Need to define the Big Data properties, expected technology capabilities and provide a
guidance/vision for future technology development

17 July 2013, UvA

Big Data Architecture Brainstorming10

** Big Data Nature: Origin and consumers (target)
Big Data Origin• Science• Telecom• Industry• Business• Living Environment,
Cities• Social media andnetworks• Healthcare

17 July 2013, UvA

Big Data Target Use• Scientific discovery• New technologies• Manufacturing,processes, transport• Personal services,
campaigns• Living environment
support• Healthcare support

Big Data Architecture Brainstorming

11

Big Data Nature: Origin and consumers (targets)
Scietific
Discovery

New
Technology

Manufactur
Transport

Personal
services,
campaigns

LivingEnvironmnt,
Infrastruct,
Utility

Healthcare
supportScience

+++++

++++

+

-

++

+++

Telecom

+

++++

++

+

++++

+

Industry

++

++++

+++++

-

-

++

Business

+

+++

++

-

+

++

Livingenvironment,
Cities

++

++

++

++

+++++

+

Social media,
networks

+

++

-

++++

++

-

Healthcare

+++

++

-

-

++

+++++

17 July 2013, UvA

Big Data Architecture Brainstorming12** From Big Data to All-Data – Paradigm Change• Really paradigm changing
factor– Data storage and processing– Security– Identification and provenance?
Big Data

Network

Big
Computer?

• Traditional model– BIG Storage and BIG computerwith FAT pipe
– Move compute to data vs Movedata to compute

Distributed Big Data
Storage

Data Bus

Visu
alisa
tion

• New Paradigm– Continuous data production– Continuous data processing
17 July 2013, UvA

Distributed Compute

Big Data Architecture Brainstorming13

* Moving to Data-Centric Models and Technologies• Current IT and communication technologies arehost based or host centric– Any communication or processing is bound to
host/computer that runs software– Especially in security: all security models are host/client
based• Big Data requires new data-centric models– Data location, search, access– Data security and access control– Data integrity and identifiability

17 July 2013, UvA

Big Data Architecture Brainstorming

14

* Data Centric Security• Paradigm shift to data centric security model– Previous and current security models are host or domain
based• New challenges and new security models– Data ownership– Data centric access control• Encryption enforced access control– Personally identified data, privacy, opacity– Trusted virtualisation platform• Dynamic trust bootstrapping

17 July 2013, UvA

Big Data Architecture Brainstorming

15

Defining Big Data Architecture Framework• Existing attempts don’t converge to something
consistent: ODCA, TMF, NIST
– See Appendix• Architecture vs Ecosystem– Big Data undergo and number of transformation during
their lifecycle– Big Data fuel the whole transformation chain• Architecture vs Architecture Framework (Stack)– Separates concerns and factors– Architecture Framework components are inter-related

17 July 2013, UvA

Big Data Architecture Brainstorming16

* Missing Component – Data Model and Lifecycle• Scientific Data and Scientific Data Lifecycle
Management (SDLM) model– Preservation is an important issue• General Big Data Lifecycle model– Actionable Data
– Not necessary preservation is a key issue– Process control, actions, etc.

17 July 2013, UvA

Big Data Architecture Brainstorming17

* Data Model: Data and Information

Model

Data (raw)

•
•

•
•
•

Metadata
Relations
Functions

Information

Presentation

Data: The lowest layer of abstraction (?) from which information can bederivedInformation: A combination of contextualised data that can provide meaningfulvalue or usage/action (scientific, business)– Actionable data••

Presentation (?)
Where is knowledge (as a target of learning)?

17 July 2013, UvA

Big Data Architecture Brainstorming18

* Data Transformation Model
Data model types?

Data/Process model(s)DatasetID={PID+Pfj}
Metadata

Data
SourceModel data,
statistical data

Metadata

PID

Metadata

Data (raw)
PID

Data
Source17 July 2013, UvA

Data
Filter/Enrich,Classification

PID

Datasets

PID

Metadata

Data (archival,
actionable)

Data (structured,datasets)
Data
Collection
andRegistratn

ModelID?=?

Data
Analytics,
Modeling,
Prediction

Security issues•• CIA and Access control ••Big Data Architecture Brainstorming

Data
Delivery,
Visualisation

Referral integrity
Traceability
Opacity

Visualisedmodels;Biz reports,
Trends;Controlled
Processes;
SocialActions

Consumer

PID=UID+time+Prj

19

** Big Data Architecture Framework (BDAF) –Target and Context for the discussion• Data Models and Structures– Data types• Big Data Lifecycle (Management) Model– Big Data transformation/staging• Big Data Infrastructure (BDI)– Storage, Compute, (High Performance Computing,) Network– Sensor network, target/actionable devices• Big Data Analytics/Tools• Big Data Applications– Target use, actionable data, presentation, visualisation• Big Data Management/Operation– Provenance, Curation, Archiving, Operational support• Big Data Security– Data Security in-rest, in-move, trusted processing environments
17 July 2013, UvA

Big Data Architecture Brainstorming

20

Big Data Architecture Framework (BDAF) –Relations between components (2)Col: Used By
Row:
Requires This

Data
Models
Structrs

Data
ModelsData
Lifecycle

BigData
Infrastr

BigData
Analytics

BigData
Aplicatn

BigData
MangntOperatn

BigDataSecurity

+++

++

+++

+++

+++

+++

+++

++

++

+++

+++

++

++

+++

+++

+++

+

++

++

++

Data
Lifecycle+++

BigDataInfastruct

+++

+++

BigData
Analytics

+++

+

++

BigData
Applicatn

++

+

+++

++

BigData
Mangnt

+++

+++

+++

+

++

BigDataSecurity

+++

+++

+++

+

+

17 July 2013, UvA

Big Data Architecture Brainstorming

+++
++

21

** Big Data Architecture Framework (BDAF) –Aggregated (1)(1) Data Models, Structures, Types– Data formats, non/relational, file systems, etc.

(2) Big Data Management– Big Data Lifecycle (Management) Model• Big Data transformation/staging– Provenance, Curation, Archiving(3) Big Data Analytics and Tools– Big Data Applications• Target use, presentation, visualisation(4) Big Data Infrastructure (BDI)– Storage, Compute, (High Performance Computing,) Network– Sensor network, target/actionable devices– Big Data Operational support(5) Big Data Security– Data security in-rest, in-move, trusted processing environments
17 July 2013, UvA

Big Data Architecture Brainstorming22

** Big Data Architecture Framework (BDAF) –Aggregated – Relations between components (2)Col: Used By
Row: Requires
This

Data
Models
Structrs

Data Models& Structures

Data
Managmnt& Lifecycle

BigData
Infrastr &Operations

BigData
BigData
Analytics & Security
Applicatn

+

++

+

++

++

++

++

++

+++

Data
Managmnt &Lifecycle++

BigData
Infrastruct &Operations

+++

+++

BigData
Analytics &Applications

++

+

++

BigDataSecurity

+++

+++

+++

17 July 2013, UvA

Big Data Architecture Brainstorming

++

+

23

Data Models, Structure, Types• Data structures– Structured data
– Unstructured data

• Data types [ref]– (a) data described via a formal data model– (b) data described via a formalized grammar– (c) data described via a standard format– (d) arbitrary textual or binary data

• Data models– Depend on target/goal, or process/object?
– Evolve or chain/stack?
[ref] NIST Big Data WG discussion http://bigdatawg.nist.gov/home.php

17 July 2013, UvA

Big Data Architecture Brainstorming24

** Evolutional/Hierarchical Data Model
Actionable Data

Papers/Reports

Archival Data

Usable Data

Processed Data (for target use)
Processed Data (for target use)
Processed Data (for target use)

Classified/Structured DataClassified/Structured DataClassified/Structured Data

Raw Data••••

Common Data Model?
Data interlinking?
Fits to Graph data type?
Metadata

17 July 2013, UvA

•
•
•
•

Referrals
Control information
PolicyData patterns

Big Data Architecture Brainstorming

25

Data
Collection&
Registration

Data
SourceData
Filter/Enrich,Classification

Data
Analytics,
Modeling,
Prediction

Data
Delivery,
Visualisation

Consumer** Big Data Ecosystem: Data, Lifecycle,
Infrastructure

Big Data Target/Customer: Actionable/Usable Data
Target users, processes, objects, behavior, etc.
Federated
AccessandDeliveryInfrastructure(FADI)Big Data Source/Origin (sensor, experiment, logdata, behavioral data)

Big Data Analytic/Tools

Storage
GeneralPurposeData Management

ComputeGeneralPurposeHigh
Performance
Computer
Clusters

StorageSpecialised
Databases
Archives(analytics DB,
In memory,
operstional)

Data
categories: metadata,(un)structured, (non)identifiable
Data
Datacategories:
categories:metadata,
metadata,(un)structured,(un)structured,(non)identifiable(non)identifiable

Intercloud multi-provider heterogeneous Infrastructure
Security Infrastructure

17 July 2013, UvA

Network InfrastructureInternal

InfrastructureManagement/Monitoring

Big Data Architecture Brainstorming26

** Big Data Infrastructure and Analytic Tools
Big Data Target/Customer: Actionable/Usable Data
Target users, processes, objects, behavior, etc.
Big Data Source/Origin (sensor, experiment, logdata, behavioral data)

Big Data Analytic/Tools
Analytics:
Refinery, Linking, Fusion

Analytics :Realtime, Interactive,
Batch, Streaming

Storage
GeneralPurposeData Management

ComputeGeneralPurposeAnalytics Applications
Link Analysis
Cluster Analysis
Entity Resolution
Complex Analysis

High
Performance
Computer
Clusters:

Federated
AccessandDeliveryInfrastructure(FADI)

StorageSpecialised
Databases
Archives

Data
categories: metadata,(un)structured, (non)identifiable
Data
Datacategories:
categories:metadata,
metadata,(un)structured,(un)structured,(non)identifiable(non)identifiable

Intercloud multi-provider heterogeneous Infrastructure
Security Infrastructure

17 July 2013, UvA

Network InfrastructureInternal

InfrastructureManagement/Monitoring

Big Data Architecture Brainstorming27

** Data Transformation/Lifecycle Model
Common Data Model?
Data Model (1)Data Model (3)Data Model (4)Data (inter)linking?

Data
Collection&
Registration

Data
SourceData
Filter/Enrich,Classification

Data
Analytics,
Modeling,
Prediction

Data
Delivery,
Visualisation

Consumer
Data Analitics
Application

Data Model (1)Data repurposing,Analitics re-factoring,
Secondary processing••

Does Data Model changes along lifecycle or data evolution?
Identifying and linking data–––

Persistent identifier
Traceability vs Opacity
Referral integrity

17 July 2013, UvA

Big Data Architecture Brainstorming

28

Data Stored on the Big Data Infrastructure• Plain, distributed, hierarchical, relational, graph data
– Streaming data (?)• Protected data
– Encrypted data
– Masked data (scrambled, padded, manipulated, etc.)– Anonymised and privacy enhanced– Identifiable and non-identifiable– Policy attached/enforced• Tiered/auto-tired

17 July 2013, UvA

Big Data Architecture Brainstorming

29Gap Analysis and Requirements to Big Data
Technologies• Based on the collection of use cases analysis• To validate the Big Data definition and Big Data
Architecture Framework definition• To be defined in a technology agnostic way– Done for the required capabilities, not selected
technologies

17 July 2013, UvA

Big Data Architecture Brainstorming

30

Big Data and Data Intensive Science• Scientific Data types• Scientific Data Lifecycle Management (SDLM)• Scientific Data Infrastructure (SDI)

17 July 2013, UvA

Big Data Architecture Brainstorming31

* Scientific Data Types
EC Open Access Initiative
Requires data linking at alllevels and stages

Publicationsand Linked
Data

Published Data

Structured Data
Raw Data

17 July 2013, UvA• Raw data collected from observationand from experiment (according to an
initial research model)• Structured data and datasets that went
through data filtering and processing(supporting some particular formal
model)• Published data that supports one or
another scientific hypothesis, research
result or statement• Data linked to publications to support
the wide research consolidation,
integration, and openness.

Big Data Architecture Brainstorming32

** Scientific Data Lifecycle Management (SDLM)
Model
Data Lifecycle Model in e-Science

User
Researcher

Data discovery

Data Curation(including retirement and clean up)
Data
recycling

Raw Data
Experimental
Data

Project/
Experiment
Planning

Data collection
andfiltering

Structured
Scientific
Data

Data analysis

DB

Data archiving

Data
Re-purpose

Data linkage
to papers

Data sharing/
Data publishing

Data Re-purpose
Data Linkage Issues•
Persistent Identifiers (PID)•
ORCID (Open Researcher andContributor ID)•
Lined Data

Data Clean up and Retirement•
Ownership and authority•
Data Detainment

Big Data Architecture Brainstorming

End of project

Open
PublicUse

Data Links

17 July 2013, UvA

Dataarchiving

Metadata &
Mngnt

33

Additional Information• Existing proposed Big Data architectures• e-Science and Scientific Data Infrastructure (SDI)• Cloud computing as a platform for SDI

17 July 2013, UvA

Big Data Architecture Brainstorming

34

Industry Initiatives to define Big Data (Architecture)• Open Data Center Alliance (ODCA) Information as aService (INFOaaS)• TMF Big Data Analytics Reference Architecture• Research Data Alliance (RDA)– All data related aspects, but not Infrastructure and tools• NIST Big Data Working Group (NBD-WG)– Range of activities

17 July 2013, UvA

Big Data Architecture Brainstorming

35

ODCA INFOaaS – Information as a Service•

Using integrated/unified
storage– New DB/storage
technologies allow
storing data during all
lifecycle[ref] Open Data Center Alliance
Master Usage model: Information
as a Service, Rev 1.0.
http://www.opendatacenteralliance.
org/docs/Information_as_a_Servic
e_Master_Usage_Model_Rev1.0.p
df

17 July 2013, UvA

Big Data Architecture Brainstorming

36

ODCA Example INFOaaS Architecture••Core Data and Information
Components
Data Integration and Distribution
Components

17 July 2013, UvA••

Presentation and Information Delivery
Components
Control and Support Components

Big Data Architecture Brainstorming

37

TMF Big Data Analytics Architecture[ref] TR202 Big Data
Analytics Reference
Model. Version 1.9, April
2013.

17 July 2013, UvA

Big Data Architecture Brainstorming

38

NIST Big Data Working Group (NBD-WG)• Deliverables target – September 2013
• Activities: Conference calls every day 17-19:00 (CET) by
subgroup - http://bigdatawg.nist.gov/home.php– Big Data Definition and Taxonomies– Requirements (chair: Jeffrey Fox)
– Big Data Security– Reference Architecture– Technology Roadmap• BigdataWG mailing list and useful documents– Input documents http://bigdatawg.nist.gov/show_InputDoc2.php– Brainstorming summary and Lessons learnt (from brainstorming)
http://bigdatawg.nist.gov/_uploadfiles/M0010_v1_6762570643.pdf– Big Data Ecosystem Reference Architecture (Microsoft)
http://bigdatawg.nist.gov/_uploadfiles/M0015_v1_1596737703.docx
17 July 2013, UvA

Big Data Architecture Brainstorming

39

NIST Proposed Reference Architecture••

Obviously not data centric
Doesn’t make data (lifecycle) management clear[ref] NIST Big Data WG mailing list discussion
http://bigdatawg.nist.gov/_uploadfiles/M0010_v1_6762570643.pdf
17 July 2013, UvA

Big Data Architecture Brainstorming

40

Big Data Ecosystem Reference Architecture (ByMicrosoft) [ref][ref] Big Data Ecosystem Reference Architecture (Microsoft)
http://bigdatawg.nist.gov/_uploadfiles/M0015_v1_1596737703.docx
17 July 2013, UvA

Big Data Architecture Brainstorming

41

LexisNexis Vision for Data Analytics
Supercomputer (DAS) [ref][ref] HPCC Systems: Introduction to HPCC (High Performance Computer Cluster), Author: A.M.
Middleton, LexisNexis Risk Solutions, Date: May 24, 201117 July 2013, UvA

Big Data Architecture Brainstorming42

LexisNexis HPCC System
Architecture
ECL – Enterprise Data Control
Language
THOR Processing Cluster (Data
Refinery)Roxie Rapid Data Delivery Engine[ref] HPCC Systems: Introduction
to HPCC (High Performance
Computer Cluster), Author: A.M.
Middleton, LexisNexis Risk
Solutions, Date: May 24, 201117 July 2013, UvA

Big Data Architecture Brainstorming

43

IBM GBS Business Analytics and Optimisation (2011).
https://www.ibm.com/developerworks/mydeveloperworks/files/basic/anonymous/api/library/48d92427-47d3-4e75-b54cb6acfbd608c0/document/aa78f77c-0d57-4f41-a92350e5c6374b6d/media&ei=yrknUbjMNM_liwKQhoCQBQ&usg=AFQjCNF_Xu6aifcAhlF4266xXNhKfKaTLw&sig2=j8JiFV_md5DnzfQl0spVrg&bvm=bv.4276
8644,d.cGE

17 July 2013, UvA

Big Data Architecture Brainstorming

44

BCP in Cloud/Intercloud Architecture Definition• NIST Cloud Computing Reference Architecture (CCRA)– Service oriented and IT/Cloud Service Management focused– NIST SP 500-292, Cloud Computing Reference Architecture, v1.0. [Online]
http://www.nist.gov/customcf/get_pdf.cfm?pub_id=909505• Intercloud Architecture Framework (ICAF) by University of Amsterdam– Leverages modern Internet (IETF, ITU-T, TMF) and SOA best practices– Intercloud Architecture for Interoperability and Integration. By Demchenko, Y.,
C.Ngo, M.Makkes, R.Strijkers, C. de Laat. In Proc. The 4th IEEE Conf. on
Cloud Computing Technologies and Science (CloudCom2012), 3 - 6
December 2012, Taipei, Taiwan.
– Cloud Reference Framework, IEFT Draft, 3 July 2013.
http://tools.ietf.org/html/draft-khasnabish-cloud-reference-framework-05

17 July 2013, UvA

Big Data Architecture Brainstorming

45

NIST Cloud Computing Reference Architecture(CCRA) 2.0 – Consolidated View

• txt

27-28 Nov 2012, HK PolyU

Cloud Standardisation

46

InterCloud Architecture Framework (ICAF) Components• Multi-layer Cloud Services Model (CSM)– Combines IaaS, PaaS, SaaS into multi-layer model with inter-layer interfaces– Including interfaces definition between cloud service layers and virtualisation platform• InterCloud Control and Management Plane (ICCMP)– Allows signaling, monitoring, dynamic configuration and synchronisation of the
distributed heterogeneous clouds– Including management interface from applications to network infrastructure andvirtualisation platform• InterCloud Federation Framework (ICFF)– Defines set of protocols and mechanisms to ensure heterogeneous clouds integration
at service and business level– Addresses Identity Federation, federated network access, etc.

•

InterCloud Operations Framework (ICOF)– RORA model: Resource, Ownership, Role, Action•

RORA model provides basis for business processes definition, SLA and access control– Broker and federation operation•

InterCloud Security Framework (ICSF)

AINA2013, 28 March 2013

InterCloud Architecture Framework

47

Multilayer Cloud Services Model (CSM)

Security InfrastructureManagement

Operations Support System

User/Client Services* Identity services (IDP)* Visualisation

User/Customer Side Functions and Resources

Administration andManagement Functions(Client)

Content/Data Services* Data * Content * Sensor * Device

1

Endpoint Functions* Service Gateway* Portal/Desktop

Inter-cloud Functions* Registry and Discovery* Federation Infrastructure

Access/Delivery
Infrastructure

SaaS

PaaS

PaaS-IaaS IF
PaaS-IaaS Interface

IaaS – Virtualisation Platform Interface

Cloud Management
Software(Generic Functions)

Cloud Management Platforms
OpenNebula

Virtualisation Platform

OpenStack

KVM

VM

VM

StorageResources

ComputeResources

Contrl&Mngnt Links

VPN

Other
CMS

XEN

VMwareNetwork
Virtualis

Proxy (adaptors/containers) - Component Services and Resources

SURFnet, 7 February 2013

Layer C5
Services
Access/Delivery

Layer C4Cloud Services(Infrastructure,
Platforms,
Applications,
Software)

Cloud Services (Infrastructure, Platform, Application, Software)

IaaS

Layer C6
User/Customer
side Functions

Hardware/PhysicalResources

Network
Infrastructure

Layer C3
Virtual Resources
Composition andControl(Orchestration)

CSM layers(C6) User/Customer side
Functions(C5) Intrecloud Services
Access and Delivery(C4) Cloud Services(Infrastructure, Platform,
Applications)(C3) Virtual Resources
Composition andOrchestration(C2) Virtualisation Layer(C1) Hardware platform anddedicated network
infrastructure

Layer C2
Virtualisation

Layer C1
PhysicalHardwarePlatform andNetwork

Control/
Mngnt Links

Data Links

Data Links

GN3+ Cloud+Slide_48

E-Science Features
•

•

•

•
•

•

Automation of all e-Science processes including data collection, storing,classification, indexing and other components of the general data curation andprovenanceTransformation of all processes, events and products into digital form bymeans of multi-dimensional multi-faceted measurements, monitoring andcontrol; digitising existing artifacts and other content
Possibility to re-use the initial and published research data with possible data
re-purposing for secondary research
Global data availability and access over the network for cooperative group of
researchers, including wide public access to scientific data
Existence of necessary infrastructure components and management tools thatallows fast infrastructures and services composition, adaptation andprovisioning on demand for specific research projects and tasks
Advanced security and access control technologies that ensure secure
operation of the complex research infrastructures and scientific instruments andallow creating trusted secure environment for cooperating groups andindividual researchers.

17 July 2013, UvA

Big Data Architecture Brainstorming

49

General requirements to SDI for emerging Big
Data Science
•
•
•
•

•
•

Support for long running experiments and large data volumes generated
at high speed
Multi-tier inter-linked data distribution and replication
On-demand infrastructure provisioning to support data sets andscientific workflows, mobility of data-centric scientific applications
Support of virtual scientists communities, addressing dynamic usergroups creation and management, federated identity management
Support for the whole data lifecycle including metadata and data source
linkage
Trusted environment for data storage and processing–••

Research need to trust SDI to put all their data on it

Support for data integrity, confidentiality, accountability
Policy binding to data to protect privacy, confidentiality and IPR

17 July 2013, UvA

Big Data Architecture Brainstorming50

Defining Architecture framework for SDI and Security• Scientific Data Lifecycle Management (SDLM) model• e-SDI multi-layer architecture model• RORA model to define relationship between resources and actors– RORA (Resource-Ownership-Role-Actor) model defines relationship between
resources, owners, managers, users– Initially defined for telecom domain– New actors in SDI (and Big Data Infrastructure)• Subject of data (e.g. patient, or scientific object/paper)• Data Manager (doctor, seller)• Security and Access Control and Accounting Infrastructure (ACAI)– Trust management infrastructure– Authentication, Authorisation, Accounting• Supported by logging service– Extended to support data access control and operations on data

17 July 2013, UvA

Big Data Architecture Brainstorming

51

SDI Architecture Model

ScientificDatasetApplicScientific
ApplicScientific
ApplicScientificFederated Access and Delivery
Infrastructure (FADI)

Layer B5
Federated Accessand Delivery
Layer

Shared Scientific Platform and Instruments(specific for scientific areas, also Grid based)

Layer B4
Scientific Platform
and Instruments

Technologies andsolutions
Scientific specialist
applications
Library resources

Optical Network
Infrastructure
Federated Identity
Management:
eduGAIN, REFEDS,
VOMS, InCommon
PRACE/DEISA

Cloud/Grid Infrastructure
Virtualisation and Management
Middleware
ComputeResources

Sensors andDevices

Network infrastructure

17 July 2013, UvA

Layer B6
Scientific
Applications

User/Scientific Applications Layer

User
portals

Metadata and Lifecycle Management

Security and AAI

Operation Support and Management Service (OSMS)

LayersLayer B3
Infrastructure
Virtualisation

Middleware
security

StorageResources

Layer B2
Datacenter andComputing Facility
Layer B1
Network Infrastructure

Big Data Architecture Brainstorming

Grid/Cloud

CloudsAutobahn, eduroam

52

SDI Architecture Layers• Layer D1: Network infrastructure layer represented by the general
purpose Internet infrastructure and dedicated network infrastructure• Layer D2: Datacenters and computing resources/facilities, including
sensor network• Layer D3: Infrastructure virtualisation layer that is represented by the
Cloud/Grid infrastructure services and middleware supporting specialised
scientific platforms deployment and operation• Layer D4: (Shared) Scientific platforms and instruments specific fordifferent research areas• Layer D5: Federated Access and Delivery Infrastructure (FADI) Layer:
Federation infrastructure components, including policy and collaborative
user groups support functionality• Layer D6: Scientific applications and user portals/clients

17 July 2013, UvA

Big Data Architecture Brainstorming

53

SDI move to Clouds• Cloud technologies allow for infrastructure virtualisation andits profiling for specific data structures or to support specificscientific workflows• Clouds provide just right technology for infrastructure virtualisation to
support data sets
• Complex distributed data require infrastructure– Demand for inter-cloud infrastructure• Cloud can provide infrastructure on-demand to support
project related scientific workflows– Similar to Grid but with benefits of the full infrastructure provisioning
on-demand• Software Defined Infrastructure Services– As wider than currently emerging SDN (Software Defined Networks)• Distributed Hadoop clusters for HPC and MPP
17 July 2013, UvA

Big Data Architecture Brainstorming

54Federated Access and Delivery Infrastructure(FADI)
Federated Cloud
Instance Customer A(VO A)

Trust
BrokerTrust
BrokerBrokerBrokerFederated Cloud
Instance Customer B(VO B)
Common FADI
Services

TrustedIntroducer

Discovery

FADI Network Infrastructure

FedIDP

17 July 2013, UvA

Directory

Directory(RepoSLA)(RepoSLA)

Gateway

Gateway

Gateway

Gateway

AAA

AAA

AAA

AAA

(I/P/S)aaS
Provider(I/P/S)aaS
Provider(I/P/S)aaS
Provider

IDP

IDP

IDP

…

Big Data Architecture Brainstorming(I/P/S)aaS
ProviderIDP

55