Overview NIST Big Data Working Group Activities
andBig Data Architecture Framework (BDAF) by UvAYuri Demchenko
SNE Group, University of Amsterdam
Big Data Analytics Interest Group
17 September 2013, 2nd RDA Plenary

Outline• Overview NIST Big Data Working Group (NBD-WG) activities anddeliverables• Proposed Big Data Architecture Framework (BDAF)– Data Models and Big Data Lifecycle– Big Data Infrastructure (BDI)• Discussion: Liaison and information exchange with NIST BD-WG

Disclaimer: Presented here information about NIST Big Data Working Group(NBD-WG) and images from the NBD-WG working documents are not official
position of the NBD-WG and are solely the authors opinion.
17 September 2013, RDA-CWGBDANIST BD-WG and UvA BDAF

Slide_2

NIST Big Data Working Group (NBD-WG)• Deliverables target – September 2013– 26 September – initial draft documents– 30 September – Workshop and F2F meeting• Activities: Conference calls every day 17-19:00 (CET) bysubgroup - http://bigdatawg.nist.gov/home.php– Big Data Definition and Taxonomies– Requirements (chair: Geoffrey Fox, Indiana Univ)– Big Data Security– Reference Architecture– Technology Roadmap• BigdataWG mailing list and useful documents– Input documents http://bigdatawg.nist.gov/show_InputDoc2.php– Big Data Reference Architecture
http://bigdatawg.nist.gov/_uploadfiles/M0226_v2_1885676266.docx– Requirements for 21 usecases
http://bigdatawg.nist.gov/_uploadfiles/M0224_v1_1076079077.xlsx
17 September 2013, RDACWG-BDANIST BD-WG and UvA BDAF3NIST Proposed Reference Architecture (beforeJuly 2013)••

Obviously not data centric
Doesn’t make data (lifecycle) management clear[ref] NIST Big Data WG mailing list discussion
http://bigdatawg.nist.gov/_uploadfiles/M0010_v1_6762570643.pdf
17 September 2013, RDACWG-BDANIST BD-WG and UvA BDAF

4

Big Data Ecosystem Reference Architecture (ByMicrosoft) [ref] – Initial contribution July 2013[ref] Big Data Ecosystem Reference Architecture (Microsoft)
http://bigdatawg.nist.gov/_uploadfiles/M0015_v1_1596737703.docx
17 September 2013, RDACWG-BDANIST BD-WG and UvA BDAF

5

NIST Reference Architecture version 0.0(August 2013)
VOLUM
E

VARIETY

DataUsage
17 September 2013, RDACWG-BDA

Lifecycle Management

System Management

Security & Privacy Management

Hardware (Storage, Networking, etc.) Management

Usage Service Abstraction

ClusterAccess

IaaS

Visualization

Platform

Analytic

PaaS

Curation

SoftwareCollection

SaaS

Transformation

Capability Service Abstraction

Data Service Abstraction

Transitional Framework

Capability Management

VELOCITY

Cloud C. Framework

Data
Sources

RETRIEVEREPORT

RENDERING

NIST BD-WG and UvA BDAF

6

NIST Reference Architecture version 0.1(September 2013)
Capabilities Provider
Big Data Framework

SW

DATA

Data Service Abstraction

Scalable Applications

Collection
Curation
Analytics
Visualization

AccessDATASW

Capabilities Service Abstraction(analytic tools, etc.)

Transformation Provider
System Service Abstraction

SW

Usage Service Abstraction

DATA

System Manager or Vertical Orchestrator

I N F O R M AT I O N F L O W / V A L U E C H A I N

Data Provider

Legacy Applications
Scalable Platforms(databases, etc.)
Legacy Platforms

Scalable
Infrastructures (VMcluster, etc.)
Legacy Infrastructures

Hardware(Storage,Networking, etc.)

Data Consumer

KEY:
Service Use

DATA

Big Data
Information FlowSW

SW Tools andAlgorithms Transfer

17 September 2013, RDACWG-BDAI T S TA C K / V A L U E C H A I NNIST BD-WG and UvA BDAF

7

Big Data Architecture Framework (BDAF)
by the University of Amsterdam• Big Data definition: from 5+1Vs to 5 parts• Big Data Architecture Framework (BDAF)
components

17 September 2013, RDACWG-BDANIST BD-WG and UvA BDAF

8

Improved: 5+1 V’s of Big Data
VolumeVariety
•
•
•
•
•
•

Structured
Unstructured
Multi-factor
Probabilistic
LinkedDynamic••••

Terabytes
Records/Arch
Tables, Files
Distributed
•
•
•
•

Batch
Real/near-time
Processes
Streams

6 Vs ofBig DataValue• Changing data
• Changing model• Linkage

Variability

Velocity•••••••••Correlations
StatisticalEvents
HypotheticalGeneric Big DataProperties• Volume• Variety• Velocity
Acquired Properties(after entering system)• Value• Veracity• Variability

Trustworthiness
Authenticity
Origin, Reputation
Availability
Accountability

Veracity
17 September 2013, RDACWG-BDANIST BD-WG and UvA BDAF

9

Big Data Definition: From 5+1V to 5 Parts (1)(1) Big Data Properties: 5V– Volume, Variety, Velocity, Value, Veracity– Additionally: Data Dynamicity (Variability)(2) New Data Models– Data Lifecycle and Variability– Data linking, provenance and referral integrity(3) New Analytics–Real-time/streaming analytics, interactive and machine learning analytics(4) New Infrastructure and Tools––––

High performance Computing, Storage, Network
Heterogeneous multi-provider services integration
New Data Centric (multi-stakeholder) service models
New Data Centric security models for trusted infrastructure and data processing
and storage(5) Source and Target– High velocity/speed data capture from variety of sensors and data sources– Data delivery to different visualisation and actionable systems and consumers– Full digitised input and output, (ubiquitous) sensor networks, full digital control
17 September 2013, RDACWG-BDANIST BD-WG and UvA BDAF10

Big Data Definition: From 5V to 5 Parts (2)Refining Gartner definition• Big Data (Data Intensive) Technologies are targeting to process (1) highvolume, high-velocity, high-variety data (sets/assets) to extract intended
data value and ensure high-veracity of original data and obtained
information that demand cost-effective, innovative forms of data andinformation processing (analytics) for enhanced insight, decision making,and processes control; all of those demand (should be supported by) new
data models (supporting all data states and stages during the whole datalifecycle) and new infrastructure services and tools that allows also
obtaining (and processing data) from a variety of sources (includingsensor networks) and delivering data in a variety of forms to different dataand information consumers and devices.
(1) Big Data Properties: 5V(2) New Data Models(3) New Analytics(4) New Infrastructure and Tools(5) Source and Target
17 September 2013, RDACWG-BDANIST BD-WG and UvA BDAF11

Big Data Nature: Origin and consumers (target)
Big Data Origin• Science• Telecom• Industry• Business• Living Environment,Cities• Social media andnetworks• Healthcare

17 September 2013, RDACWG-BDA

Big Data Target Use• Scientific discovery• New technologies• Manufacturing,processes, transport• Personal services,
campaigns• Living environment
support• Healthcare support

NIST BD-WG and UvA BDAF

12

Big Data Nature: Origin and consumers (targets)
ScietificDiscovery

New
Technology

Manufactur
Transport

Personalservices,
campaigns

LivingEnvironmnt,
Infrastruct,
Utility

Healthcare
supportScience++++++++++-+++++

Telecom+++++++++++++Industry++

++++

+++++

--++

Business++++++-+++

Livingenvironment,Cities

++

++

++

++

++++++

Social media,networks+++

-++++

++

-Healthcare+++++

--++

+++++

Rich information on usecases is available from the NIST document store
http://bigdatawg.nist.gov/show_InputDoc.php
17 September 2013, RDACWG-BDANIST BD-WG and UvA BDAF13

Moving to Data-Centric Models and Technologies• Current IT and communication technologies arehost based or host centric– Any communication or processing are bound to host/computer thatruns software– Especially in security: all security models are host/client based• Big Data requires new data-centric models– Data location, search, access– Data variability and lifecycle– Data integrity and identification– Data centric security and access control

17 September 2013, RDACWG-BDANIST BD-WG and UvA BDAF14

Defining Big Data Architecture Framework• Existing attempts don’t converge to consistent view: ODCA, TMF, NIST
– See http://bigdatawg.nist.gov/_uploadfiles/M0055_v1_7606723276.pdf• Big Data Architecture Framework (BDAF) by UvAArchitecture Framework and Components for the Big Data Ecosystem.
Draft Version 0.2
http://www.uazone.org/demch/worksinprogress/sne-2013-02-techreport-bdafdraft02.pdf• Architecture vs Ecosystem– Big Data undergo a number of transformations during their lifecycle– Big Data fuel the whole transformation chain• Data sources and data consumers, target data usage– Multi-dimensional relations between• Data models and data driven processes• Infrastructure components and data centric services• Architecture vs Architecture Framework (Stack)– Separates concerns and factors• Control and Management functions, orthogonal factors– Architecture Framework components are inter-related

17 September 2013, RDACWG-BDANIST BD-WG and UvA BDAF

15

Big Data Architecture Framework (BDAF)
for Big Data Ecosystem (BDE)(1) Data Models, Structures, Types– Data formats, non/relational, file systems, etc.

(2) Big Data Management– Big Data Lifecycle (Management) Model• Big Data transformation/staging– Provenance, Curation, Archiving(3) Big Data Analytics and Tools– Big Data Applications• Target use, presentation, visualisation(4) Big Data Infrastructure (BDI)– Storage, Compute, (High Performance Computing,) Network– Sensor network, target/actionable devices– Big Data Operational support(5) Big Data Security– Data security in-rest, in-move, trusted processing environments
17 September 2013, RDACWG-BDANIST BD-WG and UvA BDAF

16

Big Data Architecture Framework (BDAF) –Aggregated – Relations between components (2)Col: Used By
Row: Requires
This

Data
Models
Structrs

Data Models& Structures

DataManagmnt& Lifecycle

BigData
Infrastr &Operations

BigData
BigData
Analytics & Security
Applicatn+

++

+++

++

++

++

++

+++

DataManagmnt &Lifecycle++

BigData
Infrastruct &Operations

+++

+++

BigData
Analytics &Applications+++++

BigDataSecurity+++++++++

17 September 2013, RDACWG-BDANIST BD-WG and UvA BDAF+++17

DataSource

Data
Collection&Registration

Data
Filter/Enrich,Classification

Data
Analytics,Modeling,Prediction

Data
Delivery,
Visualisation

Consumer

Big Data Ecosystem: Data, Lifecycle,
Infrastructure

Big Data Target/Customer: Actionable/Usable Data
Target users, processes, objects, behavior, etc.
FederatedAccessandDelivery
Infrastructure(FADI)Big Data Source/Origin (sensor, experiment, logdata, behavioral data)

Big Data Analytic/Tools

StorageGeneral
PurposeData ManagementComputeGeneralPurpose

HighPerformance
Computer
Clusters

StorageSpecialised
Databases
Archives(analytics DB,In memory,operstional)

Data
categories: metadata,(un)structured, (non)identifiable
Data
Datacategories:
categories:metadata,
metadata,(un)structured,(un)structured,(non)identifiable(non)identifiable

Intercloud multi-provider heterogeneous Infrastructure
Security Infrastructure

17 September 2013, RDACWG-BDA

Network InfrastructureInternal

InfrastructureManagement/Monitoring

NIST BD-WG and UvA BDAF18

Big Data Infrastructure and Analytic Tools
Big Data Target/Customer: Actionable/Usable Data
Target users, processes, objects, behavior, etc.
Big Data Source/Origin (sensor, experiment, logdata, behavioral data)

Big Data Analytic/Tools
Analytics:
Refinery, Linking, Fusion

Analytics :Realtime, Interactive,Batch, Streaming

Storage
GeneralPurposeData Management

ComputeGeneral
PurposeAnalytics Applications
Link Analysis
Cluster Analysis
Entity Resolution
Complex Analysis

High
Performance
Computer
Clusters:

Federated
AccessandDelivery
Infrastructure(FADI)

StorageSpecialised
Databases
Archives

Data
categories: metadata,(un)structured, (non)identifiable
Data
Datacategories:
categories:metadata,
metadata,(un)structured,(un)structured,(non)identifiable(non)identifiable

Intercloud multi-provider heterogeneous Infrastructure
Security Infrastructure

17 September 2013, RDACWG-BDA

Network InfrastructureInternalInfrastructureManagement/Monitoring

NIST BD-WG and UvA BDAF

19

Data Transformation/Lifecycle Model
Common Data Model?
•Data Variety and Variability•
Semantic Interoperability

Data Model (1)Data Model (1)Data Model (4)Data (inter)linking?
•Persistent ID•
Identification•Privacy, Opacity

Data Storage

DataCollection&Registration

Data
Filter/Enrich,Classification

DataAnalytics,
Modeling,
Prediction

DataDelivery,
Visualisation

Consumer
Data Analitics
Application

DataSource

Data Model (3)Data repurposing,Analitics re-factoring,
Secondary processing••

Does Data Model changes along lifecycle or data evolution?
Identifying and linking data

– Persistent identifier– Traceability vs Opacity– Referral integrity
17 September 2013, RDACWG-BDANIST BD-WG and UvA BDAF

20

Scientific Data Lifecycle Management (SDLM)
Model
Data Lifecycle Model in e-Science

User
Researcher

Data discovery

Data Curation(including retirement and clean up)
Datarecycling

Raw Data
Experimental
Data

Project/ExperimentPlanning

Data collection
andfiltering

Structured
ScientificData

Data analysis

DBData archiving

DataRe-purpose

Data linkage
to papers

Data sharing/
Data publishing

Data Re-purpose
Data Linkage Issues•
Persistent Identifiers (PID)•
ORCID (Open Researcher andContributor ID)•Lined Data

Data Clean up and Retirement•
Ownership and authority•Data Detainment

NIST BD-WG and UvA BDAF

End of project

Open
PublicUse

Data Links

17 September 2013, RDACWG-BDA

Dataarchiving

Metadata &Mngnt

21

Evolutional/Hierarchical Data Model
Actionable Data

Papers/Reports

Archival Data

Usable DataProcessed Data (for target use)
Processed Data (for target use)
Processed Data (for target use)

Classified/Structured DataClassified/Structured DataClassified/Structured Data

Raw Data••••

Common Data Model?
Data interlinking?
Fits to Graph data type?
Metadata

17 September 2013, RDACWG-BDA

•
•
•
•

Referrals
Control information
PolicyData patterns

NIST BD-WG and UvA BDAF

22

Additional Information• Existing proposed Big Data architectures

17 September 2013, RDACWG-BDANIST BD-WG and UvA BDAF

23

Industry Initiatives to define Big Data (Architecture)• Open Data Center Alliance (ODCA) Information as aService (INFOaaS)• TMF Big Data Analytics Reference Architecture• Research Data Alliance (RDA)– All data related aspects, but not Infrastructure and tools• LexisNexis HPCC Systems

17 September 2013, RDACWG-BDANIST BD-WG and UvA BDAF24

ODCA INFOaaS – Information as a Service•

Using integrated/unifiedstorage– New DB/storage
technologies allow
storing data during all
lifecycle[ref] Open Data Center Alliance
Master Usage model: Information
as a Service, Rev 1.0.
http://www.opendatacenteralliance.
org/docs/Information_as_a_Servic
e_Master_Usage_Model_Rev1.0.p
df

17 September 2013, RDACWG-BDANIST BD-WG and UvA BDAF

25

ODCA Example INFOaaS Architecture••Core Data and Information
Components
Data Integration and Distribution
Components

17 September 2013, RDACWG-BDA••

Presentation and Information Delivery
Components
Control and Support Components

NIST BD-WG and UvA BDAF

26

TMF Big Data Analytics Architecture[ref] TR202 Big Data
Analytics Reference
Model. Version 1.9, April
2013.

17 September 2013, RDACWG-BDANIST BD-WG and UvA BDAF

27

LexisNexis Vision for Data Analytics
Supercomputer (DAS) [ref][ref] HPCC Systems: Introduction to HPCC (High Performance Computer Cluster), Author: A.M.
Middleton, LexisNexis Risk Solutions, Date: May 24, 201117 September 2013, RDACWG-BDANIST BD-WG and UvA BDAF28

LexisNexis HPCC System
Architecture
ECL – Enterprise Data Control
Language
THOR Processing Cluster (DataRefinery)
Roxie Rapid Data Delivery Engine[ref] HPCC Systems: Introductionto HPCC (High Performance
Computer Cluster), Author: A.M.
Middleton, LexisNexis Risk
Solutions, Date: May 24, 201117 September 2013, RDACWG-BDANIST BD-WG and UvA BDAF29

IBM GBS Business Analytics and Optimisation (2011).
https://www.ibm.com/developerworks/mydeveloperworks/files/basic/anonymous/api/library/48d92427-47d3-4e75-b54cb6acfbd608c0/document/aa78f77c-0d57-4f41-a92350e5c6374b6d/media&ei=yrknUbjMNM_liwKQhoCQBQ&usg=AFQjCNF_Xu6aifcAhlF4266xXNhKfKaTLw&sig2=j8JiFV_md5DnzfQl0spVrg&bvm=bv.4276
178644,d.cGE
September 2013, RDA-CWG-BDANIST BD-WG and UvA BDAF30