Big Data Architecture Researchat UvAYuri Demchenko
System and Network Engineering Group, University ofAmsterdam
ISO/IEC SGBD Big Data Technologies Workshop
Part of ISO/IEC Big Data Study Group meeting
13-16 May 2014

Outline• Research on Big Data and Infrastructure technologies at• Big Data definition– From 5 + 1 V’s to 5 parts Big Data Definition• Paradigm change and new challenges– Data centric model and DataBus• Defining Big Data Architecture Framework (BDAF)– From Architecture to Ecosystem to Architecture Framework• Big Data Infrastructure (BDI) and Big Data Lifecycle Managementmodel

ISO/IEC SGBD, 13-16 May2014

Big Data Architecture @ UvASlide_2

Big Data and Security Research at System andNetwork Engineering, University of Amsterdam•Long time research and development on Infrastructure services and facilities– High speed optical networking and data intensive applications– Semantic description of infrastructure and network services– Collaborative systems, Grid, Clouds and currently Big Data•

Focus on Infrastructure definition and services– Software Defined Infrastructure based on Cloud/Intercloud technologies– Dynamically provisioned security infrastructure and services• NIST Big Data Working Group– Contribution to Reference Architecture, Big Data Definition and Taxonomy, Big DataSecurity• Research Data Alliance– Interest Group on Education and Skills Development on Data Intensive Science– Big Data Analytics Interest Group• Big Data Interest Group at UvA– Non-formal but active, meets two-weekly/monthly– Provided input to NIST BD-WG and RDA activities and UvA DSRC
ISO/IEC SGBD, 13-16 May
2014

Big Data Architecture @ UvA

3

Visionaries and Drivers:
Seminal works, High level reports, Activities
The Fourth Paradigm: Data-Intensive Scientific Discovery.
By Jim Gray, Microsoft, 2009. Edited by Tony Hey, et al.
http://research.microsoft.com/en-us/collaboration/fourthparadigm/
Riding the wave: How Europe can gain fromthe rising tide of scientific data.
Final report of the High Level Expert Group on
Scientific Data. October 2010.
http://cordis.europa.eu/fp7/ict/einfrastructure/docs/hlg-sdi-report.pdf

https://www.rd-alliance.org/
NIST Big Data Working Group (NBD-WG)
https://www.rd-alliance.org/ISO/IEC SGBD, 13-16 May
2014

Big Data Architecture @ UvA

AAA Study: Study on
AAA Platforms For
Scientificdata/information
Resources in Europe,TERENA, UvA, LIBER,UinvDeb.
(2011-2012)4

Drivers at SNE/UvA• Ongoing research on Cyber Infrastructure• Demand for education on new emerging
technologies• ENVRI EU project• LifeWatch EU project• EUBrazil Cloud Connect EU-Brazil project– Consortium of 6 Brazilian institutions and 7 Europeaninstitutions– 3 scientific and research use cases

ISO/IEC SGBD, 13-16 May
2014

Big Data Architecture @ UvA

5ISO/IEC SGBD, 13-16 May
2014

Big Data Architecture @ UvA

6

ISO/IEC SGBD, 13-16 May
2014

Big Data Architecture @ UvA

7

ISO/IEC SGBD, 13-16 May
2014

Big Data Architecture @ UvA

8

Gartner Technology Hypercycle (October 2013)

Big Data
Cloud Computing

Source http://www.gartner.com/technology/research/methodologies/hype-cycle.jsp
ISO/IEC SGBD, 13-16 May2014

Big Data Architecture @ UvA

9Our/SNE Big Data Technology Research Cycle
New style of technology development
Technology consolidation

Mid-End 2013

Big Data
Cloud Computing

End 2014

2012
Mid 2014

2011

Active and productive research
Teaching on Big Data Tech/Infra
Remote BD technology following.
EU Study AAA for Research Data
Main research in Cloud/Intercloud

Component technologies mastering
Education courses development

Active research into Big Data domain definition
Building community
Source http://www.gartner.com/technology/research/methodologies/hype-cycle.jsp
ISO/IEC SGBD, 13-16 May2014

Big Data Architecture @ UvA

10

Big Data Definitions Overview•IDC definition of Big Data (conservative and strict approach) :"A new generation of technologies and architectures designed to economically extract value
from very large volumes of a wide variety of data by enabling high-velocity capture,discovery, and/or analysis“•

Gartner definition
Big data is high-volume, high-velocity and high-variety information assets that demand costeffective, innovative forms of information processing for enhanced insight and decision
making. http://www.gartner.com/it-glossary/big-data/–•

Big Data: a massive volume of both structured and unstructured data that is so large thatit's difficult to process using traditional database and software techniques.
–•

From “The Big Data Long Tail” blog post by Jason Bloomberg (Jan 17, 2013). http://www.devx.com/blog/the-big-datalong-tail.html“Data that exceeds the processing capacity of conventional database systems. The data is
too big, moves too fast, or doesn’t fit the structures of your database architectures. To gain
value from this data, you must choose an alternative way to process it.”–•Termed as 3 parts definition, not 3V definition

Ed Dumbill, program chair for the O’Reilly Strata Conference

Termed as the Fourth Paradigm *)“The techniques and technologies for such data-intensive science are so different that it isworth distinguishing data-intensive science from computational science as a new, fourthparadigm for scientific exploration.” (Jim Gray, computer scientist)*) The Fourth Paradigm: Data-Intensive Scientific Discovery. Edited by Tony Hey, Stewart Tansley, and Kristin Tolle. Microsoft, 2009.

ISO/IEC SGBD, 13-16 May
2014

Big Data Architecture @ UvA

11Improved: 5+1 V’s of Big Data
VolumeVariety
•
••
•
•
•

Structured
Unstructured
Multi-factor
Probabilistic
LinkedDynamic••••

Terabytes
Records/ArchTables, Files
Distributed

Velocity••••

6 Vs ofBig Data

• Changing data
• Changing model• Linkage

Variability

Batch
Real/near-time
Processes
Streams

•
•
•
•
•

•
•
•
•

Value

Correlations
Statistical
EventsHypothetical

Trustworthiness
Authenticity
Origin, Reputation
Availability
Accountability

Generic Big Data
Properties• Volume• Variety• Velocity
Acquired Properties(after entering system)• Value• Veracity• Variability
Commonly accepted
3V’s of Big Data

Veracity
ISO/IEC SGBD, 13-16 May
2014

Big Data Architecture @ UvA

12

Big Data Definition: From 5+1V to 5 Parts (1)(1) Big Data Properties: 5V– Volume, Variety, Velocity, Value, Veracity– Additionally: Data Dynamicity (Variability)(2) New Data Models– Data Lifecycle and Variability– Data linking, provenance and referral integrity(3) New Analytics–

Real-time/streaming analytics, interactive and machine learning analytics(4) New Infrastructure and Tools––––High performance Computing, Storage, Network
Heterogeneous multi-provider services integration
New Data Centric (multi-stakeholder) service models
New Data Centric security models for trusted infrastructure and data processingand storage(5) Source and Target– High velocity/speed data capture from variety of sensors and data sources– Data delivery to different visualisation and actionable systems and consumers– Full digitised input and output, (ubiquitous) sensor networks, full digital control
ISO/IEC SGBD, 13-16 May
2014

Big Data Architecture @ UvA

13

Big Data Definition: From 5+1V to 5 Parts (1)(1) Big Data Properties: 5V– Volume, Variety, Velocity, Value, Veracity– Additionally: Data Dynamicity (Variability)(2) New Data Models– Data linking, provenance and referral integrity– Data Lifecycle and Variability/Evolution(3) New Analytics–Real-time/streaming analytics, interactive and machine learning analytics(4) New Infrastructure and Tools––––

High performance Computing, Storage, Network
Heterogeneous multi-provider services integration
New Data Centric (multi-stakeholder) service models
New Data Centric security models for trusted infrastructure and data processingand storage(5) Source and Target– High velocity/speed data capture from variety of sensors and data sources– Data delivery to different visualisation and actionable systems and consumers– Full digitised input and output, (ubiquitous) sensor networks, full digital control
ISO/IEC SGBD, 13-16 May
2014

Big Data Architecture @ UvA

14

Big Data Definition: From 5V to 5 Parts (2)Refining Gartner definition“Big data is (1) high-volume, high-velocity and high-variety information assets thatdemand (3) cost-effective, innovative forms of information processing for (5)enhanced insight and decision making”•Big Data (Data Intensive) Technologies are targeting to process (1) high-volume,high-velocity, high-variety data (sets/assets) to extract intended data value andensure high-veracity of original data and obtained information that demand costeffective, innovative forms of data and information processing (analytics) for
enhanced insight, decision making, and processes control; all of those demand(should be supported by) new data models (supporting all data states and stages
during the whole data lifecycle) and new infrastructure services and tools thatallows also obtaining (and processing data) from a variety of sources (includingsensor networks) and delivering data in a variety of forms to different data andinformation consumers and devices.
(1) Big Data Properties: 5V(2) New Data Models(3) New Analytics(4) New Infrastructure and Tools(5) Source and Target

ISO/IEC SGBD, 13-16 May
2014

Big Data Architecture @ UvA

15

From Big Data to All-Data – Paradigm Change• Breaking paradigm changing factor– Data storage and processing– Security– Identification and provenance• Traditional model– BIG Storage and BIG Computer withFAT pipe
– Move compute to data vs Move datato compute

Move or notto move?

Big DataNetwork?
Distributed Big Data
Storage
Data Abstraction

Data Bus• New Paradigm– Continuous data production– Continuous data processing– DataBus as a Data container andProtocol
ISO/IEC SGBD, 13-16 May
2014

Big
Computer

VisualisationPresentation
ActionInfrastructure Abstraction

Distributed Computeand Analytics

Big Data Architecture @ UvA

DataBus:(1) Data Container(2) Metadata, State(3) Data Transfer
Protocol16

Moving to Data-Centric Models and Technologies• Current IT and communication technologies arehost based or host centric– Any communication or processing are bound to host/computer thatruns software– Especially in security: all security models are host/client based• Big Data requires new data-centric models– Data location, search, access– Data integrity and identification– Data lifecycle and variability– Data centric (declarative) programming models– Data aware infrastructure to support new data formats and data
centric programming models• Data centric security and access control
ISO/IEC SGBD, 13-16 May2014

Big Data Architecture @ UvA

17

Defining Big Data Architecture Framework• Architecture vs Ecosystem– Big Data undergo a number of transformations during
their lifecycle– Big Data fuel the whole transformation chain• Data sources and data consumers, target data usage– Multi-dimensional relations between• Data models and data driven processes• Infrastructure components and data centric services• Architecture vs Architecture Framework– Separates concerns and factors• Control and Management functions, orthogonal factors– Architecture Framework components are inter-related
ISO/IEC SGBD, 13-16 May
2014

Big Data Architecture @ UvA

18

Big Data Architecture Framework (BDAF) (1)(1) Data Models, Structures, Types– Data formats, non/relational, file systems, etc.

(2) Big Data Management– Big Data Lifecycle (Management) Model• Big Data transformation/staging– Provenance, Curation, Archiving(3) Big Data Analytics and Tools– Big Data Applications• Target use, presentation, visualisation(4) Big Data Infrastructure (BDI)– Storage, Compute, (High Performance Computing,) Network– Sensor network, target/actionable devices– Big Data Operational support(5) Big Data Security– Data security in-rest, in-move, trusted processing environments
ISO/IEC SGBD, 13-16 May
2014

Big Data Architecture @ UvA

19

Big Data Architecture Framework (BDAF) –Aggregated – Relations between components (2)Col: Used By
Row: Requires
This

Data
Models
Structrs

Data Models& Structures

Data
Managmnt& Lifecycle

BigData
Infrastr &Operations

BigData
BigData
Analytics & Security
Applicatn+++

+++

++

++

++

++

+++

Data
Managmnt &Lifecycle++

BigData
Infrastruct &Operations

+++

+++

BigData
Analytics &Applications

++

+++

BigDataSecurity

+++

+++

+++

ISO/IEC SGBD, 13-16 May
2014

Big Data Architecture @ UvA

++

+20

Data
SourceData
Collection&Registration

Data
Filter/Enrich,Classification

Data
Analytics,
Modeling,
Prediction

Data
Delivery,
VisualisationConsumer

Big Data Ecosystem: Data, Transformation,Infrastructure

Big Data Target/Customer: Actionable/Usable Data

Target users, processes, objects, behavior, etc.
Federated
AccessandDelivery
Infrastructure(FADI)Big Data Source/Origin (sensor, experiment, logdata, behavioral data)

Big Data Analytic/Tools

Storage
GeneralPurposeData Management

ComputeGeneral
PurposeHigh
Performance
Computer
Clusters

StorageSpecialised
Databases
Archives(analytics DB,
In memory,operstional)

Data
categories: metadata,(un)structured, (non)identifiable
Data
Datacategories:
categories:metadata,
metadata,(un)structured,(un)structured,(non)identifiable(non)identifiable

Intercloud multi-provider heterogeneous Infrastructure
Security Infrastructure

ISO/IEC SGBD, 13-16 May
2014

Network InfrastructureInternalInfrastructureManagement/Monitoring

Big Data Architecture @ UvA

21

General BDI services and components• Data management infrastructure and tools• Registries, search/indexing, ontologies, schemas,
namespace• Collaborative Environment (user/groups managements)• Heterogeneous multi-provider Inter-cloud infrastructure– Compute, Storage, Network (provisioned on-demand dynamicallyscaling)– Federated Access and Delivery Infrastructure (FADI)• Advanced high performance (programmable) network• Security infrastructure (access control, Identity and policy
management, confidentiality, privacy, trust)

ISO/IEC SGBD, 13-16 May
2014

Big Data Architecture @ UvA

22

Big Data Infrastructure and Analytic Tools
Big Data Target/Customer: Actionable/Usable Data
Target users, processes, objects, behavior, etc.
Big Data Source/Origin (sensor, experiment, logdata, behavioral data)

Big Data Analytic/Tools
Analytics:
Refinery, Linking, Fusion

Analytics :Realtime, Interactive,Batch, Streaming

StorageGeneral
PurposeData Management

ComputeGeneral
PurposeAnalytics Applications
Link Analysis
Cluster Analysis
Entity Resolution
Complex Analysis

HighPerformance
Computer
Clusters:

Federated
AccessandDelivery
Infrastructure(FADI)

StorageSpecialised
Databases
Archives

Data
categories: metadata,(un)structured, (non)identifiable
Data
Datacategories:
categories:metadata,metadata,(un)structured,(un)structured,(non)identifiable(non)identifiable

Intercloud multi-provider heterogeneous Infrastructure
Security Infrastructure

ISO/IEC SGBD, 13-16 May
2014

Network InfrastructureInternalInfrastructureManagement/Monitoring

Big Data Architecture @ UvA23

Big Data Analytics Infrastructure• High Performance Computer Clusters (HPCC)• Specialised Storage, Distributed/Replicated, Archives,Databases, SQL/NoSQL• Big Data Analytics Tools/Applications• Analytics/processing: Real-time, Interactive, Batch,Streaming• Link Analysis, Graph analysis• Cluster Analysis• Entity Resolution• Complex Analysis

ISO/IEC SGBD, 13-16 May
2014

Big Data Architecture @ UvA

24

Data Transformation/Lifecycle Model
Common Data Model?
•Data Variety and Variability•
Semantic Interoperability

Data Model (1)Data Model (1)Data Model (4)Data (inter)linking?
•Persistent ID•
Identification•
Privacy, Opacity

Data Storage

DataCollection&Registration

Data
Filter/Enrich,Classification

DataAnalytics,
Modeling,
Prediction

DataDelivery,
Visualisation

Consumer
Data Analitics
Application

DataSource

Data Model (3)Data repurposing,Analitics re-factoring,
Secondary processing••

Does Data Model changes along lifecycleor data evolution?
Identifying and linking data–––

Traceability vs Opacity
Referral integrity

Persistent identifier

ISO/IEC SGBD, 13-16 May
2014

Big Data Architecture @ UvA

25

Scientific Data Lifecycle Management (SDLM)
Model
Data Lifecycle Model in e-Science

UserResearcher

Data discoveryData Curation(including retirement and clean up)
Datarecycling

Raw Data
Experimental
DataProject/
Experiment
Planning

Data collectionandfiltering

StructuredScientific
Data

Data analysis

DB

Data archiving

DataRe-purpose

Data linkage
to papers

Data sharing/
Data publishing

Data Re-purpose
Data Linkage Issues•
Persistent Identifiers (PID)•
ORCID (Open Researcher andContributor ID)•
Lined Data

Data Clean up and Retirement•Ownership and authority•Data Detainment

Big Data Architecture @ UvA

End of project

Open
PublicUse

Data Links

ISO/IEC SGBD, 13-16 May2014

DataarchivingMetadata &Mngnt

26

Further Research• Data centric models• DataBus concept and related data centricmechanisms• Data centric security and NoSQL security• Big Data curriculum development and coordination

ISO/IEC SGBD, 13-16 May
2014

Big Data Architecture @ UvA

27

Foreseen Big Data Innovations in 2013+•Cloud-Based Big Data Solutions– Amazon’s Elastic Map Reduce (EMR) is a market leader– Expected new innovative Big Data and Cloud solutions•Real-Time Hadoop– Google’s Dremel-like solutions that will allow real-time queries on Big Data and be opensource•

Distributed Machine Learning– Mahout iterative scalable distributed back propagation machine learning and data mining
algorithm– New algorithms Jubatus, HogWild•

Big Data Appliances (also for home)– Raspberry Pi and home-made GPU clusters– Hardware vendors (Dell, HP, etc.) pack mobile ARM processors into server boxes– Adepteva's Parallella will put a 16-core supercomputer into range of $99•

Easier Big Data Tools– Open Source and easy to use drag-and-drop tools for Big Data Analytics to facilitate the BD
adoption– Commercial examples: Radoop = RapidMiner + Mahout, Tableau, Datameer, etc.
– LexisNexis: from ECL (Enterprise Control Language) to KEL (Knowledge EngineeringLanguage)

Source: Big Data in 2013 by Mike Guattieri, Forrester
ISO/IEC SGBD, 13-16 May
2014

Big Data Architecture @ UvA28

Evolutional/Hierarchical Data Model
Actionable Data

Papers/Reports

Archival Data

Usable Data

Processed Data (for target use)
Processed Data (for target use)
Processed Data (for target use)

Classified/Structured DataClassified/Structured DataClassified/Structured Data

Raw Data••••Common Data Model?
Data interlinking?
Fits to Graph data type?
MetadataISO/IEC SGBD, 13-16 May
2014

••••

Referrals
Control information
PolicyData patterns

Big Data Architecture @ UvA

29

Data Transformation Model
Data model types?

Data/Process model(s)DatasetID={PID+Pfj}

ModelID?=?

Metadata

Metadata

PID
Data
SourceModel data,
statistical data

Metadata

Data (raw)
PID

Data
SourceISO/IEC SGBD, 13-16 May
2014

Data
Filter/Enrich,Classification

Datasets

PID

Metadata

Data (archival,actionable)

Data (structured,datasets)
Data
Collection
andRegistratn

PID

Data
Analytics,
Modeling,
Prediction

Security issues•• CIA and Access control ••Big Data Architecture @ UvA

Data
Delivery,
Visualisation

Referral integrity
Traceability
Opacity

Visualisedmodels;Biz reports,
Trends;Controlled
Processes;
SocialActions

Consumer

PID=UID+time+Prj

30