Towards Defining Big Data Architecture Framework
Yuri Demchenko, Marcel Worring, Wouter Los, Cees de Laat
Big Data Definition: From 5+1 Vs to 5 Parts Definition

Big Data Paradigm Change: Moving to Data-Centric Models(1) Big Data Properties: 5+1 V

NativeProperties

Volume••••Variety••••••

Structured
Unstructured
Multi-factor
Probabilistic
LinkedDynamic

Terabytes
Records/Arch
Tables, Files
Distributed• Changing data
• Changing model• Linkage•••••

Variability
AcquiredProperties

Native properties: Volume, Variety, Velocity•

Acquired properties: Value, Veracity, Variability (Dynamicity)

Current IT and communication technologies are hostbased or host centric (service/message centric)

Move or notto move?

(2) New Data Models

Velocity••••

6 Vs ofBig Data•

Batch
Real/near-time
Processes
Streams

Big Data•

Data Lifecycle and Variability•

Data linking, provenance and referral integrity
Real-time/streaming analytics, interactive and machinelearning analytics(4) Source and Target••••

Trustworthiness
Authenticity
Origin, Reputation
Availability
Accountability•

Correlations
Statistical
EventsHypothetical•

Value•High velocity/speed data capture from variety of sensors anddata sources
Data delivery to different visualisation and actionablesystems and consumers
Full digitised input and output, (ubiquitous) sensor networks,
full digital control(5) New Infrastructure and Tools

Veracity•High performance Computing, Storage, Network•

Heterogeneous multi-provider services integration•

New Data Centric (multi-stakeholder) service models•

New Data Centric security models for trusted infrastructureand data processing and storage

Any communication or processing are bound to host/computerthat runs software•

For security: all security models are host/client based

Big Data requires new data-centric models?

(3) New Analytics•BigComputerNetwork•

Distributed Big DataStorage
Data Abstraction

Visualisation
Presentation
ActionData BusInfrastructure Abstraction
Distributed Computeand Analytics•

Data location, replication, search, access•

Data lifecycle, transformation, variability•

Data integrity, identification, ownership•

Data centric security and access control

Paradigm changing factors

DataBus:(1) Data Container(2) Metadata, State(3) Data Transfer
Protocol•Big Data properties: 5+1 V’s•

Data aggregation: multi-domain, multi-format, variability,linkage, referral integrity•

Policy granularity: variety and complex structure, for theiraccess control processing•

Virtualization: Can improve security of data processing
environment but cannot solve data security “in rest”•

Mobility of the different components of the typical data
infrastructure: data, sensors or data source, data consumer

Refining Gartner definition (http://www.gartner.com/it-glossary/big-data)
Big Data (Data Intensive) Technologies are targeting to process (1) high-volume, high-velocity, high-variety data
(sets/assets) to extract intended data value and ensure high-veracity of original data and obtained information thatdemand (3) cost-effective, innovative forms of data and information processing (analytics) for (4) enhanced insight,decision making, and processes control; all of those demand (should be supported by) (2) new data models(supporting all data states and stages during the whole data lifecycle) and (5) new infrastructure services and toolsthat allows also obtaining (and processing data) from (4) a variety of sources (including sensor networks) anddelivering data in a variety of forms to different data and information consumers and devices.

DataCollection&Registration

DataSource

DataAnalytics,Modeling,Prediction

Data
Filter/Enrich,Classification

Consumer

Big Data Infrastructure (BDI)
DataDelivery,
Visualisation

Big Data Architecture Framework (BDAF) Components(1) Data Models, Structures, Types• Data formats, non/relational, file systems, etc.
(2) Big Data Management• Big Data Lifecycle (Management) Model• Big Data transformation/staging• Provenance, Curation, Archiving(3) Big Data Analytics and Tools• Big Data Analytics Applications• Target use, presentation, visualisation(4) Big Data Infrastructure (BDI)• Storage, Compute, (High Performance Computing),
Network• Sensor network, target/actionable devices• Big Data Operational support(5) Big Data Security• Data security in-rest, in-move, trusted processing
environments

Big Data Analytics Infrastructure

Big Data Target/Customer: Actionable/Usable Data
Target users, processes, objects, behavior, etc.
Federated
AccessandDelivery
Infrastructure(FADI)Big Data Source/Origin (sensor, experiment, logdata, behavioral data)

Big Data Analytic/Tools

Data Management

ComputeGeneralPurpose

High
Performance
ComputerClusters

Target users, processes, objects, behavior, etc.
Big Data Source/Origin (sensor, experiment, logdata, behavioral data)

Big Data Analytic/Tools

StorageSpecialised
Databases
Archives(analytics DB,In memory,operstional)

Analytics:
Refinery, Linking, Fusion

Analytics :Realtime, Interactive,
Batch, Streaming

Data
categories: metadata,(un)structured, (non)identifiable
Data
Datacategories:
categories:metadata,metadata,(un)structured,(un)structured,(non)identifiable(non)identifiable

Inter-cloud multi-provider heterogeneous Infrastructure
Security Infrastructure

Network InfrastructureInternalHigh
Performance
Computer
Clusters

ComputeGeneralPurpose

Storage
GeneralPurposeInfrastructureManagement/Monitoring

Data Management

Big Data Infrastructure (BDI) Components

Analytics Applications
Link Analysis
Cluster Analysis
Entity Resolution
Complex Analysis

Federated
AccessandDelivery
Infrastructure(FADI):

StorageSpecialised
Databases
Archives

Data
Datacategories:
categories:metadata,metadata,(un)structured,(un)structured,(non)identifiable(non)identifiable
Data categories: metadata, (un)structured, (non)identifiable

Intercloud multi-provider heterogeneous Infrastructure

General BDI services and components

Big Data Analytics Infrastructure• Data Management infrastructure and tools• Registries, search/indexing, ontologies,schemas, namespace• Collaborative Environment (user/groupsmanagements)• High Performance Computer Clusters (HPCC)• Specialised Storage, Distributed/Replicated,Archives, Databases, SQL/NoSQL

Network InfrastructureInternalSecurity Infrastructure

InfrastructureManagement/Monitoring

Big Data Lifecycle Management (BDLM) Model

Big Data Analytics Tools/Applications• Real-time, Interactive, Batch, Streaming
Heterogeneous multi-provider Inter-cloud • Link Analysis, Graph analysis• Cluster Analysis
infrastructure• Compute, Storage, Network (provisioned on- • Entity Resolution• Complex Analysis
demand dynamically scaling)• Security infrastructure (access control, Identity Big Data Source and Target
and policy management, confidentiality,• Scientific Instruments, Sensor network,privacy, trust)Experiments, Technological processes• Federated Access and Delivery Infrastructure • Logdata, web/online activity, social networks(FADI)• Human activity and input (crowdsourcing)• Actionable data, reporting, visualisation

Data Model (1)Data Model (1)Data Model (3)Data Model (4)Data (inter)linking?
• Persistent ID• Identification• Privacy, Opacity

DataSource

Common Data Model?
• Data Variety and Variability• Semantic Interoperability

Data Storage

DataCollection&Registration

Data
Filter/Enrich,Classification

DataAnalytics,
Modeling,
Prediction

DataDelivery,
Visualisation

Consumer
Data Analitics
Application

Storage
GeneralPurposeBig Data Target/Customer: Actionable/Usable Data

Related links[1] Riding the wave: How Europe can gain from the rising tide of scientific data. Final report of the High Level Expert Group on
Scientific Data. October 2010. [online] Available at http://cordis.europa.eu/fp7/ict/e-infrastructure/docs/hlg-sdi-report.pdf[2] Big Data Ecosystem: Architecture Framework and Infrastructure Components. SNE Technical Report SNE-2013-02, 12
September 2013. [online] http://www.uazone.org/demch/worksinprogress/sne-2013-02-techreport-bdaf-draft02.pdf[3] Demchenko, Y., P.Membrey, P.Grosso, C. de Laat, Addressing Big Data Issues in Scientific Data Infrastructure. First International
Symposium on Big Data and Data Analytics in Collaboration (BDDAC 2013). Part of The 2013 Int. Conf. on Collaboration
Technologies and Systems (CTS 2013), May 20-24, 2013, San Diego, California, USA.
[4] NIST Big Data Working Group (NBD-WG). [online] http://bigdatawg.nist.gov/home.php

Data repurposing,Analitics re-factoring,
Secondary processing

Contributing Projects
GEANT3plus JRA1 Task 2 – Network Architectures for Cloud Services - http://www.geant.net/
COMMIT Project - http://www.commit-nl.nl/

Credits: Yuri Demchenko, Marcel Worring, Wouter Los, Cees de Laat
Contact: Yuri Demchenko <y.demchenko@uva.nl>