Cloud Based Big Data Infrastructure:
Architectural Components and AutomatedProvisioning

The 2016 International Conference on High Performance
Computing & Simulation(HPCS2016)18-22 July 2016, Innsbruck, Austria
Yuri Demchenko, University of Amsterdam
HPCS2016

Cloud based Big Data Infrastructure1Outline•

Big Data and new concepts– Big Data definition and Big Data Architecture Framework (BDAF)– Data driven vs data intensive vs data centric and Data Science•

Cloud Computing as a platform of choice for Big Data applications– Big Data Stack and Cloud platforms for Big Data
– Big Data Infrastructure provisioning automation••CYCLONE project and use cases for cloud based scientific applications
automation
Slipstream and cloud automation tools– Slipstream recipe example•

Discussion

The research leading to these results has received
funding from the Horizon2020 project CYCLONE
HPCS2016

Cloud based Big Data Infrastructure2

Big Data definition revisited: 6 V’s of Big Data
VolumeVariety
•
•
•
•
•
•

Structured
Unstructured
Multi-factor
Probabilistic
LinkedDynamic••••

Terabytes
Records/Arch
Tables, Files
Distributed••••

Adopted in general
by NIST BD-WG
HPCS2016

Batch
Real/near-time
Processes
Streams

6 Vs ofBig Data• Changing data
• Changing model• Linkage

Variability

Velocity

•
•
•
•
•

•
•
•
•

Value

Correlations
Statistical
EventsHypothetical

Trustworthiness
Authenticity
Origin, Reputation
Availability
Accountability

Veracity
Cloud based Big Data Infrastructure

Generic Big Data
Properties• Volume• Variety• Velocity
Acquired Properties(after entering system)• Value• Veracity• Variability
Commonly accepted
3V’s of Big Data
• Volume• Velocity• Variety3Big Data definition: From 6V to 5 Components(1) Big Data Properties: 6V– Volume, Variety, Velocity– Value, Veracity, Variability(2) New Data Models– Data linking, provenance and referral integrity– Data Lifecycle and Variability/Evolution(3) New Analytics–

Real-time/streaming analytics, machine learning and iterative analytics(4) New Infrastructure and Tools––––

High performance Computing, Storage, Network
Heterogeneous multi-provider services integration
New Data Centric (multi-stakeholder) service models
New Data Centric security models for trusted infrastructure and data processing
and storage(5) Source and Target– High velocity/speed data capture from variety of sensors and data sources– Data delivery to different visualisation and actionable systems and consumers– Full digitised input and output, (ubiquitous) sensor networks, full digital control
HPCS2016

Cloud based Big Data Infrastructure4Moving to Data-Centric Models and Technologies• Current IT and communication technologies arehost based or host centric– Any communication or processing are bound to host/computer thatruns software– Especially in security: all security models are host/client based• Big Data requires new data-centric models– Data location, search, access– Data integrity and identification– Data lifecycle and variability– Data centric (declarative) programming models– Data aware infrastructure to support new data formats and data
centric programming models• Data centric security and access control
HPCS2016

Cloud based Big Data Infrastructure5Moving to Data-Centric Models and Technologies• Current
IT and communication
technologies are
RDA developments(2016):
host
host centric
Data based
becomeor
Infrastructure
themselves

Any communicationor processing
are models
bound to andhost/computer•– PID,
Metadata Registries,
Data
formats thatruns software• Data Factories– Especially in security: all security models are host/client based• Big Data requires new data-centric models– Data location, search, access– Data integrity and identification– Data lifecycle and variability– Data centric (declarative) programming models– Data aware infrastructure to support new data formats and data
centric programming models• Data centric security and access control
HPCS2016

Cloud based Big Data Infrastructure

6

NIST Big Data Working Group (NBD-WG) andISO/IEC JTC1 Study Group on Big Data (SGBD)••NIST Big Data Working Group (NBD-WG) is leading the development of the Big Data
Technology Roadmap - http://bigdatawg.nist.gov/home.php– Built on experience of developing the Cloud Computing standards fully accepted byindustry
Set of documents published in September 2015 as NIST Special Publication NIST SP 1500:
NIST Big Data Interoperability Framework (NBDIF)
http://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.1500-1.pdf
Volume 1: NIST Big Data Definitions
Volume 2: NIST Big Data Taxonomies
Volume 3: NIST Big Data Use Case & Requirements
Volume 4: NIST Big Data Security and Privacy Requirements
Volume 5: NIST Big Data Architectures White Paper Survey
Volume 6: NIST Big Data Reference Architecture
Volume 7: NIST Big Data Technology Roadmap•

NBD-WG defined 3 main components of the newtechnology:–––

HPCS2016

The Big Data Paradigm consistsof the distribution of data systems
across horizontally-coupledindependent resources to achieve
the scalability needed for theefficient processing of extensive
datasets.

Big Data Paradigm
Big Data Science and Data Scientist as a new profession
Big Data Architecture
Cloud based Big Data Infrastructure

7

NIST Big Data Reference Architecture
Main components of the Big
Data ecosystem

I N F O R M AT I O N V A L U E C H A I N

•
•
•
•
•

Visualization

Analytics

Access

DATA

SW
SW

SW

Big Data Framework Provider
Processing Frameworks (analytic tools, etc.)
Horizontally Scalable

Vertically Scalable
Platforms (databases, etc.)
Horizontally Scalable
Vertically Scalable
Infrastructures
Horizontally Scalable (VM clusters)
Vertically Scalable
Physical and Virtual Resources (networking, computing, etc.)

KEY:

DATA
Service UseSW

Big Data Information
Flow
SW Tools and Algorithms
Transfer

I T VA LU E C H A I N

Preparation

Management

Collection

Security & Privacy

DATA

DATA

Data Provider

Big Data Application Provider

Data Consumer

System Orchestrator

Data Provider
Big Data Applications Provider
Big Data Framework Provider
Data Consumer
Service Orchestrator

Big Data Lifecycle andApplications Provider activities
•
•
•
•
•

Collection
Preparation
Analysis and Analytics
Visualization
AccessBig Data Ecosystem includes allcomponents that are involved into
Big Data production, processing,delivery, and consuming[ref] Volume 6: NIST Big Data Reference Architecture. http://bigdatawg.nist.gov/V1_output_docs.php
HPCS2016

Cloud based Big Data Infrastructure

8

Big Data Architecture Framework (BDAF) by UvA(1) Data Models, Structures, Types– Data formats, non/relational, file systems, etc.

(2) Big Data Management– Big Data Lifecycle (Management) Model• Big Data transformation/staging– Provenance, Curation, Archiving(3) Big Data Analytics and Tools– Big Data Applications• Target use, presentation, visualisation(4) Big Data Infrastructure (BDI)– Storage, Compute, (High Performance Computing,) Network– Sensor network, target/actionable devices– Big Data Operational support(5) Big Data Security– Data security in-rest, in-move, trusted processing environments
HPCS2016

Cloud based Big Data Infrastructure

9

Data
Collection&
RegistrationData
SourceData
Filter/Enrich,Classification

Data
Analytics,
Modeling,
Prediction

Consumer

Big Data Ecosystem: General BD Infrastructure
Data Transformation, Data Management
Data
Delivery,
Visualisation

Big Data Target/Customer: Actionable/Usable Data
Target users, processes, objects, behavior, etc.
Federated
AccessandDeliveryInfrastructure(FADI)Big Data Source/Origin (sensor, experiment, logdata, behavioral data)

Big Data Analytic/Tools

Storage
GeneralPurposeComputeGeneralPurposeHigh
Performance
Computer
Clusters

Big Data Infrastructure• Heterogeneous multi-provider
inter-cloud infrastructure• Data management
infrastructure• Collaborative Environment(user/groups managements)• Advanced high performance(programmable) network• Security infrastructure

StorageSpecialised
Databases
Archives(analytics DB,
In memory,
operstional)

categories: metadata,(un)structured, (non)identifiable
Data Management Data
Data
Datacategories:
categories:metadata,
metadata,(un)structured,(un)structured,(non)identifiable(non)identifiable

Intercloud multi-provider heterogeneous Infrastructure
Security Infrastructure

HPCS2016

Network InfrastructureInternal

InfrastructureManagement/Monitoring

Cloud based Big Data Infrastructure

10

Big Data Infrastructure and Analytics Tools
Big Data Infrastructure• Heterogeneous multi-provider
inter-cloud infrastructure• Data management infrastructure• Collaborative Environment• Advanced high performance(programmable) network• Security infrastructure• Federated Access and Delivery
Infrastructure (FADI)

Big Data Analytics
Infrastructure/Tools• High Performance Computer
Clusters (HPCC)• Big Data storage and databases
SQL and NoSQL• Analytics/processing: Real-time,
Interactive, Batch, Streaming• Big Data Analytics tools andapplications

HPCS2016

Cloud based Big Data Infrastructure11Data Lifecycle/Transformation Model
Multiple Data Models and structures•
Data Variety and Variability•
Semantic Interoperability

Data Model (1)Data Model (1)Data (inter)linking•
PID/OID•
Identification•
Privacy, Opacity•
Traceability vsOpacity

Data Model (4)Data Storage (Big Data capable)

Data
Collection&
Registration

Data
Analytics,
Modeling,
Prediction

Data
Filter/Enrich,Classification

Data
Delivery,
Visualisation

Consumer
applications

Data
SourceData Model (3)Data repurposing,
Analytics re-factoring,
Secondary processing••
HPCS2016

Data Model changes along•
data lifecycle or evolution
Data provenance is a discipline to track alldata transformations along lifecycle

Identifying and linking data–––

Persistent data/object identifiers (PID/OID)
Traceability vs Opacity
Referral integrity

Cloud based Big Data Infrastructure

12Cloud Based Big Data Services
High-performance and scalable computing system (e.g. Hadoop)

Data
Sources

Data Ingest

Data Analysis

Data Visualis(data servers)(compute servers)(query servers)

GeneralPurposeCloud
Infrastrd1

d2

Source dataset

Characteristics:
Massive data andcomputation on cloud, small
queries and results
HPCS2016

Query

Data
TargetResult(actionable
data)

Parallelfile system(e.g., GFS,
HDFS)

d3

Derived datasets

Examples:Search, scene completion
service, log processing

Cloud based Big Data Infrastructure

13

Big Data Stack components and technologies
The major structural components of the Big Data stack are grouped around the mainstages of data transformation• Data ingest: Ingestion will transform, normalize, distribute and integrate to one ormore of the Analytic or Decision Support engines; ingest can be done via ingest
API or connecting existing queues that can be effectively used for handles
partitioning, replication, prioritisation and ordering of data
• Data processing: Use one or more analytics or decision support engines toaccomplish specific task related to data processing workflow; using batch dataprocessing, streaming analytics, or real-time decision support• Data Export: Export will transform, normalize, distribute and integrate output
data to one or more Data Warehouse or Storage platforms;• Back-end data management, reporting, visualization: will support data
storage and historical analysis; OLAP platforms/engines will support dataacquisition and further use for Business Intelligence and historical analysis.

HPCS2016

Cloud based Big Data Infrastructure

14

Big Data Stack
Hook into an existing queue to get copy / subset
of data. Queue handles partitioning, replication,and ordering of data, can manage backpressure
from slower downstream components

Ingestion will transform,
normalize, distribute andintegrate to one or more of
the Analytic / Decision
Engines of choice
Use one or more Analytic /Decision engines toaccomplish specific task on
the Fast Data window

OLAP Engines tosupport data acquisition
for later historical data
analysis and BusinessIntelligence on entire
Big Data set

HPCS2016

Use direct
Ingestion
API tocaptureentirestream atwire speed

Message/Data Queue
Data Ingestion

Real-Time
Decisions and/or
Results can be fed
back “up-stream” toinfluence the “nextstep”

Data Processing
Streaming Analytics• NON QUERY• TIME WINDOW MODEL• Counting, Statistics• Rules• Triggers

Batch processing• SEARCH QUERY• Decision support• Statistics

Real-Time Decisions• CONTINUOUS QUERY• OLTP MODEL• Prog Request-Response• Stored Procedures• Export for Pipelining

Data Export
Data management, reporting, visualisation• Data Store & Historical Analytics• OLAP / DATA WAREHOUSE MODEL• PERIODIC QUERY• Business Intelligence, Reports• User Interactive

Cloud based Big Data Infrastructure

Export will transform,
normalize, distributeand integrate to one
or more Data
Warehouse or
Storage Platforms

15

Important Big Data Technologies

Microsoft Azure:
Event Hubs
Data Factory
Stream Analytics
HDInsight
DocumentDB
Google GCE:
DataFlow
BigQueryAmazon AWS:
KinesisEMR
DynamoDB
Proprietary:
Vertica
HortonWorks

HPCS2016

Event Hubs

Message/Data Queue
Samza

Data Factory, Kafka, Flume, Scribe

DataFlow

Data Ingestion

Stream Analytics,
Kinesis Storm, Cascading,
S4, Crunch, SparkStreaming

Streaming Analytics
Sqoop, HDFS

Hadoop,HDFS
EMR
HDInsight

VoltDB

Batch processing Real-Time Decisions

Data Export

HDInsight, DocumentDB, Hadoop, Hive, Spark
SQL, Druid, BigQuery, DynamoDB, Vertica,
MongoDB, EMR, CouchDB,

Data management, reporting, visualisation

Cloud based Big Data Infrastructure

Open Source
Samza
Kafka
Flume
ScribeStorm
CascadingS4
CrunchSpark
HadoopHive
Druid
MongoDB
CouchDB
VoltDB

16

Cloud Platform Benefits for Big Data•

Cloud deployment on virtual machines or containers– Applications portability and platform independence, on-demand provisioning– Dynamic resource allocation, load balancing and elasticity for tasks andprocesses with variable load••

Availability of rich cloud based monitoring tools for collecting performance
information and applications optimisation
Network traffic segregated and isolation– Big Data applications benefit from lowest latencies possible for node to node
synchronization, dynamic cluster resizing, load balancing, and other scale-out
operations– Clouds construction provides separate networks for data traffic and management
traffic– Traffic segmentation by creating Layer 2 and Layer 3 virtual networks insideuser/application assigned Virtual Private Cloud (VPC)•

Cloud tools for large scale applications deployment and automation– Provide basis for agile services development and Zero-touch services
provisioning– Applications deployment in cloud is supported by major Integrated Development
Environment (IDE)

HPCS2016

Cloud based Big Data Infrastructure17Cloud HPC and Big Data Platforms•

HPC on cloud platform– Special HPC and GPU VM instances as well as Hadoop/HPC clusters offeredby all CSPs•

Amazon Big Data services– Amazon Elastic MapReduce, Kinesis, DynamoDB, Regshift, etc•

Microsoft Analytics Platform System (APS)– Microsoft HD Insight/Hadoop ecosystems•

IBM BlueMix applications development platform– Includes full cloud services and data analytics services•

LexisNexis HPC Cluster System– Combing both HPC cluster platform and optimized data processing
languages•

Variety of Open Source tools– Streaming analytics/processing tools: Apache Kafka, Apache Storm, Apache
Spark

HPCS2016

Cloud based Big Data Infrastructure18

LexisNexis HPCC Systems Architecture••

THOR is used for massive data processing in batch mode for ETL processing
ROXIE is used for massive query processing and real-time analytics

HPCS2016

Cloud based Big Data Infrastructure19

LexisNexis HPCC Systems as an integrated Open
Source platform for Big Data Analytics
HPCC Systems data analytics environment components and HPCC Systems architecture
model is based on a distributed, shared-nothing architecture and contains two cluster••

THOR Data Refinery: Massively parallel Extract, Transform, and Load (ETL) engine that can be used forvariety of tasks such as massive: joins, merges, sorts, transformations, clustering, and scaling.
ROXIE Data Delivery: Massively parallel, high throughput, structured query response engine with realtime analytics capability

Other components of the HPCC environment: data analytics languages•••

Enterprise Control Language (ECL): An open source, data-centric declarative programming language– The declarative character of ECL language simplifies coding– ECL is explicitly parallel and relies on the platform parallelism.
LexisNexis proprietary record linkage technology SALT (Scalable Automated Linking Technology):
automates data preparation process: profiling, parsing, cleansing, normalisation, standardisation of data.
– Enables the power of the HPCC Systems and ECL
Knowledge Engineering Language (KEL) is an ongoing development– KEL is a domain specific data processing language that allows using semantic relations between
entities to automate generation of ECL code.

HPCS2016

Cloud based Big Data Infrastructure20

Cloud-powered Services Development Lifecycle:
DevOps == Continuous service improvement
Configuration Management & Orchestration

Dev
All in one
instance

Test/QA

Staging

2-tier app
with test DB

2-tier app withproduction
data

Prod
2-tier app withproduction data
Multi-AZ and HA

Amazon CloudFormation + Chef (or Puppet)••

Easily creates test environment close to real
Powered by cloud deployment automation tools–•

Continuous development – test – integration–•

To enable configuration Management and Orchestration, Deployment automation
CloudFormation Template, Configuration Template, Bootstrap Template

Can be used with Puppet and Chef, two configuration and deployment management systems for clouds[ref] Building Powerful Web Applications in the AWS Cloud” by Louis Columbus
http://softwarestrategiesblog.com/2011/03/10/building-powerful-web-applications-in-the-aws-cloud/HPCS2016

Cloud based Big Data Infrastructure

21

CYCLONE Project: Automation platform for cloud
based applications• Biomedical and Energy
applications• Multi-cloud multi-provider• Distributed and dataprocessing environment• Network infrastructure
provisioning– Dedicated and virtual
overlay over Internet• Automated applications
provisioning

HPCS2016

Cloud based Big Data Infrastructure

22

CYCLONE Components• Biomedical and Energy
applications• Multi-cloud multi-provider• Distributed and dataprocessing environment• Network infrastructure
provisioning– Dedicated and virtual
overlay over Internet• Automated applications
provisioning

HPCS2016

Cloud based Big Data Infrastructure23

SlipStream – Cloud Automation andManagement Platform
Providing complete engineering PaaS supporting DevOps processes• Deployment engine• “App Store” for sharing application definitions with other users• “Service Catalog” for finding appropriate cloud service offers• Proprietary Recipe format– Stored and shared via AppStore•All features are available through web interface or RESTful API••

Similar to Chef, Puppet, Ansible
Supports multiple cloud platforms: AWS, Microsoft Azure, StratusLab, etc.

http://sixsq.com/products/slipstream/index.html
HPCS2016

Cloud based Big Data Infrastructure24

Bioinformatics Use Cases

1.
2.
3.

Securing human biomedical data
Cloud virtual pipeline for microbial genomes analysis
Live remote cloud processing of sequencing data

On-demand bandwidth, compute as well as complex orchestration.

Functionality used for use cases deployment
The definition of an application component consists of a series of recipesthat are executed at various stages in the lifecycle of the application.
• Pre-install: Used principally to configure and initialize the operating
system’s package management.
• Install packages: A list of packages to be installed on the machine.
SlipStream supports the package managers for the RedHat and Debianfamilies of OS.
• Post-install: Can be used for any software installation that can not be
handled through the package manager.
• Deployment: Used for service configuration and initialization. This script
can take advantage of SlipStream’s “parameter database” to pass
information between components and to synchronize the configuration of
the components.
• Reporting: Collects files (typically log files) that should be collected at
the end of the deployment and made available through SlipStream.
HPCS2016

Cloud based Big Data Infrastructure

26

The master node deployment
The master node deployment script performs the following actions:• Initialize the yum package manager.
• Install bind utilities.
• Allow SSH access to the master from the slaves.
• Collect IP addresses for batch system.
• Configure batch system admin user.
• Export NFS file systems to slaves.
• Configure batch system.
• Indicate that cluster is ready for use.

HPCS2016

Cloud based Big Data Infrastructure

27

Example script to export NSF directoryss-display "Exporting SGE_ROOT_DIR..."echo -ne "$SGE_ROOT_DIR\t" > $EXPORTS_FILE
for ((i=1; i<=`ss-get
Bacterial_Genomics_Slave:multiplicity`; i++ ));donode_host=`ss-get
Bacterial_Genomics_Slave.$i:hostname`echo -ne $node_host >> $EXPORTS_FILE
echo -ne "(rw,sync,no_root_squash) ">> $EXPORTS_FILE
doneecho "\n" >> $EXPORTS_FILE # last for a newline
exportfs –av••

ss-get command retrieves a value from the parameter database.
It determines the number of slaves and then loops over each one–

HPCS2016

Retrieves each IP address (hostname) and add it to the NFS exports file.

Cloud based Big Data Infrastructure28

Questions and Discussion• More information about CYCLONE project
http://www.cyclone-project.eu/• SlipStream
http://sixsq.com/products/slipstream/index.html

HPCS2016

Cloud based Big Data Infrastructure29