Cloud based Big Data Platforms
and
New Profession of Data Scientist

Konferencję Użytkowników Komputerów Dużej
Mocy – KU KDM'16
17 March 2016, Zakopane, Poland
Yuri Demchenko, University of Amsterdam
KU KDM'16

Cloud, Big Data and Data Science1Outline• Big Data and Data Centric Computing– Need for new paradigms, architecture and platforms• Cloud Computing as a platform of choice for Big Data
applications– Big Data Stack and cloud advantages– Cloud platforms for Big Data

• Big Data, Data Science and Data Scientist profession
definition– Data Science competences, skills and body of knowledge

Acknowledgement
Slides on Big Data Stacks are credit to David Bernstein, Cloud Strategy Partners
KU KDM'16

Cloud, Big Data and Data Science2

Yuri Demchenko – Professional Summary•

Graduated from National Technical University of Ukraine “Kiev Polytechnic
Institute” (KPI) in Instrumentation and Measurement (aka Industry Automation)
–

•
•
•
•

Candidate of Science (Tech) – Dissertation on System Oriented Precision Generators (1989)

Teaching at KPI 1989-1998 – Computer Networking, Internet Technologies,
Security
Professional work in Internet technologies since 1993
Work at TERENA (Trans-European R&E Networking Association) – 1998-2002
Work at UvA with SNE group – since 2003
– Main research areas: Cloud Computing, Big Data Infrastructures, Application andInfrastructure Security, Generic AAA&Authorisation, Grid and collaborative systems– EU Projects: GEYSERS, GEANT3, Phosphorus, EGEE I-II, Collaboratory.nl– Standardisation activity – IETF, Open Grid Forum (OGF) – ISOD-RG chairing, NIST
Cloud Collaboration, NIST Big Data WG, ISO/IEC Big Data Study Group– Now/2014: Big Data Architecture, Big Data Security, Cloud Computing and Big Data
Curriculum development– Now/2015: EDSION Project coordinator: Building Data Science Profession

KU KDM'16

Cloud, Big Data and Data Science3

Visionaries and Drivers:
Seminal works, High level reports, Activities
The Fourth Paradigm: Data-Intensive Scientific Discovery.
By Jim Gray, Microsoft, 2009. Edited by Tony Hey, et al.
http://research.microsoft.com/en-us/collaboration/fourthparadigm/
Riding the wave: How Europe can gain fromthe rising tide of scientific data.
Final report of the High Level Expert Group on
Scientific Data. October 2010.
http://cordis.europa.eu/fp7/ict/einfrastructure/docs/hlg-sdi-report.pdf

https://www.rd-alliance.org/
NIST Big Data Working Group (NBD-WG)
http://bigdatawg.nist.gov/
ISO/IEC JTC1 Big Data Study Group (SGBD)
http://jtc1bigdatasg.nist.gov/home.php
KU KDM'16

Cloud, Big Data and Data Science

The Data Harvest: Howsharing research data
can yield knowledge,
jobs and growth.
An RDA Europe Report.
December 2014
https://rd-alliance.org/dataharvest-report-sharing-dataknowledge-jobs-and-growth.html

4

Big Data definition revisited: 6 V’s of Big Data
VolumeVariety
•
•
•
•
•
•

Structured
Unstructured
Multi-factor
Probabilistic
LinkedDynamic••••

Terabytes
Records/Arch
Tables, Files
Distributed••••

Adopted in general
by NIST BD-WG
KU KDM'16

Batch
Real/near-time
Processes
Streams

6 Vs ofBig Data• Changing data
• Changing model• Linkage

Variability

Velocity

•
•
•
•
•

•
•
•
•

Value

Correlations
Statistical
EventsHypothetical

Trustworthiness
Authenticity
Origin, Reputation
Availability
Accountability

Veracity
Cloud, Big Data and Data Science

Generic Big Data
Properties• Volume• Variety• Velocity
Acquired Properties(after entering system)• Value• Veracity• Variability
Commonly accepted
3V’s of Big Data
• Volume• Velocity• Variety5Big Data definition revisited: 5 parts vs 6V(1) Big Data Properties: 6V– Volume, Variety, Velocity, Value, Veracity– Additionally: Data Dynamicity (Variability)(2) New Data Models– Data linking, provenance and referral integrity– Data Lifecycle and Variability/Evolution(3) New Analytics–

Real-time/streaming analytics, interactive and machine learning analytics(4) New Infrastructure and Tools––––

High performance Computing, Storage, Network
Heterogeneous multi-provider services integration
New Data Centric (multi-stakeholder) service models
New Data Centric security models for trusted infrastructure and data processing
and storage(5) Source and Target– High velocity/speed data capture from variety of sensors and data sources– Data delivery to different visualisation and actionable systems and consumers– Full digitised input and output, (ubiquitous) sensor networks, full digital control
KU KDM'16

Cloud, Big Data and Data Science

6

NIST Big Data Working Group (NBD-WG) andISO/IEC JTC1 Study Group on Big Data (SGBD)••NIST Big Data Working Group (NBD-WG) is leading the development of the Big Data
Technology Roadmap - http://bigdatawg.nist.gov/home.php– Built on experience of developing the Cloud Computing standards fully accepted byindustry
Set of documents published in September 2015 as NIST Special Publication NIST SP 1500:
NIST Big Data Interoperability Framework (NBDIF)
http://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.1500-1.pdf
Volume 1: NIST Big Data Definitions
Volume 2: NIST Big Data Taxonomies
Volume 3: NIST Big Data Use Case & Requirements
Volume 4: NIST Big Data Security and Privacy Requirements
Volume 5: NIST Big Data Architectures White Paper Survey
Volume 6: NIST Big Data Reference Architecture
Volume 7: NIST Big Data Technology Roadmap•

NBD-WG defined 3 main components of the newtechnology:–––

KU KDM'16

The Big Data Paradigm consistsof the distribution of data systems
across horizontally-coupledindependent resources to achieve
the scalability needed for theefficient processing of extensive
datasets.

Big Data Paradigm
Big Data Science and Data Scientist as a new profession
Big Data Architecture
Cloud, Big Data and Data Science

7

NIST Big Data Reference Architecture
Main components of the Big
Data ecosystem

I N F O R M AT I O N V A L U E C H A I N

•
•
•
•
•

Visualization

Analytics

Access

DATA

SW
SW

SW

Big Data Framework Provider
Processing Frameworks (analytic tools, etc.)
Horizontally Scalable

Vertically Scalable
Platforms (databases, etc.)
Horizontally Scalable
Vertically Scalable
Infrastructures
Horizontally Scalable (VM clusters)
Vertically Scalable
Physical and Virtual Resources (networking, computing, etc.)

KEY:

DATA
Service UseSW

Big Data Information
Flow
SW Tools and Algorithms
Transfer

I T VA LU E C H A I N

Preparation

Management

Collection

Security & Privacy

DATA

DATA

Data Provider

Big Data Application Provider

Data Consumer

System Orchestrator

Data Provider
Big Data Applications Provider
Big Data Framework Provider
Data Consumer
Service Orchestrator

Big Data Lifecycle andApplications Provider activities
•
•
•
•
•

Collection
Preparation
Analysis and Analytics
Visualization
AccessBig Data Ecosystem includes allcomponents that are involved into
Big Data production, processing,delivery, and consuming[ref] Volume 6: NIST Big Data Reference Architecture. http://bigdatawg.nist.gov/V1_output_docs.php
KU KDM'16

Cloud, Big Data and Data Science8

Big Data Architecture Framework (BDAF) by UvA(1) Data Models, Structures, Types– Data formats, non/relational, file systems, etc.

(2) Big Data Management– Big Data Lifecycle (Management) Model• Big Data transformation/staging– Provenance, Curation, Archiving(3) Big Data Analytics and Tools– Big Data Applications• Target use, presentation, visualisation(4) Big Data Infrastructure (BDI)– Storage, Compute, (High Performance Computing,) Network– Sensor network, target/actionable devices– Big Data Operational support(5) Big Data Security– Data security in-rest, in-move, trusted processing environments
KU KDM'16

Cloud, Big Data and Data Science9Big Data Infrastructure and Analytics Tools
Big Data Infrastructure• Heterogeneous multi-provider
inter-cloud infrastructure• Data management infrastructure• Collaborative Environment• Advanced high performance(programmable) network• Security infrastructure• Federated Access and Delivery
Infrastructure (FADI)

Big Data Analytics
Infrastructure/Tools• High Performance Computer
Clusters (HPCC)• Big Data storage and databases
SQL and NoSQL• Analytics/processing: Real-time,
Interactive, Batch, Streaming• Big Data Analytics tools andapplications

KU KDM'16

Cloud, Big Data and Data Science

10Data Lifecycle/Transformation Model
Multiple Data Models and structures•
Data Variety and Variability•
Semantic Interoperability

Data Model (1)Data Model (1)Data (inter)linking•
PID/OID•
Identification•
Privacy, Opacity•
Traceability vsOpacity

Data Model (4)Data Storage (Big Data capable)

Data
Collection&
Registration

Data
Analytics,
Modeling,
Prediction

Data
Filter/Enrich,Classification

Data
Delivery,
Visualisation

Consumer
applications

Data
SourceData Model (3)Data repurposing,
Analytics re-factoring,
Secondary processing••
KU KDM'16

Data Model changes along•
data lifecycle or evolution
Data provenance is a discipline to track alldata transformations along lifecycle

Identifying and linking data–––

Persistent data/object identifiers (PID/OID)
Traceability vs Opacity
Referral integrity

Cloud, Big Data and Data Science

11

There were 5 exabytes of information
created between the dawn of civilization
through 2003, but that much information
is now created every 2 days, and thepace is increasing
Eric Schmidt, Google CEO,
Conference, August 4, 2010

Techonomy

Data is becoming the new raw material of business:an economic input almost on a par with capital andlabour. “Every day I wake up and ask, ‘how can I flow
data better, manage data better, analyse data
better?” says Rollin Ford, the CIO of Wal-Mart.

DemandSupplyBig Data is Driving Cloud Usage –Cloud powers Big Data applications

Source: Data, Data Everywhere, The Economist, February 25,
2010
KU KDM'16

Cloud, Big Data and Data Science12Public Clouds are All Over the Place

Cloud Centers are All Over the Place, from datacentermap.com November 2015
http://www.datacentermap.com/cloud.html
KU KDM'16

Cloud, Big Data and Data Science13

Cloud Based Big Data Services
Data-intensive computing system (e.g. Hadoop)

Externaldatasources

Paralleldata serverd1Parallelcompute serverd2

SourcedatasetParallelquery serverd3

ResultExternal
ClientParallelfile system(e.g., GFS,
HDFS)

Deriveddatasets

Characteristics:
Massive data andcomputation on cloud, small
queries and results
KU KDM'16

Query

Cloud, Big Data and Data Science

Examples:Search, scene completion
service, log processing

14

Big Data Stack
Hook into an existing queue to get copy / subset ofdata. Queue handles partitioning, replication, andordering of data, can manage backpressure fromslower downstream components

Ingestion will transform,
normalize, distribute andintegrate to one or more of
the Analytic / Decision
Engines of choice
Use one or more Analytic /Decision engines toaccomplish specific task on
the Fast Data window

The OLAP Engines ofchoice will append tohistorical data and store
for later, further Big
Data analysis on entire
data set

KU KDM'16

Message/Data Queue

Use direct
Ingestion
API tocaptureentirestream atwire speed

Real-Time Decisions
and/or Results canbe fed back “upstream” to influence
the “next step”

Data Ingestion
Data Processing
Streaming Analytics

Real-Time Decisions• NON QUERY• TIME WINDOW MODEL• Counting, Statistics• Rules• Triggers• CONTINUOUS QUERY• OLTP MODEL• Programmatic Request-Response• Stored Procedures• Export for Pipelining

Data Export
Data Store & Historical Analytics• PERIODIC QUERY• OLAP / DATA WAREHOUSE MODEL• BI, Reports• Dashboards• User Interactive

Cloud, Big Data and Data Science

Export will transform,
normalize, distributeand integrate to one
or more of the Data
Warehouse / Storage
Engines of choice15

Important Big Data Technologies
Microsoft Azure:
Event Hubs
Data Factory
Stream Analytics
HDInsight
DocumentDB

Event Hubs

Message/Data Queue
Samza,

Data Factory, Kafka, Flume, Scribe

Data Ingestion

Dataflow

Google GCE:
DataFlow
BigQuery,Stream Analytics,
KinesisStorm, Cascading,
S4, Crunch, SparkStreaming

VoltDB

Streaming Analytics
Sqoop, HDFS

Amazon AWS:
KinesisEMR
DynamoDB

Real-Time Decisions
Data Export

HDInsight, DocumentDB, Hadoop, Hive, Spark
SQL, Druid, BigQuery, DynamoDB, Vertica,
MongoDB, EMR, CouchDB,

Data Store & Historical Analytics

Open Source
Samza
Kafka
Flume
ScribeStorm
CascadingS4
CrunchSpark
HadoopHive
Druid
MongoDB
CouchDB
VoltDB

Proprietary:
Vertica
HortonWorksKU KDM'16

Cloud, Big Data and Data Science16

Cloud Platform Benefits for Big Data
• Segregated networks isolate traffic– Clouds construction provides separate networks for each type of traffic– Big Data applications benefit from lowest latencies possible for node to node
synchronization, dynamic cluster resizing, and other scale-out operations• Cloud deployment on virtual machines, containers, and bare metal– For a traditional highly load-variable problem one might consider using VM’s
as a deployment vehicle for that.
– Some clouds will offer container based isolation instead of VMs.

• Cloud tools for large scale applications deployment and automation– Supported by major IDE– Basis for agile technologies and Zero-touch services provisioning

KU KDM'16

Cloud, Big Data and Data Science17

Cloud-powered Services Development Lifecycle:
DevOps == Continuous service improvement
Configuration Management & Orchestration

Dev
All in one
instance

Test/QA

Staging

2-tier app
with test DB

2-tier app withproduction
data

Prod
2-tier app withproduction data
Multi-AZ and HA

Amazon CloudFormation + Chef (or Puppet)••

Easily creates test environment close to real
Powered by cloud deployment automation tools–•

Continuous development – test – integration–•

To enable configuration Management and Orchestration, Deployment automation
CloudFormation Template, Configuration Template, Bootstrap Template

Can be used with Puppet and Chef, two configuration and deployment management systems for clouds[ref] Building Powerful Web Applications in the AWS Cloud” by Louis Columbus
http://softwarestrategiesblog.com/2011/03/10/building-powerful-web-applications-in-the-aws-cloud/

KU KDM'16

Cloud, Big Data and Data Science18

Cloud HPC and Big Data Platforms• HPC on cloud platform– Special HPC and GPU VM instances as well as Hadoop/HPC clusters offeredby all CSPs

• Amazon Big Data services– Amazon Elastic MapReduce, Kinesis, DynamoDB, Regshift, etc• Microsoft Analytics Platform System (APS)– Microsoft HD Insight/Hadoop ecosystems• IBM BlueMix applications development platform– Includes full cloud services and data analytics services• LexisNexis HPC Cluster System– Combing both HPC cluster platform and optimized data processing
languages• Variety of Open Source tools– Streaming analytics/processing tools: Apache Kafka, Apache Storm, ApacheSpark
KU KDM'16

Cloud, Big Data and Data Science

19

AWS Cloud Big Data Services
AWS Cloud offers the following services and resources for Big Data processing• EC2 Virtual Machine (VM) instances for HPC optimized for computing (withmultiple cores) and with extended storage for large data processing.
• Amazon Elastic MapReduce (EMR) provides the Hadoop framework on
Amazon EC2 and offers a wide range of Hadoop related tools.
• Amazon Kinesis is a managed service for real-time processing of streaming big
data (throughput scaling from megabytes to gigabytes of data per second andfrom hundreds of thousands different sources).
• Amazon DynamoDB highly scalable NoSQL data stores with sub-millisecond
response latency.
• Amazon Redshift fully-managed petabyte-scale Data Warehouse in cloud atcost less than $1000 per terabyte per year. It is provided with columnar data
storage with possibility to parallelise queries.
• Amazon RDS scalable relational database.
• Amazon Glacier archival storage to AWS for long time data storage at lower costthat standard Amazon S3 object storage.
KU KDM'16

Cloud, Big Data and Data Science20

Microsoft Azure Analytics Platform System (APS)• Microsoft Azure cloud provides general IaaS services and reach Platform
as a Service (PaaS) services.
– Similar to AWS, Microsoft Azure offers special VM instances that have bothcomputational and memory advanced capabilities.

• The Analytics Platform System (APS) combines the Microsoft SQL Server
based Parallel Data Warehouse (PDW) platform with HDInsight andApache Hadoop based scalable data analytics platform.
– APS includes the PolyBase data querying technology to simplify integration of
the PDW SQL data and data from Hadoop.

• HDInsight Hadoop based platform has been co-developed with
Hortonworks– HDInsight provides comprehensive integration and management functionality
for multi-workload data processing on Hadoop platform including batch,stream, in-memory processing methods.

KU KDM'16

Cloud, Big Data and Data Science

21

HDInsight: Microsoft’s Big Data Solution• HDInsight can run both on
Azure Cloud and on Windows
Server (on premises)– Data exchange via PolyBase• Compatible with and support all
products from Apache Hadoop
stack• Supports all stages of Big Data
processing• PolyBase is a new technologythat allows integrating Microsoft
SQL Server based Parallel
Data Warehouse (PDW) with
Hadoop• Azure Blob Storage used topersistently store data
– Data are streamed to Hadoop/HDFS
for processing and pushed back to
Azure Blob Storage

KU KDM'16

Cloud, Big Data and Data Science

22

HDInsight/Hadoop Ecosystem
LegendRed = Core
Hadoop
ODBC

Query(Hive)
Distributed Processing(MapReduce)
Distributed Storage(HDFS)

Blue = Data
processing
Purple = Microsoft
integration points
and enhancements
Orange = Data
Movement

Green = Packages[ref] Microsoft Azure Training Kit. https://github.com/Azure-Readiness/MicrosoftAzureTrainingKit
KU KDM'16

Cloud, Big Data and Data Science23

Hortonworks Data Platform Architecture [ref]•••HDP includes the most recent developments of the Open Source Hadoop suite
Can run on Linux and on Windows OS
Can be deployed on premises on dedicated cluster and on cloud as a hosted application[ref] http://hortonworks.com/hdp /KU KDM'16

Cloud, Big Data and Data Science24

Hortonworks Data Platform (HDP)
http://hortonworks.com/•

HDP delivers a single integrated Hadoop platform for enterprises
–
–
–

•

YARN is treated as datacenter OS and supports multiple access methods (batch,
real-time, streaming, in-memory, and more) on a common data set––•

YARN is the architectural center of Hadoop that allows to process data simultaneously in multiple
ways
Allows creating multi-tenant data analytics applications

HDP runs natively on Linux and Windows OS––•

Provides a data platform for multi-workload data processing across an array of processing methods
including batch and interactive to real-time
Supports key capabilities of an enterprise data platform: Governance, Security and Operations
YARN and Hadoop Distributed Filesystem (HDFS) are the core components of HDP

HDP provides the basis for Microsoft’s HDInsight Service meaning complete portability of data isretained on-premise and in the cloudAvailable in integrated hardware from Teradata

Hortonworks provides a simple starters solution Hadoop Sandbox–

KU KDM'16

Hortonworks Sandbox is a single-node implementation of Hadoop based on the Hortonworks Data
Platform that includes all the typical components found in a Hadoop deployment

Cloud, Big Data and Data Science25

LexisNexis HPCC Systems as an integrated Open
Source platform for Big Data Analytics
HPCC Systems data analytics environment components and HPCC Systems architecture
model is based on a distributed, shared-nothing architecture and contains two cluster••

THOR Data Refinery: Massively parallel Extract, Transform, and Load (ECL) engine that can be used forvariety of tasks such as massive: joins, merges, sorts, transformations, clustering, and scaling.
ROXIE Data Delivery: Massively parallel, high throughput, structured query response engine with realtime analytics capability

Other components of the HPCC environment: data analytics languages•••

Enterprise Control Language (ECL): An open source, data-centric declarative programming language– The declarative character of ECL language simplifies coding– ECL is explicitly parallel and relies on the platform parallelism.
LexisNexis proprietary record linkage technology SALT (Scalable Automated Linking Technology):
automates data preparation process: profiling, parsing, cleansing, normalisation, standardisation of data.
– Enables the power of the HPCC Systems and ECL
Knowledge Engineering Language (KEL) is an ongoing development– KEL is a domain specific data processing language that allows using semantic relations between
entities to automate generation of ECL code.

KU KDM'16

Cloud, Big Data and Data Science26

LexisNexis HPCC Systems Architecture••

THOR is used for massive data processing in batch mode for ETL processing
ROXIE is used for massive query processing and real-time analytics

KU KDM'16

Cloud, Big Data and Data Science27

Data Science Profession Definition
New technologies require new competences, skills and new
professions••••

EDISON Data Science Framework
Data Science professions family
EU activities to address e-skills shortage
EDISON engagement and outreach activities

KU KDM'16

Cloud, Big Data and Data Science28

EDISON Framework and Background Developments

CF-DS

DS-BoK

MC-DS

Taxonomy & Vocabulary•Roadmap&
Sustainability

Edu&Train Inventory

EDISON Framework components
•
•
•
•
•

•

EOEE & ETMplz

CF-DS – Data Science Competence Framework
DS-BoK – Data Science Body of Knowledge
MC-DS – Data Science Model Curriculum
Data Science Taxonomy and Scientific Disciplines Classification
EOEE - EDISON Online Education Environment

Background: EU Competence Frameworks and Profiles–––

KU KDM'16

e-CFv3.0 - European e-Competence framework for IT
CWA 16458 (2012): European ICT Professional Profiles Family Tree
ESCO (European Skills, Competences, Qualifications and Occupations) framework
Cloud, Big Data and Data Science29

Identified Data Science Competence Groups•

Traditional/known Data Science competences/skills groups include
–
–
–

•

Data Analytics or Business Analytics or Machine Learning
Engineering or Programming
Subject/Scientific Domain Knowledge

EDISON identified 2 additional competence groups demandedby organisations[ref] Legacy: NIST BDWG
definition of Data Science– Data Management, Curation, Preservation– Scientific or Research Methods and/vs Business Processes/Operations•

Other skills commonly recognized aka “soft skills” or “social intelligence”–•

All groups need to be represented in Data Science curriculum and training–•

Inter-personal skills or team work, cooperativeness

Challenging task for Data Science education and training

Another aspect of integrating Data Scientist into organisation structure–––

General Data Science (or Big Data) literacy for all involved roles and management
Common agreed and understandable way of communication and information/data presentationRole of Data Scientist: Provide such literacy advice and guiding to organisation

KU KDM'16

Cloud, Big Data and Data Science30

Data Science Competence Groups - Research
Data Science Competence
includes 5 areas/groups
•
•
•
•
•

Data Analytics
Data Science Engineering
Domain Expertise
Data Management
Scientific Methods (or Business
Process Management)

Scientific Methods
•
•
•
•
•
•

Design Experiment
Collect Data
Analyse Data
Identify Patterns
Hypothesise Explanation
Test Hypothesis

Business Operations
•
•
•
•
•

KU KDM'16

Cloud, Big Data and Data Science

Operations Strategy
Plan
Design & Deploy
Monitor & Control
Improve & Re-design31Data Science Competences Groups – Business
Data Science Competence
includes 5 areas/groups

DOMAIN
EXPERTISE

•
•
•
•
•

DesignOptimisation

RESEARCH

DATA

DATA
ANALYTICS

Scientific Methods

SCIENCEANALYTIC ALGORITHMS
SYSTEMS

Monitoring

Modelling
ENGINEERINGCOMPETENCES

Business Process
Management

Data
Management
Execution

KU KDM'16

Data Analytics
Data Science Engineering
Domain Expertise
Data Management
Scientific Methods (or Business
Process Management)

ScientificMethods
Cloud, Big Data and Data Science

•
•
•
•
•
•

Design Experiment
Collect Data
Analyse Data
Identify Patterns
Hypothesise Explanation
Test Hypothesis

Business Process
Operations/Stages
•
•
•
•
•

DesignModel/Plan
Deploy & Execute
Monitor & Control
Optimise & Re-design32Identified Data Science Competence Groups
Data Analytics (DA)

1

2

3

4

5
6
7

Data Management/
Curation (DM)
Use appropriate statistical Develop andtechniques on available implement data
data to deliver insights
strategy

DS Engineering (DSE)

Scientific/Research Methods (DSRM)

DS Domain Knowledge(including Business Apps)

Use engineering principles toresearch, design, or develop
structures, instruments,
machines, experiments,
processes, systems, theories,
or technologies

Create new understandings andcapabilities by using the scientific
Understand business andmethod's hypothesis, test, and evaluation provide insight, translatetechniques; critical review; or similar
unstructured businessengineering research and development
problems into an abstract
methods
mathematical framework
Use predictive analytics
Use data to improve existing
Direct systematic study toward a fuller
to analyse big data andservices or develop new
knowledge or understanding of the
discover new relations
fundamental aspects of phenomena and services
Develops specialized data
Develop data models analysis tools to supportof observable facts, and discovers new
including metadata
executive decision making
approaches to achieve goals
Participate strategically andResearch and analyze
Integrate different data
Undertake creative work, making
tactically in financial
complex data sets,source and provide forsystematic use of investigation ordecisions that impact
combine different sources further analysis
experimentation, to discover or revise
Design, build, operatemanagement andand types of data toknowledge of reality, and uses thisrelational non-relationalorganizations
improve analysis.
databases
knowledge to devise new applications
Develop and apply
Recommends business
Develop specialized
related strategic
Develop and maintain computational solutions toanalytics to enablea historical data
domain related problems
objectives andagile decision
repository of analysis using wide range of data
Apply ingenuity to complex problems,
alternatives andmakingdevelop innovative ideas
analytics platforms
implements themCollect and manage
Ability to translate strategies into action
Provides scientific, technical,
different source of dataplans and follow through to completion.
Develop solutions for secureand analytic support services
and reliable data accessto other organisational roles
Analyse multiple data sources
Develop algorithms to analyse Influence the development ofVisualise complex and multiple source of data
for marketing purposesorganizational objectives
variable data.
Prototype new data analytics
Analyse customer data toapplications
identify/optimise customer
relations actions

KU KDM'16

Cloud, Big Data and Data Science33

Identified Data Science Skills/Experience Groups•

Group 1: Skills/experience related to competences
–
–
–
–
–
–

•

Group 2: Big Data (Data Science) tools and platforms
–
–
–
–
–
–

•

Big Data Analytics platforms
Math & Stats apps & tools
Databases (SQL and NoSQL)
Data Management and Curation platform
Data and applications visualisation
Cloud based platforms and tools

Group 3: Programming and programming languages and IDE–•

Data Analytics and Machine Learning
Data Management/Curation (including both general data management and scientific data
management)
Data Science Engineering (hardware and software) skills
Scientific/Research Methods
Application/subject domain related (research or business)
Mathematics and Statistics

General and specialized platforms for data analysis and statistics

Group 4: Soft skills or Social Intelligence–

KU KDM'16

Personal, inter-personal communication, team work (also called social intelligence or soft skills)
Cloud, Big Data and Data Science34

Identified Data Science Skill Groups
Data Analytics andMachine Learning

Data Management/
Curation

Data Science
Engineering (hardware
and software)

Scientific/ Research
Methods

1

Artificial intelligence,machine learning

Design efficientalgorithms for accessing
and analyzing large
amounts of data

Interest in data science2

Machine Learning andStatistical Modelling

Manipulating andanalyzing complex, highvolume, highdimensionality data fromvarying sources
for data improvement

Big Data solutions andadvanced data mining
tools

Machine learning
solutions and patternrecognition
techniques

Data models anddatatypes

Supervised andunsupervised learning

3

4
5

6
7
8

9
10

Personal/Interpersonalcommunication, teamwork

Application/subject
domain (research or
business)

Communication skills

Recommender or
Ranking system

Analytical, independent,
critical, curious and focusedon results

Inter-personal intrateam and external
communication

Data Analytics forcommercial purposes

Multi-core/distributed
software, preferably in a
Linux environment

Confident with large data
sets and ability to identify
appropriate tools andalgorithms

Network of contacts in
Big Data community

Data sources andtechniques for businessinsight and customer
focus

Handling vast amountsof data

Databases, databasesystems, SQL and NoSQL

Data mining

Experience of working
with large data sets

Statistical analysis
languages and tooling

Flexible analytic approach toachieve results at varying
levels of precision
Exceptional analytical skills

Markov Models,Conditional Random
Fields
Logistic Regression,
Support Vector
Machines
Predictive analysis
and statistics(including Kaggle
platform)(non)relational and (un)structured data

Cloud powered
applications design(Artificial) Neural
Networks

Metadata annotationand management

Statistics

Data citation, metadata,
PID (*)KU KDM'16

Mechanism Design
and/or Latent Dirichlet
AllocationGame Theory

Copyright and IPR

Cloud based data
storage and data
management
Data management
planning

Cloud, Big Data and Data Science35

Identified Big Data Tools and Programming
Languages
Big Data Analytics
platforms

Math& Stats tools

Databases

Data/ applications
visualization

Data Management andCuration platform

Big Data Analytics platforms

Advanced analytics tools(R, SPSS, Matlab, etc)

SQL and relational databases

Data visualization Libraries (D3.js,
FusionCharts, Chart.js, other)

Data modelling and related
technologies (ETL, OLAP, OLTP,
etc)

2

Big Data tools (Hadoop,
Spark, etc)

Data Mining tools:
RapidMiner, others

NoSQL Databases

Visualisation software (D3,Processing, Tableau, Gephi, etc)

Data warehouses platform andrelated tools

3

Distributed computing tools aplus (Spark, MapReduce,
Hadoop, Hive, etc.)

MathlabNoSQL, Mongo, Redis

Online visualization tools(Datawrapper, Google Charts,
Flare, etc)

Data curation platform,
metadata management (ETL,
Curator's Workbench, DataUp,
MIXED, etc)

4Real time and streaming
analytics systems (like Flume,
Kafka, Storm)

PythonNoSQL, Teradata

Hadoop Ecosystem/platform

R, Tableau R

Excel

SpotfireSAS

7

Azure Data Analytics
platforms (HDInsight, APS and
PDW, etc)

Scripting language, e.g.
Octave8

Amazon Data Analytics
platform (Kinesis, EMR, etc)

Statistical tools and data
mining techniques

Other cloud based Data
Analytics platforms(HortonWorks, Vertica
LexisNexis HPCC System, etc)

Other Statisticalcomputing and languages(WEKA, KNIME, IBM SPSS,
etc)

1

5
6

9

KU KDM'16

Backup and storage
management (iRODS, XArch,
Nesstar, others

•
•
•
•
•

Big Data Analytics platforms
Math& Stats tools
Databases
Data/applications visualization
Data Management and Curation
platform

Cloud, Big Data and Data Science36

Suggested e-CF extensions for Data Science
A. PLAN and Design•
A.10* Organisational workflow/processes model definition/formalisation•
A.11* Data models and data structures
B. BUILD: Develop and Deploy/Implement•
B.7* Apply data analytics methods (to organizational processes/data)•
B.8* Data analytics application development•
B.9* Data management applications and tools
15 Data Science Competences proposed•
B.10* Data Science infrastructure deployment
C. RUN: Operate•
C.5* User/Usage data/statistics analysis•
C.6* Service delivery/quality data monitoringcovering different organizational roles andworkflow stages• Data Scientist roles are crossing multiple
org roles and workflow stages

D. ENABLE: Use/Utilise•
D10. Information and Knowledge Management (powered by DS)•
D.13* Data presentation/visualisation, actionable data extraction•
D.14* Support business processes/roles with data and insight (support to D.5, D.6, D.7, D.12)•
D.15* Data management/preservation/curation with data and insight
E. MANAGE•
E.10* Support Management and Business Improvement with data and insight (support to E.5, E.6)•
E.11* Data analytics for (business) Risk Analysis/Management (support to E.3)•
E.12* ICT and Information security monitoring and analysis (support to E.8)

KU KDM'16

Cloud, Big Data and Data Science

37

Possible Data Scientist profiles/roles
as extension to CWA16458 (2012) or ESCO
• Data Analyst, Business Analyst– Data Mining– Machine Learning• Digital Librarian, Data Archivist, Data Curator, Data Steward– Data Management related competences• Data Science Engineer/Administrator/Programmer– Data Analytics applications development– Scientific programmer– Data Science/Big Data Infrastructure engineer/developer/operator• Data Science Researcher– Data Science creative– Data Science consultant/Analyst• Data Scientist in subject/research domain• Research e-Infrastructure brings its own specifics to requiredcompetences and skills definition
KU KDM'16

Cloud, Big Data and Data Science38

Data Science and Subject Domains
Data Science domain components
Data structures &databases/storage• Abstract data driven
math&compute models• Data Analytics
methods• Data and Applications
Lifecycle Management

Crossorganisational
assistive role

Data Scientist functions is to translate between two domains

Domain specific components
Domain specificdata & presentation

KU KDM'16• Models (and data types)• Methods• Processes

Cloud, Big Data and Data Science

Organisational
roles

39

Data Scientist and Subject Domain Specialist• Subject domain components– Model (and data types)– Methods– Processes– Domain specific data and presentation/visualization methods– Organisational roles and relations• Data Scientist is an assistant to Subject Domain Specialists– Translate subject domain Model, Methods, Processes into abstract data
driven form
– Implement computational models in software, build required
infrastructure and tools– Do (computational) analytic work and present it in a form understandable
to subject domain– Discover new relations originated from data analysis and advice subject
domain specialist– Interact and cooperate with different organizational roles to obtain data
and deliver results and/or actionable data
KU KDM'16

Cloud, Big Data and Data Science40

Data Science Body of Knowledge (DS-BoK)
DS-BoK Knowledge Area Groups (KAG)• KAG1-DSA: Data Analytics group including Machine Learning, statisticalmethods, and Business Analytics• KAG2-DSE: Data Science Engineering group including Software andinfrastructure engineering• KAG3-DSDM: Data Management group including data curation,
preservation and data infrastructure• KAG4-DSRM: Scientific/Research Methods group• KAG5-DSBP: Business process management group• Data Science domain knowledge to be defined by related expert groups

KU KDM'16

Cloud, Big Data and Data Science41

KAG3-DSDM: Data Management group: data curation,
preservation and data infrastructure
DM-BoK version2 “Guide for
performing data management”– 11 Knowledge Areas(1) Data Governance,(2) Data Architecture,(3) Data Modelling and Design,(4) Data Storage and Operations,(5) Data Security,(6) Data Integration andInteroperability,(7) Documents and Content,(8) Reference and Master Data,(9) Data Warehousing and BusinessIntelligence,(10) Metadata,(11) Data Quality

KU KDM'16

Other Knowledge Areas motivatedby RDA, European Open Data
initiatives, European Open Data
Cloud(12) PID, ORCID(13) Data Management Plan
(14) Research Data Infrastructure

Cloud, Big Data and Data Science42European Agenda on Skills for Digital Single
Market (DSM) in Horizon 2020
• EC document to be published in May 2016 under Dutch
presidency• Multiple activities at EC– European Open Data Cloud (EODC) report by Barend Mons, Leiden
University (bioinformatician)• To be published in April 2016

– Standardisation for Big Data technologies Workshop 14 March 2016,
Luxembourg• Call for more active contribution by European industry and experts in NIST Big
Data WG and ISO/IEC JTC1 Big Data Study Group (SGBD)– eSkills workshop 16 March 2016, Den Haag
• Addressing eSkills gap in Europe– Other events and activities by BDVA, OECD, etc

KU KDM'16

Cloud, Big Data and Data Science43Digital Economy and Society Index (EU 2015-2016)[ref] http://europa.eu/rapid/press-release_MEMO-16-385_en.htm
KU KDM'16

Cloud, Big Data and Data Science44European consultation call – Deadline 30 April 2016
https://ec.europa.eu/futurium/en/content/consultation-european-e-infrastructure•

What are the main challenges for the realisation of an integrated European e- infrastructurefrom the perspective of scientific data-related needs (from data access to sharing,
analytics, re-use, preservation, standards, interoperability, value chain and other issues)?

•

What are the challenges for reinforcing the cooperation between European einfrastructure service providers and their scientific users, including thematic research
infrastructures, to accelerate user's adoption of e-infrastructure services - such as identity
management innovation - and foster innovation in e-infrastructures?

•

What are the challenges faced by industrial actors preventing them to fully benefit fromthe services provided by European e-infrastructures and to contribute to the innovation of
the existing e-infrastructures?

•

What are the main challenges Europe is facing regarding skills and competences
required for effective data driven science, and management of research einfrastructures?

KU KDM'16

Cloud, Big Data and Data Science45

EDISON Project Engagement and Outreach• EDISON Liaison Groups: Universities, Industry, Experts• Champion universities– Summer 2016 workshop of Champions, Ambassadors and Adopters• EDISON Survey on competences and skills for Data Science
https://www.surveymonkey.com/r/EDISON_project_-_Defining_Data_science_profession• Numerous workshops• Data Science community portal http://www.edison-project.eu/– Community forum and community contribution– All major project deliverable are open for community discussion– Future: Personal profile building and competences self-assessment• Future Data Science professional certification– For graduates and self-made data scientists

KU KDM'16

Cloud, Big Data and Data Science46

EDISON Survey: Data Science Competences and Skills
Survey link https://www.surveymonkey.com/r/EDISON_project_-_Defining_Data_science_profession

KU KDM'16

Cloud, Big Data and Data Science47

Questions and Discussion

KU KDM'16

Cloud, Big Data and Data Science48

Additional information• EDISON Approach: e-CFv3.0 and CF-DS• Data Science occupations in ESCO taxonomy

KU KDM'16

Cloud, Big Data and Data Science49

EDISON Approach: e-CFv3.0 and CF-DS•

Competence Framework for Data Science (CF-DS) definition will be built based on
European e-Competence framework for IT (e-CFv3.0)– Linking scientific research cycle/flow, organizational roles, competences, skills andknowledge– Defining Data Science Body of Knowledge (DS-BoK)– Mapping CF-DS and DS-BoK to academic disciplines in a DS Model Curriculum (MC-DS)• Multiple use of e-CFv3.0 within ICT
organisations• Provides basis for individual career
path, competence assessment, training
and certification• EDISON CF-DS will be used for
defining DS-BoK and MC-DS, linking
organizational functions and required
knowledge• Provide basis for individual (self)
training and certification

KU KDM'16

Cloud, Big Data and Data Science50e-CFv3.0 Internal Structure: Refactoring for CF-DS• s• 4 Dimensions– Competence Areas– Competences– Proficiency levels– Skills and Knowledge• 5 Competence Area defined by ICT
Business Process stages– Plan– Build– Run– Enable– Manage-> Refactor to Scientific Research
cycle/workflow (and linked to Scientific Data
Lifecycle)– See example of RI manager at IG-ETRD wiki
and meeting• Each competence has 5 proficiency level– Ranging from technical to engineering tomanagement to strategist/expert level• Knowledge and skills property are definedfor/by each competence and proficiency
level (not unique)
KU KDM'16

Cloud, Big Data and Data Science51

Definitions (according to e-CFv3.0)• Competence is a demonstrated ability to apply knowledge, skills andattitudes for achieving observable results.
– Competence vs Competency (e-CF vs ACM)
• Competence is ability acquired by training or education (linked to learning outcome)• Competency is similar to skills or experience (acquired feature of a person)– Competence can be treated as outcome of learning or training• Knowledge in the context of competence definition is treated as
something to know, to be aware of, familiar with, and obtained as a partof education.
• Skills is treated as provable ability to do something and relies on the
person’s experience.

KU KDM'16

Cloud, Big Data and Data Science52

Data Science occupations in ESCO taxonomy (1)Professionals
Science andengineering
professionals

Data Science
Professionals

Data Science
professionals notelsewhere classified

Data Scientist

Data Science Researcher(Big) Data Analyst
Data Science (Application)
Programmer
Business Analyst
Database and network
professionals

Information andcommunications
technology
professionals

Data Science technology
professionals

Large scale (cloud) data
storage designers andadministrators

Large scale (cloud) database
designer*)

Database designers andadministrators

Large scale (cloud) databaseadministrator*)

Database and network
professionals notelsewhere classified

Scientific databaseadministrator*)

Data handlingprofessionals notelsewhere classified

Digital Librarian

Data Archivist
Data Steward
Data curator
KU KDM'16

Cloud, Big Data and Data Science53

Data Science occupations in ESCO taxonomy (2)Technicians and associate professionals
Science andengineering associate
professionals

Data Science Technology
Professionals

Data Infrastructureengineers andtechnicians

Big Data facilities Operators

Large scale (cloud) data
storage operators
Database and network
professionals notelsewhere classified

Scientific database operator*)

Managers
Production andspecialised services
managers

Data Science/Big Data
Infrastructure Managers

Data Science/Big Data
Infrastructure Manager
Research InfrastructureManagers

RI Manager
RI Data storage facilities
manager

Clerical support workers
General andkeyboard clerks
Data handling support Data and information
workers (alternative)
entry and access

Digital Archivists andLibrarians

Digital Librarian

Data Archivist
Data Steward
KU KDM'16

Cloud, Big Data and Data Science

Data curator

54