CN

CN
CN

CN
CN

CN

CN
CN

CN

CN
CN

5 листопада 2013, 19:00 Зустріч-дискусія у Часописі

Сучасні Хмарні технології та технології Великих
Даних: Можливості, тенденції та виклики длябізнесу та науки
Юрій Демченко - кандидат технічних наук,старший науковий співробітник Університету
Амстердама
Данило Гордиєнко & Михайло Ткаченко представники датацентра Королівства
Нідерландів і фірми SafeDataService

5 Nov 2013, Chasopys, Kiev

Теми презентації та питання для дискусії
1. Характеристики і можливості хмарних технологій, тенденції розвиткута стандартизація.
2. Приклади використання та типи впровадження комп'ютерних хмар:корпоративні, публічні, комунальні; міграція корпоративної ІТінфраструктури на хмарну платформу, необхідні передумови і рівень
"зрілості", переваги віртуалізації сервісів і ресурсів.
3. Законодавча та регуляторна база в Європі, програми підтримки
впровадження хмар в Європі.
4. Глобальні провайдери хмарних послуг і ресурсів: Amazon AWS,
Microsoft Azure, GoogleCloud: можливості, послуги, засоби розробки.
5. Великі Дані: Об'єм, Швидкість, Номенклатура, Мінливість, Цінність ,Достовірність (Volume, Velocity, Variety, Variability, Value, Veracity).
6. Великі Дані та бізнес- аналітика: приклади використання і новіможливості.
7. Проблеми Великих Даних: зберігання, передача, обробка, контроль
доступу, захист даних і персональної інформації.
8. Нові спеціальності для Хмарних технологій та Великих Даних:підготовка фахівців, тренінг та освіта.
5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

2Сучасні Хмарні технології та технології Великих Даних:Можливості, тенденції та виклики для бізнесу та науки
Cloud and Big Data Technologies:
Opportunities and Challenges for business and science

Yuri Demchenko
SNE Group, University of AmsterdamДискусія в Часописі,5 листопада 2013, Київhttp://www.uazone.org/demch/presentations/kiev2013chaspys-cloud-bigdata-biz-v02.pdf

Outline••••

Cloud Computing definition and features
Cloud Computing Reference Architecture and standardisation
Amazon AWS IaaS cloud and cloud powered design principles
European Cloud Computing strategy and legislation• Big Data and Data Intensive Science as a new technology wave
– Big Data 5+1 Vs: Volume, Velocity, Variety, Value, Variability, Veracity• Big Data in Science, Industry and Business– Where do the data come from? What are Big Data drivers?

• Defining Big Data Architecture Framework (BDAF)– Big Data Infrastructure (BDI) and Big Data Analytics tools• Data Scientist: New profession and need for Education&Training• Summary and Discussion
5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

Slide_4

Big Data Research at System and Network
Engineering (SNE), University of Amsterdam• Long time research and development on Infrastructure services– High speed optical networking and data intensive applications– Application and infrastructure security services– Collaborative systems, Grid, Clouds and currently Big Data

• Focus on Infrastructure definition and services– Software Defined Infrastructure based on Cloud/Intercloud technologies• Standardisation activity (IETF, OGF, ISO - past)– NIST Cloud Computing Reference Architecture (past) and Big Data Working Group– Research Data Alliance: Education and Data Analytics Tools• Big Data Interest Group at SNE, UvA– Non-formal but active, meets two-weekly for brainstorming sessions– Provides input to NIST BD-WG and RDA• SNE Technical Report on Architecture Framework and Components for
the Big Data Ecosystem. Draft Version 0.2, 12 September 2013
http://www.uazone.org/demch/worksinprogress/sne-2013-02-techreport-bdaf-draft02.pdf

5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

5

Gartner Technology Hypercycle
Big Data
Cloud Computing

Source http://www.gartner.com/technology/research/methodologies/hype-cycle.jsp
5 Nov 2013, Chasopys, Kiev

Cloud and Big Data6

5 yr for Cloud Computing
2 yr for Big Data adoption

5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

7

Technology Definitions and Timeline - Overview•

Service Oriented Architecture (SOA): First proposed in 1996 and revived with
the Web Services advent in 2001-2002– Currently standard for industry, and widely used
– Provided a conceptual basis for Web Services development•

Computer Grids: Initially proposed in 1998 and finally shaped in 2003 with theOpen Grid Services Architecture (OGSA) by Open Grid Forum (OGF)– Currently remains as a collaborative environment– Migrates to cloud and inter-cloud platform•

Cloud Computing: Initially proposed in 2008 – Now entering productive phase– Defined new features, capabilities, operational/usage models and actually provided
a guidance for the new technology development– Originated from the Service Computing domain and service management focused•

Big Data and Data Intensive Science: Yet to be defined– Involves more components and processes to be included into the definition– Can be better defined as Ecosystem where data are the main driving component– Need to define the Big Data properties, expected technology capabilities and provide a
guidance/vision for future technology development

5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

8

Big Data and Clouds and Mobile Technologies –From disruptive to consolidating technologies•••

Service Oriented Architecture (SOA) - Industry
Computer Grids, Distributed Computing – Research community
Cloud Computing: Initially proposed in 2008 – Now entering productive phase(Industry)– Functional Cloud Computing definition provided a guidance for the technology
development– Consolidating SOA, Distributed computing, SDN– Facilitated by Mobile technologies, Big Data•

Big Data and Data Intensive Science – Originated from science– Consolidates Cloud Computing, Mobile technologies, High Performance computing,
Data warehousing, Data analytics/science– Emerges new data centric models and technologies• Introduces new technical category Ecosystem where data are the main driving component– Need to define the Big Data properties, expected technology capabilities and provide a
guidance/vision for future technology development

5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

9Основы технологии Облачных Вычислений

• Cloud Computing technology foundation

5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

10Анатомия Облачных Технологий
• Обьединние большого количества вычислительных ресурсов изапоминающих устройств в больших Центрах Обработки Даных(ЦОД)– Экономия размера/масштаба
– Горизонтальное и вертикальное масштабирование: балансировагние
нагрузки и эластичность

• Виртуализация сервисов, ресурсов и платформ
– Virtualisation == (Pooling) – Abstraction – Composition – Deployment –(Lifecycle management)– Механизмы привязки: пространство имен ибезопасность/доверительность
– Миграция виртуальных машин (ВМ; VM - Virtual Machine)• Многоуровневая модель виртуализации сервисов и ресурсов
– Много-пользовательская среда (multi-tenancy), учет, билинг

• Всеобщий и универсальный доступ через Интернет, всеобщая
коннективность
5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

11

Top Cloud Providers (July 2013) – Ranked 1-20
1.

Salesforce.com (revenue>$3Bln)2. Amazon AWS (>$1.5Bln)
3. Microsoft (>$1.5Bln)
4. Oracle
5. Google
6. SAP
7. SoftLayer (IBM since 2013) +IBM’s SmartCloud
8. Terremart (Verizon Company)
9. Rackspace
10. NetSuite ERP cloud service
provider ($308Mln)

5 Nov 2013, Chasopys, Kiev

11. Workday HR and financial cloudservices
12. Dropbox
13. Savvis (CenturyLink Company)
Oracle and private clouds
14. Joyent
15. Navisite (Time Warner) enterpriseapplications from IBM, MS, Oracle
16. Citrix Systems
17. LogMeIn Remote Management
platform provider
18. Zoho – alternative to Salesforce,
Offcie365, Google Apps
19. Dimension Data (NTT Group, ZA)
managed hosting
20 Carbinite backup provider for SQL,MS Exchange

Cloud and Big Data

12

European Cloud Computing Strategy (2012+)
In September 2012, the European Commission adopted a strategy for”Unleashing the Potential of Cloud Computing in Europe”• To deliver 2.5 mln new European jobs, and an annual boost of EUR 160
billion to EU GDP (around 1%)
Key actions• Safe and Fair Contract Terms and Conditions
–
–
–
–
–

data preservation after termination of the contract
data disclosure and integrity
data location and transfer
ownership of the data
direct and indirect liability change of service by cloud providers andsubcontracting• Cutting through the Jungle of Standards– EU cloud framework to emerge in next 18+ months, by 2015

• Establishing a European Cloud Partnership (ECP)• Mixed reactions from the major cloud players in Europe (e.g., Microsoft,
HP, IBM, majority from US)
5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

13

NIST Cloud definition – NIST SP 800-145 (1)NIST SP 800-145 The NIST Definition of Cloud Computing (Draft)
http://csrc.nist.gov/publications/drafts/800-145/Draft-SP-800-145_cloud-definition.pdf• Five Cloud characteristics– On-demand self-service– Broad network access– Resource pooling– Rapid elasticity– Measured Service• 3 basic service models– Software as a Service (SaaS)– Platform as a Service (PaaS)– Infrastructure as a Service (IaaS)• Deployment models– Private clouds– Public clouds– Hybrid clouds– Community clouds
5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

14

NIST Cloud definition – NIST SP 800-145 (1)NIST SP 800-145 The NIST Definition of Cloud Computing (Draft)
http://csrc.nist.gov/publications/drafts/800-145/Draft-SP-800-145_cloud-definition.pdf• Пять основных характеристик Облачных технологий
– Самообслуживание по требованию
– Широкополосный сетевой доступ
– Агрегатирование (пулинг) ресурсов
– Быстрая эластичность
– Измеряемые услуги (учет обьема услуг)• 3 базовые модели услуг
– Software as a Service (SaaS)– Platform as a Service (PaaS)– Infrastructure as a Service (IaaS)• Модели реализации
– Private clouds– Public clouds– Hybrid clouds– Community clouds
5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

15

NIST Cloud definition – Draft SP 800-145 (2)Cloud computing is a model for enabling ubiquitous, convenient, on-demand
network access to a shared pool of configurable computing resources (e.g.,
networks, servers, storage, applications, and services) that can be rapidly
provisioned and released with minimal management effort or service provider
interaction. This cloud model promotes availability and is composed of five essential
characteristics, three service models, and four deployment models.
Cloud Infrastructure as a Service (IaaS)
The capability provided to the consumer is to provision processing, storage,
networks, and other fundamental computing resources where the consumer is able
to deploy and run arbitrary software, which can include operating systems andapplications. The consumer does not manage or control the underlying cloudinfrastructure but has control over operating systems, storage, deployed applications,and possibly limited control of select networking components (e.g., host firewalls).
Note: NIST Definition of Cloud – missing network provisioning, just “limited control over
network”.
5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

16

NIST Cloud Definition – PaaS, SaaS
Platform as a Service (PaaS)
The capability provided to the consumer is to deploy onto the cloud infrastructure
consumer-created or acquired applications created using programming languages,
libraries, services, and tools supported by the provider. The consumer does not
manage or control the underlying cloud infrastructure including network, servers,
operating systems, or storage, but has control over the deployed applications andpossibly configuration settings for the application-hosting environment.

Software as a Service (SaaS)
The capability provided to the consumer is to use the provider’s applications runningon a cloud infrastructure. The applications are accessible from various client devices
through either a thin client interface, such as a web browser (e.g., web-based email),or a program interface. The consumer does not manage or control the underlying
cloud infrastructure including network, servers, operating systems, storage, or evenindividual application capabilities, with the possible exception of limited user-specificapplication configuration settings.

5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

17

NIST Cloud Computing Reference Architecture(CCRA) 2.0 - Main Roles (1)5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

18

NIST Cloud Computing Reference Architecture(CCRA) 2.0 - Provider Functions (2)Service provisioning
stages:Service Delivery(Framework)• Request&SLA• Reservation/
Composition• Deployment• Operation• DecommissioningФункции провайдера облачных услуг
• Внедрение.установка сервисов иресурсов
• Взаимодействие сервисов и ресурсов
• Управление ресурсами и сервисами
• Безопасность
• Приватность
5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

19

NIST Cloud Computing Reference Architecture(CCRA) 2.0 – Consolidated View (3)• txt

5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

20

ITU-T FG on Clouds Part 2: Functional
requirements and reference architecture
Layered Cloud computing architecture
includes:•User layer (including userfunctions, partner functions,administration functions)•
Access layer (including endpoint
functions and inter-cloudfunctions). Network service
providers role is to provide intercloud transport network•
Cloud services layer (including
basic cloud services IaaS, PaaS,
SaaS, NaaS, CaaS and also
Orchestration service)•
Resources and network layer(including physical resources,
pooling and orchestration, poolingand virtualisation)

5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

21

Cloud Reference Framework(IETF I-Draft )Cloud Reference Framework. Internet Draft, by B. Khasnabish, J. Chu, S.
Ma, Y. Meng, N. So, P. Unbehagen, M. Morrow, M. Hasan, Y. Demchenko
http://tools.ietf.org/html/draft-khasnabish-cloud-reference-framework-05.txt• Multilayer Cloud Services Model (CSM)– Including Federated Access and Delivery Infrastructure layer and userside services layer• Intercloud Architecture Framework (ICAF)– InterCloud Control and Management Plane (ICCMP)• Signaling, monitoring, synchronisation between heterogeneous clouds– InterCloud Federation Framework (ICFF)• Protocols and mechanisms for heterogeneous clouds integration– InterCloud Operations and Management Framework (ICOMF)• Services and business processes management and operation– Intercloud Security Framework (ICSF)
5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

22

Multilayer Cloud Services Model (CSM)
http://www.ietf.org/id/draft-khasnabish-cloud-reference-framework-05.txt

Security InfrastructureManagement

Operations Support System

User/Client Services* Identity services (IDP)* Visualisation

User/Customer Side Functions and Resources

Administration andManagement Functions(Client)

Content/Data Services* Data * Content * Sensor * Device

1

Endpoint Functions* Service Gateway* Portal/Desktop

Federated Access andDelivery Infrastructure (FADI)

Inter-cloud Functions* Registry and Discovery* Federation Infrastructure

Cloud Services (Infrastructure, Platform, Application, Software)

IaaS

SaaS

PaaS

PaaS-IaaS IF

Layer C5
Services
Access/Delivery

Layer C4Cloud Services(Infrastructure,
Platforms,
Applications,
Software)PaaS-IaaS Interface
IaaS – Virtualisation Platform Interface

Cloud Management
Software(Generic Functions)

Cloud Management Platforms
OpenNebula

Virtualisation Platform

OpenStack

KVM

VM

VM

StorageResources

ComputeResources

Contrl&Mngnt Links

VPN

Other
CMS

XEN

VMwareNetwork
Virtualis

Proxy (adaptors/containers) - Component Services and Resources

5 Nov 2013, Chasopys, Kiev

Layer C6
User/Customer
side Functions

Hardware/PhysicalResources

Network
Infrastructure

Layer C3
Virtual Resources
Composition andControl(Orchestration)

CSM layers(C6) User/Customer side
Functions(C5) Intercloud Access andDelivery Infrastructure(C4) Cloud Services(Infrastructure, Platform,
Applications)(C3) Virtual Resources
Composition andOrchestration(C2) Virtualisation Layer(C1) Hardware platform anddedicated network
infrastructure

Layer C2
Virtualisation

Layer C1
PhysicalHardwarePlatform andNetwork

Control/
Mngnt Links

Data Links

Data Links

Cloud and Big Data

Slide_23

General use case for infrastructure provisioning:
Workflow => Logical (Cloud) InfrastructureEnterprise/Scientific workflow
Storage
Data

SpecialProc 1
Data
Filtering

Input
Data

VisualPresent
SpecialProc 2

Instrum.
Data

Campus A

Data
Archive

Visualisation

Visualisation

CE

User
Group A

Campus B

CE

User
User
User

User
User
User
VR6

Cloud 2 PaaS

VR2

VR7
VR4

VR1

VR5

Resource/
Service
ProviderEnterprise/Scientificworkflow
Is mapped
to heterogeneous cloudinfrastructure containing
IaaS, PaaS components
5 Nov 2013, Chasopys, Kiev

User
Group B

VR3

Enterprise/Project basedIntercloud Infrastructure

Cloud 1 IaaS

Resource/
Service
Provider
CN

CN
CN

CN
CN

CN

Cloud PaaS Provider

CN
CN

CN

CN
CN

Cloud IaaS Provider

Cloud and Big Data

24

Intercloud Applications Integration:
ICCM, ICFF, ICOMF
ICOMF – InterCloudOperations and Mngnt
Framework
Storage
Data

SpecialProc 1

Data
Archive

Data
Filtering

Input
Data

ICFF – InterCloud
Federation Framework

Business Processes Management and Services
Operation Support• SLA Management• Business roles and Actors• Business level Service Registry and Broker• Mobility?

Intercloud Federation
Infrastructure
Registry
User
SW&AppsVisualPresent

Layer C3-IaaS

Disco-very

BrokerBrokerGW

LayerC4-PaaS

Trust
Broker

Provider Virt&OS
Platform

SpecialProc 2

Instrum.
Data

Layer 6 – SaaS/Apps
Layer 5 - PaaS
Cloud based Platform

Visualisation

Trust

ICCMP

ProviderdefinedInfrastr.
andVirt&OS
Platform

Layer 3 – Composition
Providerdefined
Virt&OS
Platform

Trust
BrokerPaaS Domain

Layer 4 - PaaS

Layer 4 - IaaS
Cloud Infrastructure

Layer 3 - IaaS

CE

Layer 2 - Virtualisation

Visualisation

User
User
User

User
User
User
VR6

Cloud 2 PaaS

VR2

User
Group B

VR7
VR4

VR1

VR5

Resource/
Service
Provider5 Nov 2013, Chasopys, Kiev

Campus B

CE

Layer 1 - Physical
HW Platform/Network

Resource/
Operational andService
Providerbusiness issues are
typically addressedby Operations services anda framework

Provider Virt&OS
Platform

Userdefined
SW&Apps

Layer 7 - Applications

User
definedPlatform
andSW&AppsUser
Group A

GW

IdP
Attribs

IaaS Domain Cloud Layers

Campus A

User
SW&Apps

VR3

Enterprise/Project basedIntercloud Infrastructure

Cloud 1 IaaS

CN

CN
CN

CN
CN

CN

Cloud PaaS Provider

CN
CN

CN

CN
CN

Cloud IaaS Provider

Cloud and Big Data

25

Cloud IaaS – облачные инфраструктурные
сервисы

5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

26

Cloud IaaS – Основные свойства и характеристики
облачных систем и инфраструктур•Все фундаментальные свойства и характеристики компьютерных облаков должны
быть адресованы и внедреныв IaaS•Масштабируемость иэластичность
––•Доступность и надежность
–
–
–

•

–
–
Cloud and Big DataПараллельная обработка
Баланс нагрузки
Планирование работ

Доступ и переносимость
–5 Nov 2013, Chasopys, Kiev

Управляющая информация
Мониторинг систем
Биллинг

Производительность иоптимизация
–
–
–

•

Устойчивость к сбоям
Устойчивость системы
Безопасность

Управляемость исовместимость
–
–
–

•

Динамичное внедрение иустановка
Много-пользовательская среда

Стандартный доступ (через
Интернет)Тонкий клиент
Стандартизация
27

Cloud IaaS Architecture• Infrastructure as a Service (IaaS) delivers computer infrastructure
for cloud user, typically a platform virtualization environment as aservice.
• Virtualization is an enabling technique to provide an abstraction oflogical resources away from underlying physical resources.

5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

28

Multilayer Cloud Services Model (CSM)

Security InfrastructureManagement

Operations Support System

User/Client Services* Identity services (IDP)* Visualisation

User/Customer Side Functions and Resources

Administration andManagement Functions(Client)

Content/Data Services* Data * Content * Sensor * Device

1

Endpoint Functions* Service Gateway* Portal/Desktop

Access and Delivery
InfrastructureInter-cloud Functions* Registry and Discovery* Federation Infrastructure

Cloud Services (Infrastructure, Platform, Application, Software)

IaaS

SaaS

PaaS

PaaS-IaaS IF

Layer C5
Services
Access/Delivery

Layer C4Cloud Services(Infrastructure,
Platforms,
Applications,
Software)PaaS-IaaS Interface
IaaS – Virtualisation Platform Interface

Cloud Management
Software(Generic Functions)

Cloud Management Platforms
OpenNebula

Virtualisation Platform

OpenStack

KVM

VM

VM

StorageResources

ComputeResources

Contrl&Mngnt Links

VPN

Other
CMS

XEN

VMwareNetwork
Virtualis

Proxy (adaptors/containers) - Component Services and Resources

5 Nov 2013, Chasopys, Kiev

Layer C6
User/Customer
side Functions

Hardware/PhysicalResources

Network
Infrastructure

Layer C3
Virtual Resources
Composition andControl(Orchestration)

CSM layers(C6) User/Customer side
Functions(C5) Intercloud Access andDelivery Infrastructure(C4) Cloud Services(Infrastructure, Platform,
Applications)(C3) Virtual Resources
Composition andOrchestration(C2) Virtualisation Layer(C1) Hardware platform anddedicated network
infrastructure

Layer C2
Virtualisation

Layer C1
PhysicalHardwarePlatform andNetwork

Control/
Mngnt Links

Data Links

Data Links

Cloud and Big Data

Slide_29

Amazon AWS Cloud Architecture

Credits “Building Powerful Web Applications in the AWS Cloud” by Louis Columbus
http://softwarestrategiesblog.com/2011/03/10/building-powerful-web-applications-in-the-aws-cloud/

5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

30

Amazon EC2 User Application Component Services• EBS Elastic Block Store• Elastic IP Address dynamically assigned to user VMs• VPC (Virtual Private Cloud) allows organizations to use AWS
resources along with their existing infrastructure in a VPN(Virtual Private Network) to increase compute capabilities
beyond the local resources.
• CloudWatch monitoring service• Auto Scaling dynamic resource provision• Elastic Load Balancing between multiple VMs located within
a single availability zone or multiple zones• VM Import/Export for custom VM images store and load
• Cloud Formation services composition and deployment
5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

31

Amazon S3 (Simple Storage Service)
S3 is a service that stores large amounts of data, and is accessible via the Internet• Amazon S3 is intentionally built with a minimal feature set• Data stored as objects associated with unique keys
• Objects can store up to 5 TB of data and are bound to specific buckets which
can only be stored in a particular region (availability zone)“Data stored in Amazon S3 is secure by default” :-); only bucket and object
owners have access to the Amazon S3 resources they create• Amazon S3 supports multiple access control mechanisms, as well as encryption
for both secure transit and secure storage on disk
• With Amazon S3’s data protection features, you can protect your data from both
logical and physical failures, guarding against data loss from unintended user
actions, application errors, and infrastructure failures• For customers who must comply with regulatory standards such as PCI and
HIPAA, Amazon S3’s data protection features can be used as part of an overall
strategy to achieve compliance• Cost issues: free upload, paid download

5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

32

New Service: Amazon Direct Connect• AWS Direct Connect links
customer internal network to an
AWS Direct Connect location overa standard 1 Gbps or 10 Gbps
Ethernet fiber-optic cable– One end of the cable is connected
to customer router, the other to an
AWS Direct Connect router– Allows creating virtual interfaces
directly to the AWS cloud (Amazon
EC2, S3) and to Amazon Virtual
Private Cloud (Amazon VPC),
bypassing Internet service
providers in your network path

• Access is limited to Amazon Web
Services in the region
5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

33

Example: Cloud powered services design with AWS• Based on AWS seminar presentation“Building Powerful Web Applications in the AWS Cloud” by Louis
Columbus
http://softwarestrategiesblog.com/2011/03/10/building-powerful-web-applicationsin-the-aws-cloud/Pattern #1: Design for failure and nothing will fail
Pattern #2: Edge cache static content
Pattern #3: Implement Elasticity
Pattern #4: Leverage Multiple Availability Zones
Pattern #5: Isolate read and write traffic; Isolate static and dynamic traffic
Pattern #6: Hardening security at every stage
Pattern #7: Parallel Processing
Pattern #8: Go global quickly (with single API)
Pattern #9: Automate your in-cloud Development and Deployment Lifecycle
Pattern #10: Keep optimizing and see the savings in the next month's bill

5 Nov 2013, Chasopys, Kiev

Cloud and Big Data34

Hardening Security

Credits “Building Powerful Web Applications in the AWS Cloud” by Louis Columbus
http://softwarestrategiesblog.com/2011/03/10/building-powerful-web-applications-in-the-aws-cloud/

5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

35

Cloud powered development lifecycle

For the Test/QA stage:
Speed: quickly get on-demand resources
Variety: test more demo configurations
Real world load and stress testing; easy simulates
100s of clients
Repeatability: preconfigured shareable Test DB inminutes
Reproducibility: “Save As” Productions Environmentand re-launch in Test Environment
Savings: “Turn off” Testing Environment

•
•
•
•

Environmental separation
Environmental consistency
Variable resource
Different control levels

Credits “Building Powerful Web Applications in the AWS Cloud” by Louis Columbus
http://softwarestrategiesblog.com/2011/03/10/building-powerful-web-applications-in-the-aws-cloud/

5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

36Технологии Больших Даных

5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

37

Visionaries and Drivers:
Seminal works and High level reports
The Fourth Paradigm: Data-Intensive Scientific Discovery.
By Jim Gray, Microsoft, 2009. Edited by Tony Hey, et al.
http://research.microsoft.com/en-us/collaboration/fourthparadigm/
Riding the wave: How Europe can gain fromthe rising tide of scientific data.
Final report of the High Level Expert Group on
Scientific Data. October 2010.
http://cordis.europa.eu/fp7/ict/einfrastructure/docs/hlg-sdi-report.pdf

https://www.rd-alliance.org/

AAA Study: Study on
AAA Platforms For
Scientificdata/information
Resources in Europe,
TERENA, UvA, LIBER,
UinvDeb.

NIST Big Data Working Group (NBD-WG)
https://www.rd-alliance.org/
5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

38

The Fourth Paradigm of Scientific Research
1. Theory and logical reasoning
2. Observation or Experiment––E.g. Newton observed apples falling to design his theory ofmechanics
But Gallileo Galilei made experiments with falling objects from thePisa leaning tower

3. Simulation of theory or model–

Digital simulation can prove theory or model

4. Data-driven Scientific Discovery (aka Data Science)– More data beat hypnotized theory

5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

39

Big Data and Data Intensive Science - The
next technology focus
Scientific and Research Data – e-Science• Big Data is/has becoming the next buzz word
– Not much academic research and papers – Read seminal works, Dive into blogs and tweets•

Based on e-Science concept and entire information and artifacts digitising– Requires also new information and semantic models for information structuring
and presentation– Requires new research methods using large data sets and data mining• Methods to evolve and results to be improved•

Changes the way how the modern research is done (in e-Science)– Secondary research, data re-focusing, linking data and publications•

Big Data require a new infrastructure to support both distributed data(collection, storage, processing) and metadata/discovery services– High performance network and computing, distributed storage and access– Cloud Computing as native platform for distributed dynamic virtualised (datasupporting) infrastructure– Demand for trusted/trustworthy infrastructure

5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

40

Big Data Definitions Overview•

IDC definition of Big Data (conservative and strict approach) :"A new generation of technologies and architectures designed to economically extract value
from very large volumes of a wide variety of data by enabling high-velocity capture,
discovery, and/or analysis“•

Gartner definition
Big data is high-volume, high-velocity and high-variety information assets that demand costeffective, innovative forms of information processing for enhanced insight and decision
making. http://www.gartner.com/it-glossary/big-data/–•

Big Data: a massive volume of both structured and unstructured data that is so large thatit's difficult to process using traditional database and software techniques.
–•

From “The Big Data Long Tail” blog post by Jason Bloomberg (Jan 17, 2013). http://www.devx.com/blog/the-big-datalong-tail.html“Data that exceeds the processing capacity of conventional database systems. The data is
too big, moves too fast, or doesn’t fit the structures of your database architectures. To gain
value from this data, you must choose an alternative way to process it.”–•Termed as 3 parts definition, not 3V definition

Ed Dumbill, program chair for the O’Reilly Strata Conference

Termed as the Fourth Paradigm *)“The techniques and technologies for such data-intensive science are so different that it isworth distinguishing data-intensive science from computational science as a new, fourthparadigm for scientific exploration.” (Jim Gray, computer scientist)*) The Fourth Paradigm: Data-Intensive Scientific Discovery. Edited by Tony Hey, Stewart Tansley, and Kristin Tolle. Microsoft, 2009.

5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

41

Improved: 5+1 V’s of Big Data
VolumeVariety
•
•
•
•
•
•

Structured
Unstructured
Multi-factor
Probabilistic
LinkedDynamic••••

Terabytes
Records/Arch
Tables, Files
Distributed
•
•
•
•

Batch
Real/near-time
Processes
Streams

6 Vs ofBig Data• Changing data
• Changing model• Linkage

Variability

Velocity

•
•
•
•
•

Value
•
•
•
•

Correlations
Statistical
EventsHypothetical

Trustworthiness
Authenticity
Origin, Reputation
Availability
Accountability

Generic Big Data
Properties• Volume• Variety• Velocity

Acquired Properties(after entering system)• Value• Veracity• Variability
Commonly accepted
3V’s of Big Data

Veracity
5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

42

Improved: 5+1 V’s of Big DataОбьемНоменклатура
•
•
•
•
•
•

Structured
Unstructured
Multi-factor
Probabilistic
LinkedDynamic••••

Terabytes
Records/Arch
Tables, Files
Distributed••••

6 Vs ofBig Data• Changing data
• Changing model• LinkageИзменчивость

Скорость

•
•
•
•
•

Trustworthiness
Authenticity
Origin, Reputation
Availability
Accountability

Batch
Real/near-time
Processes
StreamsБазовые свойства
• Обьем
• Скорость
• Номенклатура
(разнообразие)Приданные свойства
Ценность (после обработки)• Correlations• Ценность
• Statistical• Events• Достоверность
• Hypothetical• Изменчивость
Commonly accepted
3V’s of Big DataДостоверность
5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

43

Big Data Definition: From 5+1V to 5 Parts (1)(1) Big Data Properties: 5V– Volume, Variety, Velocity, Value, Veracity– Additionally: Data Dynamicity (Variability)(2) New Data Models– Data Lifecycle and Variability– Data linking, provenance and referral integrity(3) New Analytics–

Real-time/streaming analytics, interactive and machine learning analytics(4) New Infrastructure and Tools––––

High performance Computing, Storage, Network
Heterogeneous multi-provider services integration
New Data Centric (multi-stakeholder) service models
New Data Centric security models for trusted infrastructure and data processing
and storage(5) Source and Target– High velocity/speed data capture from variety of sensors and data sources– Data delivery to different visualisation and actionable systems and consumers– Full digitised input and output, (ubiquitous) sensor networks, full digital control
5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

44

Big Data Definition: From 5+1V to 5 Parts (1)(1) Big Data Properties: 5V– Volume, Variety, Velocity, Value, Veracity– Additionally: Data Dynamicity (Variability)(2) New Data Models– Data linking, provenance and referral integrity– Data Lifecycle and Variability/Evolution(3) New Analytics–

Real-time/streaming analytics, interactive and machine learning analytics(4) New Infrastructure and Tools––––

High performance Computing, Storage, Network
Heterogeneous multi-provider services integration
New Data Centric (multi-stakeholder) service models
New Data Centric security models for trusted infrastructure and data processing
and storage(5) Source and Target– High velocity/speed data capture from variety of sensors and data sources– Data delivery to different visualisation and actionable systems and consumers– Full digitised input and output, (ubiquitous) sensor networks, full digital control
5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

45

Big Data Definition: From 5V to 5 Parts (2)Refining Gartner definition“Big data is (1) high-volume, high-velocity and high-variety information assets thatdemand (3) cost-effective, innovative forms of information processing for (5)enhanced insight and decision making”•

Big Data (Data Intensive) Technologies are targeting to process (1) high-volume,
high-velocity, high-variety data (sets/assets) to extract intended data value andensure high-veracity of original data and obtained information that demand(3) cost-effective, innovative forms of data and information processing (analytics)
for enhanced insight, decision making, and processes control; all of thosedemand (should be supported by) (2) new data models (supporting all data statesand stages during the whole data lifecycle) and (4) new infrastructure services
and tools that allows also obtaining (and processing data) from (5) a variety ofsources (including sensor networks) and delivering data in a variety of forms todifferent data and information consumers and devices.
(1) Big Data Properties: 5V(2) New Data Models(3) New Analytics(4) New Infrastructure and Tools(5) Source and Target

5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

46

Big Data Origin• Science• Internet, Web• Industry• Business• Living Environment,
Cities• Social media andnetworks• Healthcare• Telecom/Infrastructure

5 Nov 2013, Chasopys, Kiev

Data Transformation

Big Data Nature: Origin and consumers (target)

Cloud and Big Data

Big Data Target Use• Scientific discovery• New technologies• Manufacturing,processes, transport• Personal services,
campaigns• Living environment
support• Healthcare support• Social Networking

47

Volume, Velocity, Variety – Examples e-Science• Volume – Terabyte records, transactions, tables, files.
– LHC – 5 PB a month (now is under re-construction)– LOFAR, SKA – 5 PB every hour, requires processing asap to discard
non-informative data
– Large Synoptic Survey Telescope (LSST) - 10 Petabytes per year
– Genomic research – x10 TB per individual– Earth, climate and weather data

• Velocity – batch, near-time, real-time, streams.
– LHC ATLAS detector generates about 1 Petabyte raw data persecond, during the collision time about 1 ms

• Variety – structures, unstructured, semi-structured, and all
the above in a mix– Biodiversity, Biological and medical, facial research– Human, psychology and behavior research– History, archeology and artifacts

5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

48

Volume, Velocity, Variety – Examples Industry• Volume – Terabyte records, transactions, tables, files.
– A Boeing Jet engine produce out 10TB of operational data for every
30 minutes they run– Hence a 4-engine Jumbo jet can create 640TB on one Atlantic
crossing. Multiply that to 25,000 flights flown each day• Velocity – batch, near-time, real-time, streams.
– Today’s on-line ads serving requires 40ms to respond with a decision– Financial services (i.e., stock quotes feed) need near 1ms to calculate
customer scoring probabilities– Stream data, such as movies, need to travel at high speed for proper
rendering• Variety – structures, unstructured, semi-structured, and all
the above in a mix– WalMart processes 1M customer transactions per hour and feeds
information to a database estimated at 2.5PB (petabytes)– There are old and new data sources like RFID, sensors, mobile
payments, in-vehicle tracking, etc.
5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

49

Big Data technology drivers (1)• Modern e-Science in search for new knowledge– Scientific experiments and tools are becoming bigger andheavily based on data processing and mining• Traditional data intensive industry– Genomic research, drugs development, Healthcare– High-tech industry, CAD/CAM, weather/climate, etc.

• Intelligence and security• Network/infrastructure management– Network monitoring, Intrusion detection, troubleshooting

5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

50

Big Data technology drivers (2)• Consumer facing companies like Google and Facebook have
driven many of the recent advances in Big Data efficiency– Facebook has some 900+ million users and is still growing– Google handles number of search queries at 3 billion per day– Twitter handles some 400 million tweets per day count for 12 terabytes
per day• Used also for market sentiments prediction– Power companies: process up to 350 billion annual meter readings tobetter predict power consumption• Processes/activity data recording and analysis– Flight data, log data, intelligence, traffic• Business (retail) uses Big Data technologies “to search” forcustomers– Modern business concept (multi-channel) of delivering directly tocustomers requires prediction of customer behavior• Data volumes – What cause(s) and what effect?

– Big Data gives companies a fighting chance in the battle over thecustomer
5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

51

Big Data technology drivers (3) - Advertising• “… this new course of big data, gleaned from a wealth of unstructured
information on the web, has the ability to turn advertising on its head–– atleast enough to make media people rethink algorithms for maximizing
performance.” HessieJ.com– Traditional Ad Model: User profiles– More Sophisticated Ad Model: Behavioral targeting - "smart ads"
– Future Ad Model: Enter Social Data

• Example:– Mary Brown searches for information about a future trip to Mallorca• She also goes to travel sites, reads hotel reviews and has excitedly spoken to close friends on
Twitter and Facebook about her plans and preparations– Now we have not only recent behavioral activity where she’s been on the internet,
but we also are aware of her conversations that validate her behavior– It is safe to assume that Mary will “definitely” be going to Mallorca– What this information does for a travel company?
• They now have MORE information on that user that will allow them to not only serve an ad, oreven respond to that user with relevant offers, but DO so with a certain degree of confidence
that Mary will at the very least click on the ad.

5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

52

Big Data technology drivers (3a) – Service
Delivery• Consumer products and services delivery– Netflix already captures movie genre preferences by the user
and makes recommendations based on recent shows/movies
watched• Announced $2mln prize for effective customer targeting in 2003
• Netflix recommender system in use as a reference technology
implementation– It is already capturing which devices the user is watching
recent programs/shows and when
• Marrying that data with GetGlue (news feed on movies), for example,
validates the original information and supplements the usage
information– Combined and correlating, allows Netflix to optimize the movie
offering to keep you a satisfied customer– It can also capture the comments and shares from thosewatching the movie in order to drive messaging to attract newusers
5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

53

Big Data technology drivers (3b) – Managing
public campaigns, e.g. election, public relations• The rise of public opinion stored in platforms like Twitter,
Google, Facebook, etc. provide enough intelligence toinfluence the campaign development, timing, geography andeven the colour of the campaign signs– Twitter was a major source of data aggregation for the Republican
Race in the US– Multimillion-dollar contract for data management
and collection services awarded May 1, 2013 to
Liberty Work to build advanced list of voters• Article “In Data we trust” by T.Edsall in The New York Times– Book: In Data We Trust: How Customer Data is
Revolutionising Our Economy (Aug 2012)• A strategy for tomorrow's data world

5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

54

Big Data technology drivers (4) - Emerging• Social media itself – share and socialise/collaborate– Facebook, Twitter, YouTube, Flickr, etc.

• Workplace improvement– Means more data will be collected and monitored on the
personnel• Healthcare, health/physiological and medical
information– Human health monitoring – not just for ill or aged people• Early diagnostics, proactive care advising

5 Nov 2013, Chasopys, Kiev

Cloud and Big Data55

http://mattturck.com/2012/10/15/a-chart-of-the-big-data-ecosystem-take-2/

5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

56

Foreseen Big Data Innovations in 2013+•

Cloud-Based Big Data Solutions– Amazon’s Elastic Map Reduce (EMR) is a market leader– Expected new innovative Big Data and Cloud solutions•

Real-Time Hadoop– Google’s Dremel-like solutions that will allow real-time queries on Big Data and be open
source•Distributed and deep Machine Learning– Mahout iterative scalable distributed back propagation machine learning and data mining
algorithm– New algorithms Jubatus, HogWild•

Big Data Appliances (also for home)– Raspberry Pi and home-made GPU clusters– Hardware vendors (Dell, HP, etc.) pack mobile ARM processors into server boxes– Adepteva's Parallella will put a 16-core supercomputer into range of $99•

Easier Big Data Tools– Open Source and easy to use drag-and-drop tools for Big Data Analytics to facilitate the BD
adoption– Commercial examples: Radoop = RapidMiner + Mahout, Tableau, Datameer, etc.
– LexisNexis: from ECL (Enterprise Control Language) to KEL (Knowledge Engineering
Language)

Source: Big Data in 2013 by Mike Guattieri, Forrester
5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

57

NIST Big Data Working Group (NBD-WG)• Deliverables target – September 2013– 26 September – initial draft documents– 30 September – Workshop and F2F meeting• Activities: Conference calls every day 17-19:00 (CET) by subgroup http://bigdatawg.nist.gov/home.php
–
–
–
–
–

Big Data Definition and Taxonomies
Requirements (chair: Geoffrey Fox, Indiana Univ)
Big Data Security
Reference Architecture
Technology Roadmap• BigdataWG mailing list and useful documents– Input documents http://bigdatawg.nist.gov/show_InputDoc2.php– Big Data Reference Architecture
http://bigdatawg.nist.gov/_uploadfiles/M0226_v7_2611176301.docx– Big Data Architectures Survey
http://bigdatawg.nist.gov/_uploadfiles/M0151_v2_7447424902.docx– Requirements based on 51 usecases
http://bigdatawg.nist.gov/_uploadfiles/M0224_v1_1076079077.xlsx

5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

58

Defining Big Data Architecture Framework• Existing attempts don’t converge to something
consistent: ODCA, TMF, NIST
– See http://bigdatawg.nist.gov/_uploadfiles/M0151_v2_7447424902.docx• Architecture vs Ecosystem– Big Data undergo a number of transformations during theirlifecycle– Big Data fuel the whole transformation chain• Data sources and data consumers, target data usage– Multi-dimensional relations between• Data models and data driven processes• Infrastructure components and data centric services• Architecture vs Architecture Framework– Separates concerns and factors• Control and Management functions, orthogonal factors– Architecture Framework components are inter-related
5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

59

Big Data Architecture Framework (BDAF) (1)(1) Data Models, Structures, Types– Data formats, non/relational, file systems, etc.

(2) Big Data Management– Big Data Lifecycle (Management) Model• Big Data transformation/staging– Provenance, Curation, Archiving(3) Big Data Analytics and Tools– Big Data Applications• Target use, presentation, visualisation(4) Big Data Infrastructure (BDI)– Storage, Compute, (High Performance Computing,) Network– Sensor network, target/actionable devices– Big Data Operational support(5) Big Data Security– Data security in-rest, in-move, trusted processing environments
5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

60

Data
SourceData
Collection&
Registration

Data
Filter/Enrich,Classification

Data
Analytics,
Modeling,
Prediction

Data
Delivery,
Visualisation

Consumer

Big Data Ecosystem: Data, Transformation,
Infrastructure

Big Data Target/Customer: Actionable/Usable Data
Target users, processes, objects, behavior, etc.
Federated
AccessandDeliveryInfrastructure(FADI)Big Data Source/Origin (sensor, experiment, logdata, behavioral data)

Big Data Analytic/Tools

Storage
GeneralPurposeData Management

ComputeGeneralPurposeHigh
Performance
Computer
Clusters

StorageSpecialised
Databases
Archives(analytics DB,
In memory,
operstional)

Data
categories: metadata,(un)structured, (non)identifiable
Data
Datacategories:
categories:metadata,
metadata,(un)structured,(un)structured,(non)identifiable(non)identifiable

Intercloud multi-provider heterogeneous Infrastructure
Security Infrastructure

5 Nov 2013, Chasopys, Kiev

Network InfrastructureInternal

InfrastructureManagement/Monitoring

Cloud and Big Data

61

General BDI services and components• Data management infrastructure and tools• Registries, search/indexing, ontologies, schemas,
namespace• Collaborative Environment (user/groups managements)• Heterogeneous multi-provider Inter-cloud infrastructure– Compute, Storage, Network (provisioned on-demand dynamicallyscaling)– Federated Access and Delivery Infrastructure (FADI)• Advanced high performance (programmable) network• Security infrastructure (access control, Identity and policy
management, confidentiality, privacy, trust)

5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

62

Big Data Infrastructure and Analytic Tools
Big Data Target/Customer: Actionable/Usable Data
Target users, processes, objects, behavior, etc.
Big Data Source/Origin (sensor, experiment, logdata, behavioral data)

Big Data Analytic/Tools
Analytics:
Refinery, Linking, Fusion

Analytics :Realtime, Interactive,
Batch, Streaming

Storage
GeneralPurposeData Management

ComputeGeneralPurposeAnalytics Applications
Link Analysis
Cluster Analysis
Entity Resolution
Complex Analysis

High
Performance
Computer
Clusters:

Federated
AccessandDeliveryInfrastructure(FADI)

StorageSpecialised
Databases
Archives

Data
categories: metadata,(un)structured, (non)identifiable
Data
Datacategories:
categories:metadata,
metadata,(un)structured,(un)structured,(non)identifiable(non)identifiable

Intercloud multi-provider heterogeneous Infrastructure
Security Infrastructure

5 Nov 2013, Chasopys, Kiev

Network InfrastructureInternal

InfrastructureManagement/Monitoring

Cloud and Big Data

63

Big Data Analytics Infrastructure• High Performance Computer Clusters (HPCC)• Specialised Storage, Distributed/Replicated, Archives,
Databases, SQL/NoSQL• Big Data Analytics Tools/Applications• Analytics/processing: Real-time, Interactive, Batch,
Streaming• Link Analysis, Graph analysis• Cluster Analysis• Entity Resolution• Complex Analysis

5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

64

Data Transformation/Lifecycle Model
Common Data Model?
•
Data Variety and Variability•
Semantic Interoperability

Data Model (1)Data Model (1)Data Model (4)Data (inter)linking?
•
Persistent ID•
Identification•
Privacy, Opacity

Data Storage

Data
Collection&
Registration

Data
Filter/Enrich,Classification

Data
Analytics,
Modeling,
Prediction

Data
Delivery,
Visualisation

Consumer
Data Analitics
Application

Data
SourceData Model (3)Data repurposing,Analitics re-factoring,
Secondary processing••

Does Data Model changes along lifecycleor data evolution?
Identifying and linking data–––

Traceability vs Opacity
Referral integrity

Persistent identifier

5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

65

Evolutional/Hierarchical Data Model
Actionable Data

Papers/Reports

Archival Data

Usable Data

Processed Data (for target use)
Processed Data (for target use)
Processed Data (for target use)

Classified/Structured DataClassified/Structured DataClassified/Structured Data

Raw Data••••

Common Data Model?
Data interlinking?
Fits to Graph data type?
Metadata

5 Nov 2013, Chasopys, Kiev

•
•
•
•

Referrals
Control information
PolicyData patterns

Cloud and Big Data

66

Data Science: Research and Education

5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

67

Horizon2020: Challenge (2.1): Development,

deployment and operation of e-infrastructures
Call on Specific Challenge 2.1: Development, deployment and operation of einfrastructures (based on discussion draft summer 2013)
CHALLENGE 1 – High Performance Computing (HPC)
CHALLENGE 2 - CONNECTIVITY
Topic 4: Research and Education Networking – GÉANTCHALLENGE 3 - DATA
Topic 5: Community data services
Topic 6: Managing, preserving and computing with big research data
Topic 7: e-Infrastructure for Open Access
Topic 8: Towards global data e-infrastructures - RDA

CHALLENGE 4 – e-INFRASTRUCTURE INTEGRATION
Topic 9: e-Infrastructures for virtual research environments (VRE)
Topic 10: Provisioning of core services across e-Infrastructures
Topic 11: Skills and new professions for e-infrastructures

CHALLENGE 5 – POLICY AND INTERNATIONAL
Topic 12: Policy development and international cooperation
5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

68

Data Scientist: New Profession and Opportunities
McKinsey Institute on Big Data Jobs
http://www.mckinsey.com/mgi/publications/big_data/index.asp•

There will be a shortage of talent necessary for organizations to take advantage
of Big Data.
––

By 2018, the United States alone could face a shortage of 140,000 to 190,000 people with deep
analytical skills as well as
1.5 million managers and analysts with the know-how to use the analysis of big data to makeeffective decisions

SOURCE:US Bureau of Labor Statistics; US Census; Dun & Bradstreet; company interviews; McKinsey analysis
5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

69

Slide from the presentation
Demystifying Data Science(by Natasha Balac, SDSC)

5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

70

Strata Survey Skills and Data Scientist Self-ID

Analysing the Analysers. O’Reilly Strata
Survey – Harris, Murphy&Vaisman, 2013
• Based on how data scientists think
about themselves and their work
• Identified four Data Scientist clusters

5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

71

Skills and Self-ID Top Factors
Business

ML/BigData
Math/OR
Programming
Statistics

ML – Machine Learning
OR – Operations Research
5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

72

Key to a Great Data Scientist
Technical skills (Coding, Statistics, Math)+ Commitment + Creativity+ Intuition+ Presentation Skills+ Business Savvy= Great Data Scientist!
• How Long Does It Take For a Beginner to Become a Good
Data Scientist?
– 3-5 years according to KDnuggets survey [278 votes total]

5 Nov 2013, Chasopys, Kiev

Cloud and Big Data73

Summary Big Data
• Researching, learning mastering Big Data domain is a Big Data
problem itself• Cloud Computing as a native platform for Big Data
– Acceptance of clouds will grow, so demand for specialists• Demand for advanced high performance network will remain andgrow• New generically data centric models are required• New distributed data processing and analytics computing models
to be developed/re-factored• Data Scientist is a new focus for talents search by companies

5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

74Теми презентації та питання для дискусії
1. Характеристики і можливості хмарних технологій, тенденції розвиткута стандартизація.
2. Приклади використання та типи впровадження комп'ютерних хмар:корпоративні, публічні, комунальні; міграція корпоративної ІТінфраструктури на хмарну платформу, необхідні передумови і рівень
"зрілості", переваги віртуалізації сервісів і ресурсів.
3. Законодавча та регуляторна база в Європі, програми підтримки
впровадження хмар в Європі.
4. Глобальні провайдери хмарних послуг і ресурсів: Amazon AWS,
Microsoft Azure, GoogleCloud: можливості, послуги, засоби розробки.
5. Великі Дані: Об'єм, Швидкість, Номенклатура, Мінливість, Цінність ,Достовірність (Volume, Velocity, Variety, Variability, Value, Veracity).
6. Великі Дані та бізнес- аналітика: приклади використання і новіможливості.
7. Проблеми Великих Даних: зберігання, передача, обробка, контроль
доступу, захист даних і персональної інформації.
8. Нові спеціальності для Хмарних технологій та Великих Даних:підготовка фахівців, тренінг та освіта.
5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

75

Questions and Discussion

5 Nov 2013, Chasopys, Kiev

Cloud and Big Data76

Useful links•

SNE Technical Report on Architecture Framework and Components for the Big
Data Ecosystem. Draft Version 0.2, 12 September 2013
http://www.uazone.org/demch/worksinprogress/sne-2013-02-techreport-bdaf-draft02.pdf•

Addressing Big Data Issues in Scientific Data Infrastructure. By Yuri Demchenko,
et al. Paper at IEEE CTS 2013 Conf., May 20-24, 2013, San Diego, USA
http://www.uazone.org/demch/papers/bddac2013-bigdata-infrastructure-v06.pdf•

BoF on Big Data Challenges for NREN’s organised at TNC2013
https://tnc2013.terena.org/core/event/15•

NIST Big Data Working Group (NBD-WG) http://bigdatawg.nist.gov/home.php– Input documents http://bigdatawg.nist.gov/show_InputDoc2.php– Big Data Reference Architecture
http://bigdatawg.nist.gov/_uploadfiles/M0226_v7_2611176301.docx– Big Data Architectures Survey
http://bigdatawg.nist.gov/_uploadfiles/M0151_v2_7447424902.docx– Requirements for 51 usecases
http://bigdatawg.nist.gov/_uploadfiles/M0224_v1_1076079077.xlsx

5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

77

NIST Documents on Cloud Computing (1)
•
•

•

•

•

[NIST CC] NIST SP 800-145, “A NIST definition of cloud computing”, [online]
Available: http://csrc.nist.gov/publications/nistpubs/800-145/SP800-145.pdf[NIST CCRA] NIST SP 500-292, Cloud Computing Reference Architecture, v1.0.
[Online] http://www.nist.gov/customcf/get_pdf.cfm?pub_id=909505[NIST Synopsis] DRAFT NIST SP 800-146, Cloud Computing Synopsis andRecommendations. [online] Available: http://csrc.nist.gov/publications/drafts/800146/Draft-NIST-SP800-146.pdf
Draft SP 800-144 Guidelines on Security and Privacy in Public Cloud Computing.
[online] Available: http://csrc.nist.gov/publications/nistpubs/800-144/SP800144.pdf
NIST SP 500-299: NIST Cloud Computing Security Reference Architecture.
[online] http://collaborate.nist.gov/twiki-cloudcomputing/pub/CloudComputing/CloudSecurity/NIST_Security_Reference_Archit
ecture_2013.05.15_v1.0.pdf

5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

78

NIST Documents on Cloud Computing (2)•••NIST SP 500-293: US Government Cloud Computing Technology Roadmap
Volume 1, High-Priority requirements to Further USG Agency Cloud Computing
Adoption. http://www.nist.gov/itl/cloud/upload/SP_500_293_volumeI-2.pdf
NIST SP 500-293: US Government Cloud Computing Technology Roadmap
Volume II, Useful Information for Cloud Adopters (Draft)
http://www.nist.gov/itl/cloud/upload/SP_500_293_volumeII.pdf
NIST SP 500-293: US Government Cloud Computing Technology Roadmap
Volume III, Technical Considerations for USG Cloud Computing Deployment
Decisions (Draft). http://collaborate.nist.gov/twiki-cloudcomputing/pub/CloudComputing/RoadmapVolumeIIIWorkingDraft/NIST_cloud_roadmap_VIII_draft_110311.pdf

5 Nov 2013, Chasopys, Kiev

Cloud and Big Data

79