Big Data Challenges
fore-Science Infrastructure

Yuri Demchenko,
SNE Group, University of Amsterdam

AAA-Study Project
23-25 November 2012, Nanjing

COINFO2012 Conference
24-25 November 2012, Nanjing, China
Big Data Science SDI Challenges

1Outline• Introduction– System and Network Engineering (SNE) group at the University of Amsterdam (UvA)• Big Data Science as the next technology development focus• European Research Area (ERA) and SDI in Europe• Data categories and Data Lifecycle in modern e-Science• General requirements to e-Infrastructure for Big Data Science• Defining SDI architecture framework– Clouds as an infrastructure platform for complex/scientific data

• Questions and Discussion

23-25 November 2012, Nanjing

Big Data Science SDI Challenges

Slide_2

Contributing projects (FP7)• AAA Study: Study on AAA Platforms For Scientific data/information
Resources in Europe https://confluence.terena.org/display/aaastudy/AAA+Study+Home+Page– Analysis of existing Research Infrastructure in Europe use cases andrequirements– Final report and recommendations to EC, MS and NREN’s published• GEYSERS – Generalised Architecture for Infrastructure services http://www.geysers.eu/• GEANT3 JRA3 Task 3 – Composable services (GEMBus) http://www.geant.net/

23-25 November 2012, Nanjing

Big Data Science SDI Challenges

3SNE Cloud Architecture Research – Moving to
Big Data
Cloud/InterCloud oriented research• Intercloud Architecture for Interoperability and Integration, Release 1,
Draft Version 0.4. SNE Technical Report 2012-03-02, 19 June 2012
http://staff.science.uva.nl/~demch/worksinprogress/sne2012-techreport-12-05intercloud-architecture-draft04.pdf(1) Generic Cloud IaaS Architecture, Release 1, 15 April 2011
Published as http://staff.science.uva.nl/~demch/worksinprogress/sne2011-techreport-2011-03clouds-iaas-architecture-release1.pdf(2) InterCloud OS/Middleware (low level Intercloud integration)• Security Infrastructure for Cloud (dynamically provisioned)• Contributing to cloud standardisation by OGF, NIST, IEEE, IETF
• Implementation – EU projects GEYSERS, GEANT3, COMMIT– Telco and NREN driven – core network and last mile

Challenging infrastructure issues in Big Data technologies• Distributed data support, high-performance optical networks,
trustworthy ICTInterCloud Architecture 2012

InterCloud Architecture Framework

4Big Data Science as the next technology focus• Big Data is becoming the next buzz word
• Based on the e-Science concept and entire information and artifacts
digitising– Requires also new information and semantic models for information
structuring and presentation– Requires new research methods using large data sets and data mining• Methods to evolve and results to be improved• Changes the way how the modern research is done (in e-Science)– Secondary research, data re-focusing, linking data and publications• Big Data require infrastructure to support both distributed data(collection, storage, processing) and metadata/discovery services– Demand for trusted/trustworthy infrastructure– Clouds provide just right technology for infrastructure virtualisation to support
different data set

23-25 November 2012, Nanjing

Big Data Science SDI Challenges

5

EC Strategy and Policy on STI (Scientific andTechnology Information)• Covers 2 main areas– Scientific Information policies defining priorities for STI in Europe andharmonise national policies– e-Infrastructure for scientific information• Management of scientific data during their whole lifecycle: includes creation,access, (re)use, and preservation• Main targets– New technologies for research and scientific data use– Better access to support information exchange and cooperation– Preservation of data for future reuse and secondary research• EC initiative on Open Access scientific publications from publicly
funded projects• G8+O5 Global Research Data Infrastructure, Subgroup on Data,
Draft Report, 28 October 2011
– Requires “reliable infrastructures for persistent identification of data (e.g.
digital object identifiers, handle systems), researchers (e.g. digital
authors identifiers), and authentication, authorization and accounting
systems (AAA)”
23-25 November 2012, Nanjing

Big Data Science SDI Challenges

6Horizon2020 Consultation Meeting(Rome 11-12 April 2012)•

Vice-President of the European Commission Mme Neelie Kroes– "Open e-Infrastructures for Open Science" - The Digital Agenda and Access to Scientific
Information•

Working Group 1: Open Global Data Infrastructure: towards an internationalframework for collaborative scientific data infrastructure– Several contributing "position papers" produced about the subject of a "Data Web Forum" ora "Data Access and Interoperability Task Force" and reports such as “Riding the Wave” fromthe High-Level Group on Scientific Data or the report of the G8+5 working group on Data.

••

Working Group 2: Open Scientific Content: e-Infrastructure policies and services
to support access, storage, processing and exchange of scientific information
Working Group 3: Open Research Culture: Engagement of researchers andsociety with Open Science through collaborative data infrastructure.

Declaration from Rome meeting“Researchers and practitioners from any discipline are able to find, access and process the data
they need in a timely manner. They are confident in their ability to use and understand data, andthey can evaluate the degree to which that data can be trusted.
Data are stored, managed, shared, and preserved in a way that optimizes scientific discovery,
innovation, and societal benefit. Where appropriate, producers of data benefit from opening it tobroad access and routinely deposit their data in reliable repositories. A framework of repositories
work to international standards, to ensure they are trustworthy.”
23-25 November 2012, Nanjing

Big Data Science SDI Challenges

7

Open Access to Scientific Publications• EC initiative on Open Access scientific publications frompublicly funded projects– Included into Declaration from Rome meeting– Approx 3500 publicly funded ROs and 2000 privately funded ROs– Special funding scheme for reimbursing publications– Issues with China, India, Russia compliance to OA principles• Consultation at high governmental level• OpenAIRE project exploring models for open access topublications– PID (Persistent ID for data), ORCHID (Open Researcher ID), Linked
data• Community initiative - Panton Principles for Open Data in
Science (http://pantonprinciples.org/)
23-25 November 2012, Nanjing

Big Data Science SDI Challenges

8

Coordination in European Research Area (ERA)• European Commission – but not only
• EIROforum – European Intergovernmental Research Organisation– Profile committees organised by scientific domain• ESFRI – European Strategy Forum for Research Infrastructure– Coordinates projects and funding for Research Infrastructures (RI) (20022010)• eIRG – e-Infrastructure Reflection Group– High level policy development for Europe on e-Infrastructure• EEF - European e-Infrastructure Forum– Principles and practices to create synergies for distributed Infrastructures
CERN - High Energy Physics and LHC experiment•• TERENA– REFEDS – Research and Education Federations• LIBER – Association of European libraries– Growing role of scientific libraries including access to research information

23-25 November 2012, Nanjing

Big Data Science SDI Challenges

9Big Data Science and European Research Areas (1)• High Energy Physics (HEP)– Running experiment on LHC and infrastructure WLCG (Worldwide LHC
Grid)• Already producing PBytes of information• Worldwide distribution and processing– CERN and national HEP centers• Low Energy Physics and Material Science (photon, proton, laser,
spectrometry)– Number of research facilities serving international communities– Multiple short projects producing TBytes of information• Experimental data storage, identification, trusted access to multiple users(including public and private researchers)• Earth, weather and space observation– Climate research and Earth observation• With new 4? satellites to be launched starting 2017 to produce PBytes monthly– ESA (European Space Agency)

23-25 November 2012, Nanjing

Big Data Science SDI Challenges

10Big Data Science and European Research Areas (2)• Life science and biodiversity (Genomic, Biomedical and
Healthcare research)– Human genome (EMBL-EBI)• Currently centralised databases but evolving to distributed• ELSI data - Special requirements to data integrity and privacy– Living species and biodiversity• Mobile/field access, filtering and on-demand computing• Public contribution, vocational or citizen researchers– Numerous local/offline databases to be brought online– Projects: ELIXIR, EGA, LifeWatch• Humanities (History, languages, human behaviour)– Rediscovering research with total information digitising• Expected huge amount of data to digitise all human heritage– Very spread research community– Projects: CLARIN, DARIAH, EUDAT• Additional: Data collection from sensors and online activities– Intelligence, Homeland Security, Log data (+ Facebook data :-)23-25 November 2012, Nanjing

Big Data Science SDI Challenges

11

Existing and emerging Europe wide SDI• WLCG – Worldwide LHC Grid (CERN, Geneva)• EGI – European Grid Infrastructure (successor of the EGEE project)– Operational Grid infrastructure serving around 10,000 researches worldwide– Published “Seeking new horizons: EGI’s role for 2020”– Federated Cloud Infrastructure (initiative) provides an infrastructure platform for
operational and legacy Grid services• PRACE – Partnership for Advanced Computing in Europe• HELIX Nebula – The Science Cloud (prospective cloud based SDI)
– Private partnership project with wide industry participation but limited EC
support• Growing Research Infrastructures for different researchcommunities– CLARIN, EUDAT, LifeWatch, ELIXIR, etc.
• Less technology and more subject focused

23-25 November 2012, Nanjing

Big Data Science SDI Challenges

12Big Data Challenges and Initiatives• Peta and Exa scale problems– Storage, Computing, Transfer/Network– International Exascale Software Project (http://www.exascale.org/)– A Vision for Global Research Data Infrastructure(http://www.grdi2020.eu/)• G8+O5 Global Research Data Infrastructure, Subgroup on Data, Draft
Report, 28 October 2011

• International Initiative “Research Data Alliance”
http://www.rd-alliance.org/– To accelerate international data-driven innovation and discovery by facilitating
research data sharing and exchange, use and re-use, standards
harmonization, and discoverability.

– Consolidates previous initiatives• Data Web Forum (DWF) initiative by NSF• DAITF – Data Access and Interoperability Task Force initiated by EUDAT
project
23-25 November 2012, Nanjing

Big Data Science SDI Challenges

13E-Science Features
•

•

•

•
•

•

Automation of all e-Science processes including data collection, storing,classification, indexing and other components of the general data curation andprovenanceTransformation of all processes, events and products into digital form bymeans of multi-dimensional multi-faceted measurements, monitoring andcontrol; digitising existing artifacts and other content
Possibility to re-use the initial and published research data with possible data
re-purposing for secondary research
Global data availability and access over the network for cooperative group of
researchers, including wide public access to scientific data
Existence of necessary infrastructure components and management tools thatallows fast infrastructures and services composition, adaptation andprovisioning on demand for specific research projects and tasks
Advanced security and access control technologies that ensure secure
operation of the complex research infrastructures and scientific instruments andallow creating trusted secure environment for cooperating groups andindividual researchers.

23-25 November 2012, Nanjing

Big Data Science SDI Challenges

14

Scientific Data Types• Raw data collected from observationand from experiment (according to an
initial research model)
Publicationsand Linked
Data

Published Data

Structured Data
Raw Data

23-25 November 2012, Nanjing• Structured data and datasets that went
through data filtering and processing(supporting some particular formal
model)• Published data that supports one or
another scientific hypothesis, research
result or statement• Data linked to publications to support
the wide research consolidation,
integration, and openness.

Big Data Science SDI Challenges

15

Scientific Data Types
EC Open Access Initiative
Requires data linking at alllevels and stages

Publicationsand Linked
Data

Published Data

Structured Data
Raw Data

23-25 November 2012, Nanjing• Raw data collected from observationand from experiment (according to an
initial research model)• Structured data and datasets that went
through data filtering and processing(supporting some particular formal
model)• Published data that supports one or
another scientific hypothesis, research
result or statement• Data linked to publications to support
the wide research consolidation,
integration, and openness.

Big Data Science SDI Challenges

16

Traditional Data Lifecycle Model
Traditional Data Lifecycle Model
Project/
Experiment
Planning

Data collection

Data
Integration andprocessing

Publishingresearch
results

Discussion/
feedback

Archivingor
Discarding

User

•
•
•
•
•

Data collection
Data processing
Publishing research results
Discussion
Data and publications
archiving
Lack of initial data preservationand data linking to publications
23-25 November 2012, Nanjing

Big Data Science SDI Challenges

17Data Lifecycle Model in e-Science - II
Data Lifecycle Model in e-Science

User
Researcher

Data discovery

Data Curation(including retirement and clean up)
Data
recycling

Raw Data
Experimental
Data

Project/
Experiment
Planning

Data collection
andfiltering

Structured
Scientific
Data

Data analysis

DB

Data archiving

Data
Re-purpose

Data linkage
to papers

Data sharing/
Data publishing

Data Re-purpose

Data Linkage Issues•
Persistent Identifiers (PID)•
ORCID (Open Researcher andContributor ID)•
Lined Data

23-25 November 2012, Nanjing

End of project

Open
PublicUse

Data Clean up and Retirement•
Ownership and authority•
Data Detainment

Big Data Science SDI Challenges

Dataarchiving

Data Links

Metadata &
Mngnt

18

General requirements to SDI for emerging Big
Data Science
•
•
•
•

•
•
•
•

Support for long running experiments and large data volumes generated
at high speed
Multi-tier inter-linked data distribution and replication
On-demand infrastructure provisioning to support data sets andscientific workflows, mobility of data-centric scientific applications
Support of virtual scientists communities, addressing dynamic usergroups creation and management, federated identity management
Support for the whole data lifecycle including metadata and data source
linkage
Trusted environment for data storage and processing
Support for data integrity, confidentiality, accountability
Policy binding to data to protect privacy, confidentiality and IPR

23-25 November 2012, Nanjing

Big Data Science SDI Challenges

19

Defining Architecture framework for SDI and ACAI
• Scientific Data Lifecycle Management (SDLM) model• e-SDI multi-layer architecture model• RORA model to define relationship between resources andactors– RORA (Resource-Ownership-Role-Actor) model defines relationshipbetween resources, owners, managers, users– Initially defined for telecom domain– Potentially new actor in SDI – Subject of data (e.g. patient, orscientific object/paper)• Security and Access Control and Accounting Infrastructure(ACAI)– Authentication, Authorisation, Accounting• Supported by logging service– Extended to support data access control and operations on data
– Trust management infrastructure23-25 November 2012, Nanjing

Big Data Science SDI Challenges

20

SDI Architecture Model

Layer B6
Scientific
Applications

ScientificDatasetApplicUser
portals

Scientific
ApplicScientific
ApplicScientificUser/Scientific Applications Layer

Metadata and Lifecycle Management

Security and AAI

Operation Support and Management Service (OSMS)

Layers

Policy and Collaborative Groups Support

Layer B5
Federation Layer(Policy and User
Group Support)

Shared Scientific Platform and Instruments(specific for scientific area, also Grid based)

Cloud/Grid Infrastructure
Virtualisation and Management
Middleware
ComputeResources

Middleware
security

StorageResources

Network infrastructure23-25 November 2012, Nanjing

Big Data Science SDI Challenges

Layer B4
Scientific Platform
and Instruments
Layer B3
Infrastructure
Virtualisation

Layer B2
Datacenter andComputer Facility
Layer B1
Network Infrastructure

Technologies andsolutions
Scientific specialist
applications
Library resources

eduGAIN, RFEDS,
VOMS, InCommon

PRACE/DEISA

Grid/Cloud

CloudsAutobahn, eduroam

21

SDI Architecture Layers• Layer D1: Network infrastructure layer represented by the general
purpose Internet infrastructure and dedicated network infrastructure• Layer D2: Datacenters and computing resources/facilities• Layer D3: Infrastructure virtualisation layer that is represented by the
Cloud/Grid infrastructure services and middleware supporting specialised
scientific platforms deployment and operation• Layer D4: (Shared) Scientific platforms and instruments specific fordifferent research areas• Layer D5: Federation and Policy layer that includes federation
infrastructure components, including policy and collaborative user groups
support functionality• Layer D6: Scientific applications and user portals/clients

23-25 November 2012, Nanjing

Big Data Science SDI Challenges

22

SDI move to Clouds• Cloud technologies allow for infrastructure
virtualisation and its profiling for specific data
structures or to support specific scientific workflows• Clouds provide just right technology for infrastructure
virtualisation to support data sets
• Complex distributed data require infrastructure• Cloud can provide infrastructure on-demand toscientific workflows– Similar to Grid but with benefits of the full infrastructure
provisioning on-demand23-25 November 2012, Nanjing

Big Data Science SDI Challenges

23General use case for infrastructure provisioning:
Workflow => Logical (Cloud) InfrastructureEnterprise/Scientific workflow
Storage
Data

SpecialProc 1
Data
Filtering

Input
Data

VisualPresent
SpecialProc 2

Instrum.
Data

Campus A

Data
Archive

Visualisation

Visualisation

CE

User
Group A

Campus B

CE

User
User
User

User
User
User
VR6

Cloud 2 PaaS

VR2

User
Group B

VR7
VR4

VR1

VR5

Resource/
Service
ProviderVR3Enterprise/Project basedIntercloud Infrastructure

Cloud 1 IaaS

Resource/
Service
Provider
CN

CN
CN

CN
CN

CN

Cloud PaaS Provider

CN
CN

CN

CN
CN

Cloud IaaS Provider

23-25 November 2012, Nanjing

Big Data Science SDI Challenges

24General use case for infrastructure provisioning:
Workflow => Logical (Cloud) InfrastructureEnterprise/Scientific workflow
Storage
Data

SpecialProc 1
Data
Filtering

Input
Data

VisualPresent
SpecialProc 2

Instrum.
Data

Campus A

Data
Archive

Visualisation

Visualisation

CE

User
Group A

Campus B

CE

User
User
User

User
User
User
VR6

Cloud 2 PaaS

VR2

User
Group B

VR7
VR4

VR1

VR5

Resource/
Service
ProviderVR3Enterprise/Project basedIntercloud Infrastructure

Cloud 1 IaaS

Resource/
Service
Provider
CN

CN
CN

CN
CN

CN

Cloud PaaS Provider

CN
CN

CN

CN
CN

Cloud IaaS Provider

23-25 November 2012, Nanjing

Big Data Science SDI Challenges

25General use case for infrastructure provisioning:
Logical Infrastructure => Network Infrastructure (1)Resource and Cloud Provider Domains
Cloud 1 IaaS
VR3

VR1

Cloud 2 PaaS

VR5

VR7

Campus A
Infrastructure

Campus B
Infrastructure

VR2

Campus A

VR4

VR6

Cloud Carrier Network Infrastructure

Visualisation

Visualisation

CE

User
Group A

Campus B

CE

User
User
User

User
User
User
VR6

Cloud 2 PaaS

VR2

VR7
VR4

VR1

VR5

Resource/
Service
ProviderDefined as
InterCloud Architecture
Framework (ICAF)

UserGroup B

VR3

Enterprise/Project basedIntercloud Infrastructure

Cloud 1 IaaS

Resource/
Service
Provider
CN

CN
CN

CN
CN

CN

Cloud PaaS Provider

CN
CN

CN

CN
CN

Cloud IaaS Provider

23-25 November 2012, Nanjing

Big Data Science SDI Challenges

26InterCloud Architecture Framework (ICAF)
Components• Multi-layer Cloud Services Model (CSM)– Combines IaaS, PaaS, SaaS into multi-layer model with inter-layer interfaces– Including interfaces between cloud service layers and virtualisation platform• InterCloud Control and Management Plane (ICCMP)– Allows signaling, monitoring, dynamic configuration and synchronisation of
the distributed heterogeneous clouds– Including management interface from applications to network infrastructure
and virtualisation platform• InterCloud Federation Framework (ICFF)– Defines set of protocols and mechanisms to ensure heterogeneous clouds
integration at service and business level– Addresses Identity Federation, federated network access, etc.

• InterCloud Operations Framework (ICOF)– RORA model: Resource, Ownership, Role, Action– Business processes support, cloud broker and federation operation
Intercloud Architecture for Interoperability and Integration, Release 1, Draft Version 0.5. SNETechnical Report 2012-03-02, 6 September 2012
http://staff.science.uva.nl/~demch/worksinprogress/sne2012-techreport-12-05-intercloud-architecture-draft05.pdf

23-25 November 2012, Nanjing

Big Data Science SDI Challenges

27

Need for new Scientific and Academic discipline• New Scientific and Academic Discipline is needed for Big
Data Science– Included into declaration of the Rome EC Horizon2020 Consultation
meeting (3-4 April 2012)• To address– ICT infrastructure building, operation and optimisation– Big data management and distributed computing– Metadata and semantics– Security and trustworthiness of Infrastructure and Data

• Need to educate/train new specialists in Big Data
– Primarily for not ICT savvy community

23-25 November 2012, Nanjing

Big Data Science SDI Challenges

28Questions and Discussion

Thank you for your attention.

23-25 November 2012, Nanjing

Big Data Science SDI Challenges

29