Data Integration andAnalysis System© GEO Secretariat

Present DIAS &Future DIAS

© GEO Secretariat

DIAS:power of big data
and Future DIAS
Masaru Kitsuregawa
Professor, Institute of Industrial Science, The Univ. of Tokyo
Director of Earth Observation Data Integration & Fusion Research Initiative(EDITORIA),The Univ. of Tokyo
Director General of National Institute of Informatics(NII)DIAS:power of big data
and Future DIAS
Masaru Kitsuregawa
Professor, Institute of Industrial Science, The Univ. of Tokyo
Director of Earth Observation Data Integration & Fusion Research Initiative(EDITORIA),The Univ. of Tokyo
Director General of National Institute of Informatics(NII)EDITORIA(Earth Observation Data Integration & Fusion Research Initiative)Global Earth Observation System of Systems（GEOSS））

Land Observation

Ocean Observation

Satellite Imagery

WCRP CMIP3

Simulations

Data Integration & Information Fusion PlatformApp. Layer
User Apps.

User Apps.

User Apps.

User Apps.

User Apps.

Common Utility Layer•Visualizer(w display wall)•Discovery Work Flow Assist•Data Quality Manager•Data Transformer•Data Crawler•ETL•Data Mingrator•Data Navigator•Meta Data Manger

Data Management Layer •Database management system•Storage management
File System Layer •PB scale logical file
•Power management
Storage Layer
Disk Arrays

MetBroker

4V for Bigdata
VolumeVariety
Velocity
VeracityVolumeBig Data in Earth Env.
ResearchDIAS Systemdisk + tape > 20PB

Data Integration and Analysis System Project 2011-2015(DIAS-P)
Kitami Institute ofTechnology

Hokkaido Univ.

1Gbps

1Gbps×2
HINES

DIAS/GRENE
Network

SINET
Core node
Sapporo•40GbpsSINETCore node
Tokyo
UTNET
10GbpsInstitute of Industrial Science
The University of Tokyo

40Gbps
H24 revised budget•10PB Storageﾞ

•Electricity restriction at the Universityof Tokyo in the summer of 2011 causedby Tohoku Region Pacific Coast
Earthquake→Degraded operation of DIAS
Overcoming of weakness ofconcentrated operation at Univ. ofTokyo

Placing parts of nodes with DIAS functions.
•Server for data retrieve, download, Operating them together with DIAS at
simple analysis•Servers for numerical computation IIS, Univ. of Tokyo

National Institute of Informatics
Chiba Annex

VarietySo many kinds of Data on DIAS

Data Integration and Analysis Systema legacy for Japan's contributions to GEOSS

enriching data searching capability

Velocity(stock and stream)Global Data to Local Information
General Circulation Model

Data
Assimilation

Data
Assimilation

Improved prediction
Satellite data
Regional/Meso Model

Improved
InitialCondition

Proactive
Control of DAM

In-situ data
Centralized Data System

Flood Peak Reduction
3000

Optimized rules
2500

Outflow eq. inflow

3

discharge [m /s]

Outflow eq. 0
2000
1500
1000
500

DHM

0

7/8.1z7/9.1z7/10.1z

7/11.1z

2002

7/12.1z

Socio-Economic Data

Improved
Prediction

Upper Tone River Basin
Yagisawa
Naramata
Fujiwara
AimataShimagawa
Sonohara
Yamba

Flood reduction by Proactive Control of Dam
Discharge with GPV 13~18
900

570

Sonohara dam

560
550

700

540

3

800

Iwamoto gauge
3000

Optimized release
rules
Optimized

Flood peak
reduction

2000

500
400
300

520
510
500
490

100

480
470
7/9.1z7/10.1z

7/11.1z

7/12.1z

2002

1500

Water level increase due to storage

1000
500

7/10.1z

7/11.1z

7/12.1z3

7/9.1zdischarge [m /s]

0

7/8.1z530200
07/8.1z

Outflow eq. inflow

3

discharge [m /s]

Outflow eq. 0

Sim outflow
Sim inflow
Sim water levelObs water level

2002

Peak created due towater release from dams

1000
900
800
700
600
500
400
300
200
100
0

Fujiwara dam

7/8.1z

Sim outflow
Sim inflow
Sim water levelObs water level

7/9.1z7/10.1z

7/11.1z
2002

7/12.1z

650
645
640
635
630
625
620
615
610
605
600

waterlevel [m]

2500

600

waterlevel [m]1000discharge [m /s]

Water is stored until
max capacity is reachedDIAS Ensemble Flood Prediction
T12, 2011, Aimata

T12, 2011, Maebashi

T15, 2011, Murakami

T15, 2011, Maebashi

legendObserved
Ensemble Prediction

DIAS Ensemble Flood Prediction
Data Integration and Analysis System (DIAS)
Meteorological Data
Temp.

wind

rain

sunshine

Pre-processing
Model input data

River, Dam

Radar, NWP
Radar(C-band)NWP Output(GPV)

Runoff Water Level
Dam: WL, inflow, release

Real Time DataManagement System
Prediction System
Pre-processing
Radar, GPV

Ensemble Rainfall Prediction
RainfallPattern 12
RainfallRainfallPattern
Pattern N

WEB-DHM
Ensemble Flood Prediction
River Runoff
Soil
Moisture

Statistical
Analysis
Error
Estimation

Flood
Pattern 12
Flood
FloodPattern
Pattern N

Software Architecture
Real time Control

Dam Operation Simulator(Optimal)Dam Operation Simulator(Human Operation)

Veracity

DIAS METADATA
IMPLEMENTATIONISO 19115: Geographic information - Metadata
ISO 19115 and its parts defines how to describe
geographical information and associated services.
The objective of this International Standard is toprovide a clear procedure for the description ofdigital geographic datasets.
ISO 19139 provides the XML implementation
schema for ISO 19115 specifying the metadata
record format and may be used to describe,
validate, and exchange geospatial metadata
prepared in XML.

ISO19115, ISO19139

ISO19115 UML

ISO19139 XML

Collaborative development of DIAS
metadata implementation
Collaborative development activities with data providers (JAMSTEC) and system developers (Universities)
InitialSampleData
A1 – A6

Working Group
Data providers

Metadata team

Metadata
Registration
Tool

Application

Original documents
Clarifying items

Metadata standard

Data access matrices

Keyword lists

Interface requirement

Search & browse

We analyzed data providers' holding dataset documentation.
We collected metadata in the dataset documentation.
We mapped between ISO 19115 metadata elements and the dataset
documentation items.
We selected useful keyword lists in order to search effectively.

DIAS Metadata & Document Creation System
PDF
HTML

One source –multi purpose
Metadata Manual
Input

Print &
PublishDocuments

Data provider

User
ISO19139 XML

API

Automatic
Metadata extraction

XML

DIAS MetadataInterfacepublishSearchdatasetEnglish &Japanese
GCMD Keywords’
Selection Menu

Metadata

Search & Data
Discovery tools

DIAS Metadata & Document Creation System

DIAS Metadata & Document Creation System•Title•Title•Contact•Contact•Document•Document Author
Author•Dataset•Dataset Creator
Creator•Date•Date ofof Dataset
Dataset•Category•Category•Abstract•Abstract•Temporal•Temporal Extent
Extent•Geometric•Geometric Region
Region•Grid•Grid•Geographic•Geographic Description
Description•Keyword•Keyword•GCMD•GCMD Science
Science•GCMD•GCMD Platform
Platform•AGU•AGU•GEOSS•GEOSS•GEO_COP•GEO_COP•Country•Country etc.
etc.
•Online•Online Resource
Resource•Environmental•Environmental Info.
Info.
•Distribution•Distribution Info.
Info.
•Data
•Data Process
Process•Data•Data Quality
Quality•Data•Data Policy
Policy by
by Data
Data
ProviderProvider•Acknowledgement•Acknowledgement•References•References

Dataset Metadata Edit
Input items have been tabbed.
You can confirm the related projects‘ metadata
Please select an item while switching. using this button.

There are two boxes side by side when you need to
input the field in both English and Japanese.

Imported vocabulary resources•

We have imported the following resources in DIAS Interoperability portal. Properly, we
get permissions for our fair use from original resource providers.
No.

Name

Language

Data format

1

WMO Glossary

EnglishWeb pages

CEOS Missions, Instruments andMeasurements(MIM) Database

EnglishMS ExcelCEOS Systems Engineering Office(SEO)

EnglishMS ExcelGEMET(GEneral Multilingual Environmental Thesaurus)

EnglishRDF

INSPIRE((Infrastructure for Spatial Information in
the European Community) Feature Concept
Dictionary

EnglishWeb pages

NASA SWEETEnglishOWLCUAHSI(Consortium of Universities for the Advancementof Hydrologic Science)

EnglishOWLJAXA glossary

JapaneseWeb pages

Rremote sensing glossary for Japanese RS
academic society

JapaneseBooks (paper media)

GIS glossary for Japanese GIS academic society

Japanese

MS Word

2
3
4
5

6
7

8
9
10

Data Index/Search (basic keyword search)• Users can search DIAS dataset by using basic keyword search function.
①Select “Data Index/Search” tab②Input keywords(ex.”flood” or③Push “search”“AWCI”)④Visualizing dataset related to the input keyword⑤Similarity scores

Technical Term Search• Users can check the definition of terms by using ‘technical term search’ function.
①Select “Technical Term Search” tab②Select targets in list of ontologies,
terminologies, glossaries and dictionaries.

⑤Visualizing terms in each dictionaries
related to the input keyword④Push “search”③Input keywords(ex.”basin”)

3-stage Pipeline: Data upload,
Data Quality Control andMetadata registration

Noise cannot be Zerobut minimized!
We plan to introduce data linage.

Upon 4V’s criteria
DIAS is truly well-oganized‘Big Data’

How about Big Data in otherdomain?

What happens on the Cyber space
every sixty seconds?

WM/04.02 S. 40© andreas.dengel@dfki.de - 2013 - p. 40
WM/04-05 S. 40

WM/04.02 S. 41© andreas.dengel@dfki.de - 2013 - p. 41
WM/04-05 S. 41

There might be earth monitoring data
being pushed onto Cyberspace.
But I have never heard how much!!

We have to orchestrate various activities

Automatic and Active
Monitoring and Injection

Recent astronomy: Thinking telescope

Style of Data Intensive Science is evolving!

Conclusion• Extreme IT for Data Automation• More time to think and design new social
solution.
• Once you have an idea on societal innovative
service idea, we are more than happy to‘Realize’ with our IT power for you.

Nothing Is impossible for us.

inter-disciplinarity & trans-disciplinarity

Scientific Knowledge
Formal Knowledge which can be transferred and shared
among vast amount of Factual Knowledge
Deductive Inference

Hypothesis

ModelsExperiments

Observations

Inductive Inference

Factual Knowledge

Natural Worldtrans-disciplinarity

Explosive Increase of Scientific Knowledge
 Differentiation & Systematization: disciplines

Socio-economic World
Recognition World•Accumulated sub-system knowledge can not be reflected to holistic knowledge.
•Effects of a whole system can not be involved in a targeted sub-system.

 Far from fundamental solutions of issues across disciplines  inter-disciplinarity

Data System

Integration-Interlinkage System

inter-disciplinarity & trans-disciplinarity

Scientific Knowledge
Formal Knowledge which can be transferred and shared
among vast amount of Factual Knowledge
Deductive Inference

Hypothesis

ModelsExperiments

Observations

Inductive Inference

Factual Knowledge

Natural Worldtrans-disciplinarity

Explosive Increase of Scientific Knowledge
 Differentiation & Systematization

Socio-economic World
Recognition World•Accumulated sub-system knowledge can not be reflected to a whole system.
•Effects of a whole system can not be involved in a targeted sub-system.

 Far from fundamental solutions of issues across disciplines  inter-disciplinarity

Data System

Integration-Interlinkage Systemเกษตรก
รรม
D
B

Where is data?
How to access?

Ecosystem
Meteorology

农业

DB

DB

35 0
30 0
25 0
20 0

A1:高度成長社会
A2：多元化型
B1:持続発展型
B2：地域共存型

15 0
10 0
50
0

2 04
6
0
8
4
8 00
14 16
92
19 199 1996 199 20 200 20 200 2008 201 2012 20 20 201 2020

DB

DB

농업
Agriculture

DB

Quality?
Reliability?

DB
35 0
30 0
25 0
20 0

A1:高度成長社会
A2：多元化型
B1:持続発展型
B2：地域共存型

15 0
10 0
50
0

4
8 00
2 04
6
0
8
92
14 16
19 199 1996 199 20 200 20 200 2008 201 2012 20 20 201 2020

Hydrology

DB
350
300
250
200

A1: 高度 成長 社会
A2： 多元 化型
B1: 持続 発展 型
B2： 地域 共存 型

150

Climatology

Technical Terms
among
Different
Disciplines

100
50
0

98
02 004 006 008 010 012
14
92 994 996
00
19
1
1
19
20
20
2
2
2
2
2
20

6
8
0
201 201 202

DBLand Use
Health

DB
DB

DB

48

Fully interactive (biology,
chemistry, physics)
ensemble simulations inan operational mode

Computational Modeling in Two Stages;
Driving Evolution & Enabling Revolution
10

10

Data Size, Bytes

10

10

10

10

.25°x .25° resolution;
add cloud, chemistry,& radiation effects

16.25°x .25° resolution;
hurricanes, storm fronts

15

2020

2010

2005

14

2°x 2° resolution;
synoptic scales
13

Interactive
physics, biology,
chemistry;
assimilation ofsatellite data

Nanotechnology

Revolutionary

12

Homogeneous
Cluster

11

2001
Heterogeneous
Cluster

Information
Power Grid

Evolutionary

10 10

TeraFLOPS

GigaFLOPS
10

2015

ExaFLOPS

PetaFLOPS

9
9
10

10
10

11
10

10

12

13
10

10

14

Compute Speed, FLOPSReal Demonstrated Performance doing useful Science

15
10

16
10

17
10

18
10

49

Data Integration and Analysis Systema legacy for Japan's contributions to GEOSS

To create knowledge enabling us to solve
the Earth environment problems and to
generate socio-economic benefits,Data Integration and Analysis Systema legacy for Japan's contributions to GEOSS

tackling a large increase in diversity of the
Earth observation data.
Ontology

Meta data Design

Data Integration and Analysis Systema legacy for Japan's contributions to GEOSS

tackling a large increase in volume of the
Earth observation data.
IPCC AR4 (2007): 40TB

IPCC AR5 (2012): 2.6PB

2.7PB (2012)
600TB (2007)

Data Integration and Analysis Systema legacy for Japan's contributions to GEOSS

accelerating data archiving, including dataloading, QC and metadata registration

Data Integration and Analysis Systema legacy for Japan's contributions to GEOSS

accelerating data archiving, including dataloading, QC and metadata registration

Data Integration and Analysis Systema legacy for Japan's contributions to GEOSS

accelerating data archiving, including dataloading, QC and metadata registration

18 River Basin, 280stations
completed
Asia Monsoon Year
24 project, 277 stations

Climate Data
16 River Basin, 202 stations

Data Integration and Analysis Systema legacy for Japan's contributions to GEOSS

enriching data searching capability

Data Integration and Analysis Systema legacy for Japan's contributions to GEOSS

enabling us to do integrated research andto realize inter-disciplinarity

Water

Food

ClimateFishery

Biodiversityinter-disciplinarity & trans-disciplinarity

Scientific Knowledge
Formal Knowledge which can be transferred and shared
among vast amount of Factual Knowledge
Deductive Inference

Hypothesis

ModelsExperiments

Observations

Inductive Inference

Factual Knowledge

Natural Worldtrans-disciplinarity

Explosive Increase of Scientific Knowledge
 Differentiation & Systematization

Socio-economic World
Recognition World•Accumulated sub-system knowledge can not be reflected to a whole system.
•Effects of a whole system can not be involved in a targeted sub-system.

 Far from fundamental solutions of issues across disciplines  inter-disciplinarity

Data System

Integration-Interlinkage System

Sharing Data and Information
Exchanging Knowledge, Experiences and Ideas
Working Together
WorkbenchRiver
stakestakeNPO
holderManageholderr

Hydrology
Climatology
Engineering
RS
GIS

PolicystakeMaker
holderinter-disciplinarity

Climate Change
Flood, Drought
Water Quality
Food Security
Biodiversity
EnergyissuesAgriculture
EcologystakePublic
holderIT

Farmerstakesholdertrans-disciplinarity

Demonstration River Basins

Observation Metadata Registration
Complete
Close to Completion

Document Metadata Registration
Complete

Quality Controlling
Complete

Data Uploading
Complete
Bangladesh

BhutanCambodia

India

Indonesia

Japan

Korea

Lao PDR

Malaysia

Mongolia

Myanmar

Nepal

Pakistan

Philippines

Sri Lanka Thailand

Uzbekistan

VietnamWater-Climate-Agriculture Workbench in Cambodia

Stakeholder Meeting

Real-time Rain Gauge  Satellite Data Correction
 Wide Data Dissemination
Famers’ Needs & Experiences

Nation-wide Daily Soil
Moisture from Satellite
Holistic View of Water-Climate-Agriculture Problems

Local Information

Climate Change Analysis Tools

OJT for Local Practitioners

Water Cycle-Rice Production
Coupled Model

Rice Production Monitoring

Data Integration andAnalysis Systema legacy for Japan's contributions to GEOSS
by Promoting Data Sharingand Effective Use© GEO Secretariat