Meteorological DB and XML DB

Toshiyuki Amagasa
Computational Intelligence Group
Computational Informatics Division
amagasa@cs.tsukuba.ac.jp

1

External Review (CCS)

2007/10/31

The GPV/JMA Archive
http://gpvjma ccs hpcc jp
http://gpvjma.ccs.hpcc.jp

3

External Review (CCS)

2007/10/31

GPV data by JMA` Global Spectral Model (GSM)` Regional Spectral Model (RSM)` MesoMeso-Scale
Scale nonnon-hydrostatichydrostatic Model (MSM)

4External Review (CCS)

2007/10/31

GPV data by JMA (cont(cont’d)d)` Weekly ensemble` Monthly ensemble` Seasonal ensemble

5External Review (CCS)

2007/10/31

Meteorological DBs
Japan• Kitsuregawa, Toyoda lab., The University of Tokyopyjp• http://www.tkl.iis.u-tokyo.ac.jp:8080/GPV/• Center for Climate System Research, The University of Tokyo• http://www.ccsr.u-tokyo.ac.jp/• Disaster Prevention Research Institute, Kyoto University
htt //
d ik tj / b j/
• http://www.dpri.kyoto-u.ac.jp/web_j/

Other countries• National Centers for Environmental Prediction• http://www.cdc.noaa.gov/cdc/reanalysis/• European Centre for Medium-Range Weather Forecasts• http://data.ecmwf.int/data/• National Oceanic and Atmospheric Administration• http://www.ngdc.noaa.gov/stp/SOLAR/solar.html• CISL Research Data Archive• http://dss.ucar.edu/
6External Review (CCS)

2007/10/31

Background` e-Science`

Applying ICT technologies for accelerating` Exchanging scientific resources` SharingS
computational power` Human interaction` Objective`

Constructing e-Science infrastructure for meteorology` Web service system for supporting domain experts

DB query
Data acquisition
Format conversion
Rendering for visual analysis…
7

External Review (CCS)

2007/10/31

System overview

WS: KML Generation

Workflow Engine

WS: Rendering

WS: GPV Search
GPV data store
GPVGPV8

Metadata store
Metadata

GPV/JMA Archive

WS: Cropping

WS: Format Conversion

MetadataData
CControll
External Review (CCS)

2007/10/31

Metadata design for GPV/JMA data
` Using XML

Compatibility
Cibili withi h otherh relatedl d fformats` Query processing by XPath / XQuery`` Design policy`

Inherent information
Generated timestamp` Data generator` Types of physical attributes (pressure, temperature, wind vector, …)` Grid information (resolution, number, width, …)` Surface information (constant-pressure, …）` Projective method``

Additi l information
Additionali fti
URL` Data format (GSM, RSM, MSM)
`9External Review (CCS)

2007/10/31

Metadata extraction
Input

GPVwgribwgrib
ib output
Taggergg
Metadata

GPV data store

Metadata generator
GPV
GPVMetadataMeta
data

Metadata store

GPV/JMA Archive
10

External Review (CCS)

2007/10/31

Metadata extraction (cont(cont’d)d)` Generate XML data from dumped GPV data
TimeStamp
TimeStamp

C
Creator

G idId ifi i
GridIdentification

Ph i lP
PhysicalParameter

Level

RepresentationType11

External Review (CCS)

2007/10/31

Workflow specification by BPEL4WS

12External Review (CCS)

2007/10/31

Demo13External Review (CCS)

2007/10/31

Funding and output` Funding`

Grant-in-Aid for Exploratory Research by JSPS

` Publication`

14

T. Amagasa, H. Kitagawa, and T. Komano, "Constructing a Web
Service System for Large-scale Meteorological Grid Data", 3rdIEEE Int'l Conf. on e-Science and Grid Computing (e-Science2007) Bangalore2007),
Bangalore, India
India, Dec
Dec. 10-13,
10-13 2007
2007.

External Review (CCS)

2007/10/31

Collaborations` Inside CCS`Global Environmental Science Group` External`GeoGrid (AIST)` Massive sensing data from satellites (ASTER & MODIS)`gfdnavi` Toolkit for earth scientists`NationalNi l Institute
I iforf LandL d andd Infrastructure
I fManagement (NILIM)` Sensing data of river flow
` Comprehensive tie-up agreement between

Univ. of Tsukuba and NILIM
15External Review (CCS)

2007/10/31

Research plan
` DB Integration`

Lightweight tool based on FUSE

` Machine learning and DM`

Event detection

Machine learning/
Data mining
MeteorologicalOntology` “Concentrated downpour”

Metadata•ESML

Data access w/FUSELinuxGPV/JMA Archive

Meteorological DB

River flow

Precipitation

XML Databases

17

External Review (CCS)

2007/10/31

XML: Standard data format on wire
` Meta language for data representation`RRepresent
tree structure byb plainl i text

` Rapidly diffusing`

AML BML
AML,BML, CML,
CML …, ZML,ZML AAML,
AAML ABML,
ABML ACML,
ACML …

` Typical usecases`

Web documents
``

XHTML

Business documents
ODF` MS Office Open XML``

Metadata
RDF (blogs)` Scientific data`

18

<CATALOG><CD><TITLE>Empire Burlesque</TITLE><ARTIST>Bob Dylan</ARTIST><COUNTRY>USA</COUNTRY><COMPANY>Columbia</COMPANY><PRICE>10.90</PRICE><YEAR>1985</YEAR>
YEAR 1985 /YEAR
</CD>
...
</CATALOG>
External Review (CCS)

2007/10/31

Research objective and recent activities` Providing
g infrastructure for managing
g g massive

XML resources
Integration ofheterogeneous XML
resources

Analytical processing of
XML data• OLAP for XML• Proximity queries overheterogeneous XML data
• Similarity join

Storage and retrieval of
XML in various informationsystems• PC cluster systems• P2P networks
19

External Review (CCS)

2007/10/31

Analytical processing of XML
How to extract information from XML?
` Searching`

Path-based query languages

Traditional OLAP

` XPath and XQuery` Information retrieval`

XML OLAP

Keyword-based queries` CO (Content Only)` CAS (Content and Structure)

Web Mining` AnalyticalA l ti l processingi`

OLAP-like interactions` Drilling,
Drilling Cubing,Cubing Pivoting,
Pivoting …

Î Interactive analysis for mining useful information
20

External Review (CCS)

2007/10/31

Example: DBLP Bibliography<inproceedings
idimdate="2002-01-03"
d"2002 01 03" kkey="conf/er/AmagasaATK98">" f/ /AATK98"<author>Toshiyuki Amagasa</author><author>Masayoshi
Aritsugi</author>
Typical examples
of analysis: “ResearchResearch trends in the DB domaindomain”<author>Takayuki Tanaka</author>•Number of submittedpapers<author>Yoshinari
Kanamori</author>•Topic<title>Implementing
Data.</
pg Class Libraryy and Index for Managing
g g Spatio-Temporal
pptitle> •Journal / Conference<pages>532-541</pages>•Number of distinct authors
year •Topic
1998i /year<year>1998</year>•T<booktitle>ER Workshops</booktitle>•Journal / Conference<url>db/conf/er/erw98.html#AmagasaATK98</url>•NumberNumber of frequently used keywords<crossref>conf/er/98w</crossref>•Topic<ee>db/conf/er/AmagasaATK98.html</ee><cite>journals/cacm/Allen83</cite>•Journal / Conference< i >•Author<cite>...</cite></ i ><cite>conf/dexaw/AmagasaAK98</cite>•…
<cite>...</cite><cite>conf/dexa/AritsugiTAK97</cite>
21

External Review (CCS)

2007/10/31

Traditional way
FeatureExtraction

XMLXMLXML(Non-XML)
OLAP
Data Mining

Text Analysis?
22

External Review (CCS)

2007/10/31

XML OLAP way
XML-OLAPSchema Translation
XQuery / XSLTRequest in terms ofXQueryXMLXMLXML23XMLOLAPResults

External Review (CCS)

2007/10/31

Technical challenges` Supporting concept hierarchy`

XML itself contains hierarchies for XML-OLAP` Supporting non-numerical attributes

Only numerical attributes are supported in (Non-XML)OLAP` XML contains both numerical and text attributes`` How we can summarize text according to concept hierarchy?

` Heterogeneity of XML data

Irregular structure` Missing attributes / values` Heterogeneous schemas`

24

External Review (CCS)

2007/10/31Proposed systemsales

bookinfo

area
ccna
me
“math”

kant
tso

kan
saiobbbcqt

q

qtt

bb

q

na
me
“db”

kbbbbbttc

name“linalgebra”

tq

t

pt

pt

p

pttt

pq

XPath
xmlcube
tuple

tuple
p

sales

bookinfo

area

c

kanto

c

tsukuba
b1bt
A 10q

tuple
psalesbookinfoname“math”

area

c

name
“linear algebra”

kanto

name
c “db”

tsukuba
b2b

p1 b
pp

sales

bookinfo

area

cka
ns
kyotoainamec “web”

name
“cs”

p4 bb6p5bpt

1000

bpqt
2000

D 20

name“cs”

qF 303400

XQueryw/OLAP
Extension

total

total

g1

25

g2

g3

qty

g1

g2

g3

qty

External Review (CCS)

2007/10/31

Facts about an XML Data
` Facts in an XML data are data items of interest, identified by
yan XPath (fact path).
Ex A user wants to get information of book sale by giving:
Ex.
sales

pf: doc(“sales.xml”)//barea

Facts: [[ pf ]] = {b1,…, b6}

kantokansai

b1 tsukuba
b2 b3btbq

A 10t

osaka b5
b4bqtbqtb6

kyotoybqtbq

External Review (CCS) 2007/10/31
26
D 20
C 10
C 60 B 40tq

F 30

Dimensions` A dimension is given as a path expression (dimension path)

sales
area
kanto

kansaib1tsukuba
b2 b3bt

bb4osakab5

bbqtqtA 10

D20

C 10qt

b6

kyoto

bqt

Dimension:doc((“bookinfodoc
bookinfo.xmlxml”)//xml”)//b[b[t=b[ttt=doc(“sales.xml”)//bdoc(“sales.xml”)//b/tdoc( sales.xml )//b/t)//b/t]][[ppf/t ]]b1 = {“A”}[[Æ[[ docdoc((“bookinfo.xml”“bookinfo.xml”))////b[b[t=“A”
t=“A”]] ]] = {p1p1}}[[ppf/t ]]b2 = {“D”}[[{ D }
Æ[[ docdoc((“bookinfo.xml”“bookinfo.xml”))////b[b[t=“D”
t=“D”]] ]] = {p4p4}}

bq

C 60 B 40

tq

F 30

bookinfoname c“math”

cname“linearlinear algebra
algebra”

name“db”
dbcbp4

p1 bp2 bp3btpt

name
“cs”

cp tpt

p

name
“ eb”
“web”

cp5btbptp

A 1000 B 3000
C 8000
D 2000External
Review (CCS)
27 E 6000 F 3400
2007/10/31

XML Cube
Dimension

Factsales
area

kanto

b1 b2b3bbbt

namec
“ th”
“math”
name“linear algebra” ckansaitsukubab4 b5bqtqt

A 10

D20

C 10b6kyoto

osakaqtbqt

name c
“db”
bp
p4 b

name
“cs”
c name“web”
bpqt p t p t pt p t p t 5p

F 30

A 1000 B 3000 C 8000

D 2000 E 6000 F 3400t

C 60 B 40

cbp1 bp2 bp3bq

bookinfo

xmlcube
tuple

tuple

sales
area

bookinfo
namec“math”

sales
area

kanto

c “linear algebra”

kanto

tsukuba

b1 bt

q

A 10

name

p1pbp

1000

tsukuba

b2 bt

q

D 20

tuple

bookinfo
name
c“cs”
name
c “db”

p4
pbp

2000

sales
area

bookinfo
name
c“cs”

kansaic“web”

kyoto

p5 bp

b6 bt

External Review (CCS)F

nameqp30
3400
2007/10/31
28

XML Cube and concept hierarchy
XML-CubeALL
Cheap Common
Expensive[0, 1000) [1000, 5000) [5000, 10000)[] p4b1b2p1b3p3bbbptq

A

10pptqD

101000tq

C

10p2pbb4q

B

10ptq

C

10JJapan

Osaka

8000

3000

Kansai

p5b6

K
Kanto

p3pbptT k b
Tsukuba

8000

2000

b5

Concept hierarchy•Value-based•Structure based•Structure-based

bptq

F

10

Kyoto

3400

29

External Review (CCS)

2007/10/31

Implementation

Systemdid

pid

nid

0

10nnum

tname

value

psalesckanttsokansaiobname“cs”

cqqtqtttcname“db”

name
“web”

kbbq

c

name
“linalgebra”

bbbb

Approach

02

11.1

c

null

0

32

1.1.1

@name

null

Node table033

1.1.1.1

CDATA

0

4

4

1.1.2

c0…

1

9

46

1

q

t

pt

pt

pt

pp

t

q

XPath

did

sales

bookinfo

area
kanto
tsukubab1bt

qbookinfo

sales

c

area

cc

name
“linear algebra”

kanto

name
c “db”

tsukuba
b2
b

p1 b
pp

name
“cs”

p4 b

sales

bookinfo

area

cka
nsai
kyoto

name
c “web”

b6p5bpt0tuplename“math”

t0

name
“cs”

01000

D 20

XQueryw/OLAP
Extension

sales

g1

2007/10/31

g2

47

1.1

area

null

1148

1.1.1

kanto

null

1

1249

1.1.1.1

tsukuba

null

1

1350

1.1.1.1.1

bnull

pexp

jkey

/bookinfo/c/c/b/p

A

Fact/bookinfo/c/c/b/p

B/bookinfo/c/c/b/p

C

did0

qty

g1

g2

g3

/bookinfo/c/c

3

4/bookinfo/c/c/@name6

5/bookinfo/c/c/b

6

6/bookinfo/c/c/b/t

12

7/bookinfo/c/c/b/p

128/sales111/sales/area/kanto/tsukuba

1SQL
pexp

jkey

/bookinfo/c/c/b/p

A

pexp

0/bookinfo/c/c/b/p

C

table

0/bookinfo/c/c/b/pD

0/bookinfo/c/c/b/p

E/bookinfo/c/c/b/pD

0/bookinfo/c/c/b/pE

0/bookinfo
bookinfo/c/c//c/c/b/b/p

F0

jkey

XML
Data-cube

B/bookinfo/c/c/b/
p

did/bookinfo/c/c/b/p/bookinfo/c/c/b/p

Dimension
0/bookinfo/c/c/b/p

F3400

SQL

Query Translation

total

g3

43Null

10g2

g1

total

2/bookinfo/c/@name

A0

B

0

C0D0E0FqF 30

2000/bookinfo/c3

math

1

table

0

A 102

bp

q

1

null

Query Translation

tuple/bookinfo

Path table1

xmlcube
tuple

poccu

1

bpt

Pexp

null

btt

Pathcname“math”

bbookinfo

bookinfo

areab1qty

External Review (CCS)

Total

...

…

…

2

Creditcard, Cash

Cash

164

2

Creditcard, CashCreditcard

2Creditcard, Cash

2

Creditcard, Personal Check

Creditcard

133

2

Creditcard, Personal Check

Personal Check133

2

Creditcard, Personal Check

2

Money order, Cash

Cash

143

2

Money order, Cash

Money order

143

2...

g3

1

164

164

133

Money order, Cash

30...

143

...

...

Time (ms)Time for XML-cube construction
100,000,000.000
10,000,000.000
1,000,000.000
100,000.000
10,000.000
1,000.000
100.000
10 00010.000
1.000
10MB

100MB

200MB

300MB

400MB

500MB

File size
item

quantity

payment

External Review (CCS)

2007/10/31
31

payqtyTime for GROUP BY computation

Timee (ms)

10,000.000

1,000.000

100.000

10.000
10MB

100MB

200MB

300MB

400MB

500MB

File size
rollup(payment)ll ()rollup(region)ll ( i )rollup(regionpay)ll ( i)

External Review (CCS)32
2007/10/31

Related work
` XML-OLAP is becoming a hot research area.

R. Bordawakar and C. A. Lang. Analytical Processing of XML
Documents: Opportunities and Challenges. In SIGMOD Record,
volume 34(2)
34(2), pages 27–32,
27 32 2005
2005.
` Chaitanya Gokhale, Nitin Gupta, Pranav Kumar, Laks
Lakshmanan, Raymond Ng, B.
B Aditya Prakash: Complex Group
Groupby Queries for XML, ICDE2007, 2007.
` Nuwee Wiwatwattana, H.V. JJagadish,gLaks Lakshmanan, Divesh
Srivastava: X3: A Cube Operator for XML OLAP, ICDE2007,2007.
`33

External Review (CCS)

2007/10/31

Conclusions` Proposed XML-OLAP

Facts and dimensions of XML data
` Prototype implementation using relational storage` Experimental evaluation`` Future work

Algorithms for computing multiple GROUP BYs by taking XML
features into account` System
Simplementation
landd in-depth
d h analysisl`External Review (CCS)

2007/10/31
34

Award and outputs`Award``DBSJ Paper Award 2005: “Processing Large-Scale XML Data by RDBMS using Region Directory”

Recent publications
```

`

``

35

Toshiyuki Amagasa, Lianzi Wen, and Hiroyuki Kitagawa, "Proximity Search of XML Data using Ontology
and XPath Edit Similarity", 18th Int'l Conference on Database and Expert Systems Applications(DEXA2007), pp. 298-307, Regensburg, Germany, Sept., 2007.
Toshiyuki Amagasa, Kentarou Kido, and Hiroyuki Kitagawa, "Querying XML Data using PC ClusterSystem", 2nd International Workshop on XML Data Management Tools and Techniques (XANTEC'07) inconjunction with DEXA 2007, pp. 5-9, Regensburg, Germany, 2007.
Toshiyuki Amagasa, Chunhui Wu, and Hiroyuki Kitagawa, "Retrieving Arbitrary XML Fragments from
Structured Peer-to-Peer Networks,
Networks " Joint Conference of the 9th Asia-Pacific Web Conference and the8th International Conference on Web-Age Information Management (APWeb/WAIM 2007), pp. 317-328,
HuangShan? (Yellow Mountains), China, June 2007.
Chantola Kit, Toshiyuki Amagasa, and Hiroyuki Kitagawa, "OLAP Query Processing for XML Data inRDBMS," The Third IEEE International Workshop on Databases for Next-Generation Researchers(SWOD2007) ini conjunction
ji withi h ICDE2007
ICDE2007, IIstanbul,b lTTurkey,k A
Aprilil 15,15 2007.
2007
Kentarou Kido, Toshiyuki Amagasa, and Hiroyuki Kitagawa, "Processing XPath Queries in PC Clusters
Using XML Data Partitioning," The Second International Special Workshop on Databases for NextGeneration Researchers (SWOD 2006) in conjunction with ICDE 2006, pp. 114-119, Atlanta, Georgia,
p 7,, 2006.
USA,, April
Kei Fujimoto, Toshiyuki Shimizu, Dao Dinh Kha, Masatoshi Yoshikawa and Toshiyuki Amagasa, "A Mapping
Scheme of XML Documents into Relational Databases using Schema-based Path Identifiers,"
International Workshop on Challenges in Web Information Retrieval and Integration (WIRI 2005), April
8-9, 2005

External Review (CCS)

2007/10/31