Development of JVO prototype
system and its application toAstrophysics
Portal System :M. Tanaka
Data Service :Y. Shirasaki
Science Applications : S. Honda

Oct. 1, 2004

IVOA Small Projects Meeting

1

JVO : Japanese Virtual Observatory• Purpose:– Easy access to federated Astronomical databases– Interoperability through IVOAJVOInteroperability through IVOA

Researcher

Ground-base telescopes

Satellite-borne telescopes

Nobeyama

SubaruOct. 1, 2004

ASTRO-EIIIVOA Small Projects Meeting

M. Tanaka (NAOJ)

2

JVO activities• Development of JVO System– Federated Data Servers using Grid• Development of Query Language for JVO• Interoperability in IVOA
– work on VOQL WG– chair: Prof. Ohishi

Oct. 1, 2004

IVOA Small Projects Meeting

M. Tanaka (NAOJ)

3

JVO collaborators
Project Scientists
NAOJ• Mizumoto• Ohishi• Shirasaki• Tanaka• Honda• Kawanomoto
ICRC• Yasuda
Ochanomizu U.
• Masunaga
Oct. 1, 2004

System Engineers
Fujitsu Ltd.
• Monzen• Kawarai• Ishihara• Yanaka• Yamaguchi• Ishida• Saito• Abe• Tsutsumi
IVOA Small Projects Meeting

SEC Ltd.
• Morita• Nakamoto• Kobayashi• Yoshida

M. Tanaka (NAOJ)

4

JVO Portal System

Masahiro Tanaka
National Astronomical Observatory of Japan

Oct. 1, 2004

IVOA Small Projects Meeting5Development Strategy• “Top-down” approach– set “Science Use Cases”– study and design “Overall System”• How to federate distributed computers?

– build “Prototype System”• with minimal capabilities• to achieve use cases• “Build-and-scrap” prototype
Oct. 1, 2004

IVOA Small Projects Meeting

M. Tanaka (NAOJ)

6

Development history• JVO Project start ― April 2002
• Prototype 1 finish ― March 2003
• Prototype 2 finish ― March 2004• Prototype 3

Oct. 1, 2004

― under development

IVOA Small Projects Meeting

M. Tanaka (NAOJ)

7

JVO Portal Server• Web Interface to users• Interpret JVO Query Language• Retrieve query result from Distributed DataServers• VOTable and FITS image browser

Oct. 1, 2004

IVOA Small Projects Meeting

M. Tanaka (NAOJ)

8

JVO components
RegistryRegistry ServerServer
Metadata

PortalPortal ServerServerController

JVOQL
ParserScheduler

ExecuterUser InterfaceWebaccess

GridData
Data Servers
Servers
Catalog andImage DB

Grid…
Oct. 1, 2004

IVOA Small Projects Meeting

M. Tanaka (NAOJ)

9Use Case• Cross-matching z’- and i’-band catalogs ofSubaru Deep Field• Retrieving images for each object

Oct. 1, 2004

IVOA Small Projects Meeting

M. Tanaka (NAOJ)

10JVOQL
selecti.POS_EQ_RA_MAIN as ra1,
i.POS_EQ_DEC_MAIN as dec1,
i.PHOT_SDSS_I,
z.POS_EQ_RA_MAIN as ra2,
z.POS_EQ_DEC_MAIN as dec2,
z.PHOT_SDSS_Z,
i.BOX(POINT(ra1,dec1), 20 arcsec, 20 arcsec),z.BOX(POINT(ra2,dec2), 20 arcsec, 20 arcsec)
fromnaoj.sdf.i i,naoj.sdf.z z
where
XMATCH(i, z) < 10 arcsec NEAREST andBOX(POINT(201., 27.4), 5 arcmin, 5 arcmin)

Oct. 1, 2004

IVOA Small Projects Meeting

M. Tanaka (NAOJ)

11

Parsing JVOQL and Generating
Workflow• “JVOQL Parser” generates query for each host• “Scheduler” generates:– count query job for host1– count query job for host2• “Executer” executes jobs on remote hosts• “Scheduler” generates based on the result of execution– query job for host1– xmatch job for host2– image query for host1 and host2• “Executer” executes jobs on remote hosts
Oct. 1, 2004

IVOA Small Projects Meeting

M. Tanaka (NAOJ)

12

Federateion of Distributed Servers
1
JVO Portal Server
Execution Flow(Continued)

CountQueries

Count
Numbers

Catalog Searchservice

XMatch Searchservicei’ bandCatalog DBz’ band
Catalog DB

host1

Oct. 1, 2004

host2

IVOA Small Projects Meeting

M. Tanaka (NAOJ)

13

Federateion of Distributed Servers
2
JVO Portal Server
Execution Flow(Contd)
Catalog
Query
Catalog Searchservicei’ bandCatalog DB

host1

Oct. 1, 2004

Result Catalog(VOTable)

Catalog
Query

ResultCatalog(VOTable)

XMatch Searchservicez’ band
Catalog DB

ResultCatalog(VOTable)

host2

IVOA Small Projects Meeting

Image
QueryResult Image(FITS)

Image Cutout
servicei’ bandImage DB

host1ｚ’ band
Image DB

host2

M. Tanaka (NAOJ)

14

Description of Workflow• Count queries can be executed in parallel.
• Search and XMatch service must be called sequentially.
→ “Dependency” is considered.
– Parallel execution is to be implemented.

Resource {JobElement {dependElements

executedServiceName

inputResources

executedServiceURL

outputResource

fileType

serviceNames

resultserviceURLs}}
Oct. 1, 2004

IVOA Small Projects MeetingM. Tanaka (NAOJ)

15

Remote execution• Proto1:– Globus toolkit ver 2– using globus-job-run command– 1 call = 30 sec• 1 query ~10 min!

• Proto2:– Globus toolkit ver 3– using Grid Service– 1 call = 2-3 sec• overhead time is only ~ 30 ms

Oct. 1, 2004

IVOA Small Projects Meeting

M. Tanaka (NAOJ)

16

Data Transfer
Portal Server• Web/Grid Service

Query– Query result isalways returned to
Portal server

VOTable1

Data Server• GridFTP– Query result can bedirectly transferredto XMatch server

XmatchQuery

VOTable2

Data ServerPortal ServerQueryXmatch
Query

VOTable2

Data Server VOTable1 Data ServerOct. 1, 2004

IVOA Small Projects Meeting

M. Tanaka (NAOJ)

17

Data Transfer• We tried:– GridFTP, RFT– GSI-SFS• Learned:– SFS is not flexible• A server cannot be a client.

– RFT is promising in Globus environment– need support for HTTP and Web Services
Oct. 1, 2004

IVOA Small Projects Meeting

M. Tanaka (NAOJ)

18

Data Discovery• Proto1: UDDI
– UDDI is for “Service discovery”, not for “Datadiscovery”• Proto2: XMLDB– XMDB product “Karearea” is used.
– XPath search– enables both “Data discovery” and “Service discovery”

Oct. 1, 2004

IVOA Small Projects Meeting

M. Tanaka (NAOJ)

19

Resource Metadata
service metadata
content metadata

Identity metadata
curation metadata
title
short_nameidentifier
publisher
publisher_id
creatorcreator_logo
contributer
dateversioncontact_namecontact_email
service_interface_url
service_base_url
service_http_result
service_standard_uri
service_standard_url
service_msr

stringstringURI
stringURI
stringURL
stringstringstringstringe-mail address
URLURLMIME type
URIURLfloat,decimal degreesucd
unit
datatype
width
precisionarraysize

stringstringstringint
stringstringcatalog
table
columnidentity○○○

Oct. 1, 2004

curation○○○

service○○○content○○○

column××○column metadata

subject
description
sourcereference_url
type
coverage_spatial
coverage_region_of_regard
coverage_spectral
coverage_spectral_bandpass
coverage_spectral_central_wavelength
coverage_spectral_minimum_wavelength
coverage_spectral_maximum_wavelength
coverage_temporal_start_time
coverage_temporal_stop_time
coverage_depth
coverage_depth_unit
coverage_object_density
coverage_object_countcoverage_sky_fraction
resolution_spatial
resolution_spectral
resolution_temporalcontent_level
facility
instrument
formatright

IVOA Small Projects Meeting

string,liststring,free text
stringURL
string,liststring
float,decimal degrees
string,liststring,list
float
float
float
stringstringfloat
string
float
int
float
float
float
float
string,liststring,liststring,liststring,liststring

M. Tanaka (NAOJ)

20

Catalog、Table Resourceはcolumn metadata 以外を持つ。

User Interface• Java Servlet– Tomcat– Struts• Easy use by scientists (S. Honda’s talk)– needs only Web browser, no extra installation.

Oct. 1, 2004

IVOA Small Projects Meeting

M. Tanaka (NAOJ)

21

Plans toward Prototype 3• support SIAP, SSAP, SkyNode• implement ADQL

next talk by Y. Shirasaki• improve registry, employing OAI-PMHarchitecture• flexible workflow architecture• introduce User management– LDAP– User Storage Area (support VOStore?)• API to control JVO with SOAP
Oct. 1, 2004

IVOA Small Projects Meeting

M. Tanaka (NAOJ)

22

END

Oct. 1, 2004

IVOA Small Projects Meeting

M. Tanaka (NAOJ)

23