Japanese Virtual Observatory andWorkflow

Masahiro Tanaka
National Astronomical Observatory of JapanFeb 26, 2008

Astro-RG, OGF221

This talk
• Overview of JVO• JVO Workflow system• Investigation of GUI Workflow builder• Services called from Workflow

Feb 26, 2008

Astro-RG, OGF222

JVO system : Portal and Services
User

解析
解析
解析
Analysis
Servers
Grid

JVO
PortalSystemSDSS

InvokeMDS

PortalUser
Storage

QSO

SkyNode1.0…

Subaru2MASSData Services

S. Reg.

Other VOs

P. Reg.

Registry
Feb 26, 2008

Astro-RG, OGF22

P. Reg.

SIAP/SSAP

SkyNode0.7.4 3JVO Portal - top page

http://jvo.nao.ac.jp/portal/
Feb 26, 2008
4January24, 2008
ADCAstro-RG,
International
OGF22
Review4

JVO SkyNode Architecture• Uses Apache Axis and Tomcat.
• Accepts Four kinds of query languages.
JVO Portal• Query is executed through JDBC-like interface.

Internet• Query result is formatted as VOTable or CSV.

JVO SkyNode

JVOQL
ADQL

HTTP/SOAP
AXIS

SIAP
SSAP

HTTP/Parameter
TomcatFeb 26, 2008

Translator
Query Executer
Formatter

Astro-RG, OGF22

SelectSQLJDBC forSkyNode
ResultSet
getMetadata()
getString()
getFits()getJpeg()

JVO
SkyNodeDBMS5Finding Services – Registry
Searchable Registry
Publishing Registry
Publishing Registry

Query Services

DataServiceAnalysis
Service
Virtual Observatory Client

Feb 26, 2008

Astro-RG, OGF226Grid service for Subaru image
reduction•Automated job assignment with Monitor and Discovery Service (MDS)MDS Server

JVO Portal
Execute# of Jobs
CPU load
Job status

Service Request
Available Server Info
Confirm Job Completion

ClientJob Completion InfoAnalysis ServersAnalysis ServerManagement Data

Job submission
Results

Parallel Executions on Multiple ServerFeb 26, 2008

Astro-RG, OGF227JVO Workflow system

Feb 26, 2008

Astro-RG, OGF228

Example of workflow
SkyNode

Query

JVO

QSO Catalog

workflow
control

SIA ServiceSubaru
image
HyperZ
WebServices

SourceCatalog

SExtractor
WebServices

Analysis Services
Feb 26, 2008

Astro-RG, OGF229

JVO Workflow Language
XML Diagram of JVO Workflow Description Language
Parfor
10..1<<rootnode>>
ProcessFor

1

Variables

While

11..*

10..*

<<abstract>>
LoopControl<<abstract>>
Activity

Variable1..*1<<abstract>>
SequenceActivity

awk<<abstract>>
Control

1<<abstract>>
ConditionControl

1<<abstract>>
BasicActivity<<abstract>>
ActivityContainer
ifSequence

Flow

1Script

Command1

10..10..1

Input

Feb 26, 2008

invoke10..11..*ThenElseCaseSwitch
10..1Otherwise

OutputAstro-RG, OGF22

10

Current workflow builder for JVO• Editing XML directly– Difficult to write XML– Use Templates• No GUI workflow
builder for JVO

Feb 26, 2008

WF Editor

Execution monitor

Astro-RG, OGF22

11

Investigation of GUI Workflow
builder for JVO

Feb 26, 2008

Astro-RG, OGF22

12

JFLOW• Developed at CDS• Good User Interface• Does not support flow
controls:– Condition– LoopFeb 26, 2008

Astro-RG, OGF22

13

Workflow editor for NAREGI• Good Graphical WF
builder• Developed for Grid
workflow for NAREGI• Applicable to VOServices??

Feb 26, 2008

Astro-RG, OGF22

14

NAREGI• Building Grid environment using NAREGI
Middleware β2.0.1• Trying Interoperability with KEK (Institutefor high energy physics)• Problems:– Difficult to install NAREGI middleware– No support for VO Protocols
Feb 26, 2008

Astro-RG, OGF22

15

Taverna.

• Developed for Biology but usable forgeneral purposes• Evaluated by AstroGrid• JVO team has just started evaluation• Client-side application– JVO has server-side workflow

Feb 26, 2008

Astro-RG, OGF22

16

Trying Taverna• Workflow calling a single Web Service

Feb 26, 2008

Astro-RG, OGF22

17

Executing…

Feb 26, 2008

Astro-RG, OGF22

18

Result

Feb 26, 2008

Astro-RG, OGF22

19

Taverna good things• Good User Interface• Easy to call Web Services• Able to save workflow and results as XML• Good workflow engine– Job monitoring and control– Logging (intermediate results and status)

Feb 26, 2008Astro-RG, OGF22

20

Taverna issues…
• Still need knowledge on– Web Services– Input/Output data types– Programming• Need Communication with :– Client-side software– VO RegistryFeb 26, 2008

Astro-RG, OGF22

21

Scripting languages• Define new Workflow Language easier to write
than XML?
– Not difficult to define it from XML definition– We have already workflow engine.
– Need language design– Learning cost for users• Use Existing language?
– Perl, Python, Ruby, …
– No learning cost if user knows
Feb 26, 2008

Astro-RG, OGF2222SOAP call example using
Scripting Language• Ruby script :require 'soap/wsdlDriver'wsdl = 'http://ion.mtk.nao.ac.jp:8080/axis2/services/DetectLine?wsdl'url= 'http://jvo.nao.ac.jp/skynode/sdss/spectrumRequest.do?'+'db=sdss&table=spectrum&id=51630-0266-001&format=spectrum/fits'

driver = SOAP::WSDLDriverFactory.new(wsdl).create_rpc_driver
result = driver.invoke(:url=>url).return• Easy to call Web Services• Script code can be a workflow

Feb 26, 2008

Astro-RG, OGF2223

We have choices• We just started evaluation.
• Taverna is a promising tool.
• Need more experience.

Feb 26, 2008

Astro-RG, OGF22

24

Services called from Workflow

Feb 26, 2008

Astro-RG, OGF22

25

Services in VO• VO framework– Data archives• Registry, SIAP, SSAP, ConeSearch, SkyNode, TAP,

– Data storage• VO Space• Need more
– Analysis services– Visualization services– Interoperability with client tools• SAOImage, VOPlot, Aladin, …
Feb 26, 2008

Astro-RG, OGF22

26

Use Cases of Workflow• Subaru data reduction– Service: Parallel execution with cluster• Study on AGN environment– Service: SExtractor and HyperZ• Search for Metal-poor stars– Service: Line detection service

Feb 26, 2008

Astro-RG, OGF2227

Automatic Spectral Line Detection
Service• Detect every scale of line width• Wavelet-like algorithm

Feb 26, 2008

Astro-RG, OGF2228

Grid Challenge• Public Contest of HPC– Held in mid 2008 by Grid Scientists– JVO provides Subaru data and scientific scenario :• Supernova search– Good experience to build Grid services forAstronomy

Feb 26, 2008

Astro-RG, OGF2229

Useful Services• Re-usable, general-purpose services• Services which require computerresources– CPU– Storage• Services with clearly-defined interface

Feb 26, 2008

Astro-RG, OGF2230

Issues• Too few available services– Publicly available services are needed.

• WSDL can be written freely :– Variety of interface can be confusing.
– Granularity of Services?
– Need Methodology / Guideline / Standard ?

Feb 26, 2008

Astro-RG, OGF2231

Conclusion• JVO system and workflow are reviewed.
• GUI WF builder for JVO is evaluated :– Taverna is a promising tool.

• Useful services called from Workflow areneeded.

Feb 26, 2008

Astro-RG, OGF22

32