Current Status of the Japanese Virtual Observatory Portal
P063

Y. Shirasaki, Y. Komiya, M. Ohishi, Y. Mizumoto (NAOJ),
Y. Ishihara, Y. Yanaka, J. Tsutsumi, T. Hiyama (Fujitsu) ,H. Nakamoto, M. Sakamoto (SEC)

The Japanese Virtual Observatory (JVO) portal (http://jvo.nao.ac.jp/portal/) is a web portal foraccessing astronomical data and analysis system through the Internet. In 2009 and 2010, wedeveloped two new data access interfaces: JVOSky and command-line access interfaces. To enable
user to perform all sky search based on SED properties of celestial objects, we experimentally usedthe Hadoop for performing cross-match of 10 billions of photometric records in the JVO DigitalUniverse.

JVO SkyCommand-line access to JVO

JVOSky is an on-line data discovery service which displays thecoverage of observations made by various instruments on theGoogle sky. Using this interface, a user can graphically find sky
regions where data of multi-wavelength observations exist.

Although a graphical user interface (GUI) is a convenient way for
performing a simple query, it is not efficient nor flexible for performing alot of queries by changing query parameters. Such a situation happens
when a user wants to get a large number of data that may exceed themaximum number that a data service can return. We, therefore, have
implemented a command line search interface that is accessible throughtyping commands on the user's computer.

JVO portal

JVO Computing GRID

CUI for jc (jvo command)①②
Currently data of Subaru(Suprime-Cam & HDS) andSuzaku are available. More data
coming in future.

①Web GUI②④

JVO
SearchableRegistry
Suzaku observations displayed onthe JVOSky are linked to the DARTS
system operated by JAXA/ISAS. Suzaku
data can be downloaded from there.
The other data registered to DARTS(AKARI etc) will be added.

An example for using thecomputing service① Job submit② Submit to JVO grid
③ Save result on JVOSpace④ Retrieve the result

We have a plan to add
data crawled from VOservices.

③③VO Service

JVOSpace

Syntax of jc (jvo command):Reduced Subaru
Suprime-Cam and HDS areavailable.

④jc <command> [<option>] [<argument>]…

Examples:An example for using theVO query service① Submit a query② Search a VO service③ Search to VO service④ Retrieve a FITS image

jc search –i <jvoql_file>
jc registry –k <keyword>
jc copy2l <source> <destination>
jc run <program_name> <arguments>
jc join –s <votable1> -t <votable2> -o <output> --s-ra<RA_column> --s-dec <DEC_column> …

Other commands:
ls rsync passwd resume suspent abort ps union join selectCross-match using Hadoop

Experiment 1 billions records (1/20 of whole data)HDFS
JVO has a huge astronomical database
32.1 +13.4 ...
# RA
DEC BANDMAG
32.1 +13.4 ...
called Digital Universe, which contains
32.1 +13.4
R 23.2 ...
30.1 +12.1 ...
32.1 +13.4
K 20.2 ...
...
coordinates and photometric information
30.1 +12.1
J 16.3 ...
20.2 -21.3
R 16.4 ...
18.3 -23.3
B 18.3 ...
of celestial objects collected from major
20.2 -21.3 ...
18.3 -23.2
I 20.0 ...
18.3 -23.3 ...
...
survey catalogs. Currently we provide a18.3 -23.2 ...
...
functionality to search for data based on
coordinates only. However, there wouldbe a science use case where a user wants
MapReduce for Cross Match
to search based on SED properties. In Divide the whole dataset
order to provide this kind of searching
into subsets based on afunctionality, cross identification among
region of sky.
different catalogs should be performed in The Map function processes
advance. A search could be conducted
whole of the input file toagainst the whole sky, and all the data
produce cross match result
should be scanned in a reasonable time
(list of matched record ids)
scale. To achieve such a functionality we The Reduce function is not
are now developing a distributed data
executed, since each subset
search system by means of the Hadoop.
is independent each other.

 Divided into 6112 files. ~3MB/filemap<1, {1,2}><2, {3}>...

map<1, {1}><2, {2,3}>...

 Each file contains records of which pos error
circle overlaps with the same region specified
with an HTM index (level 6).
 Each file are gziped and copied to HDFS.
 Max number of task executed in parallel
1, 40, 70, 100, 160
 Hardware
10 servers: each has 2x4 core and 4 SATA HDD

Result If executed by a single task
9 days for 1G records  180 days for wholedataset (20G rec.) Parallel execution of 70
3.7 hours for 1G rec.  3 days for whole Scaling relation breaks around ~40 tasks
Overhead of writing to the local FS.
Writing time occupies ~60% of the total.