Impact of Gfarm, a Wide-area Distributed File System, upon
Astronomical Data Analysis and Virtual Observatory
Masahiro Tanaka and Osamu Tatebe (University of Tsukuba)
While 100 TB-scale astronomical data are available through Virtual Observatories, there are still several issues for large-scale data analysis that include
transferring a large amount of data and securing enough capacity of storage. We thus propose a VO-capable file system in order to offer easy access toastronomical data, by utilizing Gfarm, a wide-area distributed file system developed as an e-Science infrastructure. Gfarm is a distributed file system thatfederates storage systems in wide area. It is designed to achieve high reliability and high performance exploiting file replicas and distributed file access. Thesefeatures facilitate large-scale astronomical data analysis under research collaboration of multiple distant organizations. We discuss file system structure andsearch method which are compliant with VO standards and the initial performance of data analysis on this system.

Large area Astronomy data sharing with Gfarm• Wide-area Distributed File System• Open Source Development

NAOJ

VirtualObservatory

LaboratoryAJAXA– http://datafarm.apgrid.org//• Scalable I/O performance/subaru/akari– access locality…

• Automatic File Replica Selection/subaru/spcam– fault tolerance– avoids access concentrationdata• Fits for Grid Computing

…

/labA
… Gfarm…
/labA/personB/akari/fis…

data/archives

data

Access-protected
to Observer…

/archives/2mass
…

data

…
Analyzed Data

Public Data
Astronomical Data
in a Single Global Tree

Gfarm Components
Metadata
Server (MDS)

File System Nodes

Managementof File System

Local
Storage

Local
Storage

file1

file2

Local
StorageGfarm File System/Gfarm
ClientAccess by pathname
in a Global Directory Tree/dir1• Define unique data path– use VO ID• ivo://edu.jhu/sdss→ /edu.jhu/sdss• Data Search– VO Metadata (XML)– XPath search in Gfarm• Coordinate Search– simple proxy to VO protocol• Coordinate search is impossible with XPath.
• Proxy to SIAP service is developed.
• The Proxy transfers query to original SIAP service.
• Image data is transferred to Gfarm on demand.

Gfarm File System/edu.jhu
sdss

UserMetadata

image1HTTP request

HTTP request

http://gfs.jp?POS=...

http://sdss.edu?POS=...

(1)

(2)

Proxy to SIAP

(5)
Data path

Gfarm Filesystem(4) Datatransfersdss

VirtualObservatory

file3 file4

SIAP service(3)
URL/edu.jhu/dir2

file1 file2

Search into Gfarm with SIAP proxy

Search into Gfarm with XPathimage2
image1

image1

Large-scale data analysis with Gfarm
Scalable I/O performance in distributed
environment• Do not separate storage and CPU• Move and execute program instead of moving large-scaledata• exploiting local I/O is a key for scalable I/O performance

GXP - parallel grid shell• workflow is described as Makefile• parallel job execution• statistics log - useful for performance evaluation

 Measured using tsukubacluster of InTrigger Platform• Xeon E5410 2.33GHz• 2-sockets/node• 4-cores/socket• 32GB memory• 2TB HDD
• GigEx2 network

File System
Nodes
Job forFile 1
Local
Storagefile1

Job forFile 3

Job forFile 3
Local
Storage

Local
Storage

file3

file2

Performance evaluation of Montage workflow

Local
Storage Input file:• 2MASS J-band images• 6669 files
 Total processes:• mProjectPP :6,669• mDiff & mFitplane : 10,521• mBackground :3,474• mAdd & mShrink :324

• Files may be replicated and stored in any file system node
– fault tolerance– avoids access concentration

Makefile for Montage workflow
all: $(target)
pimages.tbl: $(RAWFITS) $(REGION)-mkdir -p p$(MAKE) -k -f mk_proj INPUT=$(INPUT) all

File System Nodes

file1

Local
Storage

Local
Storage

file1

Local
Storage
file1

diffs.tbl: pimages.tbl
mOverlaps $< $@Local
Storage

Gfarm File System//dir1
file1 file2

2 nodes
16 procs
4 nodes
32 procs

file4

Automatic File Replica Selection

Local
Storage

1 node8 procs/dir2
file3 file4

mk_diff: diffs.tbl
awk -f mk_diff.awk -v INPUT_DIR=$(INPUT_DIR) $< > $@fits.tbl: mk_diff-mkdir -p d$(MAKE) -f $< INPUT=$(INPUT) all
corrections.tbl : fits.tbl pimages.tbl
mBgModel pimages.tbl $< $@cimages.tbl: corrections.tbl pimages.tbl-mkdir -p c$(MAKE) -f mk_corr INPUT=$(INPUT) allmosaic.fits: cimages.tbl $(REGION)
mAdd -p c $< $(REGION) $@mosaic.jpg: mosaic.fits
mJPEG -ct 0 -gray $< -1.5s 60s gaussian -out $@t/tile_0_0.hdr: cimages.tbl $(REGION)-mkdir -p t
bash tile_region.sh $(REGION) $(TILE_PIXEL)
simages.tbl: t/tile_0_0.hdr cimages.tbl $(REGION)$(MAKE) -f mk_tile INPUT=$(INPUT) SHRINK_FACTOR=$(SHRINK_FACTOR) all
shrunk.fits: simages.tbl $(INPUT_DIR)/shrunken.hdr
mAdd -n -e -p s simages.tbl $(INPUT_DIR)/shrunken.hdr shrunk.fits
shrunk.jpg: shrunk.fits
mJPEG -ct 0 -gray $< -1.5s 60s gaussian -out $@8 nodes
64 procs

15 nodes120 procs