External Review for CCS
Oct 30, 2007

Gfarm Grid File System
Osamu Tatebe
University of Tsukuba
tatebe@cs.tsukuba.ac.jp

Petascale Data Intensive Computing
High Energy Physics
z CERN LHC, KEK-B Bellez ~MB/collision,
100 collisions/secz ~PB/year
z2000 physicists, 35 countries

Detector for
LHCb experiment

Detector for
ALICE experiment

Astronomical Data Analysis
z data analysis of the whole dataz TB~PB/year/telescopez Subaru telescopez 10 GB/night, 3 TB/yearPetascale Data Intensive Computing
Requirements
Storage Capacity
Peta/Exabyte scale files, millions of millions of files
Computing Power> 1TFLOPS, hopefully > 10TFLOPS
I/O Bandwidth> 100GB/s, hopefully > 1TB/s within a system andbetween systems
Global Sharing
group-oriented authentication and access control

Gfarm Grid File System [CCGrid 2002]
Commodity-based distributed file system that federates
storage of each site
It can be mounted from all cluster nodes and clients
It provides scalable I/O performance wrt the number ofparallel processes and users
It supports fault tolerance and avoids access concentration
by automatic replica selection

Globalnamespace /gfarmggfaistfile1

jp
gtrc
file2

file1
file3

Gfarm File Systemmapping
file2

file4

File replica creation

Gfarm Grid File System (2)Files can be shared among all nodes and clients
Physically, it may be replicated and stored on any filesystem node
Applications can access it regardless of its location
File system nodes can be distributed
GridFTP, samba, NFS server
GridFTP, samba, NFS server
Compute & fs node

/gfarmGfarm metadata server

metadata

Compute & fs nodeFile A

Compute & fs nodeCompute & fs nodeFile FileB C

Compute & fs nodeCompute & fs nodeFile A

Compute & fs nodeCompute & fs nodeCompute & fs nodeCompute & fs nodeUS

File B

Compute & fs nodeJapan

File CClientPC

File A
File C
File B
Gfarm
file system

NotePC…

Scalable I/O Performance
Decentralization of disk access putting priority tolocal disk
When a new file is created,
Local disk is selected when there is enough space
Otherwise, near and the least busy node is selected

When a file is accessed,
Local disk is selected if it has one of the file replicas
Otherwise, near and the least busy node having one of file
replicas is selected

File affinity scheduling
Schedule a process on a node having the specified file
Improve the opportunity to access local disk

Scalable I/O performance indistributed environment
User’s view

Physical execution view in Gfarm(file-affinity scheduling)

User A submits Job A that accesses File A

Job A is executed on a node that has File A

User B submits Job B that accesses File B

Job B is executed on a node that has File B

network
Cluster,
Cluster,GridGridCPUCPUCPUCPU

Gfarm file system
Shared network file system

File system nodes = compute nodes

Do not separate storage and CPU (SAN not necessary)Move and execute program instead of moving large-scale data
exploiting local I/O is a key for scalable I/O performanceFIRST – CCS Astrophysics Simulator
256 Compute Node
Dual Xeon for 256 nodes
Blade-GRAPE (240 nodes)
3.1 TFlops + 33 TFlops
Gfarm file system
12.8 TB (36 GB x 240 + 250 GB x 16 + 480 GB)% df /gfs/home/tatebe
Filesystem
1K-blocks
Used Available Use% Mounted ongfarmfs
13292988192 3062931612 9554800896 25% /gfs/home/tatebe

Japan Lattice Data Grid – Advanced
Nationwide Data Sharing
Nationwide distributed filesystem to share QCD data
Transparent data accessregardless of the data
location
Efficient data access withfault tolerance thanks toincorporated file replicas
management
Flexible capacity management
Kanazawa U

SINET3
Kyoto U
Hiroshima U

Osaka UKEKVOMSVO server

U Tsukuba

Gfarm Grid File System

Virtual Organization (VO)membership management
Project base, independent
from real organizations
VO based (project based)
Access control
Easy access with single signonSoftware packaging foradvanced data sharing
Commodity hardware andopen source software
Globus, VOMS, Naregi-CA,Gfarm, Uberftp, . . .
Easy deployment

Particle Physics Data Analysis• Construct 26 TB of Gfarm FS
using 1112 nodes• Store all 24.6 TB of Belleexperiment data
• 52.0GB/s in parallel read
→ 3,024 times speedup• 24.0GB/s in skimming
process for b → s γ decays
using 704 nodes→ 3 weeks to 30 minutes

I/O bandwidth [GB/sec]

O. Tatebe et al, “High Performance Data Analysis
for Particle Physics using the Gfarm File System”,
SC06 HPC Storage Challenge, Winner – LargeSystems, 2006
52.0 GB/sec

6050403020
1000200400600800
number of client nodes

100012001112nodes

PRAGMA Grid
C. Zheng, O. Tatebe et al, “Lessons Learned
Through Driving Science Applications in the
PRAGMA Grid”, Int. J. Web and Grid Services,Inderscience Enterprise Ltd., 2007
• Worldwide Grid testbed
consisting of 14 countries, 29
institutes• Gfarm file system is used forfile sharing infrastructure→ executable, input/output
data sharing possible in Grid
→ no explicit staging to a local
cluster neededMore Feature of Gfarm Grid File System
Commodity PC based scalable architecture
Add commodity PCs to increase storage capacity in operationmuch more than petabyte scale
Even PCs at distant locations via internet

Adaptive replica creation and consistent management
Create multiple file replicas to increase performance andreliability
Create file replicas at distant locations for disaster recovery
Open Source Software
Linux binary packages, ports for *BSD, . . .
It is included in Naregi, Knoppix HTC edition, and Rocks distribution

Existing applications can access w/o any modification

SC03 Bandwidth

SC|05 StorCloud SC06 HPC Storage Challenge Winner

Design and implementation ofGfarm v2

Design policy of Gfarm v2
Inherit architectural benefit of scalable I/O
performance using commodity platform
Design as a POSIX compliant file system
Solve security problems in Gfarm v1
Improve performance for small files
Reduce metadata access overhead
Grid file system -> Distributed file system
Still benefit from improvement of local filesystem
Compete with NFS, AFS, and LustreGfarm v2 testbed
Metadata server
Univ of Tsukuba
File system nodes
UnivTsukuba

AISTSDSC#nodes1483

RTT[msec]

0.202

0.787119NFS bandwidth (read 1G sep. data)

37.0 MB/s

Gfarm Scalable Bandwidth
1,433 MB/s

Univ Tsukuba

AISTSDSC

[msec]Operation latency (2~3RTT)
SDSC

238 msec

1~3 msec

AIST

0.2~0.5 msec
NFS(async)

U Tsukuba

Local FSSummary
Gfarm file system
Scalable commodity-based architecture
File replicas for fault tolerance and hot spot avoidanceCapacity increase/decrease in operation
Gfarm v1
Used for several production systems
1000+ clients and file system nodes scalability
Gfarm v2
Plug up security hole in Gfarm v1, and improve metadata accessperformance
Comparable performance with NFS for small files in LAN
0.2 ~ 0.5 milliseconds

Scalable file IO performance even in distributed environment

1433 MB/sec parallel read IO performance from 22 clients in Japan and US

Open Source Development
http://sourceforge.net/projects/gfarm