ILDG/JLDGの進捗
HPCI戦略プログラム分野５『物質と宇宙の起源と構造』全体シンポジウム
2014年3月3,4日
於：富士ソフトアキバプラザ

筑波大学計算科学研究センター
吉江友照

ILDG/JLDG Overview
JLDG: Japan Lattice Data Gridfor Lattice QCD collaborations (in Japan) using
supercomputers installed at distant sites
to manage and share daily research data

ILDG: International Lattice Data Gridfor Lattice QCD community
to share and archive QCD configurations(fundamental data of QCD simulation) worldwide
nothing to be reported here, noting new movement

Supported by HPCI Strategic Program Field 5

DiGS
DiGS: distributed grid storage
developed at EPCC, EdinburghdCacheCCSGfarm
dCachedCache: distributed data
storage caching systemdeveloped by a joint collab.
of DESY and FNAL

gfarm: grid data farmdeveloped at
AIST and Tsukuba

dCache7 sites
1Gbps VPN(HEPnet-J/sc)
File systems arebounded intoa single FS
The FS looks
the same fromall sites
Gfarm: a grid-based file system software (O. Tatebe)JLDG team と budget• JLDG team: 22名（１１機関＋企業）– 建部,天笠,吉江(筑波),松古(KEK),外川,鎌野(大阪),石川 (広島),武田(金沢),實本(東京),青木,青山,山崎,三浦(名古屋),青木,福村(京都),渡邊,土井(理研),駒,住吉(沼津高専),滝脇(国立天文台),三上,金野(日立ソリューションズ東日本)• former collaborator– 宇川,佐藤,石井,浮田(筑波)• budget– 日本学術振興会先端研究拠点事業「計算素粒子物理学の国際研究ネットワークの形成」– 国立情報学研究所CSI 委託事業「グリッド・認証技術による大規模データ計算資源の連携基盤の構築」– 国立情報学研究所「e-science 研究分野の振興を支援するCSI 委託事業」の研究課題「計算素粒子物理学の高度データ共有基盤JLDG の構築」及び 「計算素粒子物理学のデータ共有基盤JLDGの高度化」– 新学術領域・素核宇宙融合「分野横断アルゴリズムと計算機シミュレーション」– 最先端研究基盤整備事業業「ｅ－サイエンス実現のためのシステム統合・連携ソフトウェ
アの高度利用促進」– HPCI戦略プログラム分野５「物質と宇宙の起源と構造」

Progress this year: Summary• New JLDG sites: Kyoto YITP, Riken (Wako)• New servers at existing JLDG sites

9 sites 5PB– HPCI Strategic File System : 2.1PB @ Tsukuba– Others: 0.3PB @ Tsukuba, 0.55PB @ KEK• Cooperation with HPCI Shared Storage– system construction completed– start operation soon (after user/admin manual ready)• JLDG System upgrade– Zabbix (monitoring system) : completed– VOMS upgrade : completed– Multiplexing of admin servers: on-goingCooperation of JLDG and HPCI-SS
SX9@U.Osaka

HPCIシステム利用研究課題 (hp120108) として推進

easy file operation
fast data copy
10PB 11.5PB
HPCI-SSgfarm

京＠AICS-RIKEN

HA8000@U.Tokyo

gfarmJLDG

HA-PACS@U.Tsukuba
SR16000@KEK

Needs:
Fundamental data generated on a supercomputer connected to HPCI-SS (e.g. K).
Do calculations of physical quantities on other supercomputers connected to
JLDG (e.g. HA-PACS@Tsukuba).
Solution:
Fast data copy between JLDG and HPCI-SS

System Overview

SINET L3
VPN

@Tsukuba,CCS
L3 SW

JLDG(7-sites)

172.25.xx.0/23 拠点VPN

JLDG

JLDG

file
serverfile
server

L3 SW

11000 port routing

MDS (JLDG)

VOMS

11001 port
CCS NW (global IP)

HA-PACS@U.Tsukuba

mount both filesystems, copy files
in multiple streams
601 port
MDS (HPCI)K ＠AICS-RIKEN

jldghpci gfpcopy
gfarm2fs

gfarm2fs

gsissh login

authentication withHPCI grid certificate(both for JLDG,HPCI)600 port

11.5PB 10.0PB
Tokyo(Kashiwa) Riken-AICS
HPCI Shared Storage

InternetPerformance
gfpcopy from HPCI-SS to JLDG
file size : 1GB#file per stream : 16

MB/sec

sec

100

30000

90

#stream 4, 8, 16, 32, 64, 128

25000

80
70

Performance reaches
90MB/sec(over 1Gbps network)

20000

60
50

15000

4030

スループット(MB/s)

10000

時間(sec)2050001000050100# stream150

JLDG Statistics• 11 research groups, 67 users• 1.6PB data, 57M files
TB

1600
1400
1200
1000
800
600
4002000!
JLDG Disk Usagenew items in ToDo list
Security
security guideline
organization of Univ./Institute

Data integrity
silent data corruption
learn from predecessors (CERN, LANL,….)
weak/strong point of JLDG
md5sum at every action