GridFTPを使用したPHENX 実験の
RIKEN-BNL データ転送

市原卓, 渡邊康, 四日市悟
理研, RIKEN-BNL Research Center
26 July 2006 at ICEPP

26 July 2006 T. Ichihara (RIKEN)RIKEN CCJ : Overview♦♦•

Scope of CCJ− RHIC スピン物理の解析センター− PHENIX シミュレーション− PHENIXのアジア地域計算センター
Size of CCJ• 年間取扱うデータ量: 300 TB /year• ディスク容量 : ~ 50 TB,• テープ容量: ~ 1200 TB capacity (HPSS)• CPU 性能 : 256 CPU (Xeon 3.05 GHz) +260 CPU (0.8-2GHz)
History• R&D for the CC-J started in April ‘98 at RBRC in BNL• Construction began in April ‘99• CCJ started operation in June 2000• cooperation with a new RIKEN Supercomputer (2004)
26 July 2006 T. Ichihara (RIKEN)History of the CCJ construction and operaton

1998
April

RBRC(BNL)

2000

1999

R&D for CC-J

April

2001

2002

April

April

April

Prototype of CPU farms
Data Duplication facility

RIKEN
Wako

1/3 scale

CC-J Working Group
formed (Oct. 1998)
CC-J review at BNL(Dec. 1998)
HPSS Software/Hardware
Installation (March 1999)(Supplementary Budget)

April

Oct

RIKEN was reistablished as an
Independent AdministrativeInstitution (Oct. 2003)

CC-J frontend at BNL

Phase 120042003

Phase 22/3 scale

HPSS server
updatedPhase 3
Tape Robot Ta pe drive
upgradeaddition

CC-J operation
at 1/3 scale(June. 2000)

2005

April

WAN Data Transfer

RIKEN new supercomputer

HPSS cache
updated

Redwood 50GB/vol
99 40B 200BG/vol.

CC-J Operation
PHENIX Exp. at RHIC

26 July 2006 T. Ichihara (RIKEN)

2006

cooperation with new
RIKEN super computer(Mar. 2004)

Tape drive
upgradeT10000500BG/vol120MB/sPrivate:192.168.30.xx

Airplane
30MB/s x2

PHENIX
Bufferbox x6
2 Gbpstotal

BNL-HPSS
CacheDisk

BNL-HPSS
TAPE

9940B
x2

duprikThe
Pacific
NYnodeESNET
2.4 Gbps
ccjexp
NAT

9940B
X8

CCJ-HPSS

SINET
10Gbps

CCJ
ccjbox x4

RCF/BNL
FW

Global130.199.6.xxx
U.Tokyo
node

SINET
1Gbps

2Gbps
RIKEN
FW

Overview of Data transfer from PHENIX to CCJ
J"2#4$ 0&=#;25 "788 F(''OH!"#$%&$'!"#$%&$(

C4D7%4;:E!"#$%&$)!"#$%&$*!"#$%&$+(C%!M$)

J3/J
C4D7%4;:E
FG>3H
192.168BNL

HPSS

G>3 IJ::-./
0012$!

34526788

130.199-2;6&5K.LL52MM/57#MN25 F-./H!"#$%&$,

G>3

PHENIX/RCF side
C. Mickey,
Y. Dantong(RCF) et al.

C54L3/J 9:#2;<:=!25:4#2;

RTT = 200ms
HOP=10>>?@AB*+/@( C%!M(C%!M>>?@AB++/@+/@+/@>>?

>>?@AB)>>?@AB((C%!M$)

C4D7%4;:E34526788
134.160

HPSS

J3/J
C4D7%4;>>? IJ:::E>JP N75QM
26 July 2006 T. Ichihara (RIKEN)

CCJ side:
Y. Watanabe
S. Yokkaichi
S. Kametani
T. Ichihara et al.

RIKEN-BNL GridFTP データ転送マシンの環境•Hardware•CCJBOX3•CPU: Dual EM64T Xeon 3.6 GHz, 8GB Memory•Motherbord: Supermicro X60HE-XG2•Gigabit NIC : HP NC7711 (Broadcom BCM95703)
S-ATA Raid5 (4TB) via 2GB FC Host Adapter•CCJBOX4•Dual Opteon 252 2.6GHz, 8GB Memory•Motherboad: Thnder K8W•Gigabit NIC : HP NC7711 (Broadcom BCM95703)•S-ATA Raid5(4TB) via 2GB FC Host Adapter•Software
OS: Scientific Linux 4.2 (x86_64) (RHEL4 compatible)
Kernel: 2.6.9-22.0.2.106.unsupportedsmp (CentosPlus) XFS,JFS,ReisrFS
File system : XFS (data area) , ext3 (OS part)
Grid environment
The Virtual Data Toolkit (VDT 1.2.4)(http://vdt.cs.wisc.edu/index.html) (University of Wisconsin-Madison)
The Virtual Data Toolkit (VDT) is an ensemble of grid middleware that can be easily
installed and configured.
必要な Grid tool一式が pacman-2.129 で簡単にインストールできる

Grid certification
Personal CA, Host CA: DOE Grid Certificate Service
http://www.doegrids.org/ Particle Physics Data Grid (PPDG)
Gridftp/etc/grid-security/grid-mapfile設定、 grid-proxy-init, globus-url-copy6 年前は CHEP-2000 presentation (Padova Italy)

WAN performance test

RIKEN (12 Mbps) - IMnet - APAN (70 Mbps) -startap- ESnet - BNL (in 2000)•Round Trip Time for RIKEN-BNL :170 ms• File transfer rate is 47 kB/s for 8 kB TCP widowsize (Solaris default)• Large TCP-window size is necessary to obtain high-transfer rate
• RFC1323 (TCP Extensions for high performance, May 1992) describes the method of usinglarge TCP window-size (> 64 KB)
TCP windowsize

FTP transfer rate

Theoretical limit(observed)

For 170 ms RTT

8 kB

41 kB/s

47 kB/s

16 kB

87 kB/s

94 kB/s

32 kB

163 kB/s

188 kB/s

64 kB

288 kB/s

376 kB/s

128 kB453 kB/s

752 kB/s

256 kB585 kB/s

1500 kB/s

512 kB641 kB/s

3010 kB/s

Large ftp performance (641 kB/s = 5 Mbps) was obtained for a single ftp
connection using a large TCP window-size (512 kB) over the pacific ocean(RTT = 170 ms)
26 July 2006 T. Ichihara (RIKEN)Network transfer test (Autumn in 2004 )• Network bandwidth (RCF-CCJ): 498 Mbps measured
with iperf• Bottleneck maybe OC12 (622 Mbps) BNL-(ESnet)-NewYork• It will be upgraded to OC48 (2.4 Gbps) until Jan/05• ~60 MB/s maybe confirmed to transfer by network

26 July 2006 T. Ichihara (RIKEN)

Transfer rate for single TCP stream

RFC1323 (TCP Extensions for high performance, May 1992) describes the method of using large TCP window-size (> 64 KB)

RTT: (RIKEN-BNL): 200ms
Hop between WAN Router :10
RIKEN WAN bandwidth: 1Gbps

パケットロス、ボトムネックのない理想的な場合
Throughput= WindowSize/RTT

Between RIKEN and BNL (20,21,24 July 2006)

iperf

W/RTT(ideal)

現実のネットワーク(RIKEN-BNL 間）
Single TCP streamではTCP window sizeを増やしていくと 256KB ぐらいまではリニアにスループットが増大するがそれ以上はあるところで飽和し、込み具合で飽和点は変動する
Single TCP 転送の限界

Data 1 20 July 2006
Data 2-4 24 July 2006

Transfer rate for parallel tcp stream
Between RIKEN and BNL (20 July 2006), Host to Host, iperf
TCP window size

Ganglia Monitor

26 July 2006 T. Ichihara (RIKEN)

TCP window size 32k 64k 128k 256k

512k 1M 2M 4M

8M 16M 1M-80P

Data Transfer (GridFTP) between BNL and RIKEN•Single GridFTP でのデータ転送•Tcp windows size ： 1-8 MB•TCP parallel 度 ： 80 tcp streams/1 GridFTP• Memory to Memory (60-90MB/s), Disk to Disk 40-60MB/s•実際のデータ転送• データ有りの時：２-４個のGridFTPを同実行•１Grid ftp : Disk to disk 40-50MB/s• 3 Grid ftp : Disk to disk 70-100MB/s•RUN 5 (2005) 260 TB•RUN 6 (2006) 310 TB

26 July 2006 T. Ichihara (RIKEN)RIKEN WAN traffic and transferred data
MRTG of RIKEN(Wako) WAN Router
2004

2005

2006

Green : inbound, Blue :outbound traffic

PHENIX experiment uses Grid totransfer 270 TB of data to Japan This seems to be the firsttime that a data transfer ofsuch magnitude wassustained over many weeks
in actual production• http://www.cerncourier.com/main/article/45/7/15

26 July 2006 T. Ichihara (RIKEN)

/etc/sysctl.conf のサンプル(初期値）（suggested by Dangong Yu @RCF BNL)

/etc/sysctl.conf
net.ipv4.tcp_rmem = 262144 1048576 8388608
# sets min/default/max TCP read buffer, default 4096 87380 174760

net.ipv4.tcp_wmem = 262144 1048576 8388608
# sets min/pressure/max TCP write buffer, default 4096 16384 131072

net.ipv4.tcp_mem = 262144 1048576 8388608
# sets min/pressure/max TCP buffer space, default 31744 32256 32768
### CORE settings (mostly for socket and UDP effect)net.core.rmem_max = 4194304
# maximum receive socket buffer size,default 131071

net.core.wmem_max = 4194304
# maximum send socket buffer size, default 131071

net.core.rmem_default = 1048576
# default receive socket buffer size, default 65535

net.core.wmem_default = 1048576
# default send socket buffer size, default 65535

net.core.optmem_max = 1048576
# maximum amount of option memory buffers, default 10240

net.core.netdev_max_backlog = 100000
# number of unprocessed input packets before kernel starts dropping them, default 300
26 July 2006 T. Ichihara (RIKEN)