Star Asian Computing Center

Pusan National University
In-Kwon YOO

On Behalf of SACC Team

J.Lauret, D.Yu, W. Bett, J. Packard

E. Dart, E. Hjort

S.D.Lee, D.K.Kim, H.W.Kim

Star Asian Computing Center

In-Kwon YOO

Outline

ATHIC2008 @ Tsukuba

2/25

Star Asian Computing Center

1. Motivation

a. STAR Computing

JLauretDAQ

production

Raw Data

DST/event
MuDSTAnalysis• Our data processing path is- DAQ to DST- DST to MuDST : Size reduction is ~ 1 to 5

• STAR Plan
- FY 08: 370 Mevts- FY 09: 700 Mevts- FY 10:1200 Mevts- FY 11:1700 Mevts- FY 12:1700 Mevts
In-Kwon YOO

ATHIC2008 @ Tsukuba

3/25

Star Asian Computing Center

1. Motivation

a. STAR Computing

Data transfer should flow from Tier0 to Tier2

JLauretTier 01• TierBY DEFINITION:RHIC Computing Facility (RCF) at BNL• Services– Tier2
Acquisition, recording and processing of Raw data ; mai– Provide some persistent storage, source from Tier0n resource
MUST Allowof data–toTier2 only severa•– Would
host redistribution
transient datasets
requires• Services– Provide backup (additional copies) for Tier0l 100 GB– Mayrequiresa support
structureaccount– Longterm(permanent)archiving
and(userservingof all, ticket,
data response, grid
operations,…)•… Mostly
local
provideanalysis power for sbut NOTforsizedforgroups,
Monte Carlogeneration•– Functionality
pecifictopics
Allow
re-distributionof ANY data to minimally Tier1– Provides a significant added value to processing capabil• Functionality• MUST
ities ofprovide
STAR cycles (opportunistic) for at least sim– Production
reconstructionof most(all) Provide
Raw data,
provide– ulation
Half of theanalysis power
AND/ORsupportforrunningfor
embedding,simulation, …
resource
data analysis– Low requirementof Tier2
Grid operation support or co– Redistributionof data to– Has full support structure for users, code, …

In-Kwon YOO

mmon project

ATHIC2008 @ Tsukuba

4/25

Star Asian Computing Center

1. Motivation

a. STAR Computing

JLauret

6 main dedicated sites (STAR software fully installed)
• BNL

Tier0

• NERSC/PDSF

Tier1• WSU (Wayne State University)

Tier2• SPU (Sao Paulo U.)

Tier2• BHAM (Birmingham, England)

Tier2• UIC (University of Illinois, Chicago)

Tier2

Incoming• Prague• KISTI

In-Kwon YOO

Tier2
ATHIC2008 @ Tsukuba

Tier1

5/25

Star Asian Computing CenterPDSF
Berkeley LAB

Wayne State UniversityBrookhaven National LabFermi Lab

Universidadede São PauloUniversity of Birmingham

Star Asian Computing CenterAmazon.com

MIT X-grid

NPI, Czech Rep
ublic

SunGrid

STAR is also outreaching other grid resources & projects

Interoperability / outreach

Virtualization

VDT extension SRM / DPM / EGEEStar Asian Computing Center

1. Motivation

a. STAR Computing

JLauret
FY05

FY06

FY07

FY08

FY09

FY10

FY11

FY12

Real Data Volume (TB)

600570560560870

1720
1720

3000

4160
4160

4160

Reco CPU (KSI2K)

660

314

462
462

1532

3296
3296

5891

6960
6960

6960

Analys CPU (KSI2K)

360

157

210

730

1648

2805

3480

3480

Dist. Disk (TB)

150

71

105

365

749

1275

1740

1740

Cent. Disk (TB)

30

14

21

73

150

255

348

348

Annual Tape Volume (TB)

720

684

672

1044

2064

3600

4992

4992

Tape bandwidth (MB/sec)

200

200

200

200

500

800

1000

1000

WAN bandwidth (Mb/sec)

160

384

528

640

1760

2944

3800

3800

Simulation CPU (KSI2K)

153

71

101

339

742

1304

1566

1566

Simulation Data Volume (TB)120114112174344600

832832

STAR Requirement

In-Kwon YOO

ATHIC2008 @ Tsukuba

8/25

Star Asian Computing Center

1. Motivation

a. STAR Computing

JLauret

In-Kwon YOO

ATHIC2008 @ Tsukuba

9/25

Star Asian Computing Center

1. Motivation

a. STAR Computing

K$

JLauret56

78Observation: Cost seem to go into•
•

9

CPU
BUT this folds distributed disks(1/2 cost)
Reduced used of centralized disk is nonetheless third in cost

In-Kwon YOO

1011

12

Cost is clearly• Storage (~ ½)• CPU (1/3rd)• HPSS & LAN is second

ATHIC2008 @ Tsukuba

10/25

Star Asian Computing Center

1. Motivationb. KISTI Resources

Cluster system

SDLee, HWKim
Cluster system

Item
Manufacturer & Model

Phase 1Phase 2

SUN C48

SUN Fusion

Architecture

In-Kwon YOO

Cluster

Processor

AMD Opteron 2GHz
(Barcelona)Intel Xeon 3.3GHz+(Gainestown)

Operating System

Cent OSCent OS

Nodes

188

2,688

CPU cores

3,008(16/node)

21,504(8/node)Rpeak

24TFlops

286TFlops

Memory6TB

64.5TB

Disk storage

207TB

1PB

Tape storage

422TB

2PB

Interconnection network

Infiniband 4X DDRInfiniband 4X DDR

Cooling

Chilled water coolingChilled water cooling

Delivery dateJan, 2008

2Q, 2009

ATHIC2008 @ Tsukuba

11/25

Star Asian Computing Center

1. Motivationb. KISTI Resources

SDLee, HWKim

SMP system

SMP system

Item
Manufacturer & Model

Phase 1

Phase 2

IBM p595

IBM p6H

Architecture

SMP

Processor

POWER5+ 2.3GHz

POWER6 5GHz+

Operating system

AIX 5.3

AIX 5.3+

Nodes

1024

CPU cores

640(64/node)

1,536(64/node)

Rpeak

5.9TFlops

30.7TFlops

Memory

2.6TB

9.2TB

Disk storage

63TB

273TB

Tape storage

In-Kwon YOO-Interconnection network

HPS

Infiniband 4X DDRCoolingAir-coolingAir-cooling

Delivery dateSept, 2007

1Q, 2009

ATHIC2008 @ Tsukuba

12/25

Star Asian Computing Center

1. Motivationb. KISTI Resources

SDLee, HWKim• KREONETIn-Kwon YOO

ATHIC2008 @ Tsukuba

13/25

Star Asian Computing Center

1. Motivationb. KISTI Resources

SDLee, HWKim

In-Kwon YOO

ATHIC2008 @ Tsukuba

14/25

Star Asian Computing Center

In-Kwon YOO

2. Projecta. STAR Asian HUB

ATHIC2008 @ Tsukuba

15/25

Star Asian Computing Center

2. Projecta. STAR Asian HUB• Computing Infrastructure with massive data from STA
R– Frontier Research– Maximum Use of IT resources in Korea• Data Transfer• Cluster Computing with Supercomputer• Mass Storage• Korean Institute for Science and Technology Informati
ons (KISTI @ Daejoen)– Korean HUB for GLORIAD + KREONET– Super Computing Resources– Mass Storage Management Asian Supercomputing HUB :- BNL – NERSC – KISTI – SSC etc.

In-Kwon YOO

ATHIC2008 @ Tsukuba

16/25

Star Asian Computing Center

2. Projectb. Working Group• PNU– IKYoo et al.
• KISTI– SDLee, DKKim, HWKim• BNL (STAR)– JLauret, DYu, Wbett, Edart, JPackard• SSC + Tsinghua Univ. ?
– ZXiao et al. ?
In-Kwon YOO

ATHIC2008 @ Tsukuba

17/25

Star Asian Computing Center

2. Projectc. KISTI STAR Computing

SDLee, HWKim• Configuration of a testbed(16 cores) under way• Eventually SUN cluster 1st shipment (~3,000 cores) will be ded
icated to STAR early next year!
• Initial Network status : below 1 Mbps (over 10Gbps line)• Network Optimization between KISTI and BNL : since 2008-07• Target Throughput : over 2 Gbps (over LP)• KISTI’s effort- Installed 10Gbps NIC equipped server in Seattle- Local optimization
In-Kwon YOO

ATHIC2008 @ Tsukuba

18/25

Star Asian Computing Center

2. Projectc. KISTI-BNL Networking

Dantong YU

In-Kwon YOO

ATHIC2008 @ Tsukuba

19

Star Asian Computing Center

2. Projectc. KISTI-BNL Networking

Dantong YU• Network Tuning improved transfer from BNL to KISTI– Identified bottleneck with Kreonet2 peering point with Esnet. 1Gpbs=> 10Gbps– Network and TCP stack was tuned at KISTI hosts.

After Tuning

Before Tuning

In-Kwon YOO

ATHIC2008 @ Tsukuba

20/25

Star Asian Computing Center

2. Projectc. KISTI-BNL Networking

Dantong YU

There are two kinds ofnetwork events that require some further examination.

1.Delayed ramp up.
2.Sudden Drop in Bandwidth.

In-Kwon YOO

ATHIC2008 @ Tsukuba

21/25

Star Asian Computing Center

2. Projectc. KISTI-BNL NetworkingDantong YU• Performance between BNL and KISTI :not symmetrical.
• A bottleneck from KISTI back to BNL.

– Packet drops at BNL receiving host. (will be replaced).
– Old Data Transfer Nodes at BNL: being replaced– Findings being corrected:• High performance TCP parameters– Findings are still under investigation• TCP slow ramp up, and performance sudden drop.

• test/tune GridFtp tools : in next 4 weeks

In-Kwon YOO

ATHIC2008 @ Tsukuba

22/25

Star Asian Computing Center

2. Projectc. KISTI-BNL Networking

Dantong YU• Replace the old data transfer nodes– 1 Gbps per node, with expansion slots for 10Gbps.
– Large local disk for intermediate cache.

• Deploy OSG BestMan for these nodes.
• RACF firewall will be rearchitectured.
• Data transfer performance should be only
limited by the local disk buffer at both ends.
In-Kwon YOO

ATHIC2008 @ Tsukuba

23/25

Star Asian Computing Center

2. Projectd. To-Do List• KISTI needs to finish the testbed preparation• STAR Software should be installed and tested• KISTI net people need to set up lightpath between KI
STI and BNL eventually• BNL net people need to set up a host for the end-to-end test and measure the throughput• We need to complete “a real mass data-transfer” from BNL to KISTI sooner or later!
• Start Production test at KISTI
In-Kwon YOO

ATHIC2008 @ Tsukuba

24/25

Star Asian Computing Center

3. OutlookHACC• STAR Asian Computing Center (SACC) (2008 – 2011)– Experimental Data from International Facility• Computational Infrastructure for LHC / Galaxity– Asian Hub for International Coworking– Frontier Research• Heavy ion Analysis Computing Center (HACC) (2011-)– Extend to Other project (HIM) ?
– Extend to ATHIC ?
– Dedicated Resources for Heavy ion Analysis Computing

In-Kwon YOO

ATHIC2008 @ Tsukuba

25/25

Star Asian Computing Center

Appendix

PHENIX

Dantong YU

Star Asian Computing Center

Appendix

PHENIX

Dantong YU

27

Star Asian Computing Center

Appendix

PHENIX

Dantong YU

 BNL: Multiple 3.0Ghz dual CPU nodes with I
ntel copper gigabit network. Local drives connected by Raid Controller. There are PHENI
X on-line hosts.
 CCJ Site: 8 dual-core AMD Opteron based hosts, each with multiple Tera bytes SATA drives connected with a RAID controller. Each one has one gigabit broadcom network card.
 The LAN on both ends are 10Gbps.
 The data transfer tool is GridFtp.
28In-Kwon
YOO

ATHIC2008 @ Tsukuba

Star Asian Computing CenterAppendix

PHENIXPHENIX to CCJ Data Transfer Test Dantong
at theYU
beginning of 2008 (Mega Byte/second)
We can transfer up to 340M
byte/second from BNL to C
CJ, it hits the BNL firewall limitation.

In-Kwon YOO

ATHIC2008 @ Tsukuba

29

Star Asian Computing Center

Appendix

PHENIX

Dantong YU

2005 RHIC run ended on Jun
e 24, Above shows the last dayof RHIC Run.
Total data transfer to CCJ (Computer Center in Japan) is 260
TB (polarized p+p raw data)
100% data transferred via WA
N, Tool used here: GridFtp. No
747 involved.
Average Data Rate: 60~90MB/second, Peak Performance: 100 Mbytes/second recorded inGanglia Plot! About 5TB/day!
30

Star Asian Computing Center

Appendix

PHENIX

Dantong YU

In-Kwon YOO
31

ATHIC2008 @ Tsukuba