Data oriented job submission scheme
for the PHENIX user analysis in CCJ
Tomoaki Nakamura, Hideto En’yo,
Takashi Ichihara, Yasushi Watanabe
and Satoshi Yokkaichi
RIKEN Nishina Center for Accelerator-Based Science
2010/10/20

Tomoaki Nakamura

1PHENIX Computing Center in Japan (CCJ)
CCJ history:
••
•
•

•
•
•
•
•

Disk storage
and PC nodeDeveloped since 1998 as regional
analysis center for PHENIX andother RIKEN related experiments.
Construction was started at 1999.
Operation was started from 2000.
23 publications and ~30 doctoral
dissertations have been completed
by using CCJ resources.

DST production (polarized p+p).
User level data analysis (p+p, A+A).
Detector analysis, calibrations.
MC simulations.
Detector R&D.

Other experiments:••

KEK-PS E325, E559, Belle.
J-PARC, E16, E26, g-2.

2010/10/20

Tape robot
Tomoaki Nakamura

2

PHENIX experiment at RHIC
Spin physics by polarized proton beam
arXiv:1009.0505 (2010)

QGP physics by heavy-ion beam
Phys. Rev. Lett. 104, 132301 (2010)2010/10/20

http://www.bnl.gov/bnlweb/pubaf/pr/newsroom.asp
Tomoaki Nakamura

3

CCJ specification
For the data production:•Network:•

GridFTP servers:•

Tape storage:

 10Gbps from BNL to RIKEN (SINET3)
 4+2 servers (directory connected withbuffer box in PHENIX counting room)
 HPSS 7.1
 TS3500 library 2.5PB (Maxim 8PB)

For the user analysis:•

PC cluster: ~252core 160core (Joint operation with RIKEN
Integrated Cluster of Clusters)
http://accc.riken.jp/ricc_e.html

2010/10/20•

Disk storage:•

Analysis environments:•

Maintaining equivalent resources
since the start of operation (2000).

 ~320TB

Tomoaki Nakamura Offline library Data bases4

Data transfer record by GridFTP
RHIC-Run5
FY2005•••

RHIC-Run6

Polarized p+p 200GeV
Raw data
Total 263TB

FY2006•••

RHIC-Run8

Polarized p+p 200GeV
Raw data
Total 308TB

FY2008•••

RHIC-Run9

Polarized p+p 200GeV
Raw data
Total 100TB

FY2009•Polarized p+p 200/500GeV•

Total 95TB

1.5PB of raw and reconstructed data was stored as of 2010.
2010/10/20

Tomoaki Nakamura

5Effective utilization for the user analysis
Growing data size
•Improvement of beam at RHIC•

Detector upgrade:•DAQ upgrade: Luminosity
 Polarization

 ~400Kch + Silicon VTX 4.4Mch.
 500 ~ 800MB/sec at RHIC-Run10.
 Will be upgraded factor 2 or more.

For the user data analysis•

Data distribution, storage to many nodes:•

NFS export:•

Disk cash: Not easy to avoid the I/O bound problem
by the conventional methods.
 Too slow due to the multiple access.

http://www.bnl.gov/cad/

Silicon VTX

 No way, when you scan all of the data set.

Preparing special nodes with large
capacity of local disk to store theread only data in advance.
2010/10/20

Tomoaki Nakamura

http://www.rarf.riken.go.jp/lab/radiation/
6

Disk configuration and I/O performance
HP ProLiant DL180 G5••

Chassis: 2U, 12HDD-bay
CPU: Quad-core  2••

Memory: 16GB (RDIM)
HDD:•

NIC: 1Gbps

2.66GHz Xeon E5430

146GB SAS  2 for the system (RAID1)1TB SATA  10 for the data storage
Uplinked by 10Gbps switch in rack.

System••

OS: Scientific Linux 5.3 x86_64
File system: ext3

XFS has no gain for the read only usage.

HDD configuration•RAID5 configuration:•Good performance without RAID:

Advantage is only for 1 process.

Total ~550MB/sec at 8 parallel jobs.

2010/10/20

Tomoaki Nakamura

7Data allocation
PC Cluster•••

Total 18 nodes
Total 144 CPU cores
Total 180TB for the data storage

Data sets in HDD•••

Every HDD has a part of each data
set segmented by run number.
User jobs can use all of CPU coreswithout time loss at data distribution.
~40GB work area is allocated ineach HDD for the non data analysis
jobs e.g. simulation.

2010/10/20

1TB
1TB
Run91TB
Run9Run81TB
Run9Run8Run6Run9Run8Run6Run5Run8Run6WorkRun5
area
Run6WorkRun5
area

Tomoaki Nakamura

WorkRun5area
Work area8

Job submission scheme
Job submission script•

Submit the user job to the appropriate node
automatically, which has the required data.

Batch Job scheduler (LSF 7.0)••

Fair-share among users.
Set the path to the data directory and work
area for environmental variables in the job.

Lock file and directory ACL••

Create at the pre-process for the exclusive
use of a physical disk.
Second job, which will use the data in
occupied disk by the first job, stay in queue
until the end of the first job.

User got to be able to analyze
all of the existing data in local
HDD (150TB) within 9 hours.
2010/10/20

Tomoaki Nakamura

LSF9Summary• 18 calculating nodes with 180 TB local disks were introduced for
effectively analyzing huge amounts of PHENIX data.

• A data-oriented batch queuing system was developed as a wrapper ofthe LSF system to increase the total computing throughput. Indeed,
the total throughput was improved by roughly 10 times as comparedto that in the existing clusters; CPU power and I/O performance are
increased threefold and tenfold, respectively. Thus, users can analyze
data of 150TB within 9 hours.

• We have experienced just 2 times disk failure out of 180 disks in 1.5
years operation. Therefore, we could conclude that this method ishighly effective for the I/O bound data analysis.

2010/10/20

Tomoaki Nakamura

10Next step
Preparing the same set of cluster with same data set,High availability:••

Reducing down time by the redundant data.
Balance of job distribution with respect to any kind of jobs.

Easy to expand for the growing data size:••

Without any other special software.
No special servers.

By low cost:•••

144CPU core, 2GB memory/core, 180TB disk for the local data storage.
~$150K as of 2009.
The capacity of a HDD is growing and its price is going down steadily.

2010/10/20

Tomoaki Nakamura

11

Backup2010/10/20

Tomoaki Nakamura

12

CCJ specification

2010/10/20

Tomoaki Nakamura

Riken Integrated Cluster of Clusters:
http://accc.riken.jp/ricc_e.html

13

CCJ history

2010/10/20

Tomoaki Nakamura

14

HPSS data amount2010/10/20

Tomoaki Nakamura

15

RIKEN Integrated Cluster of Clusters

http://accc.riken.jp/ricc_e.html

2010/10/20

Tomoaki Nakamura

16Luminosity at hadron collider

http://www.bnl.gov/cad/
2010/10/20

Tomoaki Nakamura

17

Integrated luminosity A+A at RHIC

2010/10/20

Tomoaki Nakamura

http://www.bnl.gov/cad/18

PHENIX DAQ upgrade

SuperDAQ(i.e. sDAQ)
DAQ2010

James Nagle: Next Decade Plans for the PHENIX ExperimentWorkshop on Saturation, the Color Glass Condensate and Glasma
2010/10/20

Tomoaki Nakamura

19