RIKEN Accel. Prog. Rep. 43 (2010)

CCJ Operation in 2008-2009
T. Nakamura, H. En’yo, Y. Goto, T. Ichihara, Y. Watanabe and S. YokkaichiOverview
The RIKEN Computing Center in Japan (CCJ)1)
has been developed since April 1998 for analyzing thehuge amounts of data collected in the PHENIX experiment at RHIC. Thus far, CCJ has been providing
numerous services as Asia’s regional computing center. For instance, CCJ maintains sufficient computing power by the PC cluster operated by ScientificLinux2) for simulation and individual data analysis.
The collected data are transferred through SINET3
with a 10 Gbps bandwidth, maintained by NII3) ,from Brookheaven National Laboratory (BNL) by using GridFTP4) . The transferred data are once stored
in High Performance Storage System (HPSS)5) before starting the analysis. This HPSS is one of thejoint operations with the RIKEN Integrated Clusterof Clusters (RICC) project. One of the most successful achievements of the CCJ operation up to now is
the completion of more than 30 doctoral dissertations
with the analysis results obtained using the computing
resources at CCJ.
A summary of the basic configuration of the CCJsystem has been published elsewhere6) . Since last year,
several major hardware-level upgrades have been carried out. The center network switch has been replacedwith Catalyst4900M, which has eight 10 Gbps ports, inJuly 2009. The operation of the HPSS was started witha new version 7.1 in January 2010. In February 2010,
four sets of UPS were replaced by SANUPS ADS series (Sanyo Denki). The server (ccjnfs11) for Network
File system (NFS), which managed a 6.8 TB FC-FC
RAID box and a 8.9 TB FC-SATA RAID box, wasended the service to be upgraded in March 2010. Recently, effective use of the existing computing resources
has become difficult owing to the rapid increase in thePHENIX data size. In this report, the details of several
upgrades and developments made to the CCJ systemin the 2008-2009 period are included.
2 PC cluster specification
Until early 2008, CCJ was operated by approximately 190 PC nodes, 166 nodes of which were used
as calculation nodes for the simulation and PHENIX
data analysis. In June 2008, 112 nodes were eliminated. The remaining 36 and 18 dual CPU nodes haveIntel Pentium III (1.4 GHz) CPU and Intel Pentium 4(2.0 GHz) CPU, respectively. Each calculation node
has 1 GB memories.
In February 2009, 18 PC nodes (HP ProLiant DL180G5) were newly introduced to compensate for the low
total computing power. Each node has dual CPUs
(Quad-Core Intel Xeon E5430 2.66 GHz) and 16 GBmemories. These nodes have twelve 3.5 inch bays ofTotal speed [MB/s]

1600
1 disk500

400

8 parallel disks
RAID5 (HW)RAID5 (SW)

30020010000123456789
Number of parallel jobs

Fig. 1. Average total speed for reading 1 GB files in one
disk (square), 8 parallel disks (filled circle), hardware
RAID5 (open circle), and software RAID5 (triangle) asa function of number of parallel jobs.

HDD for each chassis. Two 146 GB SAS disks aremounted for the operating system and used by mirroring to reduce the down time originating from the troubles on the HDD. Initially, eight 1 TB SATA disks weremounted for local data storage. In December 2009, two
additional 1 TB SATA disks were added to each node.
Currently, local disks with a total capacity of 180 TB
are available for the data storage. This specificationis the key feature for I/O bound jobs, as will be described in the next section. Each node has a 1 Gbps
network interface card, and all of the nodes are connected to a network switch (Dell PowerConnect 6224)
mounted on the same rack. This network switch is up
linked to the center network switch at CCJ (Catalyst4900M) via a 10 Gbps connection. This new cluster
has the capability to process a typical PHENIX analysis job approximately three times faster than the oldPC nodes in a CPU core.
In October 2009, 20 PC nodes were setup as a cluster
at RICC for the exclusive use of the CCJ users. Eachnode has dual CPUs (Quad-Core Intel Xeon X5570
2.93 GHz) and 12 GB memories and is operated by
Scientific Linux 4.4 on VMware ESXi7) so that it isdedicate to the PHENIX analysis environment. Condor8) system is available as a batch job scheduler forthis cluster.
Thus, a total of 412 CPU cores are presently available as calculation nodes for the CCJ users.
3 Development of job submission scheme
The PHENIX data stored once in the HPSS will betransferred to several RAID boxes for the analysis. Although users can access the data in the RAID boxes
through the NFS servers, multiple access from numer-1

Table 1. Summary of nDSTs in local disk.
DatasetRun 9 p + p 200 GeV
Run 9 p + p 500 GeV
Run 8 p + p 200 GeV
Run 6 p + p 200 GeV
Run 5 p + p 200 GeV

nDST type
All type
All type
All type
w/o detector
w/o detector

Data amount
65.4 TB
31.2 TB
21.2 TB
14.6 TB
9.9 TBous calculation nodes at the same time is not possible
because of the decrease in the I/O speed. Therefore,
users must transfer the data from the RAID boxes tothe calculation node for each batch job. Since the sizeof PHENIX data is growing steadily, such data transfer
becomes a bottleneck in data analysis. This problem iseliminated with the use of the newly introduced calculating nodes, which have large capacity local disks forstoring the data a priori (see previous section). However, since these new calculating nodes have multi-coreCPUs, which is predominant in the market, data analysis remains an I/O bound type job. Therefore, itis necessary to optimize the composition of the local
HDD. We performed a benchmark test to evaluate theI/O performance for the new cluster. Figure 1 shows
the average total speed for reading 1 GB files as a function of the number of parallel jobs. Originally, each
HDD shows the I/O performance of 100 MB/s. However, the use of a multi-HDD is not advantageous with
the RAID configuration, as shown by the open circles
and triangles in Fig. 1. Since the RAID configuration
gives us a single name space, maintaining the data location becomes easy. Nevertheless, we do not chose
this configuration to maximize the I/O performance.
In October 2009, approximately 96 TB Run 9 p + p
data, so-called nano-DST, were transferred from BNL
as soon as data reconstruction was completed at theRHIC Computing Facility (RCF)9) . They were stored
in local disks along with the previously stored data.
Table 1 shows a summary of the dataset in the local
disks accessible to users by the batch queuing system.
In the calculation nodes, users can process their own
analysis code via the batch queuing system (LSF version 7.010) ) in the CCJ cluster. We have added somesoftware modules to enable the user specify the nDSTs distributed to the local disks during job submission. Figure 2 shows a brief flowchart of the dataoriented job submission scheme. Since all the subsetsof the PHENIX data have identical run numbers, themodule first obtains the location of the nDST by theuser-specified run number from a database. Then, themodule submits the user jobs to the appropriate nodeby the LSF. To avoid multiple access for a local diskfrom several jobs, the module sets a lock file for exclusive access and grants permission only to the user by
the Access Control List (ACL) method. As a result,each job dispatched to a calculating node exclusively
handles a local disk. The advantage of this method
is that the I/O performance is enhanced, as shown by2            0+123+3*45
  "#         ,%'()*&++-*./.
$%&%'()*&+   !    Fig. 2. Data-oriented job submission scheme.

the filled circles in Fig. 1. Further, a temporary work
area for the job is assigned to the same disk with thedata location. Therefore, this scheme is effective for
eliminating the I/O bound problem in case of genericjobs as well e.g. simulation. Each job are able to savethe processing time on the data transfer approximately
10 times as compared to the typical jobs in the preexisting calculating nodes. An instruction is provided inthis URL11) on how to access to the data stored in thelocal disks.
4 Summary
18 calculating nodes with 180 TB local disks wereintroduced for effectively analyzing huge amounts of
PHENIX data. A data-oriented batch queuing system
was developed as a wrapper of the LSF system to increase the total computing throughput. Indeed, thetotal throughput was improved by roughly 10 times ascompared to that in the existing clusters; CPU powerand I/O performance are increased threefold and tenfold, respectively. Thus, users can analyze data of several tens of terabytes within a few hours. This is oneof the most significant developments made to the CCJ
operation in 2009.
References
1) http://ccjsun.riken.jp/ccj/
2) http://www.scientificlinux.org/
3) http://www.nii.ac.jp/
4) http://www.globus.org/grid software/data/gridftp.php5) http://www.hpss-collaboration.org/
6) S. Yokkaichi et al., RIKEN Accel. Prog. Rep. 42, 223(2009).
7) http://www.vmware.com/products/esxi/8) http://www.cs.wisc.edu/condor/description.html
9) http://www.rhic.bnl.gov/RCF/
10) http://www.platform.com/products/LSF/11) http://ccjsun.riken.go.jp/ccj/doc/LSF/lsf-wrapper.html