Ⅱ-9. Instrumentation

RIKEN Accel. Prog. Rep. 47 (2014)

CCJ operation in 2013
Y. Ikeda,∗1 H. En’yo,∗1 T. Ichihara,∗1 Y. Watanabe,∗1 and S. Yokkaichi∗1
1

Table 1. Limitation of number of job slots from LSF queuewith cluster node.

Overview

The RIKEN Computing Center in Japan (CCJ)1)
commenced operations in June 2000 as the largest oﬀsite computing center for the PHENIX2) experimentbeing conducted at the RHIC3) . Since then, the CCJ
has been providing numerous services as a regional
computing center in Asia. We have transferred several
hundred TBs of raw data ﬁles and nano data summary tape (nDST) ﬁles, which is a term for a typeof summary data ﬁles at PHENIX, from the RHIC
Computing Facility (RCF)4) to the CCJ. The transferred data ﬁles are ﬁrst stored in a High Performance
Storage System (HPSS)5) before performing the analysis. The CCJ maintains suﬃcient computing power for
simulation and data analysis by operating a PC cluster
running a PHENIX-compatible environment.
A joint operation with the RIKEN Integrated Cluster of Clusters (RICC)6) was launched in July 2009.
Twenty PC nodes have been assigned to us for dedicated use, sharing the PHENIX computing environment.
Many analysis and simulation projects are being carried out at the CCJ, and these projects are listed on theweb page http://ccjsun.riken.go.jp/ccj/proposals/.
As of December 2013, CCJ has been contributed 31
published papers and more than 33 doctoral theses.
2Configuration

2.1 Calculation nodes
In our machine room 258/260 in the RIKEN mainbuilding, we have 28 PC nodesa) , and these nodes havebeen used for the analysis of the PHENIX nDST data.
Table 1 lists the numbers of job slots, CPU threads,and CPU cores, and the number of nodes (there isno change in 2013). These nodes are operated using
a data-oriented analysis scheme that carries out optimization using local disks7)8) . The OS on the calculation nodes is Scientiﬁc Linux (SL) 5.39) , and the same
OS works on the 20 nodes at the RICC. As a batchqueuing system, LSF 8.0.010) and Condor 7.4.211) wererun on the CCJ and RICC nodes, respectively, as ofDec 2013.
Table 2 lists the numbers of malfunctioned SATAor SAS disks in the HP servers (including NFS/AFS
servers described in the next section).
∗1a)

RIKEN Nishina Center
HP ProLiant DL180 G5 with dual Xeon E5430 (2.66 GHz,4 cores), 16 GB memory and 10 TB local SATA data disks
for each node, and HP ProLiant DL180 G6 with dual Xeon
X5650 (2.66 GHz, 6 cores), 24 GB/20 TB as above, for eachnode- 237 -CCJ-hp1
CCJ-hp2
RICC
Total

Nodes
18
101947Cores
144
120152
416

Threads
144
240152536

Jobs
180200144524

Table 2. Malfunctioned HDDs in 2011, 2012, and 2013

YypeSATASASSize
1 TB
2 TB
146 GB
300 GBTotal
192
120
38
24Malfunctioned2013
2012
1620
250
10020119411

2.2 Data servers
Two data servers (HP ProLiant DL180 G6 with 20
TB SATA raw disks) are used to manage the RAID
framework of the internal hard disks, which containthe user data and nDST ﬁles of PHENIX. The disks
are not NFS-mounted on the calculation nodes to prevent performance degradation by process and network
congestion. These disks can be accessed only using
the “rcpx” command, which is the wrapper programof “rcp” developed at CCJ, and it has an adjustable
limit for the number of processes on each server.
The Domain Name System, Network Information
System, Network Time Protocol, and Network File
System servers are operated on the server ccjnfs20b)with a 10-TB FC-RAID, where the users’ home andwork spaces are located. The home and work spaces
are formatted with VxFS 5.012) . The backup of homespaces on ccjnfs20 is saved to another disk server oncea day and to HPSS once a week. The backups on HPSS
are stored for three weeks.
2.3 HPSS
Since Dec 2008, the HPSS servers and the tape robothave been located in our machine room, although they
are owned and operated by RICC. The speciﬁcationsof the hardware used can be found in the literature13) .
The amount of data and the number of ﬁles archived inthe HPSS were approximately 1.7 PB and 2.1 million
ﬁles, respectively, as of Dec 2013. Table 3 lists the ﬁlesand the current class of service (COS) in the HPSS.
No new ﬁle has been added in 2013.
b)

SUN Enterprise M4000 with Solaris 10Ⅱ-9. Instrumentation

RIKEN Accel. Prog. Rep. 47 (2014)

Table 3. DST and raw data files in HPSS on Dec 31, 2013

Run12345689101112Total

3DST
Size [TB]
COS
42,3,100
24 2,3,4,100
102,3,6142,3
287 2,3,6,100
92
3,6,100
22
3
106
3,732
3142
333736

Raw data
Size [TB]
COS
33,205
36 1,3,5,205
46
100,205
11205
292
5,205
339
11,1001281213000854

Data transfer from BNL and the PHENIX
software environment

Data collected during the PHENIX experiment wastransferred from the RCF to the CCJ using grid-FTP16)
through the science information nework (SINET)
4 (maintained by NII17) ) with a 10 Gbps bandwidth.
The data which transfered from BNL is moved to local disks on the HP calculation nodes and the HPSS.
The ﬁles are transferred using grid-FTP at a maximum speed of about 300 MB/s. Two PostgreSQL14)
server nodes are operated for the PHENIX database,whose data size was 285 GB as of Dec 2013. The data
are copied from the RCF everyday and made accessible to the users. One AFS15) server node is operatedfor the PHENIX AFS. The size of the libraries for thePHENIX analysis setup was 1.7 TB as of Dec 2013.
The libraries are also copied from the RCF by AFSeveryday.
3.1

Uninterruptible power-supply system(UPS)
The power consumption of the CCJ system, excluding the HPSS, is about 25 kW, and the power is supplied through ﬁve UPSs (10.5 kVA each) as of Dec
2013. For the HPSS, there is one 7.5-kVA UPS for 100
V and three 10.5-kVA UPSs for 200 V purchased byCCJ. The batteries of the three UPSs expire in 2014
(One have expired in 2013).
3.2 Login server upgrade
CCJ had two login servers, ccjsun (HP ProliantDL145) with SL 5.3 and ccjgw (Supermicro 5011E)
with CentOS 3, in 2012. The ccjgw server was upgraded to the HP Proliant DL145 with SL 5.3 in 2013.
References

1) S. Yokkaichi et al.: RIKEN Accel. Prog. Rep. 44, p228
(2011).
2) http://www.phenix.bnl.gov/- 238 -3) http://www.bnl.gov/rhic/4) https://www.racf.bnl.gov/5) http://www.hpss-collaboration.org/
6) http://accc.riken.jp/ricc/
7) T. Nakamura et al.: RIKEN Accel. Prog. Rep. 43, p167
(2010)
8) J. Phys.: Conf. Ser. 331, 072025 (2011).
9) http://www.scientificlinux.org/
10) http://www-03.ibm.com/systems/technicalcomputing/platformcomputing/products/lsf/index.html
11) http://www.cs.wisc.edu/condor/description.html
12) Veritas file system (Symantec Corporation).
13) S. Yokkaichi et al.: RIKEN Accel. Prog. Rep. 42, p223
(2009).
14) http://www.postgresql.org/
15) http://www.openafs.org/
16) http://www.globus.org/toolkit/docs/latest-stable/
gridftp/17) http://www.nii.ac.jp/