CCJ operation in 2012
Y. Ikeda, H. En’yo, T. Ichihara, Y. Watanabe, and S. Yokkaichi
1Overview

The RIKEN Computing Center in Japan (CCJ)1)
commenced operations in June 2000 as the largest oﬀsite computing center for the PHENIX2) experimentbeing conducted at the RHIC3) . Since then, the CCJ
has been providing numerous services as a regional
computing center in Asia. We have transferred several hundred TBs of raw data ﬁles and nDSTs, whichis the term for a type of summary data ﬁles at thePHENIX, from the RHIC Computing Facility (RCF)4)
to the CCJ. The transferred data are ﬁrst stored ina High Performance Storage System (HPSS)5) before
starting the analysis. The CCJ maintains suﬃcientcomputing power for simulation and data analysis byoperating a PC cluster running a PHENIX-compatible
environment.
A joint operation with the RIKEN Integrated Cluster of Clusters (RICC)6) was launched in July 2009.
Twenty PC nodes have been assigned to us for dedicated use, sharing the PHENIX computing environment.
Many analysis and simulation projects are being carried out at the CCJ, and these projects are listed on theweb page http://ccjsun.riken.go.jp/ccj/proposals/.
As of December 2012, CCJ has been contributed 30
published papers and more than 33 doctoral theses.
2Configuration

2.1 Calculation nodes
In our machine room 258/260 in the RIKEN mainbuilding, we have 28 PC nodesa) , and these nodes havebeen used for the analysis of the PHENIX nDST data.
These nodes are operated by the data-oriented analysis
scheme that carries out optimization using local disks7)8). The OS on the calculation nodes is Scientiﬁc Linux
5.39) , and the same OS works on the 20 nodes at theRICC. As a batch-queuing system, LSF 8.0.010) andCondor 7.4.211) were run on the CCJ and RICC nodes,respectively, as of Dec 2012.
Table 2 shows numbers of malfunctioned SATA orSAS disks in the HP servers (including NFS/AFS
servers described in the next section).
2.2 Data servers
Two data servers (HP ProLiant DL180 G6 with 20
TB SATA raw disks) are used to manage the RAIDa)

HP ProLiant DL180 G5 with dual Xeon E5430 (2.66 GHz,4 cores), 16 GB memory and 10 TB local SATA data disksfor each node, and HP ProLiant DL180 G6 with dual Xeon
X5650 (2.66 GHz, 6 cores), 24 GB/20 TB as above, for eachnode

Table 1. Limitation of number of job slots from LSF queuewith cluster node.

CCJ-hp1
CCJ-hp2
RICC
total

Nodes
18
10
1947

Cores
144
120
152
416

Threads
144
240
152
536

Jobs
180
200
144524

Table 2. Malfunctioned HDDs in 2012 and 2011
typeSATA
SASsize
1 TB
2 TB
146 GB300 GB

total
192
120
38
24

Malfunctioned
2012
2011
20
954110
1of the internal hard disks, which contain the user dataand nDST ﬁles of PHENIX. One old server (SUN FireV40 with 10 TB FC-RAID) was terminated in March
2012. The disks are not NFS-mounted on the calculation nodes to prevent performance degradation by thecongestion of processes and the network. These disks
can be accessed only by using the “rcpx” command,
which is the wrapper program of “rcp” developed at
CCJ and has an adjustable limit for the number ofprocesses on each server.
The DNS, NIS, NTP, and NFS servers are operated
on the server ccjnfs20b) with a 10-TB FC-RAID, whereusers’ home and work spaces are located. The homeand work spaces are formatted with VxFS 5.012) . The
backup of home spaces on ccjnfs20 is saved to anotherdisk server once a day and to HPSS once a week. The
backups on HPSS are stored for 3 weeks.
2.3 HPSS
Since Dec 2008, the HPSS servers and the tape robot
have been located in our machine room, although they
are owned and operated by RICC. The speciﬁcationsof the hardware used can be found in the literature13) .
The amount of data and the number of ﬁles archived inthe HPSS were approximately 1.7 PB and 2.1 millionﬁles, respectively, as of Dec 2012.
2.4 PHENIX software environment
Two PostgreSQL14) server nodes are operated for
the PHENIX database, whose data size was 86 GB
as of Dec 2012. The data are copied from The RCF
everyday and are made accessible to the users. Oneb)

SUN Enterprise M4000 with Solaris 10

Table 3. DST and raw data files in HPSS on Dec 31, 2012

Run
12
3
4
5
6
8
910
11
12
total

DSTsize [TB]cos
42,3,100
24 2,3,4,100
102,3,6142,3287 2,3,6,100
92
3,6,100
22
3
1063,7323142333736

Raw data
size [TB]
cos
3
3,205
36 1,3,5,205
46
100,205
11205
292
5,205
339
11,100128
1213000854

4

In the period Oct 2011–May 2012, a controller of theRAID disk connected to ccjnfs20 frequently displayed
the “link down” error and stopped the operation of
CCJ several times. Replacement of the RAID controller chassis and I/F card did not solve the problem.
There was no error recurrence after the ﬁrmware of theRAID controller v3.63G.65 was updated to v3.73k.01
in June 11, 2012.
5AFS server node is operated for the PHENIX AFS.
The size of the libraries for the PHENIX analysis setup
was 2.5TB as of Dec 2012. The libraries are also copiedfrom the RCF by afs everyday.
2.5 Network configuration
The topology of the network linking the CCJ, theRICC, and the RIKEN IT division was not changed in2012. This topology was shown in the paper entitled“CCJ operation in 2011”1) .
2.6

Uninterruptible power-supply system(UPS)
The power consumption of the CCJ system, excluding the HPSS, is about 25 kW, and the power is supplied through ﬁve UPSs (10.5 kVA each) as of Dec
2012, after two old UPSs were replaced by a new UPS
module in March 2012. For the HPSS, there is one 7.5kVA UPS for 100 V and three 10.5-kVA UPSs for 200
V purchased by CCJ. For the latter three, batteries
were changed by RICC in Oct 2012.
3

Data transfer from BNL

Data collected during the PHENIX experiment havebeen transferred from the RCF to the CCJ by gridFTP16) through SINET4 (maintained by NII17) ) witha 10 Gbps bandwidth. In 2012, 144 TB of nDSTsof the PHENIX Run-10AuAu (15 TB), Run-11pp (10TB), Run-11AuAu (116 TB) and Run-12pp (3 TB)
were sent from the RCF to the CCJ, and the data
were stored in the the HPSS, and 100 TB of this data
was moved to local disks on the HP calculation nodes.
Files are transferred by grid-FTP at a maximum speedof 280 MB/s.

Lost data

In Oct 9, 2012, a ﬁle system of a user’s home directory was damaged due to disk trouble and errors
in operation when the ccjnfs20 was booted after aplanned power outage. The data were recovered fromthe backup of Oct 5 (the day before the power outage).
615)

Issue with RAID controller

Air-conditioners of server room

In Jan 2012, six air-conditioners operated in machineroom (another one had malfunctioned). Three of thembroke down and the temperature increased up to 28◦
C on May (The normal temperature is about 22 ◦ Cthrough a year). Two of broken machines were restored
and the temperature returned on June. Another wasremoved and ﬁve have been operational as of Dec 2012.
References
1) S. Yokkaichi et al.: RIKEN Accel. Prog. Rep. 44, 228(2011).
2) http://www.phenix.bnl.gov/3) http://www.bnl.gov/rhic/4) https://www.racf.bnl.gov/5) http://www.hpss-collaboration.org/
6) http://accc.riken.jp/ricc/
7) T. Nakamura et al.: RIKEN Accel. Prog. Rep. 43, 167(2010)
8) J. Phys.: Conf. Ser. 331, 072025 (2011).
9) http://www.scientificlinux.org/
10) http://www-03.ibm.com/systems/technicalcomputing/platformcomputing/products/lsf/index.html
11) http://www.cs.wisc.edu/condor/description.html
12) Veritas file system (Symantec Corporation).
13) S. Yokkaichi et al.: RIKEN Accel. Prog. Rep. 42, 223(2009).
14) http://www.postgresql.org/
15) http://www.openafs.org/
16) http://www.globus.org/toolkit/docs/latest-stable/
gridftp/17) http://www.nii.ac.jp/