Ⅱ-9. Instrumentation

RIKEN Accel. Prog. Rep. 52 (2019)

CCJ operations in 2018
S. Yokkaichi,∗1 H. En’yo,∗1 T. Ichihara,∗1 W. Nakai,∗1 and Y. Watanabe∗1Overview
The RIKEN Computing Center in Japan (CCJ)1)
commenced operations in June 2000 as the largest oﬀsite computing center for the PHENIX2) experiment being conducted at RHIC. Since then, CCJ has been providing numerous services as a regional computing centerin Asia. We have transferred several hundred TBs of raw
data ﬁles and nDSTa) ﬁles from the RHIC Computing
Facility (RCF)3) to CCJ.
Many analysis and simulation projects are being conducted at CCJ, which are listed on the web page
http://ccjsun.riken.go.jp/ccj/proposals/. As ofDecember 2018, CCJ has contributed to 43 publishedpapers and 43 doctoral theses.
Computing hardware and software
The network conﬁguration and the computing hardware (nodes) and software (OS, batch queuing systems,database engine, etc.) are almost same as described in
the previous APR.1) We have two login servers, one main
server (users’ home directory, NIS, DNS and NTP), andtwo disk servers whose disk sizes are 13 and 26 TB. The
main server has an external SAS RAID (21 TB) for thehome and work regions of users, and system usage. Also,
the server has a RAID with built-in disks (13 TB) which
can be used temporarily by users and the system.
We operate 26 computing nodes, of which 16 nodes
were purchased in Mar. 2009 and 10 nodes were purchased in Mar. 2011. Thus, in total, 368 (= 8 × 16nodes + 24 × 10 nodes) jobs can be processed simultaneously by these computing nodes using a batch queuingsystem LSF 9.1.3.4) Table 1 lists the number of malfunctioning SATA or SAS disks in the HP servers, namely,
computing nodes and NFS/AFS servers.
One database (postgreSQL5) ) server and one AFS6)
server are operated in order to share the PHENIX computing environment. It should be noted that only theSL57) environment is shared by the computing nodes,

Table 1. Number of malfunctioning HDDs in HP servers
during 2011–2018.
Type(Size)
SATA(1 TB)
SATA(2 TB)
SATA(4 TB)
SAS(146 GB)
SAS(300 GB)∗1a)total192
120
1038
26

2018 ’17 ’16 ’15 ’14 ’13 ’12 ’1116 18
8 14 11 16 2092 102 10025403153201101011001

RIKEN Nishina Centerterm for a type of summary data files in PHENIX

which have approximately 0.9 TB of library ﬁles. We
have two data-transfer servers, on which the grid environment8) is installed for data transfer to/from RCF.
Two new data-transfer servers will be in operation inearly 2019 and replace two servers being currently used.
Data transfer of the order of 100 TB from J-PARC andBNL will be performed in the future. In addition, weoperate a dedicated server for the RHICf group9) andtwo servers for the J-PARC E16 group,10) in order tokeep their dedicated compilation and library environments along with some data.
A 10-KVA UPS (supplied by NTT Facilities) was discarded without battery exchange owing to its high costof exchange. The power wiring was relocated in Nov.
2018. Subsequently, a total power in our UPS system is30 KVA and a typical average load factor is 51% without jobs and 59% with maximum number of jobs in thecalculation nodes.
Joint operation with ACCC/HOKUSAI
CCJ and the RIKEN Integrated Cluster of Clusters(RICC) have been jointly operated since July 2009. In
April 2015, a new system named “HOKUSAI Greatwave”
was launched by RIKEN ACCC11) and the joint operation with CCJ continued, including a new hierarchicalarchive system in which approximately 900 TB of CCJ
data are stored. Subsequently, the “HOKUSAI BigWaterFall” IA cluster, which has 840 nodes/33600 CPU
cores, was launched in 2017 by ACCC. Then a dedicated usage of legacy RICC 10 nodes by CCJ was ended
and the direct connection between CCJ and HOKUSAIwith the two 10G Ethernet links was also ended. CCJ
has not started to use the cluster because it does notsupport the NFS to share the computing environmentof PHENIX unlike RICC. Instead, we are in preparation to use one of the “container technologies” (such as“Docker” 12) ) to share the environment.
References

1) S. Yokkaichi et al., RIKEN Accel. Prog. Rep. 51, 188 (2018).
2) http://www.phenix.bnl.gov/ .
3) https://www.racf.bnl.gov/ .
4) https://www.ibm.com/support/knowledgecenter/en/SSETD4_9.1.3/lsf_welcome.html .
5) http://www.postgresql.org/ .
6) http://www.openafs.org/ .
7) http://www.scientificlinux.org/ .
8) http://www.globus.org/toolkit/docs/
latest-stable/gridftp/ .
9) Y. Itow et al., arXiv:1409.4860 (Proposal).
10) S. Yokkaichi, RIKEN Accel. Prog. Rep., in this report.
11) http://accc.riken.jp/
12) https://www.docker.com/- 160 -