Joint Center for Advanced High Performance Computing (JCAHPC)
Summary Joint Center for Advanced High Performance Computing (JCAHPC) decided to introduce its new supercomputer system (Post T2K system) with 25 PFLOPS peak performance. The new system is named “Oakforest-PACS” with 8,208 of next-generation manycore processors developed by Intel Corporation, and is planned to start its full operation on December Ist, 2016. The Oakforest-PACS will be the fastest supercomputer system in Japan at that time. The Oakforest-PACS will be installed at the Kashiwa Research Complex II building in the Kashiwa-no-Ha (Oakleaf) campus, the University of Tokyo. JCAHPC (http://jcahpc.jp/) was established in 2013 under agreement between the Center for Computational Sciences, University of Tsukuba (CCS) and the Information Technology Center, the University of Tokyo (ITC). JCAHPC consists of more than 20 faculty and staff members from both CCS and ITC. The primary mission of JCAHPC is designing, installing and operating the Post T2K System (Oakforest-PACS). CCS and ITC will cooperate for the procurement, installation and operation of the Oakforest-PACS under JCAHPC agreement. In addition, CCS and ITC will develop system software, numerical libraries, and large-scale applications for the Oakforest-PACS in collaboration made possible by the establishment of JCAHPC. JCAHPC is a new model for collaboration in research and development between supercomputer centers in Japan. JCAHPC is an advanced organization based on T2K Open Supercomputer Alliance by University of Tsukuba, University of Tokyo and Kyoto University (http://www.open-supercomputer.org/) established in 2006, where each of three universities introduced its own system, respectively. The Oakforest-PACS will be offered to researchers in Japan and their international collaborators through various types of programs operated by the High-Performance Computing Infrastructure (HPCI, http://www.hpci-office.jp/folders/english), by MEXT's Joint Usage/Research Centers, and by each of CCS and ITC under their original supercomputer resource sharing programs. It is expected to contribute to dramatic development of new frontiers of various field of studies, including computational science and engineering (CSE). The Oakforest-PACS will be also utilized for education and training of students and young researchers in both of CSE and high-performance computing (HPC). Both of CCS and ITC will continue to make further social contributions through operations of the Oakforest-PACS.
Details of the System
The Oakforest-PACS system has of 8,208 compute nodes, each of which consists of next-generation Intel® Xeon Phi processor (code name: Knights Landing), and Intel® Omni-Path Architecture (Intel® OPA) as a high-performance interconnect, which is a brand-new fabric developed by Intel Corporation. This is the first large scale system with such a processor in Japan. The system is integrated by Fujitsu and its PRIMERGY server is employed as each of compute node. Additionally, the system employs the shared files system (capacity: 26 PB), and the fast file cache system (940 TB), both of which are provided by Data Direct Network (DDN). Peak performance of the Oakforest-PACS is 25 PFLOPS and the total memory capacity is more than 900 TB. All compute nodes and servers of file systems are connected by fat-tree topology based on Intel® OPA, which provides full bisection bandwidth. Therefore, flexible and efficient utilization and operation of compute nodes and file systems is available. Moreover, the fast file cache system is equipped with SSD's and it is suitable for such applications that require higher file I/O performance. Supercomputing Research Division, Information Technology Center, the University of Tokyo (SCD/ITC) The Supercomputing Research Division, Information Technology Center, the University of Tokyo (http://www.cc.u-tokyo.ac.jp/) was originally established as the Supercomputing Center of the University of Tokyo in 1965, making it the oldest academic supercomputer center in Japan. The Information Technology Center (ITC) was organized in 1999, and the Supercomputing Center became the Supercomputing Division (SCD) of the ITC, joining three other divisions at that time. ITC is also a core organization of the "Joint Usage/Research Center for Interdisciplinary Large- Scale Information Infrastructures" project, and a part of HPCI (the High-Performance Computing Infrastructure) operated by the Japanese Government. The three main missions of SCD/ITC are (i) providing services for supercomputer operations and supporting supercomputer users, (ii) doing research, and (iii) providing education and training. Currently, SCD/ITC consists of more than 10 faculty members. SCD/ITC is now operating three supercomputer systems, a Hitachi SR16000/M1 based on Power7 architecture with 54.9 TFLOPS of peak performance (Yayoi), a Fujitsu PRIMEHPC FX10 System (Oakleaf-fx) at 1.13 PFLOPS, and another Fujitsu PRIMEHPC FX10 System (Oakbridge-fx) at 136.2 TFLOPS for long-time execution. From July 1 st, 2016, a new system (Reedbush) by SGI which includes 540 nodes of Intel®Xeon Broadwell-EP, and 240 NVIDIA Tesla P100 (Pascal) GPU's. 120 nodes with GPU's will start to operate March 1 st, 2017. Total number of users for these systems are more than 2,000, where 50% of them are outside of the University of Tokyo. Center for Computational Sciences, University of Tsukuba (CCS) Center for Computational Sciences, University of Tsukuba (http://www.ccs.tsukuba.ac.jp/) was established in 2004 as an advanced center from the former center named Center for Computational Physics (CCP) established in 1992. This is a research enter highly dedicated to the state-of-the-art researches in both computational sciences and high performance computing engineering. Both former and current centers are established as a unique organization in Japan where the domain scientists who need the supercomputer capability and computer scientists who work on advanced system development join for tightly coupled collaboration. From the origin, one of the most important mission of CCP was the development of highly efficient supercomputers based on "co- designing" concept to support various application fields by latest HPC technology. This concept is inherited by CCS with wider variety of scientific fields in computational sciences. CCP developed a massively parallel system named CP-PACS in 1996 and it was ranked as world fastest supercomputer in TOP500 list on November 1996 with 614 GFLOPS of peak performance. After the organization is changed to CCS, the researchers of domain science and computer science developed high-end supercomputers such as PACS-CS (high bandwidth cluster with 14 TFLOPS peak performance), FIRST (heterogeneous cluster with 38 TFLOPS peak performance in total with GRAP-6 gravity calculation engines), HA-PACS Base Cluster (GPU cluster with 800 TFLOPS peak performance) and HA-PACS/TCA (GPU cluster with original interconnection network system PEACH2 with 360 TFLOPS peak performance), and COMA (heterogeneous cluster with Intel Xeon Phi many-core processors, 1 PFLOPS peak performance).