May 2016
Supercomputing Research Division
Information Technology Center
The University of Tokyo

Integrated Supercomputer System for Data Analyses andScientific Simulations: Reedbush
Background and Current Status
The Supercomputing Research Division, Information Technology Center, the University of Tokyo(SCD/ITC/UTokyo, http://www.cc.u-tokyo.ac.jp) is operating the following three supercomputer
systems:· Yayoi: Hitachi SR16000, IBM Power7, 54.9 TFLOPS, October 2011~· Oakleaf-FX: Fujitsu PRIMEHPC FX10, SPARC IXfx, 1.13 PFLOPS, April 2012~· Oakbridge-FX: Fujitsu PRIMEHPC FX10, SPARC IXfx, 136.2 TFLOPS, April 2014~

FY05

Oakleaf-FX and Oakbridge-FX are based on same architecture, and they are commercial version ofK computer. Oakbridge-FX is mainly for long-time executions (e.g. > 7 days). Moreover, Post T2K
system based on manycore architecture will start its operation in December 2016. Post T2K system
is expected to be more than 25 PFLOPS peak performance, and is introduced under collaboration
with University of Tsukuba (Fig.1). University of Tsukuba and SCD/ITC/UTokyo established theJoint Center for Advanced High Performance Computing (JCAHPC, http://jcahpc.jp/) for thecollaboration in designing, installation and operation of the Post T2K system.
The supercomputer systems of SCD/ITC/UTokyo are shared by > 2,000 users, where more thanhalf of them are outside of the University of Tokyo. The systems are utilized for various kinds ofresearch areas, such as engineering, earth/space science, material science, energy science andphysics (Fig.2). Recently, number of users related to biology, such as biomechanics and biochemistry,has increased. Our systems are very busy, and combined average usage ratio of
Oakleaf/Oakbridge-FX was 80+% in FY.2015.
The Oakleaf-FX system will completed its mission in the end of March 2018. New system(Post FX10) is expected to start its operation in Fall 2018. Our current systems including Post T2K
are designed for computational science and engineering, and are mostly used for that purpose, asshown in Fig.2. In the Post FX10, we plan to develop new types of users, such as Big Data, Deep
Learning …
06 07 08 09 10 11 12 13 14 15 16 17 18 19
Hitachi SR11000/J2
18.8TFLOPS, 16.4TB

Fat nodes with large memory

Hitachi SR16000/M1
based on IBM Power-754.9 TFLOPS, 11.2 TB
Our last SMP, to be switched to MPP

Hitachi HA8000 (T2K)
140TFLOPS, 31.3TB(Flat) MPI, good comm. performance

Fujitsu PRIMEHPC FX10based on SPARC64 IXfx
1.13 PFLOPS, 150 TB
Turning point to Hybrid Parallel Prog. Model

Post T2K
25+ PFLOPS

Peta

京(=K)

Engineering
Earth/Space
Material
Energy/Physics
Information Sci.
Education
Industry
Bio
Economics

Initial Plan

Fig.1 Schedule for Procurement ofSCD/ITC/UTokyo

Fig.2 Areas for Usage in FY.2015 based onCPU hours (Oakleaf/Oakbridge/FX)

ReedbushConsidering these situations, we decided to introduce another small system, Reedbush. TheReedbush system has following two missions:· Supplementary computational resource for reducing congestion of Oakleaf/Oakbridge-FXsystems· Pilot system towards the Post FX10 system
We have two types of groups of compute nodes, (1) Reedbush-U (with CPU only), and (2)Reedbush-H (with accelerators). (1) and (2) could be operated separately. Summary of thespecification is as follows:(1) Reedbush-U· Each Node: Intel Xeon E5-2695v4 (Broadwell-EP 2.1 GHz 18core) x2 socket, 1,209.6 GF,
256 GB Mem, 153.6 GB/sec· Total Subsystem: 508.03 TF, 107 TB, Fat-tree with full bisection bandwidth (100 Gbps/node)(2) Reedbush-H·····

Host CPU: Same as Reedbush-U.
Accelerators: NVIDIA Tesla P100 x2, For each: 4.8-5.3 TF, 16 GB, 720 GB/sec
Host CPU~Accelerator: PCIe Gen3 x16 (15.7 GB/sec)
Accelerator~Accelerator: NVLink (20GB/s) x2 brickTotal Subsystem: 1.14-1.27 PF by Accelerators (excluded host CPU’s), 1.29-1.42 PF(CPU+GPU), Fat-tree with full bisection bandwidth (56 Gbps x 2 /node)(3) Storage· SharedStorage:
LusterFIlesystem 5.04 PB, 145.2
GB/sec.
· System for Fast File Cache:
DDN IME 209TB, 436.2
GB/sec.

Compute Nodes: 1.795-1.926 PFlops

CPU: Intel Xeon E5-2695 v4 x 2 socket(Broadwell-EP 2.1 GHz 18 core,
45 MB L3-cache)
Mem: 256GB (DDR4-2400, 153.6 GB/sec)

420

Dual-port InfiniBand FDR 4x

56 Gbps x2 /node

InfiniBand EDR 4x, Full-bisection Fat-tree
145.2 GB/s

Parallel File
System5.04 PB
Lustre Filesystem
DDN SFA14KE x3

Mellanox CS7500
634 port +SB7800/7890 36
port x 14

436.2 GB/s

High-speed
File Cache System
209 TB
DDN IME14K x6

ManagementServers
Login
Login
Login
Login
Login
Login
node
nodenodenodenodenodeUTnet

Login Node x6

Fig.3 Overview of MPT2K System
If you have any questions, please feel free to contact Kengo Nakajima or Toshihiro Hanawa(nakajima/hanawa(at)cc.u-tokyo.ac.jp).

120

SGI Rackable C1102-PL1

InfiniBand EDR 4x
100 Gbps /node· Air cooling only, 377.56 kVA(without A/C)
· Phase-I
operationfor
Reedbush-H: July 1, 2016
· Phase-II operation for fullsystem: March 1, 2017

1287.4-1418.2 TFlops
CPU: Intel Xeon E5-2695 v4 x 2 socket
Mem: 256 GB (DDR4-2400, 153.6 GB/sec)
GPU: NVIDIA Tesla P100 x 2(Pascal, SXM2, 4.8-5.3 TF,
Mem: 16 GB, 720 GB/sec, PCIe Gen3 x16,
NVLink (for GPU) 20 GB/sec x 2 brick )SGI Rackable
C2112-4GP3(4) Power, Cooling(5) Schedule

Reedbush-H (w/Accelerators)

Reedbush-U (CPU only) 508.03 TFlops

Users