University of Tsukuba | Center for Computational Sciences

COMA (PACS-IX): Massively Parallel Many-Core Accelerated Cluster
System architecture and configuration
COMA (Cluster of Many-core Architecture processors) is the 9th generation of PACS/PAX series supercomputer in CCS,
University of Tsukuba. The processor with many-core architecture is one of promised basic computation resource for next
generation of HPC. Currently, Intel Xeon Phi is the most practical and productive co-processor, but many-core processor
will be a strong main processor for HPC applications in very near future. Toward development of wide variety of scientificapplications based on many-core concept, we introduced the largest many-core based PC cluster in Japan with Intel Xeon
Phi co-processor.
COMA consists of 393 of high-end computationnode, where two Intel Xeon Phi co-processors areequipped on each node as well as dual socket
IvyBridge CPUs. All the nodes are connected by FDR
InfiniBand network under Fat-Tree configuration
with full bisection bandwidth by Mellanox HCA andswitch.
P0CPU1Intel E5-2670v2
10IntelIVB CORE
E5-2670v2
DDR-3

10 IVB CORE P1
DDR-III#2 #3 #1 DMI2

P1CPU2Intel E5-2670v2
10 IVBE5-2670v2
CORE
Intel
DDR-3
P0 10 IVB CORE
DDR-III#1Intel Xeon Phi 7110p
SLOT 1/2

Infiniband FDR
SLOT5Block diagram of computation node of COMA
Item

Specification

Computation node

Cray CS300 Cluster Unit with two Xeon Phi

CPUIntel E5-2670v2 (Ivy Bridge EP)# of cores

10 cores/socket x 2 sockets = 20 cores/node

Clock

2.5 GHz

Peak performance 400 GFLOPS/node

redundant power
supplier
Item

Specification

Peak performance

1.001 PFLOPS(MIC: 843.9TF, CPU: 157.2TF)# of nodes

393

File system

Lustre, 1.5 PB user area(DDN SFA12000)

PCI-express

generation 3 x 80 lanes (40 lanes/CPU)

Memory64 GiB, DDR3 1866MHz,
4 channel/socket, 119 GB/s/node

Infiniband network switch

Intel Xeon Phi 7110P

393 port FDR(Mellanox SX6536)

Total network bandwidth

2.75 TB/s

Language

Fortra90, C, C++

MPI

MVAPICH2, Intel MPI

System Management

Cray Advanced Cluster Engine,
SLURM

MIC
# of MICs/node

2

Peak performance 2.14 TFLOPS/node (1.07 TF/MIC)
Memory16 GiB/node (8 GiB/MIC)

Interconnection

Infiniband FDR (Mellanox ConnectX-3)

Computation node of COMA

System SpecificationNode architecuture
A computation node of COMA is equipped with two set ofXeon Phi co-processor. These co-processors and InfiniBand
FDR HCA require 40x lanes of PCIe gen-3. We decided toconcentrate all of them to be attached to single CPU while
there are two sockets of E5 CPUs. When Xeon Phi coprocessor works in “native mode” where it runs its own Linux
Operating System as like as ordinary Xeon processor, it
accesses another Xeon Phi co-processor or InfiniBand HCA
directly through PCIe switch inside Xeon CPU. It is known that
PCIe device-to-device direct communication performance isdrastically degraded through Intel QPI path, then all thesedevices are connected to one side of CPU in COMA.

IB FDR
Mellanox
Connect-X3

Intel Xeon E5-2670v2 (IvyBridge core)

SATA HDD(3.5inch 1TB x2)

Intel Xeon Phi 7110P

Chases inside of computation node

JCAHPC plan and COMA
Currently COMA is dedicated to several supercomputerutilization programs to support nation-wide computational
sciences.
University of Tsukuba and University of Tokyo plan tointroduce a joint supercomputer system under a neworganization named JCAHPC (Joint Center for Advanced HighPerformance Computing) which is located in Kashiwa
Campus of University of Tokyo. We will introduce a newsupercomputer based on many-core architecture to achieve
30 PFLOPS class of peak performance. COMA is dedicated tothe application development and performance tuning as atestbed for this new machine.
http://www.ccs.tsukuba.ac.jp/