Efficient Cooling System for Green Operation

Global Scientific Information and Computing Center, Tokyo Institute of Technology

HARDWARE
SOFTWARE
SPECIFICATIONS

Warm Water Cooling
Cooling system requires additional electric power. In order tominimize it, we employ low-power evaporative cooling tower
located on the rooffloor of the building. It is estimated that thecooling tower provides less than 32 degrees water even insummer, and the compute nodes of TSUBAME3.0 are designed
to work with the warm water.

Cooling towers on the roof of the building

CPUs and GPUs are direct water-cooled. The other componentsare air-cooled using rack-contained built-in heat exchanger(Water Coils in the figure) between water and air, allowing
low-cost, high-maintainability, and high energy efficiency. Theracks for storage and I/O systems employ rear door coolingtechnology.

tsubame3.0

Small space installation
TSUBAME2.5

GPU Accelerated Compute Node
High-Speed Network Interconnect

TSUBAME3.0
Highly efficient cooling of TSUBAME3.0 enables high density compute nodes. The space required for installation isless than half of TSUBAME2.5.

High-Speed and Highly Reliable Storage Systems
Efficient Cooling System for Green Operation
System and Application Software

System Software
The job management system and the cluster management
system are working together to manage user environment aswell as distributing computational resources to the insufficientpart by taking from the node pool. Some jobs do not use all ofcomputing resources (CPU cores, GPUs, memory, etc.) on allocated compute nodes. UNIVA Grid Engine supports cgroup andDocker to enable efficient node sharing between multiple jobs.

OSSUSE Linux Enterprise Server 12 SP2Batch System

UNIVA Grid Engine

ISV (commercial) Software( * GPU full or partial support )Compilers and DebuggersApplications

Intel Compiler (C/C++/Fortran)
PGI Compiler*(C/C++/Fortran, OpenACC, CUDA Fortran)
Allinea FORGE*

ANSYS Workbench*, Mechanical* ABAQUS*, ABAQUS CAE
ANSYS CFD, Fluent*, HFSS*MSC Nastran*, Patran, Marc*
COMSOL Multiphysics
CST STUDIO SUITE* (MW-Studio*)
LS-DYNAAMBER*
Gaussian*, Gauss View
Materials Studio, Discovery Studio
MATLAB*
Mathematica*
AVS/Express, AVS/Express PCE Maple*
Schrödinger Small-Molecule Drug Discovery Suite*

Yellow: The license for all users

White: The license for Tokyo Tech users

Magenta: The license for industrial users

Peak Perf.: 12.15 PFlops (DP), 24.3 PFlops (SP), 47.2PFlops (HP)
Full bisection bandwidth Fat-Tree, qual-rail Omni-Path network
15.9PB Lustre filesystem at 150GB/s
Warm water cooling using cooling towers

Published by Global Scientific Information and Computing Center, Tokyo Institute of Technology
2-12-1 Ookayama, Meguro-ku, Tokyo 152-8550, JAPANTEL : +81-3-5734-2087 FAX : +81-3-5734-3198 E-mail : tsubame@gsic.titech.ac.jp

http://www.gsic.titech.ac.jp/
2 0 17 0 6

GPU - Equipped High- Performance Compute Nodes

High-Speed Interconnect

TSUBAME3.0 system includes 540 compute nodes, which provides 12.15 PFlops performance in total. Each compute node isequipped with two CPUs and four GPUs in a compact design blade. In addition, high-speed network with four Omni-Path
HFIs and large capacity SSD accelerate applications’ performance in Big Data and AI areas.

Compute nodes of TSUBAME3.0 interconnected with Omni-Path Architecture of full bi-section bandwidth fat-tree
network achieving 432Tbps. End-to-End latency between the compute nodes is extremely low in microsecond-order time,
resulting in high-speed performance and high-speed connection to highly reliable storages.

Compute Node × 540

HPE SGI ICE-XA (SGI 8600) IP139-SXM2 540 nodes

1 Chassis (Compute Node × 9)

SGI ICE XA cabinet(front door opened)

CPU:Intel Xeon E5-2680 V4 (Broadwell-EP, 2.4GHz) ×2 sockets
14 cores per socket, total 28 cores per node.
GPU: NVIDIA TESLA P100 for NVlink-Optimized servers ×4.
Memory: 256GB (DDR4-2400 32GB module ×8)
SSD: Intel DC P3500 2TB (NVMe, PCI-E 3.0 x4, R2700/W1800)
Network: Intel Omni-Path Architecture HFI (100Gbps) ×4

SSD×60Each E-Rack contains up to
four chassis, and each chassis
contains nine compute nodes.

water cooling blocks
for CPU

Storage system

Gateway & Login
9 nodes

Edge Switch #1

water cooling blocks
for GPU

Gateway

Edge Switch #2

Full bi-section bandwidth
optical fat-tree network
Six optical cables are wired
between each edge and core switch
FCBN425QB1Cxx Active Optical Cable
VCSEL array technology
Core Switch #1
using multimode fiber

A compute node (cover opened)Core Switch #2

Core Switch #3

Block Diagram
TESLA P100 GPUs
x16 PCIe

DMI

x16 PCIedual port Omni-Path HFI

DIMM
DIMM
DIMM
DIMM

DIMM
DIMM
DIMM
DIMM

Xeon CPUs

PLX

x16 PCIe

CPU 0

QPI

SSD

OPA HFI
x16 PCIex16 PCIe

GPU 1

GPU 0Lustre

NVLink

x16 PCIe/gs/hs1/gs/hs2“Global WorkSpace” #1“Global WorkSpace” #2“Global WorkSpace” #3

x16 PCIex16 PCIe

PLX/gs/hs0

HOME

Home

“NFS forcompute nodes”

GPU 3

GPU 2

CPU 1

x4 PCIe

OPA HFI

Core switch : Intel Omni-Path 100 Series 24 Slot Director Class Switch (768 port) x 3Edge switch : Omni-Path Standard Switch 48-port x 120 (+ 5 for storage systems)

x16 PCIex16 PCIe

OPA HFIOPA HFI

DDN GridDirector : 45TB

Edge switch #2

PCH

Edge switch #1

Network Switches

SFA7700X + SS8460 x 2
300GB 2.5” 10Krpm SAS HDD x 226(Data: 200, Meta: 20, Spare: 6)
NAS Gateway(NFS) x 4

Tesla P100 for NVlink-Optimized servers

HOME

Campus storage

Peak performance :5.3 TFLOPS (double precision)
10.6 TFLOPS (single precision)
21.2 TFLOPS (half precision)
Shader clock : 1328 MHz (1480MHz with boost)
Number of CUDA cores (SP) : 3,584
Streaming Multiprocessors : 56On-board memory: 16GB HBM2
Memory bandwidth : 720GB / sec
TDP: 300W

DDN EXAScaler x 3sets : Total 15.9PB, 150GB/s
CIFS gateway server x 2
Each EXAScaler consists ofSFA14KXE + SS8462 x 10
EF4024(MDS) x 2 + ED4024(MDT) x 2
10TB 7.2Krpm NL-SAS HDD x 700 (incl. 20 spare)
5.3PB effective capacity (7.0PB physical)“CIFS forother machines”

DDN GridDirector : 36TB
SFA7700X + SS8460 x 2
300GB 2.5” 10Krpm SAS HDD x 186(Data: 160, Meta: 20, Spare: 6)
NAS Gateway(CIFS) x 4

High-Speed and Highly Reliable Storage Systems

NVLink is direct interconnect between GPUs.
Bandwidth of a link is 20GB/s for each direction,and four NVLinks enable 160GB/s data transfer
in addition to the PCI-E bandwidth.

TSUBAME3.0 provides massive storage volumes to serve various purposes, including 1.08 PB of SSDs embedded
in compute nodes for scratch I/O, 15.9PB of parallel file systems such as Lustre for high speed parallel I/O, and45 + 36 TB of home storage volumes for providing campus cloud storage services.

Pascal Architecture (NVIDIA GP100)

GPU - Equipped High- Performance Compute Nodes

High-Speed Interconnect

TSUBAME3.0 system includes 540 compute nodes, which provides 12.15 PFlops performance in total. Each compute node isequipped with two CPUs and four GPUs in a compact design blade. In addition, high-speed network with four Omni-Path
HFIs and large capacity SSD accelerate applications’ performance in Big Data and AI areas.

Compute nodes of TSUBAME3.0 interconnected with Omni-Path Architecture of full bi-section bandwidth fat-tree
network achieving 432Tbps. End-to-End latency between the compute nodes is extremely low in microsecond-order time,
resulting in high-speed performance and high-speed connection to highly reliable storages.

Compute Node × 540

HPE SGI ICE-XA (SGI 8600) IP139-SXM2 540 nodes

1 Chassis (Compute Node × 9)

SGI ICE XA cabinet(front door opened)

CPU:Intel Xeon E5-2680 V4 (Broadwell-EP, 2.4GHz) ×2 sockets
14 cores per socket, total 28 cores per node.
GPU: NVIDIA TESLA P100 for NVlink-Optimized servers ×4.
Memory: 256GB (DDR4-2400 32GB module ×8)
SSD: Intel DC P3500 2TB (NVMe, PCI-E 3.0 x4, R2700/W1800)Network: Intel Omni-Path Architecture HFI (100Gbps) ×4

SSD×60Each E-Rack contains up to
four chassis, and each chassis
contains nine compute nodes.

water cooling blocks
for CPU

Storage system

Gateway & Login
9 nodes

Edge Switch #1

water cooling blocks
for GPU

Gateway

Edge Switch #2

Full bi-section bandwidth
optical fat-tree network
Six optical cables are wired
between each edge and core switch
FCBN425QB1Cxx Active Optical Cable
VCSEL array technology
Core Switch #1
using multimode fiber

A compute node (cover opened)Core Switch #2

Core Switch #3

Block Diagram
TESLA P100 GPUs
x16 PCIe

DMI

x16 PCIedual port Omni-Path HFI

DIMM
DIMM
DIMM
DIMM

DIMM
DIMM
DIMM
DIMM

Xeon CPUs

PLX

x16 PCIe

CPU 0
QPI

SSD

OPA HFI
x16 PCIex16 PCIe

GPU 1

GPU 0
NVLinkLustrex16 PCIe/gs/hs1/gs/hs2“Global WorkSpace” #1“Global WorkSpace” #2“Global WorkSpace” #3

x16 PCIex16 PCIe

PLX/gs/hs0

HOME

Home

“NFS forcompute nodes”

GPU 3GPU 2

CPU 1

x4 PCIe

OPA HFI

Core switch : Intel Omni-Path 100 Series 24 Slot Director Class Switch (768 port) x 3
Edge switch : Omni-Path Standard Switch 48-port x 120 (+ 5 for storage systems)

x16 PCIex16 PCIe

OPA HFIOPA HFI

DDN GridDirector : 45TB

Edge switch #2

PCH

Edge switch #1

Network Switches

SFA7700X + SS8460 x 2
300GB 2.5” 10Krpm SAS HDD x 226(Data: 200, Meta: 20, Spare: 6)
NAS Gateway(NFS) x 4

Tesla P100 for NVlink-Optimized servers

HOME

Campus storage

Peak performance :5.3 TFLOPS (double precision)
10.6 TFLOPS (single precision)
21.2 TFLOPS (half precision)
Shader clock : 1328 MHz (1480MHz with boost)
Number of CUDA cores (SP) : 3,584
Streaming Multiprocessors : 56
On-board memory: 16GB HBM2Memory bandwidth : 720GB / sec
TDP: 300W

DDN EXAScaler x 3sets : Total 15.9PB, 150GB/s
CIFS gateway server x 2
Each EXAScaler consists ofSFA14KXE + SS8462 x 10EF4024(MDS) x 2 + ED4024(MDT) x 2
10TB 7.2Krpm NL-SAS HDD x 700 (incl. 20 spare)
5.3PB effective capacity (7.0PB physical)“CIFS forother machines”

DDN GridDirector : 36TB
SFA7700X + SS8460 x 2300GB 2.5” 10Krpm SAS HDD x 186(Data: 160, Meta: 20, Spare: 6)
NAS Gateway(CIFS) x 4

High-Speed and Highly Reliable Storage Systems

NVLink is direct interconnect between GPUs.
Bandwidth of a link is 20GB/s for each direction,and four NVLinks enable 160GB/s data transfer
in addition to the PCI-E bandwidth.

TSUBAME3.0 provides massive storage volumes to serve various purposes, including 1.08 PB of SSDs embedded
in compute nodes for scratch I/O, 15.9PB of parallel file systems such as Lustre for high speed parallel I/O, and45 + 36 TB of home storage volumes for providing campus cloud storage services.

Pascal Architecture (NVIDIA GP100)

Efficient Cooling System for Green Operation

Global Scientific Information and Computing Center, Tokyo Institute of Technology

HARDWARE
SOFTWARE
SPECIFICATIONS

Warm Water Cooling
Cooling system requires additional electric power. In order tominimize it, we employ low-power evaporative cooling tower
located on the rooffloor of the building. It is estimated that thecooling tower provides less than 32 degrees water even insummer, and the compute nodes of TSUBAME3.0 are designed
to work with the warm water.

Cooling towers on the roof of the building

CPUs and GPUs are direct water-cooled. The other components
are air-cooled using rack-contained built-in heat exchanger(Water Coils in the figure) between water and air, allowing
low-cost, high-maintainability, and high energy efficiency. Theracks for storage and I/O systems employ rear door coolingtechnology.

tsubame3.0

Small space installation
TSUBAME2.5

GPU Accelerated Compute Node
High-Speed Network Interconnect

TSUBAME3.0
Highly efficient cooling of TSUBAME3.0 enables high density compute nodes. The space required for installation isless than half of TSUBAME2.5.

High-Speed and Highly Reliable Storage Systems
Efficient Cooling System for Green Operation
System and Application Software

System Software
The job management system and the cluster management
system are working together to manage user environment aswell as distributing computational resources to the insufficientpart by taking from the node pool. Some jobs do not use all ofcomputing resources (CPU cores, GPUs, memory, etc.) on allocated compute nodes. UNIVA Grid Engine supports cgroup andDocker to enable efficient node sharing between multiple jobs.

OSSUSE Linux Enterprise Server 12 SP2Batch System

UNIVA Grid Engine

ISV (commercial) Software( * GPU full or partial support )Compilers and Debuggers

Applications

Intel Compiler (C/C++/Fortran)
PGI Compiler*(C/C++/Fortran, OpenACC, CUDA Fortran)
Arm Forge*

ANSYS Workbench*, Mechanical* ABAQUS*, ABAQUS CAE
ANSYS CFD, Fluent*, HFSS*MSC Nastran*, Patran, Marc*
COMSOL Multiphysics
CST STUDIO SUITE* (MW-Studio*)
LS-DYNAAMBER*
Gaussian*, Gauss View
Materials Studio, Discovery Studio
MATLAB*
Mathematica*
AVS/Express, AVS/Express PCE Maple*
Schrödinger Small-Molecule Drug Discovery Suite*
Yellow: The license for all users

White: The license for Tokyo Tech users

Peak Perf.: 12.15 PFlops (DP), 24.3 PFlops (SP), 47.2PFlops (HP)
Full bisection bandwidth Fat-Tree, qual-rail Omni-Path network
15.9PB Lustre filesystem at 150GB/s
Warm water cooling using cooling towers

Published by Global Scientific Information and Computing Center, Tokyo Institute of Technology
2-12-1 Ookayama, Meguro-ku, Tokyo 152-8550, JAPANTEL : +81-3-5734-2087 FAX : +81-3-5734-3198 E-mail : tsubame@gsic.titech.ac.jp

https://www.gsic.titech.ac.jp/201906