Osaka University Creates AI Supercomputerwith Supermicro SuperBlades Utilizing 3rd GenIntel® Xeon® Scalable Processors
New Features Enable Faster Results and AI Research

Figure 1: Figure or Caption Style (optional)

Introduction

INDUSTRY
University Research andIndustry Collaboration

Osaka University is a world-renowned university known for leading-edge research ina wide range of disciplines. As a 90-year-old institution, Osaka University serves awide range of researchers, students, and external experts, making available asupercomputer system that runs state-of-the-art software on the latest CPUs andGPUs.

Challenges
CHALLENGES• Faster processing andreduced time to solution• Respond quick for new
application requests• Ability to take advantage of
new CPU instructions

The Cybermedia Center at Osaka University offers high-performance systems andstorage to diverse users, both internal to the university and external researchers. The
need for faster processing and a reduced time to solution for a wide range of
disciplines was a key factor when Osaka University looked to upgrade their system.
Osaka University is expanding its research areas, including high-performance data
analysis, Artificial Intelligence (AI), Deep Learning (DL), as well as more traditional
High-Performance Computing (HPC), which required the fastest system to beconstructed in a balanced way for high performance and the lowest TCO. This fact
© Copyright 2021 SupermicroIntel®

meant that the CPUs had to be fast with an increasing number of cores compared toprevious CPU generations, but communication with other devices such as GPUs andstorage had to be increased as well.
HPC applications always need the latest technologies such as processing speeds andother system capabilities for research into diverse fields such as genome research,
medical, weather prediction, earth sciences, and manufacturing simulations. With anincreasing demand for performance for a range of innovative technologies, Osaka
University needed a new system that could address the demands of the researchers.
Thus, the SQUID (Supercomputer for Quest to Unsolved InterdisciplinaryDatascience) system was developed to meet these new challenges.
SOLUTION• Supermicro SuperBlade• Supermicro GPU systems• 3rd Gen Intel® Xeon®
Scalable Processors

BENEFITS• Faster simulations• AI Workloads with the latest
GPUs and PCI-E 4.0• Workload Matching

PRODUCTS• Compute Nodes:
SuperBlade® 8U
SBI-420P blade

Osaka University offers a wide range of software to its users that is thoroughly tested
and qualified for new systems. When moving from one generation to the next withIntel products, the binary compatibility makes this qualification relatively easy.
Although applications may not be entirely optimized for a new instruction set, theapplication should run without any issues. Re-compiling with updated compilers will
increase the optimization amounts when a new CPU is introduced, containing new
and more performant instructions. Osaka University expects the FLOPS/Watt to besignificantly improved for reducing the carbon footprint with the new system. Userswill get massive improvement with their application performance by changing theparallelization tuning to include the latest Intel® Xeon® Scalable processor'sinstructions.

Solution
Osaka University worked closely with NEC Corporation and Supermicro to design andimplement a new supercomputing class system based on a combination ofSupermicro SuperBlade® systems and 4U 8 GPU NVIDIA NVlink ® systems, which isbuilt with the 3rd Gen Intel® Xeon® Scalable Processors. These systems werebenchmarked with the new X12 SuperBlade and exceeded the expectations of thesystem architects and early users. The performance, cost, and power usage of thelatest 3rd Gen Intel® Xeon® Scalable Processors have many advantages compared toprevious generations of Intel® Xeon® Scalable Processors.

SBE-820 enclosure, 4enclosures per rack
InfiniBand HDR 200• GPU nodes:
SYS-420GP-TNAR• Front end Nodes:
SYS-620U-TNR"Researchers and scientists in Osaka University are now working on their research projects in thehope of bringing open innovation to our society. Furthermore, the Cybermedia Center at Osaka
University is in charge of delivering high performance to researchers in Japan to advancescientific research as a national joint-use facility. The new supercomputing system named SQUID,
scheduled to begin operations in May 2021, was designed and implemented to facilitate
researchers and scientists to quest to unsolved interdisciplinary data science. The test drive
conducted on SQUID before its commencement of operation showed us that the Supermicro
servers with Intel CPUs provided more performance than was expected. We are excited toleverage the latest generation of Intel and Supermicro's solutions for advanced scientificresearch projects necessitating HPC (High-Performance Computing) and HPDA (HighPerformance Data Analysis)." - Dr. Susumu Date, Cybermedia Center, Osaka University.

© Copyright 2021 SupermicroIntel®

The SQUID system is a 27 Rack cluster that contains 1520 SuperBlades, 42 nodes of
the 4U 8 GPU NVLink systems, and was integrated and tested by the Supermicro RackIntegration Services team.
The systems use the 3rd Gen Intel® Xeon® Scalable Processors, specificallythe Platinum 8368, running at 2.4GHz, 38 Cores, TDP 270W. Though this high
TDP processor, in many cases, can be air-cooled, the most effective coolingsolution for stable computing is to use liquid cooling. Supermicro Direct
Liquid Cooling solution is the most cost-effective method to reduce thecarbon footprint and PUE of their data center.

The cluster is constructed from Supermicro X12 SuperBlades consisting of1520 blades (nodes) in 80, 8U enclosures with 3rd Gen Intel Xeon Scalable
Processors for high-performance computing. All of the chassis are integrated
with the direct liquid cooling system.



A complete rack integrated solution has direct liquid cooling to provide
optimized thermal efficiency for 76 nodes in a 42U rack with InfiniBand HDR
200 connectivity between nodes.

The Supermicro Rack Integration Services team installed and configured thesystems into 42U racks and tested the systems to ensure multi-rack level
confidence upon delivery.

The GPU rack for the heterogeneous HPC/HPDA needs also has direct liquidcooling to its CPU and GPU systems with A100 GPUs and NVSwitch™ modules,
with each rack having 48 GPUs in six SYS-420GP-TNAR servers.

The 2U Ultra server, SYS-620U-TNR, with direct liquid cooling, is used as thefront end node and is equipped with the Quadro RTX6000 card and hardware
RAID configuration.

The total deployment is 27 complete racks which were all assembled, integrated, andtested at the Supermicro factory and then directly shipped to Osaka University. Below
is a table that summarizes the different nodes in the Squid supercomputing system.

SuperBlade with DLC© Copyright 2021 SupermicroIntel®

GPU System with DLC

Product

Compute node:SuperBlade® 8U
SBI-420P blade
SBE-820 enclosure, 4enclosures. per rack
GPU node:
SYS-420GP-TNAR

Front end node:
SYS-620U-TNR

Qty

3rd Gen Intel® Xeon®
Scalable processors# of Sockets

Memory Per
Node

Direct Liquid Cooling
Component

1520 blades
80 enclosures

3rd Gen Intel® Xeon®
Scalable processors,
Platinum 8368, 2.4GHz,
38 Core, TDP 270W

Dual

DDR4-3200
CPU
256GB per node

42 nodes3rd Gen Intel® Xeon®
Scalable processor,Platinum 8368, 2.4GHz,
38 Core, TDP 270W

Dual

DDR4-3200
CPU, GPU, and NVSwitch
512GB per node

10 nodes3rd Gen Intel® Xeon®
Scalable processor,
Platinum 8368, 2.4GHz,
38 Core, TDP 270W

Dual

DDR4-3200
CPU
256GB per node

Benefits
The research community's popularity and scientific needs that Osaka University
serves required that more performance be added to the computing system. For many
researchers, the run queues, which can be thought of as a backlog of applications
waiting for their turn to run with the needed resources for their research, weregrowing longer, based on more complex simulations and the absence of GPU specificworkloads. Critical studies that used the latest systems technology were delayed, andusers were looking elsewhere to run their applications.
SUPERMICROSupermicro is a global leader
in high performance, green
computing server technology
and innovation. We provide
our global customers withapplication-optimized servers
and workstations customized
with blade, storage, and GPU
solutions. Our products offer
proven reliability, superior
design, and one of theindustry’s broadest array ofproduct configurations, to fit
all computational need.
For more information, visit
https://www.supermicro.com

The arrival of the new systems from Supermicro, based on the 3rd Gen Intel® Xeon®
Scalable Processors, the X12 SuperServer solution includes the latest GPUs and has
benefited users in the following ways:
Faster time to completion of simulations. With the new system, moresimulations of similar resolution to be run in a given amount of time, andmore research can be accomplished.

Increased resolution or fidelity with existing applications. Faster CPUs enable
more complex simulations to be run in a given time frame, with the additionof more physics or finer meshes.

Workload matching – With the use of the new GPUs, applications in AI, ML,and DL, has been accelerated significantly, enabling researchers to develop
new algorithms that were not possible previously.

The new Supermicro SuperBlade systems containing 3rd Gen Intel® Xeon® Scalable
Processors enable Osaka University to conduct more research much faster thanprevious HPC systems would allow. Entire applications are now able to use the wholesystem, employing over 120,000 cores simultaneously. PCI-E 4.0 also enables fastcommunication with the latest GPUs, further enhancing and speeding up research
applications.

© Copyright 2021 SupermicroIntel®

3RD GEN INTEL® XEON® SCALABLE PROCESSORS
3rd Gen Intel® Xeon® Scalable processors, Platinum 8368
PCI-E 4.0
Support for up to 12 TB memory per socket

Further Resources, please visit:
Supermicro X12 Products:
https://www.supermicro.com/en/products/x12?utm_source=Supermicro%20Home
page&utm_medium=Homepage&utm_campaign=X12%20post%20launchSupermicro SuperBlade Products https://www.supermicro.com/en/products/superblade
Supermicro GPU Systems - https://www.supermicro.com/en/products/GPUSupermicro Rack Integration Services https://www.supermicro.com/en/solutions/rack-integration
SQUID system, Cybermedia Center, Osaka University - http://www.hpc.cmc.osakau.ac.jp/en/squid/© Copyright 2021 SupermicroIntel®