Cloud Infrastructures for AI, Data Science, and System Research

June 1 20221

mdx : an IaaS platform for bridging communities of datascience, data driven applications, HPC, and system software

Kenjiro Taura
The University of Tokyo

Who am I▪ Professor in Graduate School of Information Science and Technology, theUniversity of Tokyo▪ Also, director of Information Technology Center (ITC)▪ my main job in the last four years▪ Information Technology Center▪ research computing infrastructure▪ network within and across universities▪ IT services for education
ITC
Cloud Infrastructures for AI, Data Science, and System Research

June 1 2022

3

Research computing infrastructure in Japan▪ Supercomputers: High Performance Computing
Infrastructure (HPCI)▪ Flagship (K; Fugaku by Riken CCS)
▪ Tier-2 systems (by nine ITC-like departments in universities)▪ Wide-area network: SINET by NII▪ High bandwidth, high service network across many research
institutions▪ Data: Research Data Cloud (RDC) by NII▪ Research data management, sharing and search▪ ID management & federation: Gakunin by NII
Cloud Infrastructures for AI, Data Science, and System Research

Research Data Cloud (RDC)
June 1 2022

4

What is mdx?
▪ A medium-sized cluster with virtualization▪ Gakunin for quick access to academic researchers (in Japan for now)a user environment

Router (Ethernet 100Gbps)

a virtual LAN (private network)
Ethernetネットワーク (100GbE/25GbE)

virtual machines

RDMA/ストレージ/仮想ディスクストレージネットワーク (100GbE、RDMA通信用)

mdx
Cloud Infrastructures for AI, Data Science, and System ResearchstorageJune 1 2022

5

Who are doing it?
▪ Operated by 9 universities and 2 national research institutes▪ Open to research and development purposes▪ Launched Sep 2021
▪ Attract 70 user groups so far

Cloud Infrastructures for AI, Data Science, and System Research

June 1 2022

6

High level objectives▪ provide data-focused research computing infrastructure that are▪ more flexible and▪ more accessible

than monolithic environments (e.g., traditional supercomputers)▪ make community of people across a wide range of data-focused research
fields that▪ need data science, machine learning, AI▪ need to build data platforms▪ preprint
Cloud Infrastructures for AI, Data Science, and System Researchsupercomputers

dataJune 1 2022

7

JHPCN (community-building)▪ A virtual institution of eight university ITCs soliciting research proposals
using university resources and supporting/collaborating with researchers▪ It has been very successful in building community of researchers acrossvarious fields of computational science▪ with mdx, we aim to extend it to data science fields

JHPCN
Cloud Infrastructures for AI, Data Science, and System Research

June 1 2022

8

Requirements▪ Support data gathering and data platform services▪ Connectivity to outside and always-running services▪ Store sensitive data
▪ Help researchers not fluent in HPC (MPI/CUDA/OpenMP) programming▪ Catch up rapid progress in open-source software, esp. machine learning(torch, tensorflow, etc.), web-based programming (Jupyter) and XYZ wedon’t know today virtualized cluster
Cloud Infrastructures for AI, Data Science, and System Research

June 1 2022

9mdx spec and size
CPU nodes
Ice Lake 2 x 368
Peak DP Flops：2.1PFLOPS
Total mem BW：150.7TByte/s
GPU nodes
NVIDIA A100 x 8 x 40
Peak DP Flops：6.4PFLOPS
Peak HP Flops：100.7PFLOPS
Total mem BW：496.3TByte/sRouter (Ethernet 100Gbps)

Ethernet (100GbE/25GbE)

Fast NVMe storage
Lustre file system
1.0PB (NVMe SSD)
252GByte/s

x368

x40

Object storage
S3 Data Service
10.3PB
63.0 GByte/sRDMA/storagenetwork (100GbE、RDMA)

1.0PB

Cloud Infrastructures for AI, Data Science, and System ResearchHigh capacity HDD storage
Lustre file system
16.3PB
157.5GByte/s

16.3PB

10.3PB

June 1 2022

10

Integration to national Research Data Cloud (NII)▪ Research Data Cloud (RDC) is a research data
management/publication/discovery platform▪ of which Gakunin RDM (GRDM) is a data management/sharing component▪ mdx provides storage and computer resources for data on GRDM

mdx
Cloud Infrastructures for AI, Data Science, and System Research

Research Data CloudJune 1 2022

11

Challenges (past, on-going and ahead)▪ User portal▪ User education▪ Security (shared responsibility)▪ Auto configuring user clusters (software-defined environment)▪ Ansible▪ Efficient resource management▪ Good mix of “space sharing (long running VMs)” and “time sharing (spot VMs)”

Cloud Infrastructures for AI, Data Science, and System Research

June 1 2022

12

Road ahead▪ Hybrid “space-sharing” and “time-sharing” resource management▪ Supercomputer-class (e.g., IB) networking technology supporting secure
virtualization▪ A massive virtualized supercomputer supporting both user-managedclusters and massively parallel HPCs/MLs▪  An environment that integrates data management of individual
researchers, lab-scale computing, domain data services, massively parallel
computing (both computational and data science)Thank you for listening!
Contact: tau@eidos.ic.i.u-tokyo.ac.jp
Cloud Infrastructures for AI, Data Science, and System Research

June 1 2022

13