データ活⽤社会創成プラットフォームmdxにおけるマテリアルズ・インフォマティクス研
究・共創に向けて
Accelerating interdisciplinary research between
material science and computer scienceon the mdx data science platform
Toyotaro Suzumura and Masatoshi Hanai
Information Technology Center
The University of Tokyo

鈴村豊太郎・華井雅俊
東京大学情報基盤センター

⼤学ICT推進協議会 2021年度年次⼤会（AXIES2021）2

Outline• mdx: A platform for the data-driven future• Material Science on mdx

3Motivation• Accelerating inter-disciplinary research and collaborations• Data science has become a key driver to advance science and technology,
but more collaborations among various domain experts in both academiaand private sectors should be formulated to solve complex problems inreal-world society (e.g. material discovery, carbon-neutral society).

• How can we accomplish this ?

• Agility : Need a national-wide cloud platform (IaaS and PaaS/SaaS) thatenables researchers to promptly set up their own data analytics
environments by allowing them to install/configure their required analytics
softwares, and publish data repository services as needed• Connecting with Edge Devices (e.g. Electron Microscopes, IoT sensors)
via High-Speed Network:• Coupling with HPC environments :• Some applications need big computing powers for such applications as large-scale
simulations and large-scale AI/ML training thus the platform should be coupledwith supercomputers when needed.

• Why can’t we rely sololy on supercompters or public clouds?

• These requirements can not be fulfilled by current types ofsupercomputers mainly targeting big science, and also public clouds fromprivate sectors

Status in the U.S.
• Ian Foster, Daniel Lopresti, Bill Gropp, Mark D. Hill,Katie Schuman. “A National Discovery Cloud:Preparing the US for Global Competitiveness in theNew Era of 21st Century Digital Transformation”• 2021 Apr https://arxiv.org/abs/2104.06953• The need for national discovery cloud:• DOE and NSF supercomputers provide access topowerful simulation capabilities, but with access limited
to small communities.
• With a few notable exceptions, AI-ready datasets forresearch use are lacking. Commercial clouds areaccessible to anyone with a credit card, but there is littleof the coordination needed to create nationally useful
discovery cloud services.
5mdx: A Platform for the Data-Driven Future• Target is to leverage data utilization at all over Japan making full useof high6performance R&E network “SINET” an R&E network of Japan operated byNII (National Institute of Informatics)• Project supported by the Japanese government• Currently jointly being operated by:• 9 National Universities (Tokyo, Hokkaido, Tohoku, Tsukuba, Tokyo
Tech, Nagoya, Kyoto, Osaka, Kyushu)• NII (National Institute of Informatics)• AIST (National Institute of Advanced Industrial Science and Technology)• Invite universities and public research institutes of all over Japan to use theplatform for industry-academia and local government-academia
collaboration activities.
• Production-level operation has been started since March 2021

mdx as On-demand Collaborative Platform
Conduct data analytics from various locations via SINET or Mobile SINET
Disaster protection

Medicine

Shared infrastructure
Network / Storage / Computers

SINET5SINET
mobileAgriculture / Fishing

Data

Smart Cities

Energy / Power Grid7

mdx : Providing Secure and On-demand Virtual Platform(Optionally) with access to Supercomputers and Edge Devices
VirtualPlatform

MobileSINET

InternetPublicCloudmdx

Electron Microscopes
Shared Object
Storage(S3 compliant)

AIST ABCI

Secure and Fast Storage

Supercomputers(Wisteria/BDEC-1 etc.)

mdx: A Platform for the Data-Driven Future• Agility : Provide a rapid PoC environment to accelerate R&D 9datautilization as well as industry-academia collaboration projects.
• Sharing: Shared platform for various data utilization activities• Network: Combine a high-performance wide-area academic
network called “SINET” with high performance computing andstorage infrastructure• Seamless Integration with Edge Devices:• Users can use wide bandwidth low latency “slices”• Wide-area virtual infrastructure isolated from “the internet”• Connect edge devices with high performance computing andstorage infrastructure and supports real-time data processing• Matching: Will provide matching function of:• Data providers: various data and their owners, and• Data scientists and researchers who have skills/tools to analyze
dataHardware Overview• Facility• < 2.0 MW including Cooling, <170 m2• Same location with Wisteria/BDEC-01, same campus
with AIST’s ABCI supercomputer• Compute nodes (CPU)• 368 nodes : Each node : Intel Xeon (IceLake-SP, 38cores) x 2 CPU sockets/node• 2.1 Peta flops (double precision)• Total memory bandwidth : 150 TB/sec• Compute nodes (GPU)• 40 nodes, Intel Xeon (IceLake-SP) x2 socket• NVIDIA A100 x 8 GPUs/node• 6.4 Peta Flops (FP64), 6.7 PF (FP32), 100 PF (FP16),• Total memory bandwidth: 496 TB/sec• Storage• Fast Storage with NVMe SSD: 1.0 PB, 250 GB/sec• Large Storage with HDD: 16.3 PB. 157 GB/sec• Shared Object Storage (S3): 10 PB, 63 GB/sec• File System : Lustre• Network• Frontend (Juniper) : 25 Gbps
Ethernet• 100G to SINET• 400G to Wisteria/BDEC-01• Storage, RDMA(Mellanox/NVIDIA) : 100GEthernet with RoCEv2• Overlay with EVPN-VXLAN• Software, etc.
• VM & Container (VMware vSphere)• IaaS like management• High security, high availability
New building in Kashiwa II Campus

10Resource Management and Scheduling• Each institution is responsible for approving project applications andtheir required hardware resources
Conferences,/Events for mdx
community

Marketing

Ecosystem on mdx
Resource Poolfor Project X

Project

Resource Poolfor Project Y

2. Allocate
virtualresources
Resource Pool for Institute A(vCPU, vMem,etc)

Matching

1. Apply for theirusers(Gakunin ID ormdx-based ID）

Usage approval
for collaboration

Institute admin
0. Allocate
physicalresources

Physical resource pool in mdx(CPU, Memory, Storage, Network, etc)

mdx user(Researcher）

Service User

mdx user (project user)

mdx admin

3. Help users（mdxauthentication）

mdx user（Researcher）

UsageApprovalInstitute admin

Invoice fortheir usage

mdx adminData Community: Matching Data Providers and Data Scientists• Data Community on mdx• As of Jan 2022, we are currently in the midstof designing a PaaS level platform that
establishes a community between dataproviders and data scientists• The alpha version: January 2023 (Plan)

Example) Kaggle-like community• Requirements• Data providers can easily upload their data to”mdx” by specifying the spec of data and data
usage conditions (e.g. only for researchpurpose)• Data scientists can easily find data based on
their interests, and launch Jupyterlab

Data Providers

Data Scientists
12Outline• mdx: A platform for the data-driven future• Material Science on mdx

13Usecase: Material Science and Engineering• Key Challenges

1. Big data from laboratory instruments

Machine Learning/ Data Analysis• Image data from electron microscope• up to 1TB / experiment

Experimental Data
From ElectronMicroscope

2. High computational-powerrequirement• Physics simulations(e.g., First-principle calculation)• Machine learning / data analysis

3. Flexible & secure data management• Share confidential data among
co-researchers• Publish open data on archives(e.g., ARIM Japan
https://www.nanonet.go.jp/pages/arim/index.html)

PublishData onArchives

User

Share Data among
Co-researchers

First-Principle Calculation onSupercomputerUsecase: Material Science and Engineering

aloceL rUs rveSe• Machine learning / data
analysis on local servers• Get data from laboratory
instruments via portablestorage• Access supercomputers forFirst-principle calculation
Publish Data
• Send data to co-researchers on Archieves• etc.

Po
Sto rtabl
rag eeMachine Learning /Data Analysis

Communicate
data

Traditional ways

UserShare Data among
Co-researchers

Experimental Data From
Electron Microscope

Retriev
eresult

First-Principle
Calculation on
SupercomputerUsecase: Material Science and Engineering
System Integration with mdx:

• Install mdx as the extension of “local sever” with high performanceand large storage• Customizable VM environment (GPU-available)• Store all data in the large storage on mdx (Lustre)• Secure & High-performance Inter-University Network (SINET)• Data from laboratory instrument are via SINET (or intra-university network)• Seamless workload extension to academic supercomputers• Machine learning (GPUs on mdx) ó First-principle calculation (Supercomputer)• Publish data via Amazon S3 protocol

Academic ID

User
Co-researchers

Vertual Machine(GPUs)

Large Network Storage

Inter/IntraUniversityNetworkAmazon s3
ProtocolElectronMicroscopeSupercomputers

Comparison to Other Services
OS-level
Customizabl
Management e Resources

mdx
Local Lab
PC/Server
SupercomputerEnterprise
CloudHigh
ComputationalPower

Large-Scale
Accessible
Storage

Secure & HighEnd Academic
NetworkOpen DataPublication

https://mdx.jp/18

# of Launched Projects on mdx since October 2021
Word Cloud from Project Names# of Launched Projects on mdx
60

50

40

30

20

100
2021/10/1

2021/11/1

2021/12/1

2022/1/119

https://mdx.jp/20