2007.Oct.30-Nov.1
External Review

External Review on
Center for computational sciences〜 Plan and Strategies of CCS 〜

Mitsuhisa Sato, Director & Professor
Center for Computational Science
University of Tsukuba

Computational science

Large-scale simulations using supercomputers
Critical and cutting-edge methodology in all of science and engineering
disciplines
supercomputers
Nano materials

Particles and universe

Bioclimate

CP-PACS

Simulation

Scientific research
Synchrotrontheory
Relativity(Einstein)Experiment/
observation

Protein analysis

Subaru telescope
Meson theory(Yukawa)Particle accelerators
Renormalization theory(Tomonaga)

astronomy2Cillision experiments

Mission of CCSThe mission of the Center of Computational Sciences is topromote scientific discovery by computational sciencethrough the application of advanced computing technologies,
and to support researches of computational science inJapanese universities and institutes by operating leadingedge computing systems.
The center carries out R&D in high performancecomputing and networking systems, and applications ofnovel information technologies.
 The center is an inter-university facility open toresearchers throughout Japan.
3

Position and Role of CCSFrom the nation-wide viewpoint of development of scientific researchAn unique research “inter-university” organization which jointly carry out“interdisciplinary” researches on computational sciences, even including
development of the leading-edge high performance computer systemsComputational Science in each discipline is done by university lab and institutes specific to its discipline.
National Institute of Informatics (NII) focuses on Information Science, neither computational science nor
development of high performance computer system.
Supercomputer Centers in major university focus on just operation and service on computing facilities, andnot researches on computational science.

From the viewpoint of development of scientific research within our university

The faculty of CCS belongs to graduate schools and contribute to the education ofcomputational science
Our university supports CCS as one of the most important research organizations withrespect of budgets, faculty position and facilities.
“Academic Computing and Communications Center” (campus computer center)manages campus information infrastructures such computer systems and network
systems in the university.

4

CCS’s Vision:
COE of “Interdisciplinary Computational Science”
Computational Science is a cutting-edge and indispensable multidisciplinary
area for the development of science and technology of the 21st century.
Computational Science as the key to meet the grand challenges in nature and human, environments.
Integration of science and computer/information science to realize the computational science.
 Education System to produce the next generation of scientists who carry out research from global perspectives.


Existing “computational science”• Computational X as a partof X science• Short-term individual
projects• Make use of“supercomputer” center

New Interdisciplinary Computational Science• Integration and collaboration of computational science andcomputer science• High performance computing system will be getting bigger andmore complex in configuration. HPC technologies areindispensable to make the best use of it• Continuous development and investment are required to enable
the next generation of computational science.

• The common base of computational science brings mutual
intimate connections among various scientific science
disciplines and allows us to approach science as a wholefrom a global perspective.

We
Weaim
aimto
tobe
bean
anworld-wideworld-wideCOE
COEto
todevelop
develop“Interdisciplinary“InterdisciplinaryComputationalComputationalScience”
asanewarea
to21stcentury
advances
inscienceandScience” as a new area to 21st century advances in science andtechnology,
technology,
andsupport
itcontinuously
fromglobalandlong-termperspectives.
and support it continuously from global and long-term perspectives.
5Strategies and Plan of CCS

[Research & Development] Integrate the forefront of science research with that of computer science andinformation science, and to establish and promote new Interdisciplinary computational science Execute “Interdisciplinary Computational Science Promotion Programs” for the operation of interuniversity facilities such as PACS-CS Æ Promotion and acceleration of InterdisciplinaryComputational Science Integrate computational science with grid/network computing, sensing technologies, dataengineering and informatics Æ “new” concept of computational science[Education & Public Relations] Educate and produce the next generation of scientists who push forward
new Interdisciplinary computational science from global viewpoints.
 Dual degree (double majors) program, campus-wide courses on “computational science” forgraduate students, HPC seminar series, workshops and symposium[Infrastructure & Service] Support researches by operating leading-edge, advanced, large-scalecomputing facility for inter-university computing resources, and development of advanced computing
infrastructure Acquire & operate the next supercomputer as a part of inter-university computing resources,
making “Open supercomputer” alliance with Univ. of Tokyo and Kyoto Univ (T2K).
 Contribution to the “national” next-generation supercomputer project, and take a leadership ofnational-wide computing infrastructure.
[Collaboration & Alliance] Promote alliance and collaboration, and establish international/national
networks of researches on computational science Alliance with “supercomputer” centers of universities, the next-generation supercomputer center ofRIKEN, and research Laboratories in Tsukuba
 Promotion of international collaborations, and Support of Virtual Organization using Gridtechnologies.
6Strategies and Plan of CCS (1)[Research & Development] Integrate the forefront of science
research with that of computer science and informationscience, and to establish and promote new Interdisciplinary
computational scienceExecute “Interdisciplinary Computational Science Promotion
Programs” for the operation of inter-university facilities such as
PACS-CS Æ Promotion and acceleration of InterdisciplinaryComputational Science
Integrate computational science with grid/network computing,sensing technologies, data engineering and informatics“new” concept of computational science
7

CCS Inter-University Activity:
Interdisciplinary Computational Science Promotion Programs
Interdisciplinary collaboration program

Support to establish a network of interdisciplinary
research projects and to encourage grand challenges
of interdisciplinary computational science.
To promote interdisciplinary research projects ofdifferent disciplines.
Between Computer science and application fields （e.g.
Applied Math and Particle Physics）
 Between different application fields（e.g. Modeling ofradiation in Astrophysics and Climate simulation）


Field ＡGroup Ｘ
Proposal
requestcoordinate

Field Ｃ・ Field Ｄ
Alliance ＺＷ

proposal
Proposal
requestsupportInterdisciplinary collaboration
program support committee（ review of proposal and coordinate collaboration）
Establish a network to promoteinterdisciplinary research projectsMatchmaking of research groups from different fields
Review of Proposals by external reviewer ofInterdisciplinary collaboration coordination committee
 Coordinate research groups from different fields
 Find partners to mach the request
 Advice with expertise of different fields

Field Ｂ
Group Ｙ

Follow up the research activity produced by theinterdisciplinary collaboration

Field ＣGroup Ｚ

Field ＡGroup Ｘ

CCS
Field Ｂ
Group Ｙ

Field ＤGroup Ｗ

Large-scale scientific simulation program
Push forward the grand challenge of several fields in computational sciences by providing thecomputational power of the PACS-CS.
 Review proposals and concentrate our computational power to make new scientific discoveries Follow up the scientific results

8

“new” concept of computational science“Sensing Web” project

The Computational Media Group and the Grid computing group have started
projects “Sensing Web” in collaboration with Kyoto University and otheruniversity for the new integration with sensing technology and internet/grid.

“Initiative for Real-World Computational Informatics”

The Computational Informatics Division and High Performance
Computing System Division,Real World
Global Environment and BiologicalNature（e.g. Climate）
Sciences division propose the idea
Social phenomena
to integrate sensing, database andcomputation to capture and observe
phenomena of human and nature
Sensing
Real Worldin real world.

Sensing technology

Accepted as a strategic
project of the university for two years.

Human

Displayinteraction

Informatics Infrastructure

security

Technology ofinterface

Analysis
Modeling
Real-timeprocessing
Networking
technology

Large & highAccumulation Performance
integration
computing
Large
Distributed
database

Grid computing
ＨＰＣ technology

9Strategies and Plan of CCS (2)[Education & Public Relations] Educate and produce the next
generation of scientists who push forward newInterdisciplinary computational science from global
viewpoints.


Dual degree (double majors) program, campus-wide
courses on “computational science” for graduate students,
HPC seminar seriesWorkshops and symposium

A plan of “Long-term Computational Science Workshop” wherean advanced subject encompassing multiple disciplines of science andcomputer science is to be chosen, and participants who stay from a
few weeks to a few months are encouraged to carry out
interdisciplinary research collaborations.

International symposium once in every year

there has been 10 symposia over the past 5 years

10Educational Activities in CCS

HPC SeminarThis seminar presents knowledge, methods
and techniques for programming modernhigh performance computer systems,
including recent microprocessors, and its
performance turning, parallel programming.
 Participants: researchers and users ofcomputational science (including
researchers in companies) Periods: 2 or 3 days in summer seasonEnables a graduate student in a doctoral
program to simultaneously belong to amasters program of a different Graduate
School, and receive both a doctoral degreein science and a masters degree incomputer science, or vice versa, upon
graduation.
 Design of curriculum and courses foradvanced computational science Educate researchers who can push forward
new Interdisciplinary computational sciencefrom global viewpoints
 Plan: will be started in 2008 (Physics inDoctor course and Computer sciences inMaster course)

Held in July 2007(more than 80 participants)

Also broadcasted via internet

 Campus-wide courses on “computationalsciences” for graduate students
Faculty members of CCS give lectures
Accredited as “unit” in graduate courses.
 CoursesComputational Science Literacy
High Performance Parallel Computing
Technology for Computational Sciences(overlapped with HPC Seminar)

Start from 2008

Computational Science Dual Degree(double major) Program

Education in each graduate school

CCS allocates space for some ofstudents, and create environment toenable interactions to other fields.
11Strategies and Plan of CCS (3)[Infrastructure & Service] Support researches by operating
leading-edge, advanced, large-scale computing facility forinter-university computing resources, and development ofadvanced computing infrastructure

Acquire & operate the next supercomputer as a part of interuniversity computing resources, making “Open supercomputer”
alliance with Univ. of Tokyo and Kyoto Univ (T2K).
Contribution to the “national” next-generation supercomputer project,and take a leadership of national-wide computing infrastructure.
Explore the next-generation high performance platform.

12What’s “T2K” Open Supercomputer AllianceThree Japanese national universities: Tsukuba, Tokyo and Kyoto
We are now going to perform the procurement for next supercomputers at
the same time-flame for replacement of current ones
We agree to cooperate and make an alliance to share the main systemarchitecture on three machinesCommon Architecture - Node and Interconnection
Use of Open System/Software – based on Linux Cluster
Source code level compatibility – easy code porting & grid
Grid computing – resource sharing both on CPU and Data

T2K Open Supercomputer Alliance

13Background

Current satisfied users of PC Clusters are:

Scientific computation users with small to medium size system
Embarrassingly Parallel Application users

Many unsatisfied users still exist:
Lower performance than estimatedSuch as Monte-Carlo simulation, no communication is required

Poor performance of interconnection network … and memory bandwidth

Explore new application fields of PC Cluster

Large scale genomic informatics
Large scale archive search (database & data-mining)We
Wewillwillfindfindthethewaywayforforlarge
largescale
scalesystems
systemsand
andnew
newappapp14

Concept of “Open Supercomputer”
Openness of basic architectureOpenness of system software

Use High-performance commodity processor
 Market leading processor enables the high-performance, high-density
and low-power computing facility in the supercomputer center

Use Open source based operating system and libraries
 Seamless environment both on multi-scale and multi-site provides users
to develop various fields and size of application

Openness of user’s needs
Support much wider area of users than traditional supercomputer centers
 To explore new application fieldsArchitecture
Architectureto
tocovercoverfromfromlaboratories
laboratoriesto
toSC
SCcenters
centers
15

Design: node architecture and choice of processorThe next generation processor is a Quad-core, 2xFPU, withover 2GHz
Peak performance > 32 GFlops/socket
 Eg. Barcelona/AMD, Intel Tigerton and Kentsfield Memory bandwidth becomes more serious problem.
NUMA architecture
corecorecore

core

core

core

core

memory

core

core

core

core

core

core

core

core

memorycore

memorymemoryEach processor should haveits own memory When memory affinity fits,
it can exploit
good memory bandwidth.
 Eg. AMD HyperTransport

16

Expected basic configuration (As of March, 2007)
Node architecture








64-bit IA32 architecture> 16 cores / node
> 32 Gbyte memory/node,> 40 Gbyte/sec memory
bandwidth> 128 Gbyte local hard disk

NetworkInterface
NetworkInterface

MemoryMemoryMemoryMemoryInterconnection networkPhysical performance > 5 Gbyte/sec
MPI performance > 4Gbyte/sec

NetworkInterface
NetworkInterface

Local Disk

17

A possible configuration


core

core

core

core

core

core

core

core

core

core

core

core

core

core

IO interface

memorycore

memoryDDR infiniband x 4 orMrinet10G x 4

corememoryMemory:
8 Gbytes memory/socket
10GB/s DDR2
NUMA by HyperTransportmemoryAMD Barcelona (Quad Core)
4 socket/node
2.3 GHzIO interface

Network (DDR Infiniband x 4)
18

T2K schedule and expected performance
Three universities share the same time-line on theprocurement schedule.
 The final specification will be published on early October.
Biding will be on December.

Operation is expected to get started on June 2008

Expected performance of each university






Univ. of Tokyo> 150 TFLOPS
Univ. of Tsukuba > 80 TFLOPS (> 500 nodes)
Kyoto Univ.
> 70 TFLOPS

We expect more than 300 TFLOPS in total!

19

Toward Petascale computing“Element technology for next-generation supercomputer” Project Collaboration with Hitachi Ltd. on low-power on-chip memory architecture fornext-generation systems

Collaboration with RIKEN on the petaflops system development
Formal agreement between U. Tsukuba and RIKEN signed in September 2006
Participation of several CCS faculty in the system design as visiting
researchers
Agreement of actual collaboration in tuning of representative petascale
applications (QCD, RS-DFT and FFT)As a leading research organization, it is very important issue
for us to take a leadership concerning to operation of thepetascale system in national-wide computing infrastructure.

20

R&D of Next-generation HPC platform
The next generation high-performance computing platforms
may be:arithmetic accelerators such as GRAPE-DR, Clear-speed, GPU
many-core architectures which integrate many functions on one chip.

These architectures are promising for some special-purpose
applications.
This approach is possible just in our center.



The FIRST project is very good example we have achieved so far in thisapproach.

Is it still possible to take an approachto develop large machine by our own?
“Commitment” to design of high performance system
is really important !!

21Strategies and Plan of CCS (4)[Collaboration & Alliance] Promote alliance and collaboration,and establish international/national networks of researcheson computational science

Develop (futher!) alliance with:“Supercomputer” centers of universities (also T2K)
 The next-generation supercomputer center of RIKEN
 … and research Laboratories in TsukubaPromotion of international collaborations
A plan of “Computational Science Researchers Invitation
Program” to invite researchers who are leading the forefrontof research in science and computational science.
 Based on pear-to-pear personal/group relationshipSupport of Virtual Organization using Grid technologies.
Collaboration with NII CSI project22

International collaborations
International Lattice Data Grid in particle physics (Edinburgh University (UK),DESY(Germany), Univ. di Roma(Italy), FNAL/BNL/JLAB(USA))
Astronomical Radiation Hydrodynamics Research (National Astronomical
Observatory(Japan), SISSA(Italy))
Nano-Bio Science Collaboration (ETH(Switzerland), Princeton University/Vanderbilt
University(USA), Advanced Institute for Science and Technology(AIST), National Institutefor Materials Science (NIMS)),
Quantum Structural Biological Science Collaboration (AIST, Hyogo Prefectural
University(Japan), ETH/IBM Zurich(Switzerland)
Quantum Dynamics Collaboration (RIKEN(Japan), Washington University (USA))
Warming of Arctic Research (IARC of Alaska University/Hawai University(USA), Pusan
University(Korea), Academia Sinica(China))
France-Japan Grid Computing (National Institute of Informatics(Japan), INRIA(France),
Pennsylvania State University(USA))
Information integration/Knowledge discovery Research (Carnegie-Mellon
University/Georgia Institute of Technology(USA), Hong-Kong Chubun University).

23

Finally, …
CCS’s vision: We aim to be an world-wide COE to develop“Interdisciplinary Computational Science” as a newarea to 21st century advances in science andtechnology, and support it continuously from global
and long-term perspectives.
What is “Interdisciplinary Computational Science”?
Computational X exists!
 Interaction between computational X and computer science Interaction between computational X and computational Y with respect tomodeling and methodologies.
 What’s is “more”?


US NSF’s “cyberinfrastructure” vision

It must help to accomplish our mission
Promotion of scientific discovery
 Support for researches of computational science as inter-university activity.
24

Thanks for your attention!
Question and comments
And Discussion!

25