Fusional Integrator for Radiation-hydrodynamic Systems in Tsukuba University
for elucidating FIRST generation objects

FIRST Project
Grants-in-Aid for Specially Promoted Research, MEXT inJapan“Elucidation on the origin of first generation objects in the Universe with
Heterogeneous Multi-Computer System”
2004~2007
Total budget is 428 million yen (US$3.6 million)
Division of Computational
Astrophysics

Division of High Performance
Computing Systems

M.
M. Umemura
UmemuraK.
K. Yoshikawa
Yoshikawa (2007-)(2007-)
H.
H. Hirashita
HirashitaY.
Y. Kato
Kato (2005-)(2005-)
T.
T. Suwa
Suwa (2005-)(2005-)T.
T. Akahori
Akahori (2007-)(2007-)J.
J. SatoSato (2004-2006)(2004-2006)
T.
T. Nakamoto
Nakamoto (Tokyo(Tokyo I.Tech)
I.Tech)H.
H. Susa
Susa (U(U Kounan)
Kounan)M.
M. Mori
Mori (U(U Senshu)
Senshu)M.
M. SatoSato (2004)(2004)T.
T. BokuBoku
D.
D. Takahashi
TakahashiO.
O. Tatebe
Tatebe (2006-)(2006-)

Core Members

Self-Gravity

Radiation Transfer

Hydrodynamics

Astrophysical Radiation Hydrodynamics

Radiation Hydrodynamics
stardark matter

Gas clouds

6D Radiation Hydrodynamics (space，directions，frequencies)
operations ∝ N x N y N z ⋅ N θ N ϕ ⋅ M ν = 5 Tera

(N = 100, N ν = 500)

Development of Next Generation Massive
Parallel Computer Project (1997 - 2001)
U Tsukuba & U Tokyo

HMCS: Heterogeneous Multi-Computer System
Radiation
Hydrodynamics

Self-gravity
GRAPE-6

CP-PACS

1Tflops×8

2048PU~0.6Tflops
PCI×8Cluster(Parallel IO)
100base TX
16IOU

HUB

16IOU

RDMA: 300MB/s

Necessity for New Type of Heterogeneous Computer System
Objective• Radiation Hydrodynamics with 10 million particles• Computational time of several months
Requirements① Host： several Tflops② Gravity calculation： several 10 Tflops③ High communication bandwidthSolution① PC cluster② Embedded special-purpose processor③ High performance network

“FIRST” Simulator
A New-type of Hybrid Cluster
PC cluster embedded with special board of gravity calculations.
2U server

FIRST Simulator

Multi-port
Ether NIC

CPU1

CPU2Interconnect
Network Switch

chipset
PCI-X bus

Blade-GRAPE

special board of gravity calculations
Blade-GRAPE X64

Blade-GRAPE
Embedded Special Purpose Processor for Gravity(Newly-developed in 2005)•2 PCI-X bass full slots for 2U server•10 layers in a board•4 GRAPE6 chips = 136.8GFLOPS•electric power of 54W(from power supply for disk drive)•memory of 16MB （260 thousand particles）
GRAPE6 chips
×４Blade-GRAPE
heat-sink

Cooperation
Hamamatsu Metrics Co.

First Model of “FIRST” Simulator
May 2005
16 nodes HP Cluster with Blade-GRAPE
Performance
Host 217 Gflops
Blade-GRAPE 2.2 Tflops

16 nodesCooperations
HP Co.
Best Systems Inc.
Sumi-Sho Computer Systems Co.

Blade-GRAPE X64（Newly developed in 2006）
64 bit PCI-X version with FPGA

Cooperation
Hamamatsu Metrics Co.
K&F Computing Research Co.

“FIRST” Simulator
Completed in March, 2007
256 （16×16） nodes
496 CPU ＋
16 Blade-GRAPE
224 Blade-GRAPE X64
Total Performance = 36.1 Tflops
Host 3.1 Tflops
Blade-GRAPE 33 Tflops
Total Memory = 1.6TB
Total storage = 22TB (Gfarm)

Blade-GRAPE X64

Gfarm File System
Gfarm file system = Scalable virtual filesystem federating local file systems ofcluster nodes
SC06 Storage Challenge
Winner in Large-Systems
by attaining 52GB/s for 1112 node
system in KEK, Japan
Tatebe et al. (SC06)

Gfarm on FIRST••1 Meta-data server (first-fs2)
256 filesystem nodes(22TB)
Meta-data server

first 256 nodes

Network
Uniform Connection to 240 Port Gbit Ether Switch
for first0101-1516
240Port GbEInterconnect Switch

first0216

first0316

first1501
first1502
first1503
first1504
…

…

………

…

first0301
first0201
first0302
first0202
first0303
first0203
first0304
first0204

first1516

Blade-GRAPE Performance
100%
98%

EPRA
G
e-4
6
X

60%

adl
B

a
Bl

80%

RG
deE
P
A
40%

20%

Particle # / board

Performance of “FIRST” Simulator• PC Cluster +Blade-GRAPE
Peak speed=33.3Tflops
Effective speed=32.3Tflops (N=260,000)

without GRAPE

Gravity calculation

z=0 (NO GRAPE)

1010
times
fastertimes
fasterz=0 (Blade-GRAPE 64)

ホスト部分(その他)
PM 部分
PP部分(GRAPE 以外)
GRAPE (重力計算)
GRAPE (通信)5 times faster MPI 通信

with GRAPE
z=0 (Blade-GRAPE)

0100

200

300

400

500

600

700

Press, News, Exhibitions＜Press Release＞
First model of FIRST Simulator
March 13, 2005
＜TV News & Newspapers＞
TV News
NHK(Mito)
March 13, 2005
NHK(Capital Area) March 17, 2005
News & Net
Yomiuri, Asahi, Ibaraki, Tokyo, Jyoyo, Nikkan-Kogyo,
Nikkei-Sangyo, Kyoto

Press Release＜TV Interview＞
NHK(Mito) May 17, 2005
＜Exhibitions＞
SC05 (Seattle)
SC06 (Tampa)
SC07 (Reno) coming soon

Press Release

SC05

Radiation-SPH(RSPH) Schemewith FIRST simulator
TREE-GRAPE-SPH +Radiative transfer+ Non-equilibrium Chemistry + Thermal processes
1. Hydrodynamics
SPH (Umemura 1993; Steinmetz & Muller 1993)
2. Self-gravity
Parallel Tree-GRAPE code (Orthogonal Recursive Bisection )3. Frequency-dependent Radiative Transfer (Ray-tracing)(Kessel-Deynet & Burkert 2000, Nakamoto et al. 2001)
4. Non-equilibrium Chemistry & Thermal Processes(Susa & Kitayama 2000)

Radiation Transfer on SPH
Kessel-Deynet & Burkert (2000)
Optical depth calculations (Ray Tracing)τ TS = ∑in + n )( s(2Ei

Ei +1

Ei +1− sEi

P3TargetTσ)
P5

P2E4E1θ

E2
P1E5

E3P4SourceS

Parallelization of Radiative TransferParallelization by Domain decomposition
ORB (Orthogonal Recursive Bisection )Paralellization for Multiple Sources
MWF(Node Wave Front) Parallelization

Cosmological Radiative Transfer Codes Comparison Project II:Radiative Hydrodynamic Tests
RSPH

RH1DIlliev et al. 2007
in preparation

Radiation SPH Simulation on
Radiative Feedback on First Star Formation(Susa & Umemura 2006, 2007)
1 million dark matter / gas particles
Gas cloudUV radiation

StarUV radiation from a star generates an ionized region accompanied by a shock,which collides with a gas cloud. If the cloud density is higher than a thresholdvalue, it can collapse to form a new star.

H2 Shielded Collapse
n on = 3 ｴ 103 cm - 3

Susa & Umemura 2006

M-type IFShock-driven Evaporationn on = 3 ｴ 102 cm - 3• shock is raised by
M-type IF• shock blows the collapsingcore

P3M-GRAPE Simulation onFormation of First Stars(Suwa et al. 2007 ⇒ this afternoon)

WMAP 3 year ΛCDM cosmology

zin=15, 100kpc [comoving]3
Baryon mass: 6×106M~
Dark matter mass: 3×107M~

6 x 107 particles
for baryon + dark matter

Mass resolution: 0.3M~ in baryon
1.5M~ in DM
No change of mass resolution
throughout the simulation

First Metal Enrichment in the Universe
Mori & Umemura 2007

Total mass 108 M~，Baryon mass: 1.75x107 M~ , z=202563 meshGlobular Cluster Formation
in UV Background(Hasegawa & Umemura, 2007 ⇒ next afternoon)

UVsupersonicinfall

Radiative Transfer in Magneto-hydrodynamicAccretion Flows(Kato et al. 2007 ⇒ next afternoon)

Emergent spectrum of the Galactic center
Monte-Carlo Radiative Transfer is solved inmagnetohydrodynamic (MHD) accretion
flows plunging into a supermassive blackhole in our galactic center.

P3M-GRAPE Simulation on
Early Structure Formation in the Universe(Hirashita et al. 2007, in prep)

High-Redshift Galaxies
1.6 x 107 dark matter particles
We simulate distributions and luminosities of high-z galaxies withcosmological N-body scheme on 16 nodes of FIRST.

These galaxies are expected to be found by
next generation telescopes, e.g. ALMA.

Luminous infrared galaxies atz=6

SPH Simulation on Merger of Galaxy Clusters(Akahori et al. 2007 ⇒ next afternoon)
Non-equilibrium states of intracluster plasma
Density

Temperature

Non-equilibrium states ofintracluster plasma are goodprobes for understanding
merging clusters, in parallelwith calculations for dozensof electron non-equilibrium
states of oxygen, iron, andother heavy elements in theplasma.

6D Collisionless Boltzmann for Dark Matter(Sato & Umemura, in prep.)(⎛G m j xi − x j
∂f
∂f
1a&⎜
+v+ −2 v − 3 ∑3a j∂t∂x ⎜⎜ axi − x j⎝

6D collisionless Boltzmann
equation is solved in phasespace with CIP scheme.

) ⎞⎟ ∂f = 0⎟ ∂v⎟⎠

Summary• “FIRST” simulator has been build up to realize radiationhydrodynamic simulations in astrophysics.
• It possesses accelerator for gravity calculations, Blade-GRAPE,and the total peak speed is 36.1 Tflops.
• Intensive simulations have started with FIRST simulator on
1) Formation of First Stars
2) Formation of Globular Clusters
3) Formation of First Galaxies
4) BH Accretion Flows
5) Clusters & Large-scale Structure6) Collisionless Boltzmann