8th Symposium

学際大規模情報基盤共同利用・共同研究拠点公募型共同研究 平成２８年度採択課題

jh160055-NAHI

Ryusuke Egawa, (Cyberscience Center, Tohoku University)

Cerebrospinal Fluid Flow Analysis in Subarachnoid SpaceBackgroundA better understanding of the hydrodynamics of the Cerebrospinal fluid (CSF) is
important for the understanding of diseases related to the brain and spine. An
accurate assessment of hydrodynamics of the CSF in complex geometries ofsubarachnoid spaces (SAS) is possible by means of fully resolved direct numericalsimulation that results in meshes consisting of up to 2 billion. The simulationrequires large computational resources and an efficient implementation of theunderlying algorithms.

Team Organization

Motivation and Goals
Overall goals of this research are- Simulating the CSF in subject specific SAS, with employment of high spatial andtemporal resolutions using modern vector parallel supercomputer SX-ACE.
- Carrying out quantitative performance analysis using cutting edge
supercomputers in Germany and Japan.
Our target code is a Lattice-Boltzmann method(LBM) solver named MUSUBI of theAPEs framework. To simulate the incompressible flow, LBM has some favorable
features for large scale, transient, incompressible flow simulations in complex
geometries. It offers a relatively simple kernel with few operations that can be easily
vectorized, and it also requires a relatively high Byte/FLOP ratio. Hence animplementation on SX-ACE would offer an efficient execution.

Turbulence intensity across axial planes in the SAS of a healthy volunteer and two Chiari patients. As canbe observed, the flow is evenly distributed in the volunteer, minor fluctuations develop in the Patient1
that are highly localized in the Patient2, due to impingement of flow in areas near the cranio-vertebral
junction.

Musubi in the APES tool chain
All tools based on Treelm Mesh format
Octree structureSpace-Filling Curve Ordering
Scalable distributed processing of meshes
Mesh generation by Seeder
STL geometry as inputs
Voxelized meshes with LBM specific higher
order boundary information (q-values) asoutputs• Simulation with Musubi• Post-processing with dedicated tool••
•
•
•
•
•

Suite of mesh based simulation tools– Written in Fortran 2003
• Compiles with gfortran, Intel,
IBM and Cray compilers– Two solvers on a common basis:• Lattice-Boltzmann Solver : Musubi• Discontinuous Galerkin Solver : Ateles

Musubi is the multi-level parallel lattice
Boltzmann solver within the APES suite. It isworking on a linearized octree and uses efficient
data structures allowing adaptive parallelsimulations.

• Scalable on cluster resources– Utilizing more than 100,000 cores (Juqueen)• Unstructured meshes (indirect addressing)

Early Performance Evaluation

SX-ACE

Since MUSUBI has been running on the intel based systems and JUQUEEN, We
have ported MUSUBI to the SX-ACE with simple optimizations. We will perform
advanced intra/inter node optimizations to exploit the potential of SX-ACE. Then,based on the quantitative performance analysis among supercomputers inGermany, we will carry out design space exploration of a future HPC systemdesign.
Evaluation Environments

SX-ACESuperMUC

JUQUEEN

HornetNEC SX-ACE Processor
2,560 nodes
707 Tflop/s
130 TB Mem

Intel Xeon
9,216 nodes
3,200 Tflop/s
288 TB Mem

IBM Bluegene
28,672 nodes
5,900 Tflop/s
488 TB Mem

Intel Xeon
3,944 nodes
3,786 Tflop/s
493 TB Mem

学際大規模情報基盤共同利用・共同研究拠点 第８回シンポジウム
2016年 7月 14日，15日THE GRAND HALL （品川）