Exa FLOW

EXAFLOWENABLING EXASCALE FLUID DYNAMICS SIMULATIONSMichèle Weiland
m.weiland@epcc.ed.ac.uk

Tsukuba Workshop, 7th & 8th December 20172

ExaFLOW: key facts• EU-funded project, 3 years, started October 2015
• www.exaflow-project.eu
à Address current algorithmic bottlenecks for Exascale to enable
the use of accurate CFD codes for problems of practicalengineering interest

Tsukuba Workshop, 7th & 8th December 2017

3Project objectives• Adaptive mesh refinement in complex computationaldomains• Solver efficiency• e.g. mixed discontinuous/continuous Galerkin methods, appropriate
optimised preconditioners• Strategies to ensure fault tolerance and resilience• Heterogeneous modelling to allow for different solution

algorithms in different domain zones• Improved I/O for large data volumes
EPCC’s focus• Efficiency• Minimise energy/power consumption and time to solution
à remove inefficiencies in implementations and system setup

Tsukuba Workshop, 7th & 8th December 20174

Power and energy consumption of CFD
Test Cases

Michael Bareford, Nick Johnson, and Michèle Weiland.
On the trade-offs between energy to solution and runtime for real-world CFD test-cases.
2016. DOI: http://dx.doi.org/10.1145/2938615.2938619

Tsukuba Workshop, 7th & 8th December 2017

5ARCHER• Cray XC30 MPP, 4920 Compute Nodes• Dual Intel Xeon processors (Ivy Bridge), 24 cores, 64

GB• Tests conducted on 2-cabinet Test Development

Server• Private to EPCC, minimises resource contention

Tsukuba Workshop, 7th & 8th December 2017

6Cray XC30 Power Management Counters

Running Average Power LimitCounters

Power Management Counters

PACKAGE_ENERGY (nJ)

PM_POWER:NODE (W)

DRAM_ENERGY (nJ)

PM_ENERGY:NODE (J)PP0_ENERGY (nJ)

PM_FRESHNESS• PACKAGE = processor• two sets of RAPL counters per node

• Power instantaneous, energy cumulative

Tsukuba Workshop, 7th & 8th December 2017

7

PAT MPI Library(https://github.com/cresta-eu/pat_mpi_lib)• Acts as a wrapper that simplifies monitoring a user-defined set of hardware

performance counters during the execution of a MPI code running acrossmultiple compute nodes• Controls which MPI processes read counters (one per node for PM counters,

two per node for RAPL).

CALL pat_mpi_open(out_fn)
DO i=1,nstep
CALL pat_mpi_monitor(i,1)...
! application code...
...
CALL pat_mpi_monitor(i,2)
ENDDO
CALL pat_mpi_close()• Only one MPI process (e.g., rank 0) collates the data,

writing it to a single file.

Tsukuba Workshop, 7th & 8th December 20178

Code 1: Nektar++
Nektar++ v4.2.0 (MPI)
http://www.nektar.info
An open-source spectral element code that combines
the accuracy of spectral methods with the geometricflexibility of finite elements, specifically, hp-version
FEM
Supports several scalable solvers for many sets ofpartial differential equations, from (in)compressible
Navier-Stokes to the bidomain model of cardiac
electrophysiology

Tsukuba Workshop, 7th & 8th December 2017

9Nektar ++ test case
Aorta Blood Flow

Aortic arch mesh

Unsteady diffusion
equations with acontinuous Galerkinprojection
Resources
compiler: craynodes: 4
runs: 10Vincent, Plata, Hunt et al.,J R Soc Interface. 2011

runtime: ~16 mintime steps: 4000

Tsukuba Workshop, 7th & 8th December 2017

10Code 2: SBLISBLI v4.2.0 (MPI)
http://www.southampton.ac.uk/engineering/research/projects/sbli_computer_code.page

SBLI is a high-order fully parallelised finite difference codethat solves the full 3D compressible Navier-Stokes
equations
The code is designed for large eddy simulations oftransitional and turbulent flow
Actual code used is a customisation of v4.2.0 that includes
a particular treatment for trailing airfoil edge

Tsukuba Workshop, 7th & 8th December 201711

SBLI test case
NACA4412 Airfoil Simulation (compressible air flow)
http://library.propdesigner.co.uk/html/naca_4412.htmla)b)

Resources

Block 3ξcompiler: craynodes: 4
runs: 10W↑
wake line

Block 2Block 1

ηR

runtime: ~15 mintime steps: 1000

Tsukuba Workshop, 7th & 8th December 2017

Average Power (per node)

12

Tsukuba Workshop, 7th & 8th December 2017

Average Power (per node)13

Tsukuba Workshop, 7th & 8th December 2017

Average Power (per node)

14

Tsukuba Workshop, 7th & 8th December 2017

Average Power Distributions
Nektar++

SBLI

15

Tsukuba Workshop, 7th & 8th December 2017

Average Power Over Time16

Tsukuba Workshop, 7th & 8th December 2017

Energy Usage per time step17

Tsukuba Workshop, 7th & 8th December 2017

Energy Usage per time step18

Tsukuba Workshop, 7th & 8th December 2017

Energy Usage and CPU Frequencies

19

Tsukuba Workshop, 7th & 8th December 2017

Summary Part 1• Observed results show varying behaviour ofdifferent applications• Performance, power, energy are all variable• Between runs and within a run• If time to solution is not a concern, energysavings can be significant• Detailed monitoring can give insight intoefficiency to may inform optimisations

20

Tsukuba Workshop, 7th & 8th December 201721

Moving Nektar++ towards industrial useTsukuba Workshop, 7th & 8th December 2017Mark Taylor April 22 2017

High-order simulation of whole car geometry“Rp1” by Elemental (http://elementalcars.co.uk)

22Tsukuba Workshop, 7th & 8th December 2017

23

Task
• High-order simulations with Nektar++ are too

slow to be usable in an industrial context• EPCC tasked with understanding the bottlenecks

and (hopefully) work out a solution• Focus on Nektar++’s incompressible Navier Stokes

solver using the Rp1 test case
• Profiling on ARCHER Cray XC30
• 84 compute nodes (2016 cores), total runtime ~50mins• Cray Performance Analysis Tools (CrayPAT)

Tsukuba Workshop, 7th & 8th December 201724

Initial profiling results• Call Tree results showed significant

MPI_Waitall imbalances originating fromGSLib• GSLIB is a library for Gather/Scatter-type nearest

neighbor data exchanges• The most significant of these had an average tomaximum time difference of 150 – 420 s• Next step: investigate communication patterns

Tsukuba Workshop, 7th & 8th December 2017

Call tree visualisation of imbalance25Tsukuba Workshop, 7th & 8th December 201726

ARCHER networko Chassis with 16 compute bladeso 128 Sockets
o Inter-Aries communication overbackplaneo Per-Packet adaptive Routing

Tsukuba Workshop, 7th & 8th December 2017

Visualisation of all communication

Developed python script to turn CrayPAT
XML output into communication heatmap.

27

Tsukuba Workshop, 7th & 8th December 2017

Visualisation of off-node communication only28

Tsukuba Workshop, 7th & 8th December 201729

Communications summary

Baseline: percentage ofcore communicating pairs
Bytes: break down by
number of bytes
Count: number ofmessages
Receive/Send Time: total
receive/send timeTsukuba Workshop, 7th & 8th December 201730

Time in MPI_Waitall across all MPI ranks

Ordered bywaiting time perprocess rank, inascending order
Ranks 2,1 wait theleast amount oftime
Rank 1702, 1718
wait the mostamount of time

Tsukuba Workshop, 7th & 8th December 201731

Impact of element shape per process on wait times• Processes compute on

both prism andtetrahedra elements• Ranks 2,1: least time inwait è lots of work

• high number of tetrahedra• Ranks 1702, 1718: mosttime in wait è little work
• few tetrahedra, moreprisms

same rank order as
previous slideTsukuba Workshop, 7th & 8th December 201732

Change weight of prisms• Implication from previous graph• Prisms take less time to compute than expected• Processes are given too few prisms and run out of work

• Element weights are calculated based on shapeand boundary conditions• Graph partitioning takes into account the weights• Weights associated with prisms is too high, therefore

partitioning leads to processes being allocated too few
prisms• What happens if prism weight is halved?

Tsukuba Workshop, 7th & 8th December 2017

Changing the weight of prism elements

Good news: time per simulation step reduced on average, imbalancetime roughly halved.
Bad news: not by enough, ~1s across all time steps.
Even worse news: new load imbalances have now appearedè Work in Progress!

33

Tsukuba Workshop, 7th & 8th December 201734

Summary Part 2• Nektar++ has the potential to be an important

application for high-order modelling• However in order to make code useable in industrial

setting, we need to fix this performance bottleneck• Developed ways to visualise the profiling information to
help us work out solutions• Optimising load-balancing of complex meshes isvery difficult• Fixing one problem can lead to new unexpected

problems somewhereTsukuba Workshop, 7th & 8th December 2017

Questions?

35