Improving Strong Scalability Limitsof Finite Element Based Solvers
NICLAS JANSSONPDC Center for High Performance Computing
Introduction

Implementation of Hybrid MPI/PGAS in FEniCS

Performance Evaluation

To strongly scale finite element codes at exascale will require
unreasonable large problem sizes to balance communication
costs.

New hybrid parallelisation of the automated problem solving
environment FEniCS.

Performance evaluated on two Cray XC40, the 3944 node
Hornet at HLRS and the 1676 node Beskow at PDC, comparing•Low order methods have too little work per element tobalance communication costs•

One of the key bottlenecks is sparse matrix assembly•

Using two-sided message passing, the cost of message
matching unavoidably increase latency andsynchronisation costs•

FEniCS is written in C++ and parallelised using MPI•

Pure MPI, FEniCS using PETSc for linear algebra•

Used to build adaptive solvers for turbulent fluid flows.
Non-linear transient problems need to reassemble in eachtime-step•

MPI + UPC, FEniCS using a PGAS based linear algebra
backendSynthetic Matrix Assembly
Matrix reassembly time for Laplace’s equation in 3D on a
unstructured tetrahedral mesh with 317M elements.

We address the strong scalability limits of finite element based
solvers by changing to lightweight one-sided
communication in performance critical, latency sensitivekernels.
Complex unstructured meshes with irregularcommunication pattern during matrix assemblyCompressed Row Storage (CRS) like format optimised for
random insertion during matrix assembly•High latency costs due to message matching, requires alarge amount of elements per core to strongly scale•Row-wise data distribution•Two phase assembly. Insertion of local entries + exchange
of non local entries using remote memget operations.

•Implemented using Unified Parallel C (UPC)•Directory of object representation, allowing for arbitrary
sized global arrays, accessible by all threads

Too costly to rewrite an entire application in PGAS
Replace MPI based linear algebra backend with the newPGAS based backend

Linear Algebra𝑎 𝑢, 𝑣 = & ∇𝑣 ⋅ ∇𝑢 𝑑𝑥'Rows on thread 3

Reassembly times for the momentum and continuity equationsin an implicit LES solver, on an unstructured tetrahedral meshwith 60M elements.

UPC

𝐿 𝑣; 𝑓 = & v𝑓 𝑑𝑥

MPI

Rows on thread 2
Rows on thread 1

UPC (Mom.)

MPI (Con.)UPC (Con.)

1,00

0,100,01
128

Hybrid interface for combining with non PGAS languages.
Access data through opaque interfaces
extern "C" {int jp_mat_init(char *A,uint32_t m,uint32_t n);

typedef struct {shared[] row_stack *shared *a_dir;
row_stack *rs;...

}

} jp_mat_t;

char A[200]; /* sizeof(jp_mat_t) */int jp_mat_init(jp_mat_t *restrict A,
uint32_t m,uint32_t n);

256

512

1024

2048

4096

Cores

Computed Solution

Rows on thread 0

C++ side6144 12288 24576

Incompressible Flow Solver

MPI (Mom.)

eInterfaceuqapO

768

Cores'

jp_mat_init(A, M, N);

3072

0,01FEniCS HPC

Directory•

1536

0,10384••

Runtime (seconds)•A low latency one-sided PGAS based sparse matrix
representation.
•UPC

1,00

Runtime (seconds)

PGAS Based Sparse Matrix Representation

MPIConclusions•

New hybrid parallelisation of FEniCS combining MPI(FEM) with UPC (linear algebra)•Drop-in replacement, possible to switch at runtime•

Completely transparent, end users and solvers based onFEniCS unaffected•

Low latency, one-sided communication can improvescalability of sparse matrix assembly in finite element
based solvers•

Hybrid MPI/PGAS offers a scalable alternative to acomplete rewrite of legacy MPI codes when preparing forexascale

UPC sideNiclas Jansson
PDC Center for High Performance Computing
KTH Royal Institute of Technology
njansson@kth.se