ARCHER Webinar: eCSE 0505 Open source exascale multi-scale framework
Channel: ARCHER UK National Supercomputing Service
Date: 2017-04-20

Anton Shterenlikht, University of Bristol, Luis Cebamanos, EPCC and Lee Margetts, University of Manchester.

Fortran coarrays have been used as an extension to the standard for over 20 years, mostly on Cray systems. Their appeal to users increased substantially when they were standardised in 2010. In this work we show that coarrays offer simple and intuitive data structures for 3D cellular automata (CA) modelling of material microstructures. We show how coarrays can be used together with an MPI finite element (FE) library to create a two-way concurrent hierarchical and scalable multi-scale CAFE deformation and fracture framework. Design of a coarray cellular automata microstructure evolution library CGPACK is described.

We have developed hybrid MPI/coarray multi-scale miniapps from MPI finite element library ParaFEM and Fortran 2008 coarray cellular automata library CGPACK. We show that independently CGPACK and ParaFEM programs can scale up well into tens of thousands of cores. The miniapps represent multi-scale fracture models of polycrystalline solids. The software from which these miniapps have been derived will improve predictive modelling in the automotive, aerospace, power generation, defense and manufacturing sectors. The libraries and miniapps are distributed under BSD license, so these can be used by computer scientists and hardware vendors to test various tools including compilers and performance monitoring applications.

CrayPAT tools have been used for sampling and tracing analysis of the miniapps. Two routines with all-to-all communication structures have been identified a primary candidates for optimisation. New routines have been written implementing the nearest neighbour algorithm and using coarray collectives. Strong scaling limit for miniapps has been increased by a factor of 3, from about 2k to over 7k cores. Excessive synchronisation might be one contributing factor to the scaling limit. Therefore we conclude with a comparative analysis of synchronisation requirements in MPI and coarray programs. Specific challenges of synchronising a coarray library are discussed.