# PaRSEC: Scalability, Flexibility, and Hybrid Architecture Support for Task-based Applications in ECP

PaRSEC
39(1):147-162
©The Author(s) 2025
Reprints and permission:
sagepub.co.uk/journalsPermissions.nav
DOI: 10.1177/10943420241290520
www.sagepub.com/

SAGE

Aurelien Bouteiller<fnref n="1" />, Thomas Herault<fnref n="1" />, Qinglei Cao<fnref n="3" />, Joseph Schuchart<fnref n="1" />, and George Bosilca<fnref n="1" />,<fnref n="2" />

## Abstract

This paper highlights the most significant enhancements made to PaRSEC, a scalable task-based runtime system designed for hybrid machines, during the Exascale Computing Project (ECP). The enhancements focus on expanding the capabilities of PaRSEC to address the evolving landscape of parallel computing. Notable achievements include the integration of support for three major types of accelerators (NVIDIA, AMD, and Intel GPUs), the refinement and increased flexibility of the communication subsystem, and the introduction of new programming interfaces tailored for irregular applications. Additionally, the project resulted in the development of powerful debugging and performance analysis tools aimed at assisting users in understanding and optimizing their applications. We present a comprehensive demonstration of these advancements through a series of benchmarks and applications within ECP and beyond, thereby showcasing the enhanced capabilities of PaRSEC across the diverse architectures within the ECP, providing valuable insights into the runtime system's adaptability and performance across varied computing environments.

## Keywords

Exascale Computing Project, Task-Based Programming, Dataflow, Micro-Tasks, Runtime

## Introduction

The ECP Distributed Tasking at Exascale (DTE) project uses the PaRSEC runtime (Bosilca et al. (2013)) to explore the efficacy of task-based expression of parallelism in tackling the challenges that have emerged with the raise of many-core and heavily accelerated architectures. Under the dynamic conditions that are prevalent in hybrid systems that comprise multiple levels of memory hierarchy, the motion of data between these memory hierarchies becomes a major bottleneck to performance, and renders the execution rate on each individual processing unit (be it a CPU core, or a GPU accelerator) unpredictable, resulting in the previously prevalent programming style based on fork-join parallelism to underperform. Indeed, exascale systems offer a significant change in design, with “fat” nodes with heavy use of accelerators, raising significant concerns regarding the efficiency of application codes on these systems, especially when considering that legacy code would have to adapt to a variety of programming paradigms (e.g., CUDA, RoCM, OneAPI, OpenMP, MPI, Kokkos, etc.), and the difficulty of achieving performance portability when optimizing for a zoo of hardware and software layers under the duress of managing data motion between memory hierarchies. Unlike the currently mainstream MPI+X programming approach, PaRSEC relieves the application developer’s need to manage threads, to perform explicit communication, to handle heterogeneity, load balancing or task delegation, and, more generally, to provide any form of explicit control flow. Instead, the developer can concentrate on exposing the algorithm’s intrinsic parallelism. The dependencies between tasks are expressed by the user using one of the Domain Specific Languages (DSLs), ultimately resulting in the

computation being presented to the runtime as a Directed Acyclic Graph (DAG) where the nodes are the computational micro-tasks, and the edges are the data dependencies. This approach has multiple advantages: as the runtime is in charge of data motion and computations of independent tasks, it can schedule them to achieve overlap between independent actions automatically. As the expression of the algorithm is separated from the mapping onto the resources, the difficult task of optimizing for a given architecture is partly delegated to the system which can use runtime information to balance the load between compute resources, and minimize data movements. While PaRSEC had a proven track record in accelerating dense linear algebra workloads on many-core systems Bosilca et al. (2011a) at the start of the project, significant advancements were still required to enable the full potential on accelerated systems, and demonstrate benefits on more challenging, more irregular classes of applications. The DTE PaRSEC project has been instrumental in proving the applicability of advanced dataflow, micro-tasks programming techniques on recent leadership class hardware. Our results show that relevant, complex, large scale legacy applications can be adapted to the new programming model, and exhibit exceptional performance levels that would be barred when employing

<footnote n="1">¹The University of Tennessee, USA</footnote>

<footnote n="2">²Nvidia Corp.</footnote>

<footnote n="3">³Saint Louis University, USA</footnote>

<footnote>**Corresponding author:**
Aurelien Bouteiller, The University of Tennessee, 1122 Volunteer Blvd.,
Suite 203 Knoxville, TN 37996 USA.</footnote>

<footnote>Email: bouteill@icl.utk.edu</footnote>
traditional programming techniques. As a result, PaRSEC accelerated routines that have been deployed in multiple high-impact applications, including ECP AD projects like NWChemEx Kowalski et al. (2021), SLATE Gates et al. (2019), and international collaborations resulting in PaRSEC accelerated routines being instrumental in the Gordon Bell finalist run on the Fugaku system by the HiCMA team Cao et al. (2022a). These successes have participated in kickstarting the current popularity of task parallism in HPC communities, as well as efforts to distill the concepts in generalist programming languages like C++.

This paper summarizes the activities of the DTE project over its course, including some activities that have been published separately (cited when it is the case), and original results and description of optimization techniques that have not been published before. It is organized as follows: Sec. **Background** presents some background on the runtime system and related works; Sec. **Runtime Capabilities** describes how the PaRSEC runtime enables efficient scheduling on GPU accelerated systems, efficient communication between compute nodes, and the techniques employed to enable interoperability between PaRSEC and legacy code; Sec. **A Runtime with multiple faces: the DSLs** presents the DSLs we created or improved to meet the needs of AD applications; Sec. **Fault Tolerance** describes how the dataflow programming model enables some original fault tolerance techniques to be deployed; Sec. **Task-Based Programming Environment** outlines the development tools we have designed to support users programming micro-tasks, and their integration within ECP; Sec. **Quality Assurance** describes the steps we have taken to ensure that the software is high quality and sustainable; Finally in Sec. **Application Use Cases** we highlight some of the major results and impact from application integrations, before we conclude.

# Background

## The Exascale Computing Project

The Exascale Computing Project (ECP) was launched in 2016 by the U.S. Department of Energy (DOE) to ensure that the software ecosystem would be ready to achieve the science and engineering missions of the DOE using the new generation of leadership computing systems. Three exascale computing systems were to be installed starting in 2022: Frontier at the Oak Ridge National Laboratory (ORNL); Aurora at the Argonne National Laboratory (ANL); and El Capitan at the Lawrence Livermore National Laboratory (LLNL). The main thrust of the ECP effort targeted Application Development (AD), which encompassed 24 application projects and 6 co-design centers in areas including national security, energy, Earth systems, economic security, materials, and data science. Recognizing the significant difficulties that emerging architectures would pose to AD applications, the ECP also made significant investments in Software Technologies (ST), supporting 70 unique software products developed by 35 teams spanning programming models and runtimes, math libraries, data and visualization, and development tools.

## The PaRSEC Runtime

PaRSEC is a generic framework for architecture-aware scheduling and management of micro-tasks on distributed, multicore, heterogeneous architectures. High level APIs and/or DSLs are exposed by PaRSEC so that scientists get a familiar programming environment that preserves their productivity while providing their applications with access to an efficient state-of-the-art hybrid development platform. The first such DSL originally evolved to enable the development of the distributed, asynchronous, accelerator ready DPLASMA linear algebra library Bosilca et al. (2011a); since then, additional DSLs have been designed to expand its relevance to AD projects (see Section A Runtime with multiple faces: the DSLs).

The runtime handles the architecture-aware part, assigns computation loads to the cores, handles data coherence and movement, overlaps communication and computation and uses a dynamic fully-distributed scheduler. The PaRSEC framework includes a runtime system, specialized DSLs for linear algebra and chemistry, support libraries to facilitate the transition of algorithms using legacy programming paradigms, and performance tools to help developers efficiently porting their applications to large-scale hybrid environments. PaRSEC's task-based algorithmic representation and its dataflow runtime provide a clear separation of concerns between the different layers of the software stack, helping different communities to join their efforts in a unique collaborative environment. PaRSEC scheduling strategies are specifically designed for fine-grain tasking. Given such tight time constraints, these strategies must take into account simple criteria (minimizing data movements and maximizing data locality) to reduce the time-to-completion of an algorithm. The PaRSEC communication engine abstracts communications as dependencies between tasks executing in different distributed memory domains.

In Bosilca et al. (2013), we present the rationale behind PaRSEC, its main components, and how they interact. We refer readers to this publication for more details.

## Related Works

This section briefly refers to the task-based runtime systems focusing on data flows. OpenMP OpenMP (2021), a prominent standard for parallel programming on shared-memory architectures, now incorporates task-based functionalities, transforming it into a dynamic, task-centered framework. Parallelism is facilitated through a blend of directives and library routines, which allow for effective task management by the compiler and runtime system. OpenMP also supports heterogeneous architectures Valero-Lara et al. (2021). OmpSs Duran et al. (2009), an enhancement of OpenMP, supports heterogeneous systems, asynchronous tasks, and data dependencies, thereby increasing flexibility. Similarly, COMP Superscalar (COMPSs) Lordan et al. (2014) is designed to ease the development on distributed systems through a programming interface and a runtime system that maximizes application parallelism. IRIS Miniskar et al. (2023); Monil et al. (2023), a task-based runtime designed for extreme heterogeneous computing (e.g., CPU, GPU, DSP, FPGA), supports for today's most important
programming languages (e.g., OpenMP, OpenCL, CUDA, HIP, OpenACC). StarPU Augonnet et al. (2011) offers a framework suitable for environments with distributed, heterogeneous multicore systems. It allows for task-specific kernel annotations and efficient task scheduling and data management by the runtime system. HPX (High-Performance ParalleX) Heller et al. (2013), rooted in the ParalleX model, provides a C++ runtime system tailored for parallel and distributed environments in high-performance computing. Lastly, Legion Bauer et al. (2012) implements a unique approach in its runtime system for distributed and heterogeneous computing architectures. Legion's programming model distinctly separates the definition of tasks and data from their actual mapping onto hardware resources, enhancing automatic parallelism detection and optimizing data locality, thereby improving performance and scalability.

In this work, our focus is PaRSEC Bosilca et al. (2013). It introduces original programming paradigms and supports distributed many-core heterogeneous architectures, often resulting in great efficiency, scalability, and portability.

## Runtime Capabilities

This section discusses the advanced runtime capabilities that were developed or extended in the PaRSEC runtime during the course of ECP to enable higher performance, support ECP applications and ECP platforms, and ensure maintainability and a production-quality sustainable software. The most significant changes can be categorized in two broad efforts: an extensive support for accelerators on one hand, and the redesign of asynchronous communication management on the other.

### Accelerator support in PaRSEC

PaRSEC hybrid capabilities are provided via device managers. A device manager is a separate scheduling entity that manages both the computing and storage capabilities of the accelerator. Any 'normal' PaRSEC computing thread can become a device manager when its scheduler decides to execute an accelerated task: that task is either enqueued in the scheduled tasks for the existing device manager of the target device, or, if no PaRSEC thread is currently managing the target device, the calling thread becomes the device manager for the target device. It will remain that device manager until all work scheduled to the device is complete (see Bosilca et al. (2011b) for a more detailed presentation of the mechanism).

**Fine Grain Device Memory Management:** The device managers handle both task scheduling and data transfers between devices and the host to ensure the correct data version is available when tasks are scheduled. Efficient data movement between accelerators and main memory is crucial for performance. Simplistic approaches like unified virtual memory often lead to poor performance. In many of its DSLs, PaRSEC predicts future data usage and selects data to pre-position in accelerator memory domains. Mechanisms have been added to extend PaRSEC's capabilities, enabling data movement across memory domains and pre-positioning data copies in advance. Special flags mark data for prefetching, indicating the preferred device and whether it

<figure><img src="image_1.png" /><figcaption>**Figure 1.** Performance of a double-precision GEMM, on a Haswell E5-2650 20 cores, and one Tesla V100-PCIE-16GB, for varying problem sizes and using prefetching or memory hints</figcaption></figure>

should remain “hot” after use. The PaRSEC engine initiates data transfers based on task requirements and available space in memory.

The PaRSEC scheduler utilizes a Least Recently Used (LRU) list to keep inactive data in accelerator memory until memory pressure triggers eviction. Introducing an “always hot” property prioritizes the eviction of other data over marked data, preventing data trashing. Overall, pre-positioned data in accelerator memory reduces latency, enhances communication between host and accelerator, and improves performance, particularly when accelerator memory is limited. However, these indications are advisory, and the scheduler may disregard them.

During the course of ECP, the GPU manager has been revamped to handle data transfers and enable various features related to task scheduling and data acquisition. Tasks submitted to the GPU now undergo three distinct steps: data transfer, execution, and eviction. This allows the runtime to process problems larger than GPU memory, as demonstrated in applications like Herault et al. (2021, 2019).

**Data Prefetch Advice for Devices:** In order to improve resource sharing, the programmer can often provide hints to the runtime system related to what device should host a given memory, and when to load a given data. We extended the PaRSEC API to provide memory hints from the user code: using the new API, a user task can prefetch data on a given device, and hint that a given device is the preferred target for a given data. As the accelerator schedulers follow a owner-compute strategy, prefetching a data on an accelerator or pinning it on that accelerator both have the effect of assigning the next task that modifies this data on that device, if a tasklet for the device exists. If the tasklet does not exist, a copy of the data will be acquired by a device for which a tasklet exists, thus the operation remains a hint and users cannot generate incorrect code using the extended API (although a performance hit can be observed if the hints are not used efficiently).

Figure 1 shows the effect for a shared memory General Matrix Multiply (GEMM) operation of the added prefetch and pinning hints. The basic GEMM operation is compared with a version that uses memory hints to improve the scheduling from the tasks executing on the GPUs directly. We also implemented a simple prefetch strategy. To stress the scheduling decisions, a non-square problem is created, where the matrix B is large and square (at 71, 680 × 71, 680,
but A and C are not). We grow A and C until the three matrices are square, and their accumulated size does not fit on the accelerator memory. When the matrices start to overflow the accelerator memory, scheduling decisions become more critical as too much usage of the accelerator bus can deteriorate the performance and ruin scalability. This is what is observed in both cases when the schedulers are left to decide what operations to run without the user providing memory usage hints: as soon as the problem size overflows the accelerator memory (at $N = K = 35,840$), performance start to degrade. When the user code introduces control through memory placement, the performance continue to scale. Although the prefetch does not improve the performance in this case, operations with lower arithmetic intensity should be significantly improved by accurate prefetching orders.

**Direct Device-to-Device Memory Transfers:** This experiment lead the development of new capabilities in the runtime system: to reduce the pressure on the PCI-Express bus, we extended the CUDA driver of PaRSEC to support direct GPU-to-GPU operations, and allow a task that reuses a data currently owned by a nearby GPU to pull the required data directly from it. This change required to break thread safety assumptions in the CUDA PaRSEC drivers, and to extend the data management protocol to take into account multiple co-existing copies of data on multiple devices.

**Support for AMD and Intel GPUs:** As hardware became available, the PaRSEC runtime has been extended with a HIP/ROCM device driver (to support AMD GPUs) and a Level Zero device driver (to support Intel GPUs). The PaRSEC programming interfaces (Parameterized Task Graph (PTG) and Dynamic Tasks Discovery (DTD)) have been extended to support the writing of device kernels in DPC++, allowing them to use the entire Intel OneAPI stack, including in particular OneMKL; the support of ROCm for AMD GPUs did not require any changes in the DSLs.

**AMD GPU support: Hippify and RoCM:** Technically, the AMD GPU support is achieved thanks to the modular approach the core engine of PaRSEC features: GPU device specific management is isolated in a module that gets loaded from the generic GPU engine when the specific device type is detected at runtime. The RoCM device is produced by using AMD provided conversion tools (Hippify) in order to translate the CUDA specific module into a RoCM module. This approach improves maintainability as it avoids code duplication.

**Intel GPU support: Level Zero and SYCL:** The runtime system relies on the low level ‘Level Zero’ Intel library to manage resources: initialize the devices, allocate memory on them, schedule data transfers between the main RAM and the device memory, and betwen devices, and track the progress of computation and data transfers on the devices. To track the progress, Level Zero fences are inserted in command queues, and the GPU manager thread in PaRSEC polls these fences. The PaRSEC device driver allocates level zero command queues and lists, and these level zero objects are converted into SYCL equivalents to be exposed to the DPC++ user code that enqueues computational kernels onto the devices exposed by PaRSEC. This approach has been initially validated on Florentia, using 1 to 4 Intel GPUs,

then on the different pre-Aurora platforms, and eventually confirmed on Aurora itself.

In the ‘Application Use Cases’ Section below, we present performance results with the different accelerators supported by PaRSEC, in applicative contexts.

## Redesign of the communication system

**Concurrent MPI and PaRSEC application:** MPI remains the dominant programming model/communication library employed in today’s high performance applications. Consequently, a vast amount of legacy code and libraries are expected to remain in production for many years; in fact many ECP applications have transitionned toward more effective programming models in a gradual effort that sees the coexistence of multiple components, written in different models. Since a vast majority of existing code employs MPI, interoperability with MPI is thus paramount for any addition to the software ecosystem.

PaRSEC utilizes MPI in its runtime to handle data movement for high-level programs. It supports sequential composition with MPI programs, alternating between SPMD blocks employing MPI directly and those spawning PaRSEC algorithms. We have proposed a method to allow both libraries to operate simultaneously on compute nodes, and evaluated it using an applicative benchmark. This capability is essential for optimizing resource usage, especially with legacy libraries lacking multi-threading support. For instance, PaRSEC can efficiently utilize cores not assigned to the MPI library, enabling independent computational steps. An illustrative example is coordinating PaRSEC’s iterative main computation with legacy libraries like ParMETIS or PETSc for workload redistribution or convergence criterion calculations.

In order to demonstrate and exercise this capability, we designed an iterative stencil algorithm in PaRSEC. This algorithm represents one of the steps of a typical MultiGrid (AMG) method for solving partial differential equations Pei et al. (2020). This PaRSEC algorithm executes a fixed number of smoothing iterations on the input 2D-cyclic matrix, using multiple cores on the distributed system. The synchronization between the main thread MPI computation and the PaRSEC computation is loose. The residual computation in the main thread updates variables that are exposed to the completion callback of the PaRSEC taskpools (i.e., the container object that represents, and tracks the progress of computational tasks scheduled with the PaRSEC runtime).

Figure 2 compares the performance of the two main components of that computation when they are performed concurrently, subtracting one core from the PaRSEC computation to dedicate to the residual computation and sharing the MPI substrate, and when executed in isolation of one-another. As one can see, both operations sustain a minor decrease in performance due to the sharing of the MPI substrate, however, the overall performance shows a strong overlap of the two operations, with the PaRSEC algorithm iteration rate remaining high even when controlled by the MPI residual computation.

In order to show an interoperability between PaRSEC and an MPI library, we designed a tiled-LU factorization with partial pivoting, where the complex part, the panel
<figure><img src="image_2.png" /><figcaption>**Figure 2.** Operating simultaneously an MPI algorithm in the main application thread with a PaRSEC distributed algorithm with a control feedback. Each component executes at approximately 80% the rate it can achieve in a dedicated deployment (i.e., when not sharing resources with the other component), and execution of both components are properly overlapping each other</figcaption></figure>

factorization is implemented using MPI communications, and the update, which is the most computationally intensive and also the highly parallel part, is handled by the runtime system. The tiled-LU factorization of the panel is a tightly coupled operation that requires the lookup of the max value, a swap of lines, a scaling and, an update of the columns on the right. Expressing that operation in a parameterized task graph model is both complex and inefficient. It is complex because there is a large number of dynamic dependencies that are data dependent (due to the max operation over a vector). It is inefficient, as the generated tasks as extremely small and the runtime overheads are therefore exacerbated.

Thus, a hybrid model, where the data-dependent, communication-intensive part with low parallelism is expressed using traditional synchronous methods above MPI, and the highly-parallel predefined dataflow part is expressed with the task system is a palatable approach that can benefit from the advantages of both programming models and thus has potential to maximize the performance.

## A Runtime with multiple faces: the DSLs

PaRSEC is built around a set of DSLs or APIs that aim at enabling its users to express complex DAG of tasks in the way they prefer. There are three main interfaces that ECP applications have used to express their task-based algorithms over the PaRSEC runtime: Dynamic Task Discovery (DTD) is typically a prototyping interface; Parameterized Task Graphs (PTG) is the historical first interface of PaRSEC; and Template Task Graphs (TTG) were introduced by the TESSE and EPEXA NSF projects, and their implementation in PaRSEC was stabilized and made production-ready in the context of ECP.

### Dynamic Task Discovery

DTD is the API that most task-based runtime systems provide today allowing to express the DAG of tasks: each process sequentially iterates over the code that describes the memory accesses made by each task and tags the access with specific access needs; the runtime system part of the DSL builds a DAG representation based on this sequential loop, generating a DAG of tasks that can execute in parallel, but will provide the same result as the sequential algorithm (see Hoque et al. (2017) for details).

<figure><img src="image_3.png" /><figcaption>**Figure 3.** DAG of “hierarchical DAG” Cholesky factorization, 4 × 4 large tiles, CPU task further split into 3 × 3 small tiles</figcaption></figure>

To allow the system to have the choice between multiple implementations of the task (e.g., one running on one of the CUDA accelerators available, or one running on one of the cores available), we have exposed the concept of task class describing the low-level view of the runtime on user defined tasks, which was hidden from the user in the DTD interface. Task classes feature multiple incarnations, that represent the function eventually called by the runtime system to implement the operation of the task. By exposing the DTD task classes to the developer, we expose a mechanism that enables full utilization of any device supported by the PaRSEC runtime, including the GPU, with this interface, and give the runtime system the opportunity to dynamically select which incarnation implements the operation. That selection is guided by multiple strategies that take into account data proximity, current computing load distribution and availability of memory.

### The Parameterized Task Graph DSL

PTG is historically the first interface used in PaR-SEC (see Danalis et al. (2014) for details). In PTG, the programmer provides a synthetic internal representation of the DAGs, via the definition of the dataflow. This allows the runtime to instantiate any part of the DAG of tasks from any task, by locally discovering its local or remote successors. PTG offers a highly scalable interface to describe very large DAGs of tasks.

In addition, PaRSEC supports hierarchical DAG scheduling for distributed systems, where the task granularity is dynamically adjusted to adapt the algorithm to the available computational resource on the node, and to match their computational capabilities (see Wu et al. (2015)). This hierarchical method can be generalized to any number of hierarchies, but for the sake of the explanation we will consider two levels of hierarchy, GPU and CPU.

When a large grain task is scheduled onto a CPU, the “hierarchical DAG” capable runtime decomposes the CPU workload into a finer grain parallelism that is more adequate for this type of execution unit. The creation of the metadata representing the fine grain DAG happens online; no preprocessing or static decomposition is required. The runtime engine creates a local data descriptor, a different view of the input submatrix representing the large tile divided into smaller tiles. A new DAG is created to represent the fine
grain decomposition of the task’s algorithm applied on these smaller tiles. Tasks operating on large tiles that are scheduled for execution on the CPU are divided into finer grain tasks; the shape of the resulting multi-level graph for the Cholesky factorization is presented in Figure 3. This technique is used heavily in PTG applications that leverage GPU acceleration on ECP machines, as we discuss in the ‘Application Use Cases’ Section below.

## The Template Task Graph DSL

TTG is a task-based programming approach designed to simplify the management of distributed task-based systems targeting applications with irregular data access patterns. It can be viewed as marrying the ideas of flow programming models (e.g. DTD) with the key innovations of PTG in which each edge represents a flow of data associated with a *parameter* identifying the particular data and, equivalently, the receiving task. Simplistically, such a *parameter* represents a loop index or data structure coordinate (e.g., integer tuple addressing elements of a tensor). Thus, each edge and vertex in PTG encodes several edges and vertices in a DAG of tasks, allowing potentially massive task DAGs to be represented compactly as well as instantiated across a narrow wave front only as needed for execution. Algorithms of dense linear algebra are naturally expressed within the PTG as a workflow over mutable data that fully captures the lifetime of a datum including whether it is created, read, written, modified, or consumed by a given task. This parameterization combined with the deep understanding of what tasks are doing with data helps reduce resource utilization and data motion, and enables efficient placement and scheduling through use of temporal/spatial locality.

TTG extends the idea of PTG by generalizing the notion of parameters to arbitrary types and enabling data-dependent selection of task dependencies, which allows to dynamically build the DAG of tasks depending on computations within predecessor tasks. The TTG implementation replaces the standalone DSL for specifying PTG in PaRSEC by a high-level programming API realized as a modern C++ library (the 2020 ISO standard of C++ is used). The use of modern C++ allows for type information to be utilized to ensure correctness when constructing graphs as well as for optimizations (e.g. consuming data passed by rvalue references). Lastly, TTG also introduces some concepts from flow programming models, such as programmable terminals, that were not part of PTG. Thus TTG is a major advance of the successful idea of PTG towards general-purpose computation. However, the general-purpose character of TTG creates new challenges in retaining all of the optimizations feasible within the PTG. Enabling TTG to exploit the full capabilities of PaRSEC runtime is the focus of ongoing work (see Schuchart et al. (2022b,a) for a more complete description of the API and its implementation over PaRSEC).

Accelerated routine support has been introduced in TTG via the definition of asynchronous tasks that make use of the C++ coroutines implementations (see Fertig (2021)). Contrarily to the other interfaces of PaRSEC for accelerated tasks, TTG users require a level of control of the task life cycle that imposed a modification of the PaRSEC

GPU management engine: a TTG accelerated task may dynamically allocate new data on the device, explore its result and schedule more work on the device based on that host evaluation. Doing so with the traditional interfaces requires to split the work in as many GPU and CPU tasks that alternate their execution, introducing significant scheduling and data movement overheads. The TTG asynchronous task alleviates these issues by enabling fine grain control from the programmer with minimal overheads.

<caption>Listing 1: POTRF task written in TTG.</caption>

```c
auto potrf_fn =
    [=] (const Key1& key, MatrixTile<T>&& tile_kk)
    -> ttg::device::Task {
        /* allocate temporary workspace data */
        int info, K = key[0];
        int wsize = device potrf workspace_size tile_kk;
        auto devWS =
            ttg::make scratch (ttg::scope::Allocate, wsize);
        auto devInfo =
            ttg::make scratch (&info, ttg::scope::Allocate);
        /* the workspace and the devInfo must
         * be device-level pointers */
        co await ttg::device::select (tile_kk.buffer(),
                                     devWS, devInfo);
        /* everything is on the device, submit kernel */
        cusolverDnDpotrf (cusolver handle(),
                          CUBLAS_FILL_MODE_lower,
                          tile_kk_COLS(),
                          tile_kk.buffer() .current_device_ptr(),
                          tile_kk.lda(),
                          devWS, work_size, devInfo);
        /* wait for the kernel and transfer back info */
        co await ttg::device::wait (info);
        /* check for success */
        if (!info) throw "POTRF kernel failed!";
        /* broadcast tile to TRSM and result task */
        co await ttg::device::broadcast < 0, 1> (
            std::make_tuple (Key2 (K, K),
                              make_trsm_list (K)),
            std::move (tile_kk));
    };
return ttg::make tt < ttg::ExecutionSpace::CUDA>
    f potrf_fn,
    ttg::edges (ttg::fuse (input, input disp)),
    ttg::edges (output_result, output_trsm);
```

Listing 1 shows the POTRF kernel as used in the tiled Cholesky factorization on CUDA devices, implemented in TTG. The task receives a *key* identifying the location of the tile in the matrix and the tile itself. The task function first create scratch data that is required for the cuSolver kernel (Lines 5–10). It then suspends itself to allow the runtime to select a device based on the task inputs and make its inputs available on that device (Line 13). The kernel is submitted (Line 16) before the task again suspends itself (Line 23) waiting for the kernel to complete and the *info* to be transferred back to the host, where it is checked for success. The last step is to issue a broadcast of the result to the TRSM tasks in the column of the matrix and a task storing the tile into the output matrix (Line 27). For that purpose, the function `make_trsm_list` returns a vector of keys identifying the TRSM tasks. Lastly, the task function is passed to `make.tt` together with input and output edges (not detailed here) to create a template task that can then be connected to other template tasks and instantiated at runtime, using the `Key1` as identifier.

## Fault Tolerance

Fault tolerance (FT) is crucial to run large-scale jobs. As systems move into the exascale era, failures continue
<figure><img src="image_4.png" /><figcaption>**Figure 4.** Strong Scaling of the ABFT Cholesky operation</figcaption></figure>

to impact the execution of applications. This section describes the research towards fault tolerant operations with the PaRSEC dataflow programming model. Silent Data Corruptions (SDCs) are memory corruptions that are not detected by the hardware, and that do not escalate into an error detected by the Operating System or the runtime system. They happen in large scale systems because of the sheer size of the data exposed into RAM, transiting over buses, or being modified by computations. The specificity of SDCs is that they cannot (by definition) be detected without taking additional steps. In order to increase the reliability of the computations, we have extended PaRSEC with tools to allow applications coding application-specific resilience for SDCs.

In Cao et al. (2015), an Algorithm-Based Fault-Tolerant version of the Cholesky Factorization is expressed in the PTG DSL over the PaRSEC runtime. We show how each tile of the matrix is extended to feature their own checksum. The linear algebra operation is then applied on the extended tile, modifying both the tile itself and its checksum. Because of the mathematical properties of the operation, the checksum relationship is preserved by this extended application.

Thus, if a SDC occurs during an operation, the checksum updated by the operation will not match (with very high probability) the checksum computed again on the input data. As a consequence, this provides an application-specific way to detect the occurrence of the failure, and the update task can be re-scheduled, restarting from its initial data, if such an error is detected.

This operation incurs computational overheads: Figure 4 shows the performance of the reference Cholesky factorization (without failure), compared to the version extended for ABFT, for an increasing number of nodes. As each task adds a checksum computation before and after the operation itself, a small but significant overhead is observed.

Aside from self-checking tasks, the PaRSEC runtime has also been extended to support task-based checkpointing, and the inclusion of user-defined checkpointing strategies. The system is capable of partial rollbacks, reconstructing the missing sub-DAG of tasks based on the asynchronous checkpoint wave constructed by the location of the distributed task checkpoints. In Cao et al. (2015), we used this capability to study simple checkpointing placement heuristics in task-based systems.

## Task-Based Programming Environment

PaRSEC is not only a task-based runtime system: it also comes with an ecosystem of tools to analyze, visualize, and debug the task-based applications. These tools are paramount

for an effective programming environment, as task-based systems can become hard to understand. During the course of ECP, we have significantly extended the tools available to users.

**PAPI-SDE Interface:** PAPI-SDE (Danalis et al. (2019)) is an ECP effort of the ExaPAPI project. PAPI stands for Performance API, and SDE for Software Defined Events. PAPI-SDE provides an interface for libraries to publish performance counters that are read through the traditional PAPI interface. This is a way for the PaRSEC runtime system to provide information to users relating to internal behavior of the runtime, without enabling a full-fledge trace. Moreover, as PaRSEC already included a PAPI module inside its own profiling system, this provides an alternative way to PaRSEC users to get additional information about the behavior of the runtime system.

We defined a set of PaRSEC-related performance counters: High-level counters present A) the amount of temporary memory that has been allocated and is in use at a given moment by the runtime; B) How many tasks have been discovered, are ready to execute, are retired (are done executing); C) How many messages are sent and received, how many bytes have been transmitted at this time. More raffined counters also present to the user how many tasks are pending in scheduling queues. This is different for each scheduler, and these counters have been integrated with the Modular Component Architecture of PaRSEC to allow a variable number. At the same time, to present all the details to the user, but still allow access to simpler synthetic representations, we converged with the ExaPAPI developers to a concept of counter groups that allows to define accumulated counters that synthetically represent the status of virtual levels of waiting queues. See Jagode et al. (2019) for a more detailed description.

Figure 5 presents some of the counters read through PAPI during the execution of a Cholesky factorization of size 12000 × 12000 on a 20-cores single node machine. On the left y-axis, we present the number of pending tasks, as exposed by the PaRSEC PAPI-SDE counters of the scheduler, and the difference between the number of Ready tasks and the number of retired tasks as exposed by the high-level PaRSEC PAPI-SDE counters. On the right y-axis, we present the number of retired and ready tasks as exposed by the same counters. The high-level counters are monotonistically growing: as the execution proceeds, the number of tasks that became ready and that executed only increases. The difference between the two should represent a good approximation of the number of pending tasks; the only difference between the two is the number of executing tasks which are not counted as ready, retired or pending. The figure shows that the difference matches the number of pending tasks; it also shows that the number of pending tasks increases a lot at the beginning, to then oscillate between 0 and 600. This is due to the initial explosion of parallelism, when many tasks become ready in the few first steps of the execution. This information can guide tuning for the user.

**OTF2 Tracing Tool:** Open Trace Format 2 (see Eschweiler et al. (2012)) is the format of choice used by legacy performance tools like Vampir Trace (see Schöne et al. (2011)) to visualize performance counters and progress of
<figure><img src="image_5.png" /><figcaption>**Figure 5.** Dense Cholesky Factorization (LLT) with PaRSEC; Measuring PAPI-SDE counters as the factorization progresses</figcaption></figure>

parallel applications. PaRSEC uses its own tracing format and tool chain to provide detailed information to the users, however there is value into providing easy access to this information through existing tools. We developed an alternative tracing mechanism in PaRSEC based upon the OTF2 library to provide traditional OTF2 traces of a task system execution.

**Debugging tools for task-based applications:** We developed an infrastructure building upon the python extensions to GDB to provide a better insight for the debugging information provided by GDB when running a PaRSEC based application. When using GDB (or another debugger) to debug a PaRSEC based application, users could set breakpoints within their own code, but observing the backtrace when reaching these breakpoints would show the entire call stack, including the runtime system levels, making the interpretation complex. Moreover, extracting information from the runtime system w.r.t. the status of tasks (when they become enabled, when specific input dependencies become ready), data (why is a given data pulled on a given device, where does it reside), or progress (what tasks are not ready at a given time), was not achievable by users who do not have an intimate knowledge of the internal DAG representation of PaRSEC.

The goal of these GDB extensions is to give a cleaner, more actionable view of the status of the task-based application at a given time: we provided new GDB commands to explore the current taskpool, list tasks that are enabled or not, provide breakpoints when a new task is discovered, when it is scheduled, when its data becomes ready. The infrastructure is nascent, as it stabilize and we receive user feedback about the existing and missing features, it will be extended as needed, but the current version already provides the basic functionality to much user friendly debugging and its design is made to allow for future extensions. Table 1 shows a list of commands provided to the user

## Quality Assurance

A crucial part of the ECP effort has been on providing sustainable and high quality ST solutions that can sustain AD projects long term. In this section, we discuss the techniques we have employed to ensure that the PaRSEC runtime achieves these goals and is easily available to AD users. A byproduct of this effort is a set of processes and techniques that can be employed by HPC software developers at large to enhance the quality of their software.

<table><caption>**Table 1.** List of GDB commands provided by the extension</caption><tr><td>contexts</td><td>Lists current known PaRSEC contexts, marking the selected one with a star</td></tr><tr><td>select-context</td><td>Change the currently active PaRSEC context (w.r.t. debugging commands)</td></tr><tr><td>taskpools</td><td>Lists current taskpools, marking the selected one with a star</td></tr><tr><td>break-on-new-tp</td><td>Inserts a breakpoint when a new taskpool is created</td></tr><tr><td>select-taskpool</td><td>Change the current active taskpool (w.r.t. debugging commands)</td></tr><tr><td>tasks [type]</td><td>Lists known tasks for the currently active taskpool. Type is a selector that can show only a given type of task: ready, scheduled, retired, not-ready, not-enabled</td></tr><tr><td>break on [subcmd]</td><td>Inserts a breakpoint that will trigger if the subcommand subcmd matches. Current subcommands include specifying a task, when it is scheduled, executed, retiring, or becomes enabled. Tasks can be specified with wildcards or their name</td></tr></table>

**Code Discipline and Continuous Integration:** PaRSEC is primarily hosted on github. New features or bugfixes take place in personal forks of the main repository; code is integrated by a pull-request workflow. Pull Requests (PR) require approval and review by at least two core developers. Automated tests (GitHub actions) help ensure that each PR passes the testing harness (ctest based set of tests that cover unit testing, and application mockup testing, including accelerator testing).

In addition to the continuous integration tests, we have designed specific scripts that enable the rapid deployment of PaRSEC and its testing harness on ECP pre-eascale systems (Spock, Crusher to represent Frontier and AMD GPUs, SunSpot to represent Aurora and Intel GPUs). These tests are not fully automated (due to the difficulty of interfacing the secure enclave at DOE site with an external GitHub) but the scripts greatly simplify the validation of the software on ECP systems prior to releases.

**Availability and releases:** The main branch of PaRSEC is sufficiently validated to represent a rolling release. PaRSEC employs a semmantic versionning scheme, currently the rolling release represents the preview of the API v4. Meanwhile, API v3 represents the current stable release, available as an archive in a GitHub release. In addition, PaRSEC can be installed from Spack Gamblin et al. (2015), and E4S Heroux (2019) since release v2021.02. We have published standalone acceptance testers to the E4S testing repository; these testers focus on validating that end-users can use the PaRSEC DSLs to build and run their application using the E4S default installation.

## Application Use Cases

In this section, we present the performance obtained by a variety of applications over PaRSEC. PaRSEC targets
performance portability and algorithm reuse on platforms,
and the evaluation on a large variety of platforms and
architectures reflects this goal.

**Experimental Settings:** The experiments are mainly conducted on five large systems: Shaheen II, HAWK, Fugaku, Summit, and Frontier. Shaheen II is a Cray XC40 cluster with 6,174 compute nodes. Each node is equipped with two 16-core Intel Haswell CPUs at 2.30 GHz and 128 GB of DDR4 main memory. HAWK, an AMD-based system, includes 5,632 compute nodes, each with two 64-core AMD EPYC 7742 CPUs at 2.25 GHz and 256 GB of main memory. Fugaku features 158,976 compute nodes, each with a 48-core A64FX CPU at 2.2 GHz and 32 GB of HBM2 memory. Summit, a GPU-based IBM system, consists of 4,356 compute nodes, each with two 22-core Power9 CPUs at 3.07 GHz and 256 GB of main memory; each CPU is paired with three NVIDIA Tesla V100 GPUs. Lastly, Frontier, a GPU-based AMD cluster, has 9,408 compute nodes, each comprising a 64-core AMD Optimized 3rd Gen EPYC CPU and four AMD MI250X GPUs, with access to 512 GB of DDR4 memory. In addition, Some of the results presented below were obtained on the pre-production ECP supercomputers, in particular on Spock, Crusher (pre-production versions of Frontier with similar but less nodes), and Sunspot (a pre-production cluster with early versions of the Aurora software development kit); Sunspot nodes have 2x Intel Xeon CPU Max Series (codename Sapphire Rapids) and 6x Intel Data Center GPU Max Series (codename Ponte Vecchio).

Tensor Contraction in LibChemist

LibChemist is part of the NWChemEx ECP application;
it consists in a large collection of computational tools
for chemistry, among them a set of tensor operations
implemented over TILEDARRAY Peng et al. (2016) within
the Massively Parallel Quantum Chemistry framework
(MPQC, see Peng et al. (2020)). TILEDARRAY is itself
implemented over MADNESS, which we integrated over
PaRSEC during the course of ECP.

The whole stack integration showed some performance
advantage of using PaRSEC as the task-based runtime
system for MADNESS tasks on early hardware systems
(Spock and then Sunspot). In this integration, the
TILEDARRAY algorithms are kept unchanged. We compared
the performance of the native MADNESS task management
with the port over PaRSEC, on the Spock system, using
a dense matrix-product application. The evaluation showed
that both backends reach the same performance for very
large block sizes, up to the theoretical peak of the machine;
but in the lower range of block sizes, between 64 × 64
and 128 × 128, the PaRSEC backend is more efficient at
scheduling tasks in a way that favors data reuse and the
performance increases significantly for that range, and the
runtime system exhibits lower task management overheads.
This has significant impact for the final application. In the
final application, which features irregular data representation
and sparsity, the block size cannot be easily tunned, and the
task-based system can only schedule the tasks as they are
provided by the application.

<figure><img src="image_6.png" /><figcaption>Figure 6. Performance of the ABCD tensor contraction for $C_{65}H_{132}$ on Summit</figcaption></figure>

In a subsequent phase, we migrated certain tensor
contraction algorithms from TILEDARRAY to the PTG
DSL. Initially, this migration targeted the dense scenario
Herault et al. (2019), followed by adaptation for the block-
sparse situation Herault et al. (2021). The latter adaptation
is integral to the Coupled-Cluster Singles and Doubles
(CCSD) approach. Within the CCSD method, the intricate
tensor algebra can be streamlined by focusing on a single
representative term, commonly referred to as the ABCD
term, which typically encompasses the lion’s share (often
90% or more) of the computational workload: $R_{ab}^{ij} = \sum_{cd} T_{cd}^{ij} V_{ab}^{cd} + \dots$. Here, the elements of tensor $T$ denote the
model parameters subject to iterative refinement (typically
spanning 10-20 iterations) to nullify tensor $R$. Tensor $V$
remains constant across iterations. Given that the index
ranges scale with the system size $N$, each tensor carries
a space complexity of $N^4$, while the operation entails a
complexity of $N^6$.

This tensor contraction operation can be conceptualized
as the multiplication of matrix $T$ (where fused indices $ij$
and $cd$ serve as row and column indices, respectively) with
square matrix $V$ (with $cd$ and $ab$ as row and column indices,
respectively). In practical terms, the range of unoccupied
indices $(a, b, c, d)$ is typically 5-20 times larger than that of
occupied indices $ij$. Consequently, the transposed matricized
tensors $T$ and $R$ manifest as tall-and-skinny matrices,
characterized by aspect ratios ranging from 25 to 400.
Additionally, $V$, $T$, and $R$ all exhibit block-sparse matrix
properties, with densities potentially dipping as low as 0.3%.

Figure 6 illustrates the strong scaling performance achieved originally on Summit across three distinct tilings of the ABCD tensor contraction, corresponding to the $C_{65}H_{132}$ molecule. This molecule serves as a model for applications involving 1-dimensional polymers and quasi-linear molecules, such as certain proteins. The selection of the def2-SVP AO basis reflects the usage in medium-precision simulations within the realms of chemistry and condensed phase phenomena.

The molecule's 1-dimensional structure induces signifi-
cant sparsity in its representations, with matricized forms
exhibiting densities ranging from 3% to 17%, contingent
upon the tiling strategy and the specific matrix involved. The
figure indicates that, on the whole, performance continues to
improve up to 108 GPUs, with completion times clocking
in at less than a minute, even for the most finely grained
scenario.

Figure 7 shows the performance obtained on a synthetic
case, on one node of Sunspot, the pre-Aurora machine,
for the same implementation. We varied the input matrices
<figure><img src="image_7.png" /><figcaption>**Figure 7.** Block-Sparse GEMM on one node of Sunspot (6 Intel GPUs), tiles of random size in [64, 2048]² double elements, max. 0.5% matrix density</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>**Figure 8.** DPLASMA versus ScaLAPACK memory layout.</figcaption></figure>

(randomly generated), and for each random input, we compute the amount of floating point operation to execute (on the x-axis). We then measure the performance of the operation in GFlop/s on the y-axis. The density is bound to 0.5%, which represents an extreme case for the algorithm, and the tile size was irregular, with the smallest size 64 and the largest 2,048, in both dimensions. The performance measured shows a good strong scaling capability, as long as the overall number of floating point operations is large enough to overlap some of the communication. Yet, the problem is sparse by nature, and the performance is bound by the I/O necessary to move the data to and from GPUs.

As a result of the stabilization effort of ECP, MPQC is now porting its tensor contraction algorithms on top of TTG, the API of PaRSEC that fits best the challenges of sparsity and data-dependency of these algorithms.

## Dense Linear Algebra

**DPLASMA and ScaLAPACK over PaRSEC:** ScaLAPACK has been the state of the art for computing large dense linear algebra operations on distributed systems. To ease legacy application migration, we extended the PaRSEC library and the DPLASMA algorithms have been adapted to support the ScaLAPACK memory layout Cao et al. (2020a, 2021a). DPLASMA provides a wrapped version of the ScaLAPACK API. The wrapper hides the PaRSEC API from the application while it constructs the structures necessary for the operation with matrices represented on the ScaLAPACK memory layout. This approach reduces to a minimum the changes that need to be performed on the ScaLAPACK application.

The differences between the two memory layouts are illustrated in Figure 8. DPLASMA operates over tiles contiguously stored in memory in column-major order, while in the ScaLAPACK memory layout the entire local matrix is stored in column-major order, thus, providing sparse tiles (the columns that comprised a tile are not contiguously stored in memory). The modifications performed on PaRSEC and the DPLASMA preserve the

<figure><img src="image_9.png" /><figcaption>**Figure 9.** ScaLAPACK versus ScaLAPACK over PaRSEC, performance on 9 to 100 Summit nodes, PDPOTRF factorization, 100,000 by 100,000 matrix</figcaption></figure>

operation over contiguous tiles, those originally supported by DPLASMA, while enables the usage of sparse tiles as provided by the ScaLAPACK memory layout. Thus, avoiding the performance and memory overheads that would be introduced with a straightforward implementation based on conversion between the two formats. Instead, we enable DPLASMA algorithms to operate seamlessly over the two different memory layouts. In the implementation, tasks consuming tiles stored on local memory operate using the memory layout provided as an input, whether it is a contiguous or a sparse format. On the other hand, when data needs to be sent to tasks running on remote nodes, the tiles are compacted on the appropriate contiguous format by using the MPI datatypes, minimizing the transfer cost and the memory overhead on the receiver side. In addition, these extensions also take into account the usage of submatrices and the possibility of non completed tiles on the borders of the matrix that is introduced by the ScaLAPACK memory layout.

Figure 9 presents the performance of three implementations of the PDPOTRF factorization (distributed factorization of a double precision symmetric matrix), on 9 to 100 nodes of Summit, for a fixed problem size of 100,000 by 100,000:

* CPU - ScaLAPACK-native represents the performance obtained by the ESSL library provided on the machine, using the CPUs only. No library available on Summit provide a ScaLAPACK API with GPU acceleration.

* CPU+GPU - DPLASMA-native represents the performance obtained by the DPLASMA library, written natively in the PTG DSL on top of PaRSEC, and working on the tile layout. It exploits both CPUs and GPUs.

* CPU+GPU - ScaLAPACK o/DPLASMA represents the performance obtained by the DPLASMA library and working on the ScaLAPACK memory layout. It exploits both CPUs and GPUs.

The figure shows that DPLASMA-based runs, that can exploit GPUs transparently, provide a much higher performance than CPU-based runs. This illustrates how an application written for ScaLAPACK can transparently exploit the GPUs of Summit.

**SLATE integration:** SLATE Gates et al. (2019) is another ECP ST project that is intent on designing a modernized, C++ based interface to execute dense linear algebra operations on accelerated, distributed memory systems. While also employing the dataflow model, SLATE differs
<figure><img src="image_10.png" /><figcaption>**Figure 10.** SLATE, SLATE-PaRSEC, DPLASMA: Comparison of performance on one to four GPUs, for the PDPOTRF factorization, on a double precision matrix of variable size</figcaption></figure>

from DPLASMA in the granularity at which the task graph is created (the graph is formed from macro operations that may span multiple tiles), and it does not use a micro-task runtime, instead implementing the task execution directly using Open MP tasks.

We investigated the benefits that can be obtained from porting an algorithm that is already taskified from a custom implementation to the PaRSEC runtime. To that end, we introduced an intermediate capture layer in the SLATE software to redirect the execution of SLATE tasks from their normal Open MP executors, and instead transform them automatically into PaRSEC tasks, that are thus delegated and scheduled by the PaRSEC runtime.

Figure 10 shows the performance measured using SLATE and DPLASMA on a single node, multi-GPU system, from one to 4 NVIDIA V100 GPUs, when computing the DPOTRF factorization on matrices of variable size:

* PaRSEC device, *x* GPU represents the performance of SLATE over PaRSEC using *x* GPUs

* SLATE, target=Device represents the performance of SLATE using its own GPU target mechanism, using 1 GPU

* DPLASMA, *x* GPU represents the performance of the DPLASMA library written natively over PaRSEC through the PTG DSL.

At the time of this evaluation (since then resolved), the SLATE native GPU driver could not support more than one GPU per process; on one GPU, it achieves about 60% of the performance of the PaRSEC device for SLATE to exploit this GPU, which obtains the same performance as the DPLASMA native implementation for the same routine. When considering more GPUs per process, the PaRSEC device for SLATE increases its performance with each GPU, achieving better performance than the native DPLASMA implementation when the problem size becomes too large. The native DPLASMA implementation suffers from a lack of control flow when running on the limited memory of GPUs, that constrains its performance. The algorithms in SLATE naturally embed this control flow, thus allowing the Parsec device for SLATE to achieve better performance.

Overall, the PaRSEC-SLATE integration demonstrated that using a dataflow runtime is beneficial in that it separates the hardware concerns for the linear algebra implementors, while at the same time achieving higher level of performance due to the higher focus on optimization that has been put into the PaRSEC runtime compared to an ad-hoc task executor implementation in Open MP.

<figure><img src="image_11.png" /><figcaption>**Figure 11.** LLL Cholesky on pre-Frontier, multi-gpu scaling</figcaption></figure>

**Dense Linear Algebra on GPU systems:** We have validated the performance of PaRSEC on a variety of accelerated systems. In this section, we report the level of performance on ECP relevant application benchmarks that employ GPU acceleration, in particular when considering algorithms initially developped for the CUDA backend, and now executing on AMD hardware. To this end, we expanded the DPLASMA over PaRSEC implementation to execute over RoCM devices for some of the most impactful operations (e.g., LLL Cholesky Factorization, GEMM, etc.)

In Figure 11 we compare the performance of DPLASMA with the native RocSolver implementation of the Cholesky factorization on the Spock RoCM pre-Exascale testbed. On 1 GPU, the PaRSEC based implementation outperforms the native RoCM implementation severalfold thanks to the adaptable dataflow implementation of the algorithm in DPLASMA, which overcomes the low efficiency of the RoCM POTRF kernel by keeping the occupancy of the GPUs high with GEMM kernels. Overall, the RoCM port of DPLASMA execute at a similar efficiency (portion of peak) as on NVIDIA based systems, and scales to multiple GPUs, despite targeting new hardware with no platform specific tuning.

In Figure 16 presents the performance of the LLL Cholesky factorization on Frontier. In the interest of space, the figure contains dense and mixed-precision results; for now we focus on the dense FP64 double precision case (DP: dashed lines). Looking at a weak scaling scenario (rightmost point in each curve), the performance evolves from 225 TFlop/s on 4 nodes to 913 on 16 nodes, to 3 PFlop/s on 64 nodes, which represent near linear scaling.

## Applications: Geospatial Modeling and 3D Unstructured Mesh Deformation

This section shows the enhanced efficiency of PaRSEC at scale in addressing linear systems, alongside operations involving dense, low-rank, mixed-precision, and sparse matrices. More results are available in HiCMA-PaRSEC, the Hierarchical Computations on Manycore Architectures (HiCMA) library based on PaRSEC, which is designed to redesign existing dense linear algebra libraries to utilize the data sparsity of matrix operators introduced by Geospatial modeling and 3D unstructured mesh deformation applications Abdulah et al. (2021); Cao et al. (2019, 2020b, 2021b, 2022a,b, 2023, 2024); Mor et al. (2023).
**Geospatial Modeling:** Gaussian processes (GPs) are utilized in machine learning and Bayesian inference for modeling and predicting complex systems. GPs excel in spatial statistics due to their capability to model spatially correlated data under the assumption of stationarity. They are applied to phenomena like temperature and air quality predictions. Maximum likelihood estimation (MLE) is commonly used with GPs to model geospatial data. The GP model specifies mean and covariance functions, defining spatial relationships necessary for evaluating the likelihood of observing spatial data given model parameters. Consider a Gaussian random field $Z(s)$ with observations $Z = \{Z(s_1), \dots, Z(s_n)\}^\top$ at locations $s_1, \dots, s_n$ in $\mathbb{R}^d$. The field is assumed stationary and isotropic with a mean of zero. The covariance function $C(h; \theta)$ depends on the distance between locations, represented by a covariance matrix $\Sigma(\theta)$ that is symmetric and positive definite. Statistical inference is based on the Gaussian log-likelihood function: $\ell(\theta) = -\frac{n}{2}\log(2\pi) - \frac{1}{2}\log|\Sigma(\theta)| - \frac{1}{2}Z^\top\Sigma(\theta)^{-1}Z$. The modeling process computes $\hat{\theta}$ by defining the covariance function $C(h; \theta)$. We consider two functions:

**The 2D/3D Squared Exponential Covariance (2D/3D-sqexp)** $C(h; \theta) = \sigma^2 \exp(-h^2/\beta)$, where $h = \|h\|$.

**The 2D Matérn Covariance (2D-Matérn)** Defined as $C(h; \theta) = \sigma^2(2^{1-\nu}/\Gamma(\nu))(h/\beta)^\nu K_\nu(h/\beta)$.

**3D Unstructured Mesh Deformation:** Simulation of fluid-structure interactions with 3D moving bodies necessitates large mesh deformations. The Radial basis functions (RBF) technique, an interpolation method, generates high-quality adaptive meshes but requires solving computationally demanding large dense linear systems. RBF defines internal volume node displacement from boundary node displacement. An interpolation function for displacement $d$ over the domain is approximated by $d(x) = \sum_{i=1}^{n_b} \alpha_i \phi(||x - x_{bi}||) + p(x)$, where $x_{bi}$ are known boundary nodes, $p$ is a polynomial, $n_b$ is the boundary node count, and $\phi$ is a basis function. Coefficients $\alpha_i$ and polynomial $p$ are determined by conditions $d(x_{bi}) = d_{bi}$. The system's constraints are $\sum_{i=1}^{n_b} \alpha_i p(x_{bi}) = 0$, ensuring a unique interpolant if $\phi$ is conditionally positive definite. For basis functions of order $m \le 2$, a linear polynomial is used. Focusing on Gaussian RBF with global support, $\phi(r) = \exp(-r^2)$, where $r$ is the Euclidean distance and scaled by $\delta = \frac{1}{2} \min(||x - x_{bi}||)$ to moderate matrix condition numbers. Global support functions enhance solution accuracy by considering all mesh point interactions, albeit at the cost of dense matrices. Mesh reordering via Hilbert Space Filling Curves is utilized to improve spatial locality, compression rates, and reduce arithmetic complexity De Boer et al. (2007).

**Tile Low-rank Matrix Computations:** Figure 12 illustrates the results for extreme-scale tile low-rank (TLR) experiments, encompassing matrix sizes up to 42M geospatial locations and utilizing various node configurations for 3D-sqexp and 2D-sqexp. Each data point on the plot signifies the factorization time for the largest matrix that can be processed on a given node configuration, dictated by the memory capacity of all participating nodes. In this figure, each data point is characterized by three attributes: (1) the number of nodes, (2) the matrix size, and (3) the execution

<figure><img src="image_12.png" /><figcaption>**Figure 12.** The performance results for the largest matrices that fit in memory for 2D-sqexp and 3D-sqexp</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>**Figure 13.** Extreme scale performance on Shaheen II</figcaption></figure>

time. This provides insights into performance and scalability for various node counts. With 2D-sqexp, scalability extends up to a 32M matrix on 512 nodes, nearly 10× larger than previous capacities achieved with HiCMA-StarPU Akbudak et al. (2018). On 1024 nodes, using 32,000 cores, it scales up to a 42M matrix. 3D-sqexp, demanding more memory and computational power due to higher ranks and greater rank disparity, showcases scalability up to 42M geospatial locations on 4096 nodes utilizing 130,000 cores. These matrix sizes are particularly pertinent for computational statisticians, representing realistically large datasets previously unattainable.

**Sparse Tile Low-rank Matrix Computations:** To examine scalability for 3D unstructured mesh deformation, we deploy HiCMA-PaRSEC to larger matrices and more processors, aligning with the typical standards in the relevant scientific domain. The results, shown in Figure 13, include matrix sizes up to 52.57M and up to 2048 nodes. Each matrix size represents a strong scaling test, and each node count represents a weak scaling test. It is important to note that a matrix size of 52.57M is not commonly used in studies of TLR matrix computations. HiCMA-PaRSEC can factorize such a matrix size in 36 minutes, marking significant progress in TLR matrix approximation for extreme-scale 3D unstructured mesh deformations.

**Dense Mixed-Precision on Homogeneous Architectures:** This section investigates the performance of mixed-precision Cholesky factorization on large-scale homogeneous architectures. Both Hawk and Shaheen II lack support for half precision (HP, FP16), thus Figure 14 presents only the mixed double and single precision results for configurations 100D (100% double-precision (DP, FP64)), 10D:90S (10% DP and 90% single-precision (SP, FP32)), and 100S (100% SP). It also includes the relative speedup of 100S and 10D:90S over 100D using 1536 Hawk nodes and 4096 Shaheen II nodes. On Shaheen II, the speedup from 10D:90S
<figure><img src="image_14.png" /><figcaption>**Figure 14.** Performance of mixed double/ single on 1536 Hawk nodes (left) and 4096 Shaheen II nodes (right)</figcaption></figure>

<figure><img src="image_15.png" /><figcaption>**Figure 15.** Performance evaluation on Summit. The dashed lines in (c) show the theoretical peak performance of that precision</figcaption></figure>

to 100D is approximately 1.56X and from 100S to 100D is about 2.05X for matrix sizes larger than 2.4M. The 100D configuration achieved approximately 3.2 PFlop/s, representing about 88% of the DP Linpack benchmark on Shaheen II. Similarly, on Hawk, 100D achieved around 2.8 PFlop/s, while 10D:90S and 100S reached 4.5 PFlop/s and 5.6 PFlop/s, respectively, with speedups of up to 1.59X from 10D:90S to 100D and 1.98X from 100S to 100D.

**Dense Mixed-Precision on Heterogeneous Architectures:** We evaluate the dense mixed-precision strategy on Summit across three fundamental dimensions: weak scalability, strong scalability, and the impact of mixed-precision on FP64, as depicted in Figure 15. Regarding weak scalability, the problem size was incrementally enlarged along with the GPU count. The results exhibit nearly linear scalability, indicating the efficiency of HiCMA-PaRSEC in adapting to gradual increments in computational resources. In terms of strong scalability, we hold the matrix size constant at 798,720 while enhancing the GPU count. This analysis reveals significant strong scalability, underscoring our method's capability to distribute a constant-sized problem

<figure><img src="image_16.png" /><figcaption>**Figure 16.** Performance scalability on heterogeneous GPU system Frontier</figcaption></figure>

<table><caption>**Table 2.** MP performance of FP64/FP16 on Frontier (Pflop/s)</caption><thead><tr><th>Number of Nodes</th><th>256</th><th>1024</th><th>2048</th></tr></thead><tbody><tr><th>Performance</th><td>42.5</td><td>194.9</td><td>315.7</td></tr></tbody></table>

across more resources, thus accelerating the computational process. The performance on 384 GPUs slightly deviates from linear scaling, likely due to depletion of tasks, thereby aggravating communication and runtime overheads Slaugh-ter et al. (2020). Finally, the influence of mixed-precision on 64 Summit nodes (i.e., 384 GPUs) is examined by presenting the actual performance and its speedup relative to FP64. Here, the baseline performance of FP64 reaches 68.0% of the theoretical peak, which stands out in comparison to the 74.0% Linpack efficiency on Summit as recorded in the Top500. Regarding the mixed-precision (MP) effects, 2D-sqexp, Matern, and 3D-sqexp with adaptive precision strategy Cao et al. (2022a, 2023) outperform FP32 as the matrix size increases, achieving up to a 3.2X speedup relative to FP64.

Moreover, Figure 16 displays the performance metrics for dense/DP and dense/MP over various node counts on Frontier. The annotations regarding the “MP effect” emphasize the peak performance increase realized by MP compared to DP across different node configurations, consistently yielding a speedup of over 3×. In terms of parallel efficiency, there is a significant improvement, with enhancements of 13.6× and 10.6× noted when expanding from 4 nodes to 64 nodes for dense/DP and dense/MP, respectively. In addition, Table 2 depicts the performance of mixing FP64 and FP16 on Frontier at scale, which shows near-linear scalability.
<figure><img src="image_17.png" /><figcaption>**Figure 17.** Performance of Matérn 2D space-time of strong correlation on 4096 and 48384 Fugaku nodes</figcaption></figure>

*Mixed-Precision Dense/Low-rank Matrix Computations:*

Figure 17 illustrates the efficacy of the Matérn 2D space-time kernel with strong correlation across 4096 and 48384 Fugaku nodes. Utilizing the MP+dense/TLR methodology results in nearly an order of magnitude enhancement in speed compared to conventional dense FP64 on 4096 nodes due to higher ranks and infrequent opportunities for low precision computations. On 48384 nodes, the advantage of the MP+dense/TLR approach over pure dense FP64 diminishes owing to pronounced scalability constraints, notably a lack of sufficient tasks to fully engage the computational resources. Future enhancements could include increasing the number of tasks and refining the PaRSEC runtime system. Nonetheless, the reduction in memory usage is considerable, facilitating the management of larger problem sizes with the same resources.

## Summary

The DTE PaRSEC project set out to explore the frontier of programming techniques that would solve the emerging difficulty in programming (at least efficiently) hybrid/accelerated HPC architectures. These systems pose significant challenges to traditional approaches in the way they upend the traditional abstraction of the HPC system as a flat collection of symmetric processors that can be programmed with the SPMD programming style. In this new reality, the multiple levels of memory hierarchies, the different computing speed of components, and the myriad of software APIs to program these components and orchestrate data motion render the mapping of the application to the system hardware tedious, and prone to generating load imbalance and suboptimal performance.

Dataflow, task-based, micro-tasks programming styles have been proposed to address these shortcoming in traditional approaches, as it had been inferred that they would permit an easier mapping of parallel programs onto difficult to predict hardware conditions, but these benefits had largely yet to materialize when the DTE project started. At the end of this project, a clearer picture has emerged, where we have demonstrated conclusively that task-based programming styles, and runtimes to support them, have definite advantages. Our AD application partners, like NWChemEx, SLATE, and outside collaborators, like HiCMA, were able to express their algorithms using the PaRSEC provided DSL, and experience an unparallel level of performance as a result. Such level of performance, and the featureset that the task-based runtime enables, unlocked new application use cases that would not have been possible before, like applications with a dynamic, fine control on mixed precision computation, as illustrated in the HiCMA

Gordon Bell finalist application using PaRSEC. As the consensus that task-based programming is a critical tool to unlock further levels of performance in increasingly complex computing systems, the DOE, the HPC community, and the entire computing field at large are looking at the next step toward integrating the proven techniques, as pioneered in PaRSEC and other groundbreaking projects in the fields, and we are seeing multiple efforts sprouting to define industry standard solutions, be it as part of language constructs or runtime APIs for task-based execution.

## Declaration of conflicting interests

The author(s) declared no potential conflicts of interest concerning the research, authorship, and/or publication of this article.

## Funding

This research was supported by the Exascale Computing Project (17-SC-20-SC), a collaborative effort of the U.S. Department of Energy Office of Science and the National Nuclear Security Administration.

## References

Abdulah S, Cao Q, Pei Y, Bosilca G, Dongarra J, Genton MG, Keyes DE, Ltaief H and Sun Y (2021) Accelerating geostatistical modeling and prediction with mixed-precision computations: A high-productivity approach with PaRSEC. *IEEE Transactions on Parallel and Distributed Systems* 33(4): 964–976.

Akbudak K, Ltaief H, Mikhalev A, Charara A, Esposito A and Keyes D (2018) Exploiting data sparsity for large-scale matrix computations. In: *European Conference on Parallel Processing*. Springer, pp. 721–734.

Augonnet C, Thibault S, Namyst R and Wacrenier P (2011) StarPU: A unified platform for task scheduling on heterogeneous multicore architectures. *Concurrency Computat. Pract. Exper.* 23: 187–198.

Bauer M, Treichler S, Slaughter E and Aiken A (2012) Legion: Expressing locality and independence with logical regions. In: *International Conference for High Performance Computing, Networking, Storage and Analysis, SC*. pp. 1–11.

Bosilca G, Bouteiller A, Danalis A, Faverge M, Haidar A, Herault T, Kurzak J, Langou J, Lemarinier P, Ltaief H, Luszczek P, YarKhan A and Dongarra JJ (2011a) Flexible development of dense linear algebra algorithms on massively parallel architectures with DPLASMA. In: *25th IEEE International Symposium on Parallel and Distributed Processing - Workshop Proceedings*. IEEE, pp. 1432–1441. DOI:10.1109/IPDPS. 2011.299.

Bosilca G, Bouteiller A, Danalis A, Faverge M, Herault T and Dongarra JJ (2013) Parsec: Exploiting heterogeneity to enhance scalability. *Comput. Sci. Eng.* 15(6): 36–45. DOI: 10.1109/MCSE.2013.98.

Bosilca G, Bouteiller A, Herault T, Lemarinier P, Saengpatsa NO, Tomov S and Dongarra JJ (2011b) Performance portability of a GPU enabled factorization with the DAGuE framework. In: *International Conference on Cluster Computing (CLUSTER)*. IEEE, pp. 395–402. DOI:10.1109/CLUSTER.2011.51.
Cao C, Herault T, Bosilca G and Dongarra JJ (2015) Design for a soft error resilient dynamic task-based runtime. In: *IPDPS*.
IEEE, pp. 765–774. DOI:10.1109/IPDPS.2015.81.

Cao Q, Abdulah S, Alomairy R, Pei Y, Nag P, Bosilca G, Dongarra J, Genton MG, Keyes DE, Ltaief H et al. (2022a) Reshaping geostatistical modeling and prediction for extreme-scale environmental applications. In: *SC22: International Conference for High Performance Computing, Networking, Storage and Analysis*. IEEE, pp. 1–12.

Cao Q, Abdulah S, Ltaief H, Genton MG, Keyes D and Bosilca G (2023) Reducing data motion and energy consumption of geospatial modeling applications using automated precision conversion. In: *2023 IEEE International Conference on Cluster Computing (CLUSTER)*. IEEE, pp. 330–342.

Cao Q, Alomairy R, Pei Y, Bosilca G, Ltaief H, Keyes D and Dongarra J (2022b) A framework to exploit data sparsity in tile low-rank cholesky factorization. In: *IPDPS*. IEEE, pp. 414–424.

Cao Q, Bosilca G, Losada N, Wu W, Zhong D and Dongarra J (2021a) Evaluating data redistribution in PaRSEC. *IEEE Transactions on Parallel and Distributed Systems* 33(8): 1856–1872.

Cao Q, Bosilca G, Wu W, Zhong D, Bouteiller A and Dongarra J (2020a) Flexible data redistribution in a task-based runtime system. In: *2020 IEEE International Conference on Cluster Computing (CLUSTER)*. IEEE, pp. 221–225.

Cao Q, Herault T, Bouteiller A, SJ and Bosilca G (2024) Evaluating PaRSEC through matrix computations in scientific applications. In: *Workshop on Asynchronous Many-Task Systems and Applications (WAMTA)*.

Cao Q, Pei Y, Akbudak K, Bosilca G, Ltaief H, Keyes D and Dongarra J (2021b) Leveraging PaRSEC runtime support to tackle challenging 3D data-sparse matrix problems. In: *IPDPS*.
IEEE, pp. 79–89.

Cao Q, Pei Y, Akbudak K, Mikhalev A, Bosilca G, Ltaief H, Keyes D and Dongarra J (2020b) Extreme-scale task-based cholesky factorization toward climate and weather prediction applications. In: *Proceedings of the Platform for Advanced Scientific Computing Conference*. pp. 1–11.

Cao Q, Pei Y, Herault T, Akbudak K, Mikhalev A, Bosilca G, Ltaief H, Keyes D and Dongarra J (2019) Performance analysis of tile low-rank cholesky factorization using parsec instrumentation tools. In: *2019 IEEE/ACM International Workshop on Programming and Performance Visualization Tools (ProTools)*. IEEE, pp. 25–32.

Danalis A, Bosilca G, Bouteiller A, Herault T and Dongarra JJ (2014) PTG: an abstraction for unhindered parallelism. In: *Proceedings of the Fourth Intl. WS on Domain-Specific Languages and High-Level Frameworks for HPC, WOLFHPC '14*. IEEE, pp. 21–30. DOI:10.1109/WOLFHPC.2014.8.

Danalis A, Jagode H, Herault T, Luszczek P and Dongarra J (2019) Software-defined events through PAPI. In: *IPDPS Workshops*.
IEEE, pp. 363–372.

De Boer A, Van der Schoot MS and Bijl H (2007) Mesh Deformation Based on Radial Basis Function Interpolation. *Computers and Structures* DOI:http://dx.doi.org/10.1016/j.compstruc.2007.01.013.

Duran A, Ferrer R, Ayguadé E, Badia R and Labarta J (2009) A Proposal to Extend the OpenMP Tasking Model with Dependent Tasks. *International Journal of Parallel Programming* 37(3): 292–305.

Eschweiler D, Wagner M, Geimer M, Knüpfer A, Nagel WE and Wolf F (2012) Open trace format 2: The next generation of scalable trace formats and support libraries. In: *Applications, Tools and Techniques on the Road to Exascale Computing*. IOS Press, pp. 481–490.

Fertig A (2021) *Programming with C++ 20: Concepts, Coroutines, Ranges, and more*. Fertig Publications.

Gamblin T, LeGendre M, Collette MR, Lee GL, Moody A, De Supinski BR and Futral S (2015) The spack package manager: bringing order to HPC software chaos. In: *Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis*. pp. 1–12.

Gates M, Kurzak J, Charara A, YarKhan A and Dongarra J (2019) SLATE: Design of a modern distributed and accelerated linear algebra library. In: *Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis*. pp. 1–18.

Heller T, Kaiser H and Iglberger K (2013) Application of the ParalleX Execution Model to Stencil-based Problems. *Computer Science - Research and Development* 28(2-3): 253–261. DOI:10.1007/s00450-012-0217-1.

Herault T, Robert Y, Bosilca G and Dongarra JJ (2019) Generic matrix multiplication for multi-GPU accelerated distributed-memory platforms over PaRSEC. In: *ScalA@SC*. IEEE, pp. 33–41. DOI:10.1109/SCALA49573.2019.00010.

Herault T, Robert Y, Bosilca G, Harrison RJ, Lewis CA, Valeev EF and Dongarra JJ (2021) Distributed-memory multi-GPU block-sparse tensor contraction for electronic structure. In: *International Parallel and Distributed Processing Symposium*. IEEE, pp. 537–546. DOI:10.1109/IPDPS49936.2021.00062.

Heroux MA (2019) The extreme-scale scientific software stack (E4S). Technical report, Sandia National Lab.(SNL-NM), Albuquerque, NM (United States).

Hoque R, Herault T, Bosilca G and Dongarra JJ (2017) Dynamic task discovery in PaRSEC: a data-flow task-based runtime. In: *ScalA@SC*. ACM, pp. 6:1–6:8. DOI:10.1145/3148226.3148233.

Jagode H, Danalis A, Anzt H and Dongarra J (2019) PAPI software-defined events for in-depth performance analysis. *The International Journal of High Performance Computing Applications* 33(6): 1113–1127.

Kowalski K, Bair R, Bauman NP, Boschen JS, Bylaska EJ, Daily J, de Jong WA, Dunning Jr T, Govind N, Harrison RJ et al. (2021) From NWChem to NWChemEx: Evolving with the computational chemistry landscape. *Chemical reviews* 121(8): 4962–4998.

Lordan F, Tejedor E, Ejarque J, Rafanell R, Alvarez J, Marozzo F, Lezzi D, Sirvent R, Talia D and Badia RM (2014) Servicess: An interoperable programming framework for the cloud. *Journal of grid computing* 12(1): 67–91.

Miniskar NR, Monil MAH, Valero-Lara P, Liu FY and Vetter JS (2023) Iris-dmem: efficient memory management for heterogeneous computing. In: *2023 IEEE High Performance Extreme Computing Conference (HPEC)*. IEEE, pp. 1–7.

Monil MAH, Miniskar NR, Teranishi K, Vetter JS and Valero-Lara P (2023) Matris: multi-level math library abstraction for heterogeneity and performance portability using iris runtime. In: *Proceedings of the SC'23 Workshops of The*
*International Conference on High Performance Computing, Network, Storage, and Analysis*. pp. 1081–1092.

Mor O, Bosilca G and Snir M (2023) Improving the scaling of an asynchronous many-task runtime with a lightweight communication engine. In: *Proceedings of the 52nd International Conference on Parallel Processing*. pp. 153–162.

OpenMP (2021) OpenMP 5.2 Complete Specifications.

Pei Y, Cao Q, Bosilca G, Luszczek P, Eijkhout V and Dongarra J (2020) Communication avoiding 2d stencil implementations over parsec task-based runtime. In: *IPDPS Workshops*. IEEE Computer Society, pp. 721–729.

Peng C, Calvin JA, Pavosevic F, Zhang J and Valeev EF (2016) Massively parallel implementation of explicitly correlated coupled-cluster singles and doubles using TiledArray framework. *The Journal of Physical Chemistry A* 120(51): 10231–10244.

Peng C, Lewis CA, Wang X, Clement MC, Pierce K, Rishi V, Pavošević F, Slattery S, Zhang J, Teke N et al. (2020) Massively parallel quantum chemistry: A high-performance research platform for electronic structure. *The Journal of Chemical Physics* 153(4).

Schöne R, Tschüter R, Ilsche T and Hackenberg D (2011) The VampirTrace plugin counter interface: introduction and examples. In: *Euro-Par Parallel Processing Workshops*. Springer, pp. 501–511.

Schuchart J, Nookala P, Herault T, Valeev EF and Bosilca G (2022a) Pushing the boundaries of small tasks: Scalable low-overhead data-flow programming in TTG. In: *International Conference on Cluster Computing*. IEEE, pp. 117–128. DOI: 10.1109/CLUSTER51413.2022.00026.

Schuchart J, Nookala P, Javanmard MM, Herault T, Valeev EF, Bosilca G and Harrison RJ (2022b) Generalized flow-graph programming using template task-graphs: Initial implementation and assessment. In: *IPDPS*. IEEE, pp. 839–849. DOI:10.1109/IPDPS53621.2022.00086.

Slaughter E, Wu W, Fu Y, Brandenburg L, Garcia N, Kautz W, Marx E, Morris KS, Cao Q, Bosilca G et al. (2020) Task bench: A parameterized benchmark for evaluating parallel runtime

performance. In: SC'20. IEEE, pp. 1–15.

Valero-Lara P, Kim J, Hernandez O and Vetter J (2021) Openmp target task: Tasking and target offloading on heterogeneous systems. In: *European Conference on Parallel Processing*. Springer, pp. 445–455.

Wu W, Bouteiller A, Bosilca G, Faverge M and Dongarra JJ (2015) Hierarchical DAG scheduling for hybrid distributed systems. In: *IPDPS*. IEEE, pp. 156–165. DOI:10.1109/IPDPS.2015.56.

## Author Biographies

*George Bosilca* is a Research Associate Professor at the University of Tennessee, Knoxville, and recently joined NVIDIA corporation. His research interests evolve around distributed algorithms, parallel programming paradigms, performance modeling and optimizations, and scalable resilience. He holds a Ph.D. in computer science from the University of Paris (XI).

*Aurelien Bouteiller* is a Research Director at the University of Tennessee, Knoxville. His research centers on algorithms and designs that sustain HPC infrastructure with fault-tolerant, low-overhead communication frameworks, and explore scalable middleware and programming paradigms. He holds a Ph.D. in computer science from the University of Paris (XI).

*Qinglei Cao* is an Assistant Professor at Saint Louis University. He completed his Ph.D. in Computer Science under the guidance of Dr. Jack Dongarra (Turing Award 2021) at the University of Tennessee, Knoxville. His research interest lies in areas of High-performance Computing (HPC) and Artificial Intelligence (AI).

*Thomas Herault* is a Research Assistant Professor at the University of Tennessee, Knoxville. His research centers on resilience, task-based systems, and HPC, with a focus on bridging the gap between theoretical works and practical use. He holds a Ph.D. in computer science from the University of Paris (XI).

*Joseph Schuchart* received his PhD from the University of Stuttgart, Germany. He currently is a Research Assistant Professor with the Innovative Computing Laboratory at the University of Tennessee, Knoxville. His research interests include parallel programming models as well as performance analysis and optimization tools.