# Expressing Performance and Redundancy in OpenMP: the LIONESS Annotations and Toolchain

Sara Royuela, Eric Jenn, Hubert Guerard, Sylvain Tibério, Giuseppe Italia, Mariem Turki

► To cite this version:

Sara Royuela, Eric Jenn, Hubert Guerard, Sylvain Tibério, Giuseppe Italia, et al. Expressing Performance and Redundancy in OpenMP: the LIONESS Annotations and Toolchain. 13th European Congress of Embedded Real Time Systems (ERTS), Feb 2026, Toulouse, France. 10.82331/ERTS.2026.3. hal-05513179

HAL Id: hal-05513179

https://hal.science/hal-05513179v1

Submitted on 19 Feb 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
Expressing Performance and Redundancy in OpenMP:
the LIONESS Annotations and Toolchain

Sara Royuela¹, Eric Jenn², Hubert Guérard³, Sylvain Tibério⁴, Giuseppe Italia⁵, Mariem Turki⁶, Anthony Dentinger³

1,5 Barcelona Supercomputing Center, Barcelona, Spain
first.last@bsc.es

³Space Codesign Systems, Montreal, Canada
first.last @spacecodesign.com,

2,6 IRT Saint-Exupery, Toulouse, France
first.last@irt-saintexupery.com

⁴Airbus Defence and Space, Toulouse, France
sylvain.tiberio@airbus.com

ABSTRACT

Modern Systems-on-Chip (SoCs) incorporate multiple processing units, enabling various possibilities for parallelizing and accelerating computations. These processing capabilities can be leveraged to enhance resiliency by introducing temporal and spatial redundancy. In this paper, we propose a set of extensions to the OpenMP annotations that cover both performance and resiliency in an integrated framework. We present the implementation of these extensions and their integration into the comprehensive LIONESS toolchain, evaluating its capabilities on an image processing use case from the space domain.

1. INTRODUCTION

The processing capabilities of computing platforms are increasing continuously, driven by the need to host the increasingly complex functions required by consumers in the critical embedded systems market. Today, some high end System on Chips (SoCs) include multiple high performance and real-time processing cores, AI or graphical acceleration units, and an FPGA fabric, all on the same die. This plurality of processing units presents numerous opportunities for enhancing performance and resilience. However, the counterpart of this abundance is the complexity for the end-user to find the optimal deployment configurations. To support this task, the end-user needs means to *expose* the application's architecture and, once exposed, other means to exploit the available hardware resources.

In the domain of critical systems, the aim is to find a deployment that complies with both timing and dependability requirements (e.g. integrity or availability). The last objective is particularly crucial for space applications, as they are vulnerable to radiation-induced errors.

In this context, we propose (i) some extensions to the OpenMP Application Programming Interface (API) [1] to describe redundant architectures and recovery mechanisms, (ii) a set of tools and middleware exploiting these annotations to generate and deploy applications, and (iii) modeling means to evaluate architectural solutions.

The rest of the paper is organized as follows. Section 2 provides an overview of OpenMP and related work. Section 3 introduces the extensions to the OpenMP annotations. Section 4 presents the main components of the toolchain developed in the European Space Agency (ESA) LIONESS Project [2]. Section 5 demonstrates the application of annotations and tools to deploy an image processing function. Section 6 proposes models to support early design space exploration. Finally, Section 7 concludes the paper and discusses future applications of LIONESS.

2. BACKGROUND AND RELATED WORK

2.1. The OpenMP parallel programming model.

OpenMP is the de facto standard parallel programming model for shared-memory architectures in the High-Performance Computing (HPC) domain. It provides a portable API based on compiler directives, runtime routines, and environment variables on top of C, C++, and Fortran.

OpenMP includes a *thread-centric model* for parallelizing loop-intensive and data-parallel workflows, with directives like *for* and *sections*, and a *tasking model* for parallelizing irregular and highly dynamic workflows, with directives like *task* and *taskloop*. In contrast to the thread model, the *tasking model* is a descriptive paradigm that abstracts the complexities of the underlying hardware, enabling the runtime system to schedule and map tasks across multiple processor cores using data dependencies and memory fences for synchronization.

Additionally, OpenMP includes a *host-centric accelerator model* that, on the one hand, extends *tasking features* to enable offloading computation to accelerator devices (e.g., GPUs and FPGAs), with directives like *target*, and, on the other hand, extends *threading features* to exploit parallelism within the accelerator, with directives like *distribute* and *teams*. Pursuing performance portability, OpenMP also offers features to define function specializations through the *declare variants* directive.

2.2. State of the art.

The history of research in software-implemented fault tolerance traces back to the foundational work on SIFT [3,4], a Software Implemented Fault Tolerance method achieved through a combination of redundant hardware components and fault-tolerant software that detects errors, analyzes them,

<footnote>Sara Royuela et al. This is an open-access article distributed under the terms of the Creative Commons Attribution 4.0, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.</footnote>
and reconfigures the system to bypass faulty units. Hardware
replication has usually a strong impact on cost, weight, and
size. However, the shift to multicore systems and the
introduction of heterogeneous systems with different types of
computing units, now facilitates implementation of such fault
tolerance techniques relying on replication and N-version
programming.

Chen et al. [5] introduced an approximation algorithm for
identical processors that minimizes processor count and
maximizes utilization. Similarly, Shey et al. [6] developed
PLR, a transparent process-level redundancy method that
leverages hardware parallelism to reduce overhead. Kim et al.
[7] contributed R-BATCH, a task partitioning strategy that
allocates replicas to enhance system reliability. Building on
this, Vargas et al. [8] proposed NMR-MPar, which merges
redundancy and partitioning to improve reliability in many-
core, mixed-criticality systems. A common thread among
these studies is their primary emphasis on enhancing
scheduling performance, with less attention paid to the
implications for system programmability and developer
productivity.

In this regard, some software-based fault-tolerance methods
have been proposed using OpenMP. Fu et al. [9] introduced a
redundancy technique that duplicates parallel regions,
enhancing fault tolerance but potentially leading to
unnecessary overhead by removing the user's ability to
designate reliable and unreliable sections. Similarly, Tahan et
al. [10] developed a task-level replication mechanism from
which the runtime is entirely responsible, restricting the user's
flexibility to fine-tune replication and fault detection. This
lack of control can increase false positives, and the method is
further constrained by its support for only simple data types
when copying information for replicated tasks.

Our work builds on the principle that OpenMP alternatives
can provide a cost-effective layer of fault tolerance, as
modifying the replication scheme requires only slight changes
to compiler directives or environment variables. Therefore,
LIONESS leverages the simplicity and efficiency of OpenMP
to enhance application resilience, while granting users
significant control over both replication and consolidation
processes.

3. THE LIONESS SOLUTION

3.1. Approach

Aligned with the objective of providing efficient and safe
mechanisms for exploiting high-performance hardware
platforms in critical domains, LIONESS proposes a set of
fault-tolerance mechanisms that, leveraging OpenMP, reduce
the overhead of delivering resilience to C/C++ applications.
These mechanisms include a set of user-defined OpenMP
extensions that support (1) *task-level replication*, (2)
*consensus-and-voting* for fault detection, and (3)
*consolidation*, *timeout*, and *checkpointing* for fault recovery.

To achieve both programmability and flexibility, some parts
of the fault detection and recovery processes are handled by
the runtime system and therefore transparent or exposed
through directives (e.g., creation of software replicas, trigger
of the consolidation function), while other parts are left to the
user (e.g., definition and selection of the recovery strategy).

The proposed extensions can be integrated smoothly into
existing systems, as they are isolated from the regular flow of
the application. This design allows developers to augment
their code with resilience strategies without refactoring code
or modifying the functionalities. On top of that, OpenMP can
still be used to exploit software-based parallelism (CPU and
GPU devices) and parallelism in an FPGA fabric.

LIONESS integrates the resilience extensions to the OpenMP
API into a comprehensive toolchain for developing and
deploying aerospace applications on parallel and
heterogeneous processor architectures. The LIONESS
framework is composed of four components: TOAST,
SpaceStudio, LLVM, and Vivado. These components have
been integrated to offer a unique tool (see Section 4) that
leverages the parallel computing capabilities of the system
when deploying OpenMP applications augmented for
parallelism and resilience.

3.2. Workflow

Let us start with a standard C++ code segment that defines
some core computation, which is initially unannotated and
sequential:

int main() {
    ...
    compute(); /* unannotated computation */
    ...
}

The user then manually annotates the code to 1) expose data
and task parallelism, 2) indicates part of the computations to
be offloaded from the host processor, and 3) parts that are to
be executed multiple times on the same execution unit
(temporal redundancy) or on different execution units (spatial
redundancy), along with the associated consolidation function
and the recovery callback function. The resulting source code
augmented with specific OpenMP annotation (in bold, more
detail in the following section) is the input of the LIONESS
toolchain:

int consolidation_fnc(...);

#pragma omp declare target
void compute_fpga(...) { /* fpga exec. */ }

#pragma omp end declare target

void compute_fpga-wrap(..) {
    /* offload */
    #pragma omp target map(...) {
        compute_fpga(...);
    }
}

#pragma omp declare
variant(compute_fpga-wrap)
match device={kind(fpga)}

int main() {
    ...
    #pragma omp task replicated(2,spatial)
    with_variants(compute_fpga-wrap)
    consolidation(consolidation_fnc) {
        host_compute();
    }
    ...
}
<figure><img src="image_2.png" /><figcaption>Figure 1. The LIONESS workflow</figcaption></figure>

Figure 1 depicts the workflow implemented by the LIONESS toolchain. While data and task parallelism are handled by the traditional OpenMP runtime library, the LIONESS workflow processes the remaining annotations. First, the offloading annotations (target regions) are processed at step ① to generate two distinct source codes: one for the host device and one for the target device (another CPU and the FPGA). This step also generates the necessary host-side code for data copying to the target, computation triggering, and result retrieval ⑤. Next, the host code is processed to expand the fault tolerance patterns, generating calls to the extended OpenMP runtime library to implement replication ②. Finally, the host and target code are processed to create individual loadables, i.e., executable object code for CPUs or bitstream for an FPGA ③④⑥ and finally integrated in a single executable binary ⑦.

### 3.3. Annotations

Annotations capture the intent of the software designer in terms of performance (parallelization and acceleration) and resilience (redundancy, consolidation, and checkpointing).

Annotations are implemented using C pragmas. Those pragmas are directives to the compiler, not part of the actual code. They enable an incremental development process in which, for instance, parallelism is introduced gradually by adding annotations step-by-step to the initial, sequential version of the source code that can serve as a "reference implementation". This way, the refined version of the code can be verified against the previous version at each refinement step, allowing the developer to validate functions in a sequential computation model and introduce parallelization and redundancy using annotations with minimal modification of the actual code.

LIONESS exploits and extends the standard `target` and `task` constructs. Next, we provide a brief explanation of these annotations in the context of parallelization and acceleration and describe our extensions to support resiliency.

#### 3.3.1. Parallelization and acceleration annotations

OpenMP provides a large set of annotations to expose the parallelism of an application. Relevant directives include `parallel`, used to execute a block of code concurrently on multiple threads; `for`, used to distribute iterations of a loop among threads; and `task`, used to expose a concurrent unit of work to be scheduled by the runtime system. Other

annotations, such as the `depend` clause and the `barrier` directive, enable the synchronization of tasks and threads, respectively.

OpenMP also provides the `target` directive to offload a block of code to another device than the one – the host – executing the main thread. A device is a processing unit supported by the OpenMP runtime, which can be another CPU, a GPU, or an FPGA. In all cases, this means that the compiler or an integrated toolchain implements a specific code-generation process for the device. For a Xilinx FPGA target, this will be, for instance, Vivado HLS and Vivado.

#### 3.3.2. Redundancy annotations

To express redundancy, LIONESS extends the OpenMP task directive with several clauses that enable tuning aspects such as the replication degree, the replication strategy, the data copies, and, if any, the variant functions.

The syntax is as follows:

```
#pragma omp task
    replicated (degree,
               [spatial | temporal | spatial_temporal])
    replica_private (var [,...])
    replica_firstprivate (var [,...])
    with_variants (variant1 [,...])
{ // Default implementation }
```

The `replicated` clause specifies the `degree` of redundancy, i.e., the total number of replicas generated for a given task. It also defines the redundancy strategy: `spatial` redundancy forces each task to run on a different computation resource; `temporal` redundancy forces each task to be executed at a different point in time, enforcing a sequential execution of the replicas; finally, `spatial_temporal` redundancy combines both approaches, requiring tasks to run on different resources while also executing sequentially. If no strategy is specified, the default behavior allows tasks to run in parallel without any restrictions, thus favoring performance.

The `replica_private` and `replica_firstprivate` clauses define variables private to each task in order and manage their initialization.

The `with_variants` clause defines the function specializations (or variants) to be used by each replica, enabling N-version programming.
### 3.3.3. Error detection and masking annotations

To improve reliability or availability, redundancy must be coupled with a mechanism to detect or mask erroneous results generated by one or more replicas. These *fail-safe* and *fail-operational* behaviors are achieved using the following annotations:

```
#pragma omp task
...
    consolidation (var[:func])
    on_consolidation_error (callback:var_ptr)
    timeout (time)
    {
        // Default implementation
    }
```

Consolidation consists of comparing the outputs of the replicas and selecting a result or signaling a failure based on the comparison results. The `consolidation` clause includes the variables to be compared across replicas (var) and, optionally, the function used to compare them (func). If that function is not provided, the specification assumes a default bitwise comparison.

The `on_consolidation_error` clause specifies the callback function to execute if consolidation fails. This function is provided by the application that defines the actions to take when an error is detected.

Finally, the `timeout` clause sets a time limit for a task to complete. If the task does not finish within the specified time, it is considered to have failed, preventing failures to cause infinite delays.

### 3.3.4. Checkpointing annotations

Error recovery is generally highly dependent on the system under design. To enhance the capabilities of the user to implement the appropriate recovery mechanisms for each case, LIONESS provides a lightweight checkpointing mechanism based on three OpenMP runtime API calls: `checkpoint_initialize`, `checkpoint_store`, `checkpoint_load`, and `checkpoint_destroy`, which enable saving the state of a selected set of variables and restoring these values when needed.

This checkpointing mechanism is implemented as a lightweight C++ library integrated within the application. It supports both Plain Old Data (POD) and complex user-defined data types through callback functions that handle customized serialization and deserialization. Each checkpoint is managed by a *checkpoint_metadata* structure that maintains the allocated memory buffer, tracks used space, and stores function pointers for storing and restoring data. This design emphasizes simplicity, low overhead, and smooth integration with OpenMP's task-based execution model.

## 4. IMPLEMENTATION AND USE CASE

The implementation of the LIONESS workflow (Figure 1) relies on several tools, some of which have been developed or adapted specifically for the LIONESS project. Expanding the resilience annotations is accomplished by a modified version of the LLVM compiler and OpenMP supporting library<fnref n="1" />.

Processing the `target` directives is achieved by the TOAST tool and its supporting libraries. Selecting the actual target and generating the final loadable is performed by the SpaceStudio tool<fnref n="2" />. These tools are described below.

### 4.1. Toolchain

#### 4.1.1. Modified LLVM compiler

The OpenMP extensions proposed in this paper are implemented in the LLVM 19.0 compilation framework [12], including support in the C/C++ frontend and the runtime system.

The front-end, Clang, implements the syntax of the annotations. For *task replication*, it parses the redundancy annotations (see Section 3.3.2), transforms them into the adequate number of calls to the replicated task and includes a synchronization after the tasks. If the `with_variants` clause is present, the front-end generates a call to each variant specified. For *fault-recovery*, it parses error detection and masking annotations (see Section 3.3.3) and associates a memory object used to communicate information between the user and the runtime spaces. This object will serve the use to implement the recovery mechanisms based on the information reported by the runtime.

The OpenMP runtime, KMP, implements the logic to replicate data at run-time, when the tasks (the original ones and the replicas) are created and before they execute, and handles the concurrency of the replicas based on the type of replication specified. After the execution of the tasks, the runtime is in charge of calling the consolidation method provided in the annotations, if any, or a default bitwise comparison, otherwise. If consolidation succeeds and the correct results are in the original task, the runtime continues executing other tasks (including the sequential application). If consolidation succeeds but the correct results are not in the original task, the runtime updates the data in the original data storage. Finally, if consolidation fails, the runtime fills information about the context of the triggers the callback function for recovery provided by the user, if any.

#### 4.1.2. TOAST

TOAST takes as input a file containing an OpenMP-annotated source code and generates semantically equivalent source files ready to be processed by the SpaceStudio environment (see next section). The input of TOAST is a source file containing the host code as well as the target regions to offload, i.e., the code that will be ultimately run on the device. The output of the tool is composed of the following source files:

* One for the host, consisting of the input file with the modifications listed in the next section applied to it

* One file for each code element (called “Space Module”) generated out of a target directive.

The output of TOAST is the input of the SpaceStudio workflow, which means that the host source file is compiled with the platform's dedicated compiler (here, the LLVM modified compiler presented earlier). That way, OpenMP constructs other than the device constructs are interpreted by this compiler. Communication between the host and the target

<footnote n="1">¹ https://gitlab.bsc.es/ppc-bsc/software/llvm-taskgraph</footnote>

<footnote n="2">² https://www.spacecodesign.com/</footnote>
components of the application is handled by the SpaceStudio platform and API.

Sources for the devices will

* go through the High Level Synthesis process to produce a bitstream if the device is a FPGA. In that process, OpenMP constructs still in the device source file (such as parallel regions or tasks constructs) are discarded.

* be compiled using an appropriate toolchain depending on the device the module is mapped to. For example, a version of gcc is used to compile code for soft cores such as the microblaze. If this region contains OpenMP directives not related to the offloading, they will be interpreted by the compiler to take advantage of the parallelism expressed.

### 4.1.3. SpaceStudio

SpaceStudio (SS) is an advanced CAD tool that abstracts and orchestrates the complex workflow of developing heterogeneous system on MPSoC FPGA. It sits above traditional downstream tools, such as Xilinx Vivado and Vivado HLS, automating their execution to generate a complete, working system of an application.

SS's core functionality is its ability to analyze, partition, and synthesize the application into 3 essential, interconnected components: hardware accelerators, communication logic, and software tasks.

Offloaded functions are mapped as hardware accelerators and are synthesized using high-level synthesis (HLS) tool like Xilinx Vivado HLS. During this process, SS provides optimized HLS function implementation for the communication API.

SS analyzes the application's source code to precisely identify communication points (function calls to the communication API). Based on the application's mapping and the identified communication points, SS generates the necessary communication infrastructure (FIFOs, interconnects, adapters, DMAs, etc.)

The main host source file is mapped as a running software task. This task is compiled with the modified LLVM compiler to enable the replicated scheme as express by the redundancy annotations. The compiled source file is linked with SS's library to interface with the generated hardware and communication logic.

The output of these 3 compilations (hardware logic, communication scheme, and software) is packaged into deployable, loadable components: boot image (FPGA binary, FSBL, u-boot) and linux components (kernel, filesystem and devicetree). These unified components are then copied to an SD card for easy and rapid deployment onto the target FPGA board.

## 4.2. Use Case

**Functional chain.** Our use case is a Corner Detection Function (CDF), which is one component of larger applications such as the Lander Hazard Detection and Avoidance (LHDA) application or the Feature Tracking (FT) application. The CDF identifies the points that are pertinent elements in an image (for LHDA) or that need to be tracked from image to image to estimate displacement (for FT). The architecture of the CDF is shown in Figure 2: the upper part represents computation steps, the lower part represents the basic operators used in these steps. The input of the algorithm

is an image and its output is a list of pixel positions corresponding to detected corners in the image.

<figure><img src="image_3.png" /><figcaption>Figure 2. The Corner Detection Pipeline</figcaption></figure>

**Hardware Target.** The use case is deployed on a Xilinx Zynq UltraScale+ MPSoC ZCU102 Evaluation Kit combining a powerful processing system (PS) and a user-programmable logic (PL). The PS has three processing units: a Cortex-A53 application processing unit equipped with four Arm v8 64-bit cores, a Cortex-R5 real-time processing unit equipped with 2 Arm v7 32-bit real-time cores that can work independently or in a lock-step, and a Mali-400 GPU (which is not used in our case).

**Deployment Schemes.** LIONESS explored the deployment schemes depicted in Figure 3. Each scheme represents one point in the design space, and thanks to LIONESS, exploring this space is essentially a matter of modifying some annotations (OpenMP or HLS) and running the toolchain. In what follow, we explore the architectural space first towards the objective to improve end-to-end latency (or “performance”), then to improve integrity. In reality, the approach would certainly be carried out the other way round, first trying to comply with the dependability objectives, then improving performance.

The first configuration represents the reference case where all components of the pipeline are deployed on a unique core (CPU0). The second set of configurations explore two solutions to improve performances:

* *parallelizing* code execution on multiple cores using the OpenMP parallel annotations, with very minimal modification of the initial code (case b.1),

* *offloading* code of one or the other functional blocks to the FPGA using the TOAST tool (case b.2).

To improve dependability, the designer can act on the level of redundancy, the domain of the redundancy (spatial or temporal), the type of implementation (object code or bitstream), and the execution structure (Cortex-A53 or Cortex-R5 CPU cores or the FPGA). In order to demonstrate the versatility of the LIONESS approach, we have experimented a variety of configurations but, in reality, exploration would have been guided by the underlying fault models. For instance, replicating a function and deploying it on two cores of the same SoC improves resilience with respect to transient faults affecting one core, but not to a fault affecting the complete SoC. Similarly, executing the same function twice on the same core (pure temporal redundancy) improves resilience with respect to transient faults affecting strictly the application and not the execution infrastructure, etc. Therefore, in practice, such exploration would be strongly determined by (i) the dependability objective, (ii) the fault model.
<figure><img src="image_4.png" /><figcaption>Figure 3. Deployment schemes</figcaption></figure>

Figure 3 shows different configurations with spatial or temporal redundancy:

* *homogeneous*, i.e., the redundant components are executed on different cores of the same CPU (c.1 and d.1)

* heterogeneous, i.e., the components are executed on different execution infrastructure (c.2 and d.2)

Finally, the different schemes can be combined in different ways to optimize both *dependability* and *performance*.

The number of possible combinations is obviously very large, and we only show a subset of the combinations that were experienced during the project. For instance, configuration (d.1) combines spatial redundancy on different cores to improve the resiliency of the gradient calculation and offloading to FPGA to improve performance. Configuration (d.2) uses the FPGA to improve tolerance to design errors (implementation on CPU and FPGAs are strongly different) and tolerance to hardware faults (the execution structure are different).

Due to space restrictions, we only provide an excerpt of the code for configuration (d.1). On Lines in **red** shows LIONESS annotations for redundancy management; lines in **green** shows annotations for code offloading.

Listing 1: Code for configuration (d.1)

```c
void cornerDetection(...) {
    [... Initialization...]
    [... Processing...]
    #pragma omp parallel
    #pragma omp single
    {
        #pragma omp task \
        replicated(3,spatial) \
        shared(pImage, id1, layer0, d0, d1, d2, d3) \
        replica_firstprivate([sizeof(*pImage)]pImage) \
        replica_firstprivate([isize1]id1) \
        replica_firstprivate(layer0) \
        replica_private([size0]d0, [size1]d1, [size2]d2, \
        [size3]d3) \
        consolidation(layer0:consolidationImageGradients)
        {
            ... computeImageGradients(pImage, layer0);
        }
        #pragma omp taskwait
    }
    ...
}

#pragma omp end declare target

static void dilation(const float* pSrcData, float* pDstData) {
    ...
}
```

**Impact on functional code.** As illustrated, using pragmas reduces the coupling between the *dependability*+*performance*
annotations and the functional code. Nevertheless, when it comes to FPGA deployment, this property of independence does not completely hold. Achieving compliance with the target hardware does require modifications to the functional code. Indeed, a naive offload relying on adding `#pragma omp target` to the function fails because the functional code uses a 1Kx1K array, exceeding the available FPGA resources. To overcome this limitation, we introduce image slicing that divides the image into multiple smaller slices, significantly reducing the memory footprint on the FPGA. This implementation uses a function named `algoSliced` called iteratively on part of the image:

Listing 2: VIVADO HLS annotations

```c
#pragma omp declare target
void algoSliced(const float *pIn, float *pOut)
{
    /* algo here */
}

#pragma omp end declare target
void algo(const float in[1024*1024], float
    out[1024*1024]) {
    float inSlice[sizeof(in) / nbSlices];
    float outSlice[sizeof(out) / nbSlices];
    for(int slice=0; slice<nbSlices; slice++) {
        memcpy(inSlice, &in[slice*sizeof(inSlice)],
               sizeof(inSlice));
        #pragma omp target map(to: inSlice[:])
        map(from: outSlice[:])
            algoSliced(inSlice, outSlice);
        memcpy(&out[slice*sizeof(outSlice)], outSlice,
               sizeof(outSlice));
    }
}
```

To enhance the architecture's throughput, the functional code was further modified to enable concurrent execution of operations. This was achieved by inserting a specific Vivado HLS direction: `#pragma HLS pipeline II=1`. This optimization was applied to the innermost processing loop of the `algoSliced` function. Furthermore, the FPGA clock frequency was fixed at 200 MHz, contributing to the overall performance increase.

A design space exploration (DSE) was conducted on these architecture. The DSE varied 2 key parameters: 1) the number of slices used for image partitioning, and 2) the number of parallel slice processing units implemented on the FPGA.

### 4.3. Evaluation

The evaluation is based on six criteria that are described and analyzed hereafter.

**Execution duration** analyzes the impact of each deployment architectural choices on the execution time. In our case, the corner detection pipeline was optimized through both software-based parallelization (b.1) and FPGA acceleration (b.2). The software implementations achieved speedups between 2.7x and 3.7x when using all four APU cores, while the FPGA versions required additional adjustments to enable efficient HLS optimization (see previous section). As a result, the dilation kernel reached a 2.6x improvement for computation alone (1.65x when including data movement).

**Memory management** evaluates execution time and memory usage impact of data transfer between software and FPGA.

In our case, two categories of transfers were measured: ad-hoc copies required temporarily by TOAST/SpaceStudio, and intrinsic copies needed for kernel offloading. The ad-hoc copies introduce a small overhead of 1-7% depending on the kernel, while intrinsic copies transfers account for about 5% of execution time from host to accelerator and up to 31% from accelerator back to host.

**Software constraints** examines the effort to add LIONESS annotation and implement consolidation and recovery logic. The annotations are simple, portable, and can be enabled or disabled via compilation flags. The main developer effort lies in defining the recovery mechanisms, which depend on application semantics and cannot be automated by OpenMP. Current tool limitations also require rewriting algorithms to handle slices of data instead of whole images for offloaded functions.

**Footprint** analyzes the impact on the software and bitstream size. In the corner-detection use case, FPGA resource usage is low except for BRAM, which limits sharing with other IP. Future LIONESS improvements using DMA and data-flow processing are expected to reduce BRAM use, remove the need for slicing software data, and improve performance and parallelism.

**Fault detection** evaluates the system's ability to detect errors and trigger custom fail-safe recovery function. In the corner-detection use case, most kernels were replicated to enable rapid recovery, and testing with double replication showed that 99% of injected faults in replicated regions were successfully recovered. Faults in the OpenMP runtime remain unprotected, and replication overhead is limited to 2x for temporal replication and is negligible for spatial replication (considering the replicated functions).

**Resilience** evaluates how well the architecture maintains correct operation and prevents fault propagation. Using triple replication, the LIONESS framework achieved 99% successful recovery, employing consensus-and-voting when two replicas agree and checkpointing otherwise. Replication overhead on the APU is limited to 3x for temporal replication and negligible for spatial replication.

## 5. MODELING

### 5.1. Objectives

LIONESS proposes annotations to implement a fault tolerance strategy, but choosing a specific strategy is a difficult multi-criteria optimization problem involving multiple constraints, multiple variables and a cost – or objective – function. The objective function integrates the cost of the hardware resources (processor, LUTs, memory, etc.) and the cost of development and verification activities, which themselves may depend on the hardware target<fnref n="3" />. The constraints of our optimization process are the target probability of failure and the available type and amount of resources (RAM, LUTs). The control variables are the set of functions, which can be changed through decomposition (for the sake of performance using parallelization, acceleration, or dependability) and

<footnote n="3">³ For instance, deploying and debugging a code on an FPGA is usually much more costly than deploying and debugging the same</footnote>

<footnote>code on a processing core. However, in our context, deployment on the FPGA uses HLS, which makes faults less likely to occur and, then, debugging effort lower.</footnote>
aggregation, the mapping of functions to computation units (A-core, R-core, etc.) and the consolidation schemes.

In the absence of a completely automated optimization process, we propose two means to facilitate the exploration of the solution space and the evaluation of a given solution: one to generate candidate configurations and one to evaluate a candidate configuration.

## 5.2. Model generation with Alloy

To facilitate architectural exploration, we have developed a means to generate candidate configurations automatically according to the functional chain to be implemented, the structure of the execution platform, and some deployment constraints. Those elements are described using the Alloy modeling language and configurations satisfying constraints are generated using the SAT solver provided by Alloy's environment.

The complete Alloy model cannot be presented in the paper, but it can be found on github<fnref n="4" />. In what follows, we focus on some specific aspects of the model. For instance, the following code defines application `myApp` as being “composed” of two functional chains `fc1` and `fc2` (❶), where `fc1` is composed of functions `F1+F2a+F2b+F3+F4a+F4b+F5`, a source `source1` and a sink `sink1`. The flow itself is described as a set of relations (`->`) between functions, sources, and sinks (❷).

❶ one sig myapp extends App {}{fcs=fc1+fc2}

❷ one sig fc1 extends Fc {}{
source1+F1+F2a+F2b+F3+F4a+F4b+F5+sink1=fs and
flows = source1->F1 + F1->F2a + F1->F2b +
F2a->F3 + F2b->F3 + F3->F4a + F3->F4b + F4a-
>F5 + F4b->F5 + F5->sink1}

The next code sample shows the modeling of hardware. Here, we declare a class of hard core processor (sig CortexA, ❶) that are specialized into CortexA<i>,...</i> singleton sets with an associated reliability class and a maximum acceptable load (❷). A UltraScale is then defined as a SoC hosting 4 cortex A cores, 2 cortex R cores and a PMU (Platform Management Unit, ❸).

❶ abstract sig CortexA extends HardCore {}

❷ one sig CortexA2 extends CortexA

❸ {}{reliability = 0 and maxload = 2}

❹ one sig UltraScale extends HSoc {}
{CortexA1+CortexA2+CortexA3+CortexA4+CortexR1
+CortexR2+PMU = ps.hcs}

Two types of constraints are captured. The first type are used to produce structurally valid models. For instance, the following code expresses that all functions must be implemented and allocated (❶, there is at least one source (resp. sink) of the functional chain, and it is not the sink (resp. source) (❷,❸), and there is at least one path from the source to the flow (❶).

❶ { all f: fs | one if : Implementation
 | (if.func = f) and not no if.allocatingUnit
 and some (Source <: flows) and no( flows :>
 Source) and some (flows :> Sink) and no(
 Sink <: flows ) and some (Source <: *flows
 :> Sink) }

Besides those general constraints, we also have system-
specific constraint. For instance, the following code specifies
that function F5 must be deployed on a processing unit with a
high reliability (❶), function F1 must not be deployed directly
on the FPGA, either as a piece of software running on a soft
core (P1.scs) or directly as an IP (P1.ips) (❷).

❶ fact { F5_core.allocatingUnit.reliability>1 }

❷ all if : Implementation | (if.func = F1) => if.allocatingUnit not in (P1.scs + P1.ips)

From this set of facts and constraints, the Alloy solver is able to generate a list of valid models. A view of one of these models is given on Figure 4. On the close view, we see the functions composing functional chain 2 (in green). Function f4b has two implementations (`F4b_core2` and `F4b_ip`, in blue) which are mapped to a `CortexR1` and the `UltraScale` FPGA fabric, respectively (in yellow).

<figure><img src="image_5.png" /><figcaption>Figure 4. Example of an Alloy-generated model</figcaption></figure>

## 5.3. Modeling for evaluation with Altarica

To evaluate a fault tolerance scheme, we need the computation graph representing the produce/consume relationship between functions. At any point during the design of the architecture, a function can be decomposed into a subgraph (to expose subfunctions and increase the exploration space), or be replaced by a subgraph composed of multiple instances of the function and a consolidation function. An architecture is evaluated by transforming this computation graph into an AltaRica model that can be evaluated using stochastic simulation.

Altarica is a high-level modeling language dedicated to dependability analyses. With respect to other classical formalisms, such as Stochastic Petri Nets, Markov chain, Fault Trees, etc., the structure of an Altarica model reflects more directly the structure of the system under assessment. This strongly facilitates the transformation from the design model to the analysis model. Hereafter, we use the Altarica V3 language and the Altarica Wizard tool (both available from

<footnote n="4">⁴ https://github.com/ericjenn/erts2026</footnote>
<figure><img src="image_6.png" /><figcaption>Figure 5. Abstract view of an Altarica model</figcaption></figure>

the Altarica Association at http://www.altarica-association.org/).

To model a configuration, we propose a set of operators that can be combined to describe a configuration. Figure 5 shows a system of four functions (F1 to F4) in which F1, F2 and F4 perform the function. Function F3 is a redundancy that activates if function F2 fails to generate its output before some delay. In this figure, operator `src1` is the source of the flow and generates one "token" at the beginning of time; operator `v` joins two control flows; operator `MERGE` merges two different flows according to some policy (e.g., two-out-of-two, one-out-of-two, etc.).

Control flow operators are used to sequence the activation of FOps. Consolidation operators detects and correct errors. A functional component may fail in different ways (or modes): it may produce a manifest error, a non-manifest error, or no response. Non-functional components (control flow, error detection,...) are considered not to fail because their execution times are negligible with respect to other computations. An excerpt of the Altarica model of the system depicted in Figure 5 is given hereafter.

Line 4 instantiates function F2 that is an instance of a class of functions with 1 input and 1 output (class `Func_1_1`) and that normally produces a result after 5 units of time. It may also fails by producing no result with a probability of 0.1 (line 4). The connections of the basic operators (`source`, `F1`, `F2`,..., `Merge`, `Or`) are defined in the **assertion** section (lines 8 and following) that simply state that, at each instant, some input (e.g., `src1.TRO`) equals some output (`F1.TRI`).

### Listing 3: System model

1. `block System_timeout`

2. `Source src1;`

3. `Func_0_1 F1` (`eoc`), `F1.TRO`

4. `Func_1_1 F2` (`eocEvent`), `F2.TRO`,
   `fm_ok_event` = 0.9,
   `fm_nr_event` = 0.1);

5. `[...]`

6. `Merge merge1;`

7. `Or or1;`

8. **assertion**

9. `F1.TRI := src1.TRO; F2.TRI := F1.TRO; F2.FIP := F1.FOP;`

10. `timeout1.TRI Disarm := F1.TRO;`

11. `timeout1.TRI_DISARM := F2.TRO;`

12. `F3.TRI := timeout1.TRO; F3.FIP := F1.FOP;`

13. `or1.TRI1 := F3.TRO; or1.TRI2 := F2.TRO;`

14. `merge1.FIP1 := F2.FOP; merge1.FIP2 := F3.FOP;`

15. `F4.TRI := or1.TRO; F4.FIP := merge1.FOP;`

16. `observer Boolean out_ok = (F4.TRO != 0) and (F4.FOP == OK);`

17. `observer Boolean out_ko = (F4.TRO != 0) and (F4.FOP != OK);`

18. `end`

Once built, the behaviour of the AltaRica model can be validated using the discrete event simulator integrated in the Altarica Wizard tool. Performance evaluation also relies on stochastic simulation. Performance metrics are expressed using Altarica's `observer` construct (see lines 16 and 17). The stochastic simulator allows evaluating several indicators such as the number of occurrence of an event (e.g., a failure), or the mean-time-between occurrences of events, mean values of observables, etc. In our example, we estimate the latency and the probability of failure, as shown on Figure 6. Obviously, results could have been obtained by hand in this simple case.

<table><caption>Figure 6. Evaluation results</caption><thead><tr><th></th><th>Simulation result [hand calculation]</th></tr></thead><tbody><tr><td>Latency</td><td>19.5023<br/>[≈ pKO<sub>F2</sub> × (Δ<sub>F1</sub> + Δ<sub>to</sub> + Δ<sub>F3</sub> + Δ<sub>F4</sub>) + pOK<sub>F2</sub> × (Δ<sub>F1</sub> + Δ<sub>F2</sub> + Δ<sub>F4</sub>)]</td></tr><tr><td>PFailure</td><td>0.099425 [≈ pKO<sub>F4</sub>]</td></tr></tbody></table>

## 6. CONCLUSION

The LIONESS annotations and toolchain support the automatic deployment of applications on complex SoCs equipped with multiple processing units and accelerators. This solution strongly reduces the time to describe, deploy, and evaluate a configuration.

OpenMP annotations decouple the description of the function (as described by the code) from parallelization, acceleration, and redundancy. However, this decoupling is still imperfect. For instance, to (i) support high-level synthesis, (ii) fit on the FPGA, and (iii) obtain acceptable computation latencies, the source code needs significant modifications, including (a) splitting processing into small batches to reduce memory usage, (b) using only HLS-supported constructs, (c) adorning with HLS-specific annotations. This level of tuning means that, at some point in the development process, the code becomes target-dependent.

As expected, offloading some of the processing comes at a price due to the overhead of memory transfers between the CPU and the FPGA. Consequently, considering only performance and not resiliency, offloading a small part of the computation may reveal inefficiency. Conversely, offloading a large part may reveal it impossible due to the FPGA's limited resources. Altogether, systems require a comprehensive evaluation to make appropriate architectural and deployment choices.

Concerning the selection of configurations, the Alloy+Altarica models developed in LIONESS represent a first step towards the search for an optimal configuration. Next steps consist in translating automatically Alloy models to Altarica, implementing heuristics to guide the generation of candidate configurations. Extending the set of modeling components to cover the full set of Lioness' annotations.

## ACKNOWLEDGMENT

The authors would like to thank the European Space Agency for funding this work in the context of the OSIP call under contract number 4000141286/23/NL/GLC/ov.
REFERENCES

[1] OpenMP 6.0 specification, https://www.openmp.org/wp-content/uploads/OpenMP-API-Specification-6-0.pdf, 2024

[2] Royuela, S., Wartel, F., Tiberio, S., Jenn, E., Guérard, H., & Bois, G. (2025). LIONESS Improving and Leveraging OpenMP for the Efficient and Safe Use of New High-Performance Hardware Platforms. ACM SIGAda Ada Letters, 44(2), 71-75.

[3] J. H. Wensley, et al. SIFT: Design and analysis of a fault-tolerant computer for aircraft control. Proc. IEEE 66, 10 (1978), 1240–1255.

[4] F. Reghenzani, et al., 'Software Fault Tolerance in Real-Time Systems: Identifying the Future Research Questions', ACM Comput. Surv., vol. 55, no. 14s, pp. 1–30, Dec. 2023, doi: 10.1145/3589950.

[5] J.-J. Chen, C.-Y. Yang, T.-W. Kuo, and S.-Y. Tseng, "Real-time task replication for fault tolerance in identical multiprocessor systems," in 13th IEEE Real Time and Embedded Technology and Applications Symposium, 2007, pp. 249-258.

[6] A. Shye, J. Blomstedt, T. Moseley, V. J. Reddi, and D. A. Connors, "PLR: A software approach to transient fault tolerance for multicore architectures," IEEE Transactions on Dependable and Secure Computing, vol 6, no 2, pp. 135-148, 2008.

[7] J. Kim, K. Lakshmanan, and R. Rajkumar, "R-BATCH: Task partitioning for fault-tolerant multiprocessor real-time systems," in 10th IEEE International Conference on Computer and Information Technology, 2010, pp. 1872-1879.

[8] V. Vargas, P. Ramos, J.-F. Méhaut, and R. Velazco, "NMR-MPar: A fault-tolerance approach for multi-core and many-core processors," Applied Sciences, vol 8, no. 3, p. 465, 2018.

[9] H. Fu and Y. Ding, "Using redundant threads for fault tolerance of OpenMP programs," in International Conference on Information Science and Applications. IEEE, 2010, pp. 1-8.

[10] O. Tahan and M. Shawky, "Using dynamic task level redundancy for OpenMP fault tolerance," in International Conference on Architecture of Computing Systems. Springer, 2012, pp. 25-36.

[11] R. Leconte, E. Jenn, G. Bois, and H. Guérard, 'Make Life Easier for Embedded Software Engineers Facing Complex Hardware Architectures', in 10th European Congress on Embedded Real Time Software and Systems (ERTS 2020), Toulouse, France, Jan. 2020, p. 11.

[12] Munera, A., Quiñones, E., & Royuela, S. (2025, June). GuardianOMP: A framework for highly productive fault tolerance via OpenMP task-level replication. In 2025 IEEE International Parallel and Distributed Processing Symposium (IPDPS) (pp. 1-12). IEEE.

[13] AMD, Zynq UltraScale+ Device technical Reference Manual, UG1085, 2023.