# Communication Notification through User-Level Interrupts for the BXI Network

Charles Goedefroit, Alexandre Denis, Mathieu Barbe, Brice Goglin, Grégoire Pichon

► To cite this version:

Charles Goedefroit, Alexandre Denis, Mathieu Barbe, Brice Goglin, Grégoire Pichon. Communication Notification through User-Level Interrupts for the BXI Network. Cluster 2025 - 27th IEEE International Conference on Cluster Computing, Sep 2025, Edinburgh (Scotland), United Kingdom. hal-05150209

HAL Id: hal-05150209

https://inria.hal.science/hal-05150209v1

Submitted on 16 Sep 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Communication Notification through User-Level Interrupts for the BXI Network

Charles Goedefroit

Univ. Bordeaux & Eviden

Echirolles, France

Charles.Goedefroit@eviden.com

Alexandre Denis

Inria Bordeaux

Talence, France

Alexandre.Denis@inria.fr

Mathieu Barbe

Eviden

Echirolles, France

Mathieu.Barbe@eviden.com

Brice Goglin

Inria Bordeaux

Talence, France

Brice.Goglin@inria.fr

Grégoire Pichon

Eviden

Echirolles, France

Gregoire.Pichon@eviden.com

**Abstract**—To reduce the cost of communications in high-performance computing, it is possible to overlap communications with computations. Some communication protocols, such as rendez-vous, multi-chunk messages, and collectives, may require a completion notification to be processed before they can further progress. With active polling or passive waiting, completions are not processed while the application is busy with computation, and thus communication does not progress. However, with an event-based method like interrupts, it is expected to be much more reactive. Nevertheless, using interrupts usually involves system calls, which are avoided with high-performance networks.

The Intel Sapphire Rapids processors introduced user-level interrupts (UINTR), hardware interrupts designed to be used directly in user space, without going through the kernel. However, their current implementation is limited to inter-process communication. They cannot be triggered from a device.

In this paper, we propose new mechanisms to extend the scope of user-level interrupts, so as to be able to trigger them from a device and not only from a CPU. We have implemented these mechanisms in the BXI network from Eviden. We have evaluated their performance: we obtain a latency only 2.4 times higher than active polling (v.s. 6 times higher for interrupts with system calls). We have assessed their ability to make communication progress when overlapped with computation; we observe a near-perfect computation/communication overlap.

## I. INTRODUCTION

In high-performance computing (HPC), the performance of applications is often limited by the cost of network communications. Supercomputers use dedicated networking technologies that offer low latency and high bandwidth.

To further reduce the cost of communication, it is possible to hide it by overlapping communications with computation: the application performs computations while communications progress in background, thus reducing virtually their cost to zero. However, to make it work, the MPI library is supposed to be able to make communication actually progress in the background, which is hard to do in practice [1]. All network cards are able to send a small contiguous chunk of data using DMA or RDMA, by themselves, without requiring the help of the CPU. However, there is a point where they need to notify the user of the completion of the send or receive request. The protocol used by MPI libraries is often more

complicated than sending a plain small contiguous chunk of data; it may involve a rendez-vous protocol for large messages, submit a new request when another one completes in collective communication algorithms, or assemble multi-chunk messages when using sparse datatypes. In these cases, it is crucial for the MPI library to be able to react quickly to completion notifications from the network. Three typical mechanisms exist for notification:

* with *active polling*, the application explicitly polls the network to check whether a request is completed. When using an MPI library, it is performed through the use of *MPITest*. If the application is busy with computation, without calling MPI, then notifications do not percolate and communications do not progress.

* with *passive waiting*, the application calls a blocking system call, which makes its thread going into a sleeping state, allowing a computing thread to use the same CPU core. When the network card needs to notify the application, it triggers an interrupt (IRQ) processed by the kernel, which unblocks the system call and in turn awakes the sleeping thread. However, HPC networks are typically programmed from *user space* (OS bypass) to get low latency. The impact of the system call for passive waiting is huge on latency. This method is of marginal use in HPC for this reason.

* with an *event-based* method, while the application does not explicitly wait for a notification, the network card triggers an interrupt that is forwarded to user space, for example as a *SIGIO* signal [2]. This approach is popular [3] for networks using the TCP stack. But again, since interrupts and signals involve the kernel, they are usually not available for HPC networks programmed from user space.

When doing computation/communication overlap, the application is busy doing computation, and communication is supposed to progress by itself in background. In case the communication involves a rendez-vous, as commonly used in MPI libraries for large messages, the rendez-vous needs to
be processed by the receiver before the data can actually be
sent. If the application only performs computation and thus
does not call the MPI library, then the rendez-vous is not
processed in case of *active polling* or *passive waiting*, and thus
communication does not progress. In contrast, for *event-based*
notifications, the rendez-vous will be processed even if the
application does not call the MPI library, since it interrupts the
normal flow of instructions. The *event-based* method is then
expected to enable the best reactivity for communications, but
it suffers from high latency due to the kernel being involved.

Beginning with Intel Sapphire Rapids processors, *user-level interrupts* (UINTR) have been introduced. User-level interrupt is a hardware mechanism designed to behave like an interrupt, but calling directly user-space handler code without going through the kernel. If we are able to use them in HPC networks, we would get the best of both worlds: ability to immediately react to network messages, while maintaining a low latency thanks to an execution fully in user-space. However, in their current implementation [4], user-level interrupts are limited to inter-process communication. They can only be triggered by CPU code (user-space or kernel), not by a device.

In this paper, we propose new mechanisms to extend the
scope of user-level interrupts, so as to be able to trigger user-
level interrupts from a device and not only from a CPU. More
precisely, we make the following contributions:

* we propose a mechanism to trigger user-level interrupts from a PCIe device through the use of IOMMU;

* we implement this method in the BXI network technology [5] from Eviden;

* we extend the user-space Portals library for BXI with user-level interrupt support.

The rest of this paper is organized as follows. In Section II, we give some context about interrupts management. In Section III, we describe our contribution on user-level interrupts triggered from a PCIe device. In Section IV, we evaluate the performance of these mechanisms. In Section V, we present related work. Section VI concludes and give some insights about future work.

## II. CONTEXT: DIFFERENT TYPES OF INTERRUPTS

In this section, we first give some context about interrupts in
general, and user-level interrupts in particular. These concepts
will be needed to understand our contribution. We present
classic interrupts, the use of IOMMU, and finally user-level
interrupts.

In general, an interrupt is a signal that is sent by a device
or by a CPU to interrupt a CPU. It diverts its execution to
a *handler*, a function registered at initialization time, that
processes the event, and then returns to normal execution. In
this paper, we focus on interrupts triggered by a PCIe device.

### A. *Classic interrupts*

We call "classic interrupts" the interrupts that have existed
since the beginning of the x86 architecture. It encompasses
several generations of mechanisms to implement them. Interrupts used to rely on a dedicated IRQ line (a physical pin

on the connector), then they shifted to signals through regular
messages on the bus with MSI and MSI-X. They require kernel
privileges to be managed and are not visible from user-space.
They are sometimes called *ordinary interrupts* in the Intel
documentation [6].

To trigger a classic interrupt in the kernel, nowadays a
device uses the MSI method. MSI (Message Signaled In-
terrupts) and its extension MSI-X is described in the PCIe
specification [7] and allows to use PCI messages to emit
classic interrupts. This works with a MSI-X table located on
the device, with one entry for each interrupt. An entry stores
a PCI *Message Address* and a *Message Data*. To emit an
interrupt, the device will send a PCI message to the host. The
PCI message consists of a PCIe *memory write* with *Message
Data* to write, and the *Message Address*. So, a device can
write at a specific address to trigger an interrupt to a specific
core. Each core has 256 classic interrupts; these interrupts are
identified by a number named *vector*.

The MSI-X table entry is filled by the host kernel and is
CPU dependent. Modern Intel processors supports two MSI-
X entry formats: the *Compatibility format* for compatibility
purposes and the *Remappable format*. The IOMMU (Input-
Output Memory Management Unit) can perform physical
address translation and memory address remapping. This later
format became the default format and works with the Intel
VT-d IOMMU for virtualization. This mechanism allows to
remap device interrupts because MSI-X interrupts are memory
operation. IOMMU uses *Remapping Tables*, more specifically
for the MSI-X *Remappable format*, it uses an *Interrupt Remap-
ping Table*. This table is composed of several entries (named
IRTE) identified by an IRTE index. Intel IRTE can be in
two modes, the *remapped mode* and the *posted mode*. The
*remapped mode* is used by default and contains similar fields
as the MSI-X *Compatibility format*. The main difference is
that the fields are not located on the device, but in the host
memory and in the IOMMU cache. We will see further the
*posted mode* in the section II-B.

Figure 1 depicts the hardware path used by interrupts. In this
example, the considered PCIe device is a BXI NIC (Network
Interface Card). For classic interrupts, we follow the *blue* line
(dotted dash):

① The PCIe device receives a network event. The NIC needs to notify the application. It sends an interrupt.

② The device's PCIe interface transfers a PCI message to perform an MSI-X interrupt, which is a PCI *memory write*.

③ The IOMMU handles the PCI message and uses the IRTE index to get the IRTE (*Interrupt Remapping Table Entry*).

④ The IOMMU uses the IRTE to trigger a classic interrupt on the right core through its local APIC (=LAPIC, Local Advance Programmable Interrupt Controller).

⑤ The currently running thread is interrupted. The kernel classic interrupt handler calls the BXI kernel driver to notify the application through another mechanism (e.g. sleeping syscall, eventfd...).
<figure><img src="image_2.png" /><figcaption>Fig. 1. Classic interrupts hardware path.</figcaption></figure>

An application which uses the BXI Portals library, is allowed to use only one classic interrupt. To setup this interrupt, the Portals library calls an `ioctl` defined in the BXI kernel driver.

## B. Classic interrupts with Virtual Machines (interrupt-posting mechanism)

Even though our work does not involve virtual machines (VM), it relies on mechanisms that were initially designed for VM. Thus we present here how classic interrupts are managed in the presence of a VM.

The traditional way to interrupt a VM is to trigger an interrupt to the host kernel. Then the host kernel will interrupt the VM core, and the guest kernel will receive the interrupt.

To bypass the host kernel, Intel released a new *interrupt-posting* [8] mechanism. This mechanism extends the IOMMU interrupt remapping capability to allow to directly trigger a classic interrupt to a VM core. This mechanism comes with a new memory structure named *PID* (Posted Interrupt Descrip- tor) and needs one classic interrupt dedicated to *interrupt-posting* for each core. So, when switching the IRTE mode to *posted mode*, the IRTE stores the address of the *PID* (in the *PDA* field) and the *Virtual Vector* (VV, the classic interrupt vector that will be triggered in the guest kernel).

Each VM core can be linked to a *PID* memory structure. Details of the content of the *PID* structure are given in Figure 2. The first 256 bits of the *PID* contain the **PIR** field: each bit in this field correspond to classic interrupt vectors

<figure><img src="image_3.png" /><figcaption>Fig. 2. Posted Interrupt Descriptor (PID)</figcaption></figure>

which will be delivered to the guest kernel. The next 64 bits contain all *Notification control* fields.

When the IOMMU remaps a classic interrupt, the right IRTE is selected. If this IRTE is in *posted mode* the following steps are performed:

* The *PID* address and *VV* value is retrieved.

* A **PIR** bit is set to 1 by using the *VV* value as the *n*-th bit of the **PIR** field of the target VM *PID*.

* The **atomic operation** with the **ON** (Outstanding Notification) and **SN** (Suppress Notification) and send the classic interrupt dedicated to the *interrupt-posting*.

When a local APIC receives a classic interrupt, it uses the incoming vector to check whether this is the interrupt dedicated to *interrupt-posting*. If so, the "posted-interrupt processing" which delivers each **PIR** bit as an interrupt vector begins. The LAPIC will call the *guest kernel's* interrupt handler instead of the *host kernel's* interrupt handler.

## C. User-space Interrupt specification and definitions

User-level interrupt is a new hardware feature available on Intel Xeon processors that appeared in the Sapphire Rapids generation. It is a new kind of interrupts which triggers a handler directly in user space, but requires support from the kernel for setup.

1) *Using user-level interrupts:* Currently, user-level interrupts can only be triggered from the CPU, and thus can only be used for IPC (Inter-Process Communication) [4] between two application threads, or between a kernel thread and an application thread.

These interrupts work at application thread level: the software thread must be in user-space to be interrupted. If a user-level interrupt arrives when a task is not scheduled, this interrupt will be received when the task returns to user-space. Each thread may register its own user-level interrupt handler, which we call *ui_handler*. The *ui_handler* is called with a user vector parameter, *uvec*, between 0 and 63. That value may be used to identify the interrupt source, another application thread or a device. A *ui_handler* has some constraints like any interrupt handler: it cannot sleep, and should only use async-signal-safe functions (like signal handler). It is not allowed
<figure><img src="image_4.png" /><figcaption>Fig. 3. UPID</figcaption></figure>

to take locks, and thus cannot use functions that use locks internally, such as dynamic allocators (malloc). Moreover, the compiler only saves the 32 general registers. The FPU x87, SSE, AVX registers are not automatically saved, they should be manually saved before use in the *ui_handler*.

We call *receiver thread*, an application thread that wants to be interrupted. Several new system calls are introduced with user-level interrupt support in the Linux kernel [9]. The receiver thread first initializes the mechanism by calling *uintr_register_handler*() with the *ui_handler* address. Then it reserves a *uvec* with *uintr_vector_fd*() which returns a file descriptor, named *uvec_fd*. The descriptor refers to the chosen *uvec* and the receiver thread, it may be used by interrupt sources. When the kernel is the source, it can directly emit user-level interrupts. When an other application thread is the source, the *uvec_fd* need to be shared with *pidfd_getfd*(). Then the other thread may call *uintr_register_sender*() to be registered as sender.

2) Hardware support: The processor brings several new instructions for manipulating user-level interrupt:

* `SENDUIPI` emits user-level interrupts;

* `CLUI`, `STUI` and `TESTUI` masks or unmasks interrupts;

* `UIRET` returns from an interrupt handler (automatically inserted by the compiler<fnref n="1" /> at the end of the *ui_handler*).

Each core has a classic interrupt (a *vector*) dedicated to *UIINTR*.

New MSRs (Model-Specific Register) reserved for the *UIINTR* mechanism are added for each thread:

* `MSR_IA32_UINTR_HANDLER` to store the *ui_handler* address to call when a user-level interrupt arrived.

* `MSR_IA32_UINTR_PD` to store the *UPID* address. The *UPID* is similar to the *PID* memory structure but for application thread instead of VM core.

* `MSR_IA32_UINTR_MISC` composed of various fields, such as the *UINV* field, which corresponds to the dedicated classic interrupt vector.

* other MSR registers that we do not use in this paper.

Each receiver thread has a memory structure in host RAM named *UPID*. Figure 3 shows all *Notification control* fields in the first 64 bits of the *UPID* that represent the dedicated classic interrupt use for *UIINTR*:

* The **NV** – *Notification Vector* – field corresponding to the classic interrupt vector dedicated to user-level interrupt.

* The **NDST** – *Notification Destination* – field corresponding to the core ID (APICID) of the core where the receiver thread is currently running.

* The **SN** – *Suppress Notification* – field is a bit to suppress notification of the core stored in the **NDST**. If this field is set, the classic interrupt dedicated to *UIINTR* should not be emitted by the source. This field is set when the thread is preempted and cleared when the thread is in user-space.

* The **ON** – *Outstanding Notification* – field is a bit to indicate if a notification is outstanding. This bit is set by the *UIINTR* source and reset by the interrupt controller of the targeted core. If this field is set, the dedicated classic interrupt does not need to be emitted.

In the *UPID*, we also find the **PIR** field that represent the 64 available *uvec*.

3) **Initialization phase:** The initialization phase consists of two syscall *uintr_register_handler* and *uintr_vector_fd*. The handler registration, made by the *uintr_register_handler* syscall, performs the following steps:

1) Store the *ui_handler* address in the *MSR_IA32_UINTR_HANDLER* MSR.

2) Allocate the *UPID* and store it address in the *MSR_IA32_UINTR_PD* MSR.

3) Store the dedicated classic interrupt vector in the *NV* field of the *UPID* structure and in the *UINV* field of the *MSR_IA32_UINTR_MISC* MSR.

The *uvec* allocation, made by the *uintr_vector_fd* syscall, creates and returns a file descriptor, named *uvec_fd*, that refers to the chosen *uvec* and the *UPID* address of the targeted receiver thread.

After this initialization, the core ID (APICID) of the core where the application thread runs, is stored in the **NDST** field of the *UPID* even if the core changes.

4) **Usage phase:** After initialization, user-level interrupts may be emitted. Only a component in the CPU like a core or an *IOMMU* can emit interrupts.

To emit a user-level interrupt, the kernel or the *SENDUIPI* instruction follows this steps:

1) Retrieve the *uvec* and the target *UPID* address by using *uvec_fd*.

2) Use the *uvec* value as the *n*-th bit of the **PIR** field of the target *UPID* and set this bit to 1.

3) Perform **atomically** the following operations:

a) Check if the status *UPID* bits are clear. Status bits are composed of **ON** and **SN** fields of the target *UPID*. If these bit are not clear this atomic operation stop here.

b) Set the **ON** field to 1.

c) Read the **NDST** field to get the core ID (APICID) of the target core to interrupt and read the *NV* field of the classic interrupt to trigger.

d) Emit the right classic interrupt to the right *LAPIC*.

When the classic interrupt arrives on the target core *LAPIC*, the "user-interrupt notification identification" starts. The *LAPIC* compares the received interrupt vector with the vector stored in *UINV* field of the *MSR_IA32_UINTR_MISC* MSR. If the vector matches and the core is in users-space

<footnote n="1">¹*UIINTR* are supported by compilers since GCC 12 and since *LLVM* 15</footnote>
(Current Protection Level, CPL is 4, meaning "application"), the user-level interrupts are processed. The "user-interrupt notification processing" and "user-interrupt delivery" clear the **UPID ON** field, and trigger a user-level interrupt for every **PIR** bit set. To trigger an interrupt, the bit position in the **PIR** field is used as **uvec**. Then the **ui_handler** is invoked with the **uvec**. When the **ui_handler** returned by using **UIRET**, the next bit set in the **PIR** is delivered.

To sum up, user-level interrupts are based on the **UPID** structure and over classic interrupts.

### III. CONTRIBUTION: TRIGGER UINTR FROM A PCIe DEVICE

In this Section, we present our contribution: a mechanism to trigger user-level interrupts from a device, even though they are designed only to be triggered from the CPU.

1) **Rationale:** To trigger user-level interrupts from a PCI endpoint, our idea is to remap a MSI-X interrupt into a user-level interrupt. A user-level interrupt must be triggered by setting a **PIR** bit of the target **UPID** and performing an **atomic operation** that can emit a classic interrupt.

We have seen that the IOMMU *interrupt-posting* mechanism is similar to the UINTR mechanism. It has a dedicated classic interrupt, and performs the same steps to emit an interrupt: setting a **PIR** bit to 1 and perform the same **atomic operation** but on **PID** instead of **UPID**. In addition, setting the **PIR** bit and the atomic operation are the same actions needed by user-level interrupt. We would like to use this *posted mode* to emit user-level interrupt. Unfortunately, this mechanism is not compatible with UINTR and its **UPID**, because **PID** and **UPID** have the same fields, but not at the same location and not with the same size. In fact, the **PID**'s **PIR** field size is 256 bits and the **UPID**'s **PIR** field size is 64 bits.

However, since **UPID** and the **PID** are very similar, we explore the possibility of redirecting the IOMMU to use **UPID** rather than a traditional **PID**. Indeed, when we superpose the **UPID** and the **PID** fields, we can see that the entire **UPID** only covers the first half of the **PID**'s **PIR** field. So, we create a union of these two structures that we call **PID/UPID** union which can be used jointly by cores and IOMMU (see Figure 4). **PID** and **UPID** must be 64-byte memory aligned, thus, our union must have both superposed structures starting at the same address.

2) **Description of the mechanism:** The first step in making these objects compatible is to select a **PIR** bit that must be present on both sides. This can be done by adding 64 to the desired **uvec** and placing the result in the **VV** field of the IRTE. As the **PIR** fields of the **UPID** overlap those of **PID**, when the **uvec + 64** n-th bit of the **PID** is set, the **uvec** n-th bit of the **UPID** **PIR** is also set.

The second action, the **atomic operation**, requires the *Notification control* fields. These fields do not overlap between the **PID** and **UPID**. The *PID's Notification control* fields are located outside the **UPID**, which means we need to duplicate these fields.

<figure><img src="image_5.png" /><figcaption>Fig. 4. **PID/UPID** union (blue is **PID**, red is **UPID**)</figcaption></figure>

The **SN**, **NV** and **NDST** fields are only modified by the kernel during a core change or task switch. So we can easily duplicate them. However, for the **ON** field, it is more complicated. The **atomic operation** performed by *interrupt-posting* sets the **ON** of the **PID** and the LAPIC clears the **ON** of the **UPID**. This means that the **PID's ON** bit is never cleared and the dedicated classic interrupt will never be emitted again. Due to the mechanism that allows to defer user-level interrupts arrived when the task is preempted, we can still be interrupted later. Indeed, when any bits of the **UPID's PIR** are set when a task returns to user-space, the kernel code emits a new interrupt to the LAPIC, causing our lost user-level interrupts to be triggered.

With our tricks, the IOMMU accesses and edits bits between 64 and 320 of the **PID/UPID** union and the LAPIC manipulates the first 128 bits of the **PID/UPID** union as **UPID**.

To divert the use of *interrupt-posting* to trigger user-level interrupts, we edit an IRTE pointed by an MSI-X entry. So, we change the mode of the IRTE to *posted mode*; we fill the **PDA** field of IRTE with the **PID/UPID** union address instead of a **PID** address; and in the **VV** field of IRTE, we set the wanted **uvec** + 64.

To use our **PID/UPID** union, we need another patch to the Linux kernel patched with UINTR support. In our patch, we changed the **UPID** allocate size from 16 bytes to 64 bytes to include the following **PID** fields. This extended memory zone therefore corresponds to our **PID/UPID** union. At every place in the code where the **UPID's notification control** fields (**SN**, **NV** and **NDST**) are modified, we have also reported the value in the **PID's notification control** fields.

3) **Implementation in BXI:** In the BXI kernel driver, we add new ioctl to setup and cleanup user-level interrupts similar to the one for classic interrupts. In setup we perform the following steps:

* Retrieve the **uvec_fd** provided by the user application.

* Get the **UPID** and the **uvec** from the **uvec_fd**.

* Reserve or retrieve a MSI-X interrupt for the current application thread.

* Get the IRTE related to the MSI-X interrupt dedicated to
the application.

* Save the IRTE state.

* Change the IRTE mode from *remapped mode* to *posted mode*.

* Fill the *PDA* field of IRTE with the *PID/UPID* union address.

* Set the *VV* field of IRTE with the *uvec* plus 64.

* Call the kernel function to apply the IRTE modifications and flush the IOMMU cache.

* The setup returns the MSI-X entry number to the application. That allows to acknowledged the right interrupt.

In cleanup, we restore the IRTE state (to *remapped mode*).

In the user-level Portals library, we register a *ui_handler* (*uintr_register_handler ui_handler*)), we reserve a *uvec* to get a *uvec_fd* (*uvec_fd = uintr_vector_fd(56)*)) and we call the user-level interrupt setup ioctl (*intr->num = ioctl BXI UINTR_SETUP, uvec_fd)*). We also call the cleanup ioctl.

To clear the **ON** of the *PID* from user-space, we allow the *PID/UPID* union to be mapped in the BXI kernel driver. Then, in the Portals library, we map the union to clear the **ON** of the *PID* in the *ui_handler*.

That memory mapping can cause security issues because any application using the patched Portals library can trigger any interrupt by changing the *PID/UPID* and using the *SENDUIPI* instruction.

The BXI NIC needs to be acknowledged for each interrupt. This acknowledgment can only be done in the BXI kernel driver. We patched the BXI kernel driver and the Portals library to allow this acknowledgment to be performed in the application and we perform this acknowledgment in the *ui_handler*.

To address the interrupt handler (*ui_handler*) restrictions, we mask and unmask user-level interrupts with *CLUI*, *STUI* and *TESTUI* instructions around the critical section (lock/unlock).

4) Hardware path: The hardware path for user-level interrupts, is depicted on Figure 5 in green (dashed arrow):

① The *PCIe Device*, a BXI NIC in our example, receives a network event. The NIC's internal processor needs to notify the application and send an interrupt.

② The device's PCIe interface transfers a PCI message to perform an MSI-X interrupt, which is a PCI memory write.

③ The IOMMU handles the PCI message with the IRTE index to obtain the IRTE.

④ The IOMMU uses the IRTE to retrieve the *PID* part of the *PID/UPID* and the *VV* field. Then use the *VV* value as the *n*-th bit of the *PIR* field of the *PID* and set it to 1. Finally, the *atomic operation* is performed, the **ON** bit of the *PID* is set to 1 and the dedicated classic interrupt is triggered on the right core. The dedicated interrupt number and core number are retrieved from the *NV* and *NDST* fields of the *PID*.

<figure><img src="image_6.png" /><figcaption>Fig. 5. User-level interrupts hardware path.</figcaption></figure>

⑤ The incoming interrupt is identified as a user-level interrupt, and processing and delivery are initiated. The LAPIC retrieves the *UPID* part of the *PID/UPID* from the *MSR_IA32_UINTR PD* MSR. Clears the *UPID's ON* bit (already equal to 0 as expected). Call the *ui_handler* with the *uvec* value retrieved from the *PIR* of the *UPID*.

⑥ The thread is interrupted.

Then the application thread receives the user-level interrupt. The *ui_handler* of the Portals library is called, the **ON** bit of the *PID* part of the *PID/UPID* union is cleared, the BXI NIC is acknowledged and the network event is delivered.

## IV. EVALUATION

In this section, we measure the performance of the proposed mechanisms: the impact on communication latency, and the impact on communication/computation overlap.

### A. Evaluation environment

Our experiments are performed on a node comprised of two Intel(R) Xeon(R) Gold 6430 processors, based on the *Sapphire Rapids* micro-architecture that supports *UINTR*, and two *BXI* v2 NICs. *BXI* v2 NIC implement PCIe 3.0. Each NIC is on a different PCIe 5.0 slot linked to a different processor (a different NUMA node). Both NICs are connected back-to-back with a direct cable. The NICs are unmodified *BXI* v2 boards.

The software stack is based on a custom Linux 6.0.0 kernel with the *UIPI* Intel patches [9] in addition to our patches
presented in Section III-2. The BXI kernel driver and the
Portals library are modified with our UINTR patches<fnref n="2" />.

Each experiment involves two processes. One process is
bound on a core of the first processor and uses the closest
BXI NIC; the other process is bound on a core on the other
processor and uses the other BXI NIC.

We disable the Turbo boost and set the processor to its base
frequency (2.10GHz).

B. Latency evaluation

We first measure the end-to-end latency to assess the perfor-
mance of user-level interrupts compared to polling and classic
interrupts.

We designed a benchmark program using the portals API.
It performs a ping-pong to measure round-trip latency. We use
PtlPut to send messages, PtlEQWait to wait for a message
on receiver side, and clock_gettime() for timing. The
one-way latency is assumed to be half of the round-trip. We
measure all round-trips one by one and draw a distribution
graph.

The benchmark program is able to use different flavors
of completion notification mechanisms so as to be able to
compare their performance. The following are implemented:

* Polling: the application calls PtlEQWait from Portals library, configured to actively poll the NIC;

* Classic interrupts: the application calls *PtlEQWait* from Portals library, configured to do a system call that sleeps while waiting for an interrupt. This method is *passive waiting*;

* User-level interrupts: the application receives directly the user-level interrupt and does not call *PtlEQWait*. At initialization, the application registers a callback; when a packet arrives, a UINTR is triggered by the NIC and the registered callback is called by the *ui_handler*. The *PtlPut* for the reply of the ping-pong is sent directly from the callback. This method is *event-based*, fully in user space.

Performance results are shown in Figure 6 as a density
plot. The x-axis is the latency in nanoseconds; the y-axis
is the density of the results for this latency. We have run
1000 000 iterations. For readability, we have cropped the
graph to 10 000 ns (10 µs), which leaves out some outliers;
the number of omitted outliers is indicated on the graph for
each line.

The median latencies are 1.5 µs for polling, 3.6 µs for user-level interrupts, and 9 µs for classic interrupts.

For each mechanism, we observe two accumulation peaks
600 ns apart from each other; we do not know the cause of this
second peak. For polling, the tallest peak contains 93% of all
samples, and the second peak 6.6%. For user-level interrupts
the tallest peak contains 92% of all samples, and the second
peak 7.7%. For classic interrupts the tallest peak contains
76.6% of all samples, and the second peak to 22.9%.

<figure><img src="image_7.png" /><figcaption>Fig. 6. One-way end-to-end latency.</figcaption></figure>

Classic interrupts involve system calls and obtain, as ex-
pected, a latency much higher than polling fully in user space.
They are actually 6 times slower. User-level interrupts are only
2.4 times slower than polling, which will make them more
acceptable in a HPC context.

Regarding jitter (variations in latency), we observe on the
graph that user-level interrupts get a peak, a distribution similar
to polling, while classic interrupts results are more dispersed.
This is expected when bypassing the OS [10] and is likely to
be beneficial to applications.

C. *Overlap evaluation*

Since we expect that *event-based* notification through user-
level interrupts should improve communication/computation
overlap, we evaluate here such overlap.

To do so, we use the MadMPI Benchmark suite [11] that
defines a metric called *overhead ratio* [1]. The benchmark
makes computations with non-blocking communications run-
ning at the same time using *MPI_Isend*, *MPI_Irecv*, and
*MPI_Wait*, either on the sender side, receiver side, or both.
We use the benchmark with both sides overlapping communi-
cations with computation in this paper. The metric represents
the overhead of the measured performance compared to perfect
overlap, normalized such as 0 corresponds to perfect overlap
(black on the graphs), 1 corresponds to failed or no overlap
(red on the graphs), and values larger than 1 corresponds
to a performance worse than if no overlap would have been
attempted (yellow on the graphs). Results are drawn as a 2D
heat-map to explore the space of message size × computation
time. The white line represents the network transfer time for
the given message size; on this line, the network transfer takes
the same duration as the computation.

<footnote n="2">²Without saving the FPU x87, SSE, AVX registers in the ui_handler</footnote>
<figure><img src="image_8.png" /><figcaption>Fig. 7. Overlap ratio with polling.</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>Fig. 9. Overlap ratio with user-level interrupt.</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>Fig. 8. Overlap ratio with classic interrupt.</figcaption></figure>

The benchmark targets MPI, but porting an MPI library on top of user-level interrupts is still an on-going work. Therefore, we extended the overlap benchmark suite to directly support Portals library, with a *rendez-vous* protocol similar to what is done in MPI libraries: the sender sends a *rendez-vous request*; when the receiver is ready, it responds with a *ready to receive*; then the sender sends data.

Figure 7 shows the results obtained with active polling. We observe that there is almost no overlap. This is expected: when the application is busy with computation, it does not call the communication library, and thus it cannot reply to the received rendez-vous request. Communication starts only at the end of computation. However, the send of the rendez-vous request itself is overlapped but it is very short; it appears as half overlapping (ratio = 0.5) for small messages, when the rendez-vous itself represents a non-negligible fraction of the total communication time.

Figure 8 shows the results for classic interrupts. It uses

passive waiting, relying on a blocking system call. Therefore it is not surprising to notice that it obtains the same behavior as active polling: the system calls supposed to make communication progress are not called when the process is busy with computation, and thus the rendez-vous protocol does not progress. We observe a less smooth graph, probably due to this method being more subject to system noise since the kernel is involved.

Figure 9 shows the results for user-level interrupts. The obtained overlap is very good: it is near to perfect for small messages, because of a non-overlapable part (initialization and handler), and perfect for larger messages where this non-overlapable part becomes negligible. This is the expected behavior with *event-based* notifications: the rendez-vous is able to progress even when the application is busy doing computation, since the UINTR interrupts the computation when needed to make it progress. Network progression is performed directly in the interrupt handler.

We conclude from our evaluation that user-level interrupts have a latency much lower than classic interrupts and enable a very good communication/computation overlap, which should make them more usable in the context of HPC.

## V. RELATED WORK

Event-based notification is an old idea with existing literature on the topic.

The idea was popularized by Active Messages [12], an event-based communication library with support for interruptions; it is based on outdated hardware though (the nCUBE/2 from 1989). The same idea was present in U-Net [13], which already supported user-level interrupts in its model, but without practical implementation for user-level interrupts as far as we know.

The idea of using interrupts is widespread for networks programmed through the kernel. The balance between classic interrupt and polling as been studied and modeled [14], [15].
Events coming from interrupts may be forwarded to user space as signals, like SIGIO [2], which is widely used in non-HPC applications [3]. However, all these mechanisms rely on the kernel, which is known [16] for causing latency overhead.

For HPC networks, programmed from user space, InfiniBand uses interrupts for its *completion channel*. However, it implements only *passive waiting* through a blocking system call, and does not provide any mechanism to forward the event up to user space. Moreover, the OS being involved, performance is much lower than active polling [17].

For the BXI network, interrupts are supported [18] for in-kernel uses. In user space, they may be used for *passive waiting*.

To combine the best of both worlds, there are approaches that rely on polling threads doing active polling, then trigger events in the application, such as FastWake [19] or pioman [20]. However, they need to perform active polling and thus consume resources.

Finally, user-level interrupts [4] are a perfect solution for *event-based* programming fully in user space, but they officially only support inter-process communication, not signaling from a network card.

All in all, event-based completion notification is an idea floating in the air for good reactivity and overlap, but there is no pre-existing work fully in user space for high-performance networks.

## VI. CONCLUSION

To amortize the cost of communication through communication/computation overlap, *event-based* notifications for completion is the preferred mechanism. However, using interrupts causes a latency overhead that prevents its use in HPC communications. User-level interrupts lower the cost of interrupts, but are only usable between processes, not from a device.

We have presented mechanisms that extend the scope of user-level interrupts, so as to be able to trigger an UINTR from a PCIe discreet end-point. This proposed mechanism relies on an original use of the *interrupt posting* mechanism from the IOMMU to access memory and throw an interrupt atomically. We have implemented this mechanism in the BXI network card and software stack from Eviden. We have evaluated its performance and shown that it decreases the latency by a 2.5 factor compared to classic interrupts. We have studied the gain it brings for overlap by using a minimalist library that mimics basic MPI operations, and shown that it exhibits much better overlap properties than *active polling* and classic interrupts. The gain in latency and the good overlap should make it a good candidate for use with HPC applications.

In the future, we plan to port full MPI libraries on top of it, such as the *NewMadeleine* communication library [21] that is designed to take profit from network progression mechanisms. This will allow to study the benefit of our work on latency and overlap on real MPI applications.

## ACKNOWLEDGMENT

This project has received funding from the European High-Performance Computing Joint Undertaking (JU) under grant

agreement No 101175702. The JU receives support from the European Union's Horizon 2020 research and innovation programme and France, Greece, Germany, Spain, Italy, Switzerland.

## REFERENCES

[1] A. Denis and F. Trahay, "MPI Overlap: Benchmark and Analysis," in *International Conference on Parallel Processing*, ser. 45th International Conference on Parallel Processing, Philadelphia, United States, Aug. 2016. [Online]. Available: https://inria.hal.science/hal-01324179

[2] S. J. Leffler, W. N. Joy, and R. S. Fabry, "4.2 bsd networking implementation notes," University of California, Berkeley, Computer Science Division, Department of ..., Tech. Rep., 1983.

[3] N. Mathewson, "Fast portable non-blocking network programming with libevent," URL: http://www.wangafu.net/~nickm/libevent-book, 2012.

[4] S. Mehta, "x86 user interrupts support," *Linux Weekly News*, sep 2021, https://lwn.net/Articles/869140/ last visite 2024-04-11. [Online]. Available: https://lwn.net/Articles/869140/

[5] S. Derradji, T. Palfer-Sollier, J.-P. Panziera, A. Poudes, and F. W. Atos, "The bxi interconnect architecture," in *2015 IEEE 23rd Annual Symposium on High-Performance Interconnects*, 2015, pp. 18-25.

[6] *Intel® 64 and IA-32 Architectures, Software Developer's Manual*, https://www.intel.com/content/www/us/en/developer/articles/technical/intel-sdm.html last visite 2024-04-11. [Online]. Available: https://www.intel.com/content/www/us/en/developer/articles/technical/intel-sdm.html

[7] *PCI Express® Base Specification Revision 5.0*, https://pcisig.com/specifications/pciexpress/ last visite 2025-04-15. [Online]. Available: https://pcisig.com/specifications/pciexpress/

[8] J. Pan, "Coalesced interrupt delivery with posted msi," *Linux Weekly News*, apr 2024, https://lwn.net/Articles/970925/ last visite 2025-02-20. [Online]. Available: https://lwn.net/Articles/970925/

[9] "Intel linux kernel fork with uintr," https://github.com/intel/INTR-linux-kernel last visite 2024-04-11. [Online]. Available: https://github.com/intel/INTR-linux-kernel

[10] E. Vicente and R. Matias, "Modeling and simulating the effects of os jitter," in *2013 Winter Simulations Conference (WSC)*, 2013, pp. 2151-2162.

[11] A. Denis and F. Trahay, "MadMPI benchmark," https://pm2.gitlabpages.inria.fr/mpibenchmark/.

[12] T. Eicken, D. Culler, S. Goldstein, and K. Schauser, "Active messages: A mechanism for integrated communication and computation," in [*1992 Proceedings of the 19th Annual International Symposium on Computer Architecture*], 1992, pp. 256-266.

[13] T. von Eicken, A. Basu, V. Buch, and W. Vogels, "U-net: a user-level network interface for parallel and distributed computing," *SIGOPS Oper. Syst. Rev.*, vol. 29, no. 5, p. 40-53, dec 1995. [Online]. Available: https://doi.org/10.1145/224057.224061

[14] D. Perkovic and P. J. Keleher, "Responsiveness without interrupts," in *Proceedings of the 13th International Conference on Supercomputing*, ser. ICS '99. New York, NY, USA: Association for Computing Machinery, 1999, p. 101-108. [Online]. Available: https://doi.org/10.1145/305138.305172

[15] K. Salah, K. El-Badawi, and F. Haidari, "Performance analysis and comparison of interrupt-handling schemes in gigabit networks," *Computer Communications*, vol. 30, no. 17, pp. 3425-3441, 2007, special Issue Concurrent Multipath Transport. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0140366407002691

[16] L. Barroso, M. Marty, D. Patterson, and P. Ranganathan, "Attack of the killer microseconds," *Commun. ACM*, vol. 60, no. 4, p. 48-54, Mar. 2017. [Online]. Available: https://doi.org/10.1145/3015146

[17] P. MacArthur and R. D. Russell, "A performance study to guide rdma programming decisions," in *2012 IEEE 14th International Conference on High Performance Computing and Communication and 2012 IEEE 9th International Conference on Embedded Software and Systems*, 2012, pp. 778-785.

[18] M. Barbe, "Rapport de projet de fin d'études : Design and implement interrupt-handling code of a low-latency network adapter linux driver," 18 mars 2019 - 27 septembre 2019, chez Bull (Atos). Grenoble INP - ENSIMAG.
[19] B. Li, Z. Xiang, X. Wang, H. Ruan, J. Zhou, and K. Tan, "Fastwake: Revisiting host network stack for interrupt-mode rdma," in *Proceedings of the 7th Asia-Pacific Workshop on Networking*, ser. APNet '23. New York, NY, USA: Association for Computing Machinery, 2023, p. 1-7.
[Online]. Available: https://doi.org/10.1145/3600061.3600063

[20] A. Denis, "pioman: a pthread-based Multithreaded Communication Engine," in *Euromicro International Conference on Parallel, Distributed and Network-based Processing*, Turku, Finland, Mar. 2015. [Online].
Available: https://inria.hal.science/hal-01087775

[21] O. Aumage, E. Brunet, N. Furmento, and R. Namyst, "NewMadeleine: a Fast Communication Scheduling Engine for High Performance Networks," in *Workshop on Communication Architecture for Clusters (CAC 2007), workshop held in conjunction with IPDPS 2007*, Long Beach, California, United States, Mar. 2007. [Online]. Available: https://inria.hal.science/inria-00127356