<figure><img src="image_1.png" /></figure>

HAL
open science

Understanding Intel User Interrupts

Yves Kone, Louis Duval, Renaud Lachaize, Pascal Felber, Daniel Hagimont,
Alain Tchana

► To cite this version:

Yves Kone, Louis Duval, Renaud Lachaize, Pascal Felber, Daniel Hagimont, et al. Understanding Intel User Interrupts. Proceedings of the ACM on Measurement and Analysis of Computing Systems, 2025, Proceedings of the ACM on Measurement and Analysis of Computing Systems, 9 (2), pp.1-32. 10.1145/3727132 . hal-05115268

HAL Id: hal-05115268

https://hal.science/hal-05115268v1

Submitted on 16 Jun 2025

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Understanding Intel User Interrupts

YVES KONE, IRIT, Université de Toulouse, CNRS, Toulouse INP, UT3 Toulouse, France

LOUIS DUVAL, Univ. Grenoble Alpes, CNRS, Inria, Grenoble INP, LIG, 38000 Grenoble, France

RENAUD LACHAIZE, Univ. Grenoble Alpes, CNRS, Inria, Grenoble INP, LIG, 38000 Grenoble, France

PASCAL FELBER, University of Neuchâtel, Switzerland

DANIEL HAGIMONT, IRIT, Université de Toulouse, CNRS, Toulouse INP, UT3 Toulouse, France

ALAIN TCHANA, Univ. Grenoble Alpes, CNRS, Inria, Grenoble INP, LIG, 38000 Grenoble, France

User interrupts (UINTR) is a new hardware feature available in recent Intel Xeon processors. It enables a CPU to send, receive, and handle hardware inter-processor interrupts directly in user mode, meaning without the intervention of the OS kernel. In this paper, we shed light on the inner working of this facility and investigate, with detailed performance and overhead measurements considering deployments on physical machines and virtual machines with Linux and KVM, how it can be leveraged for more efficient inter-process communications, as an alternative to OS (Unix) signals. With UINTR, preemptive user-level thread schedulers can achieve quantum lengths in the microsecond scale.

CCS Concepts: • Software and its engineering → Process synchronization; Virtual machines; • Computer systems organization; • General and reference → Performance;

Additional Key Words and Phrases: user interrupts, inter-processor interrupts, virtualization, scheduling, performance

**ACM Reference Format:**

Yves Kone, Louis Duval, Renaud Lachaize, Pascal Felber, Daniel Hagimont, and Alain Tchana. 2025. Understanding Intel User Interrupts. *Proc. ACM Meas. Anal. Comput. Syst.* 9, 2, Article 40 (June 2025), 32 pages. https://doi.org/10.1145/3727132

# 1 Introduction

Hardware feature characterization is generally carried out by the academic community [1, 3, 12, 26, 30, 34], which is often best positioned to provide a detailed and objective understanding of the benefits and limitations of new hardware features. For example, after conducting an in-depth study of Intel PML, Bitchebe et al. [4] concluded that it only reduced the overhead of live virtual machine (VM) migration by 0.95%, despite this feature being specifically designed for that purpose. All in all, the characterization of hardware features is essential for both (systems) researchers and application developers, guiding their usage, as well as for manufacturers, potentially influencing the evolution or discontinuation of the features.

Authors' Contact Information: Yves Kone, IRIT, Université de Toulouse, CNRS, Toulouse INP, UT3 Toulouse, France, yves.kone@ens-lyon.fr; Louis Duval, Univ. Grenoble Alpes, CNRS, Inria, Grenoble INP, LIG, 38000 Grenoble, France; Renaud Lachaize, Univ. Grenoble Alpes, CNRS, Inria, Grenoble INP, LIG, 38000 Grenoble, France, renaud.lachaize@univ-grenoble-alpes.fr; Pascal Felber, University of Neuchâtel, Switzerland, pascal.felber@unine.ch; Daniel Hagimont, IRIT, Université de Toulouse, CNRS, Toulouse INP, UT3 Toulouse, France, daniel.hagimont@irit.fr; Alain Tchana, Univ. Grenoble Alpes, CNRS, Inria, Grenoble INP, LIG, 38000 Grenoble, France, alain.tchana@grenoble-inp.fr.

© 2025 Copyright held by the owner/author(s). Publication rights licensed to ACM.
This is the author's version of the work. It is posted here for your personal use. Not for redistribution. The definitive
Version of Record was published in *Proceedings of the ACM on Measurement and Analysis of Computing Systems*, https:
//doi.org/10.1145/3727132.
In this paper, we study *Intel User-level Interrupts* (UINTR, also known as “UIPIs”) [7, 17], a recent hardware feature integrated by Intel into its 4th generation (“Sapphire Rapids”) Xeon processors.<fnref n="1" /> UINTR allows a sender thread to instruct its CPU to send an interrupt to another CPU, triggering the execution of a routine previously set up by the receiver thread (within the same process or a different process), *without any intervention of the operating system (OS) kernel*. UINTR is therefore a new mechanism within the family of Inter-Process/Thread Communication (IPC) facilities, which are at the core of any operating system (OS). This feature provides an efficient means of coordination between CPU cores, which is essential to enable *µsecond-scale scheduling* [9, 10, 20, 23, 25, 26] in modern cloud applications, as explained below.

With the recent development of critical networking applications, such as in-memory key-value stores, the presence of very short jobs (on the *µsecond scale* [25]), combined with a broad service time distribution [5], has made achieving low tail latency more challenging [3]. It is more likely that very short requests end up queuing behind longer ones, resulting in head-of-line blocking [25]. OS-level schedulers, which assign time slices on the order of ms (typically 1–10 ms in Linux), cannot achieve *µs-scale tail latencies* [21, 22].

A promising approach to meet *µs-scale tail latency* is to transform application threads into user-level threads (uThreads), which are then scheduled on top of kernel-level threads (kThreads) by a specific *userspace uThread scheduler*. Several systems following this trend have been developed in recent years, including Shenengo [26], Shinjuku [20], Caladan [10], LibPreemptible [23], Junction [9], and Tiny Quanta [25]. In this approach, the quantum duration is crucial for controlling tail latency [20, 23, 26]. In order to support *µs-scale quantum sizes*, a first design option is to rely on uThreads to voluntarily yield. Tiny Quanta [25] is the latest solution using this cooperative strategy, offering quantum sizes down to 1 *µs*, yet with two limitations (as acknowledged by its authors): (1) it requires fully trusting the application code with respect to scheduling constraints, and (2) it does not guarantee precise adherence to the quantum duration as the points where the uThreads yield cannot always be finely selected. This imprecision in quantum length leads to performance unpredictability, which is detrimental for implementing auto-scaling policies.

In contrast, another possible design option is to use *preemption* when a time quantum expires. This strategy is more precise and accommodates more applications, but its viability with very small quanta depends on the latency required to send the preemption signal to the kThreads. Junction [9] and LibPreemptible [23] are the most recent works in this area. For instance, LibPreemptible [23] supports a quantum size down to 5*µs*. These articles provide insights into UINTR's potential, but since their studies do not specifically focus on UINTR, they do not fully explore it.

To our knowledge, no study so far has thoroughly characterized UINTR, which is our goal in this paper. In particular, we aim to answer questions with important practical ramifications for cloud infrastructures, such as the following ones: (1) *how fast are the basic UINTR operations?*, (2) *does UINTR maintain the same level of performance when used in a virtual machine*, given that virtualization is ubiquitous in the cloud, and (3) *what level of performance and predictability can UINTR provide for preemptive user-level thread schedulers?* Our study environment is Linux, with KVM virtualization. Intel already offers a UINTR patch for Linux.

We developed a set of micro-benchmarks and one macro-benchmark. The micro-benchmarks evaluate the new instructions introduced by UINTR. For the macro-benchmark, we develop Christine, a preemptive user-level scheduler (PUTS). We designed Christine to be as flexible as possible, aiming to test various mechanisms and approaches. In terms of mechanisms, Christine can easily support any signaling mechanism (OS signals, UINTR, etc.). Regarding approaches, we consider two

<footnote n="1">¹The concept of user-level interrupts has also garnered interest beyond Intel platforms: e.g., it has been drafted and prototyped in the RISC-V architecture [6].</footnote>
types of preemptions: with and without synchronization. Here, synchronization refers to a setup in which the next preemption notification cannot be issued until the per-core local schedulers have acknowledged the delivery and the processing of the previous preemption notification. In contrast, without synchronization, a new preemption is issued as soon as possible, when the quantum length is reached, without coordination with the local schedulers on the target cores.

We identify four important metrics for assessing a PUTS: *efficiency* (the proportion of time spent executing useful work), *minimum latency* (the minimum effective quantum length that can be achieved), *stability* (the standard deviation of the effective quantum lengths) and *focus* (the ability to guarantee a desired quantum length).

To establish a baseline, we systematically compare UINTR to OS signals (OSS for short), the standard facility for asynchronous interactions between threads or processes. Our evaluations encompass a large spectrum of deployment contexts including physical and virtual machines (with and without hardware-assisted IPI virtualization) and also study the impact of thread placement and interference.

To summarize, this paper makes the following contributions<fnref n="2" />:

* We develop benchmarks for the performance characterization of asynchronous notification mechanisms, and we perform a qualitative and quantitative study of both OSS and UINTR, including a detailed breakdown of their costs, in both native and virtualized environments.

* We design and implement Christine, a flexible preemptive user-level thread scheduler (PUTS). To better understand the behavior of microsecond-scale PUTS, we introduce a precise methodology, based on new scheduling metrics, and we empirically evaluate several versions of Christine, covering OSS and UINTR, as well as preemptions with and without synchronization.

* We derive several findings $F_X$ to guide developers, cloud operators, and hardware manufacturers: ($F_1$) UINTR outperforms OSS for the sending latencies (by up to 25× in native mode and 34× in virtualized mode), the delivery latencies (by up to 6× in native mode and 3× in virtualized mode), and also for throughput (by up to 6× in native mode and up to 2× in virtualized mode). ($F_2$) Virtualization negatively impacts the performance of UINTR operations. Nonetheless, compared to IPI emulation at the hypervisor level, Intel's IPIv feature can significantly reduce the overhead of UINTR virtualization, containing it within a maximum degradation range of 1.4× to 2.6×. ($F_3$) For Christine, UINTR allows using a scheduling quantum length as low as 2µs or even slightly below 1µs for some configurations. ($F_4$) When using UINTR instead of OSS, Christine achieves better focus (up to 7×), stability (up to 8×), efficiency (up to 2×) and latency (up to 7×) in native and virtualized modes.

The remainder of this article is organized as follows: §2 presents the necessary background about OSS and UINTR, §3 presents the experimental setup we use for evaluations, §4 and §5 present the results for the micro- and macro-benchmarks respectively, §6 discusses the related work, and §7 concludes the paper.

## 2 Background

Regarding the emission and delivery of a notification, UINTR follows the same logical steps as OSS, with the key difference being that all its steps, except for initialization, are handled by the hardware, without kernel mediation. We consider Linux for illustration. We first summarize the main aspects of OSS (§2.1) before providing a description of UINTR (in §2.2 to §2.4). Some additional details about UINTR and OSS are also available in Appendix A. Throughout this section, we consider a

<footnote n="2">²Our code is available at https://gitlab.com/krakos-INTR/understanding-INTR.git</footnote>
sender-receiver interaction pattern between two threads $T_1$ and $T_2$, which may belong to different processes or to the same one.

## 2.1 OS-level Signals (OSS)

*Signal Sending.* Consider thread $T_1$ sending signal sig to thread $T_2$ via the `pthread_kill()` function. Let us assume that $T_2$ has overridden the default disposition of sig. During the execution of the underlying `tgkill()` syscall in kernel space, a signal event structure is filled out and queued into $T_2$'s signal queue, and $T_2$'s `TIF_SIGPENDING` flag is set. This flag indicates to the kernel that there are pending signals for $T_2$, which will be processed during the next switch of $T_2$ from kernel to user mode. If $T_2$ is currently running on a CPU (noted C), the kernel fires an Inter-Processor Interrupt (IPI) at destination of C with the `RESCHEDULE VECTOR` interrupt vector. This ensures that $T_2$ promptly detects the pending signal (see below). Meanwhile, $T_1$ returns to user mode and continues execution.

*Signal Delivery.* The signal takes effect during the next context switch of $T_2$ back to user mode by calling `do_signal()`. In fact, the kernel checks for the existence of pending signals every time it finishes handling an interrupt or an exception. `do_signal()` checks $T_2$'s `TIF_SIGPENDING` flag. If the latter is set, the kernel dequeues a pending signal and invokes `handle_signal()` to process it. `handle_signal()` then jumps to the corresponding routine (transitioning the CPU from kernel to user mode) registered by $T_2$. After the routine completes execution, it automatically performs the syscall `sys_sigreturn()`, restoring the user register contents from the kernel stack frame and returning to the original $T_2$ code in user mode.

All these mode transitions are the primary cause of the slowness of OSS, leading to their significant overhead on preemptive user-level schedulers. UINTR eliminates the need for OS mediation during notification sending and handling.

## 2.2 UINTR Overview

In a nutshell, UINTR relies on two main types of (per-thread) data structures allocated and initialized in kernel space, *UPID* (*User Posted Interrupt Descriptor*) and *UITT* (*UINTR target table*), as well as several privileged CPU registers, and the *Local Advanced Programmable Interrupt Controller* (*LAPIC*) of each core.

A UPID structure corresponds to a mailbox for a recipient thread; it indicates where and how the recipient thread must be informed, e.g., which recipient LAPIC instance and which interrupt vector number to use, and it also keeps track of pending interrupt requests. A UITT is a sender-side lookup table, which contains references to the authorized UPIDs that can be targeted by a given sender CPU.<fnref n="3" /> Each entry corresponds to a (*UPID*, *request#*) pair.

A sender thread can issue a user interrupt via `SENDUIPI`, a new (unprivileged) x86 hardware instruction, which takes as input an entry number within the UITT instance currently set on the sender core. This results in posting the notification in the recipient UPID. Note that the execution of `SENDUIPI` does not always result in an inter-processor interrupt (IPI). Indeed, the delivery of a UINTR, which results in the execution of the application-provided handler, may be postponed depending on the state of the receiver. The delivery is only triggered when the recipient thread (1) is running and (2) not currently masking/buffering UINTR requests. Thus, a sender thread only triggers an IPI towards a core if, upon execution of `SENDUIPI`, the two above-mentioned conditions are met. Note that this workflow is conceptually similar to the way in which Linux conditionally leverages IPIs for the delivery of OS signals (OSS). Similarly to OSS, UINTR also provides a set of

<footnote n="3">³Although we describe the sender and receiver roles separately, each CPU/thread can act as both.</footnote>
operations to (1) read and modify the mask controlling the buffering of requests (TESTUI, CLUI, STUI), and (2) trigger the termination of a request handler (UIRET).

Although the OS kernel is not in the critical path for sending and delivering user interrupts, it is nonetheless involved for two key aspects: (1) upon initialization, enabling interrupts and registering a handler for a receiver thread, and then pairing one or several authorized sender threads with a receiver, and (2) upon a context switch (between kernel threads), adjusting the contents of the UINTR-related registers and structures, thus allowing to use UINTR at the thread-level granularity (by switching between different UPID/UITT instances and also updating the thread-CPU mappings and buffered-delivery flags).

## 2.3 UINTR Programming Interface

Listing 1 outlines each step involved when setting up UINTR-based communication between a sender and a receiver thread (within the same process or not). Initially, the receiver thread declares a routine (line 3 in Listing 1) that will be invoked by the CPU upon receiving a UINTR. It requests the kernel to create and initialize a UPID data structure (line 10 in Listing 1). Subsequently, the sender thread, aiming to trigger the handler routine of a specific receiver thread, retrieves from the kernel the UPID of the latter.<fnref n="4" /> It then instructs the kernel to initialize a UITT, adding a new entry to describe the receiver thread and a specific UINTR request type between 0 and 63 (line 18). All these steps are performed only once and for all.

To initiate the emission of a UINTR request, the sender thread simply invokes a wrapper function that encapsulates a SENDUIPI instruction.

<figure><img src="image_3.png" /><figcaption>Listing 1. Main steps required before using UINTR (pseudo-code).</figcaption></figure>

<footnote n="4">⁴For security reasons, this is only possible if the sender has access to the file descriptor linked with the UPID of the receiver, exported by the latter.</footnote>
## 2.4 UINTR Communication Workflow

Below, we describe the main steps followed by the sender and the receiver CPUs on the critical path. Note that, on both sides, some steps require accessing in-memory data structures (UITT and UPID on the sender, UPID on the receiver).

**Sender CPU.** It executes the `SENDUIPI` instruction, which consists of two steps: *UIINTR posting* and *UIINTR notification*. `UIINTR posting` is performed systematically; it consists in registering the `UIINTR` request in the receiver's UPID, by setting a bit corresponding to the request type (0-63). `UIINTR notification` involves sending an IPI to the destination CPU. This second step is only performed when the recipient thread is running; the decision is based on the current state of the receiver's UPID.

**Receiver CPU.** Note that the reaction of a receiver CPU to a `UIINTR` request is always initiated by an IPI, which is either triggered by the core of the sender thread (if the recipient thread is running upon the execution of `SENDUIPI`), or otherwise by the receiver itself upon rescheduling a recipient thread with a pending `UIINTR` (self-IPI triggered by the Linux kernel scheduler).

Upon the arrival of an IPI notifying a `UIINTR`, a receiver CPU systematically goes through the following three steps. (1) *UIINTR identification*: when a CPU receives an interrupt vector *V*, it checks if the latter vector corresponds to a `UIINTR` or a standard interrupt managed via the classic IDT (Interrupt Descriptor Table) in kernel mode. (2) *UIINTR notification processing*: The CPU saves the bitmap storing the pending interrupt requests of the UPID and resets it. It then performs the *UIINTR delivery* step for each active bit in the saved bitmap. (3) *UIINTR delivery*: The CPU jumps to the address of a pre-configured routine. Before that, it saves the current execution context, as well as the routine's input argument, onto the stack set up for the `UIINTR` handler routine. If several requests are pending, they are delivered in sequence, starting from the one with the highest type. Note that each UPID instance uses a bitmap to manage pending requests and thus cannot keep track of more than one pending request per type (0-63).

## 3 Evaluation Goals and Experimental Setup

The goal of our paper is to conduct comprehensive experiments to understand the performance levels achievable with `UIINTR`, thus enabling developers and data center operators to make more informed decisions about optimizations for granular, microsecond-scale applications. More precisely, by systematically comparing `UIINTR` with OSS as a baseline, we aim to answer the following questions, regarding both physical and virtual machines. ($Q_1$) (§4) What are the costs and the performance variability of the basic `UIINTR` operations on the critical path? How are these metrics impacted by thread placement and contention? ($Q_2$) (§5) How does `UIINTR` improve the performance of multicore microsecond-scale uThread schedulers? For which dimensions and to which extent?

**Testbed.** The test machine is a Supermicro Super Server equipped with an Intel Xeon Gold 5415+ processor. It features 8 cores, each with 2 hyperthreads, running at 2.901 GHz. Each core includes a 32KB L1 instruction cache, a 32KB L1 data cache, a 2048KB L2 cache, and a shared 23MB L3 cache across all cores. The system is equipped with 120GB of RAM. The processor supports Intel VT-x and virtualized IPIs. We use Ubuntu 22.04 LTS, Linux kernel version 6.0.0 modified with Intel's `UIINTR` patch [14], and KVM:QEMU version 6.2.0.

**Configurations.** We study multiple configurations, which differ based on the execution environment: native (noted *NATIVE*), guest OS with software-only IPI virtualization (hypervisor mediation, noted *GIPI*), guest OS with hardware-assisted IPI virtualization (Intel IPIVs, noted *GIPIV*). In the *GIPI* configuration, each guest-level IPI must be emulated by the host, which requires a VMExit
<figure><img src="image_4.png" /><figcaption>Fig. 1. Hardware topology of the testbed machine and placement configurations. In (R,S1), the sender and receiver thread are pinned on the same hyperthread. In (R,S2) they are on the same core but different hyperthreads. In (R,S3), they are on different cores.</figcaption></figure>

on the sender and receiver vCPUs. In contrast, with the GIPV configuration, a guest-level IPI is
directly handled by the hardware and does not cause any VMExit.

In addition, we study to which extent UINTR performance is impacted by the sharing of micro-
architectural resources between the sender and receiver threads: we consider the three placements
of the sender (Sᵢ) and receiver (R) as shown in Figure 1, in addition to another configuration (Self)
where the sender and receiver are the same thread.

*Methodology.* We used *rdtsc* to measure the latency of the operations being evaluated. Intel's documentation indicates that the *mfence* and *lfence* instructions should be used to improve the precision of fine-grained time measurements. We accordingly developed a function *rdtsc*() (the listing is available in Appendix B.1). For greater accuracy, we measure individually each call to the evaluated operations and then calculate the 50^th^ and 99^th^ percentiles.

*Security considerations.* The availability of unprivileged hardware-managed IPI facilities (i.e., UINTR and/or IPIv) to untrusted applications and guest virtual machines introduces new possibilities of side-channel attacks [28]. While we hope that our benchmarks and performance measurements of these facilities may be of interest to the security community, such kinds of attacks, as well as more general security considerations, are out of the scope of the present paper. A brief discussion of denial-of-service attacks is nonetheless provided in Appendix E.

# 4 Micro-benchmarks

## 4.1 Introduction

We have developed a suite of microbenchmarks to characterize the latencies of the elementary UINTR operations on the sending and receiving sides, as well as the delivery throughput. We also developed similar microbenchmarks for the equivalent OSS operations. Below, we discuss some methodological aspects. Additional information is provided in Appendix B.2.

Regarding the `SENDUIPI` operation, we consider two variants, resp. dubbed *"Posting"* and *"Posting + Notification"*. In the first variant, the sender only sets the `UIINTR` request into the target `UPID`. In the second variant, the sender also initiates the IPI notification towards the receiver core. To perform the measurements accurately for the first (resp. second) variant, we had to ensure that the target `UPID` state would never (resp. always) lead the sender to initiate an IPI. These precautions required small modification of the Linux scheduler code.

The reported numbers for the `UIINTR/OSS` delivery latencies correspond to an upper bound. Indeed, the measurements are performed from the sender-side (except for the "self" configuration, which allows more accuracy), i.e., they include the time needed to retrieve the acknowledgment of the receiver (via spinning on a shared-memory variable).
By default (contention-free case), we use a single pair of sender and receiver threads, and we ensure that the receiver is running when the sender initiates the request (except in the special cases involving pinning on the same hyperthread or self-request). We also consider contention scenarios in which (1) multiple sender threads (on different CPU hardware threads) concurrently issue requests towards the same receiver, or (2) the receiver is sharing its CPU (hardware thread) with other application threads.

The remainder of this section is organized as follows: first (§4.2), we discuss impacting factors (thread placement and virtualization), then we compare UINTR and OSS performance in contention-free scenarios (§4.3), and finally, we study the impact of contention on senders or receivers (§4.4).

## 4.2 UINTR Impacting Factors

*Thread placement Impact.* Table 1 shows the various costs associated with the SENDUIPI instruction and the latency of interrupt delivery based on sender-receiver placements. We focus here on the NATIVE configuration. The results for SENDUIPI *Posting* are fairly close for all configurations, with a slight penalty for (R,S2) due to potential interference between siblings hyperthreads. In the case of SENDUIPI *Posting + Notification*, the latencies are higher due to the extra cost of the IPI, which increases with the distance (in terms of micro-architectural wiring and cache hierarchy) to the receiver (hardware) thread. A cross-core *Posting + Notification* operation with the (R, S3) placement has a 1.8× higher 99th-percentile latency than an intra-core operation with the (R, S2) placement. In addition, with (R, S3), the overhead introduced by the *Notification* step is more noticeable with (R, S3) than with (R, S2): respectively 2.3× and 1.06×.

Regarding UINTR delivery latencies, we can see in Table 1 that they also increase with the distance: at the 99th percentile, delivery for (R, S3) takes respectively 1.3× and 1.5× than for (R, S2) and Self. However, there is a notable exception: placement (R,S1) performs the worst, and very poorly, because the two threads contend for the same CPU (hardware thread), causing deferred delivery due to head-of-line blocking.

**Take-away 1:** *Increasing the physical distance between distinct sender and receiver (hardware) threads degrades UINTR delivery latency by up to 1.3×.*

*Virtualization Impact.* Figure 2 illustrates the overhead introduced by virtualization during UINTR sending and delivery operations. For the UINTR *Posting* operation, in GIPI mode, we observe an overhead of 12% to 24% at the 50th percentile, and of 14% to 39% at the 99th percentile. Quite similarly, in GIPIV mode, the overhead ranges from 12% to 25% at the 50th percentile and 13% to 39% at the 99th percentile. Given that UINTR *Posting* (without notification) essentially consists in accessing in-memory data structures (the sender's UITT and the recipient's UPID), we mainly attribute the posting overheads in GIPI and GIPIV modes to the impact of EPT-based memory virtualization, which significantly increases the costs of TLB misses, as they require a two-dimensional page table walk [32].

We now consider the UINTR *Posting + Notification* operation. In GIPI mode, this operation is between 7× and 32× more expensive than NATIVE at the 50th percentile, and between 7× and 35× more expensive at the 99th percentile. This overhead is due to VMExits required to perform IPIs at the hypervisor level. In GIPIV mode, the virtualization overhead is less severe but still significant: the cost is approx 2× higher than NATIVE, both at the 50th percentile and the 99th percentile. This overhead is mainly due to the Posted-Interrupt mechanism for IPI virtualization. Indeed, in the GIPIV case, the UINTR *Posting + Notification* operation triggers two (hardware-managed)
<table><caption>Table 1. Latency of sending and delivery operations for OSS and UINTR in different setups (native, virtualized with GIPI or GIPIV). Each cell contains the 50th and the 99th percentiles (in cycles) separated by a dash.</caption><thead><tr><th colspan="5">NATIVE (cycles)</th></tr></thead><tbody><tr><th colspan="5">Pinning Mode</th></tr><tr><td>pthread_kill()</td><td>R,S1</td><td>R,S2</td><td>R,S3</td><td>Self</td></tr><tr><td>Signal Delivery</td><td>1.4k-1.6k<br>46M-58M</td><td>2.5k-3k<br>5.3k-5.8k</td><td>4.1k-4.7k<br>5.7k-6.1k</td><td>3.1k-3.4k<br>2.4k-2.7k</td></tr><tr><th colspan="5">SENDUIPI Posting</th></tr><tr><td>SENDUIPI Posting + Notif.</td><td>160-166<br>196-220</td><td>160-164<br>218-234</td><td>160-182<br>370-422</td><td>162-168<br>186-194</td></tr><tr><td>UIINTR Delivery</td><td>46M-58M<br>930-1k</td><td>370-422<br>1.2k-1.3k</td><td>838-856<br>1.2k-1.3k</td><td></td></tr></tbody><thead><tr><th colspan="5">GIPIV (cycles)</th></tr></thead><tbody><tr><th colspan="5">Pinning Mode</th></tr><tr><td>pthread_kill()</td><td>R,S1</td><td>R,S2</td><td>R,S3</td><td>Self</td></tr><tr><td>Signal Delivery</td><td>1.7k-1.9k<br>46M-47M</td><td>2.6k-3.6k<br>6.6k-7.4k</td><td>3.9k-7.1k<br>7.3k-8.1k</td><td>3.1k-3.6k<br>2.4k-2.7k</td></tr><tr><th colspan="5">SENDUIPI Posting</th></tr><tr><td>SENDUIPI Posting + Notif.</td><td>180-206<br>404-462</td><td>180-206<br>464-530</td><td>180-206<br>750-840</td><td>204-234<br>392-448</td></tr><tr><td>UIINTR Delivery</td><td>46M-47M<br>2.2k-2.6k</td><td>2.2k-2.6k<br>2.6k-2.9k</td><td>2.2k-2.9k<br>2.6k-2.9k</td><td>2k-2.2k<br>2k-2.2k</td></tr></tbody><thead><tr><th colspan="5">GIPI (cycles)</th></tr></thead><tbody><tr><th colspan="5">Pinning Mode</th></tr><tr><td>pthread_kill()</td><td>R,S1</td><td>R,S2</td><td>R,S3</td><td>Self</td></tr><tr><td>Signal Delivery</td><td>1.7k-1.9k<br>46M-46M</td><td>2.6k-6.8k<br>8k-9.1k</td><td>4.1k-7.7k<br>8.6k-9.5k</td><td>3.1k-3.6k<br>2.3k-2.7k</td></tr><tr><th colspan="5">SENDUIPI Posting</th></tr><tr><td>SENDUIPI Posting + Notif.</td><td>180-206<br>4.2k-5.6k</td><td>180-208<br>3.1k-3.6k</td><td>180-208<br>2.9k-3.2k</td><td>202-234<br>6k-6.8k</td></tr><tr><td>UIINTR Delivery</td><td>46M-46M<br>3.8k-4.3k</td><td>3.8k-4.3k<br>3.8k-4.2k</td><td>3.8k-4.2k<br>3.8k-4.2k</td><td>2.8k-3.1k<br>2.8k-3.1k</td></tr></tbody></table>

Posted-Interrupt operations (instead of one in the NATIVE case): the first one (similar to NATIVE) posts the request in the UPID of the recipient, and the second one (specific to GIPIV), posts the virtual interrupt to the target vCPU.

**Take-away 2:** In a guest VM, compared to a native setup, the performance of UINTR Posting + Notification operations is significantly degraded, by up to 2x with Intel IPv and up to 35x without. The UINTR Posting operation (without Notification) is degraded by up to 39%.

Significant overhead is also observed in delivery latency<fnref n="5" />. In GIPIV, this step is between 2.1x and 2.6x longer than in NATIVE. Indeed, in the case of a guest assisted with IPv, to handle a UINTR, the CPU must first evaluate the interrupt and then deliver it to the vCPU. If the interrupt is recognized and matches the UINTR vector, the processor proceeds to the “Virtualizing User-Interrupt Notifications” phase, marking the interrupt in the EOI register. The CPU then continues with the UINTR handling steps as in a native setup. These extra steps explain the observed overhead.

In GIPI, the latency is between 3.1x and 4.3x higher than in NATIVE. This high overhead is due to the IPI emulation performed by the hypervisor.

**Take-away 3:** Virtualization degrades UINTR delivery latency by up to 2.6x with Intel IPv and up to 4.3x without.

## 4.3 UINTR and OSS operations without contention

This section shows that, for most thread placements in the NATIVE and GIPIV configurations, the full bypass of the OS kernel allows all the elementary UINTR operations to drastically outperform their OSS counterparts.

Sending. According to Figures 3a and 3b (left and middle), UINTR sending operations are more efficient than their OSS equivalent, pthread_kill(). For the UINTR Posting operation (left-side

<footnote n="5">⁵ With the exception of the (R, S1) placement, where the dominating factor is the scheduling time of the receiver thread (due to the contention on the same hardware thread), as mentioned earlier.</footnote>
<figure><img src="image_5.png" /><figcaption>Fig. 2. Impact of Virtualization on UINTR. (Notice the different vertical scale for UINTR IPI in the GIPI setup.)</figcaption></figure>

part of the Figure), which corresponds to the `SENDUIPI` instruction without triggering an IPI, we see that it is between 8× and 25× faster than `pthread_kill()` for `NATIVE`, between 9× and 34× faster for `GIPIV`, and between 9× and 37× faster for `GIPI`.

For the `UIINTR Posting + Notification` operation (middle part of the Figure), which corresponds to the `SENDUIPI` instruction resulting in an IPI, we observe that it is between 7× and 17× faster than `pthread_kill()` for `NATIVE`, and between 4× and 8× faster for `GIPIV`. For `GIPI`, `pthread_kill()` performs better in some cases. In the `Self` configuration, `pthread_kill()` is 2× faster, but in the `(R,S3)` configuration, `SENDUIPI` is 1.4× to 2.4× faster.

**Take-away 4:** UINTR sending operations significantly outperform OSS, both in native mode and virtualized mode with Intel IPIv, by up to 25× and 34× respectively.

**Delivery.** UINTR provides better interrupt delivery latency. In the configurations `(R,S2)`, `(R,S3)`, and `(Self)`, UINTR is between 2.8× and 5.8× better for `NATIVE` and between 1.2× and 3× better for `GIPIV`. In the `(R,S1)` configuration, UINTR and OSS are equivalent across all three execution modes. We also observe higher latency for UINTR compared to OSS for GIPI in the `(Self)` configuration; this is because, in that case, the UINTR version requires an IPI (which itself causes a VM Exit), unlike the OSS version.

**Take-away 5:** UINTR delivery is up to 6× faster than OSS in native mode and up to 3× in virtualized mode. In virtualized mode without Intel IPIv, self delivering a UINTR is up to 20% slower than OSS.

**Throughput.** Table 2 shows the delivery throughput for OSS and UINTR for a given notification type (resp. `SIGUSR1` and `UIINTR` request #0). We can observe that UINTR provides better throughput across all execution modes, except in the `(R, S1)` configuration, for which OSS and UINTR have similar characteristics. In the `(R,S2)` configuration (which yields the highest throughput for both
<figure><img src="image_6.png" /><figcaption>Fig. 3. 50th percentile (up) and 99th percentile (down) performance improvement of UINTR w.r.t. OSS of Posting (left), Posting + Notification (middle) and delivery (right) operations. (Notice the different vertical scale for *Delivery* in both setups.)</figcaption></figure>

<table><caption>Table 2. Maximum delivery throughput and notification loss rate for OSS and UINTR. UINTR has a better maximum delivery throughput (bold values) and a worse notification loss rate than OSS. For both mechanisms, the (R,S2) thread placement leads to the best throughput.</caption><thead><tr><th rowspan="2">Execution environment</th><th colspan="3">NATIVE</th><th colspan="3">GIPIV</th><th colspan="3">GIPI</th></tr><tr><th>R,S1</th><th>R,S2</th><th>R,S3</th><th>R,S1</th><th>R,S2</th><th>R,S3</th><th>R,S1</th><th>R,S2</th><th>R,S3</th></tr></thead><tbody><tr><td><strong>Thread Placement</strong></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td><strong>OSS:</strong></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Delivery throughput (requests per sec)</td><td>31</td><td><strong>690k</strong></td><td>591k</td><td>32</td><td><strong>682k</strong></td><td>580k</td><td>31</td><td><strong>730k</strong></td><td>494k</td></tr><tr><td>Notification Loss Rate (%)</td><td>99.99</td><td>44</td><td>20</td><td>99.99</td><td>46</td><td>21</td><td>99.99</td><td>21</td><td>26</td></tr><tr><td><strong>UINTR:</strong></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Delivery throughput (requests per sec)</td><td>31</td><td><strong>4,529k</strong></td><td>3192k</td><td>31</td><td><strong>1,445k</strong></td><td>1156k</td><td>31</td><td><strong>849k</strong></td><td>831k</td></tr><tr><td>Notification Loss Rate (%)</td><td>99.99</td><td>83</td><td>79</td><td>99.99</td><td>99.95</td><td>99.95</td><td>99.99</td><td>83</td><td>85</td></tr></tbody></table>

<table><caption>Table 3. Latency (in CPU cycles) of additional operations for OSS and UINTR. Each cell contains the 50th and 99th-percentile latencies, separated by a dash.</caption><thead><tr><th>Operations</th><th>NATIVE</th><th>GIPIV</th><th>GIPI</th></tr></thead><tbody><tr><td><strong>Mask operations:</strong></td><td></td><td></td><td></td></tr><tr><td>sys_sigprocmask()</td><td>368-412</td><td>408-472</td><td>406-462</td></tr><tr><td>STUI</td><td>60-72</td><td>80-96</td><td>80-96</td></tr><tr><td>CLUI</td><td>40-50</td><td>60-74</td><td>62-74</td></tr><tr><td>TESTUI</td><td>74-76</td><td>94-98</td><td>94-98</td></tr><tr><td><strong>Handler return operations:</strong></td><td></td><td></td><td></td></tr><tr><td>sys_sigreturn()</td><td>888-940</td><td>914-1048</td><td>900-1032</td></tr><tr><td>UIRET</td><td>122-130</td><td>144-162</td><td>136-156</td></tr></tbody></table>

notification mechanisms), UINTR is 6.5×, 2.1×, and 1.16× better than OSS for, respectively, NATIVE, GIPIV, and GIPI. In the case of the (R, S3) placement (i.e., cross-core notifications), UINTR is 5.4×, 1.9×, and 1.6× better than OSS for, respectively, NATIVE, GIPIV, and GIPI. Note that both OSS and UINTR are lossy communication mechanisms, given that they only use one bit (per request type
and per recipient) to keep track of the pending notifications. Given that `UIINTR` has a significantly higher throughput, it is inherently more lossy (at full speed) than `OSS`.

**Take-away 6:** Regarding the maximum throughput between distinct hardware threads, `UIINTR` outperforms `OSS` by up to 6.5× in native mode and up to 2.1× in virtualized mode.

*Other Instructions.* Referring to Table 3, we can see that the instructions related to `UIINTR` (`STUI`, `CLUI`, `TESTUI` and `UIRET`) are more efficient than their `OSS` counterparts. Specifically, `STUI`, `CLUI` and `TESTUI` are up to 6× faster than the `sig_procmask()` syscall. The `UIRET` instruction is approximately 7× faster than its counterpart `sys_sigreturn()`.

## 4.4 `UIINTR` and `OSS` operations under contention

We now evaluate the performance of `UIINTR` and `OSS` operations under contention, on the sender or receiver side. Sender-side contention corresponds to a situation where several threads are concurrently sending notifications to the same recipient thread. This may occur, e.g., in application/runtime designs in which several dispatcher threads can notify the same worker thread or several worker threads can send responses/acknowledgments to the same dispatcher thread. Conversely, receiver-side contention corresponds to a situation where the recipient thread is competing with other threads for CPU time allocation (i.e., when the recipient thread is not necessarily running alone on a dedicated CPU hardware context). This may occur in the case of application/runtime designs requiring a large number of kernel-level threads, e.g., I/O-intensive applications based on blocking I/O primitives.

*Sender-side contention.* In this experiment, we increase the total number of concurrent sender threads towards the same receiver (each pinned on a distinct CPU hardware thread), between 1 and 14. Figure 4a presents the execution time of `SENDUIPI` as a function of the number of senders for the different execution modes. According to the Intel documentation, the workflow of this instruction involves acquiring a hardware lock on the `UPID`. For `NATIVE` and `GIPIV`, the execution time increases moderately with the number of senders (with a maximum gap of 300 cycles). In the case of `GIPI`, the increase is more sharp for the 99th percentile (up to 1300 cycles).

In contrast, when using `OSS`, the 99th-percentile latency of `pthread_kill()` calls is heavily degraded by concurrent sending operations towards the same receiver, for all modes (`NATIVE`, `GIPIV`, and `GIPI`), and even for a limited contention level (10× degradation from 1 to 2 senders, e.g., a gap of almost 50k cycles for `NATIVE`), as shown in Figure 4b. We believe that this behavior may be caused by synchronization and/or signal/IPI handling bottlenecks in the kernel code [30].

*Receiver-side contention.* As mentioned in §2.4, the delivery of a (`UIINTR` or `OSS`) request may be deferred until the next election of the receiver thread by the CPU scheduler. To study how this aspect impacts the delivery latency, we use a single sender and vary the number of additional threads competing with the receiver on the same CPU (hardware thread) between 0 and 100. In order to avoid (positive or negative) lockstep scheduling effects between the sender and the receiver, we add entropy by adding pauses of randomized durations on the sender side between successive requests. The chosen interval for the duration of the pauses (10–50 ms) takes into account the variable time slices of the Linux `CFS` scheduler and ensures that there are always multiple scheduling rounds occurring on the receiver CPU between successive request deliveries.

Figure 5 displays, for the `NATIVE` configuration, the evolution of the delivery latency (50th and 99th percentiles) for `OSS` and `UIINTR` according to the number of concurrent threads competing with the recipient thread. We make three main observations. (1) All the setups exhibit very similar measurements and trends (except for the special case with zero concurrent threads, which corresponds
<figure><img src="image_7.png" /><figcaption>(b) OSS sending operation (`pthread_kill()`) scalability.</figcaption></figure>

Fig. 4. `SENDUIPI` (up) and `pthread_kill()` (down) latencies according to the number of concurrent senders to the same recipient thread. It shows the 50th and 99th percentile for each execution mode (native and virtualized). Note that the y-axis scale is logarithmic for `pthread_kill()`.

<figure><img src="image_8.png" /></figure>

Fig. 5. Delivery latency with the receiver thread under contention. We show the 50th (left) and the 99th (right) percentiles for the NATIVE configuration with the (R,S2) and (R,S3) placements.

to the results already discussed in §4.3). (2) The latency improvements of `UINTR` over `OSS` are very low or even negligible (1% or less). (3) The delivery latency grows quite regularly with the number of concurrent threads, as the receiver thread spends more time waiting in the scheduler runqueue. For instance, in the case of (R,S2), the `UINTR` delivery latencies (99th percentile) are respectively 15.6 ms and 192.1 ms for 1 and 50 concurrent threads. We also make the same observations for the results of the `GIPIV` and `GIPI` configurations (not shown in the figure for space reasons). Overall, we conclude that, with the presence of one or more threads contending with the receiver, the delivery latencies of `OSS` and `UINTR` are practically similar, given the dominating impact of the Linux kernel scheduler's time slices.

**Take-away 7:** Overall, for sending operations, contention drastically increases the tail latency disadvantage of `OSS` compared to `UINTR`. However, in the case of receiver-side contention, `UINTR` and `OSS` achieve similar delivery latencies.
# 5 Christine: A Typical uThread Scheduler

For the purpose of this paper, we built Christine, a typical preemptible uThread scheduler (hereafter PUTS) whose main architectural traits can be seen in almost all µsecond-scale schedulers published in recent years [10, 20, 23]. We use Christine to evaluate to what extent UINTR can help reducing tail latencies on physical and virtual machines.

This section is structured as follows: §5.1 introduces the design and software architecture of Christine, §5.2 and §5.3 respectively present the evaluation metrics and methodology, and §5.4 discusses the evaluation results for native and virtualized setups.

## 5.1 Architecture

*Overview.* Christine comprises a set of *N* + 1 kernel-managed threads (hereafter “kThreads”): *N* worker threads, in charge of running user-level threads (hereafter “uThreads”), and a timer thread. Each of these kthreads is pinned to a dedicated CPU core. Listing 2 presents the pseudo-code of, respectively, the loop executed by the timer thread and the local scheduling routine executed by each worker. The timer thread periodically sends notifications to all the workers (the broadcast phase, line 10 in the pseudo-code), corresponding to the end of a quantum. Upon receiving a notification, each worker executes a handler implementing the local scheduler code, which selects the next elected uThread and swaps contexts. For simplicity, and given our focus on characterizing notification mechanisms, our prototype uses a constant quantum duration and a Round-Robin selection policy.

*Timer Thread.* It takes as input a period $Q_{target}$ (line 2 in Listing 2), which corresponds to the desired duration between two broadcast phases, i.e., the time interval between two invocations of the local scheduler on each worker CPU. It is important to ensure that the effective period closely matches the value set as a target ($Q_{target}$). Approaches based on system calls (e.g., using `sleep()` or alarms) introduce significant overheads for µsecond-scale scheduling; instead, we implement a timer based on `rdtsc()` and busy spinning (line 26 in Listing 2), similar to LibPreemptible [23].

*Synchronization Phase.* After preliminary tests, we observed that below a certain $Q_{target}$, the use of OSS as a notification mechanism hinders the progress of the uThreads by the workers. Indeed, the frequency of notifications is such that the workers spend their time executing the scheduling routine. To remedy this, we introduced a synchronization phase to ensure that notifications do not inhibit the workers and allow the uThreads to progress. In the synchronization phase, the Timer Thread waits for acknowledgments (lines 14-23 in Listing 2) from all the workers, via shared memory variables (line 38), before triggering a new broadcast phase.

*User-level Threads.* A User-level Thread, or uThread, is a software entity corresponding to an execution context. It is described by a function pointer, a stack, and a signal mask. To perform fast context switches between uThreads, the workers use `fast_swapcontext()`, an optimized implementation of LibC’s `swapcontext()`, partially inspired from the one of Shinjuku [20]. In particular, to improve performance, we remove the system call performed upon each context switch between uThreads in order to set the signal mask. This is possible under the assumption that all the tasks (uThreads) within the same process can share the same signal mask. In addition, given our focus on characterizing basic overheads, we only consider applications that do not use extended registers (e.g., SIMD registers) and these registers are not saved/restored upon a switch between uThreads.

*Christine Implementation.* We implement two main versions of Christine based on distinct notification mechanisms, i.e., using either OSS (ChristineOSS) or UINTR (ChristineUINTR), both
described below. For each version, we consider two variants: the first leverages the above-mentioned synchronization via worker acknowledgments, whereas the second has no synchronization. Overall, this leads us to four different implementations, hereafter noted *ChristineOSS-Sync*, *ChristineOSS-NoSync*, *ChristineUINTR-Sync* and *ChristineUINTR-NoSync*.

<figure><img src="image_9.png" /><figcaption>Listing 2. Main loop executed by the Timer thread and worker threads in Christine.</figcaption></figure>

## 5.2 Metrics

This section presents the three metrics that we have defined for the precise characterization of a fine-grained PUTS like Christine: *Minimum Latency*, *Efficiency* and *Precision*. Consider a CPU-intensive uThread that performs no blocking actions. Let $Q_{target}$ be the target period (set in the application configuration) for scheduling handler invocations on the worker CPUs. Let $Q_{real}^{i,w}$ be the effective period of scheduling handler invocation at iteration $i$ on worker CPU $w$.

*Efficiency*. This metric quantifies the overhead of the scheduler, meaning, for a given task, the performance degradation caused by periodic preemptions of the uThread. It represents the percentage of time spent in the scheduler (notification delivery and scheduling routine) relative to
the total time required to execute the application without preemptions, referred to as the *baseline*. We run the application *APP<sub>loop</sub>* to evaluate this metric (see details in §5.3).

*Minimum Latency.* The latency $L_p$ is the $p^{th}$ percentile of $Q_{real}^{i,w}$ (for all iterations $i$ and workers $w$) when $Q_{target}$. This metric allows quantifying the finest possible responsiveness of the scheduler. This property is important for applications with interdependent tasks, such as non-blocking producer/consumer schemes. Indeed, the producer cannot progress until the consumer has consumed, and vice versa. Therefore, the scheduler must be invoked very frequently to reduce waiting time. We run the applications *APP<sub>empty</sub>* and *APP<sub>prod_cons</sub>* to evaluate this metric (see §5.3).

*Precision.* The precision of a scheduler is its ability to guarantee $Q_{target}$ in a predictable manner. It enables the scheduler to ensure the Quality of Service (QoS) of the application. A scheduler is precise if the actual time between two invocations of the routine $Q_{real}$ is both *stable* and *focused*.

Stability is expressed by the standard deviation of $Q_{real}^{i,w}$ for all workers $w$. A scheduler is stable if it tends towards 0. This metric is important as it allows quantifying performance stability, which is a major requirement for many cloud applications.

Focus corresponds to the difference between $Q_{target}$ and the mean of $Q_{real}^{i,w}$ across all workers. A scheduler is correctly focused if it is sufficiently close to 0.

We run the application *APP<sub>empty</sub>* to evaluate this Stability and Focus (see in §5.3).

## 5.3 Evaluation Methodology

We evaluate and compare the four different variants of Christine (described in §5.1) w.r.t. the metrics introduced in §5.2 in native and virtualized mode. Our guest virtual machine has 16 vCPUs pinned on each logical core of the physical machine, 8GB RAM and Intel PIv enabled (i.e., like the GIPV setups in §4). It runs the same Linux OS as the host system.

Running concurrent activities on sibling hyperthreads (i.e., hardware threads within the same physical CPU core) is prone to interferences resulting in performance issues [10] and/or security risks [11, 29]. For these reasons, many cloud/server runtimes only deploy one worker thread per physical core [18]. Accordingly (and also to facilitate the analysis of the experimental results), for all the studied configurations of Christine, we deploy only one worker thread per physical core, resulting in a maximum limit of 7 worker threads (the 8th core is dedicated to the Timer thread).

For the purpose of our evaluation, we have developed three applications: *APP<sub>prod_cons</sub>*, *APP<sub>loop</sub>* and *APP<sub>empty</sub>*. *APP<sub>prod_cons</sub>* is based on producer-consumer pairs, where each uThread is either a producer or a consumer. Within each pair, the producer and consumer communicate through a single-entry shared memory buffer and are executed by the same worker. Each pair completes after 50k iterations. In all experiments, we have one producer-consumer pair (i.e., 2 uThreads) per worker. *APP<sub>loop</sub>* is an application in which each uThread increments a local variable in a loop. The application stops when all variables associated with the uThreads reach the value $10^8$. There is one uThread per worker. In the *APP<sub>empty</sub>* application, the uThreads execute an infinite empty loop; there is one uThread per worker. In the experiments, we vary the number of workers (1 and 7) and the period $Q_{target}$ (ranging from 0 to 500k CPU cycles, which corresponds to around $170~\mu s$).

## 5.4 Observations

*Efficiency.* Figure 6a presents the efficiency of Christine with 1 and 7 workers as a function of $Q_{target}$ in native mode, while Figure 6b shows it in a virtualized system. The closer the efficiency of a scheduler is to 100%, the better it is.

As expected, we can see that regardless of the number of workers, in both native and virtualized modes, the efficiency of the scheduler increases with $Q_{target}$. By increasing the period $Q_{target}$, the
<figure><img src="image_10.png" /><figcaption>Fig. 6. Christine Scheduler efficiency according to $Q_{target}$ with 1 and 7 workers in native and virtualized setups running $APP_{loop}$.</figcaption></figure>

Timer thread interrupts the workers less frequently, resulting in less slowdown. In native mode, ChristineUIINTR and ChristineOSS respectively achieve efficiencies of 95% and 65% with 1 worker, and 75% and 55% with 7 workers when $Q_{target}$ = 500k cycles, which corresponds to 170 µs.

With 1 worker and $Q_{target}$ = 10k cycles (approximately 3.4 µs), ChristineOSS-Sync achieves ≈ 30% efficiency, while ChristineUIINTR-Sync and ChristineUIINTR-NoSync reach 60%, effectively doubling the performance. For the same quantum, with 7 workers, ChristineOSS-Sync and ChristineOSS-NoSync achieve ≈ 35% efficiency, whereas ChristineUIINTR exceeds 50% (i.e., 1.4× better).

**Take-away 8:** For the efficiency metric, UINTR outperforms OSS by up to 2× in native mode when the targeted quantum duration is short. In virtualized mode, this gap is around 1.5×. When the target quantum duration is large, UINTR outperforms OSS by up to 1.5× in native and virtualized mode.

The efficiency decreases when increasing the number of workers. For instance, between 1 and 7 workers, ChristineUIINTR-Sync and ChristineUIINTR-NoSync show a difference of about 20% when $Q_{target}$ = 500k cycles in native mode and 15% in virtualized mode.

The efficiency of ChristineOSS-NoSync is null with 1 worker and when $Q_{target}$ < 30k cycles (≈ 10 µs). In this configuration, the timer inhibits the worker by sending signals too frequently. The worker spends all its time executing the scheduling routine and its uThreads cannot make progress.

This phenomenon does not occur for ChristineOSS-NoSync with 7 workers, nor for ChristineOSS-Sync with 1 and 7 workers. For the first configuration, the latency of the broadcast phase prevents the DoS. According to Table 1, the latency of pthread_kill() is 2k cycles (around 0.7µs) for the 50th percentile and 7.8k cycles (around 2.5 µs) for the 99th percentile. We can infer that, in most cases, the broadcast for 7 workers from the timer requires about 7 × 7.8 = 54.6k cycles, or 19 µs,
which is sufficient for the workers to execute their uThreads. ChristineOSS-Sync incorporates a synchronization phase between the timer and the workers. This phase allows the timer to calibrate the notification frequency with the workers’ delivery frequency, thereby enabling each worker to progress the execution of its uThreads. Since ChristineOSS-NoSync is not functional at $Q_{target} < 30k$ cycles ($\approx 10 \mu s$) with one worker, we do not discuss this configuration in the remainder of the paper.

We also observe the same phenomenon in virtualized mode for ChristineOSS-NoSync with 1 worker when $Q_{target} < 10k$ cycles ($\approx 3.4 \mu s$).

**Take-away 9:** For a scheduler using OS signals without a synchronization phase (see §5.1), a quantum duration below 30k cycles ($\approx 10 \mu s$) leads to a kind of DoS with a small number of workers. If the targeted quantum duration is very small, the synchronization is required. The threshold in virtualized mode for the quantum duration is 3× lower than in native mode.

The efficiency of the scheduler is impacted by virtualization. In native mode, with 1 worker, ChristineUINTR-Sync and NoSync reach around 65% of efficiency at $Q_{target} = 10k$ cycles (3.4 µs). In virtualized mode, both versions achieve almost 45%, which corresponds to a 32% efficiency drop for the same quantum. With 7 workers, the efficiency drop is up to 29% between native and virtualized modes.

**Take-away 10:** Virtualization reduces the efficiency metric of ChristineUINTR for small quantum durations. The gap between the native and the virtualized modes is up to 32%.

*Minimum Latency.* Table 4 shows only the latencies $L_{99}$ for all the implementations of Christine. We measured similar trends for other percentiles. The Minimum Latency depends on the notification delivery speed of the workers, which, in turn, depends partly on the notification speed of the timer. According to Table 1, UINTR delivery is up to 6× faster than OSS and the SENDUIPI instruction is up to 20× faster than pthread_kill(). According to Table 3, UIRET instruction is 8× faster than sys_sigreturn(). With UINTR, the Timer thread can therefore notify the workers more frequently and the workers can also deliver notifications more frequently. The results presented in Table 4 can be summarized as follows.

We observe that ChristineUINTR has better Minimum Latency than ChristineOSS, regardless of the number of workers. In native mode and with 1 worker, ChristineUINTR-NoSync and ChristineUINTR-Sync are approximately 5× and 1.7× better than ChristineOSS-Sync, respectively, achieving 1.3k cycles (0.4 µs) and 3.7k cycles (1.2 µs), compared to 6.6k cycles (2.2 µs) for ChristineOSS-Sync. With 7 workers, they are about 7.4× and 5× better, achieving 4.1k cycles (1.4 µs) and 6.1k cycles (2.1 µs), compared to 30.4k cycles (10.3 µs) for ChristineOSS-Sync. In virtualized mode, ChristineUINTR-NoSync and ChristineUINTR-Sync are approximately 3.1× and 2.4× better than ChristineOSS-Sync, with 1 worker, respectively, achieving 2.7k cycles (0.9 µs) and 3.5k cycles (1.2 µs), compared to 8.6k cycles (2.9 µs) for ChristineOSS-Sync. With 7 workers, they are resp. about 7.1× and 4.5× better, with 5.9k cycles (2 µs) and 9.2k cycles (3.1 µs), compared to 42.3k cycles (14.3 µs) for ChristineOSS-Sync.

**Take-away 11:** Considering the $L_{99}$ Minimum Latency metric, UINTR outperforms OSS by 1.7× to 7× in both native and virtualized modes. With 1 worker, UINTR allows achieving an $L_{99}$ Minimum Latency in the order of 1 µs, or even slightly below in the case of the UINTR-NoSync variant. With 7 workers, it allows reaching 1.3–3.1 µs for the same metric.

We also note that the synchronization phase degrades latency. Indeed, this phase delays the Timer Thread, impacting its notification frequency, which consequently also affects the workers’
delivery frequency. With 1 and 7 workers, ChristineUINTR-NoSync is approximately 3× and 1.4× better than ChristineUINTR-Sync in native mode. In virtualized mode, with 1 and 7 workers, ChristineUINTR-NoSync is resp. 1.3× and 1.5× better than ChristineUINTR-Sync.

<table><caption>Table 4. Christine $L_{99}$ Minimum Latency in cycles using APP<sub>empty</sub> application in Native and Guest mode with 1 and 7 workers. ChristineUINTR-NoSync achieves the best latencies (bold values).</caption><thead><tr><th colspan="5">Native (k cycles)</th></tr><tr><th>Christine</th><th>OSS-Sync</th><th>OSS-NoSync</th><th>UINTR-Sync</th><th>UINTR-NoSync</th></tr></thead><tbody><tr><th>1 worker</th><td>6.6</td><td>7.6</td><td>3.7</td><td>1.3</td></tr><tr><th>7 workers</th><td>30.4</td><td>28.1</td><td>6.1</td><td>4.1</td></tr></tbody><thead><tr><th colspan="5">Guest (k cycles)</th></tr><tr><th>Christine</th><th>OSS-Sync</th><th>OSS-NoSync</th><th>UINTR-Sync</th><th>UINTR-NoSync</th></tr></thead><tbody><tr><th>1 worker</th><td>8.6</td><td>8.4</td><td>3.5</td><td>2.7</td></tr><tr><th>7 workers</th><td>42.3</td><td>39.8</td><td>9.2</td><td>5.9</td></tr></tbody></table>

There is a noticeable degradation in the performance of Christine in virtualized mode. In virtualized mode, ChristineUINTR-Sync and ChristineUINTR-NoSync are both around 1.5× less performant than in native mode.

**Take-away 12:** Synchronization increases the Minimum Latency of ChristineUINTR by up to 3× in native mode and up to 1.5× in virtualized mode. Besides, virtualization increases the Minimum Latency of ChristineUINTR by up to 1.5×.

We also measured the impact of the Minimum Latency metric on the execution time of the application APP<sub>prod_cons</sub>, which benefits from more frequent scheduler interventions (as discussed in §5.2). Table 5 shows the minimum execution time of the application for each variant of Christine.<sup>6</sup> ChristineUINTR-NoSync achieves the best application execution time in all configurations. Indeed, it is up to 1.7× and 7.5× faster than ChristineUINTR-Sync and ChristineOSS-Sync in native mode. In virtualized mode, we observe the same trends. ChristineUINTR-NoSync is up to 1.3× and 6.7× faster than ChristineUINTR-Sync and ChristineOSS-Sync.

**Take-away 13:** For the APP<sub>prod_cons</sub> application, the total execution time is strongly impacted by the finest achievable quantum granularity of the scheduler. The performance gaps between the different scheduler versions are directly correlated with their respective $L_{99}$ Minimum Latencies. The Christine UINTR-based versions outperform the OSS versions by 1.3× to 7.5×.

**Precision.** Recall that Precision is composed of two sub-metrics: focus and stability. Figure 7 presents the focus of the different implementations of Christine with 1 and 7 workers in both

<table><caption>Table 5. Christine minimum reachable execution time of APP<sub>prod_cons</sub> for each version with 1 and 7 workers. ChristineUINTR-NoSync allows the best execution time in all configurations.</caption><thead><tr><th colspan="5">Native (M cycles)</th></tr><tr><th>Christine</th><th>OSS-Sync</th><th>OSS-NoSync</th><th>UINTR-Sync</th><th>UINTR-NoSync</th></tr></thead><tbody><tr><th>1 worker</th><td>587</td><td>2,914</td><td>171</td><td>97</td></tr><tr><th>7 workers</th><td>2,859</td><td>2,364</td><td>534</td><td>378</td></tr></tbody><thead><tr><th colspan="5">Guest (M cycles)</th></tr><tr><th>Christine</th><th>OSS-Sync</th><th>OSS-NoSync</th><th>UINTR-Sync</th><th>UINTR-NoSync</th></tr></thead><tbody><tr><th>1 worker</th><td>820</td><td>777</td><td>321</td><td>238</td></tr><tr><th>7 workers</th><td>3,834</td><td>3,512</td><td>833</td><td>564</td></tr></tbody></table>

<footnote n="6"><sup>6</sup> In this experiment, we set $Q_{target}$ to zero for all the Christine variants except for ChristineOSS-NoSync, for which we use the shortest viable quantum (resp. 30k and 10k cycles in native and guest modes, as discussed above).</footnote>
<figure><img src="image_11.png" /><figcaption>Fig. 7. Christine scheduler focus according to $Q_{target}$ with 1 and 7 workers in native and virtualized setups. Beyond 10k cycles with 1 worker and 40 cycles with 7 workers, the focus of all versions is constant near zero (≈150 cycles).</figcaption></figure>

native and virtualized modes. We notice that, for each version of Christine, there is a value $focus_{min}$ of $Q_{target}$ as it approaches 0, from which the curve begins to deviate: $focus_{min}$ corresponds to the smallest value of $Q_{target}$ for which Christine's focus remains relatively close to zero.

When $Q_{target} > focus_{min}$, the focus of Christine remains constant at around 150 cycles with 1 and 7 workers. UINTR maintains better focus for $Q_{target}$ values near zero. In native mode with 1 worker, ChristineUINTR-NoSync reaches its $focus_{min}$ at $Q_{target} = 1k$ cycles (0.3 µs), while ChristineUINTR-Sync is 2x higher at $Q_{target} = 2k$ cycles (0.6 µs). ChristineOSS-Sync reaches its limit at $Q_{target} = 6k$ cycles (2 µs), which is 6x greater than ChristineUINTR-NoSync. With 7 workers, ChristineUINTR-NoSync reaches its $focus_{min}$ at $Q_{target} = 4k$ cycles (1.4 µs), while ChristineUINTR-Sync reaches it at $Q_{target} = 6k$ cycles, 50% higher. ChristineOSS-NoSync and ChristineOSS-Sync reach their $focus_{min}$ at $Q_{target} = 28k$ cycles (9.6 µs) and $Q_{target} = 30k$ cycles, which is more than 7x higher than ChristineUINTR-NoSync.

**Take-away 14:** In native mode, ChristineUINTR with no synchronization achieves the lowest $Q_{target}$ (≈ 0.3 µs) while generally achieving the best focus compared to other configurations.

In virtualized mode, the curves exhibit the same trend as in native mode. However, we observe a performance loss illustrated by a rightward shift in the curves. With 1 worker, both ChristineUINTR-NoSync and ChristineUINTR-Sync reach their $focus_{min}$ at 3k cycles, corresponding to a 3x and 1.5x loss compared to their native mode versions, respectively. With 7 workers, ChristineUINTR-NoSync and ChristineUINTR-Sync reach their $focus_{min}$ at 5k and 8k cycles, approximately 2 µs and 3 µs, which represents a 1.25x and 1.33x loss compared to their native mode versions.
<figure><img src="image_12.png" /><figcaption>Fig. 8. Christine scheduler stability according to $Q_{target}$ with 1 and 7 workers in native and virtualized setups. Beyond 100k cycles, the stability of all Christine versions is constant. ChristineUINTR-Sync has the best stability, slightly better than ChristineUINTR-NoSync.</figcaption></figure>

**Take-away 15:** Virtualization degrades the focus of ChristineUINTR by up to 3× with 1 worker and up to 1.33× with 7 workers compared to native mode.

We also observe an anomaly with 1 worker for the implementations without synchronization. When $Q_{target} = 2k$ cycles (0.68 µs), the focus of ChristineUINTR-NoSync reaches 2.2k cycles, and when $Q_{target}$ is near focus$_{min}$, the timer emission frequency and the worker's reception capacity become very close. We therefore assume that the proximity of these two values disrupts the worker's reception frequency, degrading the scheduler's focus.

Figure 8 shows the stability of the different versions of Christine with 1 and 7 workers in native mode (8a) and virtualized mode (8b). UINTR provides better stability than OSS in all configurations. In native mode with 1 worker, ChristineUINTR-Sync and ChristineUINTR-NoSync have a stability of around 60-100 cycles. For ChristineOSS-Sync, it reaches about 480 cycles when $Q_{target} \ge 40k$ cycles (13.7 µs), which is almost 8× higher. With 7 workers, ChristineUINTR-Sync and ChristineUINTR-NoSync are up to 6× better.

In virtualized mode, the trends of the Christine variants are similar to those in native mode. We observe that the scheduler stability is not affected by virtualization, except in the case of OSS with 7 workers. In native mode with 7 workers, when $Q_{target} > 30k$ cycles, the stability of ChristineOSS-Sync reaches around 600 cycles and fluctuates between 600 and 800 cycles. In virtualized mode, it reaches about 1k cycles and fluctuates between 1k and 1.2k cycles when $Q_{target} > 38k$ cycles, which is up to 2× more than native mode.
**Take-away 16:** In native mode, ChristineUINTR allows up to 8× better stability than OSS. Virtualization has little impact on scheduler stability, except for ChristineOSS with 7 workers (degradation up to 2×).

Finally, with 1 worker, in both native and virtualized modes, ChristineUINTR-Sync is slightly more stable than ChristineUINTR-NoSync. In native mode, ChristineUINTR-Sync is up to 5.3× more stable than ChristineUINTR-NoSync when $Q_{target} = 1k$ cycles (0.3 µs). In virtualized mode, ChristineUINTR-Sync is up to 3× more stable than ChristineUINTR-NoSync when $Q_{target} = 8k$ cycles (2.7 µs).

**Take-away 17:** Synchronization improves ChristineUINTR stability by up to 5.3× with 1 worker in native mode and up to 3× in virtualized mode.

## 5.5 Discussion

We now briefly discuss how the results regarding the microbenchmarks and the characterization of Christine (presented in the previous subsections) might impact the behavior of more realistic and complex workload patterns, by considering the case-study of an in-memory key-value store (KVS) [20, 23, 25]. In this context, one of the typical scenarios involves a read-only, bimodal workload mix consisting of GET and SCAN requests, either with a large majority of GET requests (e.g., 95%) or with a more homogeneous distribution (50% of each type). The GET requests generally require a very short execution time (e.g., 0.5 µs to 10 µs), whereas the SCAN requests take much longer (one or several hundreds of microseconds, possibly above 500 µs). In order to meet the stringent tail-latency requirements of such GET requests, it is necessary to mitigate the impact of head-of-line blocking (i.e., avoiding long waiting times behind SCAN requests) via fine-grained preemption, based on a scheduling quantum whose length is quite close to (slightly greater than) the execution time of a GET request, i.e., based on the above-mentioned values, a quantum length in the range of 1 µs to 20 µs.

On our machine, based on the microbenchmarks (delivery and handler return operations in the case of cross-core interactions), we see that, in native mode, the basic cost to handle a preemption notification is above 2 µs when using OSS and below 0.5 µs when using UINTR (i.e., a difference greater than 4×). This translates into significant usability, performance and predictability improvements in the context of the fine-grained preemptions that we consider here.

As we have seen in our study, the trends of the “efficiency”, “minimum latency” and “precision” metrics show that the ChristineOSS versions suffer from prohibitive overheads and/or unstability for quantum sizes of 10 µs (30k cycles on our machine) or less, while the ChristineUINTR versions can achieve microsecond-scale (single-digit) quantum lengths with more acceptable efficiency and reliability. Regarding slightly greater quantum lengths, i.e., in the 10–20 µs range (30k-60k cycles on our machine), the ChristineOSS versions become usable in practice but remain significantly less efficient than their UINTR counterparts. Using OSS, a greater fraction of the CPU time is spent in the preemption/notification handler code (as opposed to the application code). As an illustration, in the case of a 20 µs quantum, a worker thread of the ChristineOSS version spends more than 10% of the quantum length in OSS notification overheads. In contrast, between two preemptions, the worker threads in the ChristineUINTR versions have more available time to process short GET requests as well as best-effort SCAN requests, which may yield improved performance for throughput and tail latencies.

In the case of applications running in a guest virtual machine, UINTR and OSS operations suffer from extra overheads. The relative performance degradation (with respect to native mode) is more important for UINTR, but the latter still significantly outperforms OSS. Overall, the general trends
discussed above regarding ChristineOSS versus ChristineUINTR in native mode are also applicable
to IPIV-assisted guest mode (GIPIV).

6 Related Work

Since its release, research studies utilizing UINTR have mainly focused on improvements for microsecond-scale user-level runtimes, namely LibPreemptible [23] and Junction [9]. Both systems leverage UINTR for quick preemption of uThreads but, because they focus on complete systems (respectively a threading library and a kernel-bypass engine), they do not provide an in-depth characterization of UINTR independently from other salient features of their designs (e.g., adaptive scheduling policies and NIC-level hardware assists). Vessel [24] combines Intel's UINTR and MPK mechanisms to build a userspace process abstraction, which achieves faster core reallocation (w.r.t. usual OS processes) for latency-critical applications that are colocated with best-effort applications. The Skyloft framework [19] simplifies the implementation of efficient, custom scheduling policies for Linux implemented in user space, and leverages UINTR for µs-scale preemptions. PreemptDB [13] is a new database engine that introduces preemptive scheduling, assisted via UINTR, to enable efficient execution of high-priority transactions.

The authors of xUI [2] study the microarchitectural details of UINTR via reverse engineering, and, to improve the performance of this processor feature, they propose several hardware extensions, which are prototyped and evaluated using a simulator.

Another line of work is in the field of security, where Rauscher and Gruss characterized IPIv
and UINTR as potential vectors for side channel attacks [28] and implemented several examples of
attacks or covert channels based on cross-core interrupt detection.

Our work is complementary, as it provides a detailed performance study of UINTR versus
OS signals on different setups (native and virtualized, diverse thread placements), considering
elementary operations as well as µs-scale user-level scheduling with a large range of quantum sizes
(0–150 µs). Besides, given the importance of precise timings and variability for the implementation
of side channel attacks and mitigations, we believe that the detailed measurements of our study
may also be insightful for security researchers.

7 Conclusion

In this paper, we have provided a detailed characterization of the recent Intel User Interrupt (UINTR) facility, using microbenchmarks as well as the case study of a μ-scale user-level thread scheduler. Our results show that, for inter-thread communications UINTR brings significant performance and predictability gains over OS signals, for native setups and also guest virtual machines. Our UINTR-based uThread scheduler supports quantum sizes in the range of 1–5 μs.

Besides, we notice that some intrinsic design aspects of UINTR can increase the vulnerability of a receiver thread to some types of security and safety risks (denial of service, stack smashing – described in more details in Appendix E). Our future work will aim at mitigating these issues.

Acknowledgments

We thank our shepherd, Derek Eager, and the anonymous reviewers for their insightful comments. This work was supported by the French *Agence Nationale de la Recherche* under the ANR projects "PicNic" (ANR-20-CE25-0013) and "PEPR DiVa" (ANR-23-PECL-0004), by Inria "Défi OS", and by the PAI 2021 project "Fault Tolerance for Disaggregated Rack-Scale Computing" from the French Auvergne-Rhône-Alpes Region.
## References

[1] Mohammad Alian, Yifan Yuan, Jie Zhang, Ren Wang, Myungsoo Jung, and Nam Sung Kim. 2020. Data Direct I/O Characterization for Future I/O System Exploration. In *2020 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS)*. 160–169. https://doi.org/10.1109/ISPASS48437.2020.00031

[2] Berk Aydogmus, Linsong Guo, Danial Zuberi, Tal Garfinkel, Dean Tullsen, Amy Ousterhout, and Kazem Taram. 2025. Extended User Interrupts (xUI): Fast and Flexible Notification without Polling. In *Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS '25)*. Association for Computing Machinery, New York, NY, USA.

[3] Luiz André Barroso, Mike Marty, David Patterson, and Parthasarathy Ranganathan. 2017. Attack of the killer microseconds. *Commun. ACM* 60(4) (2017), 48–54.

[4] Stella Bitchebe, Djob Mvondo, Laurent Réveillère, Noël de Palma, and Alain Tchana. 2021. Extending Intel PML for hardware-assisted working set size estimation of VMs. In *Proceedings of the 17th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments (VEE 2021)*. Association for Computing Machinery, New York, NY, USA, 111–124. https://doi.org/10.1145/3453933.3454018

[5] Sol Boucher, Anuj Kalia, David G. Andersen, and Michael Kaminsky. 2018. Putting the "Micro" Back in Microservice. In *Proceedings of the 2018 USENIX Conference on Usenix Annual Technical Conference (USENIX ATC '18)*. USENIX Association, USA, 645–650. https://www.usenix.org/conference/atc18/presentation/boucher

[6] Jiahao Chen, Dingji Li, Zeyu Mi, Yuxuan Liu, Binyu Zang, Haibing Guan, and Haibo Chen. 2023. Security and Performance in the Delegated User-level Virtualization. In *17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23)*. USENIX Association, Boston, MA, 209–226. https://www.usenix.org/conference/osdi23/presentation/chen

[7] Jonathan Corbet. 2021. User-Space Interrupts. Linux Weekly News. (2021). https://lwn.net/Articles/871113/

[8] Tu Dinh Ngoc, Bao Bui, Stella Bitchebe, Alain Tchana, Valerio Schiavoni, Pascal Felber, and Daniel Hagimont. 2019. Everything You Should Know About Intel SGX Performance on Virtualized Systems. *Proc. ACM Meas. Anal. Comput. Syst.* 3, 1, Article 5 (mar 2019), 21 pages. https://doi.org/10.1145/3322205.3311076

[9] Joshua Fried, Gohar Irfan Chaudhry, Enrique Saurez, Esha Choukse, Inigo Goiri, Sameh Elnikety, Rodrigo Fonseca, and Adam Belay. 2024. Making Kernel Bypass Practical for the Cloud with Junction. In *21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24)*. USENIX Association, Santa Clara, CA, 55–73. https://www.usenix.org/conference/nsdi24/presentation/fried

[10] Joshua Fried, Zhenyuan Ruan, Amy Ousterhout, and Adam Belay. 2020. Caladan: Mitigating Interference at Microsecond Timescales. In *14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20)*. USENIX Association, 281–297. https://www.usenix.org/conference/osdi20/presentation/fried

[11] Ben Gras, Kaveh Razavi, Herbert Bos, and Cristiano Giuffrida. 2018. Translation Leak-aside Buffer: Defeating Cache Side-channel Protections with TLB Attacks. In *27th USENIX Security Symposium (USENIX Security 18)*. USENIX Association, Baltimore, MD, 955–972. https://www.usenix.org/conference/usenixsecurity18/presentation/gras

[12] Juan Gómez-Luna, Izzat El Hajj, Ivan Fernandez, Christina Giannoula, Geraldo F. Oliveira, and Onur Mutlu. 2022. Benchmarking a New Paradigm: Experimental Analysis and Characterization of a Real Processing-in-Memory System. *IEEE Access* 10 (2022), 52565–52608. https://doi.org/10.1109/ACCESS.2022.3174101

[13] Kaisong Huang, Jiatang Zhou, Zhuoyue Zhao, Dong Xie, and Tianzheng Wang. 2025. Low-Latency Transaction Scheduling via Userspace Interrupts. In *Proceedings of the ACM SIGMOD/PODS International Conference on Management of Data (SIGMOD'25)*. Association for Computing Machinery, New York, NY, USA.

[14] Intel. [n. d.]. Linux patch of Intel user-space interrupts. https://github.com/intel/uintr-linux-kernel (Accessed: 2024-10-19). ([n. d.]).

[15] Intel. 2023. *Advanced Programmable Interrupt Controller (APIC)*. Vol. 3C. https://cdrdv2.intel.com/v1/dl/getContent/671200

[16] Intel. 2023. *APIC Virtualization and Virtual Interrupts*. Vol. 3C. https://cdrdv2.intel.com/v1/dl/getContent/671200

[17] Intel. 2023. *User Interrupts*. Vol. 3C. https://cdrdv2.intel.com/v1/dl/getContent/671200

[18] Rishabh Iyer, Musa Unal, Marios Kogias, and George Candea. 2023. Achieving Microsecond-Scale Tail Latency Efficiently with Approximate Optimal Scheduling. In *Proceedings of the 29th Symposium on Operating Systems Principles (SOSP '23)*. Association for Computing Machinery, New York, NY, USA, 466–481. https://doi.org/10.1145/3600006.3613136

[19] Yuekai Jia, Kaifu Tian, Yuyang You, Yu Chen, and Kang Chen. 2024. Skyloft: A General High-Efficient Scheduling Framework in User Space. In *Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles (SOSP '24)*. Association for Computing Machinery, New York, NY, USA, 265–279. https://doi.org/10.1145/3694715.3695973

[20] Kostis Kaffes, Timothy Chong, Jack Tigar Humphries, Adam Belay, David Mazières, and Christos Kozyrakis. 2019. Shinjuku: Preemptive Scheduling for µsecond-scale Tail Latency. In *16th USENIX Symposium on Networked Systems Design and Implementation (NSDI 19)*. USENIX Association, Boston, MA, 345–360. https://www.usenix.org/conference/nsdi19/presentation/kaffes
[21] Jacob Leverich and Christos Kozyrakis. 2014. Reconciling high server utilization and sub-millisecond quality-of-service. In *Proceedings of the Ninth European Conference on Computer Systems (EuroSys '14)*. Association for Computing Machinery, New York, NY, USA, Article 4, 14 pages. https://doi.org/10.1145/2592798.2592821

[22] Jialin Li, Naveen Kr. Sharma, Dan R. K. Ports, and Steven D. Gribble. 2014. Tales of the Tail: Hardware, OS, and Application-level Sources of Tail Latency. In *Proceedings of the ACM Symposium on Cloud Computing (SOCC '14)*. Association for Computing Machinery, New York, NY, USA, 1-14. https://doi.org/10.1145/2670979.2670988

[23] Yueying Li, Nikita Lazarev, David Koufaty, Tenny Yin, Andy Anderson, Zhiru Zhang, G. Edward Suh, Kostis Kaffes, and Christina Delimitrou. 2024. LibPreemptible: Enabling Fast, Adaptive, and Hardware-Assisted User-Space Scheduling. In *2024 IEEE International Symposium on High-Performance Computer Architecture (HPCA)*. 922-936. https://doi.org/10.1109/HPCA57654.2024.00075

[24] Jiazhen Lin, Youmin Chen, Shiwei Gao, and Youyou Lu. 2024. Fast Core Scheduling with Userspace Process Abstraction. In *Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles (SOSP '24)*. Association for Computing Machinery, New York, NY, USA, 280-295. https://doi.org/10.1145/3694715.3695976

[25] Zhihong Luo, Sam Son, Dev Bali, Emmanuel Amaro, Amy Ousterhout, Sylvia Ratnasamy, and Scott Shenker. 2024. Efficient Microsecond-scale Blind Scheduling with Tiny Quanta. In *Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (ASPLOS '24)*. Association for Computing Machinery, New York, NY, USA, 305-319. https://doi.org/10.1145/3620665.3640381

[26] Amy Ousterhout, Joshua Fried, Jonathan Behrens, Adam Belay, and Hari Balakrishnan. 2019. Shenango: Achieving High CPU Efficiency for Latency-sensitive Datacenter Workloads. In *16th USENIX Symposium on Networked Systems Design and Implementation (NSDI '19)*. USENIX Association, Boston, MA, 361-378. https://www.usenix.org/conference/nsdi19/presentation/ousterhout

[27] Onkar Patil, Latchesar Ionkov, Jason Lee, Frank Mueller, and Michael Lang. 2019. Performance characterization of a DRAM-NVM hybrid memory architecture for HPC applications using intel optane DC persistent memory modules. In *Proceedings of the International Symposium on Memory Systems (MEMSYS '19)*. Association for Computing Machinery, New York, NY, USA, 288-303. https://doi.org/10.1145/3357526.3357541

[28] Fabian Rauscher and Daniel Gruss. 2024. Cross-Core Interrupt Detection: Exploiting User and Virtualized IPIs. In *ACM Conference on Computer and Communications Security (CCS) 2024*. https://doi.org/10.1145/3658644.3690242 ACM Conference on Computer and Communications Security : CCS 2024, CCS '24 ; Conference date: 14-10-2024 Through 18-10-2024.

[29] Michael Schwarz, Moritz Lipp, Daniel Moghimi, Jo Van Bulck, Julian Stecklina, Thomas Prescher, and Daniel Gruss. 2019. ZombieLoad: Cross-Privilege-Boundary Data Sampling. In *Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security (CCS '19)*. Association for Computing Machinery, New York, NY, USA, 753-768. https://doi.org/10.1145/3319535.3354252

[30] Shumpei Shiina, Shintaro Iwasaki, Kenjiro Taura, and Pavan Balaji. 2021. Lightweight preemptive user-level threads. In *Proceedings of the 26th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming (PPoPP '21)*. Association for Computing Machinery, New York, NY, USA, 374-388. https://doi.org/10.1145/3437801.3441610

[31] Yan Sun, Yifan Yuan, Zeduo Yu, Reese Kuper, Chihun Song, Jinghan Huang, Houxiang Ji, Siddharth Agarwal, Jiaqi Lou, Ipoom Jeong, Ren Wang, Jung Ho Ahn, Tianyin Xu, and Nam Sung Kim. 2023. Demystifying CXL Memory with Genuine CXL-Ready Systems and Devices. In *Proceedings of the 56th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO '23)*. Association for Computing Machinery, New York, NY, USA, 105-121. https://doi.org/10.1145/3613424.3614256

[32] Boris Teabe, Peterson Yuhala, Alain Tchana, Fabien Hermenier, Daniel Hagimont, and Gilles Muller. 2021. (No)Compromis: Paging Virtualization is Not a Fatality. In *Proceedings of the 17th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments (VEE 2021)*. Association for Computing Machinery, New York, NY, USA, 43-56. https://doi.org/10.1145/3453933.3454013

[33] Minhu Wang, Mingwei Xu, and Jianping Wu. 2022. Understanding I/O Direct Cache Access Performance for End Host Networking. *Proc. ACM Meas. Anal. Comput. Syst.*, Vol. 6, No. 2, Article 22 (feb 2022), 37 pages. https://doi.org/10.1145/3508042

[34] Xingda Wei, Rongxin Cheng, Yuhan Yang, Rong Chen, and Haibo Chen. 2023. Characterizing Off-path SmartNIC for Accelerating Distributed Systems. In *17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23)*. USENIX Association, Boston, MA, 987-1004. https://www.usenix.org/conference/osdi23/presentation/wei-smartnic

[35] Lingfeng Xiang, Xingsheng Zhao, Jia Rao, Song Jiang, and Hong Jiang. 2022. Characterizing the Performance of Intel Optane Persistent Memory: a Close Look at its on-DIMM Buffering. In *Proceedings of the Seventeenth European Conference on Computer Systems (EuroSys '22)*. Association for Computing Machinery, New York, NY, USA, 488-505. https://doi.org/10.1145/3492321.3519556
The contents of the appendices are the following:

* Appendix A provides additional background information on Intel User Interrupts (UINTR).

* Appendix B provides additional details about our experimental methodology.

* Appendix C provides additional microbenchmark results obtained on another machine.

* Appendix D provides additional results about the evaluation of the Christine Scheduler.

* Appendix E provides a qualitative comparison regarding the features of OSS and UINTR, a well as a discussion of potential security vulnerabilites of UINTR.

## A Additional Background on UINTR

This section complements §2 to provide additional details about the Intel UINTR technology. We first position UINTR with respect to other existing inter-processor communication mechanisms available on Intel CPUs. We then describe the introduced changes at the hardware level (§A.2) and at the software level (§A.3). Finally (in §A.4), we describe in details the communication steps between a sender and a receiver thread. The interplay between the different components is illustrated in Figure 9. In addition, Figure 10 shows the different steps (involving user code, kernel code, and hardware) for a notification between a sender thread and a receiver thread; the figure compares two setups based on different notification mechanisms: the first (left) one uses OSS while the second one (right) uses UINTR.

<figure><img src="image_13.png" /><figcaption>Fig. 9. UINTR notification *sending*, *delivery*, and *processing*.</figcaption></figure>

### A.1 Inter-Processor Communication

Inter-processor communication refers to hardware mechanisms used by software layers to allow a sender processor to trigger the execution of a routine on a receiver processor. Existing mechanisms on Intel machines for this purpose are Inter-Processor Interrupts (IPIs) [15], Posted-Interrupt [16], and UINTR [17]. All these mechanisms rely on the LAPIC integrated within each CPU. To prevent potentially malicious user-space applications from disrupting other processors and, by extension, neighbor applications (denial of service), Linux protects the LAPIC, making it exclusively accessible
in kernel mode. Thus, expensive mode switches are necessary whenever application code must
trigger an interrupt.

In virtualized mode, accesses to the LAPIC are emulated by the hypervisor through VM Exits.
To speed up the sending of notifications, Shinjuku [20] maps the physical LAPIC directly into the
guest's memory. This solution poses significant security risks, as it requires the guest to be trusted.

Intel introduced a set of mechanisms that enable LAPIC virtualization (APICv) and IPI virtualization (IPIv), to interrupt-related VM Exits (for both sender and receiver CPUs) in guest VMs. UINTR is the only hardware mechanism that allows complete avoidance of mode switches for IPIs (for senders and receivers) on both native and virtualized environments. This is achieved via the (kernel-mediated) pre-registration of sender-receiver pairs at thread-level granularity.

## A.2 Main Components and hardware changes

UINTR relies on two new data structures, *User Posted-Interrupt Descriptor (UPID)* and *User Interrupt Target Table (UITT)*, several new MSR registers to control its operation, as well as multiple new x86 instructions.

*Data Structures.* A UPID is a data structure that describes a receiver CPU. It must be used by the sending CPU to post a UINTR and potentially notify the receiver CPU. UPID comprises several fields: *Outstanding Notification (ON)* indicates whether the receiver CPU has a UINTR pending; *Suppressed Notification (SN)* indicates whether an IPI must be buffered or can be delivered immediately to the receiver (in particular, buffering is required when the target thread is not currently running); *User Interrupt Notification Vector (UINV)* indicates the interrupt vector used by the IPI to specify to the destination LAPIC that this IPI should be interpreted as a UINTR; *Notification Destination (NDST)* contains the identifier of the LAPIC corresponding to the receiver CPU; and *Posted-Interrupt Requests (PIR)* records the various UINTR requests issued (up to 64 request types).

A UITT is a table describing UPIDs that the sender CPU can use. Each entry of this table includes: a *Valid Bit (V)* determining the entry's validity; a *User-Interrupt Vector (UV)* indicating the request number at the destination of the receiver CPU; a *User Posted-Interrupt linear Address (UPIDADDR)* corresponding to the linear address of the UPID describing the destination CPU.

*MSR Registers.* UINTR adds six new MSR registers. The most important are: (on the receiver CPU) *UINTR_RR*, which stores the User Interrupt Requests Register (UIRR), indicating for each bit whether the associated UINTR has been serviced (ready to be processed by the routine); (on the receiver CPU) *UINTR_HANDLER*, which stores the address of the routine to be executed; (on the receiver CPU) *UINTR_MISC*, which contain the UINTR interrupt vector number and the size of the UITT; (on the receiver CPU) *UINTR_PD*, which stores the address of the UPID; (on the sender CPU) *UINTR_TT*, which points to the UITT and indicates that the CPU is enabled to execute the SENDUIPI instruction (bit 0).

*x86 ISA Extensions.* UINTR introduces five new instructions accessible in user mode: STUI, which determines UINTR flag (UIF); (on the sender CPU) SENDUIPI, which sends a UINTR; (on the receiver CPU) CLUI, which disables UINTR delivery; (on the receiver CPU) STUI, which enables UINTR delivery; and (on the receiver CPU) UIRET, which restores the state of the registers saved before executing the routine.

## A.3 Software modifications

On the software side, Intel provides patches for Linux and gcc that implement the required initialization and compilation passes. Below, we provide details about the main changes at the OS level (Linux) and at the hypervisor level (KVM).
*OS Changes.* UINTR requires modifications in the OS code for three main reasons: enabling UINTR, initializing UPID and UITT, and ensuring proper support when the receiver thread is running or not. The latter is the only modification that can affect the cost of UINTR delivery. We briefly present the main changes which concern the kernel's scheduling process as it may impact UINTR.

A receiver thread can only be notified (IPI) and process UINTR signals while it is running on a CPU. Therefore, whenever the kernel scheduler removes a receiver thread from the CPU, it sets the SN bit of its UPID (preventing sender threads from firing IPIs). Conversely, when the scheduler places a receiver thread on a CPU, it first updates the NDST field (LAPIC identifier) of its UPID with that of the chosen CPU. Then, it unsets its SN bit and generates a self-IPI to prompt the CPU to enter the UINTR detection, delivery, and processing phases outlined above.

*UINTR in Virtual Machines.* The UINTR documentation mentions that the newly introduced instructions can also be executed within a virtual machine (VM), specifically when the CPU operates in VMX non-root ring 3 mode. However, modifications to KVM were required to support UINTR functionality within a guest OS. We have provided a patch to address this issue.

Our primary changes to KVM involved enabling the guest OS to execute the necessary operations for UINTR initialization. This process requires executing the CPUID instruction, updating the CR4 register, and accessing specific MSR registers. Since these operations cause VM Exits, our work primarily involved extending KVM's VMExit handlers to accommodate UINTR-related exit reasons.

## A.4 Detailed Communication Workflow

This subsection complements §2.4 with additional details regarding the description of the communication steps between a sender and a receiver CPU using UINTR. Figure 9 showcases the main steps followed by the sender and the receiver CPUs.

*Sender CPU.* [0] It executes the `SENDUIPI` instruction, which consists of two steps: *UINTR posting* and *UINTR notification*. [1.a] UINTR posting involves registering the UINTR request in the receiver's UPID. This step sets the bit `ON` if `SN` and `ON` were unset. [1.b] UINTR notification involves sending an `IPI` to the destination CPU, using the `NDST` and `UINV` fields of the UPID. This step is only performed when both the `ON` and `SN` bits of the recipient's UPID are unset. One of these bits is set if the recipient thread is either scheduled out or processing a UINTR.

*Receiver CPU.* It traverses the following steps. ([2] *UINTR notification identification*) When a CPU receives an interrupt vector *V*, it considers it to be a UINTR when the following two conditions are met: the processor checks if UINTR is enabled and *V* equals the value of the UINV field stored in the MSR register `UINTR_MISC`. Otherwise, it is an interrupt whose handler is defined in the classic IDT and executed in kernel mode. ([3] *UINTR notification processing*) When the interrupt is of type `UINTR`, the CPU updates the UPID whose address is stored in the MSR register `UINTR_PD`. It disables the `ON` bit and copies the PIR vector into UIRR, then sets PIR to zero. As long as UIRR is non-zero, it means that the CPU still has UINTRs to process. It remains in the *UINTR notification processing*. ([4] *UINTR delivery*) When a UINTR is identified, the CPU executes the routine whose address is stored in its MSR register `UINTR_HANDLER`. Before that, it saves the current execution context, as well as the argument taken as input by the routine, onto the stack. This argument corresponds to the most significant active bit in UIRR. As long as there are active bits in UIRR, the handler is re-executed.
<figure><img src="image_14.png" /><figcaption>Fig. 10. Execution flow of a sender-receiver interaction with OSS (left) and Intel UINTR (right) on Linux. In this scenario, thread T₁ is acting as the sender and T₂ as a receiver. The green steps are performed by the user application, the red ones by the kernel and the blue ones by the hardware. We can see that, while both mechanisms roughly follow the same logical steps, UINTR bypasses the kernel and uses the hardware directly.</figcaption></figure>

## B Experimental Methodology

This section provides additional details regarding the methodology (initially presented in §3) used for our experimental characterization of OSS and UINTR. We first detail the code used to perform the fine-grained time measurements (§B.1), and then describe the different microbenchmarks that we have implemented (§B.2).

### B.1 Time Measurement Methodology

Listing 3 displays the code of the custom routine (mentioned in §3) that we have used for our fine-grained time measurements.

<caption>Listing 3. RDTSC custom implementation.</caption>

```c
/* Custom function to read Time Stamp Counter */

static inline void rdtsc(uint64_t *time)
{
    /* all previous instructions have executed and all previous loads and stores are globally
     * visible */
    __asm__ __volatile__("mfence\n" "lfence\n");

    /* Read Time-Stamp Counter function provided by LibC */
    *time = _rdtsc();

    /* all next instructions are executed after the previous _rdtsc() call */
    __asm__ __volatile__("lfence\n");
}
```
## B.2 Microbenchmarks: uintrBench

To obtain the performance characterization results presented in §4, we developed *intrBench*, a benchmark suite for measuring the end-to-end latency of UINTR and OSS operations. Unless indicated otherwise, each micro-benchmark uses two threads: a sender and a receiver.

*intr_post*. It evaluates UINTR posting time. We setup the UPID of the receiver thread to ensure no IPI triggering at each UINTR sending. We ensure that the bits ON and SN are set at any time.

*intrnotification*. It evaluates UINTR notification time. We setup the UPID of the receiver thread to ensure IPI triggering at each UINTR sending. We ensure that the bits ON and SN are unset at any time.

*intr_delivery*. It evaluates the UINTR delivery latency when the receiver thread is running and executing its main program.

*intr_uif*. It consists of a set of three benchmarks designed to evaluate the cost of UIF accesses. Each one performs a specific UIF operation in a loop: mask (CLUI), unmask (STUI), and read (TESTUI).

*OSS*. As a baseline, we have constructed equivalents of the above micro-benchmarks for OSS. Note that, in the case of OSS, the *post* and *notification* steps are regrouped in the same benchmark.

## C Additional Results for Micro-benchmarks

We also performed a subset of the micro-benchmarks presented in §4 (Table 1, sending and delivery latencies) on a second machine with different hardware characteristics (while keeping the same OS and Linux kernel versions.). The machine is a Supermicro server equipped with an Intel Xeon Bronze 3408U processor. It features 8 cores, each with a single hardware thread, running at 1.8 GHz. Each core includes a 32KB L1 instruction cache, a 48KB L1 data cache, a 2MB L2 cache, and a shared 22.5MB L3 cache across all cores. The system is equipped with 16GB of RAM. Note that, due to the absence of hyperthreading, the (*R*, *S2*) placement is not available on this machine. Besides, we focus on the two most efficient execution modes, i.e., *NATIVE* and *Guest with IPv6* (*GIPIV*).

The results are shown in Table 6. Overall, compared to the latencies measured on the first machine (in §4), we observe some differences regarding the measured values (in number of cycles) but fairly similar trends and orders of magnitude, especially for the (*R*, *S3*) placement (i.e., cross-core notifications), which is typically the most used in practical deployments, and thus the most critical one.

In the case of the (*R*, *S1*) placement, we observe, similarly to the results from the first machine, that OSS and UINTR have similar (and very high) delivery latencies, for both the *NATIVE* and *GIPIV* setups. We can notice, however, a significant gap between the two machines regarding the number of cycles corresponding to these latencies (i.e., approx. 23M cycles on the second machine versus 46M cycles on the first machine). We believe that this difference in number of cycles may be mainly explained by the combination of the two following reasons: (1) the two machines have significantly different CPU clock frequencies, and, (2) in the specific case of the (*R*, *S1*) placement, the main impacting factor regarding the (OSS or UINTR) delivery latency is the duration of the time slices allocated by the Linux kernel scheduler<sup>7</sup> (given that the sender and the receiver are competing for the same CPU hardware thread and the delivery only occurs upon the scheduling of the receiver).

<footnote n="7"><sup>7</sup> The durations of the time slices of the Linux scheduler are chosen based on high-level settings configured in milliseconds, and they are not strongly coupled with the precise CPU clock frequency of the machine.</footnote>
<table><caption>Table 6. Latency of sending and delivery operations for OSS and UINTR in different setups (native, virtualized with GIPV) on the second machine. Each cell contains the 50th and the 99th percentile in cycle separated by a dash.</caption><thead><tr><th colspan="4">NATIVE (cycles)</th></tr></thead><tbody><tr><th scope="row">Pinning Mode</th><td>R,S1</td><td>R,S3</td><td>Self</td></tr><tr><th scope="row">pthread_kill()</th><td>1764-1854</td><td>3550-4242</td><td>3920-4116</td></tr><tr><th scope="row">Signal Delivery</th><td>23M-23M</td><td>5820-6118</td><td>2928-3114</td></tr><tr><th scope="row">SENDUIPI Posting</th><td>214-218</td><td>214-218</td><td>248-252</td></tr><tr><th scope="row">SENDUIPI Posting + Notif.</th><td>264-276</td><td>416-442</td><td>250-838</td></tr><tr><th scope="row">UINTR Delivery</th><td>23M-23M</td><td>1202-1258</td><td>902-908</td></tr></tbody><thead><tr><th colspan="4">GIPIV (cycles)</th></tr></thead><tbody><tr><th scope="row">Pinning Mode</th><td>R,S1</td><td>R,S3</td><td>Self</td></tr><tr><th scope="row">pthread_kill()</th><td>2038-2134</td><td>3958-4636</td><td>4284-4514</td></tr><tr><th scope="row">Signal Delivery</th><td>23M-23M</td><td>8470-9682</td><td>3182-3438</td></tr><tr><th scope="row">SENDUIPI Posting</th><td>240-246</td><td>240-246</td><td>480-486</td></tr><tr><th scope="row">SENDUIPI Posting + Notif.</th><td>544-566</td><td>830-1042</td><td>480-486</td></tr><tr><th scope="row">UINTR Delivery</th><td>23M-23M</td><td>2896-2986</td><td>2290-2332</td></tr></tbody></table>

## D Christine Scheduler Minimum Latency

Table 7 displays the Minimum Latency measured for the Christine Scheduler using the Appempty application. The table shows the measured Minimum Latency for the 50th, 90th and 99th percentiles. Note that this is an extended version of Table 4 shown in §5.4, which (due to a lack of space) only displayed the results for the 99th percentile.

<table><caption>Table 7. Christine Minimum Latency in cycles using Appempty application in Native and Guest mode. ChristineUINTR-NoSync achieves the best latencies (bold values).</caption><thead><tr><th colspan="4">Native</th></tr></thead><tbody><tr><th scope="row">Latencies</th><td>L<sub>50</sub></td><td>L<sub>90</sub></td><td>L<sub>99</sub></td></tr><tr><th scope="row">1 worker</th><td></td><td></td><td></td></tr><tr><th scope="row">OSS-NoSync</th><td>4640</td><td>6012</td><td>7604</td></tr><tr><th scope="row">UINTR-NoSync</th><td><strong>968</strong></td><td><strong>1052</strong></td><td><strong>1290</strong></td></tr><tr><th scope="row">OSS-Sync</th><td>6098</td><td>6282</td><td>6576</td></tr><tr><th scope="row">UINTR-Sync</th><td>1896</td><td>3614</td><td>3732</td></tr><tr><th scope="row">7 workers</th><td></td><td></td><td></td></tr><tr><th scope="row">OSS-NoSync</th><td>25468</td><td>27140</td><td>28124</td></tr><tr><th scope="row">UINTR-NoSync</th><td><strong>3794</strong></td><td><strong>4008</strong></td><td><strong>4180</strong></td></tr><tr><th scope="row">OSS-Sync</th><td>28778</td><td>29728</td><td>30408</td></tr><tr><th scope="row">UINTR-Sync</th><td>5594</td><td>5874</td><td>6110</td></tr></tbody><thead><tr><th colspan="4">Guest</th></tr></thead><tbody><tr><th scope="row">Latencies</th><td>L<sub>50</sub></td><td>L<sub>90</sub></td><td>L<sub>99</sub></td></tr><tr><th scope="row">1 worker</th><td></td><td></td><td></td></tr><tr><th scope="row">OSS-NoSync</th><td>4968</td><td>7936</td><td>8398</td></tr><tr><th scope="row">UINTR-NoSync</th><td><strong>2316</strong></td><td><strong>2552</strong></td><td><strong>2688</strong></td></tr><tr><th scope="row">OSS-Sync</th><td>8134</td><td>8358</td><td>8754</td></tr><tr><th scope="row">UINTR-Sync</th><td>3236</td><td>3292</td><td>3542</td></tr><tr><th scope="row">7 workers</th><td></td><td></td><td></td></tr><tr><th scope="row">OSS-NoSync</th><td>35966</td><td>37848</td><td>39796</td></tr><tr><th scope="row">UINTR-NoSync</th><td><strong>5652</strong></td><td><strong>5808</strong></td><td><strong>5906</strong></td></tr><tr><th scope="row">OSS-Sync</th><td>38880</td><td>40648</td><td>42264</td></tr><tr><th scope="row">UINTR-Sync</th><td>8710</td><td>8954</td><td>9194</td></tr></tbody></table>

## E Qualitative Comparison

Table 8 summarizes the qualitative differences and similarities between OSS and UINTR. As their names are fairly self-explanatory, we do not comment on them.

In addition, we discuss below two potential security vulnerabilities which are more problematic for UINTR (compared to OSS) due to its specific design and the absence of kernel mediation on the critical path for the delivery of notifications.
<table><caption>Table 8. Qualitative comparison of OSS and UINTR.</caption><thead><tr><td>Criteria</td><td>OSS</td><td>UINTR</td></tr></thead><tbody><tr><td>Recipient</td><td>Specific or arbitrary thread (within target process)</td><td>Specific thread</td></tr><tr><td># User-defined notification types</td><td>2</td><td>64</td></tr><tr><td>Distinct handler per notification type</td><td>Yes</td><td>No (but in-handler demultiplexing based on notification type is possible)</td></tr><tr><td>Interruptible handler</td><td>Only for different notification types</td><td>No but can be enabled manually (<_stui)</td></tr><tr><td>Masking granularity</td><td>Per notification type</td><td>coarse-grained</td></tr><tr><td>Handling of pending notifications upon handler completion</td><td>All <em>unmasked</em> notifications are handled before resuming application code</td><td>All notifications are handled before resuming application code</td></tr><tr><td>Inheritable</td><td>Yes</td><td>No</td></tr><tr><td>Receiver can wait for notification</td><td>Yes (<em>sigwait()</em>)</td><td>Yes (<em>uintr_wait()</em>)</td></tr><tr><td>Notification delivery may wake up receiver from blocking syscalls</td><td>Yes</td><td>No</td></tr><tr><td>Kernel intervention</td><td>Yes</td><td>Only for initialization</td></tr><tr><td>Hardware intervention</td><td>Only when IPI required</td><td>UPID update (always) + IPI (when required)</td></tr></tbody></table>

*Denial of Service Attacks.* A malicious sender that continuously emits UINTR signals could prevent the receiver from ever executing its program properly. Additionally, other legitimate senders might be affected, especially through increased latency. This problem is more difficult to address in the case of UINTR because, unlike with OSS, the kernel does not mediate the delivery of notifications. To address this issue, it is necessary to integrate a mechanism for traffic shaping and control. One solution is the implementation of a *UPID trash-backup* mechanism. This solution involves modifying the *UPIDADDR* of the malicious sender's *UITT* entry to point to a *trash* UPID address (with *SN* = 1). This effectively masks emissions from that specific sender without affecting others.

*Stack Buffer Overflow Attacks.* Indeed, upon the delivery of a notification, the return address is saved in the stack set up for the UINTR handler. Because the saved return address is accessible from user mode, unlike with OSS, UINTR handlers which copy input data to local variables are vulnerable to stack smashing attacks, resulting in potential subversion of the control flow of the receiver thread. Therefore, special care is required for the implementation of UINTR handlers, in order to ensure the absence of unsafe memory copy operations.

Received January 2025; revised April 2025; accepted April 2025