Understanding Intel User Interrupts

Yves Kone, Louis Duval, Renaud Lachaize, Pascal Felber, Daniel Hagimont,
Alain Tchana

► To cite this version:

Yves Kone, Louis Duval, Renaud Lachaize, Pascal Felber, Daniel Hagimont, et al. Understanding Intel User Interrupts. SIGMETRICS 2025 - ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems, Jun 2025, Stony Brook, United States. pp.127-129, 10.1145/3726854.3727326 . hal-05156424

HAL Id: hal-05156424

https://hal.science/hal-05156424v1

Submitted on 10 Jul 2025

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Understanding Intel User Interrupts

**Yves Kone**

IRIT, Université de Toulouse, CNRS,
Toulouse INP, UT3 Toulouse
France
yves.kone@ens-lyon.fr

**Louis Duval**

Univ. Grenoble Alpes, CNRS, Inria,
Grenoble INP, LIG, 38000 Grenoble
France

**Renaud Lachaize**

Univ. Grenoble Alpes, CNRS, Inria,
Grenoble INP, LIG, 38000 Grenoble
France
renaud.lachaize@univ-grenoble-alpes.fr

**Pascal Felber**

University of Neuchâtel
Switzerland
pascal.felber@unine.ch

**Daniel Hagimont**

IRIT, Université de Toulouse, CNRS,
Toulouse INP, UT3 Toulouse
France
daniel.hagimont@irit.fr

**Alain Tchana**

Univ. Grenoble Alpes, CNRS, Inria,
Grenoble INP, LIG, 38000 Grenoble
France
alain.tchana@grenoble-inp.fr

## Abstract

User interrupts (UINTR) is a new hardware feature available in recent Intel Xeon processors. It enables a CPU to send, receive, and handle hardware inter-processor interrupts directly in user mode, meaning without the intervention of the OS kernel. In this paper, we shed light on the inner working of this facility and investigate, with detailed performance and overhead measurements considering deployments on physical machines and virtual machines with Linux and KVM, how it can be leveraged for more efficient inter-process communications, as an alternative to OS (Unix) signals. With UINTR, preemptive user-level thread schedulers can achieve quantum lengths in the microsecond scale.

## CCS Concepts

* Software and its engineering → Process synchronization; Virtual machines; * Computer systems organization; * General and reference → Performance;

## Keywords

user interrupts, inter-processor interrupts, virtualization, scheduling, performance

## ACM Reference Format:

Yves Kone, Louis Duval, Renaud Lachaize, Pascal Felber, Daniel Hagimont, and Alain Tchana. 2025. Understanding Intel User Interrupts. In *Abstracts of the 2025 ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems (SIGMETRICS Abstracts '25)*, June 9–13, 2025, Stony Brook, NY, USA. ACM, New York, NY, USA, 3 pages. https://doi.org/10.1145/3726854.3727326

## 1 Introduction

In this paper, we study *Intel User-level Interrupts* (UINTR, also known as “UIPIs”) [2, 5], a recent hardware feature integrated by Intel into its 4th generation (“Sapphire Rapids”) Xeon processors. UINTR

is a new mechanism within the family of Inter-Process/Thread Communication (IPC) facilities: it allows a sender thread to send a physical interrupt to another CPU, triggering the execution of a routine previously set up by the receiver thread (within the same or a different process), *without any intervention of the operating system (OS) kernel*. This feature enables efficient coordination between CPU cores, which is essential to support *μsecond-scale scheduling* [3, 6, 10, 11].

Indeed, with the recent development of critical networking applications, such as in-memory key-value stores, the presence of very short jobs (on the *μsecond scale* [11]), combined with a broad service time distribution, has made achieving low tail latency more challenging [1]. It is more likely that very short requests end up queuing behind longer ones, resulting in head-of-line blocking [11]. OS-level schedulers, which assign time slices on the order of milliseconds, cannot achieve *μs-scale tail latencies* [8, 9].

A promising approach to meet *μs-scale tail latency* is to transform application threads into user-level threads (uThreads), which are then scheduled on top of kernel-level threads (kThreads) by a specific *userspace uThread scheduler* [3, 4, 6, 10–12]. In this approach, the enforcement of very short quantum durations is crucial for controlling tail latency [6, 10–12], and UINTR potentially provides a new way to achieve this goal via fine-grained preemptions, without strong assumptions on the cooperation of applications. Our main goal is to characterize the performance of UINTR and assess to which extent it can improve the performance and accuracy of workloads with such granular, *μsecond-scale inter-core coordination* requirements. To establish a baseline, we systematically compare UINTR to OS signals (OSS for short), the standard facility for asynchronous interactions between threads or processes. Our evaluations encompass a large spectrum of deployment contexts including physical and virtual machines (with and without hardware-assisted IPI virtualization) and also study the impact of thread placement and interference.

To summarize, this paper<fnref n="1" /> makes the following three main contributions.<fnref n="2" /> (1) We develop benchmarks for the performance characterization of asynchronous notification mechanisms, and we perform a qualitative and quantitative study of both OSS and UINTR,

SIGMETRICS Abstracts '25, Stony Brook, NY, USA.

© 2025 Copyright held by the owner/author(s).

This is the author's version of the work. It is posted here for your personal use. Not for redistribution. The definitive Version of Record was published in Abstracts of the 2025 ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems (SIGMETRICS Abstracts '25), June 9–13, 2025, Stony Brook, NY, USA, https://doi.org/10.1145/3726854.3727326.

<footnote n="1">¹The present document is an extended abstract of our full article [7].</footnote>

<footnote n="2">²Our code is available at https://gitlab.com/krakos-uintr/understanding-uintr.git</footnote>
including a detailed breakdown of their costs, in both native and virtualized environments. (2) We design and implement Christine, a flexible preemptive user-level thread scheduler (PUTS). To better understand the behavior of microsecond-scale PUTS, we introduce a precise methodology, based on new scheduling metrics, and we empirically evaluate several versions of Christine, covering OSS and UINTR, as well as preemptions with and without synchronization. (3) We derive several findings: (F₁) UINTR outperforms OSS for the sending latencies (by up to 25× in native mode and 34× in guest mode), the delivery latencies (by up to 6× in native mode and 3× in guest mode), and also for throughput (by up to 6× in native mode and up to 2× in guest mode). (F₂) Virtualization negatively impacts the performance of UINTR operations. Nonetheless, compared to IPI emulation at the hypervisor level, Intel's IPIv feature can significantly reduce the overhead of UINTR virtualization, containing it within a maximum degradation range of 1.4× to 2.6×. (F₃) For Christine, UINTR allows using a scheduling quantum length as low as 2µs or even slightly below 1µs for some configurations. (F₄) When using UINTR instead of OSS, Christine achieves better focus (up to 7×), stability (up to 8×), efficiency (up to 2×) and latency (up to 7×) in native and guest modes.

## 2 Background

**OS-level signals (OSS).** An OSS serves as a concise asynchronous message sent to a process or thread, carrying a unique identifier referred to as the *signal number*. Linux supports a maximum of 31 signal numbers. Signals can be issued through syscalls such as `kill()`. When a signal is sent to a process or thread, the OS may invoke a signal handler (also called *routine*) based on the current *disposition* of the signal. The disposition of a signal within a receiver context determines the action the OS will take upon delivering the signal. Signal dispositions can be modified from their defaults, enabling processes to intercept signals and execute routines in user mode or ignore signals.

**UINTR overview.** In a nutshell, UINTR relies on two main types of (per-thread) data structures allocated and initialized in kernel space, *UPID* (*User Posted Interrupt Descriptor*) and *UITT* (*UINTR target table*), as well as several privileged CPU registers, and the *Local Advanced Programmable Interrupt Controller* (*LAPIC*) of each core. A *UPID* structure corresponds to a mailbox for a recipient thread; it indicates where and how the recipient thread must be informed, e.g., which recipient LAPIC instance and which interrupt vector number to use, and it also keeps track of pending interrupt requests. A *UITT* is a sender-side lookup table, which contains references to the authorized *UPIDs* that can be targeted by a given sender CPU.<fnref n="3" /> Each entry corresponds to a (*UPID*, *request#*) pair.

A sender thread can issue a user interrupt via `SENDUIPI`, a new (unprivileged) x86 hardware instruction, which takes as input an entry number within the *UITT* instance currently set on the sender core. This results in posting the notification in the recipient *UPID*. Note that the execution of `SENDUIPI` does not always result in an inter-processor interrupt (IPI). Indeed, the delivery of a UINTR, which results in the execution of the application-provided handler, may be postponed depending on the state of the receiver. The

delivery is only triggered when the recipient thread (1) is running and (2) not currently masking UINTR requests. Thus, a sender thread only triggers an IPI towards a core if, upon execution of `SENDUIPI`, the two above-mentioned conditions are met. Note that this workflow is conceptually similar to how Linux conditionally leverages IPIs for the delivery of OS signals (OSS). Similarly to OSS, but based on unprivileged hardware instructions, UINTR also provides a set of operations to (1) read and modify the delivery mask, and (2) return from a request handler.

Although the kernel is not in the critical path for UINTR sending and delivery, it plays a role in two key phases: (1) upon initialization, enabling interrupts and registering a handler for a receiver thread, and then pairing the authorized sender thread(s) with a receiver, and (2) upon a context switch (between kernel threads), adjusting the contents of the UINTR-related registers and structures, thus allowing to use UINTR at the thread-level granularity.

## 3 Microbenchmarks

We have developed a suite of microbenchmarks to characterize the latencies of the elementary UINTR operations on the sending and receiving sides, as well as the delivery throughput. We also developed similar microbenchmarks for the equivalent OSS operations. Table 1 shows the various latencies associated with the sending/delivery operations based on different sender-receiver placements: same hyperthread (R,S1), different threads in the same core (R,S2), and different cores (R,S3). We also consider 3 execution modes: native mode (NATIVE), guest with (GIPIV) and without (GIPI) the hardware-assisted IPI virtualization feature (Intel IPIv).

<table><caption>Table 1: Latency of sending and delivery operations for OSS and UINTR in different setups. Each cell contains the 50th and the 99th percentiles separated by a dash.</caption><thead><tr><th colspan="5">NATIVE (CPU cycles)</th></tr></thead><tbody><tr><th colspan="5">Pinning Mode</th></tr><tr><td>pthread_kill()</td><td>R.S1</td><td>R.S2</td><td>R.S3</td><td>Self</td></tr><tr><td>Signal Delivery</td><td>1.4k-1.6k<br>46M-58M</td><td>2.5k-3k<br>5.3k-5.8k</td><td>4.1k-4.7k<br>5.7k-6.1k</td><td>3.1k-3.4k<br>2.4k-2.7k</td></tr><tr><th colspan="5">SENDUIPI Posting</th></tr><tr><td>SENDUIPI Posting + Notif.</td><td>160-166<br>196-220</td><td>160-164<br>218-234</td><td>160-182<br>370-422</td><td>162-168<br>186-194</td></tr><tr><td>UINTR Delivery</td><td>46M-58M</td><td>930-1k</td><td>1.2k-1.3k</td><td>838-856</td></tr></tbody><thead><tr><th colspan="5">GIPIV (CPU cycles)</th></tr></thead><tbody><tr><th colspan="5">Pinning Mode</th></tr><tr><td>pthread_kill()</td><td>R.S1</td><td>R.S2</td><td>R.S3</td><td>Self</td></tr><tr><td>Signal Delivery</td><td>1.7k-1.9k<br>46M-47M</td><td>2.6k-3.6k<br>6.6k-7.4k</td><td>3.9k-7.1k<br>7.3k-8.1k</td><td>3.1k-3.6k<br>2.4k-2.7k</td></tr><tr><th colspan="5">SENDUIPI Posting</th></tr><tr><td>SENDUIPI Posting + Notif.</td><td>180-206<br>404-462</td><td>180-206<br>464-530</td><td>180-206<br>750-840</td><td>204-234<br>392-448</td></tr><tr><td>UINTR Delivery</td><td>46M-47M</td><td>2.2k-2.6k</td><td>2.6k-2.9k</td><td>2k-2.2k</td></tr></tbody><thead><tr><th colspan="5">GIPI (CPU cycles)</th></tr></thead><tbody><tr><th colspan="5">Pinning Mode</th></tr><tr><td>pthread_kill()</td><td>R.S1</td><td>R.S2</td><td>R.S3</td><td>Self</td></tr><tr><td>Signal Delivery</td><td>1.7k-1.9k<br>46M-46M</td><td>2.6k-6.8k<br>8k-9.1k</td><td>4.1k-7.7k<br>8.6k-9.5k</td><td>3.1k-3.6k<br>2.3k-2.7k</td></tr><tr><th colspan="5">SENDUIPI Posting</th></tr><tr><td>SENDUIPI Posting + Notif.</td><td>180-206<br>4.2k-5.6k</td><td>180-208<br>3.1k-3.6k</td><td>180-208<br>2.9k-3.2k</td><td>202-234<br>6k-6.8k</td></tr><tr><td>UINTR Delivery</td><td>46M-46M</td><td>3.8k-4.3k</td><td>3.8k-4.2k</td><td>2.8k-3.1k</td></tr></tbody></table>

**UINTR impacting factors.** We observe that thread placement and virtualization have a significant impact on the latency of UINTR operations. For instance, we observe that increasing the physical distance between distinct sender and receiver (hardware) threads degrades UINTR delivery latency by up to 1.3×. In a guest VM, compared to a native setup, the performance of UINTR *Posting + Notification* operations is significantly degraded, by up to 2× with Intel IPIv and up to 35× without. The *Posting* operation is

<footnote n="3">³Each CPU/thread can act as a sender and/or as a receiver.</footnote>
degraded by up to 39%. Finally, virtualization degrades UINTR delivery latency by up to 2.6× with Intel IPIv and up to 4.3× without.

**UINTR versus OSS.** We observe a significant performance advantage for UINTR over OSS regarding sending and delivery operations. For instance, UINTR sending operations significantly outperform OSS, both in native mode and virtualized mode with Intel IPIv, by up to 25× and 34× respectively. UINTR delivery is up to 6× faster than OSS in native mode and up to 3× in virtualized mode. In virtualized mode without Intel IPIv, self delivering a UINTR is up to 20% slower than OSS. Also, regarding the maximum throughput between distinct hardware threads, UINTR outperforms OSS by up to 6.5× in native mode and up to 2.1× in virtualized mode.

# 4 Case study: microsecond-scale scheduler

For the purpose of this paper, we built Christine, a typical preemptive uThread scheduler, whose main architectural traits can be seen in almost all µsecond-scale schedulers published in recent years [4, 6, 10]. We use Christine to evaluate to what extent UINTR can help reducing tail latencies on physical and virtual machines.

**Overview.** Christine comprises a set of $N + 1$ kernel-managed threads (hereafter “kThreads”): *N* worker threads (in our experiments, we set *N* to 1 or 7), in charge of running user-level threads (hereafter “uThreads”), and a timer thread. Each of these kThreads is pinned to a dedicated CPU core. The timer thread periodically sends notifications to all the workers, corresponding to the end of a quantum. Upon receiving a notification, each worker executes a handler implementing the local scheduler code, which selects the next elected uThread and swaps contexts.

**Metrics.** We identify four important metrics for assessing a preemptive uThread scheduler: *efficiency* (the proportion of time spent executing useful work), *minimum latency* (the minimum effective quantum length that can be achieved), *stability* (the standard deviation of the effective quantum lengths), and *focus* (the ability to guarantee a desired quantum length). The last two represent the *precision* of a PUTS.

**Results.** For each of the metrics mentioned above, we measure the impact of the preemption mechanisms (UINTR and OSS) in Christine and compare the different implementations. For the *efficiency* metric, UINTR outperforms OSS by up to 2× in native mode when the targeted quantum duration is short. In guest mode, this gap is around 1.5×. When the target quantum duration is large, UINTR outperforms OSS by up to 1.5× in native and guest mode.

Considering the (99th-percentile) *minimum latency* metric, UINTR outperforms OSS by 1.7× to 7× in both native and guest modes. With 1 worker, UINTR allows achieving a *minimum latency* in the order of 1 µs, or even slightly below. With 7 workers, it allows reaching 1.3–3.1 µs for the same metric. While OSS achieves a minimum latency of around 2.4 µs with 1 worker and 10–13 µs with 7 workers respectively in native and virtualized mode.

Regarding the *precision* of Christine, UINTR allows better performance in all cases. In native mode, UINTR achieves the lowest quantum duration (≈ 0.3 µs) while generally achieving the best

*focus* compared to OSS. Virtualization degrades the focus of the UINTR-based scheduler by up to 3× with 1 worker and up to 1.33× with 7 workers compared to native mode. Although the relative degradation is less significant for OSS in virtualized mode, the UINTR-based version still achieves a better focus than its OSS counterpart. In native mode, UINTR allows up to 8× better *stability* than OSS. Virtualization has little impact on scheduler stability, except for OSS with 7 workers (degradation up to 2×).

# Acknowledgments

We thank our shepherd, Derek Eager, and the anonymous reviewers for their insightful comments. This work was supported by the French Agence Nationale de la Recherche under the ANR projects "PicNic" (ANR-20-CE25-0013) and "PEPR DiVa" (ANR-23-PECL-0004), by Inria "Défi OS", and by the PAI 2021 project "Fault Tolerance for Disaggregated Rack-Scale Computing" from the French Auvergne-Rhône-Alpes Region.

# References

[1] Luiz André Barroso, Mike Marty, David Patterson, and Parthasarathy Ranganathan. 2017. Attack of the killer microseconds. *Commun. ACM* 60(4) (2017), 48–54.

[2] Jonathan Corbet. 2021. User-Space Interrupts. Linux Weekly News. (2021). https://lwn.net/Articles/871113/

[3] Joshua Fried, Gohar Irfan Chaudhry, Enrique Saurez, Esha Choukse, Inigo Goiri, Sameh Elnikety, Rodrigo Fonseca, and Adam Belay. 2024. Making Kernel Bypass Practical for the Cloud with Junction. In *21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24)*. USENIX Association, Santa Clara, CA, 55–73. https://www.usenix.org/conference/nsdi24/presentation/fried

[4] Joshua Fried, Zhenyuan Ruan, Amy Ousterhout, and Adam Belay. 2020. Caladan: Mitigating Interference at Microsecond Timescales. In *14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20)*. USENIX Association, 281–297. https://www.usenix.org/conference/osdi20/presentation/fried

[5] Intel. 2023. *User Interrupts*. Vol. 3C. https://cdrdv2.intel.com/v1/dl/getContent/671200

[6] Kostis Kaffes, Timothy Chong, Jack Tigar Humphries, Adam Belay, David Mazieères, and Christos Kozyrakis. 2019. Shinjuku: Preemptive Scheduling for µsecond-scale Tail Latency. In *16th USENIX Symposium on Networked Systems Design and Implementation (NSDI 19)*. USENIX Association, Boston, MA, 345–360. https://www.usenix.org/conference/nsdi19/presentation/kaffes

[7] Yves Kone, Louis Duval, Renaud Lachaize, Pascal Felber, Daniel Hagimont, and Alain Tchana. 2025. Understanding Intel User Interrupts. *Proc. ACM Meas. Anal. Comput. Syst.* 9, 2, Article 40 (June 2025). https://doi.org/10.1145/3727132

[8] Jacob Leverich and Christos Kozyrakis. 2014. Reconciling high server utilization and sub-millisecond quality-of-service. In *Proceedings of the Ninth European Conference on Computer Systems (EuroSys '14)*. Association for Computing Machinery, New York, NY, USA, Article 4, 14 pages. https://doi.org/10.1145/2592798.2592821

[9] Jialin Li, Naveen Kr. Sharma, Dan R. K. Ports, and Steven D. Gribble. 2014. Tales of the Tail: Hardware, OS, and Application-level Sources of Tail Latency. In *Proceedings of the ACM Symposium on Cloud Computing (SOCC '14)*. Association for Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/2670979.2670988

[10] Yueying Li, Nikita Lazarev, David Koufaty, Tenny Yin, Andy Anderson, Zhiru Zhang, G. Edward Suh, Kostis Kaffes, and Christina Delimitrou. 2024. LibPreemptible: Enabling Fast, Adaptive, and Hardware-Assisted User-Space Scheduling. In *2024 IEEE International Symposium on High-Performance Computer Architecture (HPCA)*. 922–936. https://doi.org/10.1109/HPCA57654.2024.00075

[11] Zhihong Luo, Sam Son, Dev Bali, Emmanuel Amaro, Amy Ousterhout, Sylvia Ratnasamy, and Scott Shenker. 2024. Efficient Microsecond-scale Blind Scheduling with Tiny Quanta. In *Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (ASPLOS '24)*. Association for Computing Machinery, New York, NY, USA, 305–319. https://doi.org/10.1145/3620665.3640381

[12] Amy Ousterhout, Joshua Fried, Jonathan Behrens, Adam Belay, and Hari Balakrishnan. 2019. Shenango: Achieving High CPU Efficiency for Latency-sensitive Datacenter Workloads. In *16th USENIX Symposium on Networked Systems Design and Implementation (NSDI 19)*. USENIX Association, Boston, MA, 361–378. https://www.usenix.org/conference/nsdi19/presentation/ousterhout