# An XVisor extension using a Domain Specific Language to detect Rootkits in Virtual Machines

Lionel Hemmerlé, Guillaume - Hiet, Frédéric Tronel, Pierre Wilke, Jean-Christophe Prévotet

<figure><img src="image_1.png" /></figure>

To cite this version:

Lionel Hemmerlé, Guillaume - Hiet, Frédéric Tronel, Pierre Wilke, Jean-Christophe Prévotet. An XVisor extension using a Domain Specific Language to detect Rootkits in Virtual Machines. 2025. hal-04879065

HAL Id: hal-04879065

https://hal.science/hal-04879065v1

Preprint submitted on 10 Jan 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# An XVisor extension using a Domain Specific Language to detect Rootkits in Virtual Machines

Lionel Hemmerlé

lionel.hemmerle@centralesupelec.fr
CentraleSupélec,Inria,CNRS,University
of Rennes, France

Guillaume Hiet

guillaume.hiet@centralesupelec.fr
CentraleSupélec,Inria,CNRS,University
of Rennes, France

Frédéric Tronel

frederic.tronel@centralesupelec.fr
CentraleSupélec,Inria,CNRS,University
of Rennes, France

Pierre Wilke

lionel.hemmerle@centralesupelec.fr
CentraleSupélec,Inria,CNRS,University
of Rennes, France

Jean-Christophe Prévotet

jean-christophe.prevotet@insa-
rennes.fr
INSA Rennes : UnivRennes, INSA
Rennes, CNRS, IETR -UMR 6164,
F-35000 Rennes, France

## Abstract

Endpoint Detection Reaction (EDR) systems must remain secure even when attackers gain high privileges. To enhance protection, we propose to isolate the protected system in a Virtual Machine (VM) and implement the EDR in the hypervisor. This ensures that the EDR remains functional even if the VM is fully compromised. However, this setup introduces a semantic gap, as the EDR loses the operating system abstractions within the VM.

To address this, we developed HYPERSEC, a domain-specific lan-
guage allowing the VM to send programs to the hypervisor. These
programs leverage the VM's knowledge of its internal OS struc-
tures and enable the hypervisor to detect intrusions. To secure this
process, we enforce constraints on the set of programs accepted
by the hypervisor to prevent the exploitation of vulnerabilities
inside the hypervisor. We evaluated our approach using kernel
rootkits designed to hide processes within the VM or elevate pro-
cesses privileges, along with detection mechanisms for such attacks.
We evaluated the overhead induced by HYPERSEC programs and
showed that this overhead is acceptable for detecting most rootkits.
Furthermore, we show that overhead can be further reduced by
using compiled programs instead of an interpreter.

## CCS Concepts

* Security and privacy → Intrusion detection systems; Virtualization and security.

## Keywords

Security, Hypervisor, Intrusion Detection Systems, Rootkit

## ACM Reference Format:

Lionel Hemmerlé, Guillaume Hiet, Frédéric Tronel, Pierre Wilke, and Jean-Christophe Prévotet. 2024. An XVisor extension using a Domain Specific

Language to detect Rootkits in Virtual Machines. In *Proceedings of ACM Conference on Data and Application Security and Privacy (CODASPY '25)*. ACM, New York, NY, USA, 13 pages. https://doi.org/XXXXXXX.XXXXXXX

# 1 Introduction

Intrusion Detection Systems (IDS) can be placed at different levels to detect intrusions: they can be located on some critical network nodes or directly on the systems that must be protected. Network-based IDS (NIDS) are generally limited since they can only detect attacks leaving specific traces on the network. However they are fundamentally unable to detect local attacks (for example, a privilege escalation). Host-based IDS (HIDS), or their generalization known as Endpoint Detection and Reaction (EDR hereafter) runs directly on the system that must be protected and can detect a much broader range of attacks. Those EDR can be privileged user-mode processes and use data made accessible by system calls, or they can even run directly in kernel mode and access data in kernel memory. In both cases, malware can disable them if they execute with sufficiently high privileges [44]. For example, an attacker who can compromise an operating system (OS) can kill processes related to the EDR or send them erroneous information.

In order to safeguard an EDR, it may be deployed in a distinct environment from the protected operating system [29]. This may involve utilizing another dedicated processor that operates independently from the one hosting the protected operating system. However, this approach has a drawback since the EDR cannot access information such as system register values, which allows attackers to remain undetected [27]. To overcome this limitation, we have chosen to isolate our EDR from the virtual machine (VM) by executing it in a higher privilege mode that employs processor virtualization extensions. This enables the EDR to perform Virtual Machine Introspection (VMI) [18] and offers improved protection against potential attacks.

A hardware component called a virtualization extension allows a processor to mimic several virtual processors. These virtual processors are created and managed by a software component known as an hypervisor. By overseeing every interaction between a virtual processor and the hardware, the hypervisor can isolate these different processors. Each virtual processor can then operate its own operating system, forming a VM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org.

CODASPY '25, June 4-6, 2025, Pittsburgh, PA, USA

© 2024 Copyright held by the owner/author(s). Publication rights licensed to ACM.

ACM ISBN 978-1-4503-XXXX-X/18/06

https://doi.org/XXXXXXX.XXXXXXX
Lightweight hypervisors have been increasingly used in embed-
ded systems to regroup various systems on the same processing
units while still keeping them separated, to allow the use of differ-
ent operating systems on the same physical machine and to isolate
vulnerable programs to protect the rest of the system [21].

As the hypervisor operates at a higher privilege level than a VM, even if an attacker completely compromises a VM, they cannot attack the hypervisor or other VMs. This assumes that the hypervisor does not have any critical vulnerabilities. The hypervisor can thus be utilized to protect a VM, such as safeguarding the kernel code [42] or enforcing the integrity of specific applications [33].

The hypervisor may regularly perform some monitoring (as im-
plemented by Secure Kernel Patch Guard in Windows 10 [50], or
Samsung's Real-time Kernel Protection [40]) to check that some
security invariants always hold; or it can react to events occurring
in the VM. We choose to implement the latter because the former
would allow an attacker to perform a transient attack where invari-
ants can be temporarily broken between two snapshots, without
the hypervisor noticing.

When the EDR is installed on the hypervisor, it can fully access
the VM memory and its interactions with the hardware. However,
the EDR loses the abstractions provided by the operating system in
the VM. For example, to retrieve the list of processes running in a
VM, the EDR must clearly understand how a process is represented
in the VM memory. It should also know the storage location of
the structures representing the running processes and should be
notified of specific events occurring in the VM, such as the start
of a new process. This challenge is commonly referred to as the
"semantic gap problem" [26].

Various methods have been suggested for bridging the seman-
tic gap. Garfinkel et al. [18] proposed developing a library for a
specific version of the operating system that retrieves data from
the VM memory. This approach is highly accurate but not very
versatile, as the library needs to be rewritten for each new version
of the operating system. Additionally, it may be difficult to sup-
port closed-source operating systems using this method. Xiao et
al. [48] use heuristics to increase versatility and work on multiple
operating systems. However, they can only retrieve partial data,
such as the names and process IDs of the running processes. Zhao
et al. [17] propose a solution to reuse user-space programs as VM
 introspection tools, but their solution is vulnerable to direct kernel
object manipulation from an attacker. Dolan-Gavit et al. [13] use
a training phase to teach the intrusion detection system how to
retrieve data. If the training process is incomplete, the VMI tool
may fail to extract data from the VM memory. Westphal et al. [46]
developed a domain-specific language that allows users to specify
the sequences of events in a VM that could be caused by an intruder.
However, their approach is limited by the need for external config-
uration files that are required for each operating system used in
the various VMs.

For our EDR to be compatible with multiple operating systems, we could use a paravirtualization technique [47] by modifying the VM to communicate with the hypervisor and provide information about its state. This would allow the VM to delegate certain security checks to the hypervisor [24, 35] and send information to the hypervisor whenever specific events occur within the VM [38]. While

this approach can help to protect the VM and provide detailed in-
formation to the hypervisor, it requires modifying the guest kernel
code to enable communication between the VM and the hypervi-
sor. Additionally, it could reduce the resilience of the EDR since
an attacker could prevent the VM from communicating with the
hypervisor.

Instead, we propose a solution to detect intrusions by allowing a VM to send programs to the hypervisor during VM provisioning. These programs are then executed by the hypervisor when specific events occur inside the VM. They can read the memory and the registers' contents of the VM to determine if it has been infected by a rootkit. Before running the programs, the hypervisor verifies the programs to make sure that they do not contain any bugs that could compromise their integrity. This solution does not require any change to the kernel code but merely loads a kernel module at startup to send a sequence of programs to the hypervisor. Once the last program has been sent, the module can be unloaded.

A similar approach has been proposed by Wei and Amit [45],
but it has not been designed primarily for security purposes. Their
equivalent of our programs can be sent at any time by the VM. This
could allow a privileged attacker to turn off the security monitoring
by sending their own programs to the hypervisor. Additionally,
they make use of hyperupcalls to react to events occurring in the
hypervisor (such as memory reclaim or virtual processor preemp-
tion), whereas our solution detects events happening within a VM.
Bitdefender [5] proposes a similar approach for x86, but the seman-
tic gap problem is not solved by the protected VM but is hardcoded
in the introspection library, which is not flexible.

The primary objective of this study is to devise a methodology
for detecting intrusions against the kernel of an operating system
using a generic approach. To achieve this, VMs are permitted to
transmit programs coded in a specific language, to the hypervisor.
The contributions of this research can be succinctly summarized as
follows:

* We design HYPERSEC, a language dedicated to the specifica-
  tion of security monitors inside an hypervisor.

* We implement an interpreter for HYPERSEC programs running inside the XVisor [49] hypervisor.

* We implement three known and one new attacks that aim at hiding processes or elevating their privilege, as Linux kernel modules.

* We implement detection techniques against these attacks using HYPERSEC programs.

* We demonstrate that our detection technique is not limited to our own rootkits but also to other existing rootkits.

* We evaluate the performance and the security of our ap-
  proach using typical workloads.

The remainder of this article is organized as follows. First, Sec-
tion 2 gives a high-level overview of our approach, and Section 3 in-
troduces several malicious techniques that we aim to detect with our
solution. Section 4 presents the features of HYPERSEC, the proposed
domain-specific language, and the associated security concerns.
Section 5 reports on the implementation of HYPERSEC features on
a Cortex-A53 ARM processor. Section 6 explains how we use HY-
PERSEC programs to detect all the rootkits described in Section 3.
<figure><img src="image_3.png" /><figcaption>Figure 1: Representation of our detection system</figcaption></figure>

Finally, Section 7 discusses the limitations of our approach and gives perspectives for future works. Section 8 concludes.

## 2 Threat model and approach

We suppose that a malicious agent can entirely compromise a VM, including the OS kernel, and that one of its objectives is to remain stealthy. Consequently we focus on detecting malware that may change the kernel's internal data structures. Indeed, detecting malware that runs only in userland does not require a hypervisor (an EDR running in the kernel is sufficient, and such EDR could then be protected by our system). We simulate such malware with malicious kernel modules. We suppose that our hypervisor is safe and one of our objectives is to ensure that our approach does not introduce vulnerabilities into it. If that were the case, an attacker could compromise the hypervisor or another VM running on top of it.

To detect kernel rootkits, we introduce the HYPERSEC language. Our approach is illustrated in Figure 1. The VM to be protected will write HYPERSEC programs that describe how intrusions should be detected, compile these programs into an AST structure, and send them to the hypervisor during the early phase of the VM's life. Note that it is assumed that the VM has not been compromised during this early stage. HYPERSEC programs contain every piece of information to allow the hypervisor to cross the semantic gap. More precisely, they allow to: (1) specify events (memory writes in specific kernel data structures, privileged register modifications, kernel function calls), and (2) attach actions to these events (update of the hypervisor view of the OS, alerts raising, new events monitoring, ...). The hypervisor will verify HYPERSEC programs, and run them as soon as it detects specific events.

Safely executing programs from untrusted sources, in a high-privilege context, is a problem that has already been addressed by EBPF [9], with proof-carrying code [36] or with Singularity [22], an OS that runs every process in ring 0 and relies on properties of its language to ensure the system's safety.

Our approach is analogous to the one used by EBPF inside the Linux kernel: EBPF is a language designed to trace events occurring in the kernel, from userspace. To do so, processes running in userspace can send EBPF programs to the kernel. The kernel compiles and executes these programs and sends the results to the initial process. This approach is similar to ours since it implies getting programs from an untrusted source (a process running in userspace) and executing them in the kernel.

However, our approach differs from EBPF in the sense that the objective is not to provide the VM (the untrusted environment) insight into the state of the hypervisor (the trusted environment)

but rather the opposite. In our case, the hypervisor needs to get information about the state of the VM. Consequently, HYPERSEC programs can only access VM information, and the hypervisor will exploit their results, meaning that our solution has additional security constraints since an attacker may be able to manipulate our program's inputs (for example, the VM memory).

In our solution, we must introduce some constraints to prevent an attacker from compromising the hypervisor. For example, we must restrict sending our programs during an initial phase in which we consider that the VM has not been compromised yet (for example, during the first boot). During that time, the integrity of the kernel can be protected using another security mechanism like secure boot. We must also guarantee that programs sent by the VM and executed by the hypervisor do not introduce vulnerabilities. For example, they should be unable to access the whole hypervisor's memory. These security aspects of the HYPERSEC language will be developed in Section 4.2.

## 3 Rootkits

In this section, we present rootkits that are designed to hide their presence on a system or allow a process to execute with elevated privileges. These rootkits target the Linux kernel (version 5.15.36) but the same ideas should be easily applicable to other versions of the Linux kernel and to other operating systems.

The ps command displays the list of running processes, by parsing the content of the /proc filesystem, which contains a directory for every running process. A rootkit may hide a process by deleting or hiding the corresponding directory. To evaluate our ideas, we implemented four rootkits and used two existing ones: Diamorphine [34] and Brokepkg [7]. Four out of the six rootkits aim to hide a process while two others elevate the privileges of a process. We grouped those rootkits into four categories depending on the type of structure they are modifying.

The techniques used in the rootkits described in categories RKro, RKvfs, and RKcred have already been described in [15, 30, 41]. However, to the best of our knowledge, RKpid, described in Section 3.3, has not been described elsewhere.

These rootkits are presented from the simplest to detect, to the hardest and correspond to Type 1 or Type 2 malware according to Rutkowska's classification [39]. As a reminder, Type 1 of this classification corresponds to malware that modify areas of code or data that are supposed to remain constant during system lifetime. This type of malware is therefore the easiest to detect and mitigate, since simply ensuring the read-only property at the hypervisor level is sufficient. More insidiously, Type 2 corresponds to malware that alters data (or more rarely code<fnref n="1" />) that evolves dynamically over time. Hence, they are more difficult to detect, since we cannot simply make data read-only. We must rely on some invariants that usually have to be created from the knowledge of OS internals. Furthermore, it is not so uncommon that although such invariants exist, they may be violated by the OS code during very small time periods. As an example, we could consider the case where we want to guarantee that all processes/threads in the system appears in a doubly linked list maintained by the kernel. During normal operation, it will happen that this doubly linked list is broken, when the kernel insert

<footnote n="1">¹since code is rarely suppose to evolve.</footnote>
<figure><img src="image_4.png" /><figcaption>Figure 2: Structures changed by our rootkits</figcaption></figure>

<figure><img src="image_5.png" /></figure>

a newly created process. So we cannot rely on simply checking regularly the consistency of this list, since we will find sometimes some inconsistencies. Rather we must synchronize with the kernel code, to check the consistency at some critical points (like after the creation or destruction of a process/thread). Hence, detecting (and even more) mitigating Type 2 rootkits is much more difficult.

RK<sub>ro</sub> changes only purely static data, RK<sub>vfs</sub> changes static data on a dynamic structure (e.g., a constant field in the inode structure). Those are Type 1 rootkits. RK<sub>pid</sub> change structures that are dynamically allocated and deallocated by the kernel (pid structures) and RK<sub>cred</sub> changes structures that are not only allocated and deallocated but can also be subject to legitimate changes (task_struct and cred structures). Those two rootkits are Type 2 rootkits.

The modifications implied by our rootkits are shown in Figure 2 and will be developed in the next subsections.

## 3.1 RK<sub>ro</sub>: modifying read-only data

Processes interact with filesystems through specific system calls. In particular, listing a directory's content is done by the getdents64 (*get directory entries*) system call. When a system call is triggered, the kernel looks up the corresponding kernel function in the syscall table and calls it.

A rootkit can then hide a process by modifying the syscall table to intercept calls to this function and filter out some entries, e.g., the directories corresponding to the processes to hide (this attack is illustrated in Figure 2a, red elements are values tampered by the rootkit). To detect this type of attack from the hypervisor, it is necessary to detect and prevent any write access to the syscall table.

## 3.2 RK<sub>vfs</sub>: modifying the Virtual File System operations

The Virtual File System (VFS) in the kernel provides a uniform interface for supported filesystem types. It uses inode structures to represent files and directories and file_operations structures with

function pointers for inode operations, including iterate_shared for listing directory contents.

RK<sub>vfs</sub> replaces, in the inode structure, the pointer to the legitimate file_operations by a pointer to a structure written by the attacker (Figure 2b). This allows the attacker to execute its malicious function each time a process tries to read the content of the /proc directory.

This attack has been implemented in adore-ng [1] but this rootkit specifically targets AMD64 architecture. We re-implemented this rootkit to target the ARM64 architecture.

To detect this attack from the hypervisor, we need to detect writes to the structure file_operations used by the inode representing the directory /proc and to the inode's field pointing to that file_operations structure. More generally, the hypervisor must detect some write operations in the VM memory. Note that both the syscall table and the file_operations relative to the /proc filesystem are placed in read-only pages in the VM and are not expected to be modified during legitimate execution of the VM. However, the inode structure representing the /proc filesystem is not constant, and some fields of that structure can be legitimately modified during the normal execution of the VM. This raises new challenges for the detection of these writes, that will be solved in Section 5.1.

## 3.3 RK<sub>pid</sub>: tampering with pid structures

In the kernel, processes and threads are modeled by task_struct structures. To keep track of the running processes, the kernel uses a radix-tree containing pid structures. These pid structures contain, in turn, references to their corresponding task_struct. The /proc VFS iterates on this radix-tree to construct the list of running processes. Therefore, our third rootkit, RK<sub>pid</sub>, hides processes by tampering with the tasks field, pointing to the corresponding task_struct from the pid structure, as shown in Figure 2c.

To detect this attack, it is necessary to detect specific function calls in the kernel. Indeed, we must detect when new task_struct and pid structures are created (i.e. when function kernel_clone
u ∈ Uint64
x ∈ Var ::= string
s ∈ Symbol ::= $string
m ∈ MapId ::= string
f ∈ FunName ::= string
t ∈ Type ::= u64 | listener | [ t ; u ]
r ∈ Reg ::= pc | sp | SP_EL0 | xᵢ (i ∈ [0; 31])
e ∈ Expr ::= m | u | x | vm_reg ( r ) | s
| e₁ binop e₂ | unop e
| f (⃗e) | e₁ [ e₂ ] | [ e ; u ] | [⃗e]
i ∈ Instr ::= e₁ [ e₂ ] = e₃ ; | x : t = e ; | { i* }
| loop n { i* } | for n in m { i* }
| if e then { i₁* } else { i₂* }
| f (⃗e) | alert ; | return e ;
c ∈ Cmd ::= map m [ t₁ ] -> t₂
| fun f ( x : t ) -> tₚet { i* }
| exec f
| last_command
p ∈ Prog ::= ⃗c

<caption>Figure 3: Syntax of HYPERSEC</caption>

is called in the kernel) or when they are destroyed (i.e. when the function do_exit is called).

3.4 RKcred: altering credential structures

Processes store their various user identifiers (real UID, saved UID, effective UID) in a cred structure. A pointer to such a structure is stored in each task_struct. The cred structure is then accessed each time a process attempts to execute an action that may require special privileges (for example, opening a file). The cred structure is not supposed to be changed once it is created, but new structures are created each time a running process uses a syscall updating its credentials (for example, the setuid syscall). These updates follow strict rules to enforce the security of the system. RKcred changes the cred structure of a process to elevate its privileges even if the process was initially started by an unprivileged user (Figure 2d).

To detect this attack, we need to distinguish between legitimate
changes, which are performed by system calls, and illegitimate
ones. We also need to verify that changes made by system calls are
correct to ensure that an attacker does not use a system call and
simultaneously rewrite the credentials to hide their attack from
our detection system. This implies that a copy of the credentials of
every running process needs to be stored in the hypervisor.

4 Language specification

To detect complex attacks, we designed HYPERSEC, a language
dedicated to bridging the semantic gap between a VM and the hy-
pervisor, and specifying security properties to be enforced. The
hypervisor executes HYPERSEC programs to react to events occur-
ring inside the VM. In order to demonstrate the feasibility of the
presented approach, we need to make sure that the ability of the

map syscalls_per_pid[u64] -> u64
fun onInvoke_syscall(l: listener) {
  task_struct_ptr: u64 = vm_reg(SP_EL0);
  pid: u64 = vm_mem_4(task_struct_ptr + 0x410);
  counter: u64 = 1;
  if contains(syscalls_per_pid, pid) {
    counter = get(syscalls_per_pid, pid) + 1;
  }
  set(syscalls_per_pid, pid, counter);
}
fun init() {
  register_exec($invoke_syscall, onInvoke_syscall);
}
exec init

<caption>Figure 4: A HYPERSEC program to count the number of syscalls per process.</caption>

VM to send programs to the hypervisor does not introduce any vul-
nerability in the hypervisor. This is possible thanks to compile-time
and run-time verifications.

4.1 Language syntax

The syntax of HYPERSEC is presented in Figure 3. A program consists of a list of commands that are the messages sent by the VM to the hypervisor. These commands are: (1) `map m [ t₁ ] -> t₂`, for the definition of a new map named `m` whose keys are of type `t₁` and values of type `t₂`. Maps are key-value stores which are persistent between several invocations of the hypervisor; (2) `fun f (x : t) -> tₚet { i* }`, for the definition of a new function named `f` with arguments `x : t`, returning a value of type `tₚet`, and whose body is `i*`; (3) `exec f`, for calling function `f`, previously defined. This command is typically used to initialize the monitor environment, and requires no argument; and (4) `last_command`, which indicates the end of communication between the VM and the hypervisor.

The body of functions consists of instructions: array update,
variable declaration and initialization, loops with a fixed number of
iterations, loops over map entries, conditional statements, function
calls, emission of alerts and function returns.

Expressions may be unsigned 64-bit integers *u*, variables *x*, maps *m*, access to a register of the VM vm_reg(*r*), a kernel symbol *s*, binary or unary operations over expressions, a function call *f*(*e*̃), array access *e*₁[*e*₂], construction of an array by repeating *u* times an expression *e*: [*e*; *u*] or an array literal [*e*̃]. Observable VM registers are the program counter pc, the stack pointer sp, the system register SP_EL0 and regular registers *x*ᵢ (with *i* from 0 to 31).

The system register SP_EL0 is made available to recover the
identity of the process currently executing. According to the ARM
specification, this register is supposed to capture the value of the sp
register in user mode, for access from kernel mode. However, the
Linux kernel modifies this register so that it contains the address
of the task_struct of the current process when the VM executes
in kernel mode. This assumption is only valid if traps never occur
directly from user mode to hypervisor mode. This is always the
case under normal circumstances since we are specifically targeting
events from kernel-space. Any deviation from this behaviour can
be checked by the hypervisor and will raise an alert.
We provide several built-in functions: (1) register_write attaches a handler to a write in a memory area and returns an opaque object identifying that handler; (2) register_exec is similar to register_write but attaches a handler to the execution of a given function in the VM; (3) unregister disables a previously defined handler; (4) vm_mem_1, vm_mem_2, vm_mem_4, vm_mem_8 reads 1, 2, 4 or 8 bytes from the VM memory. (5) get stores a value in a map; (6) set reads a value from a map; (7) contains test if a key has an attributed value in a map; (8) debug prints a value.

Figure 4 shows an example of a HYPERSEC program, which counts the number of system calls made by every process (identified by their PIDs). This example is composed of 4 commands: the first one defines a new global map used to store the system call count for each process. The two next commands define two functions: onInvoke_syscall, which updates the count for the appropriate PID, and init, which registers the previous function to be called each time the invoke_syscall is executed in the VM. The last command executes the function init. The function onInvoke_syscall is able to recover the PID of the process invoking the system call by reading a pointer to its task_struct from the SP_EL0 register, and adding the field offset containing the PID (0x410 in our specific kernel version) to it.

In HYPERSEC, symbols from the kernel are prefixed by a $ symbol and are replaced (at pre-processing time, in the VM) using the /proc/kallsyms file by the corresponding addresses before sending the program to the hypervisor.

Field offsets within data structures are currently hardcoded by the programmer (in Figure 4, 0x410 correspond to the PID field in the task_struct structure). To make program more robust to re-compilation or kernel updates, we could use the debug information (BPF Type Format or BTF [6]) provided by recent Linux Kernel for eBPF usage. This could be implemented either by the HYPERSEC compiler inside the VM when initially loading the security programs or inside the hypervisor during the boot phase of the VM. These issues are typical of the semantic gap problem.

## 4.2 Security Constraints

The HYPERSEC programs that are supposed to protect the VM are sent by the VM itself. These programs will take the state of the VM as input, including its memory and registers. Therefore, we take some measures to ensure that these programs do not compromise the hypervisor and effectively protect the VM.

### 4.2.1 Restricting the emission of new commands.
First, we restrict the ability of a VM to send commands to the hypervisor. This aims to prevent attackers who may compromise the VM from sending their own malicious programs. Hence, we suppose the kernel is protected during its first boot, for example using secure boot. It can thus send its commands during this phase. When there are no more commands to be sent, the VM should send a last_command command to the hypervisor, indicating that the last command has been sent. Subsequent commands will simply be ignored by the hypervisor and may raise an alert.

### 4.2.2 Language safety.
HYPERSEC programs react to events that are possibly triggered by an attacker. Therefore, we must ensure that HYPERSEC programs cannot compromise the hypervisor's memory

or leak data from another VM. The language described in Figure 3 does not allow programs to access memory without restrictions, but rather read-only accesses through the vm_mem built-in and map manipulations. In the first case, the interpreter will ensure that the requested address belongs to the VM and is valid. In the second case, the interpreter will check the validity of the map accesses.

The termination of HYPERSEC functions is ensured by only allowing statically bounded loops, either with a fixed number of loop iterations, or iteration over a finite map, during which modifications to that map are forbidden.

Another possible source of Denial-of-Service (DoS) attacks is an excessive number of events registered for a single VM. To mitigate this, we could limit the number of registered events and trigger an alert when this threshold is reached.

## 5 Language implementation

Our implementation is based on XVisor [49], a minimalist hypervisor that can be executed on ARM processors and that is targeting embedded architectures [37]. On top of XVisor, we run a single-core VM containing a Linux kernel (version 5.15.36), built using Petalinux. The ARM processor is a Cortex-A53 processor, on a Xilinx Zynq UltraScale+ ZCU104 development board.

We extended XVisor with an interpreter for HYPERSEC programs. The VM can send commands to our interpreter thanks to an hypercall (ARM instruction HVC 0xef, where 0xef is the hypercall number that we chose for our purpose). Hypercall arguments are passed in registers x1, containing a pointer to the start of the program, and x2, containing the size in bytes of the program.

Commands are written in a human-readable format in the VM, which are preprocessed to replace symbol names with their actual addresses in the VM memory. Commands are then compiled into an Abstract Syntax Tree (AST), which is finally sent to the hypervisor. The layout of the AST consists of a header, a list of instructions, a list of expressions, and a list of data types, with references between instructions, expressions and types.

When the hypervisor receives a program, it first verifies that (1) the AST sent by the VM is well-formed (the hypervisor buffer is large enough, and there are no out-of-bound references between instructions, expressions, and types), and (2) the program itself is well-formed, i.e. every instruction, expression, and type is correct: variables are not used before their definition and initialization, all accessed maps have been defined previously, called functions are also properly defined and called with arguments of the correct type, and maps are not modified while being iterated over. Other remaining possibilities for runtime errors must be checked at runtime. In particular, we must check divisions by 0, out-of-bounds array access, and access to maps using undefined keys.

Most of the HYPERSEC language features are similar to general-purpose programming languages. However, features such as detecting writes into specific memory locations or detecting the execution of specific instructions require original implementation.
<figure><img src="image_6.png" /><figcaption>Figure 5: Address translation mechanism for a VM. Nodes in blue are controlled by the hypervisor, while those in white are controlled by the VM.</figcaption></figure>

## 5.1 Detecting Writes into Memory Locations

To detect writes in memory, depending on the size of the memory area to be watched and its alignment, we rely either on page permissions of the second stage of the address translation mechanism (which is controlled by the hypervisor), or on hardware breakpoints.

5.1.1 Page Permissions. On ARM processors, virtual memory is configured by a set of page tables organized in a radix-tree structure. In a traditional setting, without any hypervisor, the translation of a virtual address into a physical address is performed by splitting the virtual address into several indexes and looking up the first index in the table whose address can be found in the system register TTBR1_EL1 (*Translation Table Base Register* 1, where EL1 stands for *Execution Level* 1, i.e. kernel mode). This gives the address of the next-level page table to look up into, using the next index from the virtual address, and so on until the end of this lookup chain. This is illustrated by the top part (in white) of Figure 5.

When an hypervisor is present, the address translation mechanism is now split into two stages. The first stage translates VM virtual addresses into intermediate addresses (physical from the point of view of the VM, virtual from the point of view of the hypervisor), and the second stage translates intermediate addresses into physical addresses, using a process similar to the first stage, starting from the page table whose address is given in the VTTBR_EL2 register (*Virtualization Translation Table Base Register*, where EL2 now stands for *Execution Level* 2, i.e. hypervisor mode). The second stage is illustrated at the bottom part (in blue) of Figure 5.

In our setting, the page tables corresponding to the first stage (white) are controlled by the VM and therefore considered compromised, while the page tables corresponding to the second stage (blue) are controlled by the hypervisor and are considered safe.

In order to detect writes in memory at a given address (a virtual address in the address space of the kernel), the HYPERSEC interpreter first translates the virtual address into an intermediate address. This can be done with the ARM assembly at (for *Address Translate*) instruction (other methods could be used for architectures such as x86 that do not have similar instruction [51]). Then we identify the page table entry in the second stage that corresponds to the physical address of that memory area and we modify its permissions to make it read-only. Thus, any write access to that memory area will trigger a page fault exception for the hypervisor to catch.

However, exceptions are raised for write accesses over a whole page (4KB). We still need to distinguish between writes in the memory area the program wants to monitor and writes in the protected

<figure><img src="image_7.png" /><figcaption>Figure 6: Example of infinite loop caused by the EDR in the hypervisor. The assembly code comes from the compilation of the rwsem_acquire_read macro from the Linux kernel.</figcaption></figure>

page outside that memory area. For that purpose, we use the register FAR_EL2 (*Fault Address Register*), which contains the faulting address that caused the page fault exception in hypervisor mode, and the register ESR_EL2 (*Exception Syndrome Register*), which contains the size of the memory access among other values. Some x86 processors provide features such as the Sub-Page Write Permissions [25], which allow to detect writes on 128-byte sub-pages. Unfortunately, no such extension exists on ARM.

**Mega pages.** To increase its performance, XVisor uses mega pages when mapping the VM memory to the physical memory. It thus exploits the possibility for page table level 2 to point directly to physical memory instead of a new page table. This reduces the number of steps needed to translate a virtual address. Consequently, since the data we want to protect are usually smaller than a mega page, we split those mega pages into 512 regular pages by creating the necessary level 3 page table to reduce the amount of data that may unnecessarily trigger exceptions in the hypervisor. Without this split, we would modify the permissions of the mega page and would trap every write access inside that mega page, incurring a larger overhead.

**Execution of trapped writes.** Once a write has been detected, the hypervisor will run the attached HYPERSEC function as needed. Regardless, the execution of the VM must continue and the memory write must be executed. The simplest solution is to allow writes on the page containing the faulting address temporarily, execute the VM in step-by-step mode to execute exactly one instruction (the one that writes in the VM memory), and revert back the permissions on the protected page (no write allowed). This works most of the time, but the behaviour of some instructions is altered by interrupts generated by the hypervisor, which prevents the step-by-step execution of these instructions. This is typically the case for some atomic instructions detailed in the next paragraph.
**Atomic Instructions and Exclusive Monitor.** ARM v8.0 processors feature exclusive load and store instructions. ARM uses the load-linked/store-conditional semantics (also known as *ll*/*sc*) when the Large System Extensions extension [3] is present. When an exclusive load is executed, the processor stores the address of that load in a component named exclusive monitor. Then, when an exclusive store is executed, the processor compares the address in the exclusive monitor with the one written at by the instruction. The memory write only occurs if the two addresses match. The store instruction will then write the status result of the store in an additional register. Furthermore, the exclusive monitor is emptied every time another write at the address stored in the exclusive monitor is made or when a CLREX (*CLeaR* EXclusive) or ERET (*Exception Return*) instruction is executed. Both instructions have this behavior to ensure that the exclusive monitor is emptied in case of a context switch to detect the loss of atomicity of execution for the instructions encompassed between the load and the store. Figure 6 shows an example of those instructions' usage from the rwsem_acquire_read macro [31] in the Linux kernel that can indeed turn into an infinite loop if it accesses an address monitored by the hypervisor. Without the proposed protection, the ldxr instruction populates the exclusive monitor and reads at the address stored into x0 (1), then the stxr instruction compares its address to the one stored in the exclusive monitor (2) and performs the write since they are equal. If the exclusive monitor is emptied between the load and the store, the conditional jump executes again that block of code (3).

In the course of implementing the interpreter for HYPERSEC, we came across a bug that caused the kernel to freeze due to an infinite loop when a page contains both a memory area protected by the hypervisor and an address accessed using load-linked/store conditional semantic since each interrupt treated by the hypervisor empties the exclusive monitor.

In the literature multiple approaches have been proposed to address this problem: either by using other atomic instructions [14, 43], by using a Transactional Hardware Memory extension [11], or by inferring the result of the exclusive store based on the previous instructions [16, 28]. The first two approaches are not available to us because our processor does not feature these extensions, and the third is unacceptable since any instruction could be present between the ldxr and the stxr. We decide to fully emulate the exclusive monitor inside the hypervisor. When an exclusive write or an exclusive load targets an area protected by the hypervisor, the hypervisor will start to detect both reads and writes on that area (by removing read and write permissions from the page). If the faulting instruction is an exclusive load or store, the hypervisor will emulate the instruction and store the targeted address in a memory structure we call a *virtual exclusive monitor*. Furthermore, when this virtual exclusive monitor is not empty, the hypervisor must detect when the ERET instruction is called in the VM. Using the Fine-Grained Trap extension [2] of ARM, we could have generated an interrupt when ERET instructions are emitted [2]. Unfortunately, this extension is only available from version 8.6 of ARM processors, while we are targeting an ARM processor v8.0, very frequent in embedded systems. Therefore, we decided to empty the virtual exclusive monitor when any interrupt occurs in the VM context. For that, we used the technique to detect the execution

<figure><img src="image_8.png" /><figcaption>Figure 7: Attacks against detection of memory accesses.</figcaption></figure>

of specific instructions described in Section 5.2 to detect when the first instruction of every segment of the Interrupt Vector Table is executed. To reduce the overhead, we enable that detection only when the exclusive monitor is not empty.

### 5.1.2 Leveraging Hardware Watchpoints.
Detecting writes by modifying the address translation mechanism requires to protect a whole page (4096 bytes on this architecture): this incurs a large performance penalty onto the VM, especially when areas surrounding the protected one are frequently written to, and even worse when those writes use the exclusive operations described previously.

To improve the performance of our implementation, we leverage hardware watchpoints to detect writes whenever its size is 8 bytes or less, and at least one hardware watchpoint is available (only 4 are available for each core on our processor)<fnref n="2" />. We configure these watchpoints by writing in the system registers DBGWCRn_EL1 and DBGWVRn_EL1, the configuration and the address of the nth watchpoint, respectively and we enable debug-related exceptions by setting the correct flag in MDSCR_EL1.

Watchpoints can also be used by the VM, in kernel mode. Therefore, we need to configure the hypervisor to trap VM accesses to these registers by setting the corresponding flag in MDCR_EL2 (*Monitor Debug Configuration Register*).

### 5.1.3 Mitigating Evasion of the detection of memory accesses.
If an attacker knows that we are detecting memory writes, they could try to use the first step of address translation mechanism, which is controlled by the VM (as shown Figure 5) to avoid being detected. That technique is called Address Translation Redirection Attack (ATRA) [27]. For that, they can use two different methods depending on the type of hook used by the EDR.

If we use the second stage of the address translation mechanism to protect a given intermediate address (IA in Figure 7a), a rootkit could copy this memory region somewhere else (at another intermediate address IA') and remap the original virtual address VA onto the new IA', thus evading the monitoring. This is one of the attacks presented by Jang et al. [27]. To detect and mitigate this attack, we enforce read-only permissions on page table entries used to translate each memory area we want to protect (since the page tables themselves are accessed by the processor using their intermediate addresses, they cannot be tampered using ATRA-like attacks). Furthermore, we need to detect writes to TTBR1_EL1, which contains the address of the first table used in stage one of the address translation process to ensure that the intruder cannot change it.

<footnote n="2">²Watchpoints can theoretically be used to detect writes on memory areas whose size is any power of 2, but this requires impractical alignment constraints on these memory areas.</footnote>
For that, we modified the register bit TRM in the register HCR_EL2 used to configure the hypervisor extension.

If watchpoints are used to monitor access to a given virtual address, a variant of that attack can be used by the attacker: they can map the same intermediate address to an unprotected virtual address as shown in Figure 7b. To detect this attack, we use the hardware watchpoint to detect both reads and writes to an address, and before every access, we compare the current value stored at the given address to a shadow copy stored in the hypervisor (updated after every access). If those two values differ, it means that an attacker used an ATRA-like technique to bypass our detection system. This allows us to detect that attack before the kernel uses the tampered value (the kernel only accesses the value legitimately by using the virtual address monitored by the hardware watchpoint).

## 5.2 Detecting the Execution of an Instruction

HYPERSEC relies on the ability to execute only one instruction in the VM before re-enabling some protections. For example, whenever a memory operation triggers a hypervisor interrupt, the corresponding security must be temporarily disabled to allow that memory operation to perform correctly. Then it must be re-enabled right after the execution of the faulty instruction. To do so, we make use of the debug extension of ARM processors to trigger software-step exceptions each time an instruction is executed in the VM when protection is removed. This allows us to detect events properly, but require careful configuration some control register to avoid introducing security caveats. In particular, the register used to program these software step exceptions is MDSCR_EL1 (Monitor Debug System Control Register), and can, by default, be accessed from EL1, i.e. kernel mode, meaning that the compromised VM can tamper with these registers. We modify the MDSCR_EL2 (Monitor Debug Configuration Register) register so that the generated step exceptions are handled in the hypervisor rather than in the OS kernel, and to prevent the VM from accessing the MDSCR_EL1 register. Furthermore, we modify the PSTATE (Processor State) register to enable the software-step exceptions and disable other kernel mode interrupts. This prevents an attacker from taking advantage of the fact that we disabled memory protection temporarily to trigger an exception and bypass the software-step exception to access the protected memory area without restrictions.

## 5.3 Single Stepping Mode

HYPERSEC relies on the ability to execute only one instruction in the VM before re-enabling some protections. For example, whenever a memory operation triggers a hypervisor interrupt, the corresponding security must be temporarily disabled to allow that memory operation to perform correctly. Then it must be re-enabled right after the execution of the faulty instruction. To do so, we make use of the debug extension of ARM processors to trigger software-step exceptions each time an instruction is executed in the VM when protection is removed. This allows us to detect events properly, but require careful configuration some control register to avoid introducing security caveats. In particular, the register used to program these software step exceptions is MDSCR_EL1 (Monitor Debug System Control Register), and can, by default, be accessed from EL1, i.e. kernel mode, meaning that the compromised VM can tamper with these registers. We modify the MDSCR_EL2 (Monitor Debug Configuration Register) register so that the generated step exceptions are handled in the hypervisor rather than in the OS kernel, and to prevent the VM from accessing the MDSCR_EL1 register. Furthermore, we modify the PSTATE (Processor State) register to enable the software-step exceptions and disable other kernel mode interrupts. This prevents an attacker from taking advantage of the fact that we disabled memory protection temporarily to trigger an exception and bypass the software-step exception to access the protected memory area without restrictions.

# 6 Rootkits and their detection using HYPERSEC

In this section, we report on the implementation of the rootkits described in Section 3, and explain how we detect each of them using HYPERSEC programs.

## 6.1 Rootkits

All our rootkits (but one) were implemented as kernel modules for version 5.15.36 of the Linux kernel. However, to show that our approach does not depend on the way the rootkit is inserted inside the kernel, we decided for the last rootkit to leverage the exploitation of a kernel vulnerability, namely CVE 2023-2163 [12], which affects the eBPF verifier, and is present on the version of the Linux kernel we are using. This vulnerability allows arbitrary writes in the kernel memory. This in turn allows us to inject rootkit code directly in the kernel without relying on regular kernel module infrastructure, in the same vein as KOPYCAT [32].

RK<sub>ro</sub>. As an example of rootkits that change read-only data, we used Diamorphine [34], which directly tampers with the syscall table to make the kernel call its functions instead of the original ones, and Brokepkg [7] which uses the ftrace mechanism, that directly changes the kernel code to intercept function calls. Those two rootkits can then be used to hide a process.

RK<sub>vfs</sub>. To implement the rootkit modifying the VFS, we used the filp_open function that returns a pointer to a file structure containing itself a pointer to the inode of the opened file or directory. By doing so, we obtained the address of the inode representing the /proc folder, allowing us to change that structure's field pointing to the file_operations structure to perform the attack.

RK<sub>pid</sub> and RK<sub>cred</sub>. For the rootkits modifying the pid and cred structures, we iterate over the doubly linked list tasks contained in the structures task_struct, starting from the global variable named init_task_struct. Then, from a given task_struct we can either retrieve and edit the reference of the corresponding pid structure to that task_struct, making the process invisible from userspace, or we could retrieve the address of the cred structure and replace the value of the effective user ID and effective group ID with 0, which represents the root user.

RK<sub>cred</sub>. We implemented the rootkit elevating the privileges of a process using two different methods providing similar results. One method uses a module that directly changes the credentials of a given process. The other exploits a vulnerability in the BPF verifier [12]. This vulnerability allows us to load invalid BPF programs that can do writes at arbitrary addresses in the kernel. Consequently, we used it to override the identifiers of any process to make it privileged.

## 6.2 Rootkit Detection and Performance Evaluation

This section presents the various methods, using HYPERSEC programs, that have been used to detect each rootkit and the overhead incurred by our EDR. The number of lines of code we used in our programs to detect every rootkit is presented in Table 2. We needed only 8 lines of code to detect our simplest rootkit and up to 524 lines of code for RK<sub>cred</sub>. Programs used to detect RK<sub>pid</sub> and RK<sub>cred</sub> are divided into two parts: one common part detects when processes

<footnote n="3">³To be stealthy, the hypervisor could hide the introduced hypercall instructions by providing the VM with two different views of the code: one to be read, identical to the original code, and one to be executed, which contains our hypercall for execution detection.</footnote>
<table><caption>Table 1: Compilation time of binutils and execution time of the various Mibench tests when detecting the different attacks.</caption><thead><tr><th></th><th>no detection</th><th>RK<sub>ro</sub></th><th>RK<sub>vfs</sub></th><th>RK<sub>pid</sub></th><th>RK<sub>cred</sub></th><th>RK<sub>cred</sub> with C</th></tr></thead><tbody><tr><td>binutils</td><td>147m9s</td><td>147m10s (0%)</td><td>147m6s (0%)</td><td>147m46s (0%)</td><td>271m52s (85%)</td><td>205m47 (40%)</td></tr><tr><td>mibench average overheads</td><td>0%</td><td>0.4%</td><td>0.3%</td><td>0.4%</td><td>55.5%</td><td>15.1%</td></tr></tbody></table>

<table><caption>Table 2: Number of lines of code needed (empty lines and comments excluded) to detect each type of rootkit</caption><thead><tr><th>Rootkit</th><th>LoC for our programs</th></tr></thead><tbody><tr><td>RK<sub>ro</sub></td><td>8</td></tr><tr><td>RK<sub>vfs</sub></td><td>46</td></tr><tr><td>RK<sub>pid</sub></td><td>78 + 40 = 118</td></tr><tr><td>RK<sub>cred</sub></td><td>78 + 446 = 524</td></tr></tbody></table>

are created or destroyed (containing 78 lines), and one part for each rootkit that implements the detection itself.

The overheads are presented in Table 1. The columns contain the execution time of our different benchmarks. The first column is our baseline, where we launch the benchmark in the VM without running any detection in the hypervisor. The benchmarks were run for the next four columns while detecting one of the rootkits. In the last column, we present the overhead to detect RK<sub>cred</sub> but this time directly using C code in the hypervisor, to illustrate the performances we could obtain using a JIT compiler as explained in the future work in Section 7.

We evaluate the overhead of our interpreter by compiling binutils [4] (once in each configuration) and by using the MiBench [20] automotive, network, telecoms, and security benchmarks (except for the PGP test, which could not be compiled for our architecture) 250 times for each configuration.

### 6.2.1 RK<sub>ro</sub> and RK<sub>vfs</sub>.
To detect those rootkits, we need to detect writes to addresses that are constant in kernel virtual address space. In this case, detecting writes on these areas does not add any overhead for legitimate usage since we are hooking on events that are not supposed to happen.

To detect attacks on the VFS, we also detect writes on the `i_fop` field of the inode corresponding to the `/proc` directory. Indeed, this field contains the address of the corresponding `file_operations` structure. Unlike the address of the `file_operations` used by `/proc`, the address of the corresponding inode is not exported in the list of the symbols accessible from `/proc/kallsyms` and can vary between reboots, even when KASLR [8] is disabled. To find it, we use the address of the head of the `super_blocks` linked list, an exported symbol. We iterate over that list to find the superblock of `/proc`, and then, knowing the address of that superblock, we can find the inode by iterating over the list of all inodes attached to that superblock.

Since the values modified by the attacker are supposed to be constant during the kernel life, it is rather easy to detect those attacks. The hypervisor only needs to raise an alert every time a write is detected in the corresponding memory areas. This also means that detecting those attacks adds no overhead to legitimate VM executions since the hypervisor only detects events that do

not happen during the normal VM lifetime. This is confirmed by observing the overhead in Table 1, in columns relative to RK<sub>ro</sub> and RK<sub>vfs</sub>, which stays very low, most of the time there is no overhead at all.

### 6.2.2 RK<sub>pid</sub>.
To detect RK<sub>pid</sub>, the hypervisor must maintain a shadow list of running processes. To do so, we first extract the list of running processes when the command is sent from the VM to the hypervisor by iterating over the radix tree containing all the pid structures. Then, we detect newly created `task_struct` by hooking on the function `on_wake_up_new_task`, which takes the address of the new structure as an argument, and we detect process destruction by hooking on the `do_exit` function. The running process list is stored in a dedicated map as well as the addresses of the corresponding pid and `task_struct` structures.

A naive approach to detect RK<sub>pid</sub> would be to forbid writes on some fields of the `task_struct` and pid structures. Still, since the kernel accesses these structures very frequently, this method would incur an overhead that would render the VM unusable.

The `ps` command calls the function `proc_root_readdir` to list the contents of the `/proc` directory. Consequently, we only need to verify that pid and `task_struct` are untampered when that function is called. For that, at the beginning of the function call, the hypervisor iterates over all those structures to check that they correspond to the data saved earlier and that no `task_struct` or pid structure is missing. This approach does not allow us to detect that a process has been hidden directly when the rootkit hides it, but we will detect it as soon as a userspace process calls `proc_root_readdir`. When detecting that rootkit, the overhead is relatively small for most usages (see column RK<sub>pid</sub> from Table 1).

### 6.2.3 RK<sub>cred</sub>.
To detect changes on cred structures, we need to store a copy of the user and group identifiers of every process. To that aim, we modified the code that detects the previous attack to store a copy of those identifiers when processes are created.

Since some syscalls can legitimately change the credentials of a process (namely `setuid`, `setreuid`, `setresuid`, `setfsuid`, `setgid`, `setregid`, `setresgid`, `setfsgid`, `execve` and `execveat`), we detect when these are called and update the hypervisor's copy of the credentials according to what those syscalls do.

To detect syscalls, we track the execution of the `invoke_syscall` function, whose second argument is the syscall code, which allows us to know the current syscall and compute the new credentials using the arguments of the syscall. To correctly compute the credentials after the `execveat` syscall, we also need to know the owner and group of the executed file. For that, we add a hook in a specific instruction of the `bprm_execve` function to get the file structure representing the binary, which contains a pointer to the corresponding inode struct, which contains the user and group ids of the file and the `setuid` and `setgid` flags.
<figure><img src="image_9.png" /><figcaption>Figure 8: A TOCTOU attack</figcaption></figure>

We could detect the exiting of `invoke_syscall` by inserting a breakpoint at the address stored in the return address register (x30), but this greatly reduces the performance of our detection system. Instead, we check at the beginning of the next system call if the credentials have changed according to the rules of the last system call that have been made. This is illustrated by the ▲ sign in Figure 8. By doing so, we can have an average overhead of 55.5% in the mibench benchmark instead of 92.7% if we hooked the syscall ends to update the stored credentials.

Even then, an attacker could still modify the process credentials within the window between the hypervisor's initial and subsequent checks, illustrated by the red line in Figure 8. This results in a TOCTOU (Time Of Check vs. Time Of Use) vulnerability. Indeed, the `inode permission` function, which is used to check the process permission for various system calls (e.g., open or `faccessat`), will see the updated credentials that the hypervisor has not captured.

To protect against those attacks, we ensure that the corresponding fields of the `cred` and `task_struct` structure are not changed by the attacker while the kernel is using them. Since making those structures read-only adds a heavy overhead to the VM, we cannot protect them during the entire syscall. Instead, they are made read-only for the shortest time possible, only when the kernel uses them. Furthermore, since we only protect some fields of those structs, we use hardware watchpoints when available to detect writes on those fields. We implemented that protection for the `inode permission` function. Before calling this function, we check that the credentials have not been changed, and we make them read-only during the function call (see the ⊕ sign in Figure 8).

The overhead incurred by the detection of $RK_{cred}$ is 55.5% on average for the MiBench benchmark and 85% for the compilation of binutils. This overhead keeps the system usable for typical workloads and is acceptable for security-critical applications. As expected, the benchmarks that stress the file system exhibit a drastic overhead. Indeed, every interaction with inodes and process credentials involves permission checks.

The number of files opened by a process does not seem to increase the overhead. However, if several syscalls related to file systems are made simultaneously (e.g. a process issues an open syscall, is interrupted before the end of that syscall, then the newly scheduled process issues another open syscall), we may have to use page table permissions instead of hardware watchpoints to protect the credentials structure, since only 4 watchpoints are available at the same time for a given core, which worsens the performance.

# 7 Limitations and Future Work

**Interpreter vs. JIT Compiler.** Our system can detect various rootkits, including those that only tamper with dynamic objects,

but can only detect the latter by adding a large overhead (85% for binutils). This overhead is partly due to the fact that HYPERSEC programs are interpreted within XVisor rather than being compiled into native code. We could therefore reduce our overhead by compiling those programs when the hypervisor receives them. To emulate the performances of a JIT compiler, we implemented the detection of $RK_{cred}$ in C directly in XVisor and ran the benchmarks shown in Table 1. In that configuration, the overheads are more acceptable: only 15.1% on average for Mibench and 40% for compiling binutils. As a reminder, recall that Kaiser/KPTI patches introduced in Linux kernels in order to mitigate Meltdown attack induced massive overhead ranging from 1% to over 800% [19] depending on the number of syscalls per unit of times performed by applications. We believe that for mission critical applications in embedded systems a 15% slowdown can be tolerated. We are currently working on implementing a JIT compiler inside XVisor.

**Reboots.** First of all, detecting VM reboot is trivial within the hypervisor. We could easily add a new event to the language, so that programs can take into account VM reboot. They could for example, reset their internal state (maps) to reflect the fact that the VM has been restarted. Another technical obstacle that must be tackled is retrieving the loading address of the relevant kernel objects. Kernel objects are either dynamically allocated or statically allocated. Dynamically allocated objects already need to be retrieved by hand. Statically allocated objects are relocated between reboots only if KASLR [8] is enabled. Indeed in that case, kernel code and data are produced by the compilation chain to be position independent (mainly by using only relative jump and issuing memory access only using instruction pointer relative mode). The kernel is first loaded at a fixed address by the bootloader, and then mapped to random addresses by the `map_kernel` function. We could hook into that function and compute the real addresses of the concerned sections and derive the addresses of statically allocated kernel objects used by our programs.

**Kernel updates.** Kernel updates are more problematic since they may require sending new programs because structure layouts can radically change between two kernel versions or if the kernel is recompiled with the GCC randstruct plugin [10, 23], which randomizes structure layout. If the kernel structures have not changed too radically (e.g. between two minor consecutive versions of the same major version), only relying on symbolic names inside HYPERSEC programs could be sufficient to make the transition smooth enough to avoid reloading a new program. We could then only rely on embedded debugging information inside the kernel as done by EBPF. But if we need to load new programs to support either different kernel versions, or the detection of new attacks, we must support a robust method for kernel upgrades. A possible upgrading path is the following one: after kernel upgrades the VM must anyway be rebooted. We leverage the possibility offered by (most) firmware to programmatically change the boot options of the next reboot so that the VM is not restarted immediately, but rather an isolated (and hence safe) update VM is started instead. This update VM will be minimalist to reduce its attack surface, and its unique goal will be to load the latest version of the HYPERSEC programs that correspond to the upgraded kernel. The update VM will hence act as a proxy between the VM to be protected and the hypervisor. The hypervisor can then always trust the update VM, which is always
allowed to send new programs. Finally, the *update* VM ends up the
whole process by rebooting and transferring control back to the
upgraded VM.

**Support for multiple cores.** Our implementation currently only supports VM using a single core. To allow multi-core support, we could use different sets of page tables for stage 2 of the address translation process for each core. Although the current version of XVisor only treats the page tables on a per-VM basis instead of a per-core basis. Correctly implementing multi-core monitoring would require substantial implementation effort, but we do not foresee any conceptual issues.

**Attack mitigation** We are currently only detecting rootkits, but we could also mitigate some of them: attacks that only modify constant data like RK<sub>ro</sub> and RK<sub>vsf</sub> can be easily mitigated by discarding the incriminated memory writes. Rootkits that tamper with dynamic data (RK<sub>pid</sub> and RK<sub>cred</sub>) are harder to mitigate since they can change existing data structures but they could also move those resources and update the references pointing to them. This makes it harder for the hypervisor to restore them without writing in incorrect places of the VM memory.

8 Conclusion

This paper presents HYPERSEC, a dedicated language for adaptive rootkit detection in virtual machines. HYPERSEC enables virtual machines to autonomously generate detection programs tailored to their specific configurations, enhancing adaptability and security. Our prototype was implemented on a ZCU104 platform featuring an ARM Cortex-A53 processor, selected to meet constraints from ongoing project requirements<fnref n="4" />. Despite this platform’s modest virtualization performance compared to high-end Intel Xeon-based systems, our evaluation demonstrates the feasibility and potential of HYPERSEC for dynamic rootkit detection.

References

[1] Adore-ng 2024. Linux rootkit adapted for 2.6 and 3.x. https://github.com/yaoyu meng/adore-ng.

[2] Arm documentation 2024. Armv8-A Architecture Registers. https://developer arm.com/documentation/ddi0595/2020-12/AArch64-Registers/HFGITR-EL2-- Hypervisor-Fine-Grained-Instruction-Trap-Register.

[3] ARM Large System Extensions 2024. Introduction to Large System Extensions. https://learn.arm.com/learning-paths/servers-and-cloud-computing/lse/intro/.

[4] Binutils 2024. Source code. https://www.gnu.org/software/binutils/.

[5] Bitdefender 2023. Hypervisor Memory Introspection - Specification. https://hvmi.readthedocs.io/en/latest/index.html.

[6] btf 2024. BPF Type Format (BTF) — The Linux Kernel documentation. https://docs.kernel.org/bpf/btf.html.

[7] Jorge Buzeti. 2024. R3tr074/brokepkg: The LKM rootkit working in Linux Kernels 2.6.x/3.x/4.x/5.x. https://github.com/R3tr074/brokepkg.

[8] Claudio Canella, Michael Schwarz, Martin Haubenwallner, Martin Schwarzl, and Daniel Gruss. 2020. KASLR: Break It, Fix It, Repeat. In Proceedings of the 15th ACM Asia Conference on Computer and Communications Security (Taipei, Taiwan) (ASIA CCS '20). Association for Computing Machinery, New York, NY, USA, 481–493. https://doi.org/10.1145/3320269.3384747

[9] Cilium 2022. BPF and XDP Reference Guide. Cilium.

[10] Kees Cook. 2017. Introduce struct layout randomization plugin. https://lwn.net/ Articles/719732/.

[11] Emilio G Cota, Paolo Bonzini, Alex Bennée, and Luca P Carloni. 2017. Cross-ISA machine emulation for multicores. In 2017 IEEE/ACM International Symposium on Code Generation and Optimization (CGO). IEEE, Washington, D.C., USA, 210–220.

[12] CVE-2023-2163 2024. Vulnerability Details. https://www.cvedetails.com/cve/CVE-2023-2163/.

[13] Brendan Dolan-Gavitt, Tim Leek, Michael Zhivich, Jonathon Gifflin, and Wenke Lee. 2011. Virtuoso: Narrowing the Semantic Gap in Virtual Machine Introspection. In *2011 IEEE Symposium on Security and Privacy*. IEEE, Washington, D.C., USA, 297–312. https://doi.org/10.1109/SP.2011.11

[14] Dynamorio 2024. Exclusive Monitors. https://dynamorio.org/page_ldstex.html.

[15] Asaf Eitani. 2022. Detecting Drovorub's File Operations Hooking with Tracee. https://www.aquasec.com/blog/detect-drovorub-kernel-rootkit-attack-tracee/.

[16] Frida 16.1.3 Released 2023. A world-class dynamic instrumentation toolkit. https://frida.re/news/2023/07/14/frida-16-1-3-released/.

[17] Yangchun Fu and Zhiqiang Lin. 2012. Space traveling across vm: Automatically bridging the semantic gap in virtual machine introspection via online kernel data redirection. In *2012 IEEE symposium on security and privacy*. IEEE, USA, Washington, D.C., USA, 586–600.

[18] Tal Garfinkel and Mendel Rosenblum. 2003. A Virtual Machine Introspection Based Architecture for Intrusion Detection. In *Proceedings of the Network and Distributed System Security Symposium, NDSS 2003, San Diego, California, USA*. The Internet Society, 11710 Plaza America Drive, Suite 400, Reston, VA 20190, USA, 191–206. https://www.ndss-symposium.org/ndss2003/virtual-machine-introspection-based-architecture-intrusion-detection/.

[19] Brendan Gregg. 2018. KPTI/KAISER Meltdown Initial Performance Regressions. https://brendangregg.com/blog/2018-02-09/kpti-kaiser-meltdown-performanc e.html.

[20] M.R. Guthaus, J.S. Ringenberg, D. Ernst, T.M. Austin, T. Mudge, and R.B. Brown. 2001. MiBench: A free, commercially representative embedded benchmark suite. In *Proceedings of the Fourth Annual IEEE International Workshop on Workload Characterization*. WWC-4 (Cat. No.01EX538). IEEE, Washington, D.C., USA, 3–14.

[21] Gernot Heiser. 2008. The role of virtualization in embedded systems. In *Proceedings of the 1st Workshop on Isolation and Integration in Embedded Systems (Glasgow, Scotland) (IIES '08)*. Association for Computing Machinery, New York, NY, USA, 11–16. https://doi.org/10.1145/1435458.1435461

[22] Galen C Hunt and James R Larus. 2007. Singularity: rethinking the software stack. *ACM SIGOPS Operating Systems Review* 41, 2 (2007), 37–49.

[23] Nur Hussein. 2017. Randomizing structure layout. https://lwn.net/Articles/722 293/.

[24] Hypervisor-Enforced Kernel Integrity 2023. Hypervisor-Enforced Kernel Integrity (Heki) for KVM. https://lpc.events/event/17/contributions/1486/.

[25] Intel. 2016. *Intel® 64 and IA-32 Architectures Software Developer's Manual Volume 3C: System Programming Guide, Part 3*. Intel.

[26] Bhushan Jain, Mirza Basim Baig, Dongli Zhang, Donald E. Porter, and Radu Sion. 2014. SoK: Introspections on Trust and the Semantic Gap. In *2014 IEEE Symposium on Security and Privacy, SP 2014, Berkeley, CA, USA, May 18-21, 2014*. IEEE Computer Society, Washington, D.C., USA, 605–620. https://doi.org/10.110 9/SP.2014.45

[27] Daehee Jang, Hojoon Lee, Minsu Kim, Daehyeok Kim, Daegyeong Kim, and Brent Byunghoon Kang. 2014. Atra: Address translation redirection attack against hardware-based external monitors. In *Proceedings of the 2014 ACM SIGSAC Confer- ence on Computer and Communications Security*. ACM, New York, NY, USA, 167–178.

[28] Lauterbach GmbH 2024. Trace32: ARMv8-A/-R Debugger. Lauterbach GmbH. Support for exclusive monitor, page 157.

[29] Hojoon Lee, Hyungon Moon, Ingoo Heo, Daehee Jang, Jinsoo Jang, Kihwan Kim, Yunheung Paek, and Brent Byunghoon Kang. 2017. Ki-mon arm: A hardware-assisted event-triggered monitoring platform for mutable kernel object. *IEEE Transactions on Dependable and Secure Computing* 16, 2 (2017), 287–300.

[30] Linux kernel patching without LKM 2001. Linux on-the-fly kernel patching without LKM. http://phrack.org/issues/58/7.html.

[31] Linux Kernel Source 2022. lockdep.h. https://elixir bootlin.com/linux/v5.15.36/so urce/include/linux/lockdep.h#L535.

[32] Ilya V. Matveychikov. [n. d.]. KOPYCAT - Linux Kernel module-less implant (backdoor). https://github.com/milabs/kopycat.

[33] Jonathan M McCune, Yanlin Li, Ning Qu, Zongwei Zhou, Anupam Datta, Virgil Gligor, and Adrian Perrig. 2010. TrustVisor: Efficient TCB reduction and attestation. In *2010 IEEE Symposium on Security and Privacy*. IEEE, Washington, D.C., USA, 143–158.

[34] Victor Ramos Mello. 2024. Diamorphine: LKM rootkit for Linux Kernels 2.6.x/3.x/4.x/5.x/6.x (x86/x86_64 and ARM64). https://github.com/m0nad/Diamo rphine.

[35] James Morris, Mickaël Salaün, and Thara Gopinath. 2023. Linux Virtualization Based Security (LVBS). https://lpc.events/event/17/contributions/1515/.

[36] George C. Necula and Peter Lee. 1996. Safe kernel extensions without run-time checking. *SIGOPS Oper. Syst. Rev.* 30, SI (oct 1996), 229–243. https://doi.org/10.1 145/248155.238781

[37] Anup Patel, Mai Daftedar, Mohamed Shalan, and M Watheq El-Kharashi. 2015. Embedded hypervisor xvisor: A comparative analysis. In *2015 23rd Euromicro International Conference on Parallel, Distributed, and Network-Based Processing*. IEEE, IEEE, Washington, D.C., USA, 682–691.

[38] Bryan D. Payne, Martim Carbone, Monirul Sharif, and Wenke Lee. 2008. Lares: An Architecture for Secure Active Monitoring Using Virtualization. In *2008 IEEE*

<footnote n="4">⁴To maintain anonymity, these projects cannot be granted in this version.</footnote>
*Symposium on Security and Privacy (S&P 2008)*. IEEE, Washington, D.C., USA, 233–247.

[39] Joanna Rutkowska. 2006. Introducing Stealth Malware Taxonomy.

[40] Samsung. 2024. Real-time Kernel Protection. https://docs.samsungknox.com/admin/fundamentals/whitepaper/core-platform-security/real-time-kernel-protection/.

[41] scellator 2020. Linux Rootkits Part 3: A Backdoor to Root :: TheXcellerator. https://xcellerator.github.io/posts/linux_rootkits_03/.

[42] Arvind Seshadri, Mark Luk, Ning Qu, and Adrian Perrig. 2007. SecVisor: A tiny hypervisor to provide lifetime kernel code integrity for commodity OSes. In *Proceedings of twenty-first ACM SIGOPS symposium on Operating systems principles*. ACM, New York, NY, USA, 335–350.

[43] Valgring bug report 2017. Valgrind on arm64 violates the ARMv8 spec (ldxr/stxr). https://bugs.kde.org/show_bug.cgi?id=369459.

[44] Wavestone Cybersecurity and Digital Trust practice. 2022. EDRSandBlast. https://github.com/wavestone-cdt/EDRSandblast.

[45] Michael Wei and Nadav Amit. 2019. Leveraging Hyperupcalls To Bridge The Semantic Gap: An Application Perspective. *IEEE Data Eng. Bull.* 42, 1 (2019), 22–35.

[46] Florian Westphal, Stefan Axelsson, Christian Neuhaus, and Andreas Polze. 2014. VMI-PL: A monitoring language for virtual platforms using virtual machine introspection. *Digital Investigation* 11 (2014), S85–S94. Fourteenth Annual DFRWS Conference.

[47] Andrew Whitaker, Marianne Shaw, Steven D Gribble, et al. 2002. *Denali: Light-weight virtual machines for distributed and networked applications*. Technical Report. University of Washington.

[48] Jidong Xiao, Lei Lu, Haining Wang, and Xiaoyun Zhu. 2016. HyperLink: Virtual machine introspection and memory forensic analysis without kernel source code. In *2016 IEEE international conference on autonomic computing (ICAC)*. IEEE, Washington, D.C., USA, 127–136.

[49] xvisor 2024. XVisor hypervisor. https://github.com/xvisor/xvisor.

[50] Pavel Yosifovich, Mark E. Russinovich, David A. Solomon, and Alex Ionescu. 2017. *Windows Internals, Part 1: System architecture, processes, threads, memory management, and more (7th Edition)* (7th ed.). Microsoft Press, USA.

[51] Siqi Zhao, Xuhua Ding, Wen Xu, and Dawu Gu. 2017. Seeing through the same lens: introspecting guest address space at native speed. In *26th USENIX Security Symposium (USENIX Security 17)*. USENIX Association, Vancouver, BC, 799–813.

Received 6 December 2024