# CryptKSP: A Kernel Stack Protection Model Based on AES-NI Hardware Feature

Bingnan Zhong, Zicheng Wang, Yinggang Guo, Qingkai Zeng

► To cite this version:

Bingnan Zhong, Zicheng Wang, Yinggang Guo, Qingkai Zeng. CryptKSP: A Kernel Stack Protection Model Based on AES-NI Hardware Feature. 37th IFIP International Conference on ICT Systems Security and Privacy Protection (SEC), Jun 2022, Copenhagen, Denmark. pp.270-286, 10.1007/978-3-031-06975-8_16. hal-05319900

HAL Id: hal-05319900

https://inria.hal.science/hal-05319900v1

Submitted on 17 Oct 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# CryptKSP: A Kernel Stack Protection Model
Based on AES-NI Hardware Feature

Bingnan Zhong¹,²✉, Zicheng Wang¹,², Yinggang Guo¹,², and Qingkai Zeng¹,²

¹ State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China
{zbnnjucsv9,wzc,gyg}@smail.nju.edu.cn, zqk@nju.edu.cn

² Department of Computer Science and Technology, Nanjing University, Nanjing 210023, China

**Abstract.** The kernel stack is an important data structure in the kernel. It stores the return address, local variables and a large amount of state information of the kernel and application. Therefore, the kernel stack is often a valuable target for the attackers. More seriously, some types of attacks, which could bypass existing protection mechanisms, such as “return-to-schedule” rootkit, have posed a serious security threat to the security of the kernel stack. Therefore, some defensive approaches have been proposed to protect the integrity of the kernel stack by setting it read-only, however, it does not protect the confidentiality of the key data in the kernel stack. In this paper, we propose a kernel stack protection model for both the confidentiality and integrity without relying on another higher privilege layer. It takes advantage of AES-NI to encrypt the kernel stacks of the threads that are not running, and decrypt the kernel stack of the thread which will be running. In order to improve the security and adaptability, we implement two AES algorithms with different key lengths. The experimental results show that the kernel stack protection model could provide effective protection for the integrity and confidentiality of the kernel stack and do not impose high performance overhead.

**Keywords:** Kernel stack protection · AES-NI · Kernel stack integrity · Kernel stack confidentiality · Memory isolation

## 1 Introduction

The kernel stack is a critical data structure in the kernel, which stores important information, including control data (such as the return addresses) and non-control data (such as local variables). The kernel stack is located in the direct mapping area [1, 2] of the kernel address space. Once the kernel is compromised, any kernel execution unit [3] (which is defined in KSP [3], it refers to a piece of code with its own stack) can easily steal the sensitive information or destroy the crucial data stored in the kernel stack by leveraging KOH [4], DKOH [5] and return-to-schedule rootkit [6] attacks.

Since the kernel stack is an untyped large memory area and is not easy to traverse, it is difficult to protect the data in the kernel stack. Therefore, many security protection mechanisms have been proposed to protect the integrity of the kernel stack. KCOFI [7] proposes a protection method for the control data in the kernel stack, thus ensuring
the integrity of the kernel control flow. Fine-CFI [8] further implements fine-grained
protection of the return address of the kernel stack. Silhouette [9] builds a shadow
stack based on compiler technology, which effectively protects the return address of the
embedded system. Some works [10, 11, 12] use encryption to check integrity of software
to protect against computers virus. CCFI [13] uses cryptographic message authentication
codes (MACs) to protect control flow information such as function pointers, return
addresses, and vtable pointers.

Ret-to-sched is a serious threat to the return address in the kernel stack. The attacker
can change the control flow by destroying the return address in the kernel stack to point
to the rootkit. To address such threats, OSck [6] periodically checks that whether the
return address in the kernel stack belongs to a valid kernel code region. However, it is an
asynchronous event and only protects part not all data in the kernel stack. To protect the
integrity of the entire kernel stack, KSP [3] introduces a higher privilege layer, and sets
read-only protection on the kernel stack by clearing R/W bit in the corresponding page
table entries. Similarly, methods such as ShadowMoniter [14] and xMP [15] that use
hardware virtualization and EPT [16] to build a memory region protection mechanism
can also be applied to the integrity protection of the kernel stack. SaVioR [17] uses
randomization technology to protect the stack of application, and may be extended to
protect the kernel stack.

However, the above methods have the following problems:

1) Lack of confidentiality protection. It is easy to ensure the integrity of the kernel stack by taking use of read-only protection mechanism. However, only read-only protection will still put the protected data stored in kernel stack at risk, the attacker is still able to collect enough information before launching the attack.

2) Memory granularity conflict. On the x86_64 architectures, the default size of the kernel stack is 16 KB. However, the page size is 4 KB, 2 MB or 1 GB. Therefore, there is a serious memory granularity conflict between the kernel stack and the page. The kernel stack is often allocated in the kernel direct mapping area (excludes the kernel stack of the initialization process). In order to improve the memory access performance, the page size of the direct mapping memory regions is 2 MB or 1 GB, which is inconsistent with the size of the kernel stack.

3) Switching across privilege layers. Some methods introduce hypervisor, which is executed on a higher privilege than the kernel, as the trusted computing base of the system and deploy the interception and verification mechanism for the kernel stack in the hypervisor. It will be trapped into the hypervisor when the interception and verification mechanism is triggered, which will result in large performance loss. What's more, the current mainstream commercial hypervisors (including Xen [18] and KVM [19]) have a large amount of code and a complex architecture, which leads to the expansion of the TCB (Trusted Computing Base) size of the overall system.

Aiming at the above problems, we propose an integrity and confidentiality protection
model for the kernel stack based on AES-NI [20] hardware features, and implement a
prototype system CryptKSP. CryptKSP builds two different views for the kernel stack
of each thread or process (for the convenience of description, hereinafter collectively
referred to as thread) by means of encryption. One is normal view, where the kernel
stack is exposed to the kernel in plaintext. The other is an encrypted view, where the kernel stack is exposed to the kernel in ciphertext. When a thread is running normally, the corresponding kernel stack is the normal view, while when the thread is sleeping, the corresponding kernel stack is encrypted view. CryptKSP does not rely on higher privilege layers and read-only protection based on paging mechanism, and builds a trusted computing base (Intral-level kernel [21] or the same-privilege-layer trusted computing base [22], hereinafter referred to as the same-privilege-level TCB) at the same privilege level with the kernel. CryptKSP intercepts and verifies the key operations of the kernel stack, and implements the switching of different views of the kernel stack. To achieve fast view switching, CryptKSP implements the AES encryption algorithm by making use of AES-NI. In order to ensure the security of the cryptographic key, CryptKSP stores the cryptographic key in the DR register, and ensures that only the CryptKSP can access the DR register. CryptKSP configures different encryption keys for the kernel stack.

The contributions of this paper are as follows:

1) We implement a lightweight memory protection model to protect the confidentiality and integrity of the kernel stack based on the Nested Kernel [21] core architecture. It does not rely on higher privilege layers and does not need to maintain a large number of codes to support the corresponding virtualization functions, which effectively reduces the amount of code in the system and overall TCB size.

2) We use AES-NI feature to implement the encryption and decryption operations of the memory area of the kernel stack at the same privilege layer of the kernel. At the same time, we use the debug register to store the AES key and protect the AES key from being stolen or tampered by the compromised kernel.

3) We have implemented a simple prototype system of our model on Linux system and provided an evaluation for security and performance. The experiment results show that the model proposed in this paper can provide effective confidentiality and integrity protection for the kernel stack without incurring a large performance overhead.

The rest of this paper is organized as follows. Section 2 describes the background; Sect. 3 presents the threat model and assumptions; Sect. 4 and Sect. 5 describe the overall design and implementations of the CryptKSP prototype; Sect. 6 evaluates the security and performance of CryptKSP respectively; Sect. 7 presents the Related Work; Sect. 8 discuss the model and Sect. 9 describes the conclusion.

## 2 Background

**Intral-Kernel TCB.** EqualVisor [23] and Nested Kernel achieve a same-privilege-level TCB or intra-kernel TCB by leveraging the WP bit in the CR0 register and binary rewriting technology. When the kernel wants to operate the page table, it can only send a request to the same-privilege-level TCB. It eliminates the corresponding privileged instructions to prevent the untrusted kernel from destroying the memory protection. It can only make requests to the same-privilege-level TCB when the kernel wants to execute these privileged instructions. It is effective and efficient to deploy and implement kernel
protection mechanism to provide kernel security services in the same-privilege-level
TCB.

**AES-NI.** In order to increase the density of the AES encryption algorithm, Intel hardware chips have introduced a dedicated instruction set (Intel AES New Instructions, AES-NI for short). The AES-NI has six instructions to provide complete hardware support for AES. The instruction set mainly includes AESENC, AESENCLAST, AESDEC, AESDELAST, AESIMC and AESKEYGENASSIST. The AES instructions can flexibly support all encryption usages of AES, including standard key lengths (e.g. 128-bit key length), standard operating modes (e.g. ECB mode), and even some non-standard or future variants. AES-NI is turned on by default, and it can only be turned off by writing 11b to MSR_FEATURE_CONFIG MSR, but this operation will be invalid after system reset (restart).

**Vector Register.** In order to achieve media acceleration, Intel CPU provides vector instructions, including MMX [24], SSE [16], and AVX [25]. The registers of MMX are physically identical to floating point registers and are called MM registers. The SSE registers are called XMM registers, while the AVX registers are extensions of the XMM registers and are called YMM registers. The XMM register is the low 128-bit of the YMM register. On an x86_64 Intel CPU (such as Intel Haswell CPU), there are 16 64-bit GPRs, 8 64-bit MM registers, and 16 256-bit YMM registers.

**Debug Register.** To achieve hardware debug function, Intel hardware provides eight debug registers DR0-DR7. DR0-DR3 can be used to store the address of the debug point. DR6 is a status register and DR7 is a control register. DR4-DR5 are the aliases of DR6-DR7, respectively. In addition, the DR registers can only be accessed by privileged instructions, so the system can prevent the application being debugged from accessing these available registers [26].

## 3 Threat Model and Assumption

We assume that the kernel is untrusted, and the attacker can exploit the kernel vulnerabilities to attack the kernel or introduce untrusted kernel extensions. At present, according to the existing research, the attack methods against the kernel stack can be divided into two categories. Firstly, there are a large number of arbitrary read and write vulnerabilities in the kernel, and the attacker can use these vulnerabilities to arbitrarily read and write data in the kernel stack. Secondly, since main kernel and all kernel modules share the same address space, all kernel execution units [3] can access the address space of the kernel stack, and read and write data stored in the kernel stack arbitrarily. Since some existing research works [27-29] can effectively solve the first type of attack, therefore, we mainly consider the second type of attack.

CryptKSP ensures that the entire kernel boot process is trusted through the Trusted Boot hardware technologies (Intel TXT) [30]. As a module, CryptKSP will be loaded into the system during the kernel boot process and implement the corresponding initialization operation. We assume nested kernel architecture and the encryption and decryption
module in CryptKSP are trusted. Moreover, the amount of code of CryptKSP is small, and it is possible to verify its security through formal verification methods as seL4 [31] does. In addition, the hardware is assumed to be trusted, and the denial of service attack is out of scope in this paper.

# 4 System Design

## 4.1 Basic Framework of CryptKSP

Figure 1 shows the basic model of CryptKSP. The encryption module is deployed in the same-privilege-level TCB (Trusted Computing Base, or referred to as intra-kernel TCB). CryptKSP intercepts all kernel stack operations, including kernel stack allocation, kernel stack deletion and kernel stack switching. CryptKSP encrypts the kernel stack switched out and decrypt the kernel stack switched in. As far as I know, KSP is currently the main method to protect kernel stack integrity based on hypervisor. Figure 2 shows the basic framework of hypervisor-based kernel stack integrity protection approach, which introduces a higher privilege layer and maintains the shadow page table in the hypervisor. KSP protects the kernel stack by setting kernel stack read-only. The hypervisor in the KSP intercepts the all kernel stack operations, and operates the corresponding page table entries to enable or disable read-only protection for kernel stack.

<figure><img src="image_3.png" /><figcaption>Fig. 1. CryptKSP architecture</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>Fig. 2. KSP basic architecture</figcaption></figure>

**Security Requirements and Challenges.** Combined with the security purpose of the basic framework of CryptKSP and the security attributions of the same-privilege-level TCB described by NestedKernel, EqualVisor and other works, we will describe the basic security requirements and the challenges of CryptKSP. The security requirements are as follows:

1) Security of the same-privilege-level TCB. CryptKSP should meet the three basic security attributes of the same-privilege-level TCB [22], including code and data integrity, control flow integrity and fixed entry points.

2) Interception and verification of kernel stack operation. CryptKSP should ensure that it can effectively intercept key operations of the kernel stack, as well as privileged operations in the kernel, and ensure the security of inserted hook points.

3) Cryptographic Key Protection. CryptKSP should ensure the security of the encryption module and the privacy and integrity of the cryptographic key.
When we achieve to protect the integrity and confidentiality of the kernel stack
effectively and efficiently, we still need to solve the following challenges:

1) CryptKSP must make sure that the memory region of CryptKSP is isolated from the kernel. And CryptKSP must ensure that it can effectively intercept key operations of the kernel stack and.

2) CryptKSP needs to leverage the AES-NI hardware feature to build the encryption module in same-privilege-level TCB. CryptKSP should solve the problem to store the AES key and provide different cryptographic key for each thread.

The basic architecture of CryptKSP is consistent with Nested Kernel and EqualVisor, which both use the WP hardware mechanism in the CR0 register to implement the same-privilege-level TCB. Therefore, CryptKSP uses the WP hardware mechanism to build the same-privilege-level TCB in the same way, and intercepts and verifies the key operations of the kernel stack, and constructs different views for the kernel stack through the encryption module. CryptKSP uses debug register to store cryptographic key.

## 4.2 Interception and Verification of Kernel Stack Operation

The key operations of the kernel stack mainly include the kernel stack allocation, kernel stack deletion and the kernel stack switching. The kernel allocates a unique kernel stack to each thread, and maintains a one-to-one correspondence between threads and kernel stacks.

The structure of the kernel stack is shown in Fig. 3, which is a *union* data structure, including a *thread_info* and a 16 KB or 8 KB memory area. The default kernel stack size in x86 system is 8 KB (two page frames), and 16 KB for x86_64 systems. In our experiment, we use the x86_64 architectures, since the default kernel stack size is 16 KB in the rest of the paper. The bottom part of the stack is the *thread_info* data structure, while the top of the stack is area used to store the return address and local variables. The system allocates a data structure *task_struct* for each thread, and the *stack* field in the *task_struct* data structure points to the data structure *thread_info*.

CryptKSP can intercept key events in the kernel through Secure Gate [32]. To intercept the sensitive operations of the kernel stack, CryptKSP inserts appropriate hooks in the kernel. Once the hooks are triggered, the control flow is transferred to the CryptKSP. Then, CryptKSP can encrypt or decrypt part of the memory area where the kernel stack is located. At the same time, CryptKSP maintains two linked lists, one is used to maintain the base address pointer of the kernel stack (the address of the *thread_info* structure, hereafter named kernel stack address), the current macro shown in Fig. 3 can be used to obtain the kernel stack address. The other is used to maintain the stack pointer of the corresponding kernel stack (as the *rsp* in *thread_structure*, hereafter named kernel stack pointer). CryptKSP sets read-only protection for the above two linked lists. Next, this paper will detail the interception and verification of key operations of the kernel stack in CryptKSP.
<figure><img src="image_5.png" /><figcaption>Fig. 3. Kernel stack framework</figcaption></figure>

**Kernel Stack Creation.** During process creation, the system allocates a kernel stack for each thread, and initializes the *thread_info* and *task_struct* structure, which are mutually linked by means of the fields *task* and *stack*, respectively. Therefore, CryptKSP inserts the appropriate hooks in the function *dup_task_-struct*. The control flow will transfer to the CryptKSP after the thread has been created and initialized. At this time, the top of the stack does not load any execution information for the kernel. Therefore, CryptKSP only encrypts the memory area of bottom of the kernel stack, and constructs an encrypted view of the kernel stack. After the newly created kernel stack is initialized, the *thread_info* data structure at the bottom of the stack contains sensitive information such as the address of the thread’s *task_struct* structure. Then CryptKSP update the newly created kernel stack address in the linked list of kernel stack addresses and kernel stack pointer in the linked list of kernel stack pointers.

**Kernel Stack Deletion.** When a thread exits, the kernel releases all the resources it occupies, including the memory resources occupied by the kernel stack. CryptKSP intercepts the kernel stack deletion operation, but does not need to decrypt the memory area of kernel stack. At this time, the data in the memory region of the kernel stack is in the state of ciphertext, and it will not affect the management and utilization of the physical memory page. Of course, there is also a way to clear the memory area of the kernel stack before the kernel stack is released. In addition, CryptKSP will delete the released kernel stack from the two linked lists of the kernel stack address and the kernel stack pointer.

**Kernel Stack Switching.** During thread switch, the scheduler will select a thread being activated and a thread being replaced. Again, prev thread is referred to as the thread being replaced and the next thread is referred to as the thread being activated.

When performing the thread switch, the scheduler needs to switch the kernel stack and the hardware context, which concludes all the information needed by the new thread being running on CPU. Even after switching the kernel stack, the scheduler still restores some more information for the thread being replaced. Therefore, CryptKSP inserts the hooks in the function *context_switch*. CryptKSP can decrypt the kernel stack of a thread being activated before reloading the kernel stack pointer into the *rsp* register. Before
performing the decryption operation, CryptKSP will verify whether the value of rsp field in the *task_struct* structure of the next thread is legal.

CryptKSP encrypts the kernel stack being replaced at the end of the function *con-text_switch*. This means that there is a small attack surface during thread context switching, because the kernel stacks of two threads in the kernel are in the state of plaintext. The kernel execution unit that performs the context switch may be exploited by the compromised kernel to attack the kernel stack of the next thread. In order to eliminate the security threat here, CryptKSP ensures that the execution of the thread context switch is atomic. Firstly, CryptKSP applies the $W \oplus X$ mechanism to protect the integrity of kernel code. Secondly, CryptKSP ensures that the thread switch cannot be interrupted through the shadow IDT technology used by EqualVisor and NestedKernel.

**Integrity Protection for Kernel Stack Pointer.** An attacker can modify the kernel stack pointer maintained by the kernel and forge a kernel stack for a thread. To defend against such attacks, CryptKSP maintains a copy of the kernel stack pointer for each thread in the kernel stack pointer linked list. When the kernel performs the thread switch, CryptKSP can intercept the kernel stack switch and immediately check its legality. That is to say, CryptKSP can check that whether the stack pointer of the thread being activated is equal to the copy of kernel stack pointer maintained by the CryptKSP. If it is not equal, it means the stack pointer of the thread being activated has been tampered by the attacker. Therefore, CryptKSP stops thread switching and reports a serious warning. On the contrary, CryptKSP then decrypt the kernel stack of next thread and encrypt the kernel stack of prev thread. Afterwards, CryptKSP can update the kernel stack pointer linked list maintained in the CryptKSP for the next thread and prev thread.

**Anti-bypass Interception for Kernel Stack Operation.** CryptKSP applies the $W \oplus X$ mechanism to protect the integrity of the kernel code, then the hooks inserted into the kernel code cannot be destroyed. In addition, AES algorithm is public encryption algorithm, therefore, the secret key is most important for the system. CryptKSP is the only module which maintains the secret key in the kernel. Then, the kernel must send the encryption or decryption request to the CryptKSP when the kernel wants to select a sleeping thread to run. CryptKSP protects the secret key from damaging or disclosing. Then the attacker could not directly access the kernel stack protected by CryptKSP.

## 4.3 AES for Kernel Stack

CryptKSP uses the AES-NI hardware mechanism to implement the AES encryption algorithm instead of software, which can effectively improve the performance of the encryption algorithm. Since the data in the kernel are all shaped data, there may be frequent occurrences of shaped data with certain characteristics, such as zero. Therefore, in order to meet the security requirements in different scenarios, CryptKSP configures different cryptographic keys for each kernel stack. In addition, CryptKSP implements two AES algorithms with different key lengths, AES-128 and AES-256, in CBC mode. CryptKSP-128 means CryptKSP using AES-128 algorithm, and CryptKSP-256 means CryptKSP using AES-256 algorithm.

The key part of the encryption module in CryptKSP is cryptographic key generation, storage and protection.
**Cryptographic Key Storage and Creation.** CryptKSP adopts a similar method in PT_Rand [33], using the debug register to store critical information. CryptKSP only uses the DR2 and DR3 to store cryptographic key for AES. The debug registers are all 64 bits in the x86_64 architectures, and the values stored in the DR2 and DR3 registers can construct a 128-bit key for the AES-128 algorithm. Then, CryptKSP does not occupy all the registers, and the unoccupied DR0 and DR1 registers can still be used to support the hardware debugging function.

CryptKSP maintains two secret keys for each kernel stack, one is basic secret key and another is seed secret key. The basic secret key is only stored in debug registers and seed secret key is really used in AES algorithm. The seed secret key is generated by using the basic secret key and kernel stack address. CryptKSP can configure a different seed secret key for each kernel stack. CryptKSP uses the thread's kernel stack address (the address of *thread_info* structure) and the value stored in the DR2 and DR3 register to calculate the seed secret key for each thread. Because the address of kernel stack of each process is unique, CryptKSP can create a different seed secret key for each kernel stack. And the research shows that even if one bit of the secret key is different, about 50% of the bits in the ciphertext will change without changing the plaintext.

Although we can configure different seed secret key used in AES-128 for each kernel stack, the differences between the different seed secret keys are small. To increase the obfuscate of the seed secret key, we use the SHA-256 cryptographic hash algorithm to produce an output of fixed length, which is used as AES-256 key. The length of output of SHA-256 is 256 bits. In order to calculate the 256-bit seed secret key quickly, CryptKSP-256 uses the kernel stack base address and value stored in DR3 register (referred to as DR3_value) to construct a 512-bit basic block, which is used as the input of SHA-256.CryptKSP first uses the kernel stack base address and DR3_value to construct the upper 384-bit data of the block, and the last 64 bits of the basic packet are used to store the initial length of the message. In Sect. 5.2, we detail how to construct the basic block by using the DR3_value and kernel stack address.

In order to improve the security of the AES algorithm, CryptKSP uses the YMM register to store the round keys. Then, it not only improves the performance of encryption, but also avoids exposing the secret key directly to memory. And after encryption or decryption operations have been finished, CryptKSP directly clears the YMM register that stores the round key.

XMM and other registers are directly used for encryption and decryption function in the kernel, which may result in affecting the normal running of applications that need to use these registers, CryptKSP uses the *kernel_fpu_end* and *kernel_fpu_begin* functions in the kernel to save and restore FPU hardware context.

**Protection for Secret Key.** Although storing the secret key in the debug register can effectively avoid exposing the secret key to the memory, the untrusted kernel can steal and modify the secret key stored in debug register through MOV-FROM/TO-DR0-DR7 instruction. In order to ensure the integrity and confidentiality of the basic cryptographic key stored in the debug register, we eliminate the MOV-FROM/TO-DR0-DR7 instructions in the kernel and only maintains MOV-FROM/TO-DR0-DR7 in the CryptKSP. Therefore, when the kernel wants to access the DR register, it must send the corresponding request to the CryptKSP. Then, CryptKSP can verify all the request and decide
whether the request is legal. If the request is to access the DR register used to store the
basic secret key, CryptKSP considers the request is illegal and does nothing. Thence, the
attacker could not damage or disclose the critical information stored the DR registers.

The variables and memory areas that are used to store secret keys temporarily are
cleared after the encryption or decryption has been finished. Meanwhile, such memory
areas are set as non-cacheable.

**Encryption/Decryption for Kernel Stack.** In the x86_64 systems, the default size of the kernel stack is 16 KB. As shown in Fig. 3, only the *thread_info* data structure at the bottom of the kernel stack and the memory areas pointed to by rsp register at the top of the kernel stack are used. In most cases, there are large memory areas in the middle of the kernel stack are not used. To improve system performance, CryptKSP just only encrypts the *thread_info* data structure and the memory region pointed by the rsp register at the top of the kernel stack. The length of packet in AES algorithm is 16 bytes, however, the length of the *thread_info* structure is not aligned on a 16-byte.

Therefore, CryptKSP makes sure that the memory areas of *thread_info* data structure
is aligned on a 16-byte when encrypting the *thread_info* data structure. Then, CryptKSP
verifies the size of the memory area used at the top of the kernel stack. There is no need
to encrypt or decrypt the memory area at the top of the kernel stack when the size of the
memory area used at the top of the stack is 0. The *push* or *pop* operation of the kernel
stack is aligned on an 8-byte, the address of memory areas at the top of the kernel stack
should be moved forward 8 bytes from the current address when the size of the memory
area used at the top of the stack is aligned on a 16-byte. The size of the memory area also
needs to be increased by 8 bytes. It avoids that the encrypted memory areas go beyond
scope of the kernel stack.

# 5 Implementation

## 5.1 Loading and Initialization of CryptKSP

To ensure the boot process of the kernel is trusted, trusted boot technology of Intel
hardware [30] is used in the kernel boot process. CryptKSP is initialized during the
kernel boot process. Therefore, the initialization process of CryptKSP is also trusted.

CryptKSP initializes the hardware context of the entire system, and the address space
of the entire kernel page table. At the same time, CryptKSP will initialize the settings of
shadow IDT and IOMMU, as well as the corresponding registers (including CR0, CR4,
IA32_EFER, IA32_FMASK, IA32_FEATURE_CONFIG, IDTR and other registers).
Then, CryptKSP implements the initialization work related to page table protection,
deploys the W ⊕ X protection mechanism in the system, and starts to intercept and verify
the page table update operation through the para-virtualized interface, and realizes the
same-prvilege-level TCB with the kernel. The privileged instructions that need to be
eliminated in the kernel are described in Table 1.
<table><caption>**Table 1.** Privileged instructions that require interception and verification.</caption><thead><tr><td>Privileged instructions to be eliminated</td><td>Privileged instructions Os</td></tr></thead><tbody><tr><td>MOV_TO_CR0</td><td>0x 0f 22/1</td></tr><tr><td>MOV_TO_CR3</td><td>0x 0f 22/3</td></tr><tr><td>MOV_TO_CR4</td><td>0x 0f 22/4</td></tr><tr><td>lidt</td><td>0x 0f 01/3</td></tr><tr><td>lgdt</td><td>0x 0f 01/2</td></tr><tr><td>wrmsr</td><td>0x 0f 30</td></tr><tr><td>MOV_TO_DR0-DR7</td><td>0x 0f 21/r</td></tr><tr><td>MOV_FROM_DR0-DR7</td><td>0x 0f 23/r</td></tr></tbody></table>

## 5.2 Implementation of Encryption and Decryption Based on AES-NI

CryptKSP uses the AES-NI hardware mechanism to implement AES encryption algorithms with different cryptographic key lengths, such as 128-bit and 256-bit, which are called CryptKSP-128 and CryptKSP-256 respectively. We implement the encryption module in CryptKSP based on the source code provided by MemSentry [34]. However, MemSentry just realizes the AES-128 for the applications. In this article, we migrate it to the kernel layer. In addition, we extend it to AES-256 and apply it to CryptKSP-256. As MemSentry, both CryptKSP-128 and CryptKSP-256 use YMM registers to store the round keys, which not only improves the performance of encryption, but also does not expose the round keys to memory.

We prepare two secret keys in CryptKSP, one is basic secret key stored in debug register, and another is seed secret key which is really used in the AES algorithm. The seed secret key is calculated from the valued in debug register and the corresponding kernel stack address.

CryptKSP-128 performs a bit wise operation on the basic secret key stored in the DR3 register and the corresponding kernel stack address to ensure that each kernel stack has a different seed secret key. However, at the same time, the lower 14-bit of the basic secret key is retained. “!” means bitwise OR operation, “&” means bitwise AND operation, “< <” means bitwise left shift operation.

The pseudo calculation formula of the calculation is as follows:

$$ High\_seed\_key = (DR3\_value \& 0x3fff) | (DR3\_value \& task\_struct - > stack) \quad (1) $$

The seed secret key consists of upper 64-bit *High_seed_key* and lower 64-bit *Low_seed_key*. The *task_struct- > stack* is the kernel stack address and the address of *thread_info* data structure.

In order to avoid the situation that two kernel stacks have the same seed secret key, CryptKSP sets all bit-15 to bit-42 of the *DR3_value* to 1. Therefore, the bit-15 to bit-42 of the *High_seed_key* value is determined by the value of the stack pointer in the
`task_struct` data structure. The core difference of the kernel stack address of each thread is bit-15 to bit-42.

The secret key in CryptKSP-256 is calculated by hash function. The core of the calculation is to hash the basic secret key stored in the DR3 register and the kernel stack address. The pseudo calculation formula of the calculation is as follows:

$$ \text{Seed\_Key} = \text{SHA256}((\text{task\_struct-} > \text{stack} << 448) | (\text{DR3\_value} << 384) | (\text{task\_struct-} > \text{stack} << 320)|(\text{DR3\_value} << 256)|(\text{DR3\_value} << 192) | (\text{DR3\_value} << 128)|(1 << 64)|(\text{value\_length})) $$

(2)

Among them, DR3_value is the basic secret key stored in the DR3 register, and task_struct- > stack is the kernel stack address allocated by the kernel for each thread. The value_length is length of the original message. Seed_Key is 256-bit data. Task_struct- > stack, DR3_value and value_length are all 64-bit data.

## 5.3 Interception and Verification of the Kernel Stack Operation

CryptKSP draws on the Secure Gate designed in SIM [32] and Nested Kernel, and inserts appropriate Secure Gate in the process of kernel stack creation, kernel deletion and kernel stack switching in the kernel.

CryptKSP maintain the copy of kernel stack address and current stack pointer for each kernel stack. CryptKSP will verify the legality of the kernel stack address and kernel stack pointer when intercept the kernel stack operations. Meanwhile, the IST mechanism is turned off during the system implementation to ensure that all interrupts use the same kernel stack.

# 6 Experimental Evaluation and Analysis

We will analyze the kernel stack model CryptKSP from two aspects: security and performance. The hardware environment used in this experiment is Intel Core i7-9700, 3.00 GHz, memory 8 GB DDR3, 1TB SATA hard disk. The operating system in the experiment is Ubuntu14.04 64-bit LTS, the kernel version is Linux 4.2.1, and the gcc version is 7.4.0.

## 6.1 Security Evaluation

### 1) Secret Key Security

In CryptKSP, the cryptographic key consists of two parts, one is the basic secret key stored in the DR3 register, and the other is the seed secret key actually used in the AES encryption algorithm. The relationship between the basic secret key and the seed secret key in CryptKSP-128 and CryptKSP-256 is described in detail in Sect. 5.3. The seed secret key in CryptKSP must be calculated by combining the basic secret key and kernel stack address.

CryptKSP eliminates all MOV-FROM/TO-DR0-DR7 instructions in the kernel by making use of the same method in Nested Kernel and EqualVisor, and only CryptKSP
has corresponding instructions. When the kernel wants to access the DR register, it must send the request to the CryptKSP. Therefore, the untrusted kernel cannot tamper with the basic secret key to destroy the encryption module in CryptKSP, nor can steal the basic secret key maintained by CryptKSP through the MOV-FROM-DR0-DR7 privileged instruction. Since the kernel stack address of the thread stored in the process data structure *task_struct* is visible to the kernel, once the basic secret key is leaked, the attacker may deduce the seed secret key of thread. However, the attacker could not obtain the basic secret key protected by CryptKSP, so it is very difficult to deduce the seed secret key of the thread.

## 2) Security Analysis of Protecting the Kernel Stack

**Confidentiality.** When a thread is in a sleeping state, its corresponding kernel stack is in ciphertext state. The attacker can read any data in the kernel stack of any other threads, but the contents are all ciphertext, so the attacker cannot steal any effective information. When the thread is scheduled, CryptKSP will verify the validity of kernel stack address and kernel stack pointer of the thread. Then, CryptKSP decrypted the kernel stack and load the kernel stack pointer stored in *task_struct* structure into rsp register when the verification is passed. As described in the Sect. 3, there are various techniques to protect the integrity of the kernel stack of the currently running process at this time, therefore, it is orthogonal to this paper.

**Integrity.** The attacker may try to modify the ciphertext information in the other kernel stacks. When the sleeping thread is scheduled to be executed, the data in the kernel stack may be invalid after decryption, which will eventually lead to a failure or error when the system is running. CryptKSP maintains the kernel stack address and kernel stack pointer of all threads and verifies its legality during the interception of kernel stack operations. Meanwhile, CryptKSP updates the maintained kernel stack address and kernel stack pointer synchronously. It can effectively prevent attackers from destroying the integrity of the kernel stack address and kernel stack pointer.

## 6.2 Performance Evaluation

To evaluate the performance overhead of CryptKSP. We have implemented three kinds of kernel stack protection model. Based on the basic idea of Nested Kernel and some implementation technologies in EqualVisor [23], we realize a simple model Nested-KSP, which protects the integrity of kernel stack by sets read-only protection for the kernel stack at the same-privilege-level TCB. We have implemented two kinds of AES encryption algorithm in CryptKSP, one is 128-bit cryptographic key for AES (referred to as AES-128) and another is 256-bit cryptographic key for AES (referred to as AES-256). The one using AES-128 in CryptKSP is called CryptKSP-128 and another using AES-256 in CryptKSP is called CryptKSP-256.

From the following two aspects, we will discuss the impact on performance. To measure the overhead different kernel operations, we adopt lmbench benchmarking suite [35] (v.3.0-a9). Table 2 shows the average experimental results based on three identical
<table><caption>**Table 2.** Latency on process-related kernel operation (in µs, smaller is better).</caption><thead><tr><th>Benchmarks</th><th>Native</th><th>NestedKSP</th><th>CryptKSP-128</th><th>CryptKSP-256</th><th>NestedKSP slowdown</th><th>CryptKSP-128 slowdown</th><th>CryptKSP-256 slowdown</th></tr></thead><tbody><tr><td>Null call</td><td>0.05</td><td>0.05</td><td>0.05</td><td>0.05</td><td>0.0%</td><td>0.0%</td><td>0.0%</td></tr><tr><td>Null I/O</td><td>0.12</td><td>0.12</td><td>0.12</td><td>0.12</td><td>0.0%</td><td>0.0%</td><td>0.0%</td></tr><tr><td>Stat</td><td>0.37</td><td>0.37</td><td>0.37</td><td>0.38</td><td>0.0%</td><td>0.0%</td><td>2.7%</td></tr><tr><td>Open/close</td><td>0.80</td><td>0.81</td><td>0.80</td><td>0.80</td><td>1.3%</td><td>0.0%</td><td>0.0%</td></tr><tr><td>Sig inst</td><td>0.13</td><td>0.14</td><td>0.14</td><td>0.14</td><td>7.7%</td><td>7.7%</td><td>7.7%</td></tr><tr><td>Sig hndl</td><td>0.80</td><td>0.84</td><td>0.83</td><td>0.83</td><td>5.0%</td><td>3.7%</td><td>3.7%</td></tr><tr><td>Fork+exit</td><td>68.63</td><td>155.90</td><td>155.57</td><td>159.23</td><td>127.2%</td><td>126.7%</td><td>132.0%</td></tr><tr><td>Fork+exec</td><td>228.00</td><td>415.70</td><td>414.93</td><td>420.53</td><td>82.3%</td><td>82.0%</td><td>84.4%</td></tr><tr><td>Fork+/bin/sh</td><td>546.40</td><td>906.00</td><td>904.77</td><td>916.27</td><td>65.8%</td><td>65.6%</td><td>67.7%</td></tr><tr><td>Protection fault</td><td>0.379</td><td>0.547</td><td>0.547</td><td>0.545</td><td>44.3%</td><td>44.3%</td><td>43.8%</td></tr><tr><td>Page fault</td><td>0.0330</td><td>0.1273</td><td>0.1278</td><td>0.1273</td><td>285.8%</td><td>287.3%</td><td>285.8%</td></tr></tbody></table>

runs. The overhead for foremost microbencmarks are basically the same. That is to say, CryptKSP cannot introduce high performance lost for foremost test cases.

Table 3 shows the corresponding communication bandwidth, the results presents the communication bandwidth of Native Kernel, NestedKSP, CryptKSP-128 and CryptKSP-256. For most benchmark, the results of CryptKSP-128 and CryptKSP-256 is close to that of NestedKSP. However, for CryptKSP the results of the benchmark Pipe, AF Unix and TCP are smaller than that of NestedKSP. What's more, the CryptKSP-128 is closer to the native kernel than the CryptKSP-256.

<table><caption>**Table 3.** Local-communication bandwidths (in MB/s, bigger is better)</caption><thead><tr><th>Benchmarks</th><th>Native</th><th>NestedKSP</th><th>CryptKSP-128</th><th>CryptKSP-256</th><th>NestedKSP Ratio</th><th>CryptKSP-128 Ratio</th><th>CryptKSP-256 Ratio</th></tr></thead><tbody><tr><td>Pipe</td><td>6681.67</td><td>5058.33</td><td>4169</td><td>3035.33</td><td>75.7%</td><td>62.4%</td><td>45.4%</td></tr><tr><td>AF UNIX</td><td>7983.67</td><td>7046.33</td><td>6268.67</td><td>5238.33</td><td>88.3%</td><td>78.5%</td><td>65.6%</td></tr><tr><td>TCP</td><td>6820</td><td>6301.33</td><td>5826.33</td><td>5199.33</td><td>92.4%</td><td>85.4%</td><td>76.2%</td></tr><tr><td>File reread</td><td>7818.63</td><td>7722.27</td><td>7796.73</td><td>7804.87</td><td>98.8%</td><td>99.7%</td><td>99.8%</td></tr><tr><td>Mmap</td><td>13.7K</td><td>13.7K</td><td>13.7K</td><td>13.7K</td><td>100.0%</td><td>100.0%</td><td>100.0%</td></tr><tr><td>Bcopy/libc)</td><td>8025.8</td><td>7934.20</td><td>7925.03</td><td>7951.87</td><td>98.9%</td><td>98.7%</td><td>99.1%</td></tr><tr><td>Bcopy的手</td><td>5786.67</td><td>5762.43</td><td>5765.30</td><td>5741.83</td><td>99.6%</td><td>99.6%</td><td>99.2%</td></tr><tr><td>Mem read</td><td>12K</td><td>12K</td><td>12K</td><td>12K</td><td>100.0%</td><td>100.0%</td><td>100.0%</td></tr><tr><td>Mem write</td><td>8022</td><td>8002.67</td><td>7976</td><td>7996.67</td><td>99.8%</td><td>99.4%</td><td>99.7%</td></tr></tbody></table>
# 7 Related Work

**Hypervisor.** OSck [6] protects the data in the kernel stack, but only part of the data is protected. KSP [3] uses virtualization and shadow page tables to set the entire kernel stack read-only. However, the read-only protection mechanism cannot effectively protect privacy of key information in the kernel stack. The attacker can still access relevant information. In addition, there is a conflict between the size of the stack and the granularity of the pages in the direct mapping area, which increases the overhead of memory access and consumes more memory resources. CryptKSP can not only protect the confidentiality and integrity of the entire kernel stack, but also does not rely on operating page tables to protect the kernel stack, so there is no memory granularity conflict.

**Hardware Virtualization.** ShadowMonitor [14] and xMP [15] utilize EPT to create memory isolation and data protection, which can also be applied to protect the kernel stack. However, they all rely on hardware virtualization support and need to trap into root mode when intercepting kernel stack operations, because the underlying EPT cannot be manipulated in non-root mode. The AES-NI hardware features are a widely supported hardware feature. It is also a typical method to use the WP bit in the CR0 register to construct the same-privilege-level TCB.

**Encryption.** CCFI [13] also uses AES-NI to encrypt the return address in the stack, which can also be applied to the return address in the kernel stack. However, it relies on modifying the compiler, which will result in higher overhead. CryptKSP do not need to modify the compiler.

# 8 Discussion

The SHA algorithm in CryptKSP is implemented by software. We will explore to realize the SHA-256 hash function with the help of the SHA-NI hardware feature to improve the performance. We just only consider the security of the kernel stack, but there is also a lot of kernel state information in the interrupt stack. Then, it is valuable for us to continue to discuss the protection for the interrupt stack in the future research.

# 9 Conclusion

In this paper, we propose a kernel stack integrity and confidentiality protection model using the AES-NI hardware feature at the same hardware privilege level with the kernel and have implemented the prototype system CryptKSP. Compared with other related kernel stack protection schemes, CryptKSP does not rely on higher privilege layer and effectively eliminates the performance lost caused by cross-privilege layer switching. Moreover, it has obtained better security for the kernel stack.

**Acknowledgments.** This work has been partly supported by National NSF of China under Grant No. 61772266, 61431008.
## References

1. Love, R.: Linux Kernel Development. Pearson Education, London (2010)

2. Bovet, D.P., Cesati, M.: Understanding the Linux Kernel: From I/O Ports to Process Management. O'Reilly Media Inc., Newton (2005)

3. Liu, W., Luo, S., Liu, Y., et al.: A kernel stack protection model against attacks from kernel execution units. Comput. Secur. 72, 96-106 (2018)

4. KOH: Kernel object hooking rootkits (koh rootkits), March 2016. http://www rootkit com/newsread ph p?newsid=501

5. Butler, J., Hoglund, G.: Vice - catch the hookers! in Black Hat USA 2004 (2004)

6. Hofmann, O.S., Dunn, A.M., Kim, S., Roy, I., Witchel, E.: Ensuring operating system kernel integrity with OSck. In: 16th International Conference on Architectural Support for Programming Languages and Operating Systems (2011)

7. Criswell, J., Dautenhahn, N., Adve, V.: KCofI: complete control-flow integrity for commodity operating system kernels. In: IEEE Symposium on Security and Privacy, pp. 292-307. IEEE (2014)

8. Ge, X., Talele, N., Payer, M., et al.: Fine-grained control-flow integrity for kernel software. In: IEEE European Symposium on Security and Privacy (EuroS&P), pp. 179-194. IEEE (2016)

9. Zhou, J., Du, Y., Shen, Z., Ma, L., Criswell, J., Walls, R.J.: Silhouette: efficient protected shadow stacks for embedded systems. In: USENIX Security Symposium 2020, pp. 1219-1236 (2020)

10. Popek, G.J., Kampe, M., Kline, C.S., Stoughton, A., Urban, M., Walton, E.J.: UCLA secure Unix. In: Proceedings of the AFIPS National Computer Conference, vol. 48, pp. 355-364 (1979)

11. Pozzo, M.M., Gray, T.E.: An approach to containing computer viruses. Comput. Secur. 6(4), 321-331 (1987)

12. Davida, G.L., Desmedt, Y., Matt, B.J.: Defending systems against viruses through cryptographic authentication. In: IEEE Symposium on Security and Privacy, pp. 312-318 (1989)

13. Mashtizadeh, A.J., Bittau, A., Boneh, D., Mazières, D.: CCFI: Cryptographically enforced control flow integrity. In: CCS. ACM(2015)

14. Shi, B., Cui, L., Li, B., Liu, X.D., Hao, Z.Y., Shen, H.Y.: ShadowMonitor: an effective in-VM monitoring framework with hardware- enforced isolation. In: Bailey, M., Holz, T., Stamatogiannakis, M., Ioannidis, S. (eds.) Research in Attacks, Intrusions, and Defenses, vol. 11050, pp. 670-690. Springer, Berlin (2018). https://doi.org/10.1007/978-3-030-00470-5_31

15. Proskurin, S., Momeu, M., Ghavamnia, S., Kemerlis, V.P., Polychronakis, M.: xMP: selective memory protection for kernel and user space. In: Proceedings of the IEEE Symposium on Security and Privacy (S&P), pp. 233-247. IEEE (2020)

16. Intel Corporation. Intel 64 and IA-32 Architectures Software Developer's Manual. Intel Corporation (2019)

17. Lee, S., Kang, H., Jang, J., Kang, B.B.: SaVioR: thwarting stack-based memory safety violations by randomizing stack lay out. IEEE Trans. Dependable Secur. Comput. (2021)

18. Barham, P., et al.: Xen and the art of virtualization. In: Proceedings of the 19th ACM Symposium on Operating System Principles (SOSP), pp. 164-167. ACM (2003)

19. Kivity, A., Kamay, Y., Laor, D., Lublin, U., Liguori, A.: KVM: the Linux virtual machine monitor. In: Proceedings of the Linux Symposium, OLS, Ottawa, pp. 225-230 (2007)

20. Gueron, S.: Intel Advanced Encryption Standard (AES) New Instructions Set. Number 323641-001. Intel Corporation, May 2010
21. Dautenhahn, N., Kasampalis, T., Dietz, W., et al.: Nested kernel: an operating system architecture for intra-kernel privilege separation. In: Proceedings of the 20th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS). ACM (2015)

22. Deng, L.: Study on System Security Techniques in an Untrusted Kernel Environment Ph.D. Thesis. Nanjing University, Nanjing (2018). (in Chinese with English abstract)

23. Deng, L., Zeng, Q.K., Wang, W.G., Liu, Y.: EqualVisor: providing memory protection in an untrusted commodity hypervisor. In: Proceedings of the 13th IEEE International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom), pp. 300–309. IEEE (2014)

24. DuLong, C., Gutman, M., Julier, M., et al.: Complete Guide to MMX Technology. McGraw-Hill Professional, New York (1997)

25. Lomont, C.: Introduction to Intel advanced vector extensions. Intel White Paper (2011)

26. Müller, T., Freiling, F.C., Dewald, A.: TRESOR runs encryption securely outside RAM. In: Proceedings of 20th USENIX Security Symposium (2011)

27. Li, J., Wang, Z., Jiang, X., Grace, M., Bahram, S.: Defeating return-oriented rootkits with "return-less" kernels. In: EuroSys, pp. 195–208 (2010)

28. Onarlioglu, K., Bilge, L., Lanzi, A., Balzarotti, D., Kirda, E.: G-free: defeating return-oriented programming through gadget-less binaries. In: ACSAC, pp. 49–58 (2010)

29. Wartell, R., Mohan, V., Hamlen, K.W., Lin, Z.: Binary stirring: selfrandomizing instruction addresses of legacy x86 binary code. In: CCS, pp. 157–168 (2012)

30. Intel. Intel Trusted Execution Technology Preliminary Architecture Specification (2006)

31. Klein, G., et al.: seL4: formal verification of an OS kernel. In: Proceedings of the 22nd ACM Symposium on Operating Systems Principles, pp. 207–220. ACM (2009)

32. Sharif, M.I., Lee, W., Cui, W., Lanzi, A.: Secure in-VM monitoring using hardware virtualization. In: 16th ACM Conference on Computer and Communications Security (CCS), pp. 477–487. ACM (2009)

33. Davi, L., Gens, D., Liebchen, C., Sadeghi, A.R.: PT-Rand: practical mitigation of data-only attacks against page tables. In: Proceedings of 24th Annual Network and Distributed System Security Symposium, NDSS 2017 (2017)

34. Koning, K., Chen, X., Bos, H., Giuffrida, C., Athanasopoulos, E.: No need to hide: protecting safe regions on commodity hardware. In: Proceedings of the 12th European Conf. on Computer Systems (EuroSys), pp. 437–452. ACM (2017)

35. McVoy, L., Staelin, C.: Imbench: portable tools for performance analysis. In: Proceedings of USENIX Annual Technical Conference (1996)