# PaCaR: Improved Buffered I/O Locality on NUMA Systems with Page Cache Replication

Jérôme Coquisart, Julien Sopena, Redha Gouicem

<figure><img src="image_1.png" /></figure>

## To cite this version:

Jérôme Coquisart, Julien Sopena, Redha Gouicem. PaCaR: Improved Buffered I/O Locality on NUMA Systems with Page Cache Replication. EuroSys 2026 - European Conference on Computer Systems, ACM, Apr 2026, Edinburgh, Scotland, United Kingdom. 10.1145/3767295.3769359. hal-05411611

HAL Id: hal-05411611

https://hal.science/hal-05411611v1

Submitted on 11 Dec 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution - ShareAlike 4.0 International License
# PACAR: Improved Buffered I/O Locality on NUMA Systems with Page Cache Replication

Jérôme Coquisart

Julien Sopena

Redha Gouicem

RWTH Aachen University

Sorbonne Université, LIP6

RWTH Aachen University

Germany

France

Germany

## Abstract

Modern systems featuring multiple sockets suffer from the non-uniformity of performance of their memory accesses. Accessing memory of a remote node can lead to doubled latencies and halved bandwidth. This is particularly critical for I/O-intensive applications relying on the Linux page cache, where locality is paramount. Existing solutions migrating threads or memory across nodes fail to address the issue with highly parallel workloads, where a small working set is constantly accessed by multiple threads across nodes.

We present PACAR, a novel page cache replication mechanism that enhances locality by transparently replicating cached pages across Non-Uniform Memory Access (NUMA) nodes. PACAR ensures write consistency, adapts to memory pressure, and operates seamlessly without requiring application modifications. Evaluations on a dual socket NUMA server demonstrate up to 1.4x performance improvements in synthetic workloads and up to 25% gains in real-world workloads, showcasing the effectiveness of PACAR in optimizing NUMA-aware buffered I/O operations.

**CCS Concepts: Software and its engineering → Memory management; Software performance.**

**Keywords:** NUMA, Memory Management, Page Cache, Replication, Data Locality, I/O Performance

## ACM Reference Format

Jérôme Coquisart, Julien Sopena, and Redha Gouicem. 2026. PACAR: Improved Buffered I/O Locality on NUMA Systems with Page Cache Replication. In *21st European Conference on Computer Systems (EUROSYS '26), April 27–30, 2026, Edinburgh, Scotland Uk*. ACM, New York, NY, USA, 17 pages. https://doi.org/10.1145/3767295.3769359

## 1 Introduction

Modern applications are increasingly I/O-intensive, driven by the growing demand for data processing in fields such as machine learning, databases, and large-scale web services. These applications rely heavily on efficient file I/O operations

to achieve high performance. To mitigate the inherent latency of disk accesses, operating systems usually implement I/O caching mechanisms that buffer file data in memory, significantly improving access times and throughput, e.g., the Linux page cache. This caching layer is particularly crucial for workloads that involve frequent read and write operations, as it reduces the need for costly disk operations and helps maintain high application performance.

However, contemporary data centers predominantly use NUMA multi-socket servers. The resulting non-uniform memory access and cross-socket effects challenge the assumptions and performance of the traditional page cache. In NUMA systems, memory access times vary depending on whether the memory is local to the CPU or resides on a remote node, i.e., another socket. Remote memory accesses can incur doubled latencies and halved bandwidth, as shown in prior studies [25]. The Linux kernel community has actively worked on improving the support for NUMA architectures over the last decade. Since 2013, there have been at least 123 commits related to NUMA every year, with a peak in 2024 with 242 commits. In particular, automatic memory migration and thread placement have been introduced to improve data locality [10]. However, these mechanisms only target anonymous memory and rarely apply to the page cache. Consequently, I/O-intensive applications that access the page cache often suffer from degraded performance due to remote memory accesses, particularly in highly parallel workloads where threads on different nodes share the same working set.

To address this gap, we propose PACAR, a novel page cache replication mechanism that enhances data locality in NUMA systems. PACAR transparently replicates cached pages across NUMA nodes, ensuring that threads can access data locally without incurring the penalties of remote memory accesses. Our design is guided by three key goals: improving performance by minimizing remote accesses, addressing the limitations of existing NUMA balancing techniques, and ensuring transparency for applications by requiring no modifications to their code. Additionally, PACAR integrates seamlessly with the kernel's memory management subsystem, adapting to memory pressure and maintaining system stability.

Our contributions are as follows:

1. We design and implement PACAR, a transparent page cache replication mechanism that enhances locality for buffered I/O operations on NUMA systems.

<figure><img src="image_3.png" /></figure>

This work is licensed under a Creative Commons Attribution 4.0 International License.

EUROSYS '26, Edinburgh, Scotland Uk

© 2026 Copyright held by the owner/author(s).

ACM ISBN 979-8-4007-2212-7/26/04

https://doi.org/10.1145/3767295.3769359
2. We ensure write consistency across replicated pages and introduce mechanisms to adapt to memory pressure, preserving system performance under varying workloads.

3. We comprehensively evaluate PACAR on a dual-socket NUMA server, demonstrating up to 1.4× speedup in micro-benchmarks and up to 25% performance gains in real-world workloads, including YCSB, web servers, video streaming applications, and RocksDB.

4. We provide insights into the trade-offs between replication and memory footprint, highlighting the benefits of our approach for I/O-intensive and multithreaded applications.

The remainder of this paper is organized as follows. Section 2 provides background on NUMA systems and the Linux page cache. Section 3 motivates the need for page cache replication by analyzing the limitations of existing approaches. Section 4 presents the design of PACAR, followed by implementation details in Section 5. Section 6 evaluates the performance of PACAR across synthetic and real-world workloads. We discuss the implications and limitations of our approach in Section 7, review related work in Section 8, and conclude in Section 9.

# 2 Background

## 2.1 Non-Uniform Memory Access Systems

A Non-Uniform Memory Access (NUMA) system is composed of multiple *nodes*, each integrating one or more CPU sockets and directly attached memory modules. These nodes are interconnected via a high-speed bus, allowing any CPU to access any memory location in the system. However, memory accesses are not uniform: accessing local memory is significantly faster and offers higher bandwidth than accessing remote memory, which can double the latency and halve the bandwidth [25]. As a result, data locality is a critical determinant of application performance, especially for memory- and I/O-intensive workloads.

To mitigate the performance penalties of remote accesses, modern operating systems implement NUMA-aware memory management and scheduling. The Linux kernel detects the NUMA topology and exposes it to both kernel and user space. By default, Linux uses a first-touch allocation policy, placing memory on the node where it is first accessed. Developers can further optimize locality using APIs and tools such as *numactl* to pin threads and memory to specific nodes. The kernel also features automatic NUMA balancing [10], which monitors memory access patterns and migrates threads or memory pages to improve locality. Despite these mechanisms, achieving optimal locality remains challenging, particularly for highly parallel workloads where threads on different nodes frequently access shared data. In such cases, the system must balance locality with parallelism [6, 8, 20].

## 2.2 Memory management

**Page cache.** Persistent storage remains orders of magnitude slower than DRAM, making disk I/O a major bottleneck for data-intensive applications. To bridge this gap, Linux employs a page cache, transparently buffering file data in memory to accelerate subsequent accesses. The page cache is central to the performance of I/O-bound workloads, as it absorbs repeated reads and writes, reducing the frequency of slow disk operations. The default allocation policy for the page cache is first-touch: when a thread fetches data from disk, the corresponding page is allocated on the same NUMA node as the thread.

Applications interact with the page cache through two primary interfaces: memory mapped I/Os, through *mmaped* files, and buffered I/Os, through the POSIX *read* and *write* system call families. Both mechanisms trigger the loading of file data into the page cache upon access. For *mmap*, the kernel maintains reference counts to track which pages are mapped into user address spaces.

Another important characteristic of the page is its opportunistic use of free memory. Most OSes, and Linux in particular, do not allocate a pool of memory for the page cache. Instead, the page cache dynamically uses all the available free memory of the system. Under memory pressure, such as when applications explicitly allocate additional memory, the Linux kernel reclaims memory from both the page cache and anonymous memory. Historically, the kernel preferred reclaiming page cache memory over anonymous memory; however, recent changes to the Least Recently Used (LRU) algorithm have altered this behavior [1, 11].

**Physical page management.** Linux organizes physical memory into page frames, typically 4 KB in size. Each page frame is represented by a *struct page* that contains metadata such as mapping information, reference counts, and state flags. To improve efficiency and reduce management overhead, recent kernels introduced *folios*, which group together contiguous pages frames (in powers of two) that share metadata and state. Folios enable the kernel to manage memory at a coarser granularity, reducing the cost of tracking and manipulating individual pages. Importantly, folios reuse the existing *struct page*, i.e., one per folio, and do not require additional memory allocation.

The page cache stores folios within an *xarray*, a scalable data structure optimized for both dense and sparse indices. The xarray replaces older hash maps and radix tree implementations, providing efficient growth, traversal, and lookup for large numbers of cached objects.

**Page frame reclamation algorithm.** To maintain system stability under memory pressure, Linux employs a page frame reclamation algorithm based on per-node LRU lists. Both page cache folios and anonymous memory (such as application stacks and heaps) are tracked on these lists. When
free memory falls below a threshold, i.e., a watermark, the kernel evicts the least recently used folios from the LRU, freeing page frames for new allocations.

Since Linux 6.1, the page frame reclamation algorithm has been enhanced to use multi-generation LRU lists [13]. LRU lists are divided into multiple generations, improving the estimation of the age of a page and its likelihood of being accessed again. The LRU lists are maintained independently on each NUMA node, meaning that reclamation decisions are made locally. This design can inadvertently evict hot data from the cache, especially in NUMA systems where access patterns are non-uniform and shared across nodes.

## 3 Motivation

Modern NUMA systems offer tremendous memory bandwidth and capacity, but their non-uniform access characteristics introduce new challenges for system designers and application developers. In this section, we highlight four key challenges that limit the performance and scalability of I/O-intensive workloads on NUMA servers, motivating the need for new approaches to page cache management.

**Cost of Remote Accesses.** Despite advances in NUMA interconnects, remote memory accesses remain significantly slower than local accesses. As shown in Table 1, recent server-class CPUs still exhibit 60–190% higher latency and up to 80% lower bandwidth for remote memory. This performance gap is fundamental and persists across hardware generations. For data-intensive applications, even a modest fraction of remote accesses can lead to substantial throughput degradation and increased tail latencies, especially under high concurrency. On RocksDB, we observe 14% bandwidth performance penalty when the page cache is located on a distant node.

**Lack of Page Cache NUMA Locality.** The Linux page cache is a critical component for accelerating file I/O, but it is largely oblivious to NUMA topology. When applications use the *read* and *write* APIs, cached pages are allocated on the NUMA node of the first accessing thread, but subsequent accesses from other nodes incur remote memory penalties. Existing NUMA balancing [27] mechanisms in Linux focus on anonymous and memory-mapped pages, leaving the page cache unmanaged. Manual pinning of threads and memory can mitigate remote accesses, but this approach is brittle, does not scale, and often leads to resource underutilization or contention on a single node.

**Challenges with Shared Data and Parallelism.** Many modern workloads, such as databases and analytics engines, involve multiple threads distributed across NUMA nodes accessing shared data structures. In these scenarios, traditional heuristics for data and thread placement are insufficient: a single piece of hot data may be accessed concurrently from many nodes, making it impossible to achieve locality for all

threads [31]. Our experiments with RocksDB, for example, show that even with NUMA balancing and memory mapping enabled, over 50% of page cache accesses remain remote, and attempts to concentrate threads and memory on a single node result in severe bandwidth bottlenecks. This highlights the limitations of spatial placement strategies in the presence of shared, highly contended data.

**Underutilized Memory Resources.** Large NUMA servers and virtualized environments often provision memory conservatively to handle peak loads, resulting in substantial periods where significant memory remains idle [14, 17, 19, 32–35]. While memory overcommitment enables responsiveness to demand spikes, it also means that valuable DRAM resources are frequently underutilized. We observe that this idle memory could be leveraged to improve data locality and system throughput, rather than remaining unused.

Taken together, these challenges reveal a fundamental gap in current operating system support for NUMA-aware page cache management. Addressing the cost of remote accesses, the lack of cache locality, the difficulty of managing shared data, and the opportunity presented by idle memory motivates our proposal for page cache replication—a mechanism that transparently improves locality and performance for I/O-intensive, multithreaded workloads on NUMA systems.

## 4 Design

In this section, we describe the design of PACAR, our transparent page cache replication mechanism for NUMA systems. We first present the design goals that guided our approach, then provide an overview of the architecture, and finally detail the core mechanisms that enable efficient, consistent, and transparent replication of page cache data across NUMA nodes.

### 4.1 Design goals

PACAR is designed to address the fundamental limitations of page cache management on NUMA servers. Our design is guided by four primary objectives: maximizing performance for I/O-intensive workloads, overcoming the inherent challenges of data locality in parallel environments, ensuring full transparency for applications, and preserving robust memory management.

**Improving performance.** Our primary goal is to maximize the performance of buffered I/O by minimizing the cost and frequency of remote memory accesses. As shown in Section 3, remote accesses to the page cache can severely degrade throughput and increase tail latency, especially for highly parallel, read-heavy workloads. PACAR aims to ensure that the majority of page cache accesses are satisfied locally, reducing contention on the interconnect and improving both bandwidth and latency for demanding applications.
<table><caption>**Table 1.** Memory latency and bandwidth measurements on various NUMA servers</caption><thead><tr><th colspan="2">Processor</th><th colspan="3">Latency (ns)</th><th colspan="3">Bandwidth (GB/s)</th></tr><tr><th>Model</th><th>Release date</th><th>Local</th><th>Distant</th><th>Penalty</th><th>Local</th><th>Distant</th><th>Penalty</th></tr></thead><tbody><tr><td>2 x Intel Xeon Gold 5320</td><td>2023</td><td>94.6</td><td>162.3</td><td>+71.6%</td><td>122.7</td><td>55.5</td><td>-54.7%</td></tr><tr><td>2 x Intel Xeon Silver 4410Y</td><td>2023</td><td>109.1</td><td>181.7</td><td>+66.5%</td><td>141.4</td><td>63.7</td><td>-54.9%</td></tr><tr><td>2 x Intel Xeon Silver 4314</td><td>2022</td><td>89.3</td><td>152.5</td><td>+70.8%</td><td>131.5</td><td>34.5</td><td>-73.8%</td></tr><tr><td>2 x AMD EPYC 7301</td><td>2018</td><td>91.2</td><td>267.6</td><td>+193%</td><td>17.3</td><td>7.4</td><td>-57.4%</td></tr><tr><td>4 x Intel Xeon Gold 6130</td><td>2017</td><td>87.4</td><td>146.9</td><td>+68.1%</td><td>104.0</td><td>17.2</td><td>-83.4%</td></tr><tr><td>2 x Intel Xeon Gold 6130</td><td>2017</td><td>89.6</td><td>143.3</td><td>+59.9%</td><td>109.8</td><td>34.4</td><td>-68.6%</td></tr><tr><td>2 x Intel Xeon E5-2630</td><td>2012</td><td>80.8</td><td>131.9</td><td>+63.2%</td><td>38.6</td><td>18.6</td><td>-51.8%</td></tr></tbody></table>

**Navigating locality challenges.** Existing NUMA locality mechanisms, such as thread or memory migration, are ill-suited for workloads where many threads on different nodes concurrently access shared data. Migration can introduce high overheads, disrupt cache locality, and ultimately fail to provide locality for all threads. PACAR avoids these pitfalls by introducing selective, on-demand replication, enabling each node to maintain local copies of hot data and thereby addressing the scalability and contention issues of centralized placement strategies.

**Transparency.** A guiding principle of our design is transparency: PACAR requires no changes to application code or user workflows. Replication is triggered automatically and adapts to workload characteristics, ensuring that applications benefit from improved locality without explicit intervention. Furthermore, PACAR is designed to avoid introducing overhead for workloads that do not benefit from replication, such as write-heavy or single-threaded applications.

**Preserving memory management.** Replication inevitably increases the memory footprint of the page cache. To maintain system stability and fairness, PACAR integrates tightly with the kernel's memory management and reclamation subsystems. Replicated pages are subject to the same eviction policies as regular cache pages, and replication is automatically throttled or disabled under memory pressure to avoid exacerbating resource contention.

## 4.2 Design overview

PACAR is a page cache replication mechanism designed to enhance the locality of buffered I/O operations on NUMA systems. It operates at the kernel level, specifically within the Linux kernel's page cache subsystem. Figure 1 provides an overview of PACAR and how it is integrated into the kernel's page cache and memory management subsystems.

The main idea of PACAR is to replicate pages from the page cache in every NUMA node that uses them. Whenever a page is accessed through the read system call (1), we check if the page is located on the same NUMA node as the thread

accessing it. If access is local, we proceed as usual, and the page is copied to the application. If the page is located on a different NUMA node, we trigger a replication of the page on the local node, and copy the newly allocated page to the application. From now on, we name the original page *main* and the replica *twin*. We keep track of the mappings between *main* and *twin* pages (4) in order to return the proper page on every read operation.

With the introduction of replicas, we now need to ensure that the data remains consistent across all of them. If a program writes to a page in the page cache (2), all subsequent reads to that page, regardless of whether they are local or remote, should return the same data. To achieve this, we centralize all write operations to the *main* page, and invalidate all *twin* pages. This allows us to have a single source of truth for the data. If the modified page is read remotely again, it will trigger our replication mechanism on the updated page. Note that in the case of frequent write operations, we provide a mechanism to switch which replica is the *main* one. This mechanism is crucial to avoid pathological cases where a node repeatedly writes to a remote page despite having a local replica, thus reducing unnecessary interconnect traffic.

Providing replication and consistency allows PACAR reaching our goals of performance and transparency without relying on thread or memory migration techniques. We still need to preserve the kernel's memory management to maintain the system's stability and performance (5). To achieve this, we integrate PACAR into the memory reclamation subsystem, in particular with the LRU list. This allows us to take advantage of existing page reclamation mechanisms and avoid disrupting applications that need to allocate memory. Our replicated pages are treated as any page in the page cache in that sense: they get evicted if they are not frequently accessed.

## 4.3 Detailed design

**Replicate-on-read mechanism.** The main focus of our design is to enhance the locality of reads from the page cache. To do so, we introduce a duplicate on read mechanism that ensures that every page read from the page cache is
<figure><img src="image_4.png" /><figcaption>**Figure 1.** Overview of PACAR, our page cache replication mechanism.</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>**Figure 2.** Flowchart for Read and Write Routines</figcaption></figure>

local to the thread accessing it. By improving page cache locality, we reduce the access latency and the contention on the interconnect, and improve bandwidth.

To illustrate the page replication process, we refer to Figure 2, in particular to the read routine. When a thread performs a read operation, i.e., when it calls the read system call, we check if the page is local to the thread's NUMA node. We track this information with metadata associated with

each main page in the page cache, that contains information about the page's twins, i.e., its replicas. In particular, we store the address of the twin associated with each NUMA node as well as their status (valid or invalid). If the accessed main page is local, we use the regular code path and just copy the page to the application. If the accessed main page is on a remote NUMA node, we check whether a local twin already exists and is valid. In that case, we copy it to the application, else, we initiate the replication process.

The replication process consists of allocating a new page on the local NUMA node, and copying the data from the main page to this new twin page. We perform the copy in a synchronous manner, which means that the read operation will block until the replication is complete, and copy the local twin page to the application. We also explored asynchronous replication early on. While it hides initial access latency, our analysis suggests that for I/O-intensive workloads with high page re-access rates (our primary target), the overhead of managing asynchronous state and the second (delayed) remote access outweigh the benefits. We prefer synchronous replication, as it avoids the overhead of performing two separate distant reads.

Note that replication can fail if the local NUMA node is out of memory. In this case, we fall back to the regular code path, and copy the remote page to the application. We prefer this solution to freeing local memory, as this could have adverse effects on the performance of the system by freeing currently used memory.

**Write operations and replica consistency.** When pages are replicated, we need to ensure that the data remains consistent across all replicas. If a thread writes to its local replica, we need to guarantee that threads located on other nodes can
observe the write operation consistently. The general idea is to centralize all write operations to one single authoritative page, *main*, and invalidate all *twins*. Figure 2 illustrates the write routine on the right side.

When a write operation is performed, we first check if this page has replicas in our metadata. If no replica exists, i.e., the page was never replicated, we just proceed with the write operation as usual. If one or more replicas exist, we invalidate all the *twins* in our metadata and perform the write operation on the *main* page. Subsequent accesses to an invalid *twin* page will trigger an update of that page, by copying the data from the *main* page. Thus, all read operations on a *twin* page are always consistent with the associated *main* page.

Note that when invalidating a *twin* page, we do not invalidate it from the LRU or free the page. The benefit of this lazy invalidation is twofold: (i) if the *twin* page is accessed again, we do not need to allocate a new page, and simply need to synchronize the *twin* with its *main* page, and (ii) we avoid the cost of dropping the page from the LRU and freeing it, and let the page reclamation algorithm do it in the future if memory needs to be reclaimed.

In addition to ensuring the consistency of replicas, we also want to minimize the cost of write operations by improving their locality. We do so by switching the *main* page to the NUMA node that performs most of the write operations on this page. We measure the amount of consecutive remote writes to a *main* page, and if it exceeds a threshold, we make the *twin* from the NUMA node that performed the most writes the new *main* page. We then proceed with the invalidation of *twins* and perform the write to the new *main*. This switch is efficient, as it only requires updating the page reference in the page cache and our metadata.

We deliberately choose not to propagate writes synchronously to all twin copies. In scenarios where a thread performs frequent writes to a page that is infrequently read, synchronously updating all *twins* could overload the interconnect and introduce unnecessary memory contention. Our approach avoids this issue by updating *twin* copies only when they are actively being read.

**Memory evictions and memory pressure.** As explained in Section 2.2, the page cache does not use a pre-allocated pool of memory. Instead, it uses available free memory as a cache for storage devices, and relies on the memory reclamation subsystem and its LRU to make its pages available when necessary. Since we are replicating pages from the page cache, our replicas should follow the same policy.

To this end, PACAR integrates with the kernel's memory reclamation subsystem. In particular, when we create a replica of a *main* page, we insert it into the LRU list of its local NUMA node. This allows us to rely on existing mechanism from the kernel to measure the usage of our replicas, in the same way as any other page in the system. If a replica is not used for a while, and memory needs to be reclaimed,

it will be evicted from the LRU and freed. When a replica is evicted, we also update the metadata of its *main* page by removing its reference. This is necessary to avoid an evicted *twin* from being accessed in the future after being freed.

An alternate design would have been to prioritize the eviction of *twins* over the other pages in memory. However, this would have required us to implement a custom eviction policy or modify the LRU to prioritize our replicas. This would make our system more complex, but also prevent replicas from staying in memory when frequently used. Indeed, if two replicas are much more frequently accessed than another page in memory (from the page cache or not), it is better for the overall system performance to evict the less used page, and keep the replicas in memory.

Additionally, we want PACAR to preserve the behavior of the memory management subsystem, without adding unnecessary overhead. In particular, when the system suffers from memory pressure, we disable our replication mechanism to avoid adding to the existing pressure, which would in turn lead to more memory reclamation. We rely on Linux's watermark mechanism to detect memory pressure and pause our replication mechanism until the system is back to normal.

**Page pre-migration.** An interesting aspect to note with PACAR is how it offers a pre-migration mechanism for free. When a page from the page cache is initially allocated, it is placed on the NUMA node where the first access occurs. If all subsequent accesses happen on another node, the page will be replicated, and the original page, the *main* page, will be evicted by the page reclamation algorithm. The end result of this scenario is the exact same one as with traditional page migration, as implemented in Linux's NUMA balancing mechanism. However, in contrast to NUMA balancing, we perform the migration ahead of time, and do not wait to detect that a page should be migrated to do so. This is akin to a form of prefetching, but for page migration on NUMA systems. This pre-migration also proves beneficial when the page keeps being reused on multiple nodes, as it avoids the continuous ping-pong effect page migration can have.

# 5 Implementation

We implement PACAR on the latest Linux Long-Term Support (LTS) kernel as of this writing, version 6.12. We modify multiple subsystems of the kernel to introduce consistent page replication without breaking existing functionality. In particular, we modify the generic POSIX file API of the Virtual File System (VFS) and the memory reclamation and compaction code. We also thoroughly test our implementation to ensure it does not introduce any regressions or performance issues. In this section, we detail our implementation of PACAR. Our changes include 44 modified files in the Linux kernel, mainly in the memory and file system subsystems, totalling 1860 insertions and 131 deletions.
## 5.1 Replicate on read mechanism

PACAR triggers page replication when a page from the page cache is read from a remote NUMA node. Thus, we need to detect such occurrences of buffered I/O operations. We do this by hooking our replication code into the read system call, `sys_read`. During this system call, the kernel fetches the target pages from the page cache's `xarray` and copies the read data from these pages back to a user space buffer. To minimize the overhead of this operation, the system call batches a set of pages before copying back the data to user space. We use this opportunity to iterate over the pages in the batch and check if they are located on a local or a remote node. For each page, if it is local, we do nothing particular. If it is distant, we replace it with its local `twin` when available, and let the kernel copy them to user space. If no `twin` has been created yet, we initiate the replication process.

**Replication metadata.** We track the pages that have been replicated with a new kernel structure called *struct duplication*, which we associate to each *main* page. This structure holds a set of metadata for each *twin* page, including the address of the *twin*, a counter of remote write operations, a reference counter and a spinlock. Since we have at most one replica per NUMA node, the size of our metadata set is the number of NUMA nodes in the system.

To associate the *struct duplication* to *main* pages, we need to maintain a reference to it in the *struct page* of the *main* page. However, there is one *struct page* for each page frame, i.e., physical page, in the system. This means that adding a pointer to our metadata in this structure would have a non-negligible impact on the memory footprint of the system. For instance, on a 512 GB machine, adding such a pointer would increase the memory footprint of the system by 1 GB. To prevent increasing the size of the *struct page*, we add the pointer to our structure as a union with the *private* field of the *struct page*, as it is rarely used for pages in the page cache. In case this field is used, we store a copy in our *struct duplication*, and patch all accesses to the private field with a generic accessor that uses our *struct duplication* when necessary. We choose this specific field because its usage is largely confined to file systems, minimizing its impact on other kernel subsystems, which in turn simplifies debugging and maintenance.

We limit the memory overhead of our replication mechanism by making it as small as possible, with only 32 bytes per structure, per NUMA node. For every additional NUMA node, the structure size increases by 8 bytes. Furthermore, this structure can be shared by multiple pages when folios are used. Indeed, we only associate one *struct duplication* to a folio, which can potentially contain hundreds of pages.

## 5.2 Handling of writes and consistency

Handling consistency is key in a cache with replicated pages. We ensure consistency by ignoring memory-mapped I/Os

and performing proper page invalidation and update upon writes.

**Memory-mapped I/Os.** PACAR only supports buffered I/Os performed through the *read* and *write* system calls. We do not support memory-mapped I/Os, e.g., *mmaped* files, as they are accessed through regular memory operations, which makes read and write operations difficult to track without significant overhead, e.g., by triggering a page fault on every access. On the contrary, buffered I/Os are easy to intercept, as they are performed through well-defined system calls. To avoid replicating a mapped page, we check the *mapcount* of the page, which counts the number of references to the page in the page tables. We only replicate pages with a *mapcount* of 0, meaning that the page is not mapped anywhere, and mmap have their sets of drawbacks. According to Crotty et al. [12], performance may suffer due to the inability to manage memory explicitly or control page faults effectively. Additionally, mmap is ill-suited for high throughput demands on fast persistent storage devices, and scenarios where precise error handling is crucial. A more detailed discussion of mmap support is provided in Section 7.

**Consistency algorithm.** We implement the consistency algorithm described in Section 4.3, where all write operations are performed on the *main* page, while all *twin* pages are invalidated. Invalidating a *twin* is a lightweight operation, as it only consists in clearing the *uptodate* flag of the page in its *struct page*. We do not explicitly free the page, as it might be reused in the near future, and can be reclaimed by the page reclamation algorithm in case of memory pressure.

**Switching the main page.** Since we are focusing all write operations on the *main* page, we want to ensure that we are minimizing remote accesses. To this end, we count the amount of consecutive write operations to a page performed from each remote NUMA node, with no read operation in between. If one of these counters exceeds a predefined threshold of 10 consecutive writes, we make the *twin* located on that node the new *main*. Note that the threshold can be modified at run time through a sysfs interface.

Switching the *main* consists in freezing the references on both the *main* and *twin* pages, to ensure that we are the only users of these pages. We can then trigger a page migration from *main* to *twin*, which will remove *main* from the page cache and insert *twin* instead. We optimize out the copying process of the page, as it was already performed ahead of time during replication. After performing the migration, we unfreeze the pages and update the *struct duplication* to reflect the new *main* page.

## 5.3 Handling of memory evictions and memory pressure

Linux provides several advanced mechanisms for efficient memory management. In particular, PACAR must interact
<figure><img src="image_6.png" /><figcaption>**Figure 3.** Eviction code path modifications</figcaption></figure>

closely with memory reclamation, watermarks, and memory compaction subsystems. Figure 3 summarizes our changes made to the kernel’s memory subsystem.

**Memory reclamation.** The *kswapd* daemon is a kernel thread responsible for reclaiming memory pages when the system is under memory pressure. It does this by scanning the LRU list and evicting pages that are not in use. When a page is evicted, it is removed from the LRU list and added to a free list, making it available for future allocations. If a *main* page is evicted, we need to ensure that we don’t leave orphaned *twin* pages. Therefore, we modify the eviction code path to check whether this is the case. If so, we perform a switch *main* operation to make one of the *twin* pages the new *main*. We use the same algorithm described in Section 5.2.

**Watermarks.** The Linux kernel uses a watermark system to manage memory pressure. Each NUMA node has a structure called *zone* that monitors its memory status. This *zone* uses watermarks, or thresholds, to signal when the node is running low on physical memory. Once the highest watermark is reached, the kernel will start reclaiming memory pages. We use this same watermark to temporarily disable replication when the system is under memory pressure. We

resume replication once the memory pressure is relieved and the memory usage is below the watermark.

**Memory compaction.** The *kcompactd* daemon is a kernel thread that helps reduce physical memory fragmentation by moving pages around using generic memory migration functions. If a replicated page is moved by this daemon, we need to guarantee that the mappings are still correct. Since we do not add our replicas in the page cache’s *xarray*, we need to ensure that the memory migration code does not change page cache entries or block buffers when moving a replicated page.

## 5.4 File system support

PACAR has been implemented and thoroughly tested on NFS, XFS, and ext4 (without journaling), three of the most widely used file systems spanning desktop, server, and network storage environments.

Among the supported file systems, XFS stands out as the only mature implementation currently supporting large footos (up to 128 pages), which significantly reduces LRU overhead by grouping pages together. This capability enables PACAR to replicate contiguous pages in batches, requiring only a single *struct duplication* per *folio*. While ext4 and NFS
are also supported, they do not yet benefit from these folio-
based optimizations. The journaling features of ext4 are not
fully supported due to its distinct approach to handling write
operations, which is incompatible with PACAR. Once these
file systems add support for large folios, they will be able
to fully leverage the performance and memory efficiency
improvements provided by PACAR.

Porting PACAR to additional file systems is straightforward and typically requires only minimal modifications. The primary requirement is that the file system relies on standard I/O paths in Linux, ensuring that read and write operations are routed through the page cache. A second requirement is that all accesses to the *private* field of *struct page* must be performed using the generic accessor provided by PACAR. Since most file systems already adhere to these conventions, porting is generally seamless. Comprehensive instructions are included with the PACAR source code to facilitate this process.

6 Evaluation

We evaluate PACAR across three dimensions: (i) the raw
performance of our page replication mechanism, (ii) the
ability to handle memory pressure, and (iii) the performance
of real-world workloads. We also evaluate the impact of
specific features of PACAR, namely the switch *main* and the
deactivation of replication during memory pressure.

We conduct our experiments on a dual-socket NUMA
server featuring two Intel Xeon Silver 4410Y processors, each
with 12 cores and 24 threads. The server is equipped with
256 GB of DDR5 memory and three Dell DC NVMe CD7
U.2 SSD drives (960 GB each) configured in RAID0, with the
XFS file system. The NUMA characteristics of the server are
available in Table 1 (on the second row).

## 6.1 Raw performance of page cache replication

We first evaluate the potential gains of our page cache repli-
cation mechanism by running a synthetic benchmark. We
investigate the read and write bandwidths of the page cache
with different load ratios. We use fio [3] to generate loads
with different characteristics in terms of read and write ratios
on a 40 GB dataset and 100 threads. We generate workloads
with read ratios ranging from 0% (write only) to 100% (read
only), with increments of 5% for read ratios from 0% to 90%,
and increments of 2% for read ratios from 90% to 100%. We
run fio for 20 seconds for each read/write ratio, measuring
the bandwidth with and without PACAR. Figure 4 presents
the results of our experiments.

Figure 4a depicts the bandwidths of read operations only. We observe that PACAR improves the read performance over Linux starting from a 65% read ratio. This improvement peaks at a 100% read ratio, where we achieve a bandwidth increase of up to 1.6x compared to Linux. This is mainly due to the ability of fio threads to use local replicas instead of

performing remote accesses. In particular, we note that the
higher the read ratio, the better bandwidth improvement
we get. This is due to the fact that the less write operations
we have, the fewer invalidations are needed, consequently
reducing remote accesses and improving the bandwidth. We
also note a slight degradation with low read ratios, which is
due to the high number of replica invalidation, which then
lead to new replications. This behavior is expected, as we
specifically target read-intensive workloads.

Figure 4b depicts the bandwidth of write operations only. We slightly degrade the performance of write operation with low read ratios, i.e., high write ratios. This is expected as write operations require replica invalidations, while Linux just performs the write operation. More surprisingly, we note a performance improvement of the write bandwidth with increasing read ratios (from 65% onwards). We hypothesize that this is due to most read operations becoming local, freeing up the interconnect from excessive remote accesses. This reduction in contention on the interconnect allows write operations to be executed more efficiently, resulting in improved write bandwidth.

**Page cache locality.** We also measure the locality of accesses to the page cache, in order to confirm that the performance improvements we observe are indeed due to the locality of accesses. Figure 4c shows the ratio of local accesses to the page cache, i.e., the ratio of accesses that are local to the node where the thread is running. Note that what we call local access is an access where the page returned to user space was read on the local node. Therefore, it also includes accesses that trigger a page replication.

With no replication (Linux), threads and data are evenly distributed across NUMA nodes, resulting in an average locality ratio of 45% for both read and write operations. With PACAR, we drastically improve the locality of accesses to the page cache, with a steady increase with the read ratio, reaching full locality for the read-only workload. The remote accesses still present with intermediate read ratios are due to write operations, as they still trigger remote operations to the *main* page to preserve the consistency of the page cache.

We note that the even distribution of threads and data
across NUMA nodes, coupled with random access patterns,
leads to no single node predominantly writing to a specific
page. Consequently, every node engages in randomly writing
locally and remotely. This behavior undermines the concept
of switching the main page upon write, as no node consis-
tently writes to a page without encountering other access
patterns. As a result, this experiment does not demonstrate
any improvement in write locality.

## 6.2 Handling of memory evictions and memory pressure

We evaluate how our system handles memory pressure by
creating a scenario with a user application allocating a large
<figure><img src="image_7.png" /><figcaption>(a) Bandwidth of read operations in our fio synthetic workloads (higher is better)</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>(b) Bandwidth of write operations in our fio synthetic workloads (higher is better)</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>(c) Ratio of local accesses in our fio synthetic workloads (higher is better)</figcaption></figure>

<caption>**Figure 4.** Synthetic benchmark for different read/write ratios generated with fio</caption>

amount of memory. We use fio [3] to generate a read-only workload with a random access pattern on an 80 GB dataset. 72 threads are used in this experiment. After 40 seconds of executing our read-only workload, we allocate a large amount of memory (100 GB) in user space (with `malloc`) to trigger the kernel's page reclamation algorithm and see how it affects the allocation of our user program, as well as the bandwidth of our fio workload. When the allocation completes, we wait for 20 seconds, and then free the allocated memory. We perform this experiment on Linux, as well as with different configuration of PACAR: none only features

page replication and consistency, *switch main* enables the switch of the *main* page upon eviction, *pressure mitigation* enables the memory pressure mitigation mechanism, that disables replication when memory pressure is detected, as well as *pressure mitigation + switch main* which combines all features of PACAR. Figure 5a depicts the evolution of the bandwidth of our fio workload over time, while Figure 5b shows the evolution of the allocated size of the *malloc* over time. Both figures are aligned to ease the reading of correlated events.

The memory allocation is performed by a single thread, that triggers a page fault for every allocated page. This leads to a very long allocation time (roughly between 40 and 150 seconds) for all setups. During most of this allocation time, there is no memory pressure, which means our fio workload's bandwidth is steady at around 175 GB/s for PACAR and 120 GB/s for Linux.

When the system is under memory pressure, at around 140 seconds, the bandwidth of our fio workload starts to drop, as replicated pages start top get reclaimed to accommodate the memory allocation. All variants of PACAR suffer from this drop, but they always remain on par with the Linux baseline. This is particularly interesting, as the Linux baseline is not affected by memory pressure. Indeed, our server has enough memory (256 GB) to allocate the 100 GB while maintaining the entire 80 GB of the fio dataset in memory. With PACAR, the fio dataset is fully replicated, which means it takes up 160 GB of memory. Locating the 100 GB thus exceeds the available physical memory, leading to memory pressure and the eviction of replicated pages.

**Memory pressure mitigation.** We notice a significant bandwidth reduction when the memory pressure mitigation mechanism is not active. This is due to the fact that PACAR still tries to replicate page even though the page frame reclamation algorithm is trying to free up memory. This leads to a situation where the replication mechanism is constantly trying to replicate pages, while the kernel is trying to reclaim them. This results in a bandwidth drop of roughly 50 GB/s, mainly due to the increased number of activations of the swapper thread that does not manage to free memory for the user application quickly enough. This shows that disabling replication during memory pressure is crucial to avoid performance degradation.

**Switching the main page.** When enabling the *switch main* feature, we notice that our fio bandwidth recovers quickly after the user memory is freed. This is because the *main* page is switched to the local node, which allows for faster subsequent access to the data.

**Impact on the speed of allocations.** Another interesting observation in Figure 5b is that the allocation time is significantly faster when replication is enabled. This is mainly due to the fact that accesses top the page cache are much more
<figure><img src="image_10.png" /><figcaption>(a) Bandwidth of a read-only workload (fio) during memory pressure (higher is better)</figcaption></figure>

<figure><img src="image_11.png" /><figcaption>(b) Allocated size of our user space program creating memory pressure</figcaption></figure>

<caption>**Figure 5.** Experimental scenario illustrating the impact of memory pressure on PACAR</caption>

local, reducing the contention on the interconnect. Additionally, the reclaimed pages from the page cache are quickly reused by the malloc allocation, so there is no costly memory allocation to perform, i.e., recycling replicated pages is faster than allocating new pages.

**Memory footprint.** We also evaluate the memory footprint of our mechanism: approximately 950 MB of metadata (*struct duplication*) is used to replicate 80 GB worth of pages.

## 6.3 Real-world workloads

To assess the effectiveness of our mechanism under realistic conditions, we employ Filebench [24] with six representative workloads, each chosen to cover a broad spectrum of read/write ratios and file operation patterns commonly encountered in production environments.

The *fileserver* workload emulates a typical *file server* environment by performing a mix of file operations—such as creation, deletion, appending, reading, writing, and attribute modifications—across a directory tree. This pattern closely mirrors the behavior of the SPECsfs benchmark, providing a comprehensive assessment of general-purpose file server activity.

The *web-server* workload models typical web-server I/O activity by performing sequences of open-read-close operations on multiple files within a directory tree, along with log file appends.

The *videoserver* workload emulates a video server environment. It uses two file sets: one containing actively served videos and another holding inactive videos. While a single thread writes new videos to replace outdated ones in the passive set, the majority of threads concurrently serve videos from the active file set. This setup closely models the typical access patterns of a production video streaming service, with frequent reads and periodic updates.

*ycsb-a*, *ycsb-b* and *ycsb-c* workloads are inspired by the I/O patterns of YCSB [9] with read ratios of 0.55, 0.95, and 1.0. These workloads are characterized by an access pattern typical of databases: 20,000 files of 512 KB accessed using a gamma distribution, and 48 threads in total. *fileserver* is configured with 50 threads, 30,000 files, each with a size of 128 KB. *videoserver* consists of 48 threads constantly reading 10 files of 2 GB, and 30 files of 2 GB being written by a single additional thread. *webserver* is configured with 100 threads reading 10,000 files of 512 KB, additionally, each thread is also appending data to a log file.

Some *filebench* workloads have been excluded (*oltp*, *var-mail*, and *webproxy*) because they are not I/O intensive, and our mechanism will neither improve nor degrade performance. We don't measure improvements or deterioration of bandwidth when testing those workloads.

Each workload is executed for 10 minutes under conditions both with and without replication enabled. To maintain consistency across experiments, we flush the page cache at the start of each test. Upon completion, we collect data on total bandwidth, latency, and operations per second. Our findings are summarized in Figure 6.

As illustrated in Figure 6a, all workloads exhibit a clear improvement in bandwidth when replication is enabled. For the YCSB workloads (*ycsb-a*, *ycsb-b*, and *ycsb-c*), we observe that higher read ratios correspond to greater bandwidth gains, with PACAR delivering consistent improvements even for *ycsb-a*, which features a moderate read ratio of 0.55. These results confirm that our mechanism is effective across a range of access patterns, providing tangible benefits even in scenarios that are not predominantly read-intensive.

The *fileserver* workload stands out as an exception, exhibiting no significant performance improvement across any metric. The minor performance decrease observed (less than 1%) is likely due to its relatively low I/O intensity and limited opportunity for replication benefits. Furthermore, *fileserver* has the lowest read intensity among all workloads, with a read ratio of only 0.33, which further limits the potential gains from our mechanism.

Figure 6b shows that PACAR achieves a consistent reduction in average operation latency across nearly all workloads, except *fileserver*. The most significant latency decrease, up
<figure><img src="image_12.png" /><figcaption>(a) Bandwidth for different filebench workloads</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>(b) Latency for different filebench workloads</figcaption></figure>

<figure><img src="image_14.png" /><figcaption>(c) Operations per second for different filebench workloads</figcaption></figure>

to 37%, is observed for the webserver workload. Similarly, Figure 6c demonstrates that PACAR increases the number of operations per second for all workloads except *fileserver*, highlighting its effectiveness in improving throughput under diverse I/O patterns.

**RocksDB.** RocksDB is utilized to evaluate the performance of PACAR on real-world workloads. We compare the performance of Linux and PACAR across three different workloads: *multireadrandom*, *updaterandom*, and *appendrandom*. The *multireadrandom* workload is read-only, *updaterandom* involves a combination of read, modify, and write operations, while *appendrandom* is write-only, appending new values to the database. To ensure accurate performance measurements,

we configure *dbbench* — RocksDB’s benchmarking tool — to bypass its internal cache, as caching would undermine the role of PACAR. The database’s total size is 17 GB for all three workloads when started, with *appendrandom* growing it to hundreds of GB. *dbbench* is set up to use all 48 threads on the machine, spanning both NUMA nodes.

Table 2 shows the percentage differences between Linux and PACAR across three metrics: operations per second, total operations, and bandwidth. *Multireadrandom* and *updaterandom* run for 5 minutes each, while *appendrandom* is executed 10 times for 30 seconds, on a fresh database each time to avoid database overgrowth.

We observe that PACAR provides a performance improvement of approximately 7% across all three metrics for the read-only *multireadrandom* workload. However, there is no significant differences between Linux and PACAR for the other workloads, indicating that PACAR does not introduce any overhead for these types of applications. The observed performance gains are smaller compared to our previous experiments, likely because RocksDB is less I/O-intensive than the other workloads we test.

<table><caption>**Table 2.** Performance comparison of PACAR and Linux on RocksDB, showing the percentage difference in operations per second, total operations, and bandwidth across three different workloads.</caption><thead><tr><td>Workload / Metric</td><td>Ops/sec</td><td>Operations</td><td>Bandwidth</td></tr></thead><tbody><tr><td>multireadrandom</td><td>+7.9%</td><td>+7.9%</td><td>+8.0%</td></tr><tr><td>updaterandom</td><td>-0.7%</td><td>±0.0%</td><td>-0.7%</td></tr><tr><td>appendrandom</td><td>-0.2%</td><td>-0.1%</td><td>-0.3%</td></tr></tbody></table>

These results highlight the versatility and effectiveness of our mechanism in enhancing system performance under diverse and demanding I/O patterns. It successfully avoids performance degradation in write-heavy workloads, delivering up to a 25% improvement in bandwidth, a reduction of up to 37% in latency, and an increase of up to 28% in operations per second.

# 7 Limitations

**Support for Memory-Mapped Pages.** PACAR does not currently support the replication of memory-mapped (mmap) pages. The primary challenge lies in the difficulty of tracing memory accesses to mmaped regions by the Linux operating system. When a page is memory-mapped, it is directly accessible to the application through its virtual address space. The application can then access the data simply by reading from or writing to that address. These operations bypass the operating system entirely as long as the mapped pages remain valid, making it challenging for PACAR to detect accesses to the page cache when pages are mapped.
Additionally, on Linux systems, NUMA balancing already targets `mmaped` pages similarly to how it handles anonymous memory. Consequently, file-backed memory maps exhibit greater NUMA-awareness compared to accessing the page cache through file descriptors.

One potential solution to support `mmap` involves maintaining a separate page table for each NUMA node per process. Each page table would reference the appropriate local replica of a page, ensuring that threads consistently access memory local to their NUMA node. To preserve consistency across replicas, these pages could be write-protected, causing write accesses to trigger page faults. This would allow the operating system to intervene and manage coherence across replicas.

Since this approach represents a fundamentally different solution to addressing locality issues, it falls outside the scope of PACAR and has not been implemented at this time.

**Tradeoff between replication and overall page cache size.** Replication enhances performance for frequently accessed pages across multiple NUMA nodes by improving locality. However, this approach can also reduce the total volume of cached data available. In scenarios involving larger datasets that cannot be fully replicated, performance may degrade due to increased disk accesses as fewer pages remain cached. Future work could investigate strategies to limit replication, selectively targeting only the most frequently accessed pages to optimize locality. This could help reduce latency and increase bandwidth for these critical pages while avoiding penalties on less frequently accessed pages that would otherwise necessitate disk retrieval. This presents a tradeoff between the performance gains achieved through replication and the potential reduction in cache size leading to increased disk accesses.

# 8 Related Work

Over the years, numerous solutions have been proposed to address the challenges of NUMA systems. This section reviews key contributions to the field, with a particular focus on replication mechanisms.

**Replication on NUMA systems.** One common approach to mitigate the impact of remote memory accesses in NUMA systems is memory replication. Mitosis, for example, demonstrated significant performance improvements (ranging from 1.34x to 3.24x) by replicating page tables [2]. Building on this, Hydra introduced transparent, on-demand, and partial page-table replication [15], while WASP focused on enabling replication tailored to specific workloads [26]. Phoenix further advanced this field by employing a page table-aware scheduler that adapts replication based on memory locality [29].

Another notable system is NrOS, a NUMA-aware OS resembling a hypervisor, which continuously replicates all

memory across NUMA nodes, including the page cache [5]. Using a log-based architecture, NrOS efficiently propagates write operations between nodes, and features NrFS, an in-memory file system that benefits from this replication for enhanced performance, similar to PACAR. However, unlike NrOS, PACAR is implemented within the Linux kernel and does not inherently support full memory replication. Instead, it takes a more granular approach, replicating only actively used memory across nodes and releasing it as memory pressure increases.

Popcorn [4], a replicated-kernel OS based on Linux, is another system designed to improve scalability on high-core-count machines. It boots multiple kernel instances on separate cores, with each instance managing its own state while communicating with others to maintain a unified system image. This allows applications to migrate seamlessly across kernels, but it does not address page cache management.

Carrefour introduces an adaptive replication method, which activates when more than 95% of memory operations are reads and there is sufficient free memory. However, it lacks mechanisms to enhance the locality of the page cache, thus failing to optimize data access from storage [16].

Another technique aimed at improving performance in NUMA systems is kernel text replication, which reduces delays caused by accessing kernel code on remote nodes. By replicating the kernel text and read-only data across nodes, this approach ensures that each node can access its local copy, minimizing instruction fetch delays. The technique has proven particularly effective for workloads that are sensitive to memory access patterns. Russell King's work on arm64 laid the foundation for this approach, demonstrating its ability to balance memory bandwidth and improve system performance [18].

**Data placement and migration.** In addition to replication, significant research has focused on improving NUMA locality through strategic data placement and migration. Early work by Bolosky et al. laid the groundwork for understanding how process migration can optimize performance in NUMA systems [7]. Since then, several projects have advanced dynamic data and task placement techniques, which automatically adjust the location of data and/or tasks based on system demands [16, 22, 23, 27, 30]. Other efforts have focused on developing program libraries that assist developers in creating NUMA-aware applications, thereby improving system resource utilization [21, 28]. Together, these approaches aim to reduce latency and enhance overall system performance by aligning memory access patterns with the physical layout of multicore processors.

# 9 Conclusion

This paper presented PACAR, a transparent and practical page cache replication mechanism that addresses the fundamental challenges of data locality in modern NUMA systems.
Motivated by the persistent performance gap between local and remote memory accesses, and the limitations of existing NUMA-aware memory management for buffered I/O, PACAR introduces selective, on-demand replication of cached pages across NUMA nodes. Our design is guided by four key goals: maximizing I/O performance, overcoming the scalability barriers of traditional migration-based approaches, ensuring full transparency for unmodified applications, and preserving robust memory management under pressure.

PACAR operates entirely within the kernel, requiring no changes to user code or application workflows. By replicating hot pages on-demand and maintaining strong consistency through a lightweight main/twin protocol, PACAR enables threads on any node to access cached data locally, eliminating the bottlenecks and contention of remote memory accesses. Our evaluation demonstrates that PACAR delivers substantial performance improvements for I/O-intensive and highly parallel workloads, achieving up to 1.4× speedup in microbenchmarks, up to 25% gains in real-world workloads such as YCSB and web servers, and around 8% of bandwidth improvement on RocksDB. Importantly, PACAR adapts to memory pressure by integrating with the kernel's reclamation subsystem, ensuring that replication never compromises system stability or resource fairness.

By bridging the gap between NUMA-aware memory management and the page cache, PACAR unlocks the full potential of modern multi-socket servers for data-intensive applications. Our approach demonstrates that it is possible to transparently and efficiently exploit idle memory resources to improve locality, without sacrificing consistency or requiring intrusive changes to the operating system or applications.

Looking forward, a promising direction is to extend the principles of PACAR to memory-mapped file access (mmap), enabling transparent replication and locality benefits for a broader class of applications and workloads. We believe that generalizing page cache replication to all file-backed memory will further enhance the scalability and efficiency of future NUMA systems.

## Acknowledgments

We thank the anonymous reviewers and our shepherd, Reto Achermann, for their insightful comments and suggestions. Some of the experiments presented in this paper were carried out using the Grid'5000 testbed, supported by a scientific interest group hosted by Inria and including CNRS, RENATER and several Universities as well as other organizations (see https://www.grid5000.fr). This work was supported by a French government grant managed by the Agence Nationale de la Recherche under the France 2030 program, reference "ANR-23-PECL-0004"

# A Artifact Appendix

## A.1 Abstract

This document details the system requirements and installation procedure for PACAR, and describes the benchmark suite and its execution. The artifact includes PACAR as a patch to the Linux kernel, along with the benchmarking tools, scripts, and workloads needed to reproduce the experiments reported in this paper. The artifact is packaged under the GPL License 3, ensuring it remains open-source and freely redistributable.

## A.2 Description & Requirements

### A.2.1 How to access.
You can access the artifact through GitHub at: https://github.com/geromecst/pacar artifact_eurosys_26.

The artifact is also available on Zenodo at: https://doi.org/10.5281/zenodo.17054043.

### A.2.2 Hardware dependencies.
To run our experiments, you need a NUMA system with 2-nodes. We tested PACAR on x86_64-based systems, with 2×24 cores and 256 GB of memory. The scripts can scale depending on the number of threads and memory of your machine.

A block device with at least 1 TB of free storage is also required, preferably an SSD.

### A.2.3 Software dependencies.
PACAR has been tested on Debian 12 and Debian 13. We use gcc 14.2.0 (Debian 14.2.0-19), but other versions should also work. You also need to have a root partition formatted with XFS.

For the benchmarks, we mainly rely on these programs:

* fio version 3.39

* filebench commit 22620e602

* rocksdb version v9.10.10

To reproduce our results exactly, please use these versions.

### A.2.4 Benchmarks.

**Filebench.** We include the filebench workloads in the directory workloads/filebench. The workloads webserver, videoserver and fileserver have been modified so that we can make them scale with our script run-exp.sh. We are able to modify dynamically the data set size and the number of cores used. We wrote the ycsb workloads to match the respective YCSB traces A, B and C. The YCSB workloads are defined here https://github.com/brianfrankcooper/YCSB/wiki/Core-Workloads.

filebench is automatically cloned and installed by our setup script:

./pacar-setup.sh

**Fio.** fio can be obtained using apt. The workloads are included in workloads/fio. They are simulating different configuration of readers and writers accessing the same set of files.
sudo apt install fio

**RocksDB.** We are using `dbbench` to benchmark `RocksDB`, the workload used are already defined by `dbbench`. The script `run-exp.sh` contains the configuration of `RocksDB`, where we increase the database size, disable the cache and increase the number of threads used. `RocksDB` is automatically cloned and installed by our setup script:

./pacar-setup.sh

### A.3 Setup

**Root partition.** We use the XFS file system in our experiments. Make sure you are using XFS for the root file system. A tutorial to install Debian with XFS is provided in the README.

**PACAR installation.** To build and install PACAR, you need to download the Linux sources (version 6.12.20), apply the PACAR patch, and compile and install the newly patched kernel. This is achieved by our script `pacar-setup.sh`.

sudo apt install git

git clone

https://github.com/jeromecst/pacar Artifact eurosys_26

cd pacar Artifact eurosys_26

# `pacar-setup` pulls all dependencies, installs PaCaR,

and also install filebench/rocksdb/fio

sudo ./pacar-setup.sh

Now that the new kernel is installed, we can execute it directly.

# `Reboot` with `kexec`

version=6.12.20

sudo kexec -l /boot/vmlinuz-$version --initrd

/boot/initrd.img-$version --reuse-cmdline

sudo kexec -e

Finally, verify that PACAR is correctly installed on the system:

# this directory should be created

$ ls /sys/kernel/mm/duplication/

dump enabled memory pressure mitigation stats

switch_main_eviction threshold

The entire evaluation is performed on the patched kernel, with PACAR being disabled or enabled depending on the experiment.

## A.4 Evaluation workflow

### A.4.1 Major Claims.

* (C1): PACAR improves read bandwidth by 65% on read-only workloads on I/O intensive microbenchmarks. This is shown by experiment E1, described in Section 6.1, Figure 4a.

* (C2): Even under memory pressure, PACAR manages to keep a 25% bandwidth improvement over Linux

on read-only workloads on I/O intensive microbenchmarks. This is shown by experiment E2, described in Section 6.2, Figure 5.

* (C3): PACAR improves bandwidth, decreases latency and improves operations per seconds for various simulation workloads. This is shown by experiment E3, described in Section 6.3, Figure 6.

* (C4): PACAR improves bandwidth and operations per seconds on read-only workloads, with a small overhead otherwise, on RocksDB, a production-grade database. This is shown by experiment E4, described in Section 6.3, Table 2.

### A.4.2 Experiment E1: fio read ratio percentage (40 GB, 2.4 compute-hours).
In this experiment, we run fio with different percentages of read-write ratios, and measure the read and write bandwidths of PACAR and Linux.

The experiment is run, and the results plotted using the following commands:

# Run the experiment

sudo ./run-exp.sh fio_percentage

# Plot the results (the filename depends on the

current date, replace XXXXX accordingly)

python3 plot/script_fio_percentage.py

./exp_results/fio_percentage_XXXXX

### A.4.3 Experiment E2: fio memory pressure (80 GB, 50 compute-minutes).
In this experiment, we run fio and trigger a malloc spanning an important amount of memory on a single node to measure the bandwidth effects on both PACAR (with different features) and Linux.

The experiment is run and the results plotted using the following commands:

# Run the experiment

sudo ./run-exp.sh fio_malloc

# Plot the results (the filename depends on the

current date, replace XXXXX accordingly)

python3 plot/script_fio_malloc.py

./exp_results/fio_malloc_XXXXX

### A.4.4 Experiment E3: filebench (120 GB, 2 compute-hours).
In this experiment, we run filebench with different workloads, and measure the bandwidth, latency and operations per second of PACAR and Linux.

The experiment is run and the results plotted using the following commands:

# Run the experiment

sudo ./run-exp.sh filebench

# Plot the results (the filename depends on the

current date, replace XXXXX accordingly)

python3 plot/script_filebench.py

./exp_results/filebench_malloc_XXXXX

### A.4.5 Experiment E4: RocksDB (500 GB, 70 compute-minutes).
In this experiment, we run db_bench on RocksDB
in read-only mode, and measure the bandwidth, latency and
operations per second of PACAR and Linux.

The experiment is run, and the results plotted using the
following commands:

# Run the experiment

sudo ./run-exp.sh dbbench

# Plot the results (the filename depends on the

→ current date, replace XXXXX accordingly)

python3 plot/script_dbbench.py

./exp_results/dbbench_XXXXX

A.5 Notes on Reusability

Most experiments are automatically scaled up and down, de-
pending on the hardware configuration (number of cores and
memory size). The dataset size is not automatically scaling:

* (E1 and E2) The size of the dataset can be modified with the SIZE variable in run-exp.sh.

* (E3) can be scaled by modifying the file size in the workloads defined in the directory workloads/filebench.

* (E4) The dataset size can be scaled by modifying the parameters value_size and num in the load_dbbench and exp_dbbench function of run-exp.sh.

The experiments can also be run on a NUMA with 4 nodes or more, but the calls to *numactl* in *run-exp.sh* need to be updated to spread the page cache on all nodes.

A.6 Notes to support more file systems

PACAR is compatible with both XFS and NFS file systems. To
extend support to additional file systems, modifications to
their source code are necessary. Specifically, PACAR modifies
the *struct page* to track replication metadata, as described
in Section 5. The *private* field within this structure is re-
purposed to store a union. Since some file systems directly
access the *private* field (using *page->private* instead of a
getter function), these file systems must be updated to use
the *folio_get_private()* getter function for compatibility with
PACAR. This getter function has been adapted by PACAR to
ensure that the correct field is returned based on the page's
state, preventing the incorrect return of a *struct duplication*
instead of the original *private* field.

References

[1] [n. d.]. Multi-Gen LRU. https://docs.kernel.org/mm/multigen_lru.html#multi-gen-lru [Online].

[2] Reto Achermann, A. Panwar, A. Bhattacharjee, Timothy Roscoe, and Jayneel Gandhi. 2019. Mitosis: Transparently Self-Replicating Page-Tables for Large-Memory Machines. *Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems* (2019). doi:10.1145/3373376.3378468

[3] Jens Axboe. [n. d.]. Axboe/fio: Flexible I/O tester. https://github.com/axboe/fio

[4] Antonio Barbalace, Binoy Ravindran, and David Katz. 2014. Popcorn: a replicated-kernel OS based on Linux. In *Proceedings of the Linux Symposium, Ottawa, Canada*.

[5] Ankit Bhardwaj, Chinmay Kulkarni, Reto Achermann, Irina Calciu, Sanidhya Kashyap, Ryan Stutsman, Amy Tai, and Gerd Zellweger.

2021. NrOS: Effective replication and sharing in an operating system. In *15th {USENIX} Symposium on Operating Systems Design and Implementation* ({OSDI} 21). 295-312.

[6] Carlos Bilbao, J. C. Saez, and Manuel Prieto-Matias. 2023. *Divide&Content: A Fair OS-Level Resource Manager for Contention Balancing on NUMA Multicores*. *IEEE Transactions on Parallel and Distributed Systems* 34 (2023), 2928-2945. doi:10.1109/TPDS.2023.3309999

[7] W. Bolosky, R. Fitzgerald, and M. Scott. 1989. Simple but effective techniques for NUMA memory management. In *Proceedings of the Twelfth ACM Symposium on Operating Systems Principles (SOSP '89)*. Association for Computing Machinery, New York, NY, USA, 19-31. doi:10.1145/74850.74854

[8] Luc Bouganim, D. Florescu, and P. Valduriez. 2004. Load Balancing for Parallel Query Execution on NUMA Multiprocessors. *Distributed and Parallel Databases* 7 (2004), 99-121. doi:10.1023/A:1008642513285

[9] Sean Busbey, allanbank, Andy Kruth, Kevin Risden, Connor McCoy, Jason Tedor, Brian Cooper, Chris Larsen, Jaemyoun, stfeng2, Nitsan Wakart, gse89, Cindy Peng, Igor Bernstein, Uncle Betty, suraj-pasuparthy, Robert Lehmann, Eugene Blikh, Weston Platter, Jeff Yemin, Johan Oskarsson, yuyanting, Michael Nitschinger, Kirill Vlasov, Misha Brukman, Sudipto Das, danielpoltx, Christian Matser, Steffen Friedrich, and Ivan Prisyazhnyy. 2025. YCSB. https://github.com/brianfrankcooper/YCSB. https://github.com/brianfrankcooper/YCSB

[10] Jonathan Corbet. 2012. AutoNUMA: the other approach to NUMA scheduling. https://lwn.net/Articles/488709/

[11] Jonathan Corbet. 2021. The multi-generational LRU. https://lwn.net/Articles/851184/

[12] Andrew Crotty, Viktor Leis, and Andrew Pavlo. 2022. Are You Sure You Want to Use MMAP in Your Database Management System? (2022).

[13] Linux Kernel Documentation. [n. d.]. Multi-Gen LRU. https://docs.kernel.org/admin-guide/mm/multigen_lru.html

[14] Alexander Fuerst, Ahmed Ali-Eldin, Prashant Shenoy, and Prateek Sharma. 2020. Cloud-scale VM-deflation for Running Interactive Applications On Transient Servers. In *Proceedings of the 29th International Symposium on High-Performance Parallel and Distributed Computing (Stockholm, Sweden) (HPDC '20)*. Association for Computing Machinery, New York, NY, USA, 53-64. doi:10.1145/3369583.3392675

[15] Bin Gao, Qingxuan Kang, Hao-Wei Tee, Kyle Timothy Ng Chu, Alireza Sanaee, and Djordje Jevdjic. 2024. numaPTE: Managing Page-Tables and TLBs on NUMA Systems. *ArXiv* abs/2401.15558 (2024). doi:10.48550/arXiv.2401.15558

[16] Fabien Gaud, Baptiste Lepers, Justin Funston, Mohammad Dashti, Alexandra Fedorova, Vivien Quéma, Renaud Lachaize, and Mark Roth. 2015. Challenges of Memory Management on Modern NUMA System: Optimizing NUMA systems applications with Carrefour. *Queue* 13, 8 (Oct. 2015), 70-85. doi:10.1145/2838344.2852078

[17] Jing Guo, Zihao Chang, Kangjing Wang, and Guoyao Xu. 2018. Alibaba Cluster Data. https://github.com/alibaba/clusterdata/blob/master/cluster-trace-v2018/trace_2018.md.

[18] Russell King. 2023. [patch RFC 00/17] ARM64 kernel text replication. https://lwn.net/ml/linux-doc/ZHYCUVa8fzmB4XZV@shell.armlinux.org.uk/

[19] Andres Lagar-Cavilla, Junwhan Ahn, Suleiman Souhlal, Neha Agarwal, Radoslaw Burny, Shakeel Butt, Jichuan Chang, Ashwin Chaugule, Nan Deng, Junaid Shahid, Greg Thelen, Kamil Adam Yurtsever, Yu Zhao, and Parthasarathy Ranganathan. 2019. Software-Defined Far Memory in Warehouse-Scale Computers. In *Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems (Providence, RI, USA) (ASPLOS '19)*. Association for Computing Machinery, New York, NY, USA, 317-330. doi:10.1145/3297858.3304053

[20] Tong Li, Dan P. Baumberger, David A. Koufaty, and Scott Hahn. 2007. Efficient operating system scheduling for performance-asymmetric
multi-core architectures. *Proceedings of the 2007 ACM/IEEE Conference on Supercomputing (SC '07)* (2007), 1–11. doi:10.1145/1362622.1362694

[21] Zoltan Majo and Thomas R Gross. 2015. A library for portable and composable data locality optimizations for NUMA systems. *ACM SIGPLAN Notices* 50, 8 (2015), 227–238.

[22] Neethu Bal Mallya, Bhavishya Goel, and Ioannis Sourdis. 2025. MEMPLEX: A Memory System with Replication and Migration of Data for Multi-Chiplet NUMA Architectures. In *Proceedings of the 39th ACM International Conference on Supercomputing. 1219–1233*.

[23] Jaydeep Marathe, Vivek Thakkar, and Frank Mueller. 2010. Feedback-directed page placement for ccNUMA via hardware-generated memory traces. *J. Parallel and Distrib. Comput.* 70, 12 (2010), 1204–1219.

[24] Richard McDougall and Jim Mauro. 2005. FileBench - A Prototype Model Based Workload for File Systems. (2005). http://www.nfsv4bat.org/Documents/nasconf/2004/filebench.pdf

[25] A. Plaat, H. Bal, Rutger F. H. Hofman, and T. Kielmann. 1999. Sensitivity of parallel applications to large differences in bandwidth and latency in two-layer interconnects. *Proceedings Fifth International Symposium on High-Performance Computer Architecture* (1999), 244–253. doi:10.1109/HPCA.1999.744376

[26] Hongliang Qu and Zhibin Yu. 2024. WASP: Workload-Aware Self-Replicating Page-Tables for NUMA Servers. *Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2* (2024). doi:10.1145/3620665.3640369

[27] RedHat. [n. d.]. Virtualization Tuning and Optimization Guide Automatic NUMA Balancing. https://docs.redhat.com/en/documentation/redHat_enterprise_linux/7/html/virtualization_tuning_and_optimization_guide/sect-virtualization_tuning_optimization_guidenuma-auto_numa-balancing. [Accessed 20-09-2025].

[28] Vishal Shrivastav, Asaf Valadarsky, Hitesh Ballani, Paolo Costa, Ki Suh Lee, Han Wang, Rachit Agarwal, and Hakim Weatherspoon. 2019. Shoal: A network architecture for disaggregated racks. In *16th USENIX Symposium on Networked Systems Design and Implementation (NSDI '19)*. 255–270.

[29] Mohammad Siavashi, Alireza Sanaee, Mohsen Sharifi, and Gianni Antichi. 2025. Phoenix - A Novel Technique for Performance-Aware Orchestration of Thread and Page Table Placement in NUMA Systems. arXiv:2502.10923 [cs.OS] https://arxiv.org/abs/2502.10923

[30] Mustafa M Tikir and Jeffrey K Hollingsworth. 2008. Hardware monitors for dynamic page migration. *J. Parallel and Distrib. Comput.* 68, 9 (2008), 1186–1200.

[31] Ben Verghese, Scott Devine, Anoop Gupta, and Mendel Rosenblum. 1996. Operating system support for improving data locality on CC-NUMA compute servers. *ACM SIGOPS Operating Systems Review* 30, 5 (1996), 279–289.

[32] Abhishek Verma, Luis Pedrosa, Madhukar R. Korupolu, David Opdenheimer, Eric Tune, and John Wilkes. 2015. Large-scale cluster management at Google with Borg. In *Proceedings of the European Conference on Computer Systems (EuroSys). Bordeaux, France*.

[33] Midhul Vuppalapati, Justin Miron, Rachit Agarwal, Dan Truong, Ashish Motivala, and Thierry Cruanes. 2020. Building an elastic query engine on disaggregated storage. In *Proceedings of the 17th Usenix Conference on Networked Systems Design and Implementation (Santa Clara, CA, USA) (NSDI'20)*. USENIX Association, USA, 449–462.

[34] Yaohui Wang, Ben Luo, and Yibin Shen. 2023. Efficient Memory Overcommitment for I/O Passthrough Enabled VMs via Fine-grained Page Meta-data Management. In *2023 USENIX Annual Technical Conference (USENIX ATC 23)*. USENIX Association, Boston, MA, 769–783.

[35] Johannes Weiner, Niket Agarwal, Dan Schatzberg, Leon Yang, Hao Wang, Blaise Sanouillet, Bikash Sharma, Tejun Heo, Mayank Jain, Chunqiang Tang, and Dimitrios Skarlatos. 2022. TMO: transparent memory offloading in datacenters. In *Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (Lausanne, Switzerland) (ASPLOS '22)*. Association for Computing Machinery, New York, NY, USA, 609–621. doi:10.1145/3503222.3507731