# EasyDRAM: An FPGA-based Infrastructure for Fast and Accurate End-to-End Evaluation of Emerging DRAM Techniques

Oğuzhan Canpolat, Ataberk Olgun, David Novo, Oğuz Ergin, Onur Mutlu

► To cite this version:

Oğuzhan Canpolat, Ataberk Olgun, David Novo, Oğuz Ergin, Onur Mutlu. EasyDRAM: An FPGA-based Infrastructure for Fast and Accurate End-to-End Evaluation of Emerging DRAM Techniques. DSN 2025 - 55th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, Jun 2025, Naples, Italy. pp.331-344, 10.1109/DSN64029.2025.00042. lirmm-05448422

HAL Id: lirmm-05448422

https://hal-lirmm.ccsd.cnrs.fr/lirmm-05448422v1

Submitted on 8 Jan 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# EasyDRAM: An FPGA-based Infrastructure for Fast and Accurate End-to-End Evaluation of Emerging DRAM Techniques

Oğuzhan Canpolat†§

Ataberk Olgun§

David Novo‖

Oğuz Ergin‡§†

Onur Mutlu§

§ETH Zürich

†TOBB ETÜ

‖LIRMM, Univ. Montpellier, CNRS

‡University of Sharjah

In this work, we introduce EasyDRAM, a new end-to-end framework to evaluate DRAM techniques on real DRAM chips. EasyDRAM overcomes the main drawbacks of prior FPGA-based platforms with two key ideas. First, EasyDRAM removes the need for hardware description language expertise by enabling developers to implement DRAM techniques using high-level software (C++). At runtime, EasyDRAM executes the software-defined memory system design in a programmable memory controller. Second, EasyDRAM tackles a fundamental challenge in accurately modeling modern systems: real processors typically operate at significantly higher clock frequencies than DRAM, a disparity that is difficult to replicate on FPGA platforms. EasyDRAM addresses this challenge by decoupling the processor-DRAM interface and advancing the system state using a novel technique we call time scaling, which faithfully captures the timing behavior of the modeled system.

We demonstrate the ease of use of EasyDRAM by evaluating two DRAM techniques end-to-end in a real system. First, we implement bulk copy operations (i.e., RowClone) and show that these operations improve the average (maximum) copy efficiency by 13.1x (15.2x). Second, implement reduced latency DRAM access that leverages the heterogeneity in DRAM cell access latency, which results in an average (maximum) performance improvement of 2.75% (9.76%). We compare our results to prior evaluation methodologies and reveal the insight that prior approaches that do not faithfully model an end-to-end system can be significantly (>20x) inaccurate.

## 1. Introduction

DRAM-based main memory is a critical component in modern computing systems. Many prior works show that DRAM is increasingly becoming the primary bottleneck for system performance and energy efficiency [1–3] as the demand for higher performance, reduced latency, and increased capacity is growing. To meet these requirements, many prior works [4–18] introduce new unconventional operations (we call these operations DRAM techniques) such as violating timings set by the DRAM manufacturers to enhance the latency, reliability, and computing capabilities of DRAM. Evaluating the system-wide benefits of DRAM techniques is non-trivial as they require modifications to multiple layers of the computation stack (e.g., memory controllers, memory mapping and allocation in the operating system, and user applications), and require DRAM to be operated outside the current standard. For example, it is possible to reliably copy bulk data at DRAM row granularity (typically 8 KiB) by issuing row activation commands in quick succession in real DRAM chips [17–23]. However, the DRAM command sequence to perform such copy operations is unde-

fined in the current DRAM standards (e.g., [24–27]). Hence, commercial off-the-shelf computing systems do not perform those copy operations.

Software simulators [28–32] can be used to evaluate system integrations of DRAM techniques. However, simulator evaluations have two key drawbacks. First, correctly implementing the behavior of DRAM techniques would require rigorous characterization studies and modeling efforts by domain experts [4, 5, 8, 33–35] because a software simulator does not use a real DRAM chip. Second, existing simulators demonstrate slow simulation speeds (e.g., a few KHz [36]), thereby requiring a significant amount of time (e.g., days or weeks) to evaluate end-to-end full-system workloads.

To overcome the limitations of commercial systems and software simulators in evaluating DRAM techniques, prior works [22, 23, 37–40] propose a variety FPGA-based evaluation platforms that are broadly categorized into three classes: (1) DRAM testing platforms, (2) FPGA-based simulators, and (3) FPGA-based emulators. These platforms suffer from key drawbacks that prevent their users from rapidly and accurately evaluating DRAM techniques end-to-end.

**Drawbacks of Existing FPGA-Based Platforms.** First, DRAM testing platforms [23,38] allow DRAM characterization studies to discover and test DRAM techniques [4, 8, 41, 42]. These platforms do not model an end-to-end system and they cannot be used to understand the system-wide performance impacts of DRAM techniques.

Second, FPGA-based simulators [39, 40] model and evaluate modern systems at high simulation speeds. These platforms have two drawbacks that prevent them from enabling rapid and accurate DRAM technique evaluation: (1) the simulated system cannot operate real DRAM chips because the simulation interfaces with main memory through a rigid interface and (2) modifying FPGA-based simulators requires deep hardware design language (HDL) expertise.

Third, FPGA-based emulators [22, 37] model end-to-end systems and can operate real DRAM chips. However, (1) modifying the memory controller of these systems to implement DRAM techniques requires deep HDL expertise and (2) the processor of the modeled system operates at 50 MHz clock speed while the DRAM operates at a much higher clock speed (e.g., 1.6 GHz). This clock frequency discrepancy limits the accuracy of the evaluation and heavily skews the results in favor of DRAM techniques.

Table 1 presents a qualitative analysis of five evaluation platform types that can be used to evaluate DRAM techniques. Among the available types of evaluation platforms, we identify five main challenges in the design of an end-to-end evaluation
framework for DRAM techniques. The framework needs to 1) operate real DRAM chips and provide correct insights on the techniques, 2) easily accommodate necessary modifications (e.g., customize DRAM timings) for DRAM techniques, 3) be fast to allow rapid end-to-end full workload evaluation, 4) accurately model the performance of modern computing systems, and 5) have an easily configurable system to meet user needs (e.g., to easily test a wide variety of processor designs).

**Our goal** is to develop a configurable framework that addresses the outlined challenges, allowing fast and accurate evaluation of emerging DRAM techniques using real DRAM chips. To this end, we design and develop EasyDRAM. We observe that DRAM testing platforms [23, 38] and FPGA-based emulators [22, 37] provide solutions to the first, third, and fifth design challenges. We build EasyDRAM on these solutions. To solve the two remaining challenges and enable rapid end-to-end evaluation of a wide-variety DRAM techniques, EasyDRAM leverages two key ideas.

First, EasyDRAM implements an easily programmable core that executes a program to model the memory controller (e.g., perform request arbitration and scheduling), we call such programs *software memory controllers*. Compared to a rigid (i.e., not programmable) memory controller (typically implemented using an HDL), a software memory controller allows the broader community of expert system designers and architects that lack deep HDL expertise to rapidly develop and prototype emerging DRAM techniques.

Second, EasyDRAM advances system state in a way that respects modern clock frequency ratios between different system components (e.g., the processor and the DRAM interface), even in FPGA-based prototypes where it is difficult to implement memory controller designs at clock frequencies equaling today's real processor clock speeds. To do so, we implement an emulation technique called *time scaling*, which allows system components to be evaluated at a different clock speed than their FPGA clock speeds (e.g., a processor running at 50 MHz FPGA clock speed interacts with the system as if the processor were clocked at 4 GHz), building on the design provided in [39, 40]. This way, EasyDRAM retains the benefits (e.g., short evaluation time and operating on real DRAM chips) of hardware evaluation platforms (e.g., PiDRAM [22]) while overcoming their shortcomings (e.g., inaccurate system performance evaluation).

We conduct two case studies to demonstrate EasyDRAM's extensibility and evaluation accuracy. First, we implement RowClone [43] and investigate system performance improvement of data copy and initialization microbenchmarks with two EasyDRAM configurations: 1) EasyDRAM - Baseline (no time scaling, i.e., 50MHz processor frequency) and 2) EasyDRAM -

Time Scaling (i.e., configured to imitate a NVIDIA Jetson Nano SoC [44]). Our results using RowClone [43] with EasyDRAM - Baseline and EasyDRAM - Time Scaling improves performance by 306.7x and 15.0x, respectively. These results imply that evaluation frameworks (e.g., [22, 37]) that do not faithfully model a modern system skew the results (e.g., >20x error margin) in favor of DRAM techniques. Second, we implement a DRAM access latency reduction technique (similar to [4]) and evaluate the end-to-end system performance of PolyBench [45] workloads with EasyDRAM - Time Scaling configuration. Our results show that access latency reduction improves the average system performance by 2.75% across all evaluated workloads.

We make the following key contributions:

* We propose EasyDRAM, an FPGA-based framework for end-to-end evaluation of DRAM techniques. EasyDRAM overcomes the limitations of prior work by enabling developers to implement DRAM techniques using high-level software and faithfully capturing the timing behavior of a modeled modern computing system using time scaling.

* We develop the EasyDRAM user-friendly C++ API for designing and extending EasyDRAM-based software memory controllers, making DRAM techniques easier to implement and their evaluation studies more accessible to system designers that lack deep hardware description language expertise.

* We perform two case studies using EasyDRAM and demonstrate that state-of-the-art evaluation methodologies that do not faithfully model a real system can yield significantly inaccurate system performance results.

* To enable reproducibility and aid future research in a transparent manner, we open-source our EasyDRAM implementation at https://github.com/CMU-SAFARI/EasyDRAM.

## 2. Background

We provide a brief background on DRAM organization, DRAM timing parameters, and memory controllers. For more detailed background on these, we refer the reader to many prior works [9, 12, 15, 23, 38, 46–54].

### 2.1. DRAM Organization

Figure 1 illustrates the hierarchical structure of DRAM-based main memory. Typically, a system's *memory controller* links to several DRAM *modules* across multiple DRAM *channels*, each operating independently. Each DRAM *module* encompasses one or more ranks, with each rank composed of several DRAM *chips* functioning in sync. The memory controller can interact with multiple DRAM ranks by alternating the channel's I/O bus among the ranks. Since the I/O bus is shared, the memory controller sequences accesses to different ranks within the same

<table><caption>Table 1: Comparison of EasyDRAM with related state-of-the-art prototyping and evaluation platforms.</caption><thead><tr><th>Platforms</th><th>Interface with real DRAM chips</th><th>Flexible MC for DRAM techniques</th><th>Evaluation clock speed magnitude</th><th>Accurate performance evaluation</th><th>Configurable system</th></tr></thead><tbody><tr><td>Commercial computing systems</td><td>DDR3/4/5</td><td>X</td><td>GHz</td><td>✓</td><td>X</td></tr><tr><td>Software simulators [28–32]</td><td>X</td><td>✓(C/C++)</td><td>KHz</td><td>✓</td><td>✓</td></tr><tr><td>FPGA-based simulators [39, 40]</td><td>X</td><td>X</td><td>MHz</td><td>✓</td><td>✓</td></tr><tr><td>DRAM testing platforms [23, 38]</td><td>DDR3/4</td><td>X</td><td>MHz</td><td>X</td><td>X</td></tr><tr><td>FPGA-based emulators [22, 37]</td><td>DDR3/4</td><td>HDL</td><td>MHz</td><td>X</td><td>✓</td></tr><tr><td>EasyDRAM (this work)</td><td>DDR4</td><td>✓(C/C++)</td><td>MHz</td><td>✓</td><td>✓</td></tr></tbody></table>
channel.

Each DRAM chip is partitioned into multiple *banks*. These DRAM banks within a chip share common I/O circuitry. The cells within a DRAM bank are structured into a grid of *rows* and *columns*. Each *cell* within a DRAM row connects to a common *wordline* via *access* transistors. A *bitline* links a column of DRAM cells to a sense amplifier, facilitating data access and manipulation.

The two-dimensional array of DRAM cells is often split into
several DRAM *subarrays* [46, 54, 56]. Implementing a DRAM
bank as a monolithic structure requires very bitlines to connect
the row buffer to all the rows in the bank, which can significantly
increase the access latency. Instead, a bank consists of multiple
subarrays, each with its own local row-buffer. Subarrays within
a bank share i) a global row-address decoder and ii) a set of
global bitlines which connect their local row-buffers to a global
row-buffer.

<figure><img src="image_2.png" /><figcaption>Figure 1: DRAM Organization</figcaption></figure>

## 2.2. DRAM Timing Parameters

The memory controller accesses DRAM locations via three
primary steps. First, the memory controller sends an activate
(ACT) command to open a closed row within a bank, preparing
the row for column access. Second, the memory controller
issues a read (RD) or write (WR) command to read from or
write to a column in the row, respectively. Third, once all
column operations on the open row are completed, the memory
controller issues a precharge (PRE) command that closes the
row and readies the bank for a new activation.

DRAM cells are inherently leaky and lose their charge over
time due to charge leakage in the access transistor and the stor-
age capacitor. Thus, to maintain data integrity, the memory con-
troller periodically refreshes each row in a time interval called
*the refresh window* ($t_{REFW}$) (32ms for DDR5 [24] and 64ms
for DDR4 [25]). To ensure all rows are refreshed every $t_{REFW}$,
the memory controller issues REF commands with a time inter-
val called *the refresh interval* $t_{REF1}$ (3.9µs for DDR5 [24] and
7.8µs for DDR4 [25]).

To ensure correct DRAM operation, the memory controller
must adhere to standardized timings between consecutive com-
mands, known as *timing parameters* [10, 11, 55]. These parame-
ters ensure the internal DRAM circuitry has adequate time to
execute the operations required by the command.

## 2.3. Memory Controller

A memory controller manages the data exchange between the
CPU and the memory system. It plays a role in ensuring effi-
cient memory utilization, preserving system performance, and
maintaining overall system stability. The memory controller's
responsibilities include:

**Arbitration of Requests.** In systems with multiple master devices (e.g., processors), the memory controller arbitrates memory access requests from these various sources. The arbitration process is critical to avoid contention and ensure fair and efficient access to shared memory resources.

**Physical to DRAM Address Translation.** The memory controller transforms physical memory addresses (e.g., provided by the MMU in the CPUs) into DRAM addresses which consist of channels, ranks, banks, rows, and columns.

**Policy-Based Request Servicing.** The memory controller employs scheduling algorithms to determine the order in which the buffered requests are serviced. These algorithms, such as First Ready, First Come, First Serve (FR-FCFS) [57,58], aim to maximize memory throughput while maintaining fairness among multiple requesters. The memory controller must consider the state of the DRAM, timing constraints, and the type of requests while deciding the servicing order.

**Issuing DRAM Commands.** The memory controller determines the correct sequence of DRAM commands to serve requests and issues them through the DDRx Physical Interface.

The interaction of these functions within a memory con-
troller is complex, with each aspect impacting system perfor-
mance. Therefore, the design and implementation of memory
controllers require carefully considering these elements to bal-
ance system speed, energy use, and fair memory access.

## 3. Motivation

Evaluating the system-wide benefits of DRAM techniques is non-trivial as they require modifications across multiple layers of the compute stack (as described in §1). Table 1 presents a comparison of existing platforms that could be used to evaluate system integration of DRAM techniques. We find FPGA-based platforms to be a good starting point in evaluating DRAM techniques for three reasons. First, interfacing with real DRAM chips ensures the findings are relevant to real-world applications. Second, fast evaluation speeds enable rapid exploration of end-to-end systems. Third, existing open-source system design libraries [37,59] allow easy system configuration to match user needs.

However, an FPGA-based system evaluation has two challenges. First, an FPGA design is hard to model and extend because it requires hardware design expertise. Second, an FPGA design *cannot* accurately accommodate a modern system with high design complexity. For example, an FPGA implementation of a processor usually has a clock frequency of a few hundred MHz [59, 60] (an order of magnitude slower than a processor tapeout [59]), while the DRAM chip works at full capacity, e.g., 3200MHz [24]). Figure 2 depicts a sketch that compares a hypothetical real system with two FPGA systems to visualize the evaluation errors caused by the clock frequency discrepancy. Here, the x-axis shows the time taken for each section and the y-axis shows different evaluation systems.

We make three observations from Figure 2. First, FPGA sys-
tems take longer to execute the same number of processor and
scheduling cycles compared to a real system. Second, software
memory controllers proposed in this paper take considerably
<figure><img src="image_3.png" /><figcaption>Figure 2: Execution timeline of different systems</figcaption></figure>

more time than RTL memory controllers. Third, although pro-
cessor evaluation times take longer, the DRAM access latency
stays constant as all of these systems use the same real DRAM
chips.

Based on these observations we outline that FPGA-based
systems are susceptible to accuracy problems as they combine
real DRAM chips with less capable processors. To solve this
accuracy problem and evaluate realistic systems (i.e., accurate
performance of components), we propose *time scaling*. Time
scaling maintains the relative execution time between process-
ing, scheduling, and main memory. For the FPGA + Time
Scaling example in Figure 2, time scaling preserves the relative
real system performance of each component by 1) allowing the
processor to execute extra cycles ① and 2) inducing additional
latency to the main memory response ②. This modification
allows the configuration of varying performance constraints
(e.g., clock frequency) between system components even if
their FPGA implementations *cannot* support it. We provide
further details on time scaling in §4.3.

4. EasyDRAM

End-to-end evaluation of DRAM techniques using real DRAM chips requires modifications to multiple levels of the compute stack (e.g., memory controllers, memory mapping and allocation in the operating system, and user applications). To ease the evaluation process, we propose EasyDRAM, a framework that allows rapid implementation and accurate end-to-end evaluation of DRAM techniques with real DRAM chips.

**Overview.** We design EasyDRAM to 1) be easy to use, 2) allow evaluation on real DRAM chips, and 3) provide *accurate* system performance results. Figure 3 depicts a high-level overview of EasyDRAM in a simplified computing system.

<figure><img src="image_4.png" /><figcaption>Figure 3: EasyDRAM high-level overview</figcaption></figure>

EasyDRAM is implemented between the last-level cache and
the main memory. EasyDRAM has three key ideas that enables
our design goals.

① **Programmable Core.** EasyDRAM implements a pro-
grammable core as a memory controller that allows easy mod-
ifications with software (i.e., C++) changes without requiring
hardware description language (HDL, e.g., Verilog) expertise.

② Operating Real DRAM Chips. EasyDRAM interfaces with real DRAM chips and provides an easy-to-use API to the user that allows issuing arbitrary DRAM commands with arbitrary delays between any two consecutive commands.

③ **Time Scaling.** EasyDRAM overcomes the accurate performance evaluation limitations of FPGA-based systems with *time scaling*. Time scaling 1) separates hardware components (e.g., memory controller) to different emulation domains and 2) provides fine-grained control over the emulation duration and emulation rate of each domain. By doing so, EasyDRAM emulates each hardware component to match user performance constraints (e.g., processor clock frequency).

**4.1. Programmable Memory Controller**

Implementing new DRAM techniques requires invasive mod-
ifications to memory controller design. Existing frameworks
do not provide the means to easily make these changes. For
example, extending FPGA-based emulators [22, 37] to evaluate
DRAM techniques are time consuming for three reasons. First,
extending FPGA-based emulators require HDL expertise and
evaluation shortcuts are usually harder to implement (e.g., mod-
eling the high-level behaviour of a component instead of fully
implementing the component). Second, FPGA-based emulators
need to go through lengthy synthesis, floor-planning, place-
ment, and routing steps even for the smallest changes. Third,
FPGA-based emulators require components to meet timing con-
straints. As such, evaluating DRAM techniques using HDLs
and FPGA-based emulators have relatively high overheads at
multiple stages (e.g., design, debugging, emulation) of the eval-
uation process. To overcome these challenges, EasyDRAM
implments and uses a fully programmable general-purpose
processor instead of a conventional RTL memory controller.
This way, memory scheduling decisions or DRAM operations
are expressed in software without extensive HDL expertise, and
results are observed by operating real DRAM chips. Listing 1
presents a simple memory controller modeled using C++ in
EasyDRAM.

In this example, the memory controller checks for incom-
ing requests (line 3), operates the DRAM chip (lines 6-8), and
responds with the obtained data (lines 9-10). EasyDRAM pro-
vides an easy means to perform low-level operations such as
buffering the requests and executing the DRAM commands.

**Challenges of Implementing a Programmable Memory Controller.** Using a programmable core within an FPGA-based system has two challenges. First, DRAM operation requires obeying strict timing parameters. Software running on a programmable core (i.e., software memory controller) *cannot* issue DRAM commands at nanosecond granularity. The software memory controller issues a DRAM command each time it processes a memory request. The programmable core executes hundreds of instructions to process a memory request and issue a DRAM command, which may take hundreds of clock
<caption>Listing 1: A read only memory controller for EasyDRAM.</caption>

```c
void read_only_controller() {
    while (true) {
        while (req_empty()); // Wait for a request to arrive
        Request req = get_request(); // Move the request from hardware
                                      // buffers to software memory
        if (req.type == RequestType::WRITE) return; // Ignore write
                                                    // requests for this example
        auto addr = get_addr_mapping(req_addr); // Translate address to
                                                    // rank, bank, row, and column
        read_sequence(addr); // Send DRAM Bender instructions to
                              // read the data
        flush_commands(); // Signal DRAM Bender to execute the
                          // instructions
        rdback_cacheline(req_data); // Move the data from DRAM
                                   // Bender buffers to software memory
        enqueue_response(req); // Send request response to the memory
                                // bus
    }
}
```

cycles. Second, a software memory controller can receive requests every few workload processor (i.e., processors that execute a benchmark for performance evaluation) clock cycles for memory-intensive workloads. The programmable core’s request queue remains full for a long time (because the software memory controller can take hundreds of programmable core cycles to process a memory request). As such, the workload processor *cannot* execute memory instructions for hundreds of processor clock cycles, thereby reducing evaluation accuracy due to experiencing more stalls compared to a real system.

EasyDRAM addresses these two challenges. First, EasyDRAM integrates DRAM Bender, the state-of-the-art open-source DRAM testing platform, into the system with a decoupled interface to the programmable core. Second, EasyDRAM uses *time scaling*, which decouples the programmable core from system components that generate memory requests (e.g., the workload processor) to satisfy fine-grained emulation rates between these components. The following section provides a deeper look into the challenges and explains how we apply these key ideas to overcome the challenges.

## 4.2. Operating Real DRAM Chips

DRAM operation requires timely execution of DRAM commands. Modern systems include fast memory controllers that issue commands to the DDRx interface. Performing these actions with the software memory controller is challenging for two reasons. First, the software memory controller executes hundreds of instructions to produce a DRAM command. Therefore, the software *cannot* quickly (i.e., every few nanoseconds) decide on commands to be issued. Second, general-purpose programmable cores (e.g., RISC-V cores) on FPGAs have clock frequencies of a few hundred MHz [59, 60]. Therefore, the programmable controller *cannot* reliably operate the high-speed DDRx interface (e.g., issuing a command every 1.5 nanoseconds). These two requirements significantly increase software and hardware design requirements (e.g., FPGA processor design with GHz clock frequency).

EasyDRAM overcomes these challenges by coupling the programmable core with DRAM Bender [23], the state-of-the-art DRAM testing platform. DRAM Bender allows users to issue DRAM commands using the DRAM Bender instruction set architecture (ISA). EasyDRAM introduces a command buffer shared between the programmable core and DRAM Bender. The software memory controller fills this buffer with DRAM Bender instructions to issue DRAM commands. As an example, Figure 4 presents how an in-DRAM copy technique (i.e., RowClone [56]) can be implemented in a simplified depiction of this system.

<figure><img src="image_5.png" /><figcaption>Figure 4: High level Programmable core and DRAM Bender execution model</figcaption></figure>

The user implements a program ① that copies the content of a row to multiple target rows. The software memory controller running on the programmable core offloads the DRAM commands and their timings to DRAM Bender as DRAM Bender instructions ② and the *execute* call triggers DRAM Bender to start issuing commands to DRAM ③.

The software memory controller generates the DRAM commands and waits for DRAM Bender to execute the commands. Therefore, EasyDRAM’s DRAM operation contains two types of time-slices: 1) DRAM is idle (i.e., generating the DRAM commands) and 2) DRAM is operated in real-time (i.e., DRAM Bender execution). The high command scheduling latency of the software memory controller and overheads of being coupled with DRAM Bender (e.g., transferring DRAM commands) need to be considered for realistic system evaluation. To do so, EasyDRAM implements *time scaling*.

## 4.3. Time Scaling

The software memory controller runs significantly slower than a hardware memory controller. We identify two problems with the evaluation accuracy of a slow memory controller. First, serving a memory request slower causes additional memory requests to arrive and potentially change subsequent scheduling decisions (e.g., keeping a DRAM row open). Second, a processor can execute additional instructions due to increased memory access latency. Both of these problems reduce evaluation accuracy because the executed instruction traces and issued DRAM commands as the result of differing scheduling decisions do *not* faithfully emulate a real system.

For the first problem, we mark the arrival workload processor emulation clock cycle of a request and compare it to the software memory controller’s emulation clock cycle. As such, additional requests that arrive due to increased scheduling latency are ignored. For the second problem, we divide system into different emulation domains and allow fine-grained control of the progression of each domain. By doing so, EasyDRAM can stop a processor from executing additional instructions due to the increased memory access latency of a software memory controller.
<figure><img src="image_6.png" /><figcaption>Figure 5: Time scaled execution of EasyDRAM at different emulation steps</figcaption></figure>

To control the emulation rate of different hardware components, we propose a mechanism called *time scaling*. Time scaling aligns the emulation rate of each domain to the slowest progressing domain (i.e., the domain with the most discrepancy between its real and FPGA clock frequency). To do so, time scaling stalls a domain that executes ahead, thereby providing slower domain extra cycles to satisfy the desired system clock frequency requirement. EasyDRAM implements *time scaling counters* to track the emulation point (i.e., emulated clock cycle) for each emulation domain (e.g., workload processors). By doing so, EasyDRAM accurately tracks each emulated domain's emulation rate and data transfers.

Figure 5 visualizes EasyDRAM's state at various stages of the emulation process. First, (a) we show the labels of critical components and time scaling counters in a simplified system version where bus latencies and the cache hierarchy are removed for clarity. The time scaling counters for the workload processors (WP) and the software memory controller (SMC) are used to track the progress of their respective emulation points. The global counter serves as a reference timer and counts clock cycles from the system power-on. We note that all WPs share the same time scaling counter, and synchronization actions (e.g., clock-gating) affect all processors.

As the emulation starts (b), all counters are initialized to 0 and no requests or responses exist. Time scaling counters are incremented every cycle while the system has no unresolved memory requests. As the emulation proceeds (c), WP makes the first main memory request (1). The request is tagged with the current WP time scaling counter value and the WPs get clock-gated (colored gray in Figure 5), stopping their execution until the memory request is processed. When SMC detects the request (d), it enters *critical mode*. In critical mode, SMC locks the WP time scaling counter such that the WP time scaling counter *cannot* emulate ahead of the SMC time scaling counter. By doing so, the SMC controls the WP emulation rate to match the targetted clock frequency constraints. Then, SMC transfers the incoming request from hardware buffers to software memory (2). SMC makes a scheduling decision, creates a row activate (ACT) command, and sends the command to DRAM Bender (3). Once DRAM Bender finishes execution (e), SMC obtains the time spent for the ACT command (4),

calculates the cycles the WP domain should execute at the emulated system's clock frequency, and advances the SMC domain time scaling counter by that value (5). Updating the time scaling counter causes the WPs to become active and emulate the missing time scaled duration (f) where WPs send two new requests (6). When the WP time scaling counter reaches SMC time scaling counter (g), SMC transfers new requests from hardware buffers to software memory (7). Then, SMC makes the next scheduling decision and issues a read command (8). After executing the read command (h) SMC obtains the data and time spent for execution (9). The SMC tags the response with the WP time scaling counter value the response will become visible and writes it to the hardware response buffer (10). The duration spent on the scheduling is converted to the number of emulation cycles at the emulated system's clock frequency and the SMC time scaling counter is updated (11). As we conclude our example, the synchronizing WPs do *not* send new requests and SMC issues another read command (12) to the open row from the previous scheduling.

The system runs similar to our example as long as there are requests in the hardware buffer or SMC is in critical mode (e.g., SMC has *not* responded to all requests for our implementation). When critical mode ends, the time scaling counters synchronize and WPs continue their normal execution.

## 4.4. Lifetime Of A Request

Figure 6 outlines the general execution of a memory request.

<figure><img src="image_7.png" /><figcaption>Figure 6: Lifetime of a memory request with EasyDRAM</figcaption></figure>

The workload processor (WP) initiates a read request that requires access to main memory (1). This request is placed into the programmable core's hardware buffers (2) and the WP emulation is stopped (3). The software memory controller (SMC) implements a loop that i) checks for new requests, ii) makes a scheduling decision, and, iii) handles responses from DRAM. When a request is found, the SMC enters critical mode (4).
<figure><img src="image_8.png" /><figcaption>Figure 7: The system overview of EasyDRAM</figcaption></figure>

and transfers the new request from the hardware buffers to the software request table ⑤. The SMC makes a scheduling decision and transfers the DRAM commands to DRAM Bender ⑥. DRAM Bender executes the DRAM commands ⑦, then returns the data and the number of cycles taken for execution ⑧. The SMC finalizes the memory request response and transfers the response to the request buffers ⑨. Lastly, the SMC unlocks time scaling counters by advancing the MC time scaling counter ⑩ and exists critical mode. As the execution continues, the WP time scaling counter reaches the response's time scaled point. The response is transferred to the memory bus ⑪, completing the request's lifetime by reaching the processor ⑫.

## 5. EasyDRAM Implementation

Our EasyDRAM implementation takes advantage of agile hardware development platforms. Particularly, the Chipyard Framework [37]. Chipyard facilitates the design and evaluation of full-system hardware in an agile environment. We use Chisel [61] as our primary HDL.

**System Implementation.** We utilize Chipyard for two key benefits. First, Chipyard contains a diverse selection of processors and offers high system configurability. This variety provides a strong base for modifications and experimentation. Second, updates to Chipyard enhance the system feature availability, ensuring that our EasyDRAM implementation has access to future advancements. Our default EasyDRAM system implements the Berkeley Out-of-Order RISC-V Processor (BOOM) [62] as the workload processor.

**Programmable Controller.** We use Rocket [60] as our programmable core. We modify the Rocket Core by separating the core from the *RocketTile* and disabling virtual memory support. Then, we connect its memory ports to EasyTile's interconnect. These changes enable Rocket to meet our low-latency programmable core design requirements.

### 5.1. EasyDRAM's Hardware Design (EasyTile): Overview

Figure 7 shows an overview of the main components of EasyDRAM and its configuration for a dual-core system.

We pack the programmable core and DRAM Bender with various helper components of EasyDRAM inside a module we call EasyTile (Figure 7, right). EasyTile includes the following hardware components.

**Programmable Core ④.** The programmable core executes a

program that implements a software memory controller. These programs use the on-tile software memory ⑨ to store instructions and data. Section 5.2 describes the proposed API that simplifies programming a software memory controller.

**DRAM Bender ②.** DRAM Bender allows users to issue DRAM commands without dealing with the low-level DRAM interface (e.g., DDRx interface ①). Programs running on the programmable core use the DRAM Bender's interface to implement DRAM operations and techniques.

**Tile Control Logic ⑤.** Tile Control Logic allows the programmable core to offload common memory controller operations. The two most important operations are interacting with the Memory Bus Interface ⑤ (i.e., receive and respond to main memory requests) and transferring commands and data between DRAM Bender and hardware buffers.

**Command Buffer ⑦.** The command buffer stores and accumulates multiple DRAM commands before they are issued to the DRAM chip in a *timing-preserving* batch. The delay between each DRAM command in a batch is executed exactly as intended by the EasyDRAM user.

**Readback Buffer ③.** Data read from DRAM and returned to EasyDRAM by DRAM Bender is placed into this buffer. The programmable core receives the necessary data from the readback buffer for the responses of main memory requests.

**Request and Response Buffers ⑥.** EasyTile implements two buffers that store main memory requests and their responses until they are processed. The *Tile Control Logic* automatically inserts incoming main memory requests and sends responses as they become valid.

### 5.2. EasyAPI

EasyDRAM contains various hardware and software components. A programmer orchestrates these components to evaluate DRAM techniques. We introduce the user-friendly EasyAPI to simplify development with EasyDRAM. Table 2 shows some of the most commonly used library functions of EasyDRAM. We categorize EasyAPI's features into two categories: hardware (top) and software (bottom).

**Hardware Abstraction Library.** The software memory controller interfaces with various hardware modules to update the system state. For example, a conventional memory controller (e.g., FRFCFS [57]) implementation in EasyDRAM transfers requests between the hardware buffers and the software memory or updates the time scaling status to emulate the frequency
constraints requested by the user. Hardware library allows easy interaction with the underlying hardware modules.

**Software Library.** The software memory controller implements i) DRAM techniques and ii) a memory controller to serve requests by operating the DRAM module. The programmer may only want to evaluate a DRAM technique or require minimal changes to an existing memory controller design. Software library streamlines the design of these two components by providing several DRAM techniques (e.g., RowClone, $t_{RCD}$ reduction) and memory controller (e.g., FCFS, FRFCFS) implementations.

## 6. Time Scaling Validation

**Simulation.** We validate EasyDRAM's time scaling against an RTL reference system that is simulated at a 1 GHz system clock frequency (i.e., same clock for processor, busses, and caches). The reference system implements EasyDRAM's memory controller in hardware, i.e., the reference system's memory controller takes the same scheduling decisions with EasyDRAM's memory controller but does *not* require time scaling to satisfy target system configuration memory scheduling latency requirements. We configure EasyDRAM to time scale a processor simulated at 100 MHz clock frequency to 1 GHz. We evaluate 28 benchmarks from PolyBench [45] and the memory read latency benchmark from Imbench suite [63].<fnref n="1" /> Our validation against a 1 GHz RTL reference system shows that EasyDRAM's time scaling has low average (<0.1%) and maximum (<1%) execution time and memory latency inaccuracy across 29 evaluated microbenchmarks.

**Real System.** We extend Chipyard v1.8.1 with our implementation of EasyDRAM. We use a Xilinx VCU108 [64] board with a DDR4 main memory.<fnref n="2" /> For our experiments, we configure EasyDRAM to target an NVIDIA Jetson Nano SoC [44]. We adjust both EasyDRAM and BOOM core parameters to mirror the corresponding ARM Cortex A57 CPU and memory hierarchy. Figure 8 shows the latency profile in terms of processor cycles

of the real board (Cortex A57) and two versions of EasyDRAM: before (EasyDRAM - Default) and after (EasyDRAM - Time Scaling) calibration. We obtain the latency profiles after executing a microbenchmark similar to the Imbench suite's memory read latency benchmark [63]. The figure indicates the access latency for reading data from the L1D cache, L2 cache, and main memory. Note that we maintain the L2 memory size at 512 KiB, instead of the 2 MiB of the Jetson Nano, to achieve a higher main memory activity.

<figure><img src="image_9.png" /><figcaption>Figure 8: Average cycles per load benchmarked for increasing data sizes on a real ARM Cortex A57 compared to EasyDRAM - Default and EasyDRAM - Time Scaling target configurations</figcaption></figure>

From Figure 8 we observe that, EasyDRAM - Default (i.e., a system emulation that does *not* faithfully model a modern processor) has significantly lower main memory access latency compared to a real system (i.e., a system with a Cortex A57 processor). This is because the less-capable processor has slower clock speed and, thus, executes fewer cycles while the request is served from the main memory. In contrast, EasyDRAM - Time Scaling (i.e., a system emulation that mimics a board with Cortex A57) has a similar memory latency compared to a real system. We conclude that FPGA evaluation methodologies that do *not* faithfully model a modern processor fail to capture the effects of main memory accesses. For the remainder of our evaluation, we utilize EasyDRAM in these two configurations.

## 7. Case Study: Realistic performance evaluation of RowClone

RowClone [56] is a DRAM technique that copies a row's content to another row. A state-of-the-art end-to-end processing-in-memory system evaluation framework (PiDRAM [22]) shows the programming challenges of using RowClone in an end-to-end computing system. PiDRAM evaluates the performance impact of RowClone using an FPGA-based in-order CPU implementation at 50 MHz against a real DDR3 chip running at

<table><caption>Table 2: Commonly used hardware and software library functions</caption><thead><tr><td>Function Name</td><td>Description</td></tr></thead><tbody><tr><td>set_scheduling_state(bool state)</td><td>Set critical mode register</td></tr><tr><td>get_request()</td><td>Read a request from the hardware request buffer</td></tr><tr><td>ddr_activation(/precharge()/read()...)</td><td>Insert a DRAM command to the command batch</td></tr><tr><td>flush_commands()</td><td>Execute the DRAM commands in the command batch</td></tr><tr><td>add_request(Request& req)</td><td>Insert a request to the software request table</td></tr><tr><td>FRFCFS::schedule()</td><td>Select a request with the FRFCFS scheduler</td></tr><tr><td>FCFS::schedule()</td><td>Select a request with the FCFS scheduler</td></tr><tr><td>rowclone(Address src, Address dst...)</td><td>Insert a RowClone command sequence to the command batch</td></tr></tbody></table>

<footnote n="1">¹We open-source our validation reference system and microbenchmarks to enable reproducibility at https://github.com/CMU-SAFARI/EasyDRAM.</footnote>

<footnote n="2">²EasyDRAM can support additional boards with new top-level FPGA wrap- ers (referred to as an fpga-shell or a harness [37]) for Chipyard. The develop- ment effort needed to port EasyDRAM to a different FPGA board with similar I/O capability is very small as all EasyDRAM components are reusable across such FPGA boards. A user interested in porting EasyDRAM to a different FPGA board *only* needs to make the correct top-level design I/O to FPGA pin assignments for their board.</footnote>
800 MT/s. PiDRAM's CPU capabilities are significantly limited against a real DRAM chip. Therefore, PiDRAM's evaluation is skewed in favor of RowClone. To this end, we re-evaluate RowClone by leveraging the capabilities of EasyDRAM where we configure it to emulate the memory latency characteristics of a Cortex A57 chip (§6).

## 7.1. Data Mapping

We explain the limitations and requirements of the RowClone using Figure 9. Given that RowClone is strictly limited to intra-subarray row operations, the source and target of data the operation must satisfy the following four constraints: 1) data must be aligned to row boundaries (alignment problem ①), 2) data size must be a multiple of the DRAM row size (granularity problem ②), 3) both the source and target DRAM rows must share the same DRAM subarray (mapping problem ⑤), and 4) data must be resident in memory (coherency problem).

<figure><img src="image_10.png" /><figcaption>Figure 9: Showcase of RowClone limitations in a DRAM chip with 2 banks and 2 subarrays. Only operation ④ can succeed as its operands satisfy all constraints. Reproduced from [22].</figcaption></figure>

These restrictions require careful data organization to utilize the full potential of RowClone. We demonstrate how programmers can easily satisfy these constraints using EasyDRAM.

**Alignment Problem.** We program the workload processor to reserve whole DRAM rows for RowClone operations. EasyAPI includes a set of physical-to-DRAM address mappers that allow both the workload processor and software memory controller to convert an address into a bank, row, column triplet, and vice versa.

**Granularity Problem.** We configure the workload processor to select bank and row pairs that accommodate the required size. The algorithm could choose to interleave its rows across multiple banks to improve throughput, we leave the exploration and optimization of such approaches for future work.

**Coherency Problem.** Our workload processors do not implement an instruction to evict or flush cache lines. We provide the programmer with a memory-mapped register that flushes a target cache line to main memory. This feature is used to flush the content to DRAM before initiating a RowClone operation and ensuring data coherency across the system.

**Mapping Problem.** A workload processor finds a clonable row pair in five steps: 1) choose two rows within a bank, 2) initialize the rows with a known data pattern, 3) flush the rows to DRAM, 4) perform a RowClone operation, and 5) check the contents of both rows to ensure the operation is successful.

**Source and Target Row Allocation.** A workload can perform a bulk copy or initialization operation that *cannot* be served by a single source and destination row address pair. Thus, we distribute a bulk copy operation's source and destination rows across multiple subarrays. To accommodate bulk data initialization operations that span multiple subarrays, we allocate one source row in each subarray. If a RowClone operation between a source row and a destination row *cannot* be done successfully, EasyDRAM falls back to regular CPU-based read and write operations (i.e., uses load and store instructions).

## 7.2. Evaluation and Results

We implement and evaluate RowClone using EasyDRAM and a state-of-the-art cycle-level software memory simulator, Ramulator 2.0 [32, 65]. We evaluate two EasyDRAM configurations. 1) EasyDRAM - Default, where we disable time scaling (i.e., workload processor executes at 50 MHz) and 2) EasyDRAM - Time Scaling, where workload processors are emulated as Cortex A57 cores. EasyDRAM - Default configuration results allow us to compare the accuracy of time scaling to prior evaluation methodologies (e.g., [22]). We configure Ramulator 2.0 to faithfully model EasyDRAM's memory system.<fnref n="3" />

We execute two workloads: *Copy* and *Init*. Both programs take an argument *N*, where *Copy* replicates an *N*-byte source array into another *N*-byte destination array, and *Init* initializes an *N*-byte array with a predetermined pattern. Each program comes in two variants: 1) *CPU-copy* duplicates or initializes data by using load and store instructions, and 2) *RowClone* uses in-DRAM copy operations to perform copy and initialization.

We evaluate these two workloads in two settings. In the first setting, *No Flush RowClone*, the source array's data is already present in DRAM. In the second setting, *CLFLUSH RowClone*, the source array's data has cached copies in the processor that must be written back to DRAM. The first setting demonstrates RowClone's optimal performance by eliminating any setup overhead associated with the operation (e.g., flushing caches), and the second setting provides a common case performance study.

**No Flush RowClone.** Figure 10 presents the execution time of No Flush RowClone for Copy (a) and Init (b) workloads on increasing array sizes, normalized to each configuration's CPU baseline that copies or initializes data with load and store instructions. Axes respectively show the size of the carried operation in bytes (x axis) and the execution time speedup (y axis) normalized execution time. Different bars identify the EasyDRAM - Default (50 MHz processor), EasyDRAM - Time Scaling (Cortex A57 emulation), and Ramulator 2.0.

We make three observations. First, without time scaling, RowClone Copy and Init improve the average (maximum) system throughput across different copied data sizes by 306.7x (423.1x) and 36.7x (51.3x), respectively. Second, with time scaling average (maximum) improvements drop for Copy and Init to 15.0x (17.4x) and 1.8x (2.0x), respectively. We attribute

<footnote n="3">³Ramulator 2.0's processor system model (a simple out-of-order core and a last-level cache) significantly differs from that of EasyDRAM's real processor system implementation.</footnote>
<figure><img src="image_11.png" /><figcaption>**Figure 10: RowClone Copy (a) and Init (b) execution time speedup normalized to a baseline system that copies or initializes the data with load and store instructions**</figcaption></figure>

the significant decrease in RowClone's benefits to time scaling more faithfully representing a modern CPU's performance, thereby strengthening the baseline performance the RowClone is compared against. Third, with Ramulator 2.0, RowClone Copy and Init improve the average (maximum) system throughput across all tested data sizes by 27.2x (33.0x) and 17.3x (21.0x), respectively. All source and destination row pairs can successfully perform RowClone operations in Ramulator 2.0 simulations. We attribute Ramulator 2.0's higher speedup than EasyDRAM - Time Scaling for Init to the lack of real DRAM characterization in software simulation. Software simulation fails to capture the overhead of falling back to CPU initialization for rows that *cannot* be initialized with RowClone.

**CLFLUSH RowClone.** Figure 11 presents the normalized execution time speedup of CLFLUSH RowClone for copy (a) and initialize (b) operations on increasing array sizes. Axes respectively show the size of the carried operation in bytes (x axis) and the execution time speedup (y axis) normalized to each configuration's CPU baseline without using RowClone. Different bars identify the EasyDRAM - Default (50 MHz processor), EasyDRAM - Time Scaling (Cortex A57 emulation), and Ramulator 2.0.

<figure><img src="image_12.png" /><figcaption>**Figure 11: RowClone Copy (a) and Init (b) speedup normalized to a baseline system that copies or initializes the data with load and store instructions**</figcaption></figure>

We make five observations. First, with and without time scaling, RowClone Copy improves the average (maximum) system throughput across all data sizes by 4.04x (6.62x) and 3.1x (4.83x), respectively. Second, with and without time scaling, RowClone Init improves the average (maximum) system throughput for data sizes above 256KB by 1.19x (1.34x) and above 32KB by 3.09x (4.08x), respectively. RowClone Init with and without time scaling degrade system performance for array sizes that are $\le 256$ KB and 32 KB, respectively. Third, when compared to No Flush RowClone (Figure 10), the copy and initialize operations on small source data array sizes suffer significantly as the dirty cache blocks need to be flushed to DRAM

for data coherency. Coherency overheads are less evident in large data sizes because dirty block flush operations overlap with more data accesses. Fourth, as the source data array size increases, CLFLUSH RowClone system performance benefits similarly increase. Five, Ramulator 2.0 yields a different speedup profile than EasyDRAM. We attribute the difference to the difference in processor systems modeled by Ramulator 2.0 and EasyDRAM.

Based on our experiments, we conclude that evaluation frameworks that do *not* faithfully model a modern processor (e.g., >1 GHz clock frequency) report relatively high benefits in favor of DRAM techniques with large error margins (e.g., >20x). We note that RowClone still improves the system performance when compared to a modern CPU baseline, especially at large source data array sizes (e.g., >1 MiB).

# 8. Case Study: Access Latency Reduction

Several techniques have been proposed in prior work [4–11] to improve the main memory latency. To understand the system-wide benefits of these approaches we implement and evaluate Row-to-Column command Delay ($t_{RCD}$) timing parameter reduction with EasyDRAM.

## 8.1. $t_{RCD}$ Reduction Implementation

We implement $t_{RCD}$ reduction in two stages. First, we profile DRAM to determine the rows that serve correct data under reduced $t_{RCD}$ values (strong DRAM rows). Second, we extend our memory controller to intelligently access strong DRAM rows with the reduced timing parameter to reduce DRAM access latency.

**DRAM Characterization.** We perform our DRAM characterization using both the workload processor and the software memory controller. We extend the software memory controller to support a new type of request called *profiling request* that tests a cache line for a given $t_{RCD}$ value. The software memory controller serves a profiling request in three steps: 1) initialize the target cache line with a known data pattern, 2) access the target cache line using the requested $t_{RCD}$ value, and 3) report to the workload processor whether the reduced value resulted in correct access. During execution, the software memory controller serves both main memory and profiling requests and the workload processor iterates over banks, cache lines, and rows to test while keeping track of the locations that can be accessed reliably under reduced $t_{RCD}$ values.

Figure 12 showcases a profiling heatmap for the first two banks across 4K rows per bank.

We make three observations from this figure. First, the percentage of strong DRAM rows (84.5%) significantly exceeds the weak DRAM rows (15.5%). Second, weak DRAM rows are clustered within specific banks and areas. Third, even the weakest DRAM rows can be reliably operated at lower $t_{RCD}$ values than the manufacturer recommends. We conclude that a vast number of rows in DRAM *can* be operated faster to reduce the memory latency.
<figure><img src="image_13.png" /><figcaption>Figure 12: Minimum reliable $t_{RCD}$ of rows across two banks</figcaption></figure>

<figure><img src="image_14.png" /><figcaption>Figure 13: Execution time speedup of $t_{RCD}$ reduction</figcaption></figure>

## 8.2. Scheduler Implementation

Storing the minimum $t_{RCD}$ value of all cache lines is *not* scalable as DRAM capacity increases (e.g., >64 GiB). We introduce a two-part strategy to address this challenge. First, we identify the weakest cache line in each row and use its $t_{RCD}$ value to represent the whole row. This summarizes the profiling information while capturing the variability across rows. Second, we implement a bloom filter in the software memory controller that tracks weak DRAM rows. We use weak rows as keys for our bloom filter. Therefore, bloom filter false positives do *not* risk system reliability. We generate the bloom filter and the hash functions on the host machine and load them to software memory controller before emulation begins. Each time a DRAM row is opened, software memory controller checks this filter to determine appropriate $t_{RCD}$ value to use. These two steps enhance the applicability of $t_{RCD}$ reduction by greatly reducing the storage requirement and access latency of the DRAM technique.

We study the evaluation speed of EasyDRAM and Ramulator 2.0. Figure 14 presents the evaluation speed of EasyDRAM and Ramulator 2.0 across benchmarks. The x-axis shows the evaluated workload, and the y-axis shows the evaluation speed in MHz.

<figure><img src="image_15.png" /><figcaption>Figure 14: EasyDRAM's emulation speed across benchmarks</figcaption></figure>

## 8.3. Evaluation and Results

We implement and evaluate $t_{RCD}$ reduction using EasyDRAM and Ramulator 2.0 [32, 65]. For EasyDRAM, we leverage the fast emulation capabilities of EasyDRAM and run workloads end-to-end to study system behavior. For Ramulator 2.0, we generate traces of workloads and simulate each workload for 500M instructions. We evaluate workloads from the large dataset configuration of PolyBench [45], a suite of benchmark kernels widely used in high-performance computing.

Figure 13 presents the execution time normalized to the baseline system *without* $t_{RCD}$ reduction. Here, the x-axis shows the evaluated workload, for each workload the purple (left) and orange (right) bars are EasyDRAM and Ramulator 2.0, respectively. Where the y-axis shows the execution time speedup, normalized to a system with *no* $t_{RCD}$ reduction.

We make two observations from Figure 13. First, with EasyDRAM, $t_{RCD}$ reduction improves performance on average (maximum) by 2.75% (9.76%). Second, with Ramulator 2.0, $t_{RCD}$ reduction improves performance on average (maximum) by 2.58% (7.04%). EasyDRAM and Ramulator 2.0 can yield different results for individual workloads (e.g., correlation) because Ramulator 2.0 1) simulates only a part of the workload and 2) implements a simple out-of-order processor core model whereas EasyDRAM executes the workload end-to-end and implements a real out-of-order processor core (BOOM [62]).

From Figure 14 we observe that EasyDRAM's average (maximum) evaluation speed is 5.86x (20.25x) faster than Ramulator 2.0. We conclude that EasyDRAM allows easy characterization and fast full-system evaluation of DRAM techniques with using DRAM chips.

## 9. Related Work

We have already provided qualitative comparisons to the most closely related work that could be used to evaluate DRAM techniques [22, 23, 28–32, 38–40]. In this section, we present other related work.

**Programmable memory controllers (e.g., [66, 67]).** These works design high-performance programmable memory controllers. Programmable memory controllers allow dynamic specialization of the memory controller for any running application. These programmable memory controllers (e.g., [66, 67]) are *not* a good platform to evaluate DRAM techniques for three reasons. First, these works have limited programmability because they aim to achieve hardware design timing closure at very high clock speeds. Second, similar to commercial systems, these works do *not* accommodate easy modifications to evaluate varying system configurations. Third, these works require users to write programs using their specialized instruction set architecture.
ture (ISA).⁴ In contrast, EasyDRAM 1) allows every aspect of
the memory controller to be customized, 2) accommodates easy
configuration of multiple components of the system (e.g., cache
sizes and processor capabilities), and 3) can be programmed
with a common high-level language (i.e., C++).

**MEG [68].** MEG is a system emulation infrastructure for near-data processing that uses High-Bandwidth Memory (HBM). MEG implements a soft core as a processor in the FPGA and is designed to efficiently retrieve data from main memory to perform computation in the processor. MEG's goal is *not* to allow for rapid and accurate evaluation of DRAM techniques. However, parts of MEG's system design can be integrated into EasyDRAM to allow for rapid and accurate evaluation of DRAM techniques using real HBM chips.

**ComputeDRAM [19].** ComputeDRAM demonstrates in-memory computation in real off-the-shelf DDR3 DRAM chips. ComputeDRAM describes a software framework built around SoftMC [38] to estimate in-memory computation throughput. This framework is subject to the limitations of SoftMC [38] that are described in Section 1.

**Other related work.** Many prior work propose or demonstrate several DRAM techniques (e.g., processing in memory [19,21,34,56,69–72], access latency reduction [4,11–13,73], variable rate refresh [41, 74, 75]). None of these works provide a framework that can rapidly and accurately evaluate DRAM techniques end-to-end. The techniques proposed by these works can be implemented in EasyDRAM and their performance benefits evaluated, similar to how we do so for RowClone [56] and SolarDRAM [4].

**Clock frequency emulation on FPGAs.** Previous FPGA-based simulators [39, 40] emulate different clock frequencies by throttling parts of the system depending on their targetted simulation speed. This is accomplished by treating all input and output signals as tokens. A module (e.g., the memory controller) only advances a time step (i.e., produces an output token) if it can consume one token from all of its inputs. The token-based method is mainly suited for hardware-based memory controllers and incurs impractical performance overheads for a software memory controller because the programmable core inefficiently processes input and output tokens.

## 10. Conclusion

We develop EasyDRAM, a fast, accurate, and extensible framework for evaluating memory controllers and DRAM techniques in an end-to-end system. Our EasyDRAM prototypes allow easy end-to-end evaluation of new memory controller ideas and DRAM techniques on real DRAM chips.

⁴A compiler could ease the programmability of existing programmable memory controllers. However, EasyDRAM is superior to a compiler for programmable memory controllers (e.g., [66, 67]) in the evaluation of DRAM techniques due to two reasons. First, EasyDRAM provides a flexible and expressive software-defined memory controller that can be used to rapidly evaluate varying DRAM techniques. In contrast, a compiler would *not* improve the expressiveness (e.g., adding new operations such as RowClone) of existing programmable memory controllers to a point where they could match EasyDRAM's. Second, without time scaling, a programmable memory controller is subject to the problems we describe in §3, which prevents accurate evaluation of new and existing DRAM techniques.

With EasyDRAM, we successfully implement and evaluate two ideas: RowClone (an in-DRAM bulk data copy and initialization technique) and $t_{RCD}$ reduction (a method to decrease DRAM access latency). Our RowClone evaluation demonstrates significant improvements in data copy and initialization throughput, and our results show that evaluation frameworks that do *not* take real system frequency discrepancies into account skew results in favor of DRAM techniques. Similarly, our $t_{RCD}$ reduction results show a modest increase in throughput without incurring any additional overheads.

In conclusion, EasyDRAM provides designers with the ability to i) easily implement and evaluate memory controller and DRAM techniques with a C++ programming environment, and ii) conduct reliable studies on end-to-end systems using real DRAM chips.

## Acknowledgments

We thank the anonymous reviewers of ASPLOS 2024 and DSN 2025 for the encouraging feedback. We thank the SAFARI Research Group members for valuable feedback and the stimulating scientific and intellectual environment. We acknowledge the generous gift funding provided by our industrial partners (especially Google, Huawei, Intel, Microsoft, VMware). This work was also in part supported by the Google Security and Privacy Research Award, the Microsoft Swiss Joint Research Center, the ETH Future Computing Laboratory (EFCL), and the F3CAS project (ANR-20-CE25-0010) of the French National Research Agency.

## References

[1] O. Mutlu, "Memory Scaling: A Systems Architecture Perspective," in *IMW*, 2013.

[2] O. Mutlu, S. Ghose, J. Gómez-Luna, and R. Ausavarungnirun, "A Modern Primer On Processing In Memory," in *Emerging Computing: From Devices To Systems - Looking Beyond Moore And Von Neumann*, 2021.

[3] S. Ghose, A. Boroumand, J. S. Kim, J. Gómez-Luna, and O. Mutlu, "Processing-in-memory: A Workload-driven Perspective," *IBM JRD*, 2019.

[4] J. Kim, M. Patel, H. Hassan, and O. Mutlu, "Solar-DRAM: Reducing DRAM Access Latency by Exploiting the Variation in Local Bitlines," in *ICCD*, 2018.

[5] Y. Wang, A. Tavakkol, L. Orosa, S. Ghose, N. M. Ghiasi, M. Patel, J. S. Kim, H. Hassan, M. Sadrosadati, and O. Mutlu, "Reducing DRAM Latency Via Charge-level-aware Look-ahead Partial Restoration," in *MICRO*, 2018.

[6] A. G. Yağlikci, A. Olgun, M. Patel, H. Luo, H. Hassan, L. Orosa, O. Ergin, and O. Mutlu, "HiRA: Hidden Row Activation for Reducing Refresh Latency of Off-the-shelf DRAM Chips," in *MICRO*, 2022.

[7] H. Luo, T. Shahroodi, H. Hassan, M. Patel, A. G. Yağlikci, L. Orosa, J. Park, and O. Mutlu, "CLR-DRAM: A Low-cost DRAM Architecture Enabling Dynamic Capacity-latency Trade-off," in *ISCA*, 2020.

[8] A. Das, H. Hassan, and O. Mutlu, "VRL-DRAM: Improving DRAM Performance via Variable Refresh Latency," in *DAC*, 2018.

[9] H. Hassan, G. Pekhimenko, N. Vijaykumar, V. Seshadri, D. Lee, O. Ergin, and O. Mutlu, "ChargeCache: Reducing DRAM Latency by Exploiting Row Access Locality," in *HPCA*, 2016.

[10] D. Lee, Y. Kim, V. Seshadri, J. Liu, L. Subramanian, and O. Mutlu, "Tiered-latency DRAM: A Low Latency and Low Cost DRAM Architecture," in *HPCA*, 2013.

[11] D. Lee, Y. Kim, G. Pekhimenko, S. Khan, V. Seshadri, K. Chang, and O. Mutlu, "Adaptive-latency DRAM: Optimizing DRAM Timing for the Common-case," in *HPCA*, 2015.

[12] K. K. Chang, A. Kashyap, H. Hassan, S. Ghose, K. Hsieh, D. Lee, T. Li, G. Pekhimenko, S. Khan, and O. Mutlu, "Understanding Latency Variation in Modern DRAM Chips: Experimental Characterization, Analysis, and Optimization," in *SIGMETRICS*, 2016.

[13] K. K. Chang, "Understanding and Improving the Latency of DRAM-based Memory Systems," Ph.D. dissertation, Carnegie Mellon Univ., 2017.

[14] J. Choi, W. Shin, J. Jang, J. Suh, Y. Kwon, Y. Moon, and L.-S. Kim, "Multiple Clone Row DRAM: A Low Latency and Area Optimized DRAM," in *ISCA*, 2015.

[15] K. K. Chang, P. J. Nair, D. Lee, S. Ghose, M. K. Qureshi, and O. Mutlu, "Low-cost Inter-linked Subarrays (LISA): Enabling Fast Inter-subarray Data Movement in DRAM," in *HPCA*, 2016.
[16] Y. H. Son, O. Seongil, Y. Ro, J. W. Lee, and J. H. Ahn, "Reducing Memory Access Latency with Asymmetric DRAM Bank Organizations," in *ISCA*, 2013.

[17] İ. E. Yüksel, Y. C. Tuğrul, F. N. Bostancı, G. F. Oliveira, A. G. Yağlıkçı, A. Olgun, M. Soysal, H. Luo, J. Gómez-Luna, M. Sadrosadati et al., "Simultaneous Many-row Activation in Off-the-shelf DRAM Chips: Experimental Characterization and Analysis," in *DSN*, 2024.

[18] İ. E. Yüksel, Y. C. Tuğrul, A. Olgun, F. N. Bostancı, A. G. Yağlıkçı, G. F. Oliveira, H. Luo, J. Gómez-Luna, M. Sadrosadati, and O. Mutlu, "Functionally-complete Boolean Logic in Real DRAM Chips: Experimental Characterization and Analysis," in *HPCA*, 2024.

[19] F. Gao, G. Tziantzioulis, and D. Wentzlaff, "ComputeDRAM: In-memory Compute using Off-the-shelf DRAMS," in *MICRO*, 2019.

[20] F. Gao, G. Tziantzioulis, and D. Wentzlaff, "FracDRAM: Fractional Values in Off-the-shelf DRAM," in *MICRO*, 2022.

[21] A. Olgun, M. Patel, A. G. Yağlıkçı, H. Luo, J. S. Kim, F. Nisa Bostancı, N. Vijaykumar, O. Ergin, and O. Mutlu, "QUAC-TRNG: High-throughput True Random Number Generation using Quadruple Row Activation in Commodity DRAM Chips," in *ISCA*, 2021.

[22] A. Olgun, J. G. Luna, K. Kanellopoulos, B. Salami, H. Hassan, O. Ergin, and O. Mutlu, "PiDRAM: A Holistic End-to-end FPGA-based Framework for Processing-in-DRAM," *TACO*, 2022.

[23] A. Olgun, H. Hassan, A. G. Yağlıkçı, Y. C. Tuğrul, L. Orosa, H. Luo, M. Patel, E. Oğuz, and O. Mutlu, "DRAM Bender: An Extensible and Versatile FPGA-based Infrastructure to Easily Test State-of-the-art DRAM Chips," *TCAD*, 2023.

[24] JEDEC, *JESD79-5: DDR5 SDRAM Standard*, 2020.

[25] JEDEC, *JESD79-4C: DDR4 SDRAM Standard*, 2020.

[26] JEDEC, *JESD209-4B: Low Power Double Data Rate 4 (LPDDR4) Standard*, 2017.

[27] JEDEC, *JESD209-5A: LPDDR5 SDRAM Standard*, 2020.

[28] N. Binkert, B. Beckmann, G. Black, S. K. Reinhardt, A. Saidi, A. Basu, J. Hestness, D. R. Hower, T. Krishna, S. Sardashti et al., "The gem5 Simulator," *ACM SIGARCH Computer Architecture News*, 2011.

[29] Y. Kim, W. Yang, and O. Mutlu, "Ramulator: A Fast and Extensible DRAM Simulator," *CAL*, 2016.

[30] B. E. Forlin, P. C. Santos, A. E. Becker, M. A. Alves, and L. Carro, "Sim2PIM: A Complete Simulation Framework for Processing-in-memory," *Journal Of Systems Architecture*, 2022.

[31] J. Power, J. Hestness, M. S. Orr, M. D. Hill, and D. A. Wood, "Gem5-GPU: A Heterogeneous CPU-GPU Simulator," *IEEE CAL*, 2014.

[32] H. Luo, Y. C. Tuğrul, F. N. Bostancı, A. Olgun, A. G. Yağlıkçı, and O. Mutlu, "Ramulator 2.0: A Modern, Modular, and Extensible DRAM Simulator," *IEEE CAL*, 2024.

[33] Y. Kim, R. Daly, J. Kim, C. Fallin, J. H. Lee, D. Lee, C. Wilkerson, K. Lai, and O. Mutlu, "Flipping Bits in Memory without Accessing Them: An Experimental Study of DRAM Disturbance Errors," in *ISCA*, 2014.

[34] J. S. Kim, M. Patel, H. Hassan, L. Orosa, and O. Mutlu, "D-RANGe: Using Commodity DRAM Devices to Generate True Random Numbers with Low Latency and High Throughput," in *HPCA*, 2019.

[35] L. Orosa, A. G. Yağlıkçı, H. Luo, A. Olgun, J. Park, H. Hassan, M. Patel, J. S. Kim, and O. Mutlu, "A Deeper Look Into RowHammer's Sensitivities: Experimental Analysis of Real DRAM Chips and Implications on Future Attacks and Defenses," in *MICRO*, 2021.

[36] D. Kim, A. Izraelevitz, C. Celio, H. Kim, B. Zimmer, Y. Lee, J. Bachrach, and K. Asanović, "Strober: Fast and Accurate Sample-based Energy Simulation for Arbitrary RTL," *ACM SIGARCH Computer Architecture News*, vol. 44, no. 3, pp. 128–139, 2016.

[37] A. Amid, D. Biancolin, A. Gonzalez, D. Grubb, S. Karandikar, H. Liew, A. Magyar, H. Mao, A. Ou, N. Pemberton et al., "Chipyard: Integrated Design, Simulation, and Implementation Framework for Custom SoCs," *IEEE MICRO*, 2020.

[38] H. Hassan, N. Vijaykumar, S. Khan, S. Ghose, K. Chang, G. Pekhimenko, D. Lee, O. Ergin, and O. Mutlu, "SoftMC: A Flexible and Practical Open-source Infrastructure for Enabling Experimental DRAM Studies," in *HPCA*, 2017.

[39] D. Biancolin, S. Karandikar, D. Kim, J. Koenig, A. Waterman, J. Bachrach, and K. Asanovic, "FASED: FPGA-accelerated Simulation and Evaluation of DRAM," in *ISFPGA*, 2019.

[40] S. Karandikar, H. Mao, D. Kim, D. Biancolin, A. Amid, D. Lee, N. Pemberton, E. Amaro, C. Schmidt, A. Chopra et al., "FireSim: FPGA-accelerated Cycle-exact Scale-out System Simulation in the Public Cloud," in *ISCA*, 2018.

[41] M. Patel, J. S. Kim, and O. Mutlu, "The Reach Profiler (REAPER): Enabling the Mitigation of DRAM Retention Failures via Profiling at Aggressive Conditions," in *ISCA*, 2017.

[42] A. G. Yağlıkçı, Y. C. Tuğrul, G. F. Oliveira, İ. E. Yüksel, A. Olgun, H. Luo, and O. Mutlu, "Spatial Variation-aware Read Disturbance Defenses: Experimental Analysis of Real DRAM Chips and Implications on Future Solutions," in *HPCA*, 2024.

[43] V. Seshadri et al., "Rowclone: Fast And Efficient In-dram Copy And Initialization Of Bulk Data," in *MICRO*, 2013.

[44] NVIDIA, "NVIDIA Jetson Nano," https://developer.nvidia.com/embedded/jetson-nano

[45] L.-N. Pouchet, "Polybench: The Polyhedral Benchmark Suite," https://www.cs.colostate.edu/~pouchet/software/polybench/.

[46] J. Carter, W. Hsieh, L. Stoller, M. Swanson, L. Zhang, E. Brunvand, A. Davis, C.-C. Kuo, R. Kuramkote, M. Parker et al., "Impulse: Building a Smarter Memory Controller," in *HPCA*, 1999.

[47] K. K.-W. Chang, D. Lee, Z. Chishti, A. R. Alameldeen, C. Wilkerson, Y. Kim, and O. Mutlu, "Improving DRAM Performance by Parallelizing Refreshes with Accesses," in *HPCA*, 2014.

[48] K. Chang et al., "Understanding Reduced-voltage Operation in Modern DRAM Devices: Experimental Characterization, Analysis, and Mechanisms," in *SIGMETRICS*, 2017.

[49] P. Frigo, E. Vannacc, H. Hassan, V. Van Der Veen, O. Mutlu, C. Giuffrida, H. Bos, and K. Razavi, "TRRespass: Exploiting The Many Sides of Target Row Refresh," in *S&P*, 2020.

[50] S. Ghose, A. G. Yaglikci, R. Gupta, D. Lee, K. Kudrolli, W. Liu, H. Hassan, K. Chang, N. Chatterjee, A. Agrawal, M. O'Connor, and O. Mutlu, "What Your DRAM Power Models Are Not Telling You: Lessons From A Detailed Experimental Study," in *SIGMETRICS*, 2018.

[51] H. Hassan, M. Patel, J. S. Kim, A. G. Yağlıkçı, N. Vijaykumar, N. M. Ghiasi, S. Ghose, and O. Mutlu, "CROW: A Low-cost Substrate for Improving DRAM Performance, Energy Efficiency, and Reliability," in *ISCA*, 2019.

[52] E. Ipek, O. Mutlu, J. F. Martinez, and R. Caruana, "Self-optimizing Memory Controllers: A Reinforcement Learning Approach," in *ISCA*, 2008.

[53] A. Olgun, Y. C. Tuğrul, N. Bostancı, I. E. Yüksel, H. Luo, S. Rhyner, A. G. Yağlıkci, G. F. Oliveira, and O. Mutlu, "ABACuS: All-bank Activation Counters for Scalable and Low Overhead Rowhammer Mitigation," in *USENIX Security*, 2024.

[54] O. Canpolat, A. G. Yağlıkçı, A. Olgun, İ. E. Yüksel, Y. C. Tuğrul, K. Kanellopoulos, O. Ergin, and O. Mutlu, "BreakHammer: Enhancing RowHammer Mitigations by Carefully Throttling Suspect Threads," *MICRO*, 2024.

[55] Y. Kim, V. Seshadri, D. Lee, J. Liu, O. Mutlu, Y. Kim, V. Seshadri, D. Lee, J. Liu, and O. Mutlu, "A Case for Exploiting Subarray-level Parallelism (SALP) in DRAM," in *ISCA*, 2012.

[56] V. Seshadri, Y. Kim, C. Fallin, D. Lee, R. Ausavarungniran, G. Pekhimenko, Y. Luo, O. Mutlu, P. B. Gibbons, M. A. Kozuch et al., "RowClone: Fast and Energy-efficient In-DRAM Bulk Data Copy and Initialization," in *MICRO*, 2013.

[57] W. K. Zuravleff and T. Robinson, "Controller for A Synchronous DRAM That Maximizes Throughput by Allowing Memory Requests And Commands to be Issued Out Of Order," US Patent: 5,630,096, 1997.

[58] S. Rixner, W. J. Dally, U. J. Kapasi, P. Mattson, and J. D. Owens, "Memory Access Scheduling," in *ISCA*, 2000.

[59] OpenHW Group, "OpenHW Group - Website," https://www.openhwgroup.org/

[60] K. Asanovic, R. Avizienis, J. Bachrach, S. Beamer, D. Biancolin, C. Celio, H. Cook, D. Dabbelt, J. Hauser, A. Izraelevitz et al., "The Rocket Chip Generator," *EECS Department, University Of California, Berkeley, Tech. Rep. UCB/EECS-2016-17*, 2016.

[61] J. Bachrach, H. Vo, B. Richards, Y. Lee, A. Waterman, R. Avizienis, J. Wawrzynek, and K. Asanović, "Chisel: Constructing Hardware in a Scala Embedded Language," in *DAC*, 2012.

[62] C. Celio, "A Highly Productive Implementation of an Out-of-order Processor Generator," Ph.D. dissertation, UC Berkeley, 2018.

[63] L. W. McVoy, C. Staelin et al., "LMbench: Portable Tools for Performance Analysis." in *USENIX ATC*, 1996.

[64] Xilinx Inc., "Xilinx VCU 108 FPGA Board," https://www.xilinx.com/products/boards-and-kits/ek-u1-vcu108-g.html.

[65] SAFARI Research Group, "Ramulator 2.0," https://github.com/CMU-SAFARI/ramulator2.

[66] S. Goossens, J. Kuijsten, B. Akesson, and K. Goossens, "A Reconfigurable Real-time SDRAM Controller for Mixed Time-criticality Systems," in *CODES+ISSS*, 2013.

[67] M. N. Bojnordi and E. Ipek, "PARDIS: A Programmable Memory Controller For The DDRx Interfacing Standards," in *ISCA*, 2012.

[68] J. Zhang, Y. Zha, N. Beckwith, B. Liu, and J. Li, "MEG: A RISCV-based System Emulation Infrastructure for Near-data Processing Using FPGAs and High-bandwidth Memory," in *TRETS*, 2020.

[69] J. Kim, M. Patel, H. Hassan, and O. Mutlu, "The DRAM Latency PUF: Quickly Evaluating Physical Unclonable Functions By Exploiting The Latency-reliability Tradeoff In Modern DRAM Devices," in *HPCA*, 2018.

[70] B. M. S. Bahar Talukder, J. Kerns, B. Ray, T. Morris, and M. T. Rahman, "Exploiting DRAM Latency Variations for Generating True Random Numbers," in *ICCE*, 2019.

[71] L. Orosa, Y. Wang, M. Sadrosadati, J. S. Kim, M. Patel, I. Puddu, H. Luo, K. Razavi, J. Gómez-Luna, H. Hassan, N. Mansouri-Ghiasi, S. Ghose, and O. Mutlu, "CODIC: A Low-cost Substrate for Enabling Custom In-DRAM Functionalities and Optimizations," in *ISCA*, 2021.

[72] V. Seshadri, D. Lee, T. Mullins, H. Hassan, A. Boroumand, J. Kim, M. A. Kozuch, O. Mutlu, P. B. Gibbons, and T. C. Mowry, "Ambit: In-memory Accelerator For Bulk Bitwise Operations Using Commodity DRAM Technology," in *MICRO*, 2017.

[73] D. Lee, S. Khan, L. Subramanian, S. Ghose, R. Ausavarungniran, G. Pekhimenko, V. Seshadri, and O. Mutlu, "Design-induced Latency Variation in Modern DRAM Chips: Characterization, Analysis, and Latency Reduction Mechanisms," in *SIGMETRICS*, 2017.

[74] M. Qureshi, D.-H. Kim, S. Khan, P. Nair, and O. Mutlu, "AVATAR: A Variable-retention-time (VRT) Aware Refresh for DRAM Systems," in *DSN*, 2015.

[75] J. Liu, B. Jaiyen, R. Veras, and O. Mutlu, "RAIDR: Retention-aware Intelligent DRAM Refresh," in *ISCA*, 2012.