Optimising DRAM caches for latency in datacenter servers
Creators: name: Shahab, Amna
Publisher: name: The University of Edinburgh
Year: 2024
Subjects: main memory latency
multi-level cache hierarchy
die stacking technology
DRAM stacks
DRAM stack latency

description: The unyielding growth in the amount of data processed by datacenter servers warrants an unabated increase in cache capacities. Modern servers already dedicate a big proportion of their die real-estate to on-chip caches, with the largest last-level cache (LLC) accounting for up to a third of the die. Problematically, the slowdown in technology scaling constrains the expansion of LLC capacity as demanded by growing datasets. In the face of on-chip cache capacity constraints, systems today are increasingly being equipped with multi-gigabyte die-stacked DRAM caches. These DRAM stacks provide high access bandwidth through a combination of many DRAM banks and a wide interface, but fall short in improving access latency compared to main memory. Datacenter applications are performance sensitive to access latency as they exhibit low memory-level parallelism (MLP). High-bandwidth, high-latency stacked DRAM caches fail to benefit these datacenter applications. This thesis aims to optimise stacked DRAM caches for low latency by first identifying the sources of latency and then devising cache organisations that address them. This thesis observes that the factors contributing to the high access latency of DRAM caches are: (i) on-chip interconnect (NOC) routing delay to reach the DRAM cache controller, (ii) queuing delay in the DRAM cache controller, (iii) horizontal traversal between the processor die and the DRAM stack, (iv) addressing and access latency in the DRAM core. We find that the aforementioned factors may be addressed at the architecture level to latency-optimise the DRAM cache, and propose two cache organisations. We propose On-PaCkage Partitioned DRAM Victim Cache (CARVE), which minimises the various interconnect latencies by partitioning the DRAM stack into logically independent units, vaults, and utilising each vault as a victim cache for a shared on-chip LLC slice. This design retains the conventional on-chip cache hierarchy, and augments it with a new level of victim cache in DRAM vaults on package. We demonstrate that through a combination of fine-grained connections between the processor die and the DRAM stack, DRAM stack partitioning, and DRAM technology latency optimisation it is possible to architect a low-latency on-package DRAM cache. We propose a novel Die-Stacked Private LLC Organisation (SILO) – which combines on-chip private caches with per-core LLC slices in die-stacked DRAM, which enables further reduction of interconnect latency. Per-core private caches overcome the latency bottleneck of shared caches by limiting the length of in- terconnect that needs to be traversed on an access. To avoid long interconnect delays and maintain the latency benefits of a private cache, SILO organises the DRAM into vaults, each of which sits above a processor core. To summarise, this thesis addresses the factors that lead to the long access latency of DRAM caches through two DRAM cache designs which minimise interconnect delays. We show, through simulation, that datacenter applications running on processors equipped with latency-optimised DRAM caches observe significant performance improvement compared to when running on conventional server processors.