Memory Management in the Cloud

Pierre Louis Aublin
IIJ Research Laboratory
2023年12月19日

Meet Tanaka san●Software engineer at Super Infinity Cloud Provider (IaaS)

2

Tanaka san in charge of VM infrastructure●VM allocation●VM migration●

Efficient resource utilization●Fault-tolerance●Isolation and security

3

VM Memory usage example
Host 1

Host 2

Assumptions:● Each host has 3 CPU cores●

At most 3 VMs machine per
hostNetwork link

Host 3

Host 44

A new VM needs to be allocatedHost 1

Host 2

Network link

Host 3

New VM!

Where to allocate?

Host 45Migrate VMs
Host 1

Host 2

Network link

Host 3

New VM!

Where to allocate?

Host 46

Allocate new VM
Host 1

Host 2

Network link

Host 3

New VM!

Success!

Host 47

Is migration the solution?
Host 1

Host 3

Host 4●

Need to move data/computation acrossthe cluster●

Interruption of service for the migrated
VMs

8

Cluster is full; Memory stranding appears…
Host 1

Host 2●

Unused memory●

Cannot be used to allocate a new VM○ "All cores have been rented, butthere is still free memory on theserver"●Microsoft Azure reports >25% of strandedmemory
Li et al., "Pond: CXL-Based Memory Pooling Systems for Cloud Platforms", ASPLOS 2023●
Host 3

Host 4

Studies at Microsoft, Google, Alibaba
show 50% of memory is not utilizedZhang et al., "Redy: Remote Dynamic Memory Cache", 2021

9

Let's solve memory stranding via memory pooling!
●Memory pooling○○○

Host 3

Dynamic memory allocation scheme
Divides system memory into blocks
Each block can be allocated to / reclaimed from a VM to follow memory usage

Host 4

Host 3

Host 4

Host 3

Host 410

Where to store the reclaimed memory?
+Host 3

Host 411

Network-attached memory●

Remote Direct Memory Access○○○Access a computer's memory from another computer
Operating System/CPU not involved
Low-latency operation

RDMADRAMNIC
CPUDRAMNIC

TCP/IPCPU12

VM memory access via RDMA
RDMAScenario 1:
VM migration

Host 3

Scenario 2: no
VM migration●

Host 5

Host 4

RDMARDMAAccess toVM reclaimed memory ("cold memory") incurs latency penalty
compared to DRAM access

13

Memory disaggregation●

Separate memory from the compute resources●Memory of a single machine can be shared across a network of servers●Improves memory utilization and scalability
Host 3

Scenario 2: noVM migration

Host 4

RDMARDMA14

Leap: a solution to manage remote memory●"Effectively Prefetching Remote Memory with Leap"○○●

Al Maruf and Chowdhury. Michigan university
USENIX ATC 2020

Problem○○○Remote memory access is slow (μs compared to ns)
This slows down memory intensive applications
How do we choose which data to allocate remotely?

RDMA

Host 1

Host 215

Hot vs Cold memory pages●Hot memory page○○●

Frequent accessMove to faster, local memory for better performance

Cold memory page○○

Infrequent accessCan be moved to slower, farther memory

RDMA

Host 1

Host 216

Leap architecture●Implemented in Linux kernel○●Online prefetcher○●

Identify remote memory
accesses patterns

Local cache○○●

Applications not modified

Avoid pollution
Increase cache hit rate

https://github.com/SymbioticLab/Leap

17

Leap performance evaluation●

System setup○○●

Real-world benchmarks○●

56 Gbps Infiniband cluster
Each machine: 64 GB RAM, 2x Intel Xeon E5-2650 v2 (16 cores)

PowerGraph, NumPy, VoltDB, Memcached

Performance gain: up to 10x compared to Infiniswap, a state-of-the-art solution

18

Where to store the reclaimed memory? (Revisited)+Host 3

Host 419

●Compute-eXpress Link○○●

High-speed cache-coherent
interconnect○●

Industry-backed open standard
https://www.computeexpresslink.org/●Device types○

Type 1: CXL.io and CXL.cache■ e.g., smartNIC○

Type 2: CXL.io, CXL.cache andCXL.mem■ e.g., GPU or FPGA○

Type 3: CXL.io, and CXL.mem■ e.g., memory expansion board

Built on top of PCIeProtocols○

CXL.io: provides configuration, discovery,etc.

○

CXL.cache: allow devices to coherently
access and cache host CPU memory○CXL.mem: allow host CPU to coherently
access cached device memory20Three CXL Specifications
HostHost

Host1

Host2

DRAM

DRAM

DRAM

DRAM

CPU

CPU

CPU

CPU

DRAM
CPU

DRAM
CPU

DRAM
CPU

DRAM
CPU

Accelerator(e.g., NIC,GPU)

Accelerator(e.g., NIC,GPU)

Accelerator(e.g., NIC,GPU)

Accelerator(e.g., NIC,GPU)

1.1: CPU -> accelerator
access cache-coherent

2.0: CPU <-> accelerator
access cache-coherent

3.0: P2P accesscache-coherent

21What it means for Tanaka san's company●

VM resources spread across entire
rack/cluster○●

Host 1

Host 2

Decoupling between compute nodes(CPUs) and resource nodes (memory)○●

Remote memory access with very low
performance penalty compared to local
DRAM

Memory disaggregation

PCIe link

Better scalability and resourceutilization

Host 3

Host 422

Pond: a CXL-Based Memory Pooling System●

Pond: CXL-Based Memory Pooling Systems for Cloud Platforms○○●

Research question: How can VMs efficiently use DRAM?
○○○●Li et al., Virginia tech & Microsoft Azure
ASPLOS 2023

DRAM is expensive (50% of hardware cost at Azure)
Local memory node accesses faster than remote accesses
Cloud provider should not inspect what is running inside VMs

Existing solutions○○

Add substantial latency (~μs)
Require changes to the VM

23

Memory usage at Azure●

Analysis of traces from 100 production clusters and 158 workloads●

Consider CXL access to be similar to remote NUMA node access●Grouping memory of 16 CPUs together in a single pool achieves "sufficient"
DRAM saving while adding <100ns latency○●

Overhead of pooling compared to same-NUMA node memory access○○●

7% DRAM saving, which corresponds to ~100M$ of savings

Within 5% for 40% of the workloads>25% for 21% of the workloads~50% of all VMs touch less than 50% of their rented memory○

We can allocate the remaining rented memory on a remote node ("zNUMA") with no
performance penalty
24

The Pond System●

Rely on CXL○●

Predict VM memory allocation behaviour○○●

To fix wrong predictions
Based on hardware counters

Hardware changes○●Is it ok to allocate memory on a remote CXL node?
ML modelMonitor memory access○○●

Pooled (remote) memory access with ns latency

Implement a new EMC (External Memory Controller)

Open-source: https://github.com/vtess/Pond
25

Performance evaluation●

Simulate CXL on a 2 CPUs machine○

One CPU acts as a zNUMA node: cores offline, only memory is used

Model false positive rate
Performance Degradation Margin (PDM):
max acceptable slowdown

Memory savings
26

CXL hardware●CPUs○○○○●Intel Agilex
Xilinx Adaptive Compute Acceleration Platforms Versal Premium lineup

CXL 2.0

Memory module○●

CXL 1.1FPGA○○●Intel Sapphire Rapids
AMD Zen 4 Epyc ("Genoa" and "Bergamo")
Arm Neoverse V2
AmpereOne (https://amperecomputing.com/)

Samsung Memory-Semantic SSD

Complete Memory system○○

Panmnesia (https://panmnesia.com/)
Unifabrix (https://www.unifabrix.com/)

CXL 3.027

Latest update!!! CXL 3.1 brings Trusted Execution support●Enables Confidential Computing○○●

Securely run workloads without exposing
data to untrusted party (OS, other VM, etc.)
Targets Trusted VMs (TVMs)

TSP protocol○

Extends CXL specification to include CXL
devices into the TVM trust boundary28

CXL and disaggregated memory are hot topics

29

Concluding words●Cloud providers need to understand their memory usages
to offer the best performance to their customers●

Memory disaggregation offers several advantages
○
○
○

●

Better utilization
Better scalability
Reduces costs

CXL provides foundation for memory disaggregation
at high speed (ns)
PCIelink

pierrelouis@iij.ad.jp

30