Ethernet-based
SoftwareDefinedRack Disaggregation Using
Network (SDN)

PCIe Networking

Cloud Computing Research Center for
Mobile Applications (CCMA)
Industrial Technology Research Institute
雲端運算行動應用研究中心

1

Rack Disaggregation• Rack as the basic building block for cloud-scale data centers• Rack disaggregation: pooling of HW resources for global
allocation and independent upgrade cycle for each resource
type• CPU/memory/NICs/Disks embedded in individual hosts–––

Disk pooling in a rack
NIC/Disk pooling in a rack
Memory/NIC/Disk pooling in a rack

• Enabling technology: High-speed rack-area networking thatallows direct memory access– PCI Express is a promising candidate (Gen3 x 8 = 64Gbps)– New top-of-rack (TOR) switch consists of PCIe ports and Ethernet ports2

I/O Device Disaggregation••••

Reduce cost: One I/O device per rack rather than one per host
Maximize Utilization: Statistical multiplexing benefit
Power efficient: intra-rack networking and device countReliability: Pool of devices available for backup
Virtualized
Host
VM

VM

VM

Hypervisor

Virtualized
HostVM

VM

VM

Hypervisor

Virtualized
HostVMVM

Non-virtualized
HostNon-virtualized
HostLinuxWindowsVMHypervisor

Problem: MR-IOV capable PCIe switches and devices are not available
PCIe SwitchFlashMemory

HDD-basedRAIDs

EthernetNICs
3

ITRI Software-based MR-SRIOV• Standard PCIeswitches and device:• No MR-IOV capability• Native driver• Zero data copying• Native Performance:• Control path goes to acentral authority• Direct data path toand from devices• Secure Sharing:• Access control using
existing technologies
4I/O Virtualization• Direct communication:• Device directly assignedto VMs• Bypass the hypervisor• Physical Function:• configure and manage
the SR-IOV functionality• Virtual Function:• lightweight PCIefunction with resources
necessary for datamovement• VT-x and VT-d
• CPU/Chipset support forVMs and devices
However, works only in a single host (hence Single Root)
5

Single Root (SR) Architecture

6

Multi Root (MR) Architecture

1.Enhanced switch

2.MR-capable devices
7Non-Transparent Bridge (NTB)• PCI Enumeration• Host1’s enumeration finds
X and NTB-D• Host 2’s enumeration finds
Y and NTB-EHost 1• Translations betweendomains• NTB-D exposes a portion ofphysical memory address
range of Host2’s domain• Memory address:• From primary side andsecondary side
• PCI device ID:• Querying the ID lookup
table (LUT)

NTB-E

Host 28

NTB Address Mapping• NTB maps from:– <the primary side to the secondary side>• Mapping from addrA at primary sideto addrB at the secondary side

• Example:– AddrA = 0x8000 at BAR4 from Host1– AddrB = 0x10000 at Host2’s DRAM

• One-way Translation:– Read/write at addrA in Host 1==
read/write addrB in Host 2– Read/write at addrB in Host 2 has
nothing to do with addrA in Host 1

host1

host2

NTB9

Ladon: a hundred-heads dragon
who guards the Garden of Hesperides

Ladon: A software-defined
PCIe network architecturethat supports secure MR-IOV
using SR-IOV devices

10

System Components• Management Host (MH)• Manage the shared I/O devices• Compute Host (CH)• Non-virtualized host OS candirectly access VF• Virtualized host with VMs candirectly access VF• Non-Transparent Bridge (NTB)• Each CH connects to the fabric
via an NTB• PCIe Switch & SR-IOV device• PF and multiple VFs
11

I/O Virtualization• Direct communication:• Device directly assigned toVMs• Bypass the hypervisor
overhead• Physical Function:• configure and manage theSR-IOV functionality• Virtual Function:• lightweight PCIe function
with resources necessary
for data movement• VT-x and VT-d
• CPU/Chipset support for
VMs and devices
However, applicable in a single host (SR -> Single Root)
12

Global Physical Address Space
Physical Address Space of MH
NTB

PhysicalMemory

CH1
IOMMU

NTBMSI-XIOMMU

248

MSI-X

VF1NTB
IOMMU

VF2:VFn

CSR/MMIO

Physical
Memory

MSI-X

MSI-X

Physical
MemoryPhysicalMemory

MH

CH2CHn013

Virtual NIC Configuration• Each virtual NIC in a CH is backed by a VF of an SRIOV NIC• Identify the virtual NIC’s CSR, MMIO, MSI-X and DMA payload area
• Install mappings in the BARs of the asscociated NTB port

14

Secure Threats: 4 Cases
Malicious VM, CH and MH
MH

CH1

CH2

VM1

VM2VM1VM2VFVFVFVF

Main Memory

PFVMM

VMM

PCIe Switch Fabric
PFVF1VF2VF3VF4SR – IOV Device

VF1 is assigned to VM1 in CH1, but it can screw multiple memory areas.

Device assignment
Unauthorized Access15

Security Guarantees: Summary• Intra-host– A VF assigned to a VM can only access to the physical memory of its
associated VM– Protected by CH’s IOMMU• Inter-host– A VF can only access the physical memory of its associated CH– A host cannot touch other CHs’ or MH’s memory– Protected by CHs’ LUT & IOMMU• Inter-VF / inter-device– A VF can not write to other VFs’ registers– Protected by MH’s IOMMU• Even a compromised MH cannot touch other CHs’ memory
16Optimizations• Zero driver modification– DMA operands’ addresses are in MH’s address space rather than in CH’s or
VM’s physical address space– Solution: transparent trapping of DMA access API calls• Zero-copy– BAR window size is limited by BIOS (between 3G-4G)– Solution: Mapping the entire CH’s physical memory space to avoid copying.
MH PCI domain

Main
MemoryRX/TX
Descriptor

CH PCI domain
User-level
Application

DriverLink Virt
NTBHTTPNTB mapareaFTPcopy OS Network

devSocket
Buffer
PoolSubsystem

copy_to_user
copy_from_user

17

Inter-Host Communications• RDMA: Remote DMA from one host’s memory toanother host’s memory• Cross-Machine Memory Copying (CMMC): from theaddress space of one process on one host to theaddress space of another process on another host
• Ethernet over PCIe (EOP):– Destination node is inside the rack: sent out via PCIe
– Destination node is outside the rack: sent out via shared
Ethernet NIC
18

PLX 8619

PLX 8619

Intel 82599

PLX 8696

Let’s see some performance numbers!

TESTBED EVALUATIONVideo Available: http://www.youtube.com/watch?v=B_-GesOjkG019

Prototype Implementation
CH:Intel i7 3.4GHz
8-core CPU
8 GB of memory

Link: Gen2 x8 (32Gb)

MH:
8-core Intel Xeon 3.4GHz
8GB memory

CH1

CH2

VM:
Pin 1 core, 2GB RAM

OS/hypervisor:
Fedora15 / KVM
Linux 2.6.38
NTB/Switch:
PLX8619
PLX8696

CH4
NIC: Intel 82599
20

Latency
SRIOV

MRIOV

MRIOV+

Latency (microseconds)

300

250
200

150

MTU=1500

100

500

Message Size (bytes)
21

TCP RX/TX Throughput
SRIOV
MRIOV
Copying Overhead

MRIOV+109Bandwidth (Gbps)

8765432106432

1684Message Size (Kbytes)

2122

Scalability / Compatibility• How many CHs/VMs can Ladon support?
– Each CH: 2 BARs for CSR and MMIO, 2 BARs for DMA and MSI-X– As many VMs as a CH could run– As many CHs as are allowed by the PCIe switch• How many SRIOV devices can Ladon shares?
– As many as are allowed by the PCIe switch• Vendor-neutrality– Ladon only uses the basic features of NTB: address translation, device IDtranslation and DMA– Intel’s on-board NTB and DMA support: Xeon C5500/C3500– PCIe device: physical device driver needs modification but SRIOV driver
does not23

Summary• A software-defined PCIe-based rack-area networkarchitecture– Allows multiple servers to share and directly access SRIOV PCIedevices• A secure I/O device sharing scheme– Prevents unauthorized accesses to shared I/O devices by leveraging
EPT, IOMMU, BAR translation tables, and LUT• A fully operational Ladon prototype– Successfully demonstrates the feasibility and efficiency of thesoftware-based MRIOV– Zero throughput/latency penalty when compared with SRIOV– Full HA support for MH• Integration of PCIe with Intel’s silicon photonics technology?
24

Strawman Proposal for Disaggreated I/O• Every host comes with an NTB, a DMA engine and a PCIeexpansion adapter• A transparent PCIe switch• A set of SRIOV PCIe devices• An MH or controller that
– Enumerates the available SRIOV devices– Allocates and reclaims the virtual functions on SRIOV devices– Includes HA support• On each CH– An original SRIOV device driver for each SRIOV device– A specially-built PCIe driver that communicates with MH
25

Thank You!
For More Information:
Y.F. Juan / 阮耀飛
Deputy Director
Strategy & Business Development
Cloud and Mobile Computing CenterIndustrial Technology Research Institutet: +886 (0) 3.591.6173
m: +886 (0) 975.876.919
e: yf.juan@itri.org.tw26