Keystone Enclave

An Open-Source Secure Enclave for RISC-V
Dayeol Lee1,2, David Kohlbrenner, Kevin Cheang1, Cameron Rasmussen1,Kevin Laeufer1, Ian Fang, Akash Khosla, Chia-Che Tsai2, Sanjit Seshia1,Dawn Song2,3, and Krste Asanovic1,2
University of California, Berkeley ※Collaborators: Ilia Lebedev4, and Srinivas Devadas4
1※All authors are affiliated with the UCB

234

What is a Secure Enclave?
Applications

Trustworthy
HardwareUser
Programand DataOSIntegrity

Confidentiality

Enclave contents

RemoteAttestation

Secure Enclave as a Cornerstone Security Primitive● Strong security capabilities○

Authenticate itself (device)○

Authenticate software○

Guarantee the integrity and privacy of remote execution● A cornerstone for building new security applications

3○

Confidential computing in the cloud (e.g., machine learning)○

Secure IoT sensor network

Why do we need an Open-Source Enclave?
● Existing enclave systems are proprietary and difficult to experiment with○

Closed-source commercial hardware (e.g., Intel SGX, ARM TrustZone)○

Lack of good research infrastructure● A Lot of Challenges for Enclaves○Hardware vulnerabilities: Intel SGX - ForeShadow (USENIX’18), AMD SEV - SEVered(EuroSec’18)○Side channel attacks and physical attacks○

Important questions: do patches really fix the problem? Are there any other issues?

Open Source Design• Provides transparency & enables high assurance• Builds a community to help people work on the same problems
4Keystone Enclave

5Keystone: Open Framework for Secure Enclaves● The First Full-Stack Open-Source Enclave for Minimal Requirements○

Root of trust, security monitor, device driver, SDK, …

○Memory isolation, secure bootstrapping, remote attestation, …

● Memory Isolation only with Standard RISC-V Primitives○

RISC-V Privileged ISA (U-, S-, and M-mode support)○

Physical Memory Protection (PMP)○

Demonstrate in unmodified processors● Open Framework: Built Modular & Portable for Easy Extension

6○

Platform-agnostic isolated execution environment○

Platform-specific threat models (cross-core side channels, untrusted external memory, etc)○

Use various entropy sources/roots of trust in different platforms

github.com/keystone-enclave

Earlier Work: Sanctum● The First Enclave Design in RISC-V ISA○

V. Costan et al., USENIX Security ’16○

Proof of concept in C++(https://github.com/pwnall/sanctum)● Non-standard Hardware Extension○

PMP was introduced in 2017 (RISC-V Priv. v1.10)● Keystone and Sanctum

7○

Keystone was built from scratch○

Keystone shares many good practices from prior experiences of Sanctum○

The primary goal of Keystone is to make an open end-to-end framework

What Hardware Do We Need?
● RISC-V Physical Memory Protection (PMP)
Processor Package● RISC-V U-, S-, and M-mode

Tamper-proof
hardwareCores● (RISC-V) Device Gasket PMP (i.e., iopmp)

ZSBLDevices● An Entropy Source available at bootKey store

Entropy Src.

Memory Encryption/Integrity● Root of Trust (preferably a crypto engine)○Measuring & signing the security monitor○

Platform key store● If untrusted/external DRAM –memory encryption/integrity engine
8(not implemented yet)

ioctl()Operating System-

Untrusted device driver
Allocates contiguous memory
Provides the interface to user-

The application to execute in theenclave
syscalls,
traps,…
Enclave Runtime-A part of the enclave running in Smode

U-modeUntrusted app hosting an enclave

Enclave Application

S-mode-Trusted, Isolated

Untrusted

OverviewHostof Application
Keystone

Silicon Root of Trust
9Stored in tamper-proof hardware
Zeroth-stage bootloader (ZSBL)
Tamper-proof platform key store(preferably a crypto engine)

Keystone Security Monitor (SM)measure,sign-Manages enclaves and PMP entries
Multicore PMP synchronization
Remote attestation

M-mode(Trusted)

SBIKeystone Overview (Simplified)
Remote Machine

HostApplication

Enclave
Application

HostOS

Enclave
RuntimeYou
UntrustedNetworkmeasuressigns

10

measuressigns

Keystone Security Monitor
controls
Root of Trust

PMPKeystone Overview (Simplified)
Remote Machine

HostApplication

Enclave
Application

HostOS

Enclave
RuntimeYou
Untrusted
Networkmeasures
signs

11measures
signs

How does PMP work?

Keystone Security Monitor
controls
Root of Trust

PMPMemory Isolation with RISC-V PMP● Physical Memory Protection (PMP)o Special registers to control permissions of U- and S-mode
oooooaccesses to a specified memory region# of PMP entries can vary (e.g., default Rocket has 8)
Statically prioritized by the order of entry indicesWhitelist-based
Dynamically configurable by M-mode
Addressing modes: NAPOT (>= 4-bytes), Base/Bound● How Keystone uses PMP

12○Top/bottom PMP entries are reserved for SM/OS○

1 PMP entry for each “active” enclave○

NAPOT > 4KB (fragmentation / Linux buddy allocation)

Isolation via Switching PMP Permission Bits
rwx permissions

000

pmp0
pmp1
pmp2
…

Priority

PMP entries

address range

111

pmpN

S/U accessibility
accessible
not accessible

SM Boots
13SM

OSDRAM(0x80000000-)OS Boots

PMP entries

Creating an Isolated Enclavepmp0000

pmp1
pmp2
…
111

pmpN

S/U accessibility
not accessible

14SM

free
ELF
RTpages

PTaccessible

OSDRAM(0x80000000-)OS allocates a contiguous chunk of memory using __get_free_pages() and initializes
the free pages with the enclave page table, and the enclave program (runtime + enclaveapplication)

Creating an Isolated Enclave
PMP entries

SM sets PMP entry and finalizes the enclave hashpmp0000000

pmp1000

pmp2
…
111

pmpN

S/U accessibility
not accessible

15SM

Enclave 1
free
ELF
RTpages
MemoryPTaccessible

OSDRAM(0x80000000-)

Enclave 2
MemoryPMP entries

Creating an Isolated Enclave
pmp0OS can ask SM to create as many enclaves as the numberof remaining PMP entries

000000pmp1000

pmp2
…
111

pmpN

S/U accessibility
not accessible

16SM

Enclave 1
free
ELF
RTpages
MemoryPTaccessible

OSDRAM(0x80000000-)

Enclave 2
MemoryPMP entries

Executing an Enclave
pmp0000000

pmp1000

pmp2
…
111

pmpN

S/U accessibility
not accessible

17SM

Enclave 1
free
ELF
RTpages
MemoryPTaccessible

OSDRAM(0x80000000-)

Enclave 2
MemoryExecuting an Enclave
PMP entries

SM flips the PMP permission bits of pmp2 and pmpN to execute Enclave 2
pmp0000000

pmp1000

pmp2…
111

pmpN

S/U accessibility
not accessible

18SM

Enclave 1
free
ELF
RTpages
MemoryPTaccessible

OSDRAM(0x80000000-)

Enclave 2
MemoryExecuting an Enclave
PMP entries

SM flips the PMP permission bits of pmp2 and pmpN to execute Enclave 2
pmp0000000

pmp1

111

pmp2
…
000

pmpN

S/U accessibility
accessible
not accessible

19SM

Enclave 1
MemoryOSDRAM(0x80000000-)

Enclave 2
MemoryPMP entries(Asynchronous) Exit and Resume
pmp0000

The enclave can only exit by an SM SBI call.
The SM flips the permissions before entering the untrusted context.
000

pmp1000

pmp2
…
111

pmpN

S/U accessibility
not accessible

20

SM

Enclave 1
free
ELF
RTpages
MemoryPTaccessible

OSDRAM(0x80000000-)

Enclave 2
MemoryPMP entries

Destroying an Enclave
pmp0000
000

pmp1
pmp2
…

111

pmpN

S/U accessibility
not accessible

21

SM

Enclave 1
free
ELF
RTpages
MemoryPTaccessible

OS
DRAM(0x80000000-)

Untrusted Shared Buffer
PMP entries

The OS can allocate a shared buffer in OS memory
The SM uses the last PMP entry to allow the enclave to access the buffer.

pmp0000000

pmp1

111

pmp2
…

111

pmpN

S/U accessibility
accessible
not accessible

22SM

Enclave 1Memory

OSDRAM(0x80000000-)

Enclave 2
MemorySharedBufferKeystone Overview Revisited
Remote Machine

Host
Application

Enclave
Application

HostOS

Enclave
RuntimeWhat is a Runtime?

YouUntrusted
Networkmeasuressigns

24

measuressigns

Keystone Security Monitor
controls
Root of TrustPMPS-Mode Enclave Runtime● Provides Kernel-like Functionalityo Syscalls, traps
o thread and page table management● Useful Layer of Abstractiono Least privilege of U-mode codeo Additional functionality withoutcomplicating the SMo SM < 2K LoC + 5K LoC crypto lib.

● Reusabilityo Compatible with multiple user
programso Can act as a shield system25(e.g., Haven, Graphene) in SGX

Keystone Overview Revisited
Remote Machine

Host
Application

Enclave
Application

HostOS

Enclave
RuntimeYou
Untrusted
Networkmeasuressigns

26

measuressigns

Keystone Security Monitorcontrols

Root of Trust

How to implement?

PMPSilicon Root of Trust● Tamper-proof hardware that cryptographically hashes the security monitor,provisions an attestation key, and signs them with device’s secret key.
● Various ways to implement the root of trust○

Various entropy sources, various platform key store, and implementation of the crypto engine● Keystone uses Sanctum’s root of trust which uses ECDSA and SHA-327

Keystone Overview Revisited
Remote Machine

MITM

Host
Application

Enclave
Application

You
UntrustedNetwork
SecretData28

measures
Enclave
HowRuntime
does thesignsenclave
authenticate itself andKeystone create
Security Monitor
a secure channel?
HostOSmeasuressignscontrols

Root of Trust

PMPRemote Attestation● SM measures the enclave upon enclave creation● Enclave may bind a key to the enclave report● SM signs the enclave report and hands it (+ SM report) to the user
The Full Process of Attestation

29Measurement Layout…

Project Status● Testable in Various Platforms○

Latest RISC-V QEMU: functionality test, development○

Latest FireSim (v1.4.0): performance analysis, hardware modification○

SiFive Unleashed: runs on a real quadcore in-order processor!

● Ongoing Efforts○

Formal verification of PMP-based security monitor○

Mitigating cache side-channel attacks using platform features● Contributions Needed!

32○

Building software stack: more use cases, libraries, edge compiler, …

○

Adding software/hardware extensions
e.g., demand paging, memory encryption/integrity, multithreading, CMA integration, …

Project Links● Deployment:
○

QEMU:

https://github.com/keystone-enclave/keystone○

FireSim:https://github.com/keystone-enclave/keystone-firesim○

SiFive Unleashed:https://github.com/keystone-enclave/keystone-hifive-unleashed● Keystone Repository:○

Keystone-SDK:https://github.com/keystone-enclave/keystone-sdk○

Device Driver:https://github.com/keystone-enclave/riscv-linux○

Security Monitor:

https://github.com/keystone-enclave/riscv-pk○

A Simple Runtime:https://github.com/keystone-enclave/keystone-runtime

○

Demo:

https://github.com/keystone-enclave/keystone-demo● Documentation (more coming):

33○

Website/Blog:

https://keystone-enclave.org○

Development Docs:

https://docs.keystone-enclave.org

Demo34

A Remote Enclave with Secure Channel● SiFive Unleashed board + simulated non-standard hardware○

Root of trust: Modified FU540 FSBL with hard-coded device key● Successfully ported libsodium for ECDH Key Exchange
x86_6435

Conclusion

keystone-enclave.org● Keystone: an Open-Source Full-Stack Enclave for RISC-V○

Runs on standard RISC-V cores○

Modular design for better extensibility & portability● Use Cases○

Secure hardware research (e.g., LLC side-channel defense w/ way partitioning + PMP)○

Building secure systems (e.g., Secure IoT network)● Opens up Research Opportunities around Hardware Security

36○Formal Verification of PMP and Security Monitor Implementation○Performance Analysis○

Defending Side Channels & Physical Attacks○

Multi-level Security (MLS) for Sensitive Data Analytics

Thank You!

37

Dayeol Lee (dayeol@Berkeley.edu)
David Kohlbrenner (dkohlbre@berkeley.edu)
Forum (keystone-enclave@googlegroups.com)