duet: Combining a
Trustworthy Controller with aConfidential Computing
EnvironmentIstemi Ekin Akkus, Ivica Rimac 08.07.2024
7th Workshop on System Software for
Trusted Execution (SysTEX 2024)

1© 2024 Nokia

Trusted Execution Environments (TEEs)
Confidentiality and integrity protection- Confidential Virtual Machines❖ AMD SEV-SNP, Intel TDX- Application Enclaves❖ Intel SGX, ARM TrustZone➢Protect data in-useoConfidentiality of data during computation➢Enable remote attestation
oIntegrity of code computing over data

➢Base root of trust on hardwareoNo trust in the cloud provider or operator

Verifiable Computation2© 2024 Nokia

Verifiable Computation with TEEs – A Comparison
Confidential Virtual Machines (CVMs) run entire
operating systems in the TEE

Application Enclaves run only certain parts of
the application in the TEE

ProsPros•

Lift and shift with no change to applications••

Flexible and easy-to-use••

Access to confidential GPUs (e.g., NVIDIA H-100)

ConsRemote attestation integrity only for initial state•

Owner has full control (i.e., “insider threat”)••

Owner VerifiableComputation© 2024 Nokia

Transparent integrity of code via remote attestation(i.e., no changes after start)

Cons•

3

Minimized trusted computing baseRequires re-architecting and splitting applications
into "trusted" and "untrusted" code
No access to specialized hardware

3rd Party VerifiableComputation

Verifiable Computation with TEEs – A Guideline
Q1. Whose data is it?
Q2. Who is doing the computation?
Confidential VMs➢Same entity:➢ "Owner self-run": use confidential VMs for its
advantages➢Not the same entity:➢ "as-a-Service provider": use application enclaves• Lift-and-shift• Flexible• Confidential GPUs

• Insider threat• Transparent integrity• No insider threat• Re-architect• No GPUs

to ensure trust for users

Application Enclaves
4© 2024 Nokia

GoalTo combine
the transparent integrity of application enclaves
withthe flexibility and ease-of-use of confidential VMs

5© 2024 Nokia

Agenda•Motivation•

Background & Assumptions•

CVM integrity•

Actors•Threat Model•

duet Overview•

Prototype Implementation

6© 2024 Nokia

CVM Integrity Checks•

Attestation report covers only the firmware•

But not the rest of boot binaries

Gives only the initial
CVM state integrity
••

•

7

Measured boot approaches with other boot binaries•

QEMU and OVMF patches: kernel, kernel command line, initRAMdisk•

vTPM as a Secure VM Service Module (SVSM)“Read-only disk” hosting the confidentiality-offering
service•

Revelio [Middleware’23]•Confidential Containers (CoCo)© 2024 Nokia•

Good for “Ownerverifiability”
Not good for “3rd partyverifiability”

Gives runtime integrity••

Good for “3rd partyverifiability”
Not good for “flexibility”
and “maintenance”

Actors
Service Owner

Service UserOffers a service withconfidentiality protections

Uses the confidentialityoffering service• “Confidential/Private
Machine Learning”

8© 2024 Nokia•

Has confidential/private
assets but does not wantto run the service

Infrastructure Provider

Supplies the underlying
hardware infrastructurewith up-to-date TEEs
• No other interaction withservice owner and user

Threat Model & Assumptions•

Attacks on TEE hardware out-of-scope•

Infrastructure provider not interfering with disk integrity••

Publicly available OS and software packages••9Well-known Ubuntu image, Ubuntu packages

Non-public software packages are from well-known sources••

Can be combined with integrity protection solutions (done once per OSimage)

NVIDIA GPU drivers signed by NVIDIA

Publicly available confidentiality-offering service code

© 2024 Nokia

Agenda•Motivation•

Background & Assumptions•

duet Overview•10•High-level overview•Service Owner workflow•

Service User workflow

Prototype Implementation© 2024 Nokia

duet Overview
Approach in a nutshellGoals•

Flexibility and ease-of-use for serviceowner•

•••

11

1. An application enclave gives transparent
runtime integrity

Need CVMs for specialized hardware like confidential
GPUs
Need updates to the OS packages and service for newfunctionality

CVM’s runtime integrity•

Idea

Service owner == CVM owner
How do we prevent the ser vice owner from installing
anything malicious after the initial boot?

© 2024 Nokia➢

SGX MRENCLAVE value

2. Let the application enclave run a controller
application to “own” the CVM•

Service owner =/= CVM owner

Trustworthy Controller (SGX)duet
Combining best of both worlds
1. Deploy controller
and instruct it todeploy the service

Trustworthy Controller (SGX)

2. Provision and attest CVMServiceOwner

3. Deploy and maintain
services, and log all
commands and outputs

4. Establish trust in thecontroller (via remoteattestation) and check its
logs for the CVM integrity

Service User

Confidential VM &services

duet12© 2024 Nokia

5. Use services withconfidential dataOwning the CVM

Trustworthy Controller (SGX)

Trustworthy controller operation•••

ephemeral
SSH key

Provisioning resources for the CVM
From cloud provider or on-premise server

2. Provision and attest CVM
withCVM access only possible by the controller•

3. Deploy and maintain ser vices,and log all commands andoutputs

Controller generates an ephemeral SSH key whileprovisioning the CVM; other login options are disabled

Confidential VM &services

The SSH key stays inside theSGX enclave [confidential]➢Controller has full control of CVM•

13

Deploying and maintaining the service possible via thecontroller API© 2024 Nokia

duetDeploying and Maintaining the Confidential Service
Service owner workflowServiceOwner

Trustworthy Controller (SGX)

1. /mark_cvm(“in-update”)
2. /run_cvm_commands
3. /mark_cvm(“in-service”)•••

Service owner can manage packages
and service via controller interface
Controller logs all commands, applies
them to the CVM and logs all outputs

Controller can reject commands if
certain conditions are not satisfied
14© 2024 Nokia“in-service”/”in-update”

API
CVM’s
ephemeral
SSH key

CVM’s mode

CVM’s attestation
report

CVM’s commands andoutputs

Initial state

Modifications
after bootCVM’s current
runtime stateConfidential Service Operation
Checking the runtime integrity of the CVM by service users
Trustworthy Controller (SGX)

Service User

“in-service”/”in-update”

API
1. /quote
2. /get_cvm_state••

CVM’sephemeral
SSH key

CVM’s mode

CVM’s attestation
report

CVM’s commands andoutputs

Service users establish trust in the controller
via SGX remote attestation

Users check CVM’s runtime integrity•

Obtain “current CVM state”•Check attestation report, logs and outputs➢ If all good, share confidential/private datawith confidential service15© 2024 Nokia

Controller code and Initial statelogs are protectedby SGX [integrity]

Modifications
after bootCVM’s current
runtime stateAgenda•Motivation•

Background & Assumptions•

duet Overview•

Prototype Implementation•

16Controller & API client© 2024 Nokia

Prototype Implementation••

Generic controller implementation•~1K lines of Python code + gramine libOS for containerization•

Controller API client with ~425 lines of Python code

Initial set of CVM commands for attestation report••

Provider-specific provisioning commands••

AMD SEV-SNP and Intel TDX on Microsoft Azure

Standard Ubuntu 22.04 as CVM image•

17With Microsoft Azure Attestation

Designated for confidential computing by Canonical© 2024 Nokia

Trustworthy Controller (SGX)Limitations & Future Work••18Microsoft Attestation Service and firmware closed source•

No direct access to the TEE•

Other providers allow access to the TEE hardware

Disk integrity assumed after initial boot•SNPGuard may help•

Protection would be applied once per OS image; not per deployed service© 2024 Nokia

Summary➢Alternative approach for verifying CVM runtime integrity•

An application enclave as the CVM owner/controller•

Transparent integrity checks for 3rd party verifiability•

Flexibility and ease-of-use for service ownerConfidential VMs• Lift-and-shift• Flexible• Confidential GPUs
• Transparent integrity• No insider threat

Application Enclaves➢Source code available: https://github.com/Nokia-Bell-Labs/tee-duet19© 2024 Nokia