Project: Keylime (Provisioning / Security & Compliance)
File: tci-acm.pdf

Bootstrapping and Maintaining Trust in the Cloud
Nabil Schear
MIT Lincoln Laboratory
nabil@ll.mit.eduPatrick T. Cable II
Threat Stack, Inc.
pat@threatstack.comThomas M. Moyer
MIT Lincoln Laboratory
tmoyer@ll.mit.edu
Bryan Richard
MIT Lincoln Laboratory
bryan.richard@ll.mit.eduRobert Rudd
MIT Lincoln Laboratory
robert.rudd@ll.mit.edu
ABSTRACT
Today's infrastructure as a service (IaaS) cloud environ-
ments rely upon full trust in the provider to secure appli-
cations and data. Cloud providers do not oer the ability
to create hardware-rooted cryptographic identities for IaaS
cloud resources or sucient information to verify the in-
tegrity of systems. Trusted computing protocols and hard-
ware like the TPM have long promised a solution to this
problem. However, these technologies have not seen broad
adoption because of their complexity of implementation, low
performance, and lack of compatibility with virtualized en-
vironments. In this paper we introduce keylime , a scal-
able trusted cloud key management system. keylime pro-
vides an end-to-end solution for both bootstrapping hard-
ware rooted cryptographic identities for IaaS nodes and for
system integrity monitoring of those nodes via periodic at-
testation. We support these functions in both bare-metal
and virtualized IaaS environments using a virtual TPM.
keylime provides a clean interface that allows higher level
security services like disk encryption or conguration man-
agement to leverage trusted computing without being trusted
computing aware. We show that our bootstrapping proto-
col can derive a key in less than two seconds, we can detect
system integrity violations in as little as 110ms, and that
keylime can scale to thousands of IaaS cloud nodes.
This material is based upon work supported by the As-
sistant Secretary of Defense for Research and Engineering
under Air Force Contract No. FA8721-05-C-0002 and/or
FA8702-15-D-0001. Any opinions, ndings, conclusions or
recommendations expressed in this material are those of the
author(s) and do not necessarily reect the views of the As-
sistant Secretary of Defense for Research and Engineering.
c2016 Massachusetts Institute of Technology. Delivered
to the U.S. Government with Unlimited Rights, as dened
in DFARS Part 252.227-7013 or 7014 (Feb 2014). Notwith-
standing any copyright notice, U.S. Government rights in
this work are dened by DFARS 252.227-7013 or DFARS
252.227-7014 as detailed above. Use of this work other than
as specically authorized by the U.S. Government may vio-
late any copyrights that exist in this work.
ACM acknowledges that this contribution was authored or co-authored by an em-
ployee, or contractor of the national government. As such, the Government retains
a nonexclusive, royalty-free right to publish or reproduce this article, or to allow oth-
ers to do so, for Government purposes only. Permission to make digital or hard copies
for personal or classroom use is granted. Copies must bear this notice and the full ci-
tation on the ﬁrst page. Copyrights for components of this work owned by others than
ACM must be honored. To copy otherwise, distribute, republish, or post, requires prior
speciﬁc permission and/or a fee. Request permissions from permissions@acm.org.
ACSAC ’16, December 05-09, 2016, Los Angeles, CA, USA
c2016 ACM. ISBN 978-1-4503-4771-6/16/12. . . $15.00
DOI:http://dx.doi.org/10.1145/2991079.29911041. INTRODUCTION
The proliferation and popularity of infrastructure-as-a-
service (IaaS) cloud computing services such as Amazon
Web Services and Google Compute Engine means more cloud
tenants are hosting sensitive, private, and business critical
data and applications in the cloud. Unfortunately, IaaS
cloud service providers do not currently furnish the build-
ing blocks necessary to establish a trusted environment for
hosting these sensitive resources. Tenants have limited abil-
ity to verify the underlying platform when they deploy to
the cloud and to ensure that the platform remains in a good
state for the duration of their computation. Additionally,
current practices restrict tenants' ability to establish unique,
unforgeable identities for individual nodes that are tied to a
hardware root of trust. Often, identity is based solely on a
software-based cryptographic solution or unveriable trust
in the provider. For example, tenants often pass unprotected
secrets to their IaaS nodes via the cloud provider.
Commodity trusted hardware, like the Trusted Platform
Module (TPM) [40], has long been proposed as the solution
for bootstrapping trust, enabling the detection of changes to
system state that might indicate compromise, and establish-
ing cryptographic identities. Unfortunately, TPMs have not
been widely deployed in IaaS cloud environments due to a
variety of challenges. First, the TPM and related standards
for its use are complex and dicult to implement. Second,
since the TPM is a cryptographic co-processor and notan
accelerator, it can introduce substantial performance bottle-
necks (e.g., 500+ms to generate a single digital signature).
Lastly, the TPM is a physical device by design and most
IaaS services rely upon virtualization, which purposefully
divorces cloud nodes from the hardware on which they run.
At best, the limitation to physical platforms means that
only the cloud provider would have access to the trusted
hardware, not the tenants [17, 20, 31]. The Xen hypervisor
includes a virtualized TPM implementation that links its se-
curity to a physical TPM [2, 10], but protocols to make use
of the vTPM in an IaaS environment do not exist.
To address these challenges we identify the following de-
sirable features of an IaaS trusted computing system:
Secure Bootstrapping { the system should enable
the tenant to securely install an initial root secret into
each cloud node. This is typically the node's long term
cryptographic identity and the tenant chains other se-
crets to it to enable secure services.
System Integrity Monitoring { the system should
allow the tenant to monitor cloud nodes as they oper-ate and react to integrity deviations within one second.
Secure Layering (Virtualization Support) { the
system should support tenant controlled bootstrapping
and integrity monitoring in a VM using a TPM in the
provider's infrastructure. This must be done in collab-
oration with the provider in least privilege manner.
Compatibility { the system should allow the ten-
ant to leverage hardware-rooted cryptographic keys in
software to secure services they already use (e.g., disk
encryption or conguration management).
Scalability { the system should scale to support boot-
strapping and monitoring of thousands of IaaS resources
as they are elastically instantiated and terminated.
Prior cloud trusted computing solutions address a subset
of these features, but none achieve all. Excalibur [31] sup-
ports bootstrapping at scale, but does not allow for system
integrity monitoring or oer full support for tenant trusted
computing inside a VM (i.e., layering). Manferdelli et al.
created a system that supports secure layering and boot-
strapping, but does not support system integrity monitor-
ing, is incompatible with existing cryptographic services,
and has not demonstrated cloud scale operation [25]. Fi-
nally, the Cloud Verier [34] enables system integrity mea-
surement and cloud scalability but does not fully address
secure layering or enable secure bootstrapping.
In this paper, we introduce keylime ; an end-to-end IaaS
trusted cloud key management service that supports all the
above desired features. The key insight of our work is to
utilize trusted computing to bootstrap identity in the cloud
and provide integrity measurement to support revocation,
but then allow high-level services that leverage these iden-
tities to operate independently. Thus, we provide a clean
and easy to use interface that can integrate with existing
security technologies (see Figure 1).
We introduce a novel bootstrap key derivation protocol
that combines both tenant intent and integrity measurement
to install secrets into cloud nodes. We then leverage the
Cloud Verier [34] pattern of Schiman et al. to enable pe-
riodic attestation that automatically links to identity revo-
cation. keylime supports the above with secure layering in
both bare-metal and virtualized IaaS resources in a manner
that minimizes trust in the cloud provider. We demonstrate
the compatibility of keylime by securely enabling cloud pro-
visioning with cloud-init1, encrypted communication with
IPsec, conguration management with Puppet2, secret man-
agement with Vault3, and storage with LUKS/dm-crypt en-
crypted disks. Unlike existing solutions [39, 25], these ser-
vices don't need to be trusted computing aware, they just
need to use an identity key and respond to key revocations.
Finally, we show that keylime can scale to handle thou-
sands of simultaneous nodes and perform integrity checks
on nodes at rates up to 2,500 integrity reports (quotes) ver-
ied per second. We present and evaluate multiple options
for deploying our integrity measurement verier both in the
cloud, in a low-cost cloud appliance based on a Raspberry
Pi, and on-premises. We show that the overhead of securely
1http://launchpad.net/cloud-init
2http://puppetlabs.com/
3http://hashicorp.com/blog/vault.html
TPM / Platform Manufacturer Enrollment Software-based Cryptographic Services Software ID Keys ID key revoked? KeylimeTrusted Computing Services Signed EKs  Valid TPM? Figure 1: Interface between trusted hardware and
existing software-based security services via the
keylime trusted computing service layer.
provisioning a key using keylime takes less than two sec-
onds. Finally, we nd that our system can detect integrity
measurement violations in as little as 110ms.
2. BACKGROUND
Trusted Computing The TPM provides the means for
creating trusted systems that are amenable to system in-
tegrity monitoring. The TPM, as specied by the Trusted
Computing Group (TCG)4, is a cryptographic co-processor
that provides key generation, protected storage, and crypto-
graphic operations. The protected storage includes a set of
Platform Conguration Registers (PCRs) where the TPM
stores hashes. The TPM uses these registers to store mea-
surements of integrity-relevant components in the system.
To store a new measurement in a PCR, the extend opera-
tion concatenates the existing PCR value with the new mea-
surement, securely hashes5that value, and stores the result-
ing hash in the register. This hash chain allows a verier to
conrm that a set of measurements reported by the system
has not been altered. This report of measurements is called
anattestation , and relies on the quote operation, which ac-
cepts a random nonce and a set of PCRs. These PCRs can
include measurements of the BIOS, rmware, boot loader,
hypervisor, OS, and applications, depending on the congu-
ration of the system. The TPM reads the PCR values, and
then signs the nonce and PCRs with a key that is only ac-
cessible by the TPM. The key the TPM uses to sign quotes
is called an attestation identity key ( AIK ). We denote a
quote using Quote AIK(nonce; PCR i:di:::) for a quote us-
ingAIK from the TPM with the associated nonce and one
or more optional PCR numbers PCR iand corresponding
data dithat will be hashed and placed in PCR i.
The TPM contains a key hierarchy for securely storing
cryptographic keys. The root of this hierarchy is the Stor-
age Root Key (SRK) which the owner generates during TPM
initialization. The SRK in turn protects the TPM AIK (s)
when they are stored outside of the TPM's nonvolatile stor-
age (NVRAM). Each TPM also contains a permanent cre-
dential called the Endorsement Key ( EK). The TPM man-
ufacturer generates and signs the EK. The EK uniquely
identies each TPM and certies that it is a valid TPM
hardware device. The private EK never leaves the TPM,
is never erased, and can only be used for encryption and
decryption during AIK initialization to limit its exposure.
4http://trustedcomputinggroup.org
5TPM specication version 1.2 uses SHA-1 for measure-
ments. TPM specication version 2.0 adds SHA-256 to ad-
dress cryptographic weaknesses in SHA-1.Integrity Measurement To measure a system component,
the underlying component must be trusted computing-aware .
The BIOS in systems with a TPM supports measurement of
rmware and boot loaders. TPM-aware boot loaders can
measure hypervisors and operating systems [22, 19, 29]. To
measure applications, the operating system must support
measurement of applications that are launched, such as the
Linux Integrity Measurement Architecture [30, 21]. One
limitation of approaches like IMA is the inability to mon-
itor the run-time state of the applications. Nexus aims to
address this limitation with a new OS that makes trusted
computing a rst-class citizen, and supports introspection
to validate run-time state [37]. Several proposals exist for
providing run-time integrity monitoring including LKIM [24]
and DynIMA [8]. These systems ensure that a running sys-
temis in a known state, allowing a verier to validate not
only that what was loaded was known, but that it has not
been tampered with while it was running.
In addition to operating system validation, others have
leveraged trusted computing and integrity measurement to
support higher-level services, such as protected access to
data when the client is oine [23], or to enforce access
policies on data [26]. Others have proposed mechanisms
to protect the server from malicious clients, e.g., in online
gaming [1], or applications from a malicious operating sys-
tem [6, 7, 15]. However, these proposals do not account for
the challenges of migrating applications to a cloud environ-
ment, and often assume existing infrastructure to support
trusted computing key management.
IaaS Cloud Services In the IaaS cloud service model, users
request an individual compute resource to execute their ap-
plication. For example, users can provision physical hard-
ware, virtual machines, or containers. In this paper, we
refer to any of these tenant-provisioned IaaS resources as
cloud nodes . Users provision nodes either by uploading a
whole image to the provider or by conguring a pared-down
base image that the provider makes available. Users often
begin by customizing a provider-supplied image, then create
their own images (using a tool like Packer6) to decrease the
amount of time it takes for a node to become ready.
cloud-init is a standard cross-provider (e.g., Amazon
EC2, Microsoft Azure...) mechanism that allows cloud ten-
ants to specify bootstrapping data. It accepts a YAML-
formatted description of what bootstrapping actions should
be taken and supports plugins to take those actions. Exam-
ples of such actions include: adding users, adding package
repositories, or running arbitrary scripts. Users of cloud
computing resources at scale typically spawn new cloud in-
stances using an application programming interface and pass
along enough bootstrapping information to allow the in-
stance to communicate with a conguration management
platform (such as Chef7or Puppet, etc.) for further instance-
specic conguration. These bootstrapping instructions are
not encrypted, meaning that a provider could intercept se-
crets passed via the bootstrapping instructions. In our re-
search, we found that organizations will either (a) send an
unprotected pre-shared key for Puppet in their cloud-init
bootstrapping actions, or (b) rely on some weaker method of
proving identity such as going o the certicate's common
name (hostname).
6https://www.packer.io/
7https://www.chef.io/chef/3. DESIGN
To address the limitations of current approaches, we con-
sider the union of trusted computing and IaaS to provide
a hardware root-of-trust that tenants leverage to establish
trust in the cloud provider's infrastructure and in their own
systems running on that infrastructure. This section consid-
ers the threats that keylime addresses, and how to leverage
existing trusted computing constructs in a virtualized envi-
ronment while limiting complexity and overhead.
3.1 Threat Model
Our goal is to minimize trust in the cloud provider and
carefully account for all concessions we must make to enable
trusted computing services. We assume the cloud provider is
semitrusted, i.e., they are organizationally trustworthy but
are still susceptible to compromise or malicious insiders. We
assume the cloud provider has processes, technical controls,
and policy in place to limit the impact of such compromise
from spreading across their entire infrastructure. Thus, in
the semitrusted model, we assume that some fraction of the
cloud provider's resources may be under the control of the
adversary (e.g., a rogue system administrator may control a
subset of racks in an IaaS region).
Specically, we assume that the adversary can monitor or
manipulate compromised portions of the cloud network or
storage arbitrarily. We assume that the adversary may not
physically tamper with any host's (e.g., hypervisor or bare
metal node) CPU, bus, memory, or TPM8. In virtualized en-
vironments, the security of our system relies upon keeping
cryptographic keys in VM memory. Therefore, we assume
that the provider does not purposefully deploy a hypervisor
with the explicit capability to spy on tenant VM memory
(e.g., Ether [9]). We assume that TPM and system manu-
facturers have created the appropriate endorsement creden-
tials and have some mechanism to test their validity (i.e.,
signed certicates)
Finally, we assume that the attacker's goal is to obtain
persistent access to a tenant system in order to steal, dis-
rupt, or deny the tenant's data and services. To accomplish
persistence the attacker must modify the code loading or
running process. We assume that such modications would
be detected by load-time integrity measurement of the hy-
pervisor or kernel [19], runtime integrity measurement of the
kernel [24], and integrity measurement of applications [30].
3.2 Architecture
To introduce the architecture of keylime we rst describe
a simplied architecture for managing trusted computing
services for a single organization, or cloud tenant, without
virtualization. We then expand this simplied architecture
into the full keylime architecture, providing extensions that
allow layering of provider and tenant trusted computing ser-
vices and supporting multiple varieties of IaaS execution
isolation (i.e., bare metal, virtual machines, or containers).
Figure 3 depicts the full system architecture with layering.
The rst step in bootstrapping the architecture is to create
a tenant-specic registrar. The registrar stores and certies
the public AIK s of the TPMs in the tenant's infrastructure.
In the simplied architecture, the tenant registrar can be
hosted outside the cloud in the tenant's own infrastructure
8This is similar to the threat model assumed by the TPM,
where physical protections are not a strict requirement to
be compliant with the specication.Table 1: Keys used by keylime and their purpose.
Key Type Purpose
EK RSA 2048 Permanent TPM credential that identies the TPM hardware.
SRK RSA 2048 TPM key that protects TPM created private keys when they are stored outside the TPM.
AIK RSA 2048 TPM key used to sign quotes.
Ke AES-256 Enrollment key created by the registrar and used to activate the AIK.
Kb AES-256 Bootstrap key the tenant creates. keylime securely delivers to the node.
U; V 256bit random Trivial secret shares of Kb, derived with random 256bit V:U=KbV.
NK RSA 2048 Non-TPM software key used to protect secret shares U; V in transit.
NodeRegistrarID,AIKpub,EKpubEncEK(H(AIKpub),Ke)HMACKe(ID)
Figure 2: Physical node registration protocol.
or could be hosted on a physical system in the cloud. The
registrar is only a trust root and does not store any tenant
secrets. The tenant can decide to trust the registrar only
after it attests its system integrity. Since the registrar is a
simple a component with static code, verifying its integrity
is straight forward.
To create a registrar, we can leverage existing standards
for the creation and validation of AIK s by creating a TCG
Privacy CA [38]. To avoid the complexity of managing a
complex PKI and because there's no need for privacy within
a single tenant's resources, we created a registrar that simply
stores valid TPM AIK public keys indexed by node UUID.
Clients request public AIK s from the registrar through a
server-authenticated TLS channel.
To validate the AIK s in the registrar, we developed a
TPM-compatible enrollment protocol (see Figure 2). The
node begins by sending its ID and standard TPM creden-
tials ( EKpub; AIK pub) to the registrar. The registrar then
checks the validity of the TPM EK with the TPM manu-
facturer. Importantly, the generation and installation of the
EKby the TPM manufacturer roots the trust upon which
the rest of our system relies. If the EKis valid, the regis-
trar creates an ephemeral symmetric key Keand encrypts it
along with a hash of the public AIK , denoted H(AIK pub),
with the TPM EKpub. The node uses the ActivateIden-
tity TPM command to decrypt Ke. The TPM will only
decrypt Keif has EKprivand if it has AIK privcorrespond-
ing to H(AIK pub). The nodes uses an HMAC to prove that
it can decrypt the ephemeral key Ke. The registrar then
marks that AIK as being valid so that it can be used to
validate quotes.
The core component of keylime is an out of band cloud
verier (CV) similar to the one described by Schiman et
al. [34]. Each cloud organization will have at least one CV
that is responsible for verifying the system state of the orga-
nization's IaaS resources. The tenant can host the CV in the
IaaS cloud or on-premises at their own site (we give options
for tenant registrar and CV deployment in Section 3.2.1).
The CV relies upon the tenant registrar for validating that
theAIK s used to sign TPM quotes are valid, or more specif-
ically, that the AIK s are recognized by the tenant as beingassociated with machines (physical or virtual) owned by the
tenant. The registrar, CV, and cloud node service are the
only components in keylime that manage and use keys and
public key infrastructures associated with the TPM.
The CV participates in a three party key derivation pro-
tocol (we describe in detail in Section 3.2.2) where the CV
and tenant cooperate to derive a key, Kb, at the cloud node
to support initial storage decryption. The tenant uses Kbto
protect tenant secrets and trust relationships. The tenant
can use this key to unlock either its disk image or to unlock
tenant-specic conguration provided by cloud-init .
This protocol is akin to the method by which a user can
use the TPM to decrypt his or her disk in a laptop. To allow
the decryption key to be used to boot the laptop, the user
must enter a password (demonstrating the user's intent) and
TPM PCRs must match a set of whitelisted integrity mea-
surements (demonstrating the validity of the system that
will receive the decryption key). In an IaaS cloud environ-
ment, there is neither a trusted console where a user can
enter a password nor is there a way to pre-seed the TPM
with the storage key or measurement whitelist. Our pro-
tocol uses secret sharing to solve these problems by rely-
ing externally upon the CV for integrity measurement and
by having the tenant directly interact with the cloud node
to demonstrate intent to derive a key. The protocol then
extends beyond bootstrapping to enable continuous system
integrity monitoring. The CV periodically polls each cloud
node's integrity state to determine if any runtime policies
have been violated. The frequency with which the CV re-
quests and veries each node's integrity state will dene the
latency between an integrity violation and detection.
To cleanly link trust and integrity measurement rooted
in the TPM to higher-level services, we create a parallel
software-only PKI and a simple service to manage it. The
goal is to remove the need to make each service trusted
computing-aware, e.g. integrating Trusted Network Connect
into StrongSwan9. We refer to this parallel software-only
service as the software CA. To bootstrap this service, we
use the key derivation bootstrap protocol to create a cloud
node to host the service. Since the bootstrap key derivation
protocol ensures that the node can only derive a key if the
tenant authorizes it and if the node's integrity state is ap-
proved, we can encrypt the private key for the software CA
and pass it to the node upon provisioning. Once established,
we can then start other cloud nodes and securely pass them
keys signed by this CA. The linkage to the hardware root
of trust, the secure bootstrapping of relevant keys, and user
intent to create new resources are again ensured using the
9https://wiki.strongswan.org/projects/strongswan/wiki/
TrustedNetworkConnectTPM / Platform  Manufacturer Tenant Registrar  Tenant CV  Software CA (Cloud Node VM) vTPM Cloud Node VM  vTPM Tenant  
Virtual Enrollment Bound at manufacturing Bootstrap Key derivation 
Identity Key Bootstrap Key KbCertificate authority Trust Hypervisor  TPM Provider Registrar  Deep Quote Provider Whitelist Authority  Hypervisor  TPM Enrollment 
Revocation Service  Signed whitelists 
AIK good? TPM Good? AIK good? vAIK good? 
unwrap Figure 3: Layered keylime trusted computing architecture.
bootstrap key derivation protocol. Once established, stan-
dard tools and services like IPsec or Puppet can now directly
use the software CA identity credentials that each node now
possesses.
To complete the linkage between the trusted computing
services and the software identity keys, we need a mechanism
to revoke keys in the software PKI when integrity violations
occur in the trusted computing layer. The CV is responsible
for notifying the software CA of these violations. The CV
includes metadata about the nature of the integrity viola-
tion, which allows the software CA to have a response policy.
The software CA supports standardized methods for certi-
cate revocation like signed revocation lists or by hosting an
OCSP responder. To support push notications of failures,
the software CA can also publish signed notications to a
message bus. This way services that directly support revo-
cation actions can subscribe to notications (e.g., to trigger
a re-key in a secret manager like Vault).
3.2.1 Layering Trust
We next expand this architecture to work across the layers
of virtualization common in today's IaaS environments. Our
goal is to create the architecture described previously that
cleanly links common security services to a trusted com-
puting layer in a cloud tenant's environment. Thus, in a
VM hosting environment like Amazon EC2 or OpenStack,
we aim to create trusted computing enabled software CAs
and tenant nodes inside of virtual machine instances. Note
that, in a bare-metal provisioning environment like IBM
Softlayer10, HaaS [13], or OpenStack Ironic11, we can di-
rectly utilize the simplied architecture where there is no
trust layering.
We observe that IaaS-based virtual machines or physical
hosts all provide a common abstraction of isolated execu-
tion. Each form of isolated execution in turn needs a root of
trust on which to build trusted computing services. Due to
the performance and resource limitations of typical TPMs
(e.g., taking 500 or more milliseconds to generate a quote,
and only supporting a xed number of PCRs), direct multi-
plexed use of the physical TPM will not scale to the numbers
10http://www.softlayer.com
11https://wiki.openstack.org/wiki/Ironicof virtual machines that can be hosted on a single modern
system. As described by Berger et al. [2] and as implemented
in Xen [10], we utilize a virtualized implementation of the
TPM. Each VM has its own software TPM (called a vTPM)
whose trust is in turn rooted in the hardware TPM of the
hosting system. The vTPM is isolated from the guest that
use it, by running in a separate Xen domain.
The vTPM interface is the same as a hardware TPM. The
only exception to this, is that the client can request a deep-
quote12that will get a quote from the hardware TPM in
addition to getting a quote from the vTPM. These quotes
are linked together by including a hash of the vTPM quote
and nonce in the hardware TPM quote's nonce. Deep quotes
suer from the slow performance of hardware TPMs, but as
we'll show in later this section, we can limit the use of deep
quotes while still maintaining reasonable performance and
scale and maintaining security guarantees.
To assure a chain of trust that is rooted in hardware, we
need the IaaS provider to replicate some of the trusted com-
puting service infrastructure in their environment and allow
the tenant trusted computing services to query it. Specif-
ically, the provider must establish a registrar for their in-
frastructure, must publish an up-to-date signed list of the
integrity measurements of their infrastructure (hosted by a
whitelist authority service), and may even have their own
CV. The tenant CV will interact with the whitelist author-
ity service and the provider's registrar to verify deep quotes
collected by the infrastructure.
Despite the fact that most major IaaS providers run closed-
source hypervisors and would provide opaque integrity mea-
surements [16], we nd there is still value in verifying the
integrity of the provider's services. By providing a known-
good list of integrity measurements, the provider is com-
mitting to a version of the hypervisor that will be deployed
widely across their infrastructure. This prevents a targeted
attack where the a single hypervisor is replaced with a mali-
cious version designed to spy on the tenant (e.g., the provider
is coerced by a government to monitor a specic cloud ten-
12We use similar notation for quotes as we do for deep quotes,
DeepQuote AIK;vAIK (nonce; PCR i:di; vPCR j:dj), ex-
cept that PCRs may be from both physical and virtual sets
of PCRS. We use virtual PCR #16 to bind data.NodeTenant RegistrarID,vAIKpub,vEKpubEncvEK(H(AIKpub),Ke)ProviderRegistrarIDAIKServer	TLS	DeepQuoteAIK,vAIK(H(Ke),v16:H(ID,vAIKpub,vEKpub))OKFigure 4: Virtual node registration protocol.
ant). Thus, an attacker must subvert both the code loading
process on all the hypervisors and the publishing and signing
process for known-good measurements. In our semitrusted
threat model, we assume the provider has controls and mon-
itoring which limit the ability of a rogue individual to ac-
complish this.
As in Section 3.2, we begin with the establishment of a
tenant registrar and cloud verier. There are multiple op-
tions for hosting these services securely: 1) in a bare metal
IaaS instance with TPM, 2) on-tenant-premises in tenant-
owned hardware, 3) in a small trusted hardware appliance
deployed to the IaaS cloud provider, and 4) in an IaaS vir-
tual machine. The rst three of these options rely upon
the architecture and protocols we've already discussed. The
last option requires the tenant to establish an on-tenant-
premises CV and use that to bootstrap the tenant registrar
and CV. This on-tenant-premises CV identies and checks
the integrity of the tenant's virtualized registrar and CV,
who then in turn are responsible for the rest of the tenant's
virtualized infrastructure.
The primary motivations for a tenant choosing between
these options are the detection latency for integrity viola-
tions, scale of IaaS instances in their environment, band-
width between the tenant and the IaaS cloud, and cost. Op-
tion 1 provides maximum performance but at higher cost.
Option 2 will by limited by bandwidth and requires more
costs to maintain resources outside of the cloud. Option 3 is
a good trade-o between cost and performance for a small
cloud tenant with only tens of nodes or who can tolerate a
longer detection latency. Finally, Option 4 provides compat-
ibility with current cloud operations, good performance and
scalability, and low cost at the expense of increased complex-
ity. In Section 5, we examine the performance trade-os of
these options including a low-cost registrar and CV appli-
ance (Option 3) we implemented on a Raspberry Pi.
Once we have created the tenant registrar and CV, we can
begin securely bootstrapping nodes into the environment.
As before, the rst node to establish is a virtualized software
CA and we do this by creating a private signing key oine
and protecting it with a key that will be derived by the
bootstrap key derivation protocol. The following process
will be the same for all tenant cloud nodes. When a node
boots, it will get a vTPM from the IaaS provider.
The process of enrolling a vTPM into the tenant regis-
trar needs to securely associate the vTPM credentials, e.g.,
(vEK ,vAIK ), with a physical TPM in the provider's infras-
tructure (see Figure 4). The tenant registrar cannot directly
verify the authenticity of the vEK because it is virtual and
has no manufacturer. To address this, we use a deep quote to
bind the virtual TPM credentials to a physical TPM AIK .
The vTPM enrollment protocol begins like the physical
TPM enrollment protocol by sending ID;(EKpub; AIK pub)to the tenant registrar. The tenant registrar then returns
EncvEK(H(AIK pub); Ke) without additional checks. The
virtual node then decrypts Keusing ActivateIdentity func-
tion of its vTPM. The node next requests a deep quote using
a hash of Keas the nonce to both demonstrate the freshness
of the quote and knowledge of Keto the tenant registrar.
It also uses virtual PCR #16 to bind the vTPM credentials
and ID to the deep quote. Upon receiving the deep quote,
the tenant registrar asks the provider registrar if the AIK
from the deep quote is valid. The tenant registrar also re-
quests the latest valid integrity measurement whitelists from
the provider. Now the tenant registrar can check the va-
lidity of the deep quote's signature, ensure that the nonce
isH(Ke), conrm the binding data in PCR #16 matches
what was provided in the previous step, and check the phys-
ical PCRs values in the deep quote against the provider's
whitelist. Only if the deep quote is valid will the tenant
registrar mark the vAIK as being valid.
When considering the cost of performing a deep quote,
the provider must carefully consider the additional latency
of the physical TPM. Deep quotes provide a link between
the vTPM and the physical TPM of the machine, and new
enrollments should always include deep quotes. When con-
sidering if deep quotes should be used as part of periodic
attestation, we must understand what trusted computing
infrastructure the provider has deployed. If the provider is
doing load time integrity only (e.g., secure boot), then deep
quotes will only reect the one-time binding at boot between
the vTPM and the physical TPM and the security of the
vTPM infrastructure. If the provider has runtime integrity
checking of their infrastructure, there is value in the ten-
ant performing periodic attestation using deep quotes. In
the optimal deployment scenario, the provider can deploy
keylime and provide tenants with access to the integrity
state of the hypervisors that host tenant nodes. To limit
the impact of slow hardware TPM operations, the provider
can utilize techniques like batch attestation where multiple
deep quote requests from dierent vTPMs can be combined
into a single hardware TPM operation [28, 31].
3.2.2 Key Derivation Protocol
We now introduce the details of our bootstrap key deriva-
tion protocol (Figure 5). The goal of this protocol is for the
cloud tenant to obtain key agreement with a cloud node they
have provisioned in an IaaS system. The protocol relies upon
the CV to provide integrity measurement of the cloud node
during the protocol. The tenant also directly interacts with
the cloud node to demonstrate their intent to spawn that re-
source and allow it to decrypt sensitive contents. However,
the tenant does not directly perform integrity measurement.
This separation of duties is benecial because the attesta-
tion protocols may operate in parallel and it simplies de-
ployment by centralizing all integrity measurement, white
lists, and policy in the CV.
To begin the process, the tenant generates a fresh random
symmetric encryption key Kb. The cloud tenant uses AES-
GCM to encrypt the sensitive data to pass to the node dwith
Kb, denoted EncKb(d). The tenant then performs trivial
secret sharing to split Kbinto two parts U, which the tenant
will retain and pass directly to the cloud node and V, which
the tenant will share with the CV to provide to the node
upon successful verication of the node's integrity state. To
obtain these shares the tenant generates a secure randomTenantCloud Veri0ierNodeRegistrarID, V, IP, port, whitelistnCV, maskQuoteAIK(nCV,16:H(Nkpub),xi :yi),NKpubValid AIK?EncNK(V)ntQuoteAIK(nt,16:H(NKpub))),NKpubValid AIK?
A
B
CMutual TLS Server TLS No TLS Legend	EncNK(U),HMACKb(ID)Figure 5: Three Party Bootstrap Key Derivation Protocol.
value Vthe same length as Kband computes U=KbV.
In the next phase of the protocol, the tenant requests the
IaaS provider to instantiate a new resource (i.e., a new vir-
tual machine). The tenant sends EncKb(d) to the provider
as part of the resource creation. The data dmay be congu-
ration metadata like a cloud-init script13. Upon creation,
the provider returns a unique identier for the node uuid
and an IP address at which the tenant can reach the node.
After obtaining the node uuid and IP address, the tenant
noties the CV of their intent to boot a cloud node (see area
Ain Figure 5). The tenant connects to the CV over a secure
channel, such as mutually authenticated TLS, and provides
v,uuid, node IP, and a TPM policy. The TPM policy speci-
es a white list of acceptable PCR values to expect from the
TPM of the cloud node. At this point the CV and tenant
can begin the attestation protocol in parallel.
The attestation protocol of our scheme is shared between
the interactions of the CV and the cloud node ( B) and that
of the tenant and the cloud node ( C) with only minor dif-
ferences between them (Figure 5). The protocol consists
of two message volleys the rst for the initiator (either CV
or tenant) to request a TPM quote and the second for the
initiator to provide a share of Kbto the cloud node upon
successful validation of the quote. Since we use this pro-
tocol to bootstrap keys into the system, there are no ex-
isting software keys with which we create a secure channel.
Thus, this protocol must securely transmit a share of Kb
over an untrusted network. We accomplish this by creat-
ing an ephemeral asymmetric key pair on the node, denoted
NK, outside of the TPM14. As in Section 3.2.1, we use PCR
#16's value in a TPM quote to bind NK to the identity of
13Because Kbmay not be re-used in our protocol, the cost
of re-encrypting large disk images for each node may be
prohibitive. We advocate for encrypting small sensitive data
packets like a cloud-init script, and then establish local
storage encryption with ephemeral keys.
14NK could also be generated and reside inside the TPM.
However, since it is ephemeral, is only used for transport
security and it is authenticated by the TPM using the quote,
we found the added complexity of also storing it in the TPM
unneeded.the TPM thereby authenticating NK. The initiator can
then encrypt its share of Kbusing NKpuband securely re-
turn it to the cloud node.
The dierences in the attestation protocol between CV
and tenant arise in how each validates TPM quotes. Be-
cause we wish to centralize the adjudication of integrity
measurements to the CV, the TPM quote that the tenant
requests only veries the identity of the cloud node's TPM
and doesn't include any PCR hashes. Since the tenant gen-
erates a fresh Kbfor each cloud node, we are not concerned
with leaking Uto a node with invalid integrity state. Fur-
thermore, because Vis only one share of Kb, the CV cannot
be subverted to decrypt resources without user intent.
We now describe the attestation protocol in detail. The
initiator rst sends a fresh nonce ( ntfor the tenant as in B
from Figure 5 and nCVfor the cloud verier as in Cfrom
Figure 5) to the cloud node along with a mask indicating
which PCRs the cloud node should include in its quote. The
CV sets the mask based on TPM policy exchanged earlier
and the tenant creates an empty mask. We extend a hash of
NKpubinto a freshly reset PCR #16. The initiator requests
a quote from the TPM with the given PCR mask. The node
then returns Quote AIK(n;16 :H(NKpub); xi;:yi); NK pub
to the initiator. Additional PCR numbers xiand values yi
are only included in the quote returned to the cloud verier
based on the TPM policy it requested. During the protocol
to provide U, the tenant also supplies HMAC Kb(ID) to
the node. This provides the node with a quick check to
determine if Kbis correct.
The initiator then conrms that the AIK is valid accord-
ing to the tenant registrar over server authenticated TLS. If
the initiator is the CV, then it will also check the other PCRs
to ensure they are valid according to the tenant-specied
whitelist. If the node is virtual, then the quote to the CV
will also include a deep quote of the underlying hardware
TPM. The CV will in turn validate it as described in the
previous section. Upon successful verication, the initiator
can then return their share of Kb. Thus, the tenant sends
EncNK(U) and the cloud verier sends EncNK(V) to the
node. The cloud node can now recover Kband proceed withthe boot/startup process.
The cloud node does not retain KborVafter decryption
ofd. To support node reboot or migration, the cloud node
stores Uin the TPM NVRAM to avoid needing the tenant
to interact again. After rebooting, the node must again
request verication by the CV to obtain Vand re-derive
Kb. If migration is allowed, the provider must take care to
also securely migrate vTPM state to avoid losing U.
4. IMPLEMENTATION
We implemented keylime in approximately 5,000 lines of
Python in four components: registrar, node, CV, and ten-
ant. We use the IBM Software Trusted Platform module
library [18] to directly interact with the TPM rather than
going through a Trusted Software Stack (TSS) like Trousers.
The registrar presents a REST-based web service for en-
rolling node AIK s. It also supports a query interface for
checking the keys for a given node UUID. The registrar use
HMAC-SHA384 to check the node's knowledge of Keduring
registration.
The node component runs on the IaaS machine, VM, or
container and is responsible for responding to requests for
quotes and for accepting shares of the bootstrap key Kb. It
provides an unencrypted REST-based web service for these
two functions.
To support vTPM operations, we created a service the
IaaS provider runs to manage hardware TPM activation and
vTPM creation/association. This service runs in a desig-
nated Xen domain and has privileges to interact with the
Xen vtpmmgr domain [11]. We then implemented a utility
for the deep quote operation. Since the Xen vTPM imple-
mentation does not directly return the PCR values from the
virtual TPM (i.e., the shallow quote) during a deep quote,
we chose to rst execute a shallow quote, hash its contents
with the provided nonce, and place them in the nonce eld
of the deep quote. This operation cryptographically binds
them together. This operation is not vulnerable to man-in-
the-middle attack since there is no other interface to directly
manipulate the nonce of a deep quote [36]. We then return
both the shallow and deep quotes and require the verier
checks both signatures and sets of PCR values.
The cloud verier hosts a TLS-protected REST-based web
service for control. Tenants add and remove nodes to be
veried and also query their current integrity state. Upon
being notied of a new node, the CV enqueues metadata
about the node onto the quote_request queue where a con-
gurable pool of worker processes will then request a deep
quote from the node. Upon successful verication of the
quote, the CV will use an HTTP POST to send Vto the
node. The CV uses PKCS#1 OAEP and with RSA 2048
keys to protect shares of Kbin transit.
The tenant generates a random 256-bit AES key and en-
crypts and authenticates the bootstrap data using AES with
Galois Counter Mode [27]. The tenant uses trivial XOR-
based secret sharing to split KbintoVandU. The tenant
executes a simplied version of the same protocol that the
CV uses. The tenant checks with the registrar to determine
if the quote signing AIK is valid and owned by the tenant.
Upon receiving UandV, the node can then combine them
to derive Kb. To limit the impact of rogue CVs or tenants
connecting to the node's unauthenticated REST interface,
the node stores all received UandVvalues and iteratively
tries each combination to nd the correct Kb. Once thenode has correctly derived Kb, it mounts a small in-memory
le system using tmpfs and writes the key there for other
applications to access.
4.1 Integration
While the key derivation protocol of keylime is generic
and can be used to decrypt arbitrary data, we believe the
most natural cloud use-case for it is to decrypt a small IaaS
node-specic package of data. To enable this use-case we
have integrated keylime with the cloud-init package, the
combination we call trusted-cloud-init . As described in
Section 2, cloud-init is widely adopted mechanism to de-
ploy machine-specic data to IaaS resources. To integrate
keylime and cloud-init , we patched cloud-init to sup-
port AES-GCM decryption of the user-data (where cloud-
init stores tenant scripts and data). We modied the up-
start system in Ubuntu Linux to start the keylime node
service before cloud-init . We then congure cloud-init
to nd the key that keylime creates in the tmpfs mounted
le system. After successful decryption, cloud-init deletes
the key and scrubs it from memory.
To support applications that need node identities that do
not manage their own PKIs, we implemented a simple soft-
ware CA. The tenant provisions the software CA by creating
the CA private key oine and delivering it to a new node
using trusted-cloud-init . We also deliver certicates to
the software CA that allow it and the tenant to mutually au-
thenticate each other via trusted-cloud-init . To demon-
strate the clean separation between the trusted computing
layer and the software key management layer, we use the
ZMQ Curve secure channel implementation [14]. This sys-
tem uses an elliptic curve cryptography scheme dissimilar
from the cryptographic algorithms, keys, and other tech-
niques the TPM uses.
To enroll a new node, the tenant rst generates a node
identity key pair using the software CA client. The soft-
ware CA supports a plugin architecture that allows the ten-
ant to specify what type of key pairs to create (e.g., X.509
RSA 2048). The tenant then connects securely to the soft-
ware CA over ZMQ and gets the node's identity certicate
signed. The tenant can now provision a new node with this
identity using trusted-cloud-init . The software CA also
supports receiving notications from the CV if a node later
fails integrity measurement.
To support transparent integration with an IaaS plat-
form, we patched OpenStack Nova and libvirt to support
the creation of companion vTPM Xen domains for each
user created instance. We link the OpenStack UUID to the
keylime provider registrar. We then implemented a wrap-
per for the OpenStack Nova command line interface that
enables trusted-cloud-init . Specically, our wrapper in-
tercepts calls to nova boot and automatically encrypts the
provided user-data before passing it to OpenStack. It then
calls the keylime tenant, which begins the bootstrapping
protocol. This allows OpenStack users to transparently use
keylime andcloud-init without needing to fully trust the
OpenStack provider not to tamper or steal the sensitive con-
tents of their user-data .
4.2 Demonstration Applications
We next describe how keylime can securely bootstrap
and handle revocation for existing non-trusted computing-
aware applications and services common to IaaS cloud de-ployments.
IPsec To enable secure network connectivity similar to
TNC [39], we implemented trusted-cloud-init scripts to
automatically encrypt all network trac between a tenant's
IaaS resources. The scripts use the OpenStack API for IP
address information and then build congurations for the
Linux IPsec stack and raccoon15. This conguration is also
easily extensible to a TLS-based VPN like OpenVPN16.
Puppet To enable secure system conguration manage-
ment, we integrated keylime with Puppet We do so by gen-
erating the signed RSA keys that Puppet uses to communi-
cate with the Puppet master using the Software CA process
described previously. These steps bypass the need to either
use the insecure autosign option in the Puppet master to
blindly accept new nodes or to have an operator manually
approve/deny certicate signing requests from new nodes.
To support continuous attestation and integrity measure-
ment, we implemented a plug-in for the CV that noties the
tenant's Puppet master when a node fails its integrity mea-
surements. The master can then revoke that node's access
to check-in and receive the latest conguration data.
Vault While tools like Puppet are often used to provi-
sion secrets and keys, tenant operators can instead use a
dedicated secret management system that supports the full
lifecycle of cryptographic keys directly. To demonstrate this,
we have integrated keylime with Vault, a cloud-compatible
secret manager. Like Puppet, we use the Software CA to
provision RSA certicates for each node and congure Vault
to use them. We also implemented a revocation plugin for
the CV that noties Vault to both revoke access to a node
that fails integrity measurement and to re-generate and re-
distribute any keys to which that node had access.
LUKS Finally, to demonstrate our ability to provision
secrets instead of cryptographic identities, we implemented a
trusted-clout-init script that provides the key to unlock
an encrypted volume on boot.
5. EV ALUATION
In this section we evaluate the overhead and scalability of
keylime in a variety of scenarios. We ran our experiments
on a private OpenStack cluster, a Xen host, and a Raspberry
Pi. In OpenStack, we used standard instance avors where
the m1.small has 1 vCPU, 2GB RAM, and a 20GB disk,
and the m1.large has 4 vCPUs, 8GB RAM, an 80GB disk.
We used Ubuntu Linux 14.10 as the guest OS in OpenStack
instances. The Xen host had one Xeon E5-2640 CPU with
6 cores at 2.5Ghz, 10Gbit NIC, 64 GB RAM, a WinBond
TPM, and ran Xen 4.5 on Ubuntu Linux 15.04. The Rasp-
berry Pi 2 had one ARMv7 with 4-cores at 900Mhz, 1GB
RAM, 100Mbit NIC, and ran Raspbian 7. We ran each of
the following experiments for 1-2 minutes and present aver-
ages of the performance we observed.
5.1 TPM Operations
We rst establish a baseline for the performance of TPM
operations with the IBM client library, our Python wrap-
per code, the Xen virtual TPM, and the physical TPM. We
benchmarked both TPM quote creation and verication on
the Xen host (Table 2). We collected the physical TPM mea-
surements on the same system with a standard (non-Xen)
15http://ipsec-tools.sourceforge.net/
16https://openvpn.net/Table 2: Average TPM Operation Latency (ms).
TPM vTPM Deep quote
Create Quote 725 68.5 1390
Check Quote 4.64 4.64 5.33
kernel. We collected both vTPM quote and deep quote mea-
surements from a domain running on Xen. As expected, op-
erations that require interaction with the physical TPM are
slow. Verication times, even for deep quotes that include
two RSA signature verications, are comparatively quick.
5.2 Key Derivation Protocol
We next investigate the CV latency of dierent phases
of our protocol. In Figures 6 and 7, we show the aver-
aged results of hundreds of trials of the CV with 100 vTPM
equipped VMs. Each operation includes a full REST in-
teraction along with the relevant TPM and cryptographic
operations. We also benchmarked the latency of the pro-
tocol phases emulating zero latency from the TPM (Null
TPM). This demonstrates the minimum latency of our CV
software architecture including the time required to verify
quotes. The results from the Null TPM trials indicate that
our network protocol and other processing impose very little
additional overhead, even on the relatively modestly pow-
ered Raspberry Pi. The bare metal system had a slightly
larger network RTT to the nodes it was verifying causing it
to have a higher latency than the less powerful m1.large.
In Figure 7, we see that latency for the quote retrieval
process is primarily aected by slow TPM operations and is
comparable to prior work [31]. The bootstrapping latency
is the sum of the latencies for retrieving a quote and provid-
ing V. We nd the bootstrapping latency for bare metal and
virtual machines to be approximately 793ms and 1555ms re-
spectively. Virtual nodes doing runtime integrity measure-
ment after bootstrapping benet from much lower latency
for vTPM operations. Thus, for a virtual machine with a
vTPM, keylime can detect integrity violations in as little as
110ms. The detection latency for a system with a physical
TPM (781ms for our Xen host) is limited by the speed of
the physical TPM at generating quotes.
5.3 Scalability of Cloud Veriﬁer
Next we establish the maximum rate at which the CV can
get and check quotes for sustained integrity measurement.
This will dene the trade-o between the number of nodes
a single CV can handle and the latency between when an
integrity violation occurs and the CV detects it. Since the
CV quote checking process is a simple round robin check of
each node, it is easy to parallelize across multiple CVs fur-
ther enhancing scalability. We emulate an arbitrarily large
population of real cloud nodes using a xed number test
cloud nodes. These test cloud nodes emulate a zero latency
TPM by returning a pre-constructed quote. This way the
test nodes appear like a larger population where the CV will
never have to block for a lengthy TPM operation to com-
plete. We found that around 500 zero latency nodes were
sucient to achieve the maximum quote checking rate.
We show the average number of quotes veried per second
for each of our CV deployment options in Figure 8. Because
of our process-based worker pool architecture, the primary
factor aecting CV scalability is the number of cores and 0 10 20 30 40 50 60
Provide V Null TPMLatency (ms)Bare Metal
m1.large
m1.small
Raspberry PiFigure 6: Latency of keylime boot-
strapping protocol.
 0 200 400 600 800 1000 1200 1400 1600
vTPM Quote TPM Quote DeepQuoteLatency (ms)Bare Metal
m1.large
m1.small
Raspberry PiFigure 7: Latency of TPM opera-
tions in bootstrapping protocol.
 0 500 1000 1500 2000 2500
Bare Metal m1.large m1.smallRaspberry PiIntegrity Check Rate (quotes/s)Figure 8: Maximum CV quote
checking rate of keylime .
 0 500 1000 1500 2000 2500
 0  50  100  150  200  250  300Rate (quotes/s)
Num CV Processes
Figure 9: Scaling the CV on bare metal
RAM available. These options provide a variety of choices
for deploying the CV. For small cloud tenants a low cost
VM or inexpensive hardware appliance can easily verify hun-
dreds of virtual machines with moderate detection latency
(5-10s). For larger customers, a well-resourced VM or dedi-
cated hardware can scale to thousands with similar latency.
For high security environments, all options can provide sub-
second detection and response time. In future work, we plan
to implement a priority scheme that allows the tenant to set
the rate of verications for dierent classes of nodes.
We next show how our CV architecture can scale by adding
more cores and parallelism. We use the bare metal CV and
show the average rate of quotes retrieved and checked per
second for 500 test nodes in Figure 9. We see linear speed-
up until we exhaust the parallelism of the host CPU and
the concurrent delay of waiting for many cloud nodes. This
performance represents a modest performance improvement
over Schiman et al.'s CV which was able to process approx-
imately 2,000 quotes per second on unspecied hardware [34]
and a substantial improvement over the Excalibur monitor's
ability to check approximately 633 quotes per second [31].
5.4 On-Premises Cloud Veriﬁer
Finally, we investigate the performance of the CV when
hosted at the tenant site away from the cloud. We show
the results of our bare metal system's quote verication
rate and the bandwidth used for a variety of network de-Table 3: On-Premises bare metal CV verifying 250
Cloud Nodes using 50 CV processes.
Network RTT Rate Bandwidth
(ms) (quotes/s) (Kbits/s)
4ms (native) 937 3085
25ms 613 2017
50ms 398 1310
75 ms 282 928.3
100 ms 208 684.7
150 ms 141 464.2
lays we inserted using the comcast17tool in Table 3. These
results show that it is possible to run the CV on-premises at
the tenant site at the cost of a reduction in quote checking
rate (and therefore detection latency) and several Mbit/s
of bandwidth. As such, we recommend the highest perfor-
mance and lowest cost option is to run the CV in the cloud
alongside the nodes it will verify.
6. RELATED WORK
Many of the challenges that exist in traditional enterprise
networks exist in cloud computing environments as well.
However, there are new challenges and threats including
shared tenancy and additional reliance on the cloud provider
to provide a secure foundation [4, 16]. To address some of
the challenges, many have proposed trusted computing.
The existing specications for trusted computing rely on
trusted hardware, and assume a single owner of the system.
With the advent of cloud computing, this assumption is no
longer valid. While both the standards community [41] and
prior work [2] is beginning the process of supporting virtu-
alization, no end-to-end solution exists. For example, the
cTPM system [5] assumes a trustworthy cloud provider and
requires modications to trusted computing standards. An-
other proposal for higher-level validation of services provides
a cryptographically signed audit trail that the hypervisor
provides to auditors [12]. The audit trail captures the exe-
cution of the applications within the virtual machine. This
proposal does not provide a trusted foundation for the audit
trail, and assumes a benign hypervisor. Bleikertz, et al., pro-
pose to use trusted computing to provide cryptographic ser-
vices for cloud tenants[3]. Their Cryptography-as-a-Service
(CaaS) system relies on trusted computing, but does not ad-
dress bootstrapping and requires hypervisor modications
17https://github.com/tylertreat/Comcastthat cloud providers are unlikely to support.
To address the issues of scalability, several proposals ex-
ist to monitor a cloud infrastructure, and allow for valida-
tion of the virtual machines controlled by the tenants of the
cloud [34, 32, 33, 35]. The cloud verier pattern proposed
by Schiman, et al., allows a single verier to validate trust
in the cloud infrastructure, and in turn the cloud verier
\vouches" for the integrity of the cloud nodes. This removes
the need for tenants to validate the integrity of the hyper-
visor hosts prior to instantiating cloud nodes on them and
avoids the need for nodes to mutually attest each other be-
fore communicating. The tenant simply provides their in-
tegrity verication criteria to the verier, and the verier en-
sures that the tenant's integrity criteria are satised as part
of scheduling resources. We utilize the cloud verier pat-
tern in our work, with some important dierences. First we
extend it to support secure system bootstrapping for both
bare metal and virtualized IaaS environments. Second, we
do not host any tenant-owned parts of the integrity mea-
surement infrastructure in the provider's control as they do.
This means that our solution is substantially less invasive
to the cloud provider's infrastructure (e.g., they required
nearly 5,000 lines of code to be added to OpenStack) and
is less prone to compromise. For example, keylime relies
upon the vTPM integrity measurements inside tenant nodes
rather than enabling the cloud provider to have explicit vir-
tual machine introspection (i.e., secret stealing) capabilities.
Excalibur works to address the scalability problems of
trusted computing by leveraging ciphertext policy attribute-
based encryption (CPABE) [31]. This encryption scheme
allows data to be encrypted using keys that represent at-
tributes of the hypervisor hosts in the IaaS environment
(e.g., software version, country, zone). Using Excalibur,
clients can encrypt sensitive data, and be assured that a
hypervisor will only be given access to the data if the policy
(the specied set of attributes) is satised. Excalibur only
addresses trusted bootstrapping for the underlying cloud
platform. Therefore, a compromised tenant node would be
neither detected nor prevented. The Excalibur monitor is
a provider-owned (but attested) component that holds the
encryption keys that allow a node to boot on a particu-
lar hypervisor. keylime uses secret sharing to avoid having
bootstrap key stored (and therefore vulnerable) in any cloud
systems except for in the cloud node for which they are in-
tended.
The CloudProxy Tao system provides building blocks to
establish trusted services in a layered cloud environment [25].
The Tao environment relies on the TPM to establish iden-
tity and load-time integrity of the nodes and software in
the system. Their system does not support system integrity
monitoring as they assume that all interactions will only be
with other trusted programs running in the Tao environ-
ment. Tao relies upon mutual attestation for all communi-
cating nodes, but is unable to use TPM-based keys because
they are not fast enough to support mutual attestation. Us-
ing the out-of-band CV, we avoid mutual attestation while
maintaining rapid detection of integrity violations. The Key
Server in Tao holds all the secret keys to the system, must
interact with hosts to load new applications, and must be
fully trusted. The Key Server does not oer compatible de-
ployment options for IaaS environments, especially for small
tenants who cannot aord secure facilities or hardware se-
curity modules. Furthermore, CloudProxy Tao does not de-tail the secure bootstrapping of their Key Server or Privacy
CA component for TPM initialization. keylime explicitly
describes bootstrapping of all relevant components and en-
ables multiple realistic secure deployment options for CV
and registrar hosting.
7. CONCLUSION
In this paper, we have shown that keylime provides a
fully integrated solution to bootstrap and maintain hardware-
rooted trust in elastically provisioned IaaS clouds. We have
demonstrated that we can bootstrap hardware-rooted cryp-
tographic identities into both physical and virtual cloud
nodes, and leverage those identities in higher-level security
services, without requiring each service to become trusted
computing-aware. keylime uses a novel key derivation pro-
tocol that incorporates a tenant's intent to provision new
cloud resources with integrity measurement. Finally, we
have demonstrated and evaluated several deployment sce-
narios for our system's critical component, the cloud ver-
ier. keylime can securely derive a key in less than two
seconds during the provisioning and bootstrapping process,
and requires as little as 110ms to respond to an integrity
violation. Furthermore, we have shown that keylime can
scale to support thousands of IaaS nodes while maintaining
quick response to integrity violations.
8. REFERENCES
[1] S. Balfe and A. Mohammed. Final fantasy { securing
on-line gaming with trusted computing. In B. Xiao,
L. Yang, J. Ma, C. Muller-Schloer, and Y. Hua,
editors, Autonomic and Trusted Computing , volume
4610 of Lecture Notes in Computer Science , pages
123{134. Springer Berlin Heidelberg, 2007.
[2] S. Berger, R. C aceres, K. A. Goldman, R. Perez,
R. Sailer, and L. van Doorn. vtpm: Virtualizing the
trusted platform module. In Proceedings of the 15th
Conference on USENIX Security Symposium - Volume
15, USENIX-SS'06, Berkeley, CA, USA, 2006.
USENIX Association.
[3] S. Bleikertz, S. Bugiel, H. Ideler, S. N urnberger, and
A.-R. Sadeghi. Client-controlled
cryptography-as-a-service in the cloud. In
M. Jacobson, M. Locasto, P. Mohassel, and
R. Safavi-Naini, editors, Applied Cryptography and
Network Security , volume 7954 of Lecture Notes in
Computer Science , pages 19{36. Springer Berlin
Heidelberg, 2013.
[4] S. Bouchenak, G. Chockler, H. Chockler,
G. Gheorghe, N. Santos, and A. Shraer. Verifying
cloud services: Present and future. SIGOPS Oper.
Syst. Rev. , 47(2):6{19, July 2013.
[5] C. Chen, H. Raj, S. Saroiu, and A. Wolman. ctpm: A
cloud tpm for cross-device trusted applications. In
11th USENIX Symposium on Networked Systems
Design and Implementation (NSDI 14) , Seattle, WA,
Apr. 2014. USENIX Association.
[6] X. Chen, T. Garnkel, E. C. Lewis, P. Subrahmanyam,
C. A. Waldspurger, D. Boneh, J. Dwoskin, and D. R.
Ports. Overshadow: A virtualization-based approach
to retrotting protection in commodity operating
systems. SIGPLAN Not. , 43(3):2{13, Mar. 2008.[7] J. Criswell, N. Dautenhahn, and V. Adve. Virtual
ghost: Protecting applications from hostile operating
systems. SIGARCH Comput. Archit. News ,
42(1):81{96, Feb. 2014.
[8] L. Davi, A.-R. Sadeghi, and M. Winandy. Dynamic
integrity measurement and attestation: towards
defense against return-oriented programming attacks.
InProceedings of the 2009 ACM workshop on Scalable
trusted computing , pages 49{54. ACM, 2009.
[9] A. Dinaburg, P. Royal, M. Sharif, and W. Lee. Ether:
Malware analysis via hardware virtualization
extensions. In Proceedings of the 15th ACM
Conference on Computer and Communications
Security , CCS '08, pages 51{62, New York, NY, USA,
2008. ACM.
[10] M. Fioravante and D. D. Graaf. vTPM.
http://xenbits.xen.org/docs/unstable/misc/vtpm.txt,
November 2012.
[11] D. D. Graaf and Q. Xu. vTPM manager.
http://xenbits.xen.org/docs/unstable/misc/vtpmmgr.txt.
[12] A. Haeberlen, P. Aditya, R. Rodrigues, and
P. Druschel. Accountable virtual machines. In
Proceedings of the 9th USENIX Conference on
Operating Systems Design and Implementation ,
OSDI'10, pages 1{16, Berkeley, CA, USA, 2010.
USENIX Association.
[13] J. Hennessey, C. Hill, I. Denhardt, V. Venugopal,
G. Silvis, O. Krieger, and P. Desnoyers. Hardware as a
service - enabling dynamic, user-level bare metal
provisioning of pools of data center resources. In 2014
IEEE High Performance Extreme Computing
Conference , Waltham, MA, USA, Sept. 2014.
[14] P. Hintjens. Curvezmq authentication and encryption
protocol. http://rfc.zeromq.org/spec:26, 2013.
[15] O. S. Hofmann, S. Kim, A. M. Dunn, M. Z. Lee, and
E. Witchel. Inktag: Secure applications on an
untrusted operating system. SIGPLAN Not. ,
48(4):265{278, Mar. 2013.
[16] W. Huang, A. Ganjali, B. H. Kim, S. Oh, and D. Lie.
The state of public infrastructure-as-a-service cloud
security. ACM Comput. Surv. , 47(4):68:1{68:31, June
2015.
[17] IBM. Ibm and intel bring new security features to the
cloud. http://www.softlayer.com/press/
ibm-and-intel-bring-new-security-features-cloud,
September 2004.
[18] IBM. Software trusted platform module.
http://sourceforge.net/projects/ibmswtpm/, April
2014.
[19] Intel. Intel Trusted Boot (tboot).
https://software.intel.com/en-us/articles/
intel-trusted-execution-technology.
[20] Intel. Cloud integrity technology. http://www.intel.
com/p/en US/support/highlights/sftwr-prod/cit,
2015.
[21] T. Jaeger, R. Sailer, and U. Shankar. Prima:
Policy-reduced integrity measurement architecture. In
Proceedings of the Eleventh ACM Symposium on
Access Control Models and Technologies , SACMAT
'06, pages 19{28, New York, NY, USA, 2006. ACM.
[22] B. Kauer. Oslo: Improving the security of trusted
computing. In Proceedings of 16th USENIX SecuritySymposium on USENIX Security Symposium , SS'07,
pages 16:1{16:9, Berkeley, CA, USA, 2007. USENIX
Association.
[23] R. Kotla, T. Rodeheer, I. Roy, P. Stuedi, and
B. Wester. Pasture: Secure oine data access using
commodity trusted hardware. In Presented as part of
the 10th USENIX Symposium on Operating Systems
Design and Implementation (OSDI 12) , pages
321{334, Hollywood, CA, 2012. USENIX.
[24] P. A. Loscocco, P. W. Wilson, J. A. Pendergrass, and
C. D. McDonell. Linux kernel integrity measurement
using contextual inspection. In Proceedings of the 2007
ACM Workshop on Scalable Trusted Computing , STC
'07, pages 21{29, New York, NY, USA, 2007. ACM.
[25] J. Manferdelli, T. Roeder, and F. Schneider. The
cloudproxy tao for trusted computing. Technical
Report UCB/EECS-2013-135, EECS Department,
University of California, Berkeley, Jul 2013.
[26] P. Maniatis, D. Akhawe, K. Fall, E. Shi, S. McCamant,
and D. Song. Do you know where your data are?:
Secure data capsules for deployable data protection. In
Proceedings of the 13th USENIX Conference on Hot
Topics in Operating Systems , HotOS'13, pages 22{22,
Berkeley, CA, USA, 2011. USENIX Association.
[27] D. A. McGrew and J. Viega. The galois/counter mode
of operation (gcm). NIST , 2005.
[28] T. Moyer, K. Butler, J. Schiman, P. McDaniel, and
T. Jaeger. Scalable web content attestation. IEEE
Transactions on Computers , Mar 2011.
[29] S. Munetoh. GRUB TCG Patch to Support Trusted
Boot. http://trousers.sourceforge.net/grub.html.
[30] R. Sailer, X. Zhang, T. Jaeger, and L. van Doorn.
Design and implementation of a tcg-based integrity
measurement architecture. In Proceedings of the 13th
Conference on USENIX Security Symposium - Volume
13, SSYM'04, pages 16{16, Berkeley, CA, USA, 2004.
USENIX Association.
[31] N. Santos, R. Rodrigues, K. P. Gummadi, and
S. Saroiu. Policy-sealed data: A new abstraction for
building trusted cloud services. In Presented as part of
the 21st USENIX Security Symposium (USENIX
Security 12) , pages 175{188, Bellevue, WA, 2012.
USENIX.
[32] J. Schiman, T. Moyer, C. Shal, T. Jaeger, and
P. McDaniel. Justifying integrity using a virtual
machine verier. In Computer Security Applications
Conference, 2009. ACSAC '09. Annual , pages 83{92,
Dec 2009.
[33] J. Schiman, T. Moyer, H. Vijayakumar, T. Jaeger,
and P. McDaniel. Seeding clouds with trust anchors.
InProceedings of the 2010 ACM Workshop on Cloud
Computing Security Workshop , CCSW '10, pages
43{46, New York, NY, USA, 2010. ACM.
[34] J. Schiman, Y. Sun, H. Vijayakumar, and T. Jaeger.
Cloud verier: Veriable auditing service for iaas
clouds. In Services (SERVICES), 2013 IEEE Ninth
World Congress on , pages 239{246, June 2013.
[35] J. Schiman, H. Vijayakumar, and T. Jaeger.
Verifying system integrity by proxy. In Proceedings of
the 5th International Conference on Trust and
Trustworthy Computing , TRUST'12, pages 179{200,
Berlin, Heidelberg, 2012. Springer-Verlag.[36] A. Segall. Using the tpm: Machine authentication and
attestation. http://opensecuritytraining.info/
IntroToTrustedComputing les/Day2-1-auth-and-att.
pdf, Oct 2012.
[37] E. G. Sirer, W. de Bruijn, P. Reynolds, A. Shieh,
K. Walsh, D. Williams, and F. B. Schneider. Logical
attestation: An authorization architecture for
trustworthy computing. In Proceedings of the
Twenty-Third ACM Symposium on Operating Systems
Principles , SOSP '11, pages 249{264, New York, NY,
USA, 2011. ACM.
[38] Trusted Computing Group. TCG Infrastructure
Working Group A CMC Prole for AIK Certicate
Enrollment. http:
//www.trustedcomputinggroup.org/les/resource
les/738DF0BB-1A4B-B294-D0AF6AF9CC023163/
IWG CMC Prole Cert Enrollment v1r7.pdf.
[39] Trusted Computing Group. Trusted Network
Communications.
http://www.trustedcomputinggroup.org/developers/
trusted network communications.
[40] Trusted Computing Group. Trusted Platform Module.
http://www.trustedcomputinggroup.org/developers/
trusted platform module.
[41] Trusted Computing Group. Virtualized Platform.
http://www.trustedcomputinggroup.org/developers/
virtualized platform.