Patent ID: 12189542

DETAILED DESCRIPTION OF THE DRAWINGS

While the concepts of the present disclosure are susceptible to various modifications and alternative forms, specific embodiments thereof have been shown by way of example in the drawings and will be described herein in detail. It should be understood, however, that there is no intent to limit the concepts of the present disclosure to the particular forms disclosed, but on the contrary, the intention is to cover all modifications, equivalents, and alternatives consistent with the present disclosure and the appended claims.

References in the specification to “one embodiment,” “an embodiment,” “an illustrative embodiment,” etc., indicate that the embodiment described may include a particular feature, structure, or characteristic, but every embodiment may or may not necessarily include that particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same embodiment. Further, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is submitted that it is within the knowledge of one skilled in the art to effect such feature, structure, or characteristic in connection with other embodiments whether or not explicitly described. Additionally, it should be appreciated that items included in a list in the form of “at least one A, B, and C” can mean (A); (B); (C); (A and B); (A and C); (B and C); or (A, B, and C). Similarly, items listed in the form of “at least one of A, B, or C” can mean (A); (B); (C); (A and B); (A and C); (B and C); or (A, B, and C).

The disclosed embodiments may be implemented, in some cases, in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried by or stored on a transitory or non-transitory machine-readable (e.g., computer-readable) storage medium, which may be read and executed by one or more processors. A machine-readable storage medium may be embodied as any storage device, mechanism, or other physical structure for storing or transmitting information in a form readable by a machine (e.g., a volatile or non-volatile memory, a media disc, or other media device).

In the drawings, some structural or method features may be shown in specific arrangements and/or orderings. However, it should be appreciated that such specific arrangements and/or orderings may not be required. Rather, in some embodiments, such features may be arranged in a different manner and/or order than shown in the illustrative figures. Additionally, the inclusion of a structural or method feature in a particular figure is not meant to imply that such feature is required in all embodiments and, in some embodiments, may not be included or may be combined with other features.

Referring now toFIG.1, a computing device100for secure I/O with an accelerator device includes a processor120and one or more I/O devices. In use, as described further below, the computing device100includes a virtual machine monitor (VMM) and one or more trusted execution environments (TEE). The VMM and the TEE do not mutually trust each other (e.g., the VMM is not included in the trusted computing base (TCB) of the TEE). The computing device100further includes a trusted agent that is trusted by both the VMM and the TEE. The trusted agent verifies the identity of I/O devices capable of trusted I/O (TIO devices), provisions secrets to the TIO devices, locks, unlocks, and configures the TIO devices, and provides attestation information and device configuration information for the TIO device and the trusted agent to the TEE. Thus, the computing device100allows enables trusted I/O between a TIO device and a TEE by providing secure configuration and management of trusted I/O devices. The trusted agent operates under the overall control and management of the VMM while still providing assurances to the TEE that the TIO device is configured correctly and will operate as configured during trusted I/O. The computing device100does not require that the VMM be within the TCB of the TEE and is thus suitable for datacenter use or other multi-tenant systems.

The computing device100may be embodied as any type of device capable of performing the functions described herein. For example, the computing device100may be embodied as, without limitation, a computer, a server, a workstation, a laptop computer, a tablet computer, a notebook computer, a mobile computing device, a smartphone, a wearable computing device, a multiprocessor system, and/or a consumer electronic device. As shown inFIG.1, the illustrative computing device100includes a processor120, an I/O subsystem124, a memory128, and a data storage device130. Additionally, in some embodiments, one or more of the illustrative components may be incorporated in, or otherwise form a portion of, another component. For example, the memory128, or portions thereof, may be incorporated in the processor120in some embodiments.

The processor120may be embodied as any type of processor capable of performing the functions described herein. For example, the processor120may be embodied as a single or multi-core processor(s), digital signal processor, microcontroller, or other processor or processing/controlling circuit. As shown, the processor120illustratively includes secure enclave support122, which allows the processor120to establish a trusted execution environment known as a secure enclave, in which executing code may be measured, verified, and/or otherwise determined to be authentic. Additionally, code and data included in the secure enclave may be encrypted or otherwise protected from being accessed by code executing outside of the secure enclave. For example, code and data included in the secure enclave may be protected by hardware protection mechanisms of the processor120while being executed or while being stored in certain protected cache memory of the processor120. The code and data included in the secure enclave may be encrypted when stored in a shared cache or the main memory128. The secure enclave support122may be embodied as a set of processor instruction extensions that allows the processor120to establish one or more secure enclaves in the memory128. For example, the secure enclave support122may be embodied as Intel® Software Guard Extensions (SGX) technology.

The memory128may be embodied as any type of volatile or non-volatile memory or data storage capable of performing the functions described herein. In operation, the memory128may store various data and software used during operation of the computing device100such as operating systems, applications, programs, libraries, and drivers. As shown, the memory128is coupled to the processor120and/or the I/O subsystem124via a multi-key total memory encryption engine (MKTME)126, which may be included in or otherwise coupled to a memory controller, integrated memory controller hub, or other memory interface. The MKTME126allows the computing device100to transparently encrypt the contents of the memory128. The MKTME126maintains a table or other internal, protected structure with multiple encryption keys, which are used to encrypt and decrypt data as it is stored to and read from the memory128, respectively. The encryption keys are illustratively 128-bit AES XTS keys although may be embodied as any symmetric, asymmetric, or other encryption key. The encryption key may be selected by the MKTME126on a per-page basis, for example based on a key identifier included in one or more otherwise unused upper bits of the physical memory page address for a particular memory access. In those embodiments, an operating system, virtual memory monitor, or other supervisory component of the computing device100may control access to particular memory pages by configuring one or more page tables and/or extended page tables with the appropriate key identifiers. MKTME keys may be generated by the MKTME126, in which case they are not disclosed outside of the SoC, or may be supplied by software. In some embodiments, the MKTME126may include support for Intel Trust Domain Extensions (TDX). With TDX, the MKTME126may accept an external “domain” key, also called a “user” or “tenant” key. The MKTME126may also use a default key that is self-generated to protect memory used by MKTME and Intel SGX as well as Intel TDX from privileged adversaries on the platform. Although illustrated as coupled between the memory128and the processor120and I/O subsystem124, it should be understood that in some embodiments, the MKTME126may be included in the processor120, in the I/O subsystem124, or other component of the computing device100.

As shown, the processor120is communicatively coupled to the I/O subsystem124, which may be embodied as circuitry and/or components to facilitate input/output operations with the processor120, the memory128, and other components of the computing device100. For example, the I/O subsystem124may be embodied as, or otherwise include, memory controller hubs, input/output control hubs, sensor hubs, host controllers, firmware devices, communication links (i.e., point-to-point links, bus links, wires, cables, light guides, printed circuit board traces, etc.) and/or other components and subsystems to facilitate the input/output operations. As shown, the memory128may be directly coupled to the processor120, for example via an integrated memory controller hub. Additionally, in some embodiments, the I/O subsystem124may form a portion of a system-on-a-chip (SoC) and be incorporated, along with the processor120, the memory128, the accelerator134, and/or other components of the computing device100, on a single integrated circuit chip. Additionally or alternatively, in some embodiments the processor120may include an integrated memory controller and a system agent, which may be embodied as a logic block in which data traffic from processor cores and I/O devices converges before being sent to the memory128.

The data storage device130may be embodied as any type of device or devices configured for short-term or long-term storage of data such as, for example, memory devices and circuits, memory cards, hard disk drives, solid-state drives, non-volatile flash memory, or other data storage devices. The computing device100may also include a communications subsystem132, which may be embodied as any communication circuit, device, or collection thereof, capable of enabling communications between the computing device100and other remote devices over a computer network (not shown). The communications subsystem132may be configured to use any one or more communication technology (e.g., wired or wireless communications) and associated protocols (e.g., Ethernet, Bluetooth®, Wi-Fi®, WiMAX, 3G, 4G LTE, etc.) to effect such communication.

The accelerator134may be embodied as a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a graphics processor unit (GPU), a coprocessor, an I/O device, or other digital logic device capable of performing accelerated functions (e.g., accelerated application functions, accelerated network functions, or other accelerated functions). Illustratively, the accelerator134is an FPGA, which may be embodied as an integrated circuit including programmable digital logic resources that may be configured after manufacture. The FPGA may include, for example, a configurable array of logic blocks in communication over a configurable data interchange.

As shown, the computing device100may further include one or more peripheral devices136. The peripheral devices136may include any number of additional input/output devices, interface devices, hardware accelerators, and/or other peripheral devices. For example, in some embodiments, the peripheral devices136may include a touch screen, graphics circuitry, a graphical processing unit (GPU) and/or processor graphics, an audio device, a microphone, a camera, a keyboard, a mouse, a network interface, and/or other input/output devices, endpoints, interface devices, and/or peripheral devices. As shown, one or more of the peripheral devices136may be coupled to the I/O subsystem124via corresponding I/O switches138. The I/O switches138may be embodied as PCI Express (PCIe) switches, PCIe bridges, and/or other switching or routing components of the computing device100. Thus, the computing device100may include a hierarchical system of connected I/O devices, switches, buses, links, and/or other I/O components.

Referring now toFIG.2, in an illustrative embodiment, the computing device100establishes an environment200during operation. The illustrative environment200includes a host partition202including a provisioning agent204, multiple trusted execution environments (TEEs)208, a virtual machine monitor (VMM)214, and trusted I/O (TIO) devices216. The various components of the environment200may be embodied as hardware, firmware, software, or a combination thereof. As such, in some embodiments, one or more of the components of the environment200may be embodied as circuitry or collection of electrical devices (e.g., provisioning agent circuitry204, TEE circuitry208, VMM circuitry214, and/or TIO device circuitry216). It should be appreciated that, in such embodiments, one or more of the provisioning agent circuitry204, the TEE circuitry208, the VMM circuitry214, and/or the TIO device circuitry216may form a portion of the processor120, the I/O subsystem124, the accelerator134, the peripheral devices136, and/or other components of the computing device100. Additionally, in some embodiments, one or more of the illustrative components may form a portion of another component and/or one or more of the illustrative components may be independent of one another.

The VMM214may be embodied as any virtual machine monitor, hypervisor, or other component that manages execution of one or more virtualized workloads on the computing device100. The VMM214may manage workloads using hardware support of the processor120, for example by executing in a privileged state such as VMX root, ring level −1, or another virtualization root mode. As described further below, the VMM214is responsible for managing hardware resources of the computing device100, including the TIO devices216. The VMM214is configured to launch the host partition202and/or the provisioning agent204, and to launch the TEEs208. The VMM214is configured to assign TIO devices216to particular TEEs208. The VMM214may be configured to provide configuration commands to the TIO devices216as described below. The VMM214may be configured to enforce account permissions assigned to the provisioning agent204. The VMM214may be configured to request the provisioning agent204to request that a TEE208releases an assigned TIO device216.

The host partition202may be embodied as a root partition, host operating system, service virtual machine, or other partition of the computing device100that is trusted by the VMM214to access hardware of the computing device100(e.g., the TIO devices216). As shown, the host partition202includes the provisioning agent204and one or more host device drivers206. The provisioning agent204may be embodied as a trusted execution environment or other trusted agent of the computing device100. The provisioning agent204executes as part of a user-mode service, daemon, or other process in a non-privileged execution mode (e.g., ring level 3 or other user mode). Illustratively, the provisioning agent204is a secure enclave established using Intel SGX technology. Each host device driver206may be embodied as an operating system driver, plugin, framework, or other system component that is configured to interact with and control the TIO devices216. Each host device driver206may execute in kernel mode, ring level 0, or in another privileged execution mode. The host device drivers206have a communication channel to the provisioning agent204that may be used to service certain requests that originate from the TEE208. For example, certain requests from paravirtualized guest device drivers212may be forwarded to the provisioning agent204for service (e.g., to provide attestation reports as described below). The provisioning agent204may also register with a plug-n-play (PnP) manager or other device hotplug manager of the host partition202to receive device change events, so the provisioning agent204can respond appropriately (e.g., by processing newly attached TIO devices216).

The provisioning agent204is configured to securely command a TIO device216to enter a trusted I/O mode. Securely commanding the TIO device216may include protecting a configuration command with a provisioning key and providing the protected configuration command to the VMM214. The VMM214provides the protected configuration command to the TIO device216as described above. The provisioning agent204is further configured to securely command the TIO device216to set a global lock on configuration registers and to receive in response configuration data indicative of the configuration registers of the TIO device216. The provisioning agent204is further configured to securely command the TIO device216to atomically release the global lock and set a fine-grained lock on one or more of the configuration registers. The provisioning agent204is further configured to provide the configuration data to a TEE208in response setting the fine-grained lock.

The provisioning agent204may be further configured to perform an attestation protocol with the TIO device216to generate a device attestation report, verify the device attestation report, and securely provision the I/O device with the provisioning key in response to verifying the device attestation report. The provisioning agent204may be further configured to sign the device attestation report with a private key to generate a signed device attestation report, to include the signed device attestation report and a trusted agent attestation report in an attestation report; and to provide the combined attestation report to the TEE208.

The provisioning agent204may be further configured to determine whether the TEE208has released the TIO device216and, in response, to securely command the TIO device216to release the fine-grained lock. The provisioning agent204may be further configured to determine whether any other trusted execution environment is using the TIO device216and securely command the TIO device216to exit the trusted I/O mode if no other TEE208is using the TIO device216. The provisioning agent204may be further configured to receive a request to reclaim the I/O device from the VMM214and, in response, to request the TEE208to release the TIO device216.

Each TIO device216may be embodied as an I/O device capable of secure configuration and trusted I/O operations as described further below. For example, the TIO devices216may include one or more data storage devices130, network interface controllers132, accelerators134, peripheral devices136, or other endpoint devices and/or one or more I/O switches138or other switches that connect endpoint devices to a root complex of the host. Each TIO device216is configured to receive a configuration command and to determine whether the configuration command is restricted to the provisioning agent204. Configuration commands restricted to the provisioning agent204may include commands to enter or exit the trusted I/O mode, or commands to lock or unlock a configuration register. Each TIO device216is further configured to determine, for each configuration command restricted to the provisioning agent204, whether the configuration command is received from the provisioning agent204and, if so, to process the configuration command and, if not, to reject the configuration command. The TIO device216may determine whether the configuration command is received from the provisioning agent204by determining whether the configuration command is protected by the provisioning key, as described above. Each TIO device216is further configured to determine, for each configuration command that is not restricted to the provisioning agent204, whether the TIO device216is in the trusted I/O mode. If not the TIO mode, the TIO device216is configured to process the configuration command. If in the TIO mode, the TIO device216is further configured to determine whether a configuration register associated with the configuration command is locked and, if so, whether the configuration command is received from the provisioning agent204. The TIO device216is configured to reject the configuration command if the configuration register is locked and the configuration command is not received from the provisioning agent204. The TIO device216is configured to process the configuration command if the configuration register is not locked, or if the configuration register is locked and the configuration command is received from the provisioning agent204.

As shown inFIG.2, the TIO devices216may be included in a device tree such as a PCI Express (PCIe) device tree. The device tree may also include one or more non-TIO devices218. Each non-TIO device218may be embodied as an I/O device (e.g., a data storage device130, network interface controller132, accelerator134, peripheral device136, and/or other endpoint, switch, or bridge device) that is not capable of secure configuration and trusted I/O operations.

Each of the TEEs208may be embodied as a trust domain, virtual machine, virtual network function (VNF), guest operating system, or other trusted environment of the computing device100. Each TEE208is isolated or otherwise protected from the VMM214, host partition202, and other TEEs208with hardware support of the computing device100. Thus, the VMM214is outside of the trusted computing base (TCB) of each TEE208. Illustratively, each TEE208is a trust domain (TD) that is protected with the MKTME126. As shown, each TEE208may include a guest application210and one or more guest device drivers212. The guest device drivers212may access a paravirtualized application programming interface (API) or other interface supplied by a host device driver206, the host partition202, and/or the VMM214. Each TEE208is configured to verify the attestation report received from the provisioning agent204and, in response, to request the provisioning agent204to prepare an assigned TIO device216for trusted I/O.

Referring now toFIG.3, interaction diagram300illustrates a method for secure device configuration and management that may be executed by the computing device100. It should be appreciated that, in some embodiments, the operations of the method shown in the interaction diagram300may be performed by one or more components of the environment200of the computing device100as shown inFIG.2. The interaction diagram300begins with interaction302, in which the VMM214launches the provisioning agent204. The VMM214loads the provisioning agent204with a particular account or other permissions and thus trusts the provisioning agent204. In interaction304, the provisioning agent204performs an attestation and provisioning process with a TIO device216. Potential embodiments of methods for attestation and provisioning that may be executed by the computing device100are illustrated inFIGS.4and6and described further below.

In interaction306, the VMM214launches the TEE208and assigns the TIO device216to the TEE208. In interaction308, the provisioning agent204and the TEE208perform an attestation and provisioning process. After attestation and provisioning, the provisioning agent204is trusted by the TEE208. Thus, the provisioning agent204is trusted by the both the VMM214and the TEE208, but the VMM214may remain outside of the trusted computing base (TCB) of the TEE208. Potential embodiments of methods for attestation and provisioning that may be executed by the computing device100are illustrated inFIGS.7and9and described further below.

In interaction310, the provisioning agent204locks the configuration of the TIO device216and reads configuration data from the TIO device216. In interaction312, the provisioning agent204sends an attestation/configuration report to the TEE208. Potential embodiments of methods for locking the device configuration are illustrated inFIGS.4and8and described further below. In interaction314, the TEE208verifies the configuration of the TIO device216. Potential embodiments of a method for verifying device configuration are illustrated inFIG.9and described further below. After verifying the device configuration, in interaction316the TEE208and the TIO device216perform trusted I/O.

In interactions318,320,322, the VMM214, the provisioning agent204, and the TEE208manage trusted I/O access with the TIO device216. In interaction318, the TEE208may release the TIO device to the VMM214, for example when finished with an I/O session or otherwise done accessing the TIO device216. In interaction320, the VMM214may request that the provisioning agent204cause the TEE208to release the TIO device216. Additionally or alternatively, in interaction322the TEE208may release the TIO device216to the provisioning agent204, for example when finished with an I/O session or otherwise done accessing the TIO device216. After the TIO device216is released, in interaction324the provisioning agent204unlocks or otherwise configures the TIO device216to allow use by other components of the computing device100. Thus, the VMM214may manage and otherwise maintain control over hardware components of the computing device100such as the TIO device216. Potential embodiments of methods for securely managing TIO devices216are illustrated inFIGS.5,8, and9and described further below.

Referring now toFIG.4, in use, the computing device100may execute a method400for secure configuration command processing. It should be appreciated that, in some embodiments, the operations of the method400may be performed by one or more components of the environment200of the computing device100as shown inFIG.2, such as by a TIO device216. For example, the method400may be executed by an I/O endpoint such as the accelerator134or a peripheral device136, or by an I/O switch138. The method400begins with block402, in which the TIO device216performs a device attestation protocol with the provisioning agent (PA)204. As part of the device attestation protocol, the TIO device216may provide an attestation report to the PA204that includes one or more verifiable assertions such as the identity, type, manufacturer, firmware version, or other attributes of the TIO device216. The TIO device216and the PA204may perform any attestation protocol, such as PCIe attestation.

In block404, the TIO device216performs an authenticated key exchange protocol with the PA204, such as an authenticated Diffie-Hellman key exchange. In block406, the TIO device216receives one or more encryption keys that may be used for protecting control messages. Illustratively, the TIO device216and the PA204exchange a shared secret key Ka. The TIO device216receives a provisioning key Kpfrom the PA204that is wrapped by the key Ka. As described further below, control messages may be protected by the provisioning key Kp. In block408, the TIO device216stores the encryption keys (e.g., Kaand Kp) in protected storage accessible to the TIO device216. For example, the keys may be stored in private memory of the TIO device216.

In block410, the TIO device216receives a configuration command. The configuration command may be embodied as an access to a configuration space, configuration register, or other configuration mechanism of the TIO device216. The configuration command may be issued from software executed by the processor120such as operating systems, device drivers, and/or application software. For example, as described further below, a valid configuration command may be generated by the PA204and issued to the TIO device216via the host device driver206, the VMM214, or other supervisory component of the computing device100.

In block412, the TIO device216determines whether the configuration command is restricted to being performed by the PA204. Certain sensitive configuration commands may only be originated from the PA204and not by other entities, such as the VMM214and/or host device driver206. Illustratively, commands to enter or exit trusted I/O (TIO) mode and commands to lock or unlock configuration registers are restricted to the PA204. In block414, the TIO device216checks whether the command is restricted to the PA204. If so, the method400branches ahead to block420, described below. If the command is not restricted to the PA204, the method400advances to block416.

In block416, the TIO device216determines whether the TIO device216is operating in the TIO mode. As described further below, the TIO device216may be commanded into and out of the TIO mode by the PA204. If the TIO device216is not operating in TIO mode, the method400branches to block422, described below. If the TIO device216is operating in the TIO mode, the method400advances to block418. In block418, the TIO device216determines whether the target register of the configuration command (e.g., an identified register in a configuration space of the TIO device216) is locked. As described further below, the TIO device216may be commanded to lock and unlock particular registers by the PA204. If the register is not locked, the method400branches to block422, described below. If the register is locked, the method400advances to block420.

In block420, the TIO device216determines whether the configuration command originated from the PA204. The TIO device216may, for example, determine whether the configuration command has been encrypted, signed, or otherwise protected with the provisioning key Kp. In some embodiments, the configuration command may be created and protected by the PA204and then submitted to the host device driver206and/or the VMM214for submission to the TIO device216. In those embodiments, verifying the command with the provisioning key Kpmay ensure that the configuration command originates from the PA204and has not been altered. If the configuration command originates from the PA204, the method400branches to block422, described below. If the configuration command does not originate from the PA204, the method400branches to block428, in which the TIO device216rejects the configuration command. The TIO device216may, for example, prevent changes to the configuration space of the TIO device216or otherwise reject the configuration command. After rejecting the command, the method400loops back to block410to receive additional configuration commands.

Referring back to blocks416,418,420, if the TIO device216is not in TIO mode, if the target register of the configuration command is not locked, or if the configuration command originates from the PA204, the method400branches to block422. In block422, the TIO device216processes or otherwise accepts the configuration command. The TIO device216may change the configuration space of the TIO device216or perform other actions in response to the configuration command. In some embodiments, in block424the TIO device216may enter or exit the TIO mode based on the configuration command. As described above, when operating in the TIO mode, the TIO device216may perform certain additional checks before processing configuration commands. In some embodiments, in block426, the TIO device216may lock or unlock a configuration register identified in the configuration command. As described above, when a register is locked, commands to change that register may be allowed only from the PA204. After processing the configuration command, the method400loops back to block410to receive additional configuration commands.

Referring now toFIG.5, in use, the computing device100may execute a method500for secure device management. It should be appreciated that, in some embodiments, the operations of the method500may be performed by one or more components of the environment200of the computing device100as shown inFIG.2, such as the VMM214. The method500begins with block502, in which the computing device100boots. The computing device100may boot in response to a platform power on, reboot, or other power cycle event. In response to the boot event, the computing device100loads the VMM214, host partition202, and/or other privileged components of the computing device100. The VMM214and/or host partition202may perform physical device discovery and enumeration (e.g., PCIe enumeration) to enumerate TIO devices216and load corresponding host device drivers206.

In block504, the computing device100executes a trusted provisioning agent (PA)204with assigned permissions. As described above, the PA204may be embodied as any trusted execution environment (TEE) or other trusted agent of the computing device100that is trusted by the VMM214. The provisioning agent204may be launched as part of a user-mode service, daemon, or other process in a non-privileged execution mode (e.g., ring level 3 or other user mode). Illustratively, the PA204is a secure enclave established using the secure enclave support122(e.g., with Intel SGX technology). Thus, the PA204protects its code and data from privileged adversaries, for example using the secure enclave support122. The illustrative PA204executes within a user process executed by the host partition202. The PA204is executed within a particular predetermined operating system account or with other predetermined permissions that are enforced by the VMM214. Thus, the PA204is trusted by the VMM214.

In block506the VMM214computing device100launches a trusted execution environment (TEE)208and assigns one or more TIO devices216to that TEE208. Illustratively, the TEE208is a trust domain enforced with the MKTME126. Accordingly, the contents of the TEE208are protected in hardware from the VMM214and from other TEEs208. The computing device100may use any technique to assign the TIO devices216to the TEE208. In some embodiments, in block508the VMM214may set up one or more guest device drivers212in the TEE208. The guest device drivers212may be embodied as paravirtualized drivers or other components that communicate with a corresponding host device driver206or other component of the host partition202and/or the VMM214.

In block510, the VMM214manages requests issued between the TEE208, the PA204, and the physical TIO device216. As described further below, the PA204and the TEE208may securely configure the TIO device216and then perform trusted I/O between the TEE208and the TIO device216. During those interactions, the VMM214manages direct access to hardware such as the TIO devices216. Commands between the TEE208and the PA204may be transferred or otherwise processed by components of the VMM214, the host partition202, and/or the host device driver206. To maintain control over the TIO device216, the VMM214may drop or otherwise deny commands that are submitted by the TEE208and/or the PA204. However, as described further below, the VMM214may not alter or, in some embodiments, read the contents of those commands.

In block512, the VMM214determines whether to reclaim a TIO device216from the TEE208. As described further below, while performing trusted I/O, the TIO device216does not accept any configuration changes that do not originate from the provisioning agent204. In order to regain complete control of the TIO device216, the VMM214must reclaim the TIO device216. A TIO device216may be reclaimed, for example, in order to allow the VMM214to assign the device to another TEE208. If the VMM214determines not to reclaim the TIO device216, the method500loops back to block506, in which the VMM214may launch additional TEEs208and manage additional TIO devices216. If the VMM214determines to reclaim the TIO device216, the method500advances to block514.

In block514, the VMM214requests the PA204to take the TIO device216out of TIO mode. As described further below, the PA204may request the TEE208to release the TIO device216or otherwise ensure that the TIO device216is no longer in use by any TEE208. After taking the TIO device216out of the TIO mode, the VMM214may have complete control over the configuration of the TIO device216. For example, the TIO device216may be assigned by the VMM214to a different TEE208. After taking the TIO device216out of the TIO mode, the method500loops back to block506, in which the VMM214may launch additional TEEs208and manage additional TIO devices216.

Referring now toFIG.6, in use, the computing device100may execute a method600for secure device enumeration and attestation. It should be appreciated that, in some embodiments, the operations of the method600may be performed by one or more components of the environment200of the computing device100as shown inFIG.2, such as the PA204. The method600begins with block602, in which the computing device100launches the PA204. The PA204may be launched by the VMM214as described above in connection with block504ofFIG.5.

In block604, the PA204enumerates a device tree for all I/O devices coupled to the computing device100. For example, the PA204may enumerate all PCI buses, switches, bridges, and/or endpoints connected to a root complex of the computing device100. The PA204may enumerate the I/O devices using a user mode API provided by the host partition202and/or the VMM214.

In block606, the PA204determines whether a current I/O device of the device tree is capable of trusted I/O (TIO). The PA204may use any technique to determine whether the device is TIO-capable (i.e., if the I/O device is a TIO device216and not a non-TIO device218). For example, the PA204may read one or more predetermined registers from a configuration space of the I/O device that indicate whether the I/O device is TIO-capable. The register may indicate that the TIO-capable device216provides a secure device configuration interface to host software, including that the TIO device216is capable of performing an authenticated key exchange and is capable of being provisioned with a key for receiving encrypted and integrity protected control messages from the PA204. In block608the PA204checks whether the current I/O device is TIO-capable. If not (e.g., if the current I/O device is a non-TIO device218), the method600branches ahead to block626, described below. If the current I/O device is TIO-capable, the method600advances to block610.

In block610, the PA204performs an attestation protocol with the current TIO device216. The PA204and the TIO device216may perform any appropriate attestation protocol, such as PCIe attestation. As part of the attestation protocol, the PA204receives an attestation report from the TIO device216that includes one or more verifiable assertions such as the type, manufacturer, firmware version, or other attributes of the TIO device216. In block612, the PA204verifies the device attestation report. The PA204may verify that the device attestation report is authentic, for example by verifying one or more digital signatures of the report. The PA204may also verify one or more attributes of the TIO device216included in the device attestation report. For example, the PA204may verify that the TIO device216has an allowed firmware version or other device attribute. In block614, the PA204saves the device attestation report. The device attestation report may be saved in an in-memory data structure that is protected from unauthorized disclosure or alteration, for example using the secure enclave support122of the processor120. In other embodiments, the device attestation report may be saved in secure storage accessible to the PA204.

In block616, the PA204determines whether the TIO device216was successfully verified. If not, the method600branches ahead to block626, described below. If the TIO device216was successfully verified, the method600advances to block618.

In block618, the PA204provisions one or more encryption keys to the TIO device216. As described above, those encryption keys may be used for protecting control messages issued by the PA204to the TIO device216and corresponding responses. The PA204may use any technique to securely provision the encryption keys. In some embodiments, in block620the PA204may perform an authenticated key exchange protocol with the TIO device216, such as an authenticated Diffie-Hellman key exchange. Illustratively, the TIO device216and the PA204exchange a shared secret key Ka. The PA204then wraps a provisioning key Kpwith the key Kaand provides the wrapped provisioning key Kpto the TIO device216. As described above, control messages and responses may be integrity-protected using the provisioning key Kp.

In some embodiments, in block622the PA204may securely provision one or more link encryption keys to the TIO device216. The link encryption keys may be used to protect I/O data transferred to and/or from the TIO device216from certain hardware attacks. The link encryption keys may be different from the provisioning key Kpused to protect configuration commands.

In block624, the PA204seals (e.g., encrypts) the provisioning keys (e.g., Kaand Kp) and securely stores the sealed keys. For example, the PA204may encrypt Kaand Kpwith an enclave sealing key provided by the secure enclave support122of the processor120. Sealing the keys Kaand Kpmay allow the PA204to recover the provisioned TIO devices216in response to unexpected crashes, resets, power cycles, or other events.

In block626, the PA204determines whether additional I/O devices remain for attestation. For example, the PA204may select additional I/O devices by traversing a device tree as described above. If additional I/O devices remain, the method600loops back to block606to continue processing I/O devices. If no additional I/O devices remain, the method600is completed. After completing the method600, the PA204may provide device attestation information to one or more TEEs208as described below in connection withFIG.7.

Referring now toFIG.7, in use, the computing device100may execute a method700for secure device and trusted agent attestation. It should be appreciated that, in some embodiments, the operations of the method700may be performed by one or more components of the environment200of the computing device100as shown inFIG.2, such as the PA204. The method700begins with block702, in which the PA204receives a request from a TEE208for an attestation report associated with a TIO device216. The PA204may receive the request from the TEE208via a host device driver206or other components of the computing device100. For example, the TEE208may generate the request for the attestation report by accessing a paravirtualized guest device driver212, which in turn accesses a corresponding host device driver206in the host partition202. The host device driver206may forward the request to the PA204for servicing.

In block704, the PA204looks up the device attestation report for the requested TIO device216. As described above in connection withFIG.6, the PA204has previously enumerated and performed an attestation protocol with all TIO devices216. Thus, the PA204maintains a data structure that includes device attestation reports for each TIO device216, as well as information on which devices have been attested and provisioned and how the devices are connected to each other (the topology).

In block706, the PA204signs the device attestation report with a private key associated with the PA204. In block708, the PA204adds an attestation report associated with the PA204to the device attestation report. The PA attestation report includes one or more verifiable assertions such as the identity, signature, vendor, authority, or other attributes of the PA204. In some embodiments, the PA attestation report may be generated using hardware support of the computing device100. For example, the PA attestation report may be generated using one or more processor instructions to perform an SGX local or remote attestation protocol. The PA attestation report may also include a public key and/or certificate of the PA204.

In block710, the PA204provides the combined, signed attestation report to the TEE208. The PA204may provide the signed attestation report to the TEE208via the host device driver206or other components of the computing device100. For example, the PA204may provide the signed report to the host device driver206, which in turn may provide the response to the paravirtualized guest device driver212of the TEE208. As described further below in connection withFIG.9, the TEE208(e.g., the paravirtualized guest device driver212, the guest application210, or other component of the TEE208) may use the signed attestation report to verify both the PA204and the TIO device216. After providing the attestation report, the method700loops back to block702, in which the PA204may receive additional requests for attestation reports.

Referring now toFIG.8, in use, the computing device100may execute a method800for secure device configuration. It should be appreciated that, in some embodiments, the operations of the method800may be performed by one or more components of the environment200of the computing device100as shown inFIG.2, such as the PA204. The method800begins with block802, in which the PA204receives a request from the TEE208to prepare a TIO device216for trusted I/O. The request may be received after the PA204has attested and verified both the TEE208and the TIO device216. The PA204may receive the request from the TEE208via a host device driver206or other components of the computing device100. For example, the TEE208may generate the request for trusted I/O by accessing a paravirtualized guest device driver212, which in turn accesses a corresponding host device driver206in the host partition202. The host device driver206may forward the request to the PA204for servicing.

In block804, the PA204commands the TIO device216into the TIO mode. As described above in connection withFIG.4, the PA204may send a configuration command to the TIO device216that includes the command to enter the TIO mode. The PA204indicates that the command originates with the PA204, for example by encrypting, signing, or otherwise protecting the configuration command with the provisioning key Kp. The TIO device216verifies that the configuration command originates with the PA204as described above.

The VMM214retains the ability to approve or disapprove all configuration commands sent by the PA204to the TIO device216. For example, the PA204may send configuration commands to the TIO device216via the host device driver206, which in turn may access the TIO device216using the host partition202and/or the VMM214. Thus, the host partition202and/or the VMM214may block any configuration commands to the TIO device216at the physical driver level, which allows the VMM214to remain in control of the physical resources of the computing device100. If any configuration command is blocked by the host partition202and/or the VMM214, TIO operations may be aborted.

In block806, the PA204commands the TIO device216to set a global lock on all configuration registers of the TIO device216. As described above, the TIO device216verifies that the global lock command originated with the PA204. After setting the global lock on the configuration registers, the TIO device216will reject any changes to those registers that do not originate from the PA204.

In block808, the PA204commands the TIO device216to return configuration data to the PA204. The configuration data may be embodied as or otherwise include binary data indicative of a configuration space of the TIO device216or other machine-readable data indicative of the configuration of the TIO device216. In block810, the PA204validates the device configuration data. The PA204determines whether the TIO device216is ready to perform secure TIO, for example by evaluating one or more configuration register settings. For example, the PA204may determine whether an endpoint device is ready to perform trusted I/O. As another example, the PA204may determine whether a PCIe switch is configured correctly to enforce Access Control Services (ACS), which may prevent Requester ID spoofing or prevent undesirable peer-to-peer transactions between devices.

In block812, the PA204determines whether the device configuration data was successfully validated. If not, the method800branches to block826, in which the PA204may command the TIO device216out of TIO mode, as described below. If the device configuration data was successfully validated by the PA204, the method800advances to block814.

In block814, the PA204commands the TIO device216to atomically release the global lock on the configuration registers and to set a fine-grained lock on one or more sensitive configuration registers. The fine-grained lock may be set on any registers that may impact trusted I/O if misconfigured. For example, the fine-grained lock may be set on one or more PCI base address registers (BARs) of the TIO device216. As another example, the fine-grained lock may prevent privileged software from changing ACS settings for PCIe switches. As described above, the TIO device216verifies that the atomic unlock/lock command originated with the PA204. Because the global lock is released and the fine-grained lock is set in one atomic operation, the device configuration of the TIO device216may not be changed before setting the fine-grained lock. After setting the fine-grained lock, the TIO device216will reject any changes to the locked registers that do not originate from the PA204. Other configuration registers may be changed during trusted I/O by other entities, such as the TEE208.

In block816, the PA204returns the device configuration data to the TEE208. The device configuration data may be returned to the TEE208via the host device driver206and the paravirtualized guest device driver212. As described further below, the TEE208may also validate the device configuration data. After validating the device configuration data, the TEE208may perform trusted I/O with the TIO device216. As described above in connection withFIG.4, the TIO device216may allow the TEE208to access certain unlocked configuration registers, and may prevent the TEE208from accessing locked configuration registers.

In block818, the PA204determines whether a request to release the TIO device216has been received from the VMM214. As described above, the VMM214manages hardware resources of the computing device100and may send a request to the PA204to release one or more TIO devices216. If a request to release the TIO device216is not received, the method800branches ahead to block822, described below. If a request to release the TIO device216is received, the method800advances to block820, in which the PA204sends a request to the TEE208to release the TIO device216.

In block822, the PA204determines whether the TEE208has released the TIO device216. The TEE208may release the TIO device216in response to closing a resource, terminating a process, exiting the TEE208, or otherwise stopping use of the TIO device216. The TEE208may release the TIO device216in response to a request from the PA204or independently of any request (e.g., in response to completing TIO processing). If the TEE208has not released the TIO device216, the method800loops back to block818to continue monitoring for release of the TIO device216. If the TEE208has released the TIO device216, the method800advances to block824.

In block824, the PA204commands the TIO device216to release the fine-grained lock on the configuration registers. As described above, the TIO device216verifies that the register unlock command originated with the PA204. In block826, the PA204determines whether any other TEE208is using the TIO device216. For example, the PA204may maintain a list or other data structure of TEEs208that have been set up for TIO with the TIO device216. If any other TEE208is still using the TIO device216, the method800loops back to block802to process additional requests to prepare the TIO device216for trusted I/O. If no other TEE208is using the TIO device216, the method800advances to block828, in which the PA204commands the TIO device216out of trusted I/O mode. As described above, the TIO device216verifies that the command to exit TIO mode originated with the PA204. After exiting TIO mode, the TIO device216may be configured or otherwise used by entities other than the PA204, for example for ordinary, untrusted I/O. After taking the TIO device216out of TIO mode, the method800loops back to block802to process additional requests to prepare the TIO device216for trusted I/O.

Referring now toFIG.9, in use, the computing device100may execute a method900for secure device configuration. It should be appreciated that, in some embodiments, the operations of the method900may be performed by one or more components of the environment200of the computing device100as shown inFIG.2, such as the TEE208. The method900begins with block902, in which the computing device100launches a trusted execution environment (TEE)208. For example, the TEE208may be launched by the VMM214as described above in connection withFIGS.3and5. The TEE208may be embodied as a trust domain (TD) protected with the MKTME126, a virtual machine, or other trusted execution environment protected using hardware support of the computing device100. The TEE208may be protected from unauthorized disclosure and/or tampering by the VMM214, the host partition202, and/or other TEEs208.

In block904, the TEE208requests attestation of a TIO device216from the provisioning agent (PA)204. As described above, the TEE208may be assigned one or more TIO devices216by the VMM214. Those TIO devices216may be exposed to the TEE208as one or more virtual devices that are accessible using one or more paravirtualized guest device drivers212. As described above, the TEE208may submit the request for attestation via the paravirtualized guest device driver212, which submits the request to a host device driver206. The host device driver206forwards the request to the PA204for servicing.

In response to the request, in block906, the TEE208receives an attestation report from the PA204. As described above, the attestation report may be received from the PA204via the host device driver206and the guest device driver212. As also described above, the attestation report includes device attestation report information associated with the TIO device216as well as PA attestation report information associated with the PA204. The attestation report may be signed with the private key of the PA204and may include a public key and/or certificate of the PA204.

In block908, the TEE208verifies the attestation report. The TEE208may use any appropriate technique to verify the integrity and the contents of the attestation report. In block910, the TEE208verifies the PA attestation report information included in the attestation report. In some embodiments, the PA attestation report may be verified using hardware support of the computing device100, for example, by executing one or more processor instructions to perform an SGX local or remote attestation protocol. In block912, the TEE208verifies the device attestation report information included in the attestation report. The TEE208may verify the integrity the device attestation report using the public key of the PA204. The TEE208may also verify the information included in the device attestation report (e.g., the type, manufacturer, firmware version, or other attributes of the TIO device216).

In block914, the TEE208determines whether the attestation report was successfully verified. If not, the method900branches to block924, in which the TEE208raises a verification error. The TEE208does not perform any trusted I/O with the TIO device216, which may protect against a malicious TIO device216. Referring back to block914, if the attestation report is successfully verified, the method900advances to block916.

In block916, the TEE208requests that the PA204prepares the TIO device216for trusted I/O. In block918, the TEE208receives device configuration information from the PA204. As described above, the device configuration information may be embodied as or otherwise include binary data indicative of a configuration space of the TIO device216or other data indicative of the configuration of the TIO device216.

In block920, the TEE208verifies the device configuration information and associated host configuration data. For example, the device configuration information may include one or more PCI base address registers (BARs) associated with the TIO device216. The TEE208may verify that MMIO registers mapped by the VMM214into the guest address space of the TEE208match the device configuration information.

In block922, the TEE208checks whether the device configuration information and associated host configuration data was successfully verified. If not, the method900branches to block924, in which the TEE208raises a verification error. Referring back to block922, if the device configuration information and associated host configuration data is successfully verified, the method900advances to block926.

In block926, the TEE208performs trusted I/O operations with the TIO device216. The TEE208(e.g., the guest device driver212) may program one or more direct memory access (DMA) registers of the TIO device216and cause the TIO device216to perform TIO operations. Data transferred by the TIO operations may be protected from software and/or hardware attacks. For example, the TEE208and the TIO device216may encrypt or otherwise protected transferred I/O data with one or more link keys or other encryption keys.

In block928, the TEE208determines whether trusted I/O with the TIO device216is complete. TIO operations may be completed, for example, in response to the TEE208receiving a request from the PA204to release the TIO device216. As another example, the TIO operations may be completed based on actions or parameters of a guest application210, in response to user input, or otherwise be completed independently of the PA204. If the TIO operations are not completed, the method900loops back to block926to continue performing TIO with the TIO device216. If the TIO operations are done, the method900advances to block930.

In block930, the TEE208releases the TIO device216to the PA204. As described above in connection withFIG.8, the PA204may command the TIO device216to unlock the device configuration and, if no other TEEs208are using the TIO device216, may cause the TIO device216to leave the TIO mode. After releasing the TIO device216, the method900loops back to block904, in which the TEE208may validate and perform TIO with additional TIO devices216.

Referring now toFIG.10, in an illustrative embodiment, the computing device100establishes an environment1000during operation. The illustrative environment1000includes a host partition1002, a trusted agent1008, multiple trust domains (TDs)1014, a virtual machine monitor (VMM)1020, a secure arbitration mode (SEAM)1022, and trusted I/O (TIO) devices1024. The various components of the environment1000may be embodied as hardware, firmware, software, or a combination thereof. As such, in some embodiments, one or more of the components of the environment1000may be embodied as circuitry or collection of electrical devices (e.g., host partition circuitry1002, trusted agent circuitry1008, TEE circuitry1014, VMM circuitry1020, SEAM circuitry1022, and/or TIO device circuitry1024). It should be appreciated that, in such embodiments, one or more of the host partition circuitry1002, the trusted agent circuitry1008, the TEE circuitry1014, the VMM circuitry1020, the SEAM circuitry1022, and/or the TIO device circuitry1024may form a portion of the processor120, the I/O subsystem124, the accelerator134, the peripheral devices136, and/or other components of the computing device100. Additionally, in some embodiments, one or more of the illustrative components may form a portion of another component and/or one or more of the illustrative components may be independent of one another.

The VMM1020may be embodied as any virtual machine monitor, hypervisor, or other component that manages execution of one or more virtualized workloads on the computing device100. The VMM1020may manage workloads using hardware support of the processor120, for example by executing in a privileged state such as VMX root, ring level −1, or another virtualization root mode. As described further below, the VMM1020is responsible for managing hardware resources of the computing device100, including the TIO devices1024. The VMM1020may be configured to launch the host partition1002, the trusted agent1008, and/or the TDs1014.

The SEAM1022is a trusted firmware component of the computing device100. The SEAM1022executes in a privileged state similar to VMX root, ring level −1, or another virtualization root mode. Thus, the SEAM1022is a peer entity to the VMM1022. The SEAM1022establishes defined interfaces to trust domains (e.g., TDs1014) and the VMM1020. The SEAM1022may be embodied as a firmware library that is loaded by a SEAM loader and measured by hardware of the computing device100(e.g., by the processor120). The measurement of the SEAM1022may be provided to any requesting trust domain1014or other TEE, which may use the measurement to verify the authenticity of the SEAM1022. The SEAM1022provides trusted services to trust domains1014. For example, the SEAM1022may configure page tables for each trust domain1014, which may allow the VMM1022to be removed from the trusted computing base (TCB) of each trust domain1014.

The host partition1002may be embodied as a root partition, service virtual machine, or other partition of the computing device100that is trusted by the VMM1020to access hardware of the computing device100(e.g., the TIO devices1024). As shown, the host partition1002includes a host operating system1006and one or more host device drivers1004. As described further below, the host operating system1006and/or the host device drivers1004may directly control or otherwise interact with the TIO devices1024and/or non-TIO devices1026. The host operating system1006is configured to determine wherein an I/O device is capable of trusted I/O (e.g., is a TIO device1024) and, if so, send a request to authenticate the TIO device1024to the trusted agent1008. The host operating system1006may be further configured to program the configuration of the TIO device1024. The host operating system1008is further configured to command the TIO device1024to lock configuration of the TIO device1024, receive a device configuration report from the TIO device1024, and provide the device configuration report to the trusted agent1008. The device configuration report may be indicative of configuration registers of the TIO device1024. In some embodiments, those functions may be performed by one or more other components, such as the host device drivers1004and/or the VMM1020.

The trusted agent1008may be embodied as a trust domain or other trusted execution environment of the computing device100. The trusted agent1008may execute in a guest mode (e.g., VMX non-root) or other non-root virtualization mode. Illustratively, the trusted agent1008executes as or is included in a trust domain that is protected with the MKTME126. The trusted agent1008may be embodied as an architecturally defined trust domain that may be recognized by other entities of the computing device100, including the VMM1022and/or the SEAM1024. The trusted agent1008is configured to provide a trusted agent attestation report to the SEAM1024and to perform an authenticated key exchange protocol with a TD1014in response to providing the trusted agent attestation report. The trusted agent1008is further configured to receive the request to authenticate the TIO device1024from the host operating system1006, authenticate the TIO device1024, and perform an authenticated key exchange protocol with the TIO device1024in response to authenticating the TIO device1024. Performing the authenticated key exchange protocol may include provisioning a shared secret key to the TIO device1024. The trusted agent1006is further configured to provision a link encryption key to the TIO device1024in response performing the authenticated key exchange protocol with the TIO device1024. The trusted agent1008is further configured to receive the device configuration report from the host operating system1006and verify the device configuration report. The trusted agent1008is further configured to determine whether the TIO device1024is already assigned to a TD1014and, if not, record a binding between the TIO device1024and the TD1014. The trusted agent1008is further configured to send the device configuration report to the TD1014in response to verifying the device configuration report and recording the binding. The trusted agent1008is further configured to remove the binding between the TIO device1024and the TD1014in response to the TIO device1024being released by the TD1014or receiving a request to reclaim the TIO device1024from the host operating system1006. In some embodiments, those functions may be performed by one or more sub-components, such as an authenticator1010and/or a configuration verifier1012.

Each TIO device1024may be embodied as an I/O device capable of secure configuration and trusted I/O operations as described further below. For example, the TIO devices1024may include one or more data storage devices130, network interface controllers132, accelerators134, peripheral devices136, or other endpoint devices and/or one or more I/O switches138or other switches that connect endpoint devices to a root complex of the host. Each TIO device1024may be configured to perform trusted I/O with a TD1014. Performing trusted I/O may include securely transferring I/O data using a link encryption key (e.g., encrypting or decrypting the I/O data using the link encryption key).

As shown inFIG.10, the TIO devices1024may be included in a device tree such as a PCI Express (PCIe) device tree. The device tree may also include one or more non-TIO devices1026. Each non-TIO device1026may be embodied as an I/O device (e.g., a data storage device130, network interface controller132, accelerator134, peripheral device136, and/or other endpoint, switch, or bridge device) that is not capable of secure configuration and trusted I/O operations.

Each of the TDs1014may be embodied as a trust domain, virtual machine, virtual network function (VNF), guest operating system, or other trusted environment of the computing device100. Each TD1014is isolated or otherwise protected from the VMM1020, host partition1002, and other TDs1014with hardware support of the computing device100. Thus, the VMM1020is outside of the trusted computing base (TCB) of each TD1014. Illustratively, each TD1014is a trust domain (TD) that is protected with the MKTME126. As shown, each TD1014may include a guest application1016and one or more guest device drivers1018. The guest device drivers1018may access a paravirtualized application programming interface (API) or other interface supplied by a host device driver1004, the host operating system1006, and/or the VMM1020. Each TD1014is configured to receive the trusted agent attestation report from the SEAM1022and verify the trusted agent attestation report. Each TD1014may be further configured to verify the device configuration report received from the trusted agent1014and to perform trusted I/O with the TIO device1024in response to verifying the device configuration report. Verifying the device configuration report may include verifying a device feature of the TIO device1024, verifying a register address indicated by the device configuration report, and/or verifying a link encryption status indicated by the device configuration report.

Referring now toFIG.11, in use, the computing device100may execute a method1100for secure device management. It should be appreciated that, in some embodiments, the operations of the method1100may be performed by one or more components of the environment1000of the computing device100as shown inFIG.10, such as the VMM1020. The method1100begins with block1102, in which the computing device100boots. The computing device100may boot in response to a platform power on, reboot, or other power cycle event. In response to the boot event, the computing device100loads the VMM1020, the SEAM1022, the host partition1002, the trusted agent1008, and/or other privileged or architectural components of the computing device100.

In block1104, the computing device100enumerates an I/O device and loads any associated device drivers. For example, the VMM1020and/or host operating system1006may perform physical device discovery and enumeration (e.g., PCIe enumeration) at boot time to enumerate I/O devices and load corresponding host device drivers1004. The computing device100may also enumerate I/O devices as they are attached, hot-plugged, or otherwise added to the computing device100. The I/O devices may include one or more data storage devices130, network interface controllers132, accelerators134, peripheral devices136, or other endpoint devices and/or may include one or more I/O switches138or other switches that connect endpoint devices to a root complex of the host.

In block1106, the computing device100determines whether the I/O device is capable of trusted I/O (TIO). The computing device100may use any technique to determine whether the device is TIO-capable (i.e., if the I/O device is a TIO device1024and not a non-TIO device1026). For example, the VMM1020, the host operating system1006, and/or the host device driver1004may read one or more predetermined registers from a configuration space of the I/O device that indicate whether the I/O device is TIO-capable. The register may indicate that the TIO-capable device1024provides a secure device configuration interface to host software, including that the TIO device1024is capable of authentication and secure communication with the trusted agent1008, is capable of locking configuration of one or more partitions of the TIO device1024, and is capable of securely and accurately reporting its configuration and state to the trusted agent1008. In block1108the PA204checks whether the current I/O device is TIO-capable. If not (e.g., if the current I/O device is a non-TIO device1026), the method100branches ahead to block1112, described below. If the current I/O device is TIO-capable, the method1100advances to block1110.

In block1110, the computing device100requests that the trusted agent1008authenticate and provision the current TIO device1024. For example, the VMM1020, the host operating system1006, and/or the host device driver1004may send a request to the trusted agent1008that identifies the current TIO device1024. The request may include a device certificate or other identification data that may be used by the trusted agent1008to authenticate and provision the TIO device1024. One potential embodiment of a method for authenticating and provisioning the TIO device1024is described below in connection withFIG.12.

In block1112, the computing device100determines whether additional I/O devices remain for authentication and provisioning. If so, the method1100loops back to block1104to continue enumerating I/O devices. After all I/O devices have been enumerated, the method1100advances to block1114.

In block1114, the computing device100launches a trust domain (TD)1014. As described above, each TD1014is a trusted execution environment (TEE) that is protected with the MKTME126support. Accordingly, the contents in memory of the TD1014are protected in hardware from the VMM1020and from other TDs2014.

In block1116, the computing device100assigns a TIO device1024to the TD1014. The TIO device1024may be assigned, for example, in response to the TD1014requesting access to the TIO device1024. The computing device100may use any technique to assign the TIO device1024to the TD1014. In some embodiments, the VMM1020and/or the host operating system1006may configure the TD1014to allow the TD1014to access a configuration space, MMIO registers, or other memory addresses that can control the TIO device1024. The TIO device1024may be securely bound to the TD1014using the SEAM1022and/or the VMM1020.

In block1118, the computing device100programs the device configuration of the TIO device1024. The VMM1020, the host operating system1006, and/or the host driver1004may set registers, read and/or write memory addresses, or otherwise configure the TIO device1024.

In block1120, the computing device100commands the TIO device1024(or a partition of the TIO device1024such as a virtual function) to lock its configuration and return a device configuration report. The command to lock configuration may be sent by the VMM1020, the host operating system1006, and/or a host driver1004. After being locked, the TIO device1024only allows trusted controlling entities (e.g., the TD1014) to read or modify certain registers that control device behavior. The command to lock the TIO device1024may include a nonce used for replay detection as described further below. The device configuration report returned from the TIO device1024includes data that represents the identity and/or current configuration of the TIO device1024. For example, the device configuration report may be embodied as or otherwise include binary data indicative of a configuration space of the TIO device1024or other machine-readable data indicative of the configuration of the TIO device1024.

In block1122, the computing device100requests the trusted agent1008to verify the device configuration report. The request may be sent by the VMM1020, the host operating system1006, and/or a host driver1004. The trusted agent1008may verify the device configuration report, for example, by verifying the integrity and authenticity of the report, verifying that the TIO device1024has not been previously assigned to another TD1014, and/or performing other verification operations. If the device configuration report is verified by the trusted agent1008, the device configuration report may be accepted by the TD1014, and then the TD1014may use the TIO device1024for trusted I/O. Potential embodiments of methods for verifying and accepting the device configuration report are described below in connection withFIGS.13-14.

In block1124, in some embodiments, the computing device100may reclaim the TIO device1024. The TIO device1024may be reclaimed by the VMM1020, the host operating system1006, and/or the host driver1004. The TIO device1024may be reclaimed if the TD1014closes or otherwise releases the TIO device1024or may be reclaimed preemptively (e.g., if the TD1014is terminated or crashes, if the TIO device1024resets or is disconnected, in response to hotplug events, or in other circumstances). The method1100loops back to block1114to continue launching TDs1014and assigning TIO devices1024.

Referring now toFIG.12, in use, the computing device100may execute a method1200for secure device authentication and provisioning. It should be appreciated that, in some embodiments, the operations of the method1200may be performed by one or more components of the environment1000of the computing device100as shown inFIG.10, such as the trusted agent1008. The method1200begins with block1202, in which the computing device100launches the trusted agent1008. As described above, the trusted agent1008is an architectural trust domain that may be launched by the VMM1020.

In block1204, the SEAM1022measures the trusted agent1008as the trusted agent1008is launched. The SEAM1022stores the measurement for later use. The measurement may be indicative of the contents and identity of the trusted agent1008. As described further below, the SEAM1022may provide an attestation report or other data including the measurement to the TDs1014, and the TDs1014may use the attestation report to authenticate the trusted agent1008.

In block1206, the trusted agent1008receives a request to authenticate a TIO device1024from the host operating system1006. The request may be sent during device enumeration during initial boot and/or in response to a device being added to the computing device (e.g., a hotplug event). As described above, the request may include a device certificate associated with the TIO device1024.

In block1208, the trusted agent1008authenticates the TIO device1024using the provided device certificate. The trusted agent1008and the TIO device1024may perform any appropriate device authentication protocol, such as a protocol as described by a PCI Express device authentication specification. For example, the trusted agent1008and the TIO device1024may perform a challenge-response protocol in which the trusted agent1008sends a random nonce to the TIO device1024, the TIO device1024signs the nonce with a private key and returns the result, and the trusted agent1008verifies the response using the certificate (and corresponding public key). In block1210, the trusted agent1008determines whether the TIO device1024was successfully authenticated. If not, the method1200branches to block1214, described below. If the TIO device1024was successfully authenticated, the method1200advances to block1212.

In block1212, the trusted agent1008executes an authenticated key agreement protocol with the TIO device1024. For example, the trusted agent1008and the TIO device1024may perform an authenticated Diffie-Hellman key exchange. After performing the key exchange, both the TIO device1024and the trusted agent1008possess a shared secret key (e.g., a session key or other symmetric encryption key).

In block1214, the trusted agent1008determines whether additional TIO devices1024remain to be authenticated. If so, the method1200loops back to block1206to continue authenticating TIO devices1024. If no additional TIO devices1024remain to be authenticated, the method1200advances to block1216.

In block1216, the trusted agent1008provisions one or more link encryption keys to the trusted TIO devices1024. The trusted agent1008may provision the link encryption keys securely, for example, by wrapping each link encryption key with the secret key shared with the TIO device1024. The TIO device1024may use the link encryption key to protect I/O data transferred between the TIO device1024and the host over one or more I/O links (e.g., PCIe links). In some embodiments, in block1218the trusted agent1008may provision the TIO devices1024for end-to-end encryption. In such embodiments, the trusted agent1008may provision matching link encryption keys to an endpoint TIO device1024and the root port of the computing device100. In those embodiments, the I/O data is encrypted and/or decrypted at each end of the I/O link (i.e., the root port and the TIO device1024). The encrypted I/O data may be forwarded by intermediate I/O devices between the root port and the TIO device1024without encryption or decryption. Thus, end-to-end encryption may be used securely even in the presence of untrusted or non-TIO-capable I/O switches138or other intermediate I/O devices.

In some embodiments, in block1220the trusted agent1008may provision the TIO devices1024for point-to-point encryption. In such embodiments, matching link encryption keys are provisioned to the end nodes of each intermediate I/O link between the root port and the TIO device1024. In those embodiments, each intermediate I/O device (e.g., I/O switches138or other devices) must be TIO-capable and provisioned with link encryption keys accordingly. Accordingly, the trusted agent1008may assemble a graph or other representation of the topology of the I/O devices in order to determine whether any devices in the I/O path are untrusted or not capable of link encryption. The trusted agent1008may not enable link encryption for a TIO device1024that is connected to the root port via one or more untrusted or non-TIO-capable intermediate devices. After provisioning the link encryption keys, the method1200is completed. The trusted agent1008may continue to verify TIO device1024configuration as described below in connection withFIG.13.

Referring now toFIG.13, in use, the computing device100may execute a method1300for secure device configuration. It should be appreciated that, in some embodiments, the operations of the method1300may be performed by one or more components of the environment1000of the computing device100as shown inFIG.10, such as the trusted agent1008. The method1300begins with block1302, in which the trusted agent1008establishes a secure channel with a TD1014. As part of establishing the secure channel, the TD1014may authenticate the trusted agent1008as described further below in connection withFIG.14. In block1304, the trusted agent1008and the TD1014perform an authenticated key agreement protocol, such as an authenticated Diffie-Hellman key exchange. After performing the exchange, both the trusted agent1008and the TD1014possess a shared secret key. The shared key may be used to protect a secure communication channel between the trusted agent1008and the TD1014. The trusted agent1008may similarly establish secure channels with each TD1014executed by the computing device100.

In block1306, the trusted agent1008receives a device configuration report from the host operating system1006. As described above, the device configuration report includes data that represents the identity and/or current configuration of the TIO device1024. For example, the device configuration report may be embodied as or include binary data indicative of a configuration space of the TIO device1024, register values, or other machine-readable data indicative of the configuration of the TIO device1024. The device configuration report may include any information indicative of the current configuration and/or identity of the TIO device1024, such as device identity, link security status, assigned memory ranges, assigned interfaces, and/or other configuration information. The device configuration report may also include authentication data such as a nonce (received with the device lock and report command) and a message authentication code (MAC) or other integrity tag computed over the device configuration report using the session key shared by the trusted agent1008and the TIO device1024.

In block1308, the trusted agent1008verifies the device configuration report. In block1310, the trusted agent1008checks the integrity of the device configuration report using the MAC. In block1312, the trusted agent1008checks against potential replay attacks by comparing the nonce included in the report with the nonce previously sent by the trusted agent1008to the TIO device1024. In block1314, the trusted agent1008checks whether the device configuration report was successfully verified. If not, the method1300loops back to block1306, in which the trusted agent1008may verify additional TIO devices1024. If the device configuration report is successfully verified, the method1300advances to block1316.

In block1316, the trusted agent1008checks a database of bindings to determine if the TIO device1024is currently assigned to a different TD1014. For example, the trusted agent1008may maintain a Requester ID table or other data structure with bindings between TIO devices1024and TDs1014. In block1318, the trusted agent1008determines whether the TIO device1024is currently unassigned. If not (i.e., if the TIO device1024is already assigned to a TD1014), the method1300loops back to block1306, in which the trusted agent1008may verify additional TIO devices1024. If the TIO device1024is unassigned, the method1300advances to block1320.

In block1320, the trusted agent1008records a binding between the TIO device1024and the TD1014. The trusted agent1008may, for example, add or update an entry in the Requester ID table. In block1322, the trusted agent1008sends the device configuration report over the secure channel to the TD1014. In some embodiments, the trusted agent1008may send a modified device configuration report to the TD1014. For example, the modified report may mask details that should not be exposed to a guest in a virtualized environment (e.g., host physical addresses of MMIO registers). The trusted agent1008may, for example, replace the host physical addresses of MMIO registers in the report with a cryptographic hash of the host physical addresses. As described further below in connection withFIG.14, the TD1014may evaluate the device configuration report and determine whether to accept the TIO device1024. If accepted, the TD1014performs trusted I/O with the TIO device1024.

In block1324, the trusted agent1008waits for the TD1014to release the TIO device1024or for the VMM1022to request the TIO device1024be reclaimed. The TD1014may close or otherwise release the TIO device1024when the TD1014exits or otherwise stops performing trusted I/O. The VMM1022may reclaim the TIO device1024, for example, if the TD1014is terminated or crashes, if the TIO device1024resets or is removed, or in other circumstances. In block1326, the trusted agent1008checks whether the TIO device1024has been released. If not, the method1300loops back to block1324. If the TIO device1024has been released, the method1300advances to block1328.

In block1328, the trusted agent1008removes the binding between the TD1014and the TIO device1024. The trusted agent1008may, for example, remove or modify an appropriate entry from the Requester ID table. As described above, removing the binding may allow the TIO device1024to be reassigned to a TD1014in the future. In some embodiments, the binding may be removed when the TD1014is still active, for example if the TIO device1024is removed from the computing device100(e.g., hot-unplugged) or if the TIO device1024is reset. In some embodiments, in block1330, the trusted agent1008may verify with the SEAM1022that the TD1014has been terminated. Verifying that the TD1014has actually been terminated with a trusted component such as the SEAM1022may protect from a malicious VMM1020or other untrusted components. For example, a malicious VMM1020may attempt to reclaim a device1024that is still in use by an active TD1014. If the SEAM1022indicates that the TD1014is still active, then the trusted agent1008does not remove the binding, which may prevent the TIO device1024from being reassigned (potentially maliciously) to another TD1014. After removing the binding, the method1300loops back to block1306, in which the trusted agent1008may verify additional TIO devices1024.

Referring now toFIG.14, in use, the computing device100may execute a method1400for secure device configuration. It should be appreciated that, in some embodiments, the operations of the method1400may be performed by one or more components of the environment1000of the computing device100as shown inFIG.10, such as the TD1014. The method1400begins with block1402, in which the computing device100launches a trust domain (TD)1014. For example, the TD1014may be launched by the VMM1020as described above in connection withFIG.11. The TD1014may include a virtual machine, guest operating system, or other trusted execution environment protected using hardware support of the computing device100. The TD1014may be protected from unauthorized disclosure and/or tampering by the VMM1022, the host partition1002, and/or other TDs1014.

In block1404, the TD1014requests a SEAM attestation report from the SEAM1022. As described above, the SEAM attestation report includes a measurement of the SEAM1022generated by hardware of the computing device100. In block1406, the TD1014requests a trusted agent attestation report from the SEAM1022. The trusted agent attestation report is indicative of the contents and identity of the trusted agent1008, and as described above, the trusted agent1008supplies the attestation report to the SEAM1022. In block1408, the TD1014verifies the SEAM attestation report and the trusted agent attestation report. The TD1014may, for example, verify the authenticity and integrity of the reports. The TD1014may also verify one or more identities, versions, or other attributes included in the reports. In block1410, the TD1014determines whether the SEAM attestation report and the trusted agent attestation report were both successfully verified. If not, the method1400branches to block1412, in which the TD1014raises a verification error. If the SEAM attestation report and the trusted agent attestation report are both verified, then the SEAM1022and the trusted agent1008are trusted, and the method1400advances to block1414.

In block1414, the TD1014establishes a secure channel with the trusted agent1008using an authenticated key agreement protocol, such as an authenticated Diffie-Hellman key exchange. As described above, after performing the exchange, both the trusted agent1008and the TD1014possess a shared secret key. The shared key may be used to protect a secure communication channel between the trusted agent1008and the TD1014.

In block1416, the TD1014requests use of a TIO device1024. The TD1014may submit the request, for example, to the host device driver1004, the host operating system1006, and/or the VMM1020. The VMM1020assigns the TIO device1024to the TD1014, configures the TIO device1024, and locks the configuration of the TIO device1024as described above in connection withFIG.11. The trusted agent1008verifies a device configuration report received from the TIO device1024as described above in connection withFIG.13. In block1418, the TD1014receives the device configuration report from the trusted agent1008. The device configuration report may be received via the secure channel established between the TD1014and the trusted agent1008.

In block1420, the TD1014verifies the device configuration of the TIO device1024as indicated in the device configuration report. As described above, the trusted agent1008verifies the integrity of the device configuration report and prevents replay attacks. Thus, the TD1014may perform policy-based evaluations or other evaluations of the device configuration report to determine whether to proceed with trusted I/O using the TIO device1024. In some embodiments, in block1422, the TD1014may verify a device ID, firmware ID, and/or version (e.g., firmware version, device revision, etc.) indicated in the device configuration report. The TD1014may, for example, allow certain devices or certain firmware versions, deny certain devices or certain firmware versions, require updated firmware, or otherwise verify the device ID, firmware ID, and/or version. In some embodiments, in block1424the TD1014may verify device register or memory configuration indicated in the device configuration report. In some embodiments, in block1426the TD1014may verify link encryption status indicated in the device configuration report. For example, the TD1014may determine whether the link traffic is encrypted.

In block1428, the TD1014determines whether the device configuration report was successfully verified. If not, the method1400loops back to block1416in which the TD1014may request use of another TIO device1024. If the device configuration report was successfully verified, the method1400advances to block1430.

In block1430, the TD1014performs trusted I/O with the TIO device1024. The TD1014may perform one or more MMIO transactions, DMA transactions, or other I/O operations with the TIO device1024. The I/O data may be protected in transit from the TIO device1024to the root port of the computing device100using one or more link encryption keys as described above. The I/O data may be transferred between the root port and memory of the TD1014and thus may be protected from the VMM1020, the host operating system1006, and other TDs1014. In block1432, the TD1014determines whether the TD1014is done performing trusted I/O, whether the TIO device1024has reset or disconnected, or if the trusted I/O session is otherwise terminated. If not, the method1400loops back to block1430to continue performing trusted I/O. If the trusted I/O session is terminated, the method1400advances to block1434.

In block1434, the TD1014notifies the host operating system1006and the trusted agent1008that the TD1014is releasing the TIO device1024. As described above, the trusted agent1008may also remove a binding between the TD1014and the TIO device1024, allowing the TIO device1024to be reassigned. After notification, the method1400loops back to block1416, in which the TD1014may request use of another TIO device1024.

Referring now toFIG.15, in use, the computing device100may execute a method1500for trusted I/O. It should be appreciated that, in some embodiments, the operations of the method1500may be performed by one or more components of the environment1000of the computing device100as shown inFIG.10, such as by a TIO device1024. For example, the method1500may be executed by an I/O endpoint such as the accelerator134or a peripheral device136, or by an I/O switch138. The method1500begins with block1502, in which the TIO device1024is reset. The TIO device1024may reset, for example, in response to a power cycle event, when connected or disconnected from the computing device100, or at other times. During reset, the TIO device1024clears any stored session keys, link encryption keys, or other volatile state.

In block1504, the TIO device1024provides attestation information to the trusted agent1508. The attestation information be provided as part of a device authentication protocol. The trusted agent1008and the TIO device1024may perform any appropriate device authentication protocol, such as a protocol as described by a PCI Express device authentication specification. For example, the trusted agent1008and the TIO device1024may perform a challenge-response protocol in which the trusted agent1008sends a random nonce to the TIO device1024, the TIO device1024signs the nonce with a private key and returns the result, and the trusted agent1008verifies the response using the certificate (and corresponding public key).

In block1506, the TIO device1024performs a secure key exchange with the trusted agent1008. The trusted agent1008and the TIO device1024may perform an authenticated key agreement protocol, such as an authenticated Diffie-Hellman key exchange. After performing the exchange, both the TIO device1024and the trusted agent1008possess a shared secret key (e.g., a session key or other symmetric encryption key).

In block1508, the TIO device1024receives a link encryption key from the trusted agent1008. The link encryption key may be wrapped with the session key exchanged with the trusted agent1008or otherwise securely provisioned to the TIO device1024. As described above, the TIO device1024may use the link encryption key to protect I/O data communicated over an I/O link with the root port of the computing device100(or an intermediate device such as an I/O switch138).

In block1510, the TIO device1024locks device configuration for the TIO device1024(or a partition of the TIO device1024) and returns a device configuration report. The TIO device1024may receive a command to lock the device configuration from the VMM1020, the host operating system1006, and/or a host driver1004. After being locked, the TIO device1024only allows trusted controlling entities (e.g., the TD1014) to read or modify certain registers that control device behavior. As described above, the device configuration report includes data that represents the identity and/or current configuration of the TIO device1024. In some embodiments, in block1512the TIO device1024may sign the device configuration report with the session key exchanged with the trusted agent1008or other key provisioned by the trusted agent1008. The TIO device1024may, for example, generate a message authentication code (MAC) over the device configuration report using the session key.

After locking the device configuration, in block1514the TIO device1024rejects configuration changes from non-trusted entities. For example, the TIO device1024may only permit trusted controlling entities (e.g., an assigned TD1014) to read or modify certain registers that control device behavior. In block1516, the TIO device1024performs trusted I/O with a TD1014. As describe above, the TIO device1024may perform DMA and/or MMIO transactions with the TD1014. In some embodiments, in block1518the TIO device1024may encrypt or decrypt I/O data using the link encryption key provisioned by the trusted agent1008. After performing trusted I/O, the method1500loops back to block1514to continue performing trusted I/O with locked device configuration. The TIO device1024may be reset or the method1500may be otherwise restarted in response to the TD1014releasing the TIO device1024, the VMM1020reclaiming the TIO device1024, or otherwise.

It should be appreciated that, in some embodiments, the methods300,400,500,600,700,800,900,1100,1200,1300,1400, and/or1500may be embodied as various instructions stored on a computer-readable media, which may be executed by the processor120, the I/O subsystem124, the accelerator134, peripheral devices136and/or other components of the computing device100to cause the computing device100to perform the respective method300,400,500,600,700,800,900,1100,1200,1300,1400, and/or1500. The computer-readable media may be embodied as any type of media capable of being read by the computing device100including, but not limited to, the memory128, the data storage device130, firmware devices, other memory or data storage devices of the computing device100, portable media readable by a peripheral device136of the computing device100, and/or other media.

EXAMPLES

Illustrative examples of the technologies disclosed herein are provided below. An embodiment of the technologies may include any one or more, and any combination of, the examples described below.

Example 1 includes a computing device for secure device configuration, the computing device comprising an I/O device; and a trusted agent to securely command the I/O device to enter a trusted I/O mode; securely command the I/O device to set a global lock on configuration registers of the I/O device in response to a secure command of the I/O device to enter the trusted I/O mode; receive configuration data indicative of the configuration registers of the I/O device in response to a securely command of the I/O device to set the global lock; securely command the I/O device to atomically release the global lock and set a fine-grained lock on one or more of the configuration registers in response to receipt of the configuration data; and provide the configuration data to a trusted execution environment of the computing device in response to a secure command of the I/O device to atomically release the global lock and set the fine-grained lock.

Example 2 includes the subject matter of Example 1, and wherein the I/O device is to receive a configuration command; determine whether the configuration command is restricted to the trusted agent; determine whether the configuration command is received from the trusted agent in response to a determination that the configuration command is restricted to the trusted agent; process the configuration command in response to a determination that the configuration command is not restricted to the trusted agent or a determination that the configuration command is received from the trusted agent; and reject the configuration command in response to the determination that the configuration command is restricted to the trusted agent and a determination that the configuration command is not received from the trusted agent.

Example 3 includes the subject matter of any of Examples 1 and 2, and wherein to determine whether the configuration command is restricted to the trusted agent comprises to determine whether the configuration command comprises a command to enter or exit the trusted I/O mode or a command to lock or unlock a configuration register.

Example 4 includes the subject matter of any of Examples 1-3, and wherein the I/O device is further to determine whether the I/O device is in the trusted I/O mode in response to the determination that the configuration command is not restricted to the trusted agent; determine whether a configuration register associated with the configuration command is locked in response to a determination that the I/O device is in the trusted I/O mode; and determine whether the configuration command is received from the trusted agent in response to a determination that the configuration register is locked; wherein to reject the configuration command further comprises to reject the configuration command in response to the determination that the configuration register is locked and a determination that the configuration command is not received from the trusted agent.

Example 5 includes the subject matter of any of Examples 1-4, and wherein the trusted agent is further to perform an attestation protocol with the I/O device to generate a device attestation report; verify the device attestation report; and securely provision the I/O device with a provisioning key in response to verification of the device attestation report.

Example 6 includes the subject matter of any of Examples 1-5, and wherein to securely command the I/O device comprises to protect a configuration command with the provisioning key; and the I/O device is to determine whether the configuration command is received from the trusted agent based on the provisioning key.

Example 7 includes the subject matter of any of Examples 1-6, and wherein the trusted agent is further to sign the device attestation report with a private key of the trusted agent to generate a signed device attestation report; include the signed device attestation report and a trusted agent attestation report in an attestation report; and provide the attestation report to the trusted execution environment.

Example 8 includes the subject matter of any of Examples 1-7, and wherein the trusted execution environment is to verify the attestation report; and request the trusted agent to prepare the I/O device for trusted I/O in response to verification of the attestation report; wherein to securely command the I/O device to enter the trusted I/O mode comprises to securely command the I/O device to enter the trusted I/O mode in response to a request to the trusted agent to prepare the I/O device for trusted I/O.

Example 9 includes the subject matter of any of Examples 1-8, and wherein the trusted execution environment is to verify the configuration data.

Example 10 includes the subject matter of any of Examples 1-9, and further including a virtual machine monitor, wherein to securely command the I/O device comprises to (i) protect a configuration command with a provisioning key, and (ii) provide the configuration command to the virtual machine monitor in response to protection of the configuration command; and the virtual machine monitor is to provide the configuration command to the I/O device.

Example 11 includes the subject matter of any of Examples 1-10, and wherein the trusted agent is further to determine whether the trusted execution environment has released the I/O device; and securely command the I/O device to release the fine-grained lock in response to a determination that the trusted execution environment has released the I/O device.

Example 12 includes the subject matter of any of Examples 1-11, and wherein the trusted agent is further to determine whether no other trusted execution environment is using the I/O device in response to the determination that the trusted execution environment has released the I/O device; and securely command the I/O device to exit the trusted I/O mode in response to a determination that no other trusted execution environment is using the I/O device.

Example 13 includes the subject matter of any of Examples 1-12, and further including a virtual machine monitor, wherein the trusted agent is further to receive a request to reclaim the I/O device from the virtual machine monitor; and request the trusted execution environment to release the I/O device in response to receipt of the request to reclaim the I/O device.

Example 14 includes the subject matter of any of Examples 1-13, and further including a virtual machine monitor to enforce an account permission assigned to the trusted agent.

Example 15 includes the subject matter of any of Examples 1-14, and wherein the trusted agent comprises a secure enclave established with secure enclave support of a processor of the computing device; and the trusted execution environment comprises a trust domain established with multi-key total memory encryption support of the processor.

Example 16 includes a method for secure device configuration, the method comprising securely commanding, by a trusted agent of a computing device, an I/O device of the computing device to enter a trusted I/O mode; securely commanding, by the trusted agent, the I/O device to set a global lock on configuration registers of the I/O device in response to securely commanding the I/O device to enter the trusted I/O mode; receiving, by the trusted agent, configuration data indicative of the configuration registers of the I/O device in response to securely commanding the I/O device to set the global lock; securely commanding, by the trusted agent, the I/O device to atomically release the global lock and set a fine-grained lock on one or more of the configuration registers in response to receiving the configuration data; and providing, by the trusted agent, the configuration data to a trusted execution environment of the computing device in response to securely commanding the I/O device to atomically release the global lock and set the fine-grained lock.

Example 17 includes the subject matter of Example 16, and further including receiving, by the I/O device, a configuration command; determining, by the I/O device, whether the configuration command is restricted to the trusted agent; determining, by the I/O device, whether the configuration command is received from the trusted agent in response to determining that the configuration command is restricted to the trusted agent; processing, by the I/O device, the configuration command in response to determining that the configuration command is not restricted to the trusted agent or determining that the configuration command is received from the trusted agent; and rejecting, by the I/O device, the configuration command in response to determining that the configuration command is restricted to the trusted agent and determining that the configuration command is not received from the trusted agent.

Example 18 includes the subject matter of any of Examples 16 and 17, and wherein determining whether the configuration command is restricted to the trusted agent comprises determining whether the configuration command comprises a command to enter or exit the trusted I/O mode or a command to lock or unlock a configuration register.

Example 19 includes the subject matter of any of Examples 16-18, and further including determining, by the I/O device, whether the I/O device is in the trusted I/O mode in response to determining that the configuration command is not restricted to the trusted agent; determining, by the I/O device, whether a configuration register associated with the configuration command is locked in response to determining that the I/O device is in the trusted I/O mode; and determining, by the I/O device, whether the configuration command is received from the trusted agent in response to determining that the configuration register is locked; wherein rejecting the configuration command further comprises rejecting the configuration command in response to determining that the configuration register is locked and determining that the configuration command is not received from the trusted agent.

Example 20 includes the subject matter of any of Examples 16-19, and further including performing, by the trusted agent, an attestation protocol with the I/O device to generate a device attestation report; verifying, by the trusted agent, the device attestation report; and securely provisioning, by the trusted agent, the I/O device with a provisioning key in response to verifying the device attestation report.

Example 21 includes the subject matter of any of Examples 16-20, and wherein securely commanding the I/O device comprises protecting, by the trusted agent, a configuration command with the provisioning key; and determining, by the I/O device, whether the configuration command is received from the trusted agent based on the provisioning key.

Example 22 includes the subject matter of any of Examples 16-21, and further including signing, by the trusted agent, the device attestation report with a private key of the trusted agent to generate a signed device attestation report; including, by the trusted agent, the signed device attestation report and a trusted agent attestation report in an attestation report; and providing, by the trusted agent, the attestation report to the trusted execution environment.

Example 23 includes the subject matter of any of Examples 16-22, and further including verifying, by the trusted execution environment, the attestation report; and requesting, by the trusted execution environment, the trusted agent to prepare the I/O device for trusted I/O in response to verifying the attestation report; wherein securely commanding the I/O device to enter the trusted I/O mode comprises securely commanding the I/O device to enter the trusted I/O mode in response to requesting the trusted agent to prepare the I/O device for trusted I/O.

Example 24 includes the subject matter of any of Examples 16-23, and further including verifying, by the trusted execution environment, the configuration data.

Example 25 includes the subject matter of any of Examples 16-24, and wherein securely commanding the I/O device comprises protecting, by the trusted agent, a configuration command with a provisioning key; providing, by the trusted agent, the configuration command to a virtual machine monitor in response to protecting the configuration command; and providing, by the virtual machine monitor, the configuration command to the I/O device.

Example 26 includes the subject matter of any of Examples 16-25, and further including determining, by the trusted agent, whether the trusted execution environment has released the I/O device; and securely commanding, by the trusted agent, the I/O device to release the fine-grained lock in response to determining that the trusted execution environment has released the I/O device.

Example 27 includes the subject matter of any of Examples 16-26, and further including determining, by the trusted agent, whether no other trusted execution environment is using the I/O device in response to determining that the trusted execution environment has released the I/O device; and securely commanding, by the trusted agent, the I/O device to exit the trusted I/O mode in response to determining that no other trusted execution environment is using the I/O device.

Example 28 includes the subject matter of any of Examples 16-27, and further including receiving, by the trusted agent, a request to reclaim the I/O device from a virtual machine monitor of the computing device; and requesting, by the trusted agent, the trusted execution environment to release the I/O device in response to receiving the request to reclaim the I/O device.

Example 29 includes the subject matter of any of Examples 16-28, and further including enforcing, by a virtual machine monitor of the computing device, an account permission assigned to the trusted agent.

Example 30 includes the subject matter of any of Examples 16-29, and wherein the trusted agent comprises a secure enclave established with secure enclave support of a processor of the computing device; and the trusted execution environment comprises a trust domain established with multi-key total memory encryption support of the processor.

Example 31 includes a computing device comprising a processor; and a memory having stored therein a plurality of instructions that when executed by the processor cause the computing device to perform the method of any of Examples 16-30.

Example 32 includes one or more non-transitory, computer readable storage media comprising a plurality of instructions stored thereon that in response to being executed result in a computing device performing the method of any of Examples 16-30.

Example 33 includes a computing device comprising means for performing the method of any of Examples 16-30.

Example 34 includes a computing device for secure device configuration, the computing device comprising: an I/O device; and a trusted agent to: authenticate the I/O device of the computing device; perform an authenticated key exchange protocol with the I/O device in response to authentication of the I/O device; verify a device configuration report in response to performance of the authenticated key exchange protocol, wherein the device configuration report is indicative of configuration registers of the I/O device; and send the device configuration report to a trusted execution environment of the computing device in response to verification of the device configuration report.

Example 35 includes the subject matter of Example 34, and wherein the trusted agent is further to: provide a trusted agent attestation report to a trusted firmware component of the computing device; and perform an authenticated key exchange protocol with the trusted execution environment in response to providing of the trusted agent attestation report.

Example 36 includes the subject matter of any of Examples 34 and 35, and wherein the trusted agent comprises a trusted execution environment having a predetermined identity, wherein the predetermined entity is known to the trusted firmware component.

Example 37 includes the subject matter of any of Examples 34-36, and wherein: the trusted agent is further to receive a request to authenticate an I/O device from a host operating system of the computing device; and to authenticate the I/O device comprises to authenticate the I/O device in response to receipt of the request to authenticate the I/O device.

Example 38 includes the subject matter of any of Examples 34-37, and wherein: to receive the request to authenticate the I/O device comprises to receive a device certificate; and to authenticate the I/O device comprises to authenticate the I/O device with the device certificate.

Example 39 includes the subject matter of any of Examples 34-38, and wherein the host operating system is to: determine wherein the I/O device is capable of trusted I/O; and send the request to authenticate the I/O device to the trusted agent in response to a determination that the I/O device is capable of trusted I/O.

Example 40 includes the subject matter of any of Examples 34-39, and wherein: to perform the authenticated key exchange protocol with the I/O device comprises to provision a shared secret key to the I/O device; and to verify the device configuration report comprises to verify integrity of the device configuration report with the shared secret key.

Example 41 includes the subject matter of any of Examples 34-40, and wherein the trusted agent is further to receive the device configuration report from a host operating system of the computing device.

Example 42 includes the subject matter of any of Examples 34-41, and wherein the host operating system is to: command the I/O device to lock configuration of the I/O device; receive the device configuration report from the I/O device in response to commanding of the I/O device to lock the configuration; and provide the device configuration report to the trusted agent.

Example 43 includes the subject matter of any of Examples 34-42, and wherein: the host operating system is further to program the configuration of the I/O device; and to command the I/O device to lock the configuration comprises to command the I/O device to lock the configuration in response to programming of the configuration.

Example 44 includes the subject matter of any of Examples 34-43, and wherein: the trusted agent is further to determine whether the I/O device is assigned to a different trusted execution environment in response to verification of the device configuration report; the trusted agent is further to record a binding between the I/O device and the trusted execution environment in response to a determination that the I/O device is not assigned to a different trusted execution environment; and to send the device configuration report to the trusted execution environment further comprises to send the device configuration report to the trusted execution environment in response to recordation of the binding.

Example 45 includes the subject matter of any of Examples 34-44, and wherein the trusted agent is further to remove the binding between the I/O device and the trusted execution environment in response to release of the I/O device by the trusted execution environment.

Example 46 includes the subject matter of any of Examples 34-45, and wherein: the trusted agent is further to receive a request to reclaim the I/O device from a host operating system; the trusted agent is further to verify, with a trusted firmware component of the computing device, that the trusted execution environment is terminated in response to receipt of the request to reclaim the I/O device; and to remove the binding between the I/O device and the trusted execution environment further comprises to remove the binding between the I/O device and the trusted execution environment in response to verification that the trusted execution environment is terminated.

Example 47 includes the subject matter of any of Examples 34-46, and wherein the trusted agent is further to: receive a request to reclaim the I/O device from a host operating system; verify that the I/O device is removed from the computing device in response to receipt of the request to reclaim the I/O device; and remove the binding between the I/O device and the trusted execution environment in response to verification that the I/O device is removed.

Example 48 includes the subject matter of any of Examples 34-47, and wherein the trusted agent is further to provision a link encryption key to the I/O device in response to performance of the authenticated key exchange protocol with the I/O device.

Example 49 includes the subject matter of any of Examples 34-48, and wherein the trusted execution environment is to: receive the trusted agent attestation report from the trusted firmware component; and verify the trusted agent attestation report; wherein to perform the authenticated key exchange protocol with the trusted execution environment comprises to perform the authenticated key exchange protocol with the trusted execution environment in response to verification of the trusted agent attestation report.

Example 50 includes the subject matter of any of Examples 34-49, and wherein the trusted execution environment is to: verify the device configuration report in response to sending of the device configuration report; and perform trusted I/O with the I/O device in response to verification of the device configuration report.

Example 51 includes the subject matter of any of Examples 34-50, and wherein to perform the trusted I/O comprises to securely transfer, by the I/O device, I/O data with a link encryption key.

Example 52 includes the subject matter of any of Examples 34-51, and wherein to verify the device configuration report comprises to verify a device feature of the I/O device.

Example 53 includes the subject matter of any of Examples 34-52, and wherein to verify the device configuration report comprises to verify a register address indicated by the device configuration report.

Example 54 includes the subject matter of any of Examples 34-53, and wherein to verify the device configuration report comprises to verify a link encryption status indicated by the device configuration report.

Example 55 includes the subject matter of any of Examples 34-54, and wherein the trusted agent comprises a first trust domain and wherein the trusted execution environment comprises a second trust domain.

Example 56 includes a method for secure device configuration, the method comprising: authenticating, by a trusted agent of a computing device, an I/O device of the computing device; performing, by the trusted agent, an authenticated key exchange protocol with the I/O device in response to authenticating the I/O device; verifying, by the trusted agent, a device configuration report in response to performing the authenticated key exchange protocol, wherein the device configuration report is indicative of configuration registers of the I/O device; and sending, by the trusted agent, the device configuration report to a trusted execution environment of the computing device in response to verifying the device configuration report.

Example 57 includes the subject matter of Example 56, and further comprising: providing, by the trusted agent, a trusted agent attestation report to a trusted firmware component of the computing device; and performing, by the trusted agent, an authenticated key exchange protocol with a trusted execution environment of the computing device in response to providing the trusted agent attestation report.

Example 58 includes the subject matter of any of Examples 56 and 57, and wherein the trusted agent comprises a trusted execution environment having a predetermined identity, wherein the predetermined entity is known to the trusted firmware component.

Example 59 includes the subject matter of any of Examples 56-58, and further comprising: receiving, by the trusted agent, a request to authenticate an I/O device from a host operating system of the computing device; wherein authenticating the I/O device comprises authenticating the I/O device in response to receiving the request to authenticate the I/O device.

Example 60 includes the subject matter of any of Examples 56-59, and wherein: receiving the request to authenticate the I/O device comprises receiving a device certificate; and authenticating the I/O device comprises authenticating the I/O device with the device certificate.

Example 61 includes the subject matter of any of Examples 56-60, and further comprising: determining, by the host operating system, wherein the I/O device is capable of trusted I/O; and sending, by the host operating system, the request to authenticate the I/O device to the trusted agent in response to determining that the I/O device is capable of trusted I/O.

Example 62 includes the subject matter of any of Examples 56-61, and wherein: performing the authenticated key exchange protocol with the I/O device comprises provisioning a shared secret key to the I/O device; and verifying the device configuration report comprises verifying integrity of the device configuration report with the shared secret key.

Example 63 includes the subject matter of any of Examples 56-62, and further comprising receiving, by the trusted agent, the device configuration report from a host operating system of the computing device.

Example 64 includes the subject matter of any of Examples 56-63, and further comprising: commanding, by the host operating system, the I/O device to lock configuration of the I/O device; receiving, by the host operating system, the device configuration report from the I/O device in response to commanding the I/O device to lock the configuration; and providing, by the host operating system, the device configuration report to the trusted agent.

Example 65 includes the subject matter of any of Examples 56-64, and further comprising: programming, by the host operating system, the configuration of the I/O device; wherein commanding the I/O device to lock the configuration comprises commanding the I/O device to lock the configuration in response to programming the configuration.

Example 66 includes the subject matter of any of Examples 56-65, and further comprising: determining, by the trusted agent, whether the I/O device is assigned to a different trusted execution environment in response to verifying the device configuration report; and recording, by the trusted agent, a binding between the I/O device and the trusted execution environment in response to determining that the I/O device is not assigned to a different trusted execution environment; wherein sending the device configuration report to the trusted execution environment further comprises sending the device configuration report to the trusted execution environment in response to recording the binding.

Example 67 includes the subject matter of any of Examples 56-66, and further comprising removing, by the trusted agent, the binding between the I/O device and the trusted execution environment in response to the trusted execution environment releasing the I/O device.

Example 68 includes the subject matter of any of Examples 56-67, and further comprising: receiving, by the trusted agent, a request to reclaim the I/O device from a host operating system; and verifying, by the trusted agent with a trusted firmware component of the computing device, that the trusted execution environment is terminated in response to receiving the request to reclaim the I/O device; wherein removing the binding between the I/O device and the trusted execution environment further comprises removing the binding between the I/O device and the trusted execution environment in response to verifying that the trusted execution environment is terminated.

Example 69 includes the subject matter of any of Examples 56-68, and further comprising: receiving, by the trusted agent, a request to reclaim the I/O device from a host operating system; verifying, by the trusted agent, that the I/O device is removed from the computing device in response to receiving the request to reclaim the I/O device; and removing, by the trusted agent, the binding between the I/O device and the trusted execution environment in response to verifying that the I/O device is removed.

Example 70 includes the subject matter of any of Examples 56-69, and further comprising provisioning, by the trusted agent, a link encryption key to the I/O device in response to performing the authenticated key exchange protocol with the I/O device.

Example 71 includes the subject matter of any of Examples 56-70, and further comprising: receiving, by the trusted execution environment, the trusted agent attestation report from the trusted firmware component; and verifying, by the trusted execution environment, the trusted agent attestation report; wherein performing the authenticated key exchange protocol with the trusted execution environment comprises performing the authenticated key exchange protocol with the trusted execution environment in response to verifying the trusted agent attestation report.

Example 72 includes the subject matter of any of Examples 56-71, and further comprising: verifying, by the trusted execution environment, the device configuration report in response to sending the device configuration report; and performing, by the trusted execution environment, trusted I/O with the I/O device in response to verifying the device configuration report.

Example 73 includes the subject matter of any of Examples 56-72, and wherein performing the trusted I/O comprises securely transferring, by the I/O device, I/O data with a link encryption key.

Example 74 includes the subject matter of any of Examples 56-73, and wherein verifying the device configuration report comprises verifying a device feature of the I/O device.

Example 75 includes the subject matter of any of Examples 56-74, and wherein verifying the device configuration report comprises verifying a register address indicated by the device configuration report.

Example 76 includes the subject matter of any of Examples 56-75, and wherein verifying the device configuration report comprises verifying a link encryption status indicated by the device configuration report.

Example 77 includes the subject matter of any of Examples 56-76, and wherein the trusted agent comprises a first trust domain and wherein the trusted execution environment comprises a second trust domain.

Example 78 includes a computing device comprising a processor; and a memory having stored therein a plurality of instructions that when executed by the processor cause the computing device to perform the method of any of Examples 56-77.

Example 79 includes one or more non-transitory, computer readable storage media comprising a plurality of instructions stored thereon that in response to being executed result in a computing device performing the method of any of Examples 56-77.

Example 80 includes a computing device comprising means for performing the method of any of Examples 56-77.