Patent Publication Number: US-11050844-B2

Title: User controlled hardware validation

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
     This application is a continuation of U.S. application Ser. No. 15/085,787 entitled “HARDWARE VALIDATION”, filed Mar. 30, 2016, which is incorporated herein by reference for all purposes. 
    
    
     BACKGROUND 
     As an increasing number of applications and services are being made available over networks such as the Internet, an increasing number of content, application, and service providers are turning to technologies such as cloud computing. Cloud computing, in general, is an approach to providing access to electronic resources through services, such as Web services, where the hardware and/or software used to support those services is dynamically scalable to meet the needs of the services at any given time. A customer typically will rent, lease, or otherwise pay for access to resources through the cloud, such that the customer does not have to purchase and maintain the hardware and software to provide access to these resources. A potential disadvantage to such an approach, at least from a customer point of view, is that the resources typically are at a location under control of the provider of those resources, and thus are out of the direct control of the customer. Further, the resources are often shared with other customers such that data, software, and resource security and integrity may be compromised. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       Various embodiments in accordance with the present disclosure will be described with reference to the drawings, in which: 
         FIG. 1  illustrates an example environment in which various embodiments can be implemented. 
         FIG. 2  illustrates an example configuration for a virtual machine manager running multiple guest virtual machines that can be used in accordance with various embodiments. 
         FIG. 3  illustrates an example environment including a trusted co-processor that can be utilized in accordance with various embodiments. 
         FIG. 4  illustrates an example environment wherein a trusted co-processor can be provided to a customer through a customer environment and/or a resource provider environment that can be utilized in accordance with various embodiments. 
         FIG. 5  illustrates an example process for enabling a customer to verify state of a host computing device before performing tasks on that device that can be utilized in accordance with various embodiments. 
         FIG. 6  illustrates an example process for enabling a customer to verify operation of a provider-owned host computing device using a customer-owned trusted co-processor, installed on the host computing device, that can be utilized in accordance with various embodiments. 
         FIG. 7  illustrates example components of a computing device that can be utilized in accordance with various embodiments. 
     
    
    
     DETAILED DESCRIPTION 
     Systems and methods in accordance with various embodiments of the present disclosure may overcome one or more of the aforementioned and other deficiencies experienced in conventional approaches to managing security in a computing environment. In particular, various embodiments provide a trusted enclave inside a host machine operated by a resource provider, or other such entity, where that trusted enclave is logically owned by a separate party, such as a customer of the resource provider. The trusted enclave in some embodiments can take the form of a peripheral card including a trusted co-processor, where the card is connected to a high speed bus, such as a peripheral component interconnect (PCI bus), in the host machine. The trusted co-processor can have at least one API that the customer can use to call into the trusted co-processor, and the card itself may be at least somewhat customer programmable. When the customer is allocated such a trusted co-processor, the provider can indicate one or more expected values for the card based at least in part upon information such as the configuration, software installed, etc. The customer can submit a validation request to the trusted co-processor, and if the returned information matches the expected values, or is otherwise verifiable, the customer can take logical ownership of the trusted co-processor and lock out the provider. 
     The customer can have some level of access and visibility into the host machine, and can use this trusted enclave as an observation point to verify certain aspects of the operation of the host machine, such as software versions installed, guests operating on the host, etc. The customer can periodically submit verification requests and receive back verification data that can be compared against the expected values. If any unexpected change has occurred, the customer can take a remedial action, such as to cease operation on that host, contact the provider for a new allocation, etc. 
     Various other applications, processes, and uses are presented below with respect to the various embodiments. 
       FIG. 1  illustrates an example environment  100  in which aspects of the various embodiments can be implemented. In this example, a user is able to utilize a computing device  102  to submit a call or request across at least one network  104  to be received to a shared resource environment  106 , such as a data center or “cloud” environment, among other such options. The computing device  102  can include any appropriate device, as may include client devices such as personal computers, cell phones, handheld messaging devices, laptop computers, set-top boxes, personal data assistants, electronic book readers, and the like. The at least one network  104  can include any appropriate wired and/or wireless network, including an intranet, the Internet, a cellular network, a local area network, or any other such network or combination thereof. Components used for such a system can depend at least in part upon the type of network and/or environment selected. Protocols and components for communicating via such a network are well known and will not be discussed herein in detail. 
     The resource environment  106  in this example includes an interface layer  118 , as may include components such as application programming interfaces (APIs), routers, switches, Web servers, and other such components known or used for routing communications from a user computing device  102  to an appropriate resource in the environment. In this example, the resource environment  106  includes a number of racks  108 , each rack including a number of host computing devices  110 , as well as an optional rack support computing system  116  in this example embodiment. The host computing systems  110  on one of the illustrated racks  108  each host one or more virtual machines  114  in this example, as well as a distinct virtual machine manager  112  associated with the virtual machines on that host computing system. The virtual machine manager (VMM) is tasked with managing the virtual machines (VMs) on the respective host device, and handling various aspects of virtualization. Each virtual machine  114  can act as an independent computing resource for executing one or more tasks on behalf of a user, where the virtual machine functions as a dedicated resource for the user. The environment can also include additional host computing systems that do not include distinct virtual machines, but may nonetheless each act as a computing resource for one or more users. The rack support computing system  116  may provide various utility services for other computing systems local to its rack (e.g., long-term program storage, metering, and other monitoring of program execution and/or of non-local block data storage access performed by other computing systems local to the rack, etc.), as well as possibly to other computing systems located in the environment  106 . Each computing system may also have one or more local attached storage devices (not shown), such as to store local copies of programs and/or data created by or otherwise used by the executing programs, as well as various other components. 
     It will be appreciated that the example of  FIG. 1  has been simplified for the purposes of explanation, and that the number and organization of host computing systems and other devices may be much larger than what is depicted in  FIG. 1 . For example, as one illustrative embodiment, there may be approximately tens of thousands of computing systems in a cloud environment, with at least some of those computing systems being host computing systems that may each host multiple virtual machines. 
     A number of virtualization techniques can be used to simultaneously operate a plurality of guest virtual machines (VMs) or guest operating systems (OSs) on a given host machine.  FIG. 2  illustrates an example  200  of utilizing one virtualization technique using a virtual machine manager (VMM), such as a hypervisor, in accordance with various embodiments. The hardware  204  (e.g., the central processor and other such components) of the host computing device  202  is able to interface with the VMM  206  running directly on the hardware  204 , such in the case of a “bare metal” or native hypervisor. Examples of hypervisors useful for such purposes include Xen, Hyper-V®, and the like. Hypervisors typically run at a higher, more privileged processor state than any other software on the machine, and provide services such as memory management and processor scheduling for dependent layers and/or domains. The most privileged of such layers and/or domains resides in the service domain layer, which may include a host domain  208  that may include an administrative operating system for configuring the operation and functionality of the hypervisor  206 , as well as that of domains of lower privilege, such as the domains of the guest virtual machines  210 ,  212 ,  214  or other operating systems, which may be heterogeneous (i.e., running different operating systems than each other). The host domain  208  (e.g., the DOM-0) may have direct access to the hardware resources  204  of the host computing device  202  by way of the hypervisor  206 , while the guest virtual machine domains  210 ,  212 ,  214  may not. 
     In certain embodiments, there can be various modes of operations. For example, to update the microcode on a host computing device, the host computing device can receive an update from a trusted source and enter a system management mode (SMM), such as by receiving a system management interrupt (SMI) to interrupt normal hypervisor execution. Entering SMM causes the execution of all the physical processor cores except for a single physical core (e.g., the bootstrapping core) in the processor to be suspended in at least some conventional designs. The single physical core that remains executing will be responsible for applying the update. In some embodiments, it may be ensured that the processor finishes executing all of the update for the particular instruction that it was executing at the time of detecting the SMI or other interrupt signal before suspending the execution of the physical cores. The received update (e.g., for firmware, software, or microcode) can be signed and/or encrypted by a trusted source, such as the service provider or processor manufacturer. In one example, the update can be signed using a private key or other such credential of a trusted source in order to ensure authenticity of the update. If the update is signed using a private key, the host computing device may need to validate the signature of the update before applying the patch. In some embodiments, the public key used to verify the signature can be stored in a trusted platform module (TPM) (not shown) on the host computing device. In some embodiments, the update can be encrypted using asymmetric or symmetric cryptography. For example, the update may be encrypted using a public key in order to ensure privacy of the update. If the update is encrypted, then any intermediate parties that handle the update cannot read the update or ascertain its contents unless they have access to the private key used to the decrypt the update. If the update is encrypted, then the private key to decrypt the update can be stored to the TPM, similarly to the public key for verifying the update, as previously described. Once the update is authenticated and/or decrypted, the update can be applied to the host computing device  202 . If SMM is used to apply the update, then the instructions to apply the update can be implemented in an SMI handler, or SMI handler can cryptographically verify and call a patch function pre-configured in a hypervisor. A potential downside to such an approach, however, is that the virtual machine manager (VMM) on a host machine that is exposed to one or more outside users can become compromised. As a result, one of the guest virtual machines (VMs)  210 ,  212 ,  214  on the host computing device  202  can potentially access the TPM in the host hardware  204  through the hypervisor  206 . 
     Approaches in accordance with various embodiments can attempt to improve security in such an environment while ensuring that customer secrets, keys, secure data, and other such objects are well secured within that environment, and that no unintended or unexpected modification of the environment has occurred.  FIG. 3  illustrates an example environment  300  that can be used to provide such security. It should be noted that reference numbers for similar components can be carried over between figures for purposes of simplicity of explanation, but such usage should not be interpreted as a limitation of the scope of the various embodiments unless otherwise specifically stated. In this example, the host computing device  202  includes (or is capable of receiving) a trusted co-processor  304 , or secure co-processor, that can communicate with the VMM  206  and other components of the host machine. The trusted co-processor  304  can be provided via a peripheral device, for example, such as a PCI device that is removable from the host computing device  202 . The trusted co-processor can include one or more processors, memory, one or more interfaces, and any other components or connections known for providing a peripheral device or additional processor to such a system. In one embodiment, the trusted co-processor is an independent hardware component that takes the form of a separate PCIE card that includes specific firmware. The configuration of the trusted co-processor enables it to be logically considered to be within a trust boundary, cryptographic boundary, trust region, or trusted enclave, associated with the customer VM. In at least some embodiments, the peripheral card can be manufactured, configured, or otherwise provided by a provider of the resource environment, or an entity trusted by the provider, such that the provider can have a higher level of trust in the security of the peripheral card, or other such additional or removable hardware, than in the security of the raw host computing device  202 . 
     A trusted co-processor  304 , or other such hardware device, can provide the advantage of being able to offload many of the administrative tasks from the host computing device  202  such that the host computing device can be substantially dedicated to hosting customer virtual machines, processing customer data, or performing other such tasks. The computational resources dedicated to the customer can thus be provided by the host computing device, enabling the host hardware usage for customer tasks to be maximized, while the administration of those resources is managed by the trusted co-processor. This has the further advantage that all administrative processes will be run in a trusted computing environment  308 , or within a trust boundary corresponding to the hardware with the higher level of provider trust. A further advantage is that the provider can provide security assurances that provider access is limited to only the necessary resources, and the system is running trusted software. 
     The components such as the trusted co-processor  304  within the trusted computing environment  308  thus can be logically considered a sub-system on the host device, where that subsystem has relatively limited functionality. This separate, “limited” subsystem is contained within the associated cryptographic boundary, and is referred to herein as “1-sys.” At least a portion of the hardware of the host can then be dedicated to providing and maintaining the virtual machines, or virtual hosts, running on the host machine. This “main” subsystem, contained within a separate cryptographic boundary, is referred to herein as “m-sys.” The provider access thus can be at least somewhat limited to the 1-sys for purposes of provider administrative functions for the guest machines hosted in the m-sys. 
     Even with such separation and limiting of provider access, however, the provider will still need to be able to provide assurances to customers (with virtual instances running in the virtual environment) that the virtual image for each customer instance is in a “pristine” or expected state, without unexpected modification, and that the instance is running in a trusted execution environment. Assurances can be provided using a tiered or chained credentialing mechanism, whereby subsystems can have their own credentialing mechanisms enabling the subsystems to be identified and the software and virtual systems running on top of subsystems to be verified as authentic. Each subsystem in at least some embodiments can have its own cryptographic boundary, secure storage, and trusted computing capabilities, as well as an ability to run additional firmware and software. 
     Further, at least some subsystems can utilize separate memory and processor space as part of a trusted computing environment, such as may be modeled as a trusted platform module (TPM) or other such component. A TPM can be used to perform remote attestation, such as to remotely verify the authenticity of a system running on a host. By providing a series, chain, or hierarchy of TPMs, for example, the subsystems can be identified and the software and virtual systems running a top those sub-systems can be verified as authentic. Although examples discussed herein relate to a two-system host, including the 1-sys and m-sys, it should be understood that various other numbers and/or configuration of subsystems can be used as appropriate within the scope of the various embodiments. 
     One approach for verifying that the embedded systems were not unexpectedly modified or tampered with, and that the embedded systems and virtual machines will function as expected, involves remote attestation. Remote attestation provides for the detection of unauthorized or unexpected changes to a computing environment, such as various subsystems described herein. 
     These changes can include modifications to attempt to circumvent security measures. During a conventional remote attestation process, a host machine can generate a certificate indicating the software currently running on the host. The certificate can then be provided to the customer, or another appropriate party, in order to prove that the software currently executing in the environment is unmodified or as expected. Remote attestation can take advantage of public key encryption, for example, to ensure that the information regarding the software is only exposed to the party requesting the attestation, or other party having obtained the appropriate key. A TPM in general is a dedicated microprocessor that can secure hardware by installing cryptographic keys into the hardware. A TPM can utilize a unique encryption key that is inaccessible outside the TPM in order to enforce expected behaviors on the hardware and software of the host. In some embodiments the unique key is an endorsement key, such as a 2048-bit RSA public and private key pair, that is immutable and unexportable from the TPM. The public key for the endorsement key can be contained within a certificate for the TPM. A TPM can perform remote attestation in some embodiments by generating a hash summary of the hardware and software configuration of the corresponding environment or subsystem. The hash value can be generated using any appropriate cryptographic hashing algorithm, such as may generate MD5, SHA-1, SHA-2, or SHA-3 hashes. 
     Approaches in accordance with various embodiments can utilize a first TPM  306  in the 1-sys, or peripheral with trusted co-processor  304  in this example, and a second TPM  302  in the m-sys, or remainder of the host computing device  202  in this example, to perform remote attestation. The trusted 1-sys environment  308  can function as a root of trust for the attestation of one or more other subsystems, such as the m-sys, on the device, capable of performing an attestation of the m-sys. The 1-sys can generate an attestation integrity key pair and a host identity key pair, with the public key bound to a certificate from a certificate authority as discussed previously. That information can be used to uniquely identify the device within the infrastructure, as well as to generate a hash for attestation. In some embodiments, an attestation request will include some type of question, task, or challenge, and the device receiving the request can gather the information with the question, task, or challenge and generate a hash to be returned to the source of the request (or another appropriate party or destination). The attestation request can be received through an API and triggered through a console or other such interface. In some embodiments the attestation request can come from within the customer virtual machine, such as from a virtual TPM or other such attestation device. The requesting party can then use its copy of the appropriate key to ensure that the correct challenge was included, as well as the expected values for all information for the challenge. This can help to ensure that the software, hardware, configuration, and/or other aspects of the 1-sys are as expected and no unexpected modification has occurred. Once the 1-sys is verified to be operating as expected, the 1-sys (which can include firmware under control of the provider) can boot or otherwise initialize the m-sys, or upper level system. The m-sys can also have a TPM that can generate an attestation integrity key pair and a host identity key pair, with the public key bound to an appropriate certificate. That information can be used to uniquely identify the device within the infrastructure, as well as to generate a hash for attestation. In this example, the 1-sys can be the certificate issuing authority for the m-sys. The 1-sys, or trusted co-processor, can thus issue the certificate, act as a certificate sub-authority, and certify that it is physically attached to the m-sys TPM, since the trusted co-processor has a physical connection to the host. Such an approach can provide a full attestation chain. A customer or other entity wanting to attest a host can obtain an attestation through the certificate of the m-sys, corresponding to the host of the designated virtual machine, that is chained to the 1-sys, and the peripheral card or trusted co-processor, which is chained to the original certificate authority. 
     In one example, a customer system is communicating with a customer virtual machine running on a virtual host in the resource provider environment. The customer system can be aware of information such as an instance identifier (ID) bound to the virtual machine, as well as a host ID for the host computing device running the virtual machine. The customer system can then send an attest command, or attestation request, to the customer virtual machine. The virtual machine can receive the request, which may arrive through the peripheral device or 1-sys in some embodiments. The virtual machine can receive the instance ID and the host ID and determine that the virtual machine has correctly received the request, and that an attestation should be provided. In order to provide the attestation, information needs to be returned for both the trusted root system, or 1-sys, and the main system, or m-sys. In this example, the virtual machine can send an attest command to the 1-sys, which can generate the appropriate hash using the expected information and the challenge of the attest command. The 1-sys can also send an attest command to the m-sys, which can generate a similar hash for the m-sys information. The hash of the m-sys can contain information for the expected boot image of the customer for the virtual machine. The virtual machine can obtain both hash values, for example, and can send back the appropriate information for the attestation. This can include, for example, the hash values for 1-sys and m-sys, as well as the certificate chain for the 1-sys and m-sys. The customer system receiving the information can verify that the image is as expected and can trust the attestation chain. A customer system getting quotes (or other measurements, which may be signed) for each subsystem can also verify the signature on each quote, as well as the values within those signatures. The customer system can also verify that the m-sys certificate was issued by the 1-sys, and the 1-sys certificate was issued by a trusted certificate authority. As long as the customer trusts the certificate authority, the customer should be able to verify the information indicating that the customer virtual machine is operating as expected. 
     The information provided as part of the hash for verification purposes can include any appropriate information for the relevant subsystem. For the 1-sys where the trusted co-processor is on a peripheral card, for example, the hash can include values for the PCR registers, or information for the firmware for the trusted co-processor. Various other types of information can be utilized as well. A determination of which information to include can balance the usefulness of including a lot of information, which provides increased security that the system or environment is operating as expected, with the need to maintain knowledge of the current state of the system or environment such that the customer system can know or determine the appropriate hash to expect for the current state. An advantage to using a chain of attestation is that the 1-sys, in addition to being a more trusted environment due to the nature of the subsystem, is a more controlled and simple environment, such that the likelihood of a hashing or matching error is significantly reduced, and the likelihood of compromise is significantly reduced as well. Since both the hardware and software of the m-sys will be sourced from outside the resource provider environment, in many cases, there can be fewer or weaker assurances that the environment is secure. The nature of the subsystems makes it beneficial for the trusted subsystem to be able to certify the other due to the level of trust in the 1-sys and the physical connection of the 1-sys to the m-sys, among other such advantages. 
     As mentioned, a benefit of using a device such as a PCI device is that it is difficult to behave maliciously over a high speed bus, such as a PCI bus. The PCI device typically will be involved with high speed transactions that deal directly with the memory of the host system. The ability to provide a trusted co-processor on a PCI bus of a host computing device, particularly one that is programmable, provides for an increased level of trust and functionality over conventional approaches. In some embodiments, the ability to provide such a device can modify the ownership model for various resources. For example, customers of a resource provider environment might want to take advantage of various services and offerings, such as scalable computing and Web services, but the customers might not receive the level of trust needed to embrace these offerings. These customers may want to be able to verify operation and state of the resources performing tasks on behalf of those customers, regardless of whether the resources are in a customer environment or a resource provider environment, such as a data center. 
     Accordingly, approaches in accordance with various embodiments can become at least the logical owners of such observation point devices, whether those devices are located in computing devices at a client site or in a resource provider environment, among other such options. These observation point devices can be PCI devices, for example, that communicate via a PCI bus and enable the logical owner to perform observational tasks such as to interrogate memory, analyze processes running on the device, etc. The access can be limited to the observation point device, however, such that the logical owner cannot gain unauthorized access to other resources or data on the host computing device. This can provide a customer, or other logical owner, with assurance that the host computing device is operating as expected while preventing the customer from performing unauthorized tasks on the computing device. 
     As illustrated in the example situation  400  of  FIG. 4 , an observation point device can be placed in a customer environment  406  or a resource provider environment  106 , or both. The device can be a PCI device, a trusted co-processor, or another mechanism for providing such a trusted enclave. A first trusted co-processor  404  is illustrated on a specific host machine  402  (similar to the host computing device  300  of  FIG. 3 ) in the provider environment, and a second trusted co-processor  412  is illustrated on a host machine  410  in the customer environment  406 . In such an example, a customer can use a client device  408  to request observational data from either of the trusted co-processors  404 ,  412 . In various instances the trusted co-processors, or PCI devices, can be provided by the resource provider, or service provider, such that the provider can configure the device to only be able to read or write to certain areas of memory on the respective host device. In some embodiments the device might only be aware of certain areas of memory. As discussed previously with respect to  FIG. 3 , the trusted co-processor can include a TPM for storing cryptographic materials and performing cryptographic functions. In at least some embodiments such a device can thus measure its own software to ensure that the installed software is as expected. The client device  408  can connect to the device, such as the first trusted co-processor  404 , and can know reliably that the client device is connected to the trusted co-processor. The client device can also determine that the trusted co-processor is running the software that it is expected to be running. 
     The trust in the co-processor can be rooted in a variety of different ways. As an example, a disclosure can be made as to the makeup of the PCI device and the reasons why the device should be trusted. This disclosure can come from a trusted party such as a third party auditor or a disclosure of the source code for the device. The customer can then be sure that there is no man-in-the middle, or malicious firmware running on the trusted co-processor. Once the customer is able to trust that the customer is communicating directly with the communication point in a trusted way, the customer can run various queries to determine information such as the processes running on the host computing device, the users logged into the device, a list of guests running on the host and proof that those guests correspond to the customer, etc. 
     The trusted co-processor can also have at least some measure of programmability. In some embodiments the device will provide at least one API. The trusted co-processor can include enough processing power that the customer can upload code that can run on the PCI device to perform a set of checks and provide notification of anything unexpected. The API can also be accessible over the network, such as may be queried by the client device  408 . In some embodiments the customer have execute auditing software, on the customer site or in a data center, for example, that can query the API and fetch the data for analysis and verification. 
     Such an approach is an improvement over conventional approaches at least for the fact that conventional approaches do not allow customers to take ownership of a trusted component, particularly a programmable component, operating in a host computing device and capable of performing the functionality discussed herein. In some embodiments the trusted co-processor includes an IOMMU as discussed previously, which provides a memory management unit specifically for I/O operations. A host computing device can accept input from potentially untrusted parties, such as through remote direct memory access (RDMA), which is a low latency, low overhead approach. In order to quickly move data, the hardware can, at a very low level, move data to be stored in RAM on a destination machine, then notify software executing on the destination machine. Such an approach incurs cost for the bus and RAM bandwidth, but incurs no cost for any CPU cycles that would otherwise be involved in moving the data. A customer, however, generally will not want to provide a remote party with the ability to write to random addresses in a host machine. Various approaches provide at least some level of filtering, such as where a user can write to certain ranges if specified conditions are true, and is prevented from writing to other ranges no matter what. 
     Approaches in accordance with various embodiments can instead take advantage of the trusted co-processor that is highly programmable. The trusted co-processor in at least some embodiments can verify the presence of a processor connected to a bus interface, which includes an IOMMU and other bus interface bits. The trusted co-processor can also verify that the bus interface is plugged into a PCI bus or other bus within the computer. That IOMMU and the bus interface bits can be intelligent enough to indicate actions to be taken if a specified processor requests memory in a certain range, or deny a request to perform a memory operation in a different range. 
     A trusted co-processor, or PCI device, can effectively have two owners in at least some embodiments. A resource provider in one example will own the hardware, including the bus interface, the IOMMU, and other such components. The customer can be the logical owner of the compute enclave with the TPM, which can enable the customer to verify that the provider (or at least the host machine) is operating as expected. In at least some embodiments it can appear to the customer that the customer&#39;s trusted co-processor has full PCI bus access, but that access will in fact be limited or partial, and the main processor on the host that is owned by the resource provider can also have at least partial access on the host computing device. 
     In one example, a trusted co-processor can be used to verify that there is no “man-in-the-middle” that can impact the operation. In a TPM bootstrap approach, there is an initialization step that the customer trusts. This can involve, for example, the customer hand assembling the PCI device, including burning the keys into the TPM, and then watching (directly or through a trusted third party) those devices be placed into the corresponding host devices. The process can continue depending upon the level of trust, and in some cases can include watching the server get placed in the rack and then monitoring the rack to ensure no swapping or modification occurs. 
     Another approach involves an endorsement key being included in the TPM on the trusted co-processor. The endorsement key can be burnt into the hardware of the TPM at the factory. Some manufacturers will sign the endorsement key, turning the key into a certificate. Manufacturers can also sign the certificate signing request generated using the endorsement key, turning it into a certificate. A customer or other entity who did not build the TPM can nevertheless verify that the conversation being had is with a TPM manufactured by a specified manufacturer and includes a key with a specified fingerprint, and thus determine that use of the TPM is likely acceptable. 
     In one embodiment, measurements of a virtual machine manager (i.e., hypervisor) running on a host machine can be published. A resource provider can indicate the expected measurement (i.e., a SHA-256 hash value) for a range of memory locations. If the host is operating and/or configured as expected, the measurement value for the software over those memory ranges should not change. A customer wanting to verify operation of the box can select to only run “verifiable” virtual machine instances on hosts that include a trusted co-processor that is logically owned by the customer. In some embodiments a customer can be given a mapping that indicates which instances are running on which host machines, or other hardware, that can be verified using a particular identifier. The customer can be provided with a cryptographic key that the customer should expect to receive when attempting to connect to the verifier. When the customer connects to the verifier, the verifier can produce the expected key embedded in a certificate that is signed by a certificate authority that is trusted by the customer. This can enable the customer to be confident that they are connected to the correct verifier. The customer can then issue commands to that verifier, in order to verify that the software running on the verifier is correct and as expected. Once the customer is comfortable with the verifier, the customer can begin to query the verifier regarding the host computer. 
     A device-side IOMMU is not required for all trusted enclaves discussed and suggested herein. On the device side an IOMMU can help to partition the devices that are part of, for example, a system on a chip implementation. There can be different contexts, and there can be limits placed on the device memory (e.g., DRAM) that is attached to the system on chip and accessible during the various contexts. An IOMMU is not a necessary component for providing access to the host DRAM. In some embodiments a motherboard on the host side could be programmed from the trusted enclave, but could require some software cooperation. 
     As mentioned, customers can have the ability to submit queries regarding a specific host computer to verify that they are communicating with the expected computer. A customer can also have the ability to cause certain memory patterns or other specific indicators to be stored for a virtual machine, for example, that can enable the customer to determine that something very close to their virtual machine instance is running on the computer. The customer can then use the verifier to, over the PCI bus, query for the memory ranges that cover the virtual machine manager. The customer can generate the expected checksum and compare that to a checksum for the memory ranges to verify that the software located in the pages indicated by the virtual machine manager match. In some embodiments a customer can submit various API calls that enable them to interpret various management tables of the virtual machine manager, as well as to verify that there is one instance running on this box that is owned by the customer with no other instances running, etc. The customer can verify the trusted environment and use that environment to execute their software. Interrogating the PCI bus using the trusted co-processor and receiving expected results that only the customer would have can provide some level of assurance that the box that their instance is running on is in a valid and trustworthy configuration. 
     Various functionality discussed herein can be implemented without a dedicated co-processor, PCI device, or other such dedicated hardware. In some embodiments a protected enclave can be provided a trusted boot-type virtual machine. An advantage to a hardware device is that the device can be shipped to a customer and installed in a customer host in order to enable the customer to obtain desired functionality. In one example, the installing of such a device can enable the customer capacity to be virtualizable using the specified APIs and the resource provider network. In that case, the customer can have a machine that is owned by the customer and a PCI device or trusted co-processor that is owned by the resource provider. In various embodiments discussed herein, however, the host machine is owned by the resource provider and contained in the provider network, and the trusted co-processor is logically owned by the customer. Customers may also be more likely to trust hardware they can inspect than a process running in a remote environment, particularly where the trusted co-processor is completely separate from the virtual machine manager or the host operating system. 
     In an embodiment where dedicated hardware is not used, it can be desirable to perform a trusted boot of the hardware, including the virtual machine manager. This information then would also need to be communicated to customers in a way that they trust. A trusted measurement process can be used to verify the contents of the instance as well. A series of trusted interfaces can be provided to the guest as well, where the trust comes at least in part from measurement of the virtual machine manager, the BIOS, and other such aspects. Difficulty arises from the variability of the software at different levels. Any patches or updates to an application can cause the measurement to change, and changes to various software instances can come from any of a number of different sources. Capturing information for the entire chain and measuring all of those layers, as well as understanding what the measurements mean, can become very challenging, particularly when there can be thousands of valid combinations. With dedicated hardware such as a trusted co-processor or PCI device, however, there can be single-image deployment, such as a flat file or chunk, which can be strongly versioned. 
     As mentioned, the availability of a trusted, programmable device can provide for various functionality for customers in a resource provider environment. While PCI devices are discussed, USB and other types of devices can be used as well within the scope of the various embodiments. A driver might run in the host that the customer does not trust, as such software may be relatively easy to modify unexpectedly. It can be much tougher to maliciously modify a device connected to a PCI bus and operating at full PCI bus speed. Further, PCI devices provided to customers can have limited memory, which can limit the ability to maliciously modify operation of the device. For example, devices with substantial amounts of memory can store alternate versions of firmware, such as one for execution and one for read-only operations. A customer performing a checksum on the firmware in such a situation might get a valid and expected checksum based on the read-only version, but the other firmware would actually be the firmware executed. By constraining the memory not under control of the customer, the inclusion of alternate versions by unintended parties is significantly minimized. In some embodiments customers can provide their own implementations of drivers. A network interface can be forwarded into a guest, and the trusted co-processor then performs the networking. A block device can be forwarded into a guest, then the trusted co-processor manages the block storage. If a customer wants to write their own encryption driver or their own networking driver, the customer can be allowed to forward devices into a guest, run their driver, and load keys into the trusted enclave that the customer owns and have measured from the ground up. The resource provider in such an example would never see, or have access to, the key. 
       FIG. 5  illustrates an example process  500  for enabling a customer to utilize a trusted enclave that can be utilized in accordance with various embodiments. It should be understood that, for any process discussed herein, there can be additional, fewer, or alternative steps, performed in similar or different orders, or in parallel, within the scope of the various embodiments unless otherwise stated. In this example, a trusted co-processor is provided  502  (i.e., installed, configured, or allocated) on a trusted interface of a host computing device. In some embodiments, this can include installing a peripheral card, such as a PCI card or USB stick, into an appropriate port or slot on a server or other host computing device. The host computing fleet with the trusted co-processor can be added  504  to the fleet of resources available for use by, or on behalf of, various customers. This can include, for example, installing the device in a rack in a data center and connecting the server to a network of resources in a resource provider environment. The device can be in communication with a resource manager, or other such system or service, configured to allocate resources for performing various customer tasks. 
     A request can subsequently be received  506  from (or on behalf of) a customer of the resource provider, where the request involves access to at least one host computing device that includes a trusted co-processor or other trusted enclave as discussed or suggested herein. In response to the request, a host computing device with a trusted co-processor (an available capacity, appropriate configuration, etc.) can be allocated (partially or completely) for use by that customer, at least for a determined period of time. The customer can be provided with address information for communicating directly with the device, or routing information can be updated for the network, among other such options. In addition to providing access to the resource, the customer can be provided  508  with “expected” information pertaining to the trusted co-processor and/or host computing device. This can include, for example, indication of a cryptographic key stored by the host and known by the customer, such as an endorsement key burned into a trusted platform module (TPM) of the peripheral device. The expected information can also include one or more expected measurement values for the configuration of the peripheral device, as may include checksums or hashes of various memory ranges, firmware, or other such information as discussed elsewhere herein. At least one API can also be exposed  510  that enables the customer to submit queries to the trusted co-processor, which can involve various observational or interrogational tasks. 
     Subsequently, a request for verification can be received  512  from, or on behalf of, the customer. This can be received to the exposed API of the trusted co-processor in at least some embodiments. In response to the request, the customer can be provided  514  with verification data from the host device. This verification data can correspond to the type of expected information that was previously provided. For example, if a hash value was given for a specific memory range then the verification data can include a hash value corresponding to the information currently in those memory ranges. A digital signature or other such information can be provided as well as discussed herein. A determination can be made  518  as to whether the verification passes, such as by the verification information verifiable or otherwise as expected. If so, logical ownership of the trusted co-processor and/or peripheral device can be transferred  520  over to the customer, even though the card is installed in a host computing device owned by the provider. The customer can thus be enabled  522  to take over control (i.e., root level control) of the trusted co-processor, which in some cases can include locking out the provider from accessing the trusted co-processor. If the information cannot be verified (cryptographically or otherwise), or is determine to be different than expected, another host can be selected and verification information provided until the customer accepts one of the peripheral cards as a trusted appliance. It should be understood, however, that in some embodiments a customer may take logical ownership and/or control of the trusted co-processor before performing the measurement, which can prevent any unintended modification from occurring between the verification and the transfer of control in at least some embodiments. 
     Once the customer has taken ownership of the peripheral device, the customer can use that device as an observation point to verify operation, configuration, state, or other such information for the provider-owned host.  FIG. 6  illustrates an example process  600  that can be utilized in accordance with various embodiments. In this example, logical ownership of the trusted co-processor, peripheral card, or other trusted enclave is transferred  602  to the customer from the resource provider. As mentioned, in at least some embodiments the provider and any other entity will be locked out from the trusted enclave, and the trusted enclave can provide limited access into the host computing device in which the enclave exists. A verification request can be received  604  from the customer, such as to an API associated with the trusted co-processor on the peripheral device. The request can be analyzed to determine  606  at least one observational task to be performed on the host computing device, which can be used for verification or other such purposes. Assuming the customer is authorized to have the task performed and receive the results, as may be determined using various authorization mechanisms discussed or suggested herein, the observational task can be performed  608  in order to determine the appropriate verification data. This can include, for example, a hash of data contained in various memory ranges or registers, information about processes running on the host, information about guests running on the host and associations of those guests, etc. The verification data can also provide information about particular binaries or libraries that are located, information about whether such loaded binaries or libraries are intact, process table data, lists of logged in users, and peripherals identified in a configuration register, among other such verifiable data. The verification data can then be provided  610  to the customer for use in verifying the state, configuration, operation, or other aspect(s) of the host computing device. 
     At a subsequent point in time, the customer may want to perform an observation and/or verification action in order to ensure that the operational state is as expected. Accordingly, a subsequent request can be received  612  to the trusted co-processor, such as to an API of a PCI card, to perform one or more observational tasks. As mentioned, the trusted co-processor can function as an observation point in the provider host machine that is logically owned by the customer, and is on a fast bus that makes it more difficult to falsify the information obtained. This can include, for example, hashes of memory ranges or software code, among other such options discussed and suggested herein. If the verification data matches the expected data values, the operation can continue with the customer confident that the host, software, and other components (to the extent determinable) have not been compromised or otherwise unexpectedly modified. If it is determined  614  that the verification does not pass, such as by the verification data not being as expected, one or more responsive actions can be taken  616 . These can include, for example, notifying the customer of the failed verification, ceasing customer operations on the host, modifying customer operations, disconnecting the host from the network, deleting one or more previously executing processes or functions, contacting the provider, etc. Some systems may allow a customer to force information, data, state, or configuration back to an expected or intended value, or potentially other verifiable values. In some embodiments the configuration of components such as drives on a host can be analyzed, and if parameters such as the encryption parameters have changed then a customer can potentially override those values with the expected values and/or other appropriate values, depending at least in part upon the system. In some embodiments slight variations might be allowed, and in some embodiments the responsive action may depend at least in part upon the type of difference experienced or amount of the difference. As mentioned, the customer visibility to the host may be limited, but the customer should still be able to verify the appropriate information on the host, even when that host is owned by a provider and located in a provider environment, or other such location inaccessible to the customer. 
     As mentioned, the trusted co-processor can be logically owned by the customer, but physically owned and operated by the provider, with the customer being able to lock the provider from accessing at least some functionality on the trusted co-processor. The trusted co-processor can take the form of a peripheral card that is shipped in the host machine, and physically bolted or otherwise connected inside the host. The level of visibility is thus limited in at least some embodiments to the host in which the card is installed. As mentioned elsewhere herein, such functionality can be provided without dedicated hardware. Such approaches can use various software mechanisms to provide a trusted customer enclave inside a provider host machine as discussed elsewhere herein. 
     In some embodiments, the trusted enclave may not be provisioned with verifiable credentials such as a certificate. For these situations, a specific provisioning process can be utilized where physical access to the host can be used instead. In some embodiments the attesting of a remote system can require special privileges. Certified credentials for the requesting entity can be established, where possible, and used to validate the authenticity of the attestation request. Most notably, a certified asymmetric key pair can be used and the attestation request signed. In some embodiments, neither the host identity nor the virtual image identifier need to be known for attestation. The host identity does not necessarily need to be known at the time of attestation request generation, and the virtual image identifier can be reported back as part of the attestation quote, whereby the acceptable values can be determined. 
     In at least some embodiments the quote does not need to be restricted to the defined quote mechanism and may include additionally signed information using the identity key of the system. The host attestation response can include sufficient information to determine the authenticity of multiple virtual images. This can be performed in some embodiments by returning multiple quotes, or combining multiple results into a single quote. Further, while a two system hierarchy is presented for purposes of explanation, it should be stated that any appropriate tree or hierarchical structure with a parent node of the certificate authority can be used as well within the scope of the various embodiments, wherein each node in the tree is signed by the parent node. Similarly there can be any number of root certificate authorities in at least some embodiments. In one such system, the trusted co-processor forms a root for a family of subsystems contained within the single host. 
       FIG. 9  illustrates a logical arrangement of a set of general components of an example computing device  900 . In this example, the device includes a processor  902  for executing instructions that can be stored in a memory device or element  904 . As would be apparent to one of ordinary skill in the art, the device can include many types of memory, data storage, or non-transitory computer-readable storage media, such as a first data storage for program instructions for execution by the processor  902 , a separate storage for images or data, a removable memory for sharing information with other devices, etc. The device typically will include some type of display element  906 , such as a touch screen or liquid crystal display (LCD), although devices such as portable media players might convey information via other means, such as through audio speakers. As discussed, the device in many embodiments will include at least one input element  908  able to receive conventional input from a user. This conventional input can include, for example, a push button, touch pad, touch screen, wheel, joystick, keyboard, mouse, keypad, or any other such device or element whereby a user can input a command to the device. In some embodiments, however, such a device might not include any buttons at all, and might be controlled only through a combination of visual and audio commands, such that a user can control the device without having to be in contact with the device. In some embodiments, the computing device  900  of  FIG. 9  can include one or more network interface elements  908  for communicating over various networks, such as a Wi-Fi, Bluetooth, RF, wired, or wireless communication systems. The device in many embodiments can communicate with a network, such as the Internet, and may be able to communicate with other such devices. 
     Example environments discussed herein for implementing aspects in accordance with various embodiments are primarily Web-based, as relate to Web services and cloud computing, but it should be appreciated that, although a Web-based environment is used for purposes of explanation, different environments may be used, as appropriate, to implement various embodiments. Client devices used to interact with various embodiments can include any appropriate device operable to send and receive requests, messages, or information over an appropriate network and convey information back to a user of the device. Examples of such client devices include personal computers, smart phones, handheld messaging devices, laptop computers, set-top boxes, personal data assistants, electronic book readers, and the like. The network can include any appropriate network, including an intranet, the Internet, a cellular network, a local area network, or any other such network or combination thereof. Components used for such a system can depend at least in part upon the type of network and/or environment selected. Protocols and components for communicating via such a network are well known and will not be discussed herein in detail. Communication over the network can be enabled by wired or wireless connections, and combinations thereof. 
     It should be understood that there can be several application servers, layers, or other elements, processes, or components, which may be chained or otherwise configured, which can interact to perform tasks as discussed and suggested herein. As used herein the term “data store” refers to any device or combination of devices capable of storing, accessing, and retrieving data, which may include any combination and number of data servers, databases, data storage devices, and data storage media, in any standard, distributed, or clustered environment. The application server can include any appropriate hardware and software for integrating with the data store as needed to execute aspects of one or more applications for the client device, handling a majority of the data access and business logic for an application. The application server provides access control services in cooperation with the data store, and is able to generate content such as text, graphics, audio, and/or video to be transferred to the user, which may be served to the user by the Web server in the form of HTML, XML, or another appropriate structured language in this example. The handling of all requests and responses, as well as the delivery of content between a client device and a resource, can be handled by the Web server. It should be understood that the Web and application servers are not required and are merely example components, as structured code discussed herein can be executed on any appropriate device or host machine as discussed elsewhere herein. 
     A data store can include several separate data tables, databases, or other data storage mechanisms and media for storing data relating to a particular aspect. The data store is operable, through logic associated therewith, to receive instructions from a server, and obtain, update, or otherwise process data in response thereto. In one example, a user might submit a search request for a certain type of item. In this case, the data store might access the user information to verify the identity of the user, and can access the catalog detail information to obtain information about items of that type. The information then can be returned to the user, such as in a results listing on a Web page that the user is able to view via a browser on the user device. Information for a particular item of interest can be viewed in a dedicated page or window of the browser. 
     Each server typically will include an operating system that provides executable program instructions for the general administration and operation of that server, and typically will include a non-transitory computer-readable medium storing instructions that, when executed by a processor of the server, allow the server to perform its intended functions. Suitable implementations for the operating system and general functionality of the servers are known or commercially available, and are readily implemented by persons having ordinary skill in the art, particularly in light of the disclosure herein. 
     The environment in one embodiment is a distributed computing environment utilizing several computer systems and components that are interconnected via communication links, using one or more computer networks or direct connections. However, it will be appreciated by those of ordinary skill in the art that such a system could operate equally well in a system having fewer or a greater number of components than are described. Thus, the depictions of various systems and services herein should be taken as being illustrative in nature, and not limiting to the scope of the disclosure. 
     Various aspects can be implemented as part of at least one service or Web service, such as may be part of a service-oriented architecture. Services such as Web services can communicate using any appropriate type of messaging, such as by using messages in extensible markup language (XML) format and exchanged using an appropriate protocol such as SOAP (derived from the “Simple Object Access Protocol”). Processes provided or executed by such services can be written in any appropriate language, such as the Web Services Description Language (WSDL). Using a language such as WSDL allows for functionality such as the automated generation of client-side code in various SOAP frameworks. 
     Most embodiments utilize at least one network that would be familiar to those skilled in the art for supporting communications using any of a variety of commercially-available protocols, such as TCP/IP, FTP, UPnP, NFS, and CIFS. The network can be, for example, a local area network, a wide-area network, a virtual private network, the Internet, an intranet, an extranet, a public switched telephone network, an infrared network, a wireless network, and any combination thereof. 
     In embodiments utilizing a Web server, the Web server can run any of a variety of server or mid-tier applications, including HTTP servers, FTP servers, CGI servers, data servers, Java servers, and business application servers. The server(s) also may be capable of executing programs or scripts in response requests from user devices, such as by executing one or more Web applications that may be implemented as one or more scripts or programs written in any programming language, such as Java®, C, C #or C++, or any scripting language, such as Perl, Python, or TCL, as well as combinations thereof. The server(s) may also include database servers, including without limitation those commercially available from Oracle®, Microsoft®, Sybase®, and IBM®. 
     The environment can include a variety of data stores and other memory and storage media as discussed above. These can reside in a variety of locations, such as on a storage medium local to (and/or resident in) one or more of the computers or remote from any or all of the computers across the network. In a particular set of embodiments, the information may reside in a storage-area network (“SAN”) familiar to those skilled in the art. Similarly, any necessary files for performing the functions attributed to the computers, servers, or other network devices may be stored locally and/or remotely, as appropriate. Where a system includes computerized devices, each such device can include hardware elements that may be electrically coupled via a bus, the elements including, for example, at least one central processing unit (CPU), at least one input device (e.g., a mouse, keyboard, controller, touch screen, or keypad), and at least one output device (e.g., a display device, printer, or speaker). Such a system may also include one or more storage devices, such as disk drives, optical storage devices, and solid-state storage devices such as random access memory (“RAM”) or read-only memory (“ROM”), as well as removable media devices, memory cards, flash cards, etc. 
     Such devices also can include a computer-readable storage media reader, a communications device (e.g., a modem, a network card (wireless or wired), an infrared communication device, etc.), and working memory as described above. The computer-readable storage media reader can be connected with, or configured to receive, a computer-readable storage medium, representing remote, local, fixed, and/or removable storage devices as well as storage media for temporarily and/or more permanently containing, storing, transmitting, and retrieving computer-readable information. The system and various devices also typically will include a number of software applications, modules, services, or other elements located within at least one working memory device, including an operating system and application programs, such as a client application or Web browser. It should be appreciated that alternate embodiments may have numerous variations from that described above. For example, customized hardware might also be used and/or particular elements might be implemented in hardware, software (including portable software, such as applets), or both. Further, connection to other computing devices such as network input/output devices may be employed. 
     Storage media and other non-transitory computer readable media for containing code, or portions of code, can include any appropriate media known or used in the art, including storage media and communication media, such as but not limited to volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules, or other data, including RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by the a system device. Based on the disclosure and teachings provided herein, a person of ordinary skill in the art will appreciate other ways and/or methods to implement the various embodiments. 
     The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense. It will, however, be evident that various modifications and changes may be made thereunto without departing from the broader spirit and scope of the invention as set forth in the claims.