APPARATUS, METHOD, AND SYSTEM FOR SCHEDULING APPLICATION UNITS UTILIZING TRUSTED EXECUTION ENVIRONMENTS IN COMPUTING CLUSTERS

A method, system, and apparatus for deploying application units within a computing cluster is disclosed. The apparatus includes memory circuitry, machine-readable instructions, and processor circuitry configured to identify a plurality of worker nodes, each with a hardware-based security resource. The apparatus receives deployment requests specifying security requirements, selects compatible worker nodes based on these requirements, and schedules the application units for execution on the selected nodes.

BACKGROUND

In modern distributed computing environments, deploying application units such as pods and containers requires precise scheduling to ensure security and efficiency. Traditional scheduling mechanisms do not incorporate hardware-based security resources like Trusted Execution Environments (TEEs), leading to potential security vulnerabilities and inefficient resource utilization. Existing systems typically rely on simple labeling, which does not account for the dynamic nature of resource availability and specific security requirements of applications. Therefore, there may be a desire for a fine-grained scheduling apparatus that dynamically tracks and manages security resources across a cluster.

DETAILED DESCRIPTION

Specific details are set forth in the following description, but examples of the technologies described herein may be practiced without these specific details. Well-known circuits, structures, and techniques have not been shown in detail to avoid obscuring an understanding of this description. “An example/example,” “various examples/examples,” “some examples/examples,” and the like may include features, structures, or characteristics, but not every example necessarily includes the particular features, structures, or characteristics.

Some examples may have some, all, or none of the features described for other examples. “First,” “second,” “third,” and the like describe a common element and indicate different instances of like elements being referred to. Such adjectives do not imply that the described element item must be in a given sequence, either temporally or spatially, in ranking, or in any other manner. “Connected” may indicate elements are in direct physical or electrical contact with each other, and “coupled” may indicate elements cooperate or interact with each other, but they may or may not be in direct physical or electrical contact.

As used herein, the terms “operating,” “executing,” or “running” as they pertain to software or firmware in relation to a system, device, platform, or resource are used interchangeably and can refer to software or firmware stored in one or more computer-readable storage media accessible by the system, device, platform, or resource, even though the instructions contained in the software or firmware are not actively being executed by the system, device, platform, or resource.

The description may use the phrases “in an example/example,” “in examples/examples,” “in some examples/examples,” and/or “in various examples/examples,” each of which may refer to one or more of the same or different examples. Furthermore, the terms “comprising,” “including,” “having,” and the like, as used with respect to examples of the present disclosure, are synonymous.

It should be noted that the example schemes disclosed herein are applicable for/with any operating system and a reference to a specific operating system in this disclosure is merely an example, not a limitation.

FIG.1illustrates a block diagram of an apparatus100or device100example. Apparatus100comprises circuitry configured to provide the functionality of the apparatus100. For example, apparatus100ofFIG.1comprises interface circuitry40, processing circuitry30, (optional) storage circuitry20, and machine-readable instructions20a. For example, the processing circuitry30may be coupled with the interface circuitry40and optionally with the storage circuitry20.

For example, the processing circuitry30may be configured to provide the functionality of the apparatus100in conjunction with the interface circuitry40. For example, the interface circuitry40may be configured to exchange information, e.g., with other components inside or outside the apparatus100and the storage circuitry20. Likewise, the device100may comprise means that is/are configured to provide the functionality of the device100.

The components of the device100are defined as component means, which may correspond to, or be implemented by, the respective structural components of the apparatus100. For example, device100ofFIG.1comprises means for processing30, which may correspond to or be implemented by the processing circuitry30, means for communicating40, which may correspond to or be implemented by the interface circuitry40, and (optional) means for storing information20, which may correspond to or be implemented by the storage circuitry20. In the following, the functionality of the device100is illustrated with respect to the apparatus100. Features described in connection with the apparatus100may thus likewise be applied to the corresponding device100.

In general, the functionality of the processing circuitry30or means for processing30may be implemented by the processing circuitry30or means for processing30executing machine-readable instructions. Accordingly, any feature ascribed to the processing circuitry30or means for processing30may be defined by one or more instructions of a plurality of machine-readable instructions. The apparatus100or device100may comprise the machine-readable instructions, e.g., within the storage circuitry20or means for storing information140.

The interface circuitry40or means for communicating40may correspond to one or more inputs and/or outputs for receiving and/or transmitting information, which may be in digital (bit) values according to a specified code, within a module, between modules or between modules of different entities. For example, the interface circuitry40or means for communicating120may comprise circuitry configured to receive and/or transmit information.

For example, the processing circuitry30or means for processing30may be implemented using one or more processing units, one or more processing devices, or any means for processing, such as a processor, a computer, or a programmable hardware component being operable with accordingly adapted software. In other words, the described function of the processing circuitry30or means for processing30may be implemented in software, which is then executed on one or more programmable hardware components. Such hardware components may comprise a general-purpose processor, a Digital Signal Processor (DSP), a microcontroller, etc.

For example, the storage circuitry20or means for storing information20may comprise at least one element of the group of a computer-readable storage medium, such as a magnetic or optical storage medium, e.g., a hard disk drive, a flash memory, Floppy-Disk, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read Only Memory (PROM), Erasable Programmable Read Only Memory (EPROM), an Electronically Erasable Programmable Read Only Memory (EEPROM), or a network storage. For example, the storage circuitry20may store a (UEFI) BIOS.

The memory circuitry20may be a non-transitory, computer-readable medium comprising a program code20athat, when the program code20ais executed on a processor, a computer, or a programmable hardware component30, causes the processor, computer, or programmable hardware component30to perform the embodiments disclosed herein.

The processing circuitry30may be configured to identify a plurality of worker nodes within a cluster, where each worker node comprises a hardware-based security resource. Processing circuitry30may further receive a deployment request for an application deployment unit, wherein the deployment request includes a security requirement for a type of hardware-based security resource. Then, processing circuitry30may select, based on the security requirement, a compatible worker node of the plurality of worker nodes, wherein the compatible worker node comprises the type of hardware-based security resource. Finally, processing circuitry30may schedule the application deployment unit for execution on the compatible worker node.

A scheduling apparatus may comprise memory circuitry, machine-readable instructions, and processor circuitry configured to perform scheduling tasks. A worker node may be a computing node within a cluster that can execute application deployment units. An application deployment unit may be a software package ready for deployment, such as a pod or container.

The hardware-based security resource may be a trusted execution environment (TEE). A hardware-based security resource may be a physical component that provides security functionalities, such as a TEE. A TEE may be a secure processor area that ensures the integrity and confidentiality of code and data loaded inside. The scheduling apparatus ofFIG.1may ensure that applications requiring specific security features are only deployed on compatible nodes, thereby enhancing the security of the deployed applications by leveraging available hardware-based security resources. Using a TEE as the hardware-based security resource ensures that the deployment takes advantage of the highest level of hardware security available.

A cluster may comprise a plurality of types of TEEs. A cluster is a collection of interconnected computers, known as nodes, that work together as a single system to provide high availability, scalability, and performance. Each node of the cluster may have one or more types of TEEs. In computing and data processing, a cluster is designed to distribute workloads across multiple machines to improve efficiency, reliability, and redundancy. There are various types of TEEs, including SGX, SEV, TrustZone, TDX, CCA, etc. SGX (Software Guard Extensions) is a set of security-related instruction codes built into some modern CPUs, providing enclaves for secure computation. SEV (Secure Encrypted Virtualization) is a feature of some processors that encrypts virtual machines' memory to protect against unauthorized access. TrustZone is a security extension integrated into some processors that creates an isolated secure world alongside the normal execution environment. TDX (Trusted Domain Extensions) is an extension of virtualization technology that provides hardware-enforced confidentiality and integrity protections for virtual machines. CCA (Confidential Compute Architecture) is an architecture designed to secure sensitive data and code execution within a hardware-isolated environment on certain processors.

The application deployment unit may be a pod or a container. This ensures that the scheduling apparatus can handle different application deployment units, providing versatility in deploying a wide range of applications. Pod or container orchestration systems are platforms designed to automate containers' deployment, scaling, management, and networking across clusters of machines. These systems ensure efficient resource utilization, high availability, and fault tolerance by scheduling containers to run on suitable nodes based on resource requirements and availability. They provide features like service discovery, load balancing, automated rollouts and rollbacks, and storage orchestration. By abstracting the underlying infrastructure, container orchestration systems enable developers to focus on application development while maintaining consistent and reliable operations in diverse computing environments. Examples include Kubernetes (K8s), Docker Swarm, Apache Mesos, and HashiCorp Nomad.

A pod is a lightweight, portable, and self-sufficient computing environment that encapsulates one or more applications and their dependencies. It provides a consistent runtime environment, ensuring applications run reliably across different computing environments. Within a pod, containers share the same network namespace and storage resources, allowing them to communicate easily and share data.

A container is a standardized unit of software that packages up code and all its dependencies so the application runs quickly and reliably from one computing environment to another. Containers and the host system are isolated from one another, ensuring each container operates in its own environment with its own file system, CPU, memory, and process space. This isolation provides consistency, security, and portability across different infrastructure environments, whether on-premises or in the cloud.

Certain platforms, like K8s, automate the deployment, scaling, and management of containerized applications. They manage clusters of virtual machines and schedule containers to run on those clusters based on their available computing resources and the application's requirements.

The examples disclosed herein address the pod or container scheduling in a cluster that relies on hardware-based TEE resources. The present disclosure treats hardware-based TEE info in each node as security resources and dynamically tracks those resources. Pods or containers may be scheduled with TEE-related requirements with an enhanced scheduler. The TEE resources may be dynamically tracked or monitored by the control plane or notified by the scheduler in the cluster. Then, these security resources could be scheduled even with quality of service (QOS) purposes.

The scheduler apparatus100receives worker node data from the plurality of worker nodes within the cluster. Worker node data is information about each worker node, including its available resources and capabilities. This enables the apparatus to gather detailed information from each worker node, which can be used to make informed and fine-grained scheduling decisions.

The Apparatus100discloses a fine-grained scheduler based on collected TEE resources. The collected hardware-TEE resources are utilized in the scheduler to enhance the scheduling; thus, the pods or containers may be scheduled accurately. With this approach, the failure of pods or containers to start at the destination or working node may be prevented.

Schedulers are challenged when integrating nodes with hardware-based tee resources into the cluster because the security components are not treated as schedulable or first-class citizen resources for scheduling. The cluster's orchestration components or control planes will not treat hardware TEE resources in each node as resources. So, the current scheduler based on hardware-based TEE resources is very simple. For example, in K8s, there are two main solutions. First, a label-based scheduler, where the K8s labels each node with SGX or TDX label for the node, then schedules the pods or containers according to a user's YAML file (a configuration file). YAML (YAML Ain′t Markup Language) is a human-readable data serialization format commonly used to configure files and exchange between languages with different data structures. YAML is designed to be easy to read and write, making it ideal for configuration management, data interchange, and application storage. However, any machine or human-readable format can be used for a deployment request, allowing flexibility in specifying application requirements and configurations. And second, a scheduler based on extended resources.

However, K8s can use a device plugin framework to manage the device numbers in each host and then schedule the pods or containers according to the explicit TEE resource description. The resource tracking for the TEE resources is not accurate. Because the resource report is one-time and static, this will not correctly guide the scheduler. It may fail to schedule the pods due to the unchanged old info. Because the pods or containers will need additional security resources to start instead of only device numbers. For example, EPC (enclave page cache) is required for SGX; available TDX keys and memories are necessary for starting pods with TDX-protected virtual machines (VMs, i.e., TD-VMs).

Usually, hardware-based TEE-related components are not treated as security resources. Currently, in the popular project K8s, there are two kinds of schedulers: First, a label-based scheduler. This means that the scheduler can schedule the pods or containers to a node with the predefined label on that node. For example, if a node is labeled with SGX/TDX info, the pods with the required label in their YAML files will be scheduled to those nodes.

Second, a scheduler based on extended resources. Each node can have some extended resource info. Then, the scheduler receives the request to schedule the pods with the extended resource in the YAML files, which will be scheduled with the extended resources. For example, if there is 1 SGX device request in the YAML file, the scheduler will schedule the pods into the node with SGX devices discovered previously by the device plugin work.

The existing two schedulers in K8s-managed distributed clusters are inaccurate. The scheduler is not aware of the TEE-based security resource. So, pod scheduling is quite simple and coarse-grained. A pod with containers scheduled into a destination host may not start due to a shortage of resources. Then, there will be many paused pods, and rescheduling is needed. Even rescheduling cannot address this issue.

The examples disclosed herein address the scheduling of application deployment units within a cluster (e.g., pods or containers), which require hardware-based TEE resources. To do this, the system should first treat hardware-based TEE info in each node as security resources to start pods or containers and these resources. Those resources will be dynamically tracked/monitored by the control plane or notified by the scheduler in the cluster. Then, these security resources could be scheduled even with QoS purposes.

And second, the system should employ a fine-grained scheduler based on collected TEE resources. The disclosed embodiments utilize the collected hardware-based TEE resources in the scheduler and enhance the scheduling. Thus, the pods or containers may be scheduled accurately. The disclosed approach can prevent the failure of pods to start at the destination node.

The hardware-based TEE technique (e.g., SGX/TDX) may be fully exploited with the example schemes disclosed herein in cluster usage. With the disclosed example schemes, the multiple tenancy usage based on SGX/TDX may be explored well, and the techniques disclosed herein may continue to be promoted. There is a service to remotely verify and assert the trustworthiness of computing assets (TEEs, devices, Root of Trusts). For example, the hardware-based TEE-related resources in each node may be collected and used for scheduler usage. The disclosed examples may provide the TEE or host-related attestation and each node's dynamic TEE resource change to the orchestrator.

Users may manage the TEE resource in a fine-grained manner. The disclosed embodiments may efficiently schedule the pods or containers and then efficiently use the nodes with TEE resources. They can decrease their TCO.

In the disclosed examples, hardware-based TEE components and related resources from each node in the cluster may be treated as security resources, which may be managed by the control plane in the cluster and aware by the scheduler in the cluster.

Worker node data may include the type of hardware-based security resource, a trusted memory size, the number of devices, and the number of supported keys. Trusted memory size may be the amount of secure memory available on a node. The number of devices may be the count of hardware security modules or processors with TEEs. The number of supported keys may be the number of cryptographic keys that the security resource can manage. This embodiment allows for more detailed and granular data collection, ensuring the scheduler has all the necessary information to optimize security and resource allocation.

FIG.2shows diagram200of an example fine-grained scheduler for a hardware-based TEE. The main idea may be composed of the following two parts:

First, the embodiments are described for dynamically tracking and monitoring hardware TEE resources. As shown inFIG.2, the TEE resource in each working node220-A,220-B,220-C may be managed by the TEE resource management module222in each node, and it should be reported to the node resource manager212of the scheduler apparatus or scheduling node210. An example of a TEE resource is defined in Table 1. Table 1 shows an example of the definition of hardware TEE. The structure may be defined as 4 KB (or 1 KB, or 2 KB, or 8 KB, or 32 KB, etc.) or a similar size. This structure may be extended with more fields if different hardware TEEs are available.

TABLE 1/* These fields in this structure are used to track the TEE resource */Struct Tee_resources{/* This field is designed to store the TEE type. For example, value 0 may be used torepresent SGX; value 1 can present TDX; */Uint16_t TEE_type;/* This field is used to store the protected memory size in MB related with this TEEmodule. For example, if TEE_type is SGX, then the memory_size should beinterpreted as the EPC memory size. If the TEE_type is TDX, it should be interpretedas TD's trusted memory.*/Uint64_t trusted_memory_size;/*This field is used to track the total available device numbers in this platform */Uint32_t num_devces;/*This field defines the supported key numbers to protect the memories in thisplatform */Uin32_t supported_key_nums;/* Customized fields for a special TEE */Char custom_fields[1024];};

According to Table 1, a node's TEE resources may be reported to the node source manager in the scheduler node210. Moreover, each node has a TEE-based event monitor module to track the available TEE resources. This is one of the key innovations which can differentiate existing solutions. For example, if working node-A220-A inFIG.2has the following security TEE resources,<TEE_TYPE=SGX, trusted_memory_size=8 GB, num_devices=4, supported_key_nums=16, . . . >.

Then, the EPC size of the node is added with another 8 GB. Then, the node resource manager should have the following TEE info of the working node A220-A inFIG.2. For example,<TEE_TYPE=SGX, trusted_memory_size=16 GB, num_devices=4, supported_key_nums=16, . . . >.

This means that if there are dynamic changes in the TEE in each node, the node resource manager will always get the latest info from the corresponding TEE resource management module and adjust the scheduler. This means there will always be an event to promote the TEE resource change info from the working node to the scheduler apparatus210. Moreover, the working nodes220-A,220-B,220-C inFIG.2may be an Xeon host and an IPU. When a new working node is added to the cluster, the scheduler node may do an attestation.

The scheduler apparatus220may update the worker node data based on a change in the hardware-based security resource in each worker node. This ensures worker node data is continuously updated, maintaining the accuracy of the scheduler's decisions based on the most current resource availability.

FIG.3shows an improved scheduler based on the collected hardware-based TEE resource pool and users' request300. According to the embodiments disclosed herein, by defining TEE resources, the scheduler apparatus310can have the TEE resource of each node. Then, the scheduler can use such info to design a more fine-grained scheduler based on the TEE resources. Usually, the simplest algorithm may be matching the fields by order. For example, the apparatus can map the first field (i.e., TEE_EXAMPLE) and then use SQL-like language to select the destination node if Etcd-like databases are used. Etcd-like databases may refer to distributed, key-value stores designed to reliably store data across a cluster of machines, ensuring data consistency and availability. These databases are often used for configuration management, service discovery, and coordination in distributed systems. They provide strong consistency guarantees, allowing clients to read the most recent data written to any node in the cluster, and typically support features such as leader election, distributed locking, and watch mechanisms for real-time updates. After the accurate matching, the related resource numbers may be reduced.

For example, Table 2 shows example TEE resource info of each node.

FIG.3further shows a user request coming for the pod, with the following requested info defined in the YAML file,<TEE_TYPE=SGX, trusted_memory_size=16 GB, num_device=1>.

Without the scheduling disclosed herein, the pod may be scheduled into Node A with a 50% probability if we only leverage the labeled scheduler. But with the algorithms disclosed herein, it will be accurately scheduled into working Node C.

After starting the pods, the resources will be changed into Table 3. When the pods are destroyed, the resources will be converted to the value shown in Table 2. Table 3 shows an example of the TEE resource info of each node after the scheduling.

FIG.3shows a deployment request315in the form of a pod YAML file describing the pod requirements.

The requested info is: <TEE_TYPE=TDX, num_device=1, trusted_memory_size=2048M>. Then, the scheduler apparatus310already applies the schedule, leverages the request interpreted from the YAML file and provides it to the scheduler as described above. In subsequent steps, scheduler apparatus310may allocate the related resources successfully and create the containers with the TDX TEE protection.

According to another embodiment, the scheduler apparatus may receive security resource data from the compatible worker node after scheduling the application deployment unit. Security resource data may be information about the current state and usage of security resources on a node. Obtaining this data ensures that the scheduler can monitor the security resource usage after deployment, maintaining the application's security and performance.

According to another embodiment, the scheduler may reschedule the application deployment unit for execution on a second compatible worker node of the plurality of worker nodes when the security resource data no longer satisfies the security requirement. This may provide a mechanism for rescheduling applications if the security resources on a node change, ensuring continuous compliance with security requirements.

When the scheduler selects a working node, the node accepts the request via the node request handle module224, as shown inFIG.2. If there is a TEE resource change after the scheduler selects this node. If the node does not have enough resources to execute the application deployment unit, it will notify the scheduler immediately for rescheduling purposes. This may eliminate the failure to start an application deployment unit or pod.

FIG.4shows an example of TEE resource change inside schedulers. The scheduler apparatus may further receive cluster data, wherein the cluster data includes an addition of a new worker node to the cluster and/or a removal of an existing worker node from the cluster. Cluster data may include information about the state of the cluster, including changes in the composition of worker nodes. This may allow the scheduler to adapt to changes in the cluster, such as adding or removing nodes, ensuring dynamic and flexible scheduling.

Adding or removing a working node can change the collected hardware TEE resources. When a working node with a TEE resource is added, then a new TEE resource with this node should be added to the node resource manager. This may be done by the worker node reporting itself to the cluster or scheduler node. When a working node is removed from the cluster, the related TEE resource node in the node resource manager should be removed. This may be done by either an active report by the departing worker node or a passive cluster survey. When there is a dynamic change of a TEE resource in a worker node, the worker node should report the change so that the TEE resource state changes in the node resource manager of the scheduler node.

When an application deployment unit (e.g., pod or container) with a TEE resource request is created in the scheduler, the resource state for a working node that will serve the unit is reduced. When the pod with a TEE resource request is destroyed, the resource state for the working node that served the unit is increased.

According to another embodiment, the deployment request may include a trusted memory size, a number of devices, and a number of supported keys. This may provide for the specification of detailed security requirements in the deployment request, allowing for a more precise matching of application needs with node capabilities.

According to another embodiment, a subset of the plurality of worker nodes may comprise a plurality of hardware-based security resources. This may allow for the deployment of applications on nodes with multiple hardware-based security resources or TEEs, enhancing the flexibility and robustness of the scheduling apparatus.

FIG.1further shows a worker apparatus100. The worker apparatus100may include memory circuitry, one or more hardware-based security resources, machine-readable instructions, and processor circuitry to execute the machine-readable instructions. The worker apparatus may provide security resource data for each of the one or more hardware-based security resources to a scheduling node and receive an application deployment unit for execution on a compatible hardware-based security resource of the one or more hardware-based security resources.

A worker apparatus may be a computing device within a cluster that provides security resources and executes scheduled applications. This may enable worker nodes to provide up-to-date security resource information to the scheduler, facilitating informed and accurate scheduling decisions.

A worker apparatus100may further provide updated hardware-based security resource data for each hardware-based security resource. This may ensure that worker nodes continuously update their security resource data, maintaining the accuracy of the scheduler's information.

A worker apparatus100may further determine whether the application deployment unit can execute in the compatible hardware-based security resource and notify the scheduling node when the application deployment unit cannot execute in the compatible hardware-based security resource. This may allow worker nodes to verify whether they can meet the security requirements of the application deployment unit, improving reliability and reducing deployment failures.

More details and aspects of the concept for offloading a workload may be described in connection with examples discussed below (e.g.,FIGS.5A-8).

FIGS.5A and5Bshow a flowchart of a method for a scheduler node and a worker node in a cluster.FIG.5Ashows method500for scheduling an application deployment unit on a compatible worker node within a cluster. The method may include identifying510, a plurality of worker nodes within a cluster, where each worker node comprises a hardware-based security resource. Method500may then include receiving520a deployment request for the application deployment unit, wherein the deployment request includes a security requirement for a type of hardware-based security resource;

Method500may then include selecting520, based on the security requirement, the compatible worker node of the plurality of worker nodes, wherein the compatible worker node comprises the type of hardware-based security resource. Finally, method500may include scheduling540, the application deployment unit, for execution on the compatible worker node.

Method500may include receiving505worker node data from the plurality of worker nodes within the cluster and updating507the worker node data based on a change of the hardware-based security resource in each worker node.

Method500may include receiving503cluster data, wherein the cluster data includes an addition of a new worker node to the cluster and/or a removal of an existing worker node from the cluster. Method500may include receiving543security resource data from the compatible worker node after scheduling the application deployment unit. Method500may include rescheduling545, the application deployment unit, for execution on a second compatible worker node of the plurality of worker nodes, when the security resource data no longer satisfies the security requirement.

FIG.5Bshows method550for executing an application deployment unit on a worker apparatus. Method550includes providing560security resource data for each of the one or more hardware-based security resources to a scheduling node, which receives570in an application deployment unit for execution on a compatible hardware-based security resource of the one or more hardware-based security resources.

The method550may include providing555updated hardware-based security resource data for each hardware-based security resource. Method500may further include determining575whether the application deployment unit can execute in the compatible hardware-based security resource and notifying577the scheduling node when the application deployment unit cannot execute in the compatible hardware-based security resource.

Method550may include notifying553the scheduling node when the worker apparatus is added to the cluster and/or when the worker apparatus leaves the cluster. Method550may include providing573security resource data after receiving the application deployment unit. Method550may further include executing580, the application deployment unit in the hardware-based security resource

The example schemes in this disclosure are to treat hardware-based TEE in each node as a security resource, and the pods or containers scheduler in the cluster may be aware of those hardware-based TEE resources accurately and dynamically. Then, a fine-grained scheduler may be provided based on the HW TEE resources according to the requirements of users' pods or containers, and even conduct the QoS on security resources (e.g., TEE resources). With the example schemes disclosed herein, the scheduler is much more efficient than the existing approaches (e.g., the label-based scheduler or extended resource scheduler in K8s). Promoting SGX/TDX-based TEE techniques in confidential container usage scenarios will be helpful.

More details and aspects of the concept for offloading a workload may be described in connection with examples discussed above (e.g.,FIGS.1-4) or below (e.g.,FIGS.6-8).

FIG.6is a block diagram of an electronic apparatus600incorporating at least one electronic assembly and/or method described herein. Electronic apparatus600is-merely one example of an electronic apparatus in which forms of the electronic assemblies and/or methods described herein may be used. Examples of an electronic apparatus600include but are not limited to, personal computers, tablet computers, mobile telephones, game devices, MP3 or other digital music players, etc. In this example, electronic apparatus600comprises a data processing system that includes a system bus602to couple the various components of the electronic apparatus600. System bus602provides communications links among the various components of the electronic apparatus600. It may be implemented as a single bus, a combination of buses, or in any other suitable manner.

An electronic assembly610as describe herein may be coupled to system bus602. The electronic assembly610may include any circuit or combination of circuits. In one embodiment, the electronic assembly610includes a processor612, which may be of any type. As used herein, “processor” means any type of computational circuit, such as but not limited to a microprocessor, a microcontroller, a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a graphics processor, a digital signal processor (DSP), multiple core processor, or any other type of processor or processing circuit.

Other types of circuits that may be included in electronic assembly610are a custom circuit, an application-specific integrated circuit (ASIC), or the like, such as, for example, one or more circuits (such as a communications circuit614) for use in wireless devices like mobile telephones, tablet computers, laptop computers, two-way radios, and similar electronic systems. The IC can perform any other type of function.

The electronic apparatus600may also include an external memory620, which in turn may include one or more memory elements suitable to the particular application, such as a main memory622in the form of random access memory (RAM), one or more hard drives624, and/or one or more drives that handle removable media626such as compact disks (CD), flash memory cards, digital video disk (DVD), and the like.

The electronic apparatus600may also include a display device616, one or more speakers618, and a keyboard and/or controller630, which can include a mouse, trackball, touch screen, voice-recognition device, or any other device that permits a system user to input information into and receive information from the electronic apparatus600.

More details and aspects of the concept for offloading a workload may be described in connection with examples discussed above (e.g.,FIGS.1-5B) or below (e.g.,FIGS.7-8).

FIG.7illustrates a computing device700in accordance with one implementation of the invention. The computing device700houses a board702. The board702may include a number of components, including but not limited to a processor704and at least one communication chip706. The processor704is physically and electrically coupled to the board702. In some implementations the at least one communication chip706is also physically and electrically coupled to the board702. In further implementations, the communication chip706is part of the processor704. Depending on its applications, computing device700may include other components that may or may not be physically and electrically coupled to the board702. These other components include but are not limited to, volatile memory (e.g., DRAM), non-volatile memory (e.g., ROM), flash memory, a graphics processor, a digital signal processor, a crypto processor, a chipset, an antenna, a display, a touchscreen display, a touchscreen controller, a battery, an audio codec, a video codec, a power amplifier, a global positioning system (GPS) device, a compass, an accelerometer, a gyroscope, a speaker, a camera, and a mass storage device (such as hard disk drive, compact disk (CD), digital versatile disk (DVD), and so forth). The communication chip706enables wireless communications for the transfer of data to and from the computing device700. The term “wireless” and its derivatives may be used to describe circuits, devices, systems, methods, techniques, communications channels, etc., that may communicate data through the use of modulated electromagnetic radiation through a non-solid medium. The term does not imply that the associated devices do not contain any wires, although, in some embodiments, they might not. The communication chip706may implement any of a number of wireless standards or protocols, including but not limited to Wi-Fi (IEEE 802.11 family), WiMAX (IEEE 802.16 family), IEEE 802.20, long term evolution (LTE), Ev-DO, HSPA+, HSDPA+, HSUPA+, EDGE, GSM, GPRS, CDMA, TDMA, DECT, Bluetooth, derivatives thereof, as well as any other wireless protocols that are designated as 3G, 4G, 5G, and beyond. The computing device700may include a plurality of communication chips706. For instance, a first communication chip706may be dedicated to shorter-range wireless communications such as Wi-Fi and Bluetooth, and a second communication chip706may be dedicated to longer-range wireless communications such as GPS, EDGE, GPRS, CDMA, WiMAX, LTE, Ev-DO, and others. The processor704of the computing device700includes an integrated circuit die packaged within the processor704. In some implementations of the invention, the integrated circuit die of the processor includes one or more devices that are assembled in an cPLB- or cWLB-based POP package that includes a mold layer directly contacting a substrate, in accordance with implementations of the invention. The term “processor” may refer to any device or portion of a device that processes electronic data from registers and/or memory to transform that electronic data into other electronic data that may be stored in registers and/or memory. The communication chip706also includes an integrated circuit die packaged within the communication chip706. In accordance with another implementation of the invention, the integrated circuit die of the communication chip includes one or more devices that are assembled in an ePLB- or eWLB-based POP package that includes a mold layer directly contacting a substrate, in accordance with implementations of the invention.

More details and aspects of the concept for offloading a workload may be described in connection with examples discussed above (e.g.,FIGS.1-6) or below (e.g.,FIG.8).

FIG.8shows an example of a higher-level device application for the disclosed embodiments. The MAA cantilevered heat pipe apparatus embodiments may be found in several parts of a computing system. In an embodiment, the MAA cantilevered heat pipe is part of a communications apparatus such as is affixed to a cellular communications tower. The MAA cantilevered heat pipe may also be referred to as an MAA apparatus. In an embodiment, a computing system2800includes but is not limited to, a desktop computer. In an embodiment, a system2800includes but is not limited to, a laptop computer. In an embodiment, a system2800includes but is not limited to, a netbook. In an embodiment, a system2800includes but is not limited to, a tablet. In an embodiment, a system2800includes but is not limited to, a notebook computer. In an embodiment, a system2800includes but is not limited to, a personal digital assistant (PDA). In an embodiment, a system2800includes but is not limited to, a server. In an embodiment, a system2800includes but is not limited to, a workstation. In an embodiment, a system2800includes but is not limited to, a cellular telephone. In an embodiment, a system2800includes but is not limited to, a mobile computing device. In an embodiment, a system2800includes but is not limited to, a smartphone. In an embodiment, a system2800includes but is not limited to, an internet appliance. Other types of computing devices may be configured with the microelectronic device that includes MAA apparatus embodiments.

In an embodiment, the processor2810has one or more processing cores2812and2812N, where2812N represents the Nth processor core inside processor2810where N is a positive integer. In an embodiment, the electronic device system2800using a MAA apparatus embodiment that includes multiple processors including2810and2805, where the processor2805has logic similar or identical to the logic of the processor2810. In an embodiment, the processing core2812includes, but is not limited to, pre-fetch logic to fetch instructions, decode logic to decode the instructions, execution logic to execute instructions, and the like. In an embodiment, the processor2810has a cache memory2816to cache at least one of instructions and data for the MAA apparatus in the system2800. The cache memory2816may be organized into a hierarchal structure, including one or more levels of cache memory.

In an embodiment, the processor2810includes a memory controller2814, which is operable to perform functions that enable the processor2810to access and communicate with memory2830, which includes at least one of a volatile memory2832and a non-volatile memory2834. In an embodiment, the processor2810is coupled with memory2830and chipset2820. The processor2810may also be coupled to a wireless antenna2878to communicate with any device configured to at least one of transmit and receive wireless signals. In an embodiment, the wireless antenna interface2878operates in accordance with but is not limited to, the IEEE 802.11 standard and its related family, Home Plug AV (HPAV), Ultra Wide Band (UWB), Bluetooth, WiMax, or any form of wireless communication protocol.

The memory2830stores information and instructions to be executed by the processor2810. In an embodiment, the memory2830may also store temporary variables or other intermediate information while the processor2810is executing instructions. In the illustrated embodiment, the chipset2820connects with processor2810via Point-to-Point (PtP or P-P) interfaces2817and2822. Either of these PtP embodiments may be achieved using an MAA apparatus embodiment as set forth in this disclosure. The chipset2820enables the processor2810to connect to other elements in the MAA apparatus embodiments in a system2800. In an embodiment, interfaces2817and2822operate in accordance with a PtP communication protocol such as the QuickPath Interconnect (QPI) or the like. In other embodiments, a different interconnect may be used.

In an embodiment, the chipset2820is operable to communicate with the processor2810,2805N, the display device2840, and other devices2872,2876,2874,2860,2862,2864,2866,2877, etc. The chipset2820may also be coupled to a wireless antenna2878to communicate with any device configured to at least do one of transmit and receive wireless signals.

The chipset2820connects to the display device2840via the interface2826. The display2840may be, for example, a liquid crystal display (LCD), a plasma display, a cathode ray tube (CRT) display, or any other form of visual display device. In an embodiment, the processor2810and the chipset2820are merged into an MAA apparatus in a system. Additionally, the chipset2820connects to one or more buses2850and2855that interconnect various elements2874,2860,2862,2864, and2866. Buses2850and2855may be interconnected together via a bus bridge2872such as at least one MAA apparatus embodiment. In an embodiment, the chipset2820couples with a non-volatile memory2860, a mass storage device(s)2862, a keyboard/mouse2864, and a network interface2866by way of at least one of the interface2824and2874, the smart TV2876, and the consumer electronics2877, etc.

While the modules shown inFIG.28are depicted as separate blocks within the MAA apparatus embodiment in a computing system2800, the functions performed by some of these blocks may be integrated within a single semiconductor circuit or may be implemented using two or more separate integrated circuits. For example, although cache memory2816is depicted as a separate block within processor2810, cache memory2816(or selected aspects of2816) may be incorporated into the processor core2812.

Where useful, the computing system2800may have a broadcasting structure interface such as for affixing the MAA apparatus to a cellular tower.

As used herein, the term “module” refers to logic that may be implemented in a hardware component or device, software or firmware running on a processing unit, or a combination thereof, to perform one or more operations consistent with the present disclosure. Software and firmware may be embodied as instructions and/or data stored on non-transitory computer-readable storage media. As used herein, the term “circuitry” can comprise, singly or in any combination, non-programmable (hardwired) circuitry, programmable circuitry such as processing units, state machine circuitry, and/or firmware that stores instructions executable by programmable circuitry. Modules described herein may, collectively or individually, be embodied as circuitry that forms a part of a computing system. Thus, any of the modules may be implemented as circuitry. A computing system referred to as being programmed to perform a method may be programmed to perform the method via software, hardware, firmware, or combinations thereof.

Any of the disclosed methods (or a portion thereof) may be implemented as computer-executable instructions or a computer program product. Such instructions can cause a computing system or one or more processing units capable of executing computer-executable instructions to perform any of the disclosed methods. As used herein, the term “computer” refers to any computing system or device described or mentioned herein. Thus, the term “computer-executable instruction” refers to instructions that may be executed by any computing system or device described or mentioned herein.

The computer-executable instructions or computer program products as well as any data created and/or used during implementation of the disclosed technologies may be stored on one or more tangible or non-transitory computer-readable storage media, such as volatile memory (e.g., DRAM, SRAM), non-volatile memory (e.g., flash memory, chalcogenide-based phase-change non-volatile memory) optical media discs (e.g., DVDs, CDs), and magnetic storage (e.g., magnetic tape storage, hard disk drives). Computer-readable storage media may be contained in computer-readable storage devices such as solid-state drives, USB flash drives, and memory modules. Alternatively, any of the methods disclosed herein (or a portion) thereof may be performed by hardware components comprising non-programmable circuitry. In some examples, any of the methods herein may be performed by a combination of non-programmable hardware components and one or more processing units executing computer-executable instructions stored on computer-readable storage media.

The computer-executable instructions may be part of, for example, an operating system of the computing system, an application stored locally to the computing system, or a remote application accessible to the computing system (e.g., via a web browser). Any of the methods described herein may be performed by computer-executable instructions performed by a single computing system or by one or more networked computing systems operating in a network environment. Computer-executable instructions and updates to the computer-executable instructions may be downloaded to a computing system from a remote server.

Further, it is to be understood that implementation of the disclosed technologies is not limited to any specific computer language or program. For instance, the disclosed technologies may be implemented by software written in C++, C#, Java, Perl, Python, JavaScript, Adobe Flash, C#, assembly language, or any other programming language. Likewise, the disclosed technologies are not limited to any particular computer system or type of hardware.

Furthermore, any of the software-based examples (comprising, for example, computer-executable instructions for causing a computer to perform any of the disclosed methods) may be uploaded, downloaded, or remotely accessed through a suitable communication means. Such suitable communication means include, for example, the Internet, the World Wide Web, an intranet, cable (including fiber optic cable), magnetic communications, electromagnetic communications (including RF, microwave, ultrasonic, and infrared communications), electronic communications, or other such communication means.

As used in this application and the claims, a list of items joined by the term “and/or” can mean any combination of the listed items. For example, the phrase “A, B and/or C” can mean A; B; C; A and B; A and C; B and C; or A, B and C. As used in this application and the claims, a list of items joined by the term “at least one of” can mean any combination of the listed terms. For example, the phrase “at least one of A, B or C” can mean A; B; C; A and B; A and C; B and C; or A, B, and C. Moreover, as used in this application and the claims, a list of items joined by the term “one or more of” can mean any combination of the listed terms. For example, the phrase “one or more of A, B and C” can mean A; B; C; A and B; A and C; B and C; or A, B, and C.

The disclosed methods, apparatuses, and systems are not to be construed as limiting in any way. Instead, the present disclosure is directed toward all novel and nonobvious features and aspects of the various disclosed examples, alone and in various combinations and sub-combinations with one another. The disclosed methods, apparatuses, and systems are not limited to any specific aspect or feature or combination thereof, nor do the disclosed examples require that any one or more specific advantages be present or problems be solved.

Although the operations of some of the disclosed methods are described in a particular, sequential order for convenient presentation, it is to be understood that this manner of description encompasses rearrangement, unless a particular ordering is required by specific language set forth herein. For example, operations described sequentially may in some cases be rearranged or performed concurrently. Moreover, for the sake of simplicity, the attached figures may not show the various ways in which the disclosed methods may be used in conjunction with other methods. More details and aspects of the concept for adapting a processor to a workload may be described in connection with examples discussed above (e.g.,FIGS.1-7).

An example (e.g., example 1) relates to a scheduler apparatus comprising memory circuitry, machine-readable instructions, and processor circuitry to execute the machine-readable instructions to: identify a plurality of worker nodes within a cluster, where each worker node comprises a hardware-based security resource, receive a deployment request for an application deployment unit, wherein the deployment request includes a security requirement for a type of hardware-based security resource; select, based on the security requirement, a compatible worker node of the plurality of worker nodes, wherein the compatible worker node comprises the type of hardware-based security resource; and schedule the application deployment unit for execution on the compatible worker node.

Another example (e.g., example 2) relates to a previously described example (e.g., example 1), wherein the hardware-based security resource is a trusted execution environment (TEE).

Another example (e.g., example 3) relates to a previously described example (e.g., example 2), wherein the cluster comprises a plurality of types of TEEs.

Another example (e.g., example 4) relates to a previously described example (e.g., one of the examples 1-3), wherein the application deployment unit is at least one of: a pod; and a container.

Another example (e.g., example 5) relates to a previously described example (e.g., one of the examples 1-4), further comprising machine-readable instructions to receive worker node data from the plurality of worker nodes within the cluster.

Another example (e.g., example 6) relates to a previously described example (e.g., example 5), wherein worker node data includes the type of hardware-based security resource, a trusted memory size, a number of devices, and a number of supported keys.

Another example (e.g., example 7) relates to a previously described example (e.g., example 5), further comprising machine-readable instructions to update the worker node data based on a change of the hardware-based security resource in each worker node.

Another example (e.g., example 8) relates to a previously described example (e.g., example 5), wherein further comprising machine-readable instructions to: receive cluster data, wherein the cluster data includes at least one of: an addition of a new worker node to the cluster; and a removal of an existing worker node from the cluster.

Another example (e.g., example 9) relates to a previously described example (e.g., one of the examples 1-8), wherein the deployment request further includes a trusted memory size, a number of devices, and a number of supported keys.

Another example (e.g., example 10) relates to a previously described example (e.g., one of the examples 1-9), further comprising machine-readable instructions to receive security resource data from the compatible worker node after scheduling the application deployment unit.

Another example (e.g., example 11) relates to a previously described example (e.g., example 10), further comprising machine-readable instructions to reschedule the application deployment unit for execution on a second compatible worker node of the plurality of worker nodes when the security resource data no longer satisfies the security requirement.

Another example (e.g., example 12) relates to a previously described example (e.g., one of the examples 1-11), wherein a subset of the plurality of worker nodes comprise a plurality of hardware-based security resources.

An example (e.g., example 13) relates to a worker apparatus within a cluster, the apparatus comprising memory circuitry, one or more hardware-based security resources, machine-readable instructions, and processor circuitry to execute the machine-readable instructions to: provide security resource data for each of the one or more hardware-based security resources to a scheduling node; and receive an application deployment unit for execution on a compatible hardware-based security resource of the one or more hardware-based security resources.

Another example (e.g., example 14) relates to a previously described example (e.g., example 13), further comprising machine-readable instructions to provide updated hardware-based security resource data for each of the hardware-based security resources.

Another example (e.g., example 15) relates to a previously described example (e.g., one of the examples 13-14), further comprising machine-readable instructions to: determine whether the application deployment unit can execute in the compatible hardware-based security resource; and notify the scheduling node when the application deployment unit cannot execute in the compatible hardware-based security resource.

Another example (e.g., example 16) relates to a previously described example (e.g., one of the examples 13-15), wherein each hardware-based security resource is a trusted execution environment (TEE).

Another example (e.g., example 17) relates to a previously described example (e.g., one of the examples 13-16), relates to a previously described example (e.g., example 2), wherein the cluster comprises a plurality of types of TEEs.

Another example (e.g., example 18) relates to a previously described example (e.g., one of the examples 13-17), wherein the application deployment unit is at least one of: a pod; and a container.

Another example (e.g., example 19) relates to a previously described example (e.g., examples 13-18), wherein security resource data includes the type of hardware-based security resource, a trusted memory size, a number of devices, and a number of supported keys.

Another example (e.g., example 20) relates to a previously described example (e.g., examples 13-19), further comprising machine-readable instructions to update the scheduling node when the worker apparatus is added to the cluster and/or when the worker apparatus leaves the cluster.

Another example (e.g., example 21) relates to a previously described example (e.g., one of the examples 13-20), further comprising machine-readable instructions to provide the security resource data after receiving the application deployment unit.

An example (e.g., example 22) relates to a system and/or a cluster comprising a scheduler apparatus and/or a scheduler node according to a previously described example (e.g., one of the examples 1-12) and a worker apparatus and/or worker node according to a previously described example (e.g., one of the examples 13-21).

Another example (e.g., example 23) relates to a method for scheduling an application deployment unit on a compatible worker node within a cluster, the method comprising: identifying a plurality of worker nodes within a cluster, where each worker node comprises a hardware-based security resource, receiving a deployment request for the application deployment unit, wherein the deployment request includes a security requirement for a type of hardware-based security resource; selecting, based on the security requirement, the compatible worker node of the plurality of worker nodes, wherein the compatible worker node comprises the type of hardware-based security resource; and scheduling the application deployment unit for execution on the compatible worker node.

Another example (e.g., example 24) relates to a previously described example (e.g., example 23), wherein the hardware-based security resource is a trusted execution environment (TEE).

Another example (e.g., example 25) relates to a previously described example (e.g., example 24), wherein the cluster comprises a plurality of types of TEEs.

Another example (e.g., example 26) relates to a previously described example (e.g., one of the examples 23-25), wherein the application deployment unit is at least one of: a pod; and a container.

Another example (e.g., example 27) relates to a previously described example (e.g., one of the examples 23-26), further comprising receiving worker node data from the plurality of worker nodes within the cluster.

Another example (e.g., example 28) relates to a previously described example (e.g., example 27), wherein worker node data includes the type of hardware-based security resource, a trusted memory size, a number of devices, and a number of supported keys.

Another example (e.g., example 29) relates to a previously described example (e.g., example 27), further comprising updating the worker node data based on a change of the hardware-based security resource in worker each node.

Another example (e.g., example 30) relates to a previously described example (e.g., example 27), wherein further receiving cluster data, wherein the cluster data includes at least one of: an addition of a new worker node to the cluster; and a removal of an existing worker node from the cluster.

Another example (e.g., example 31) relates to a previously described example (e.g., one of the examples 23-20), wherein the deployment request further includes a trusted memory size, a number of devices, and a number of supported keys.

Another example (e.g., example 32) relates to a previously described example (e.g., one of examples 23-31), further comprising receiving security resource data from the compatible worker node after scheduling of the application deployment unit.

Another example (e.g., example 33) relates to a previously described example (e.g., example 32), further comprising rescheduling the application deployment unit for execution on a second compatible worker node of the plurality of worker nodes when the security resource data no longer satisfies the security requirement.

Another example (e.g., example 34) relates to a previously described example (e.g., one of the examples 23-33), wherein a subset of the plurality of worker nodes comprise a plurality of hardware-based security resources.

An example (e.g., example 35) relates a method for executing an application deployment unit on a worker apparatus, the method comprising: providing security resource data for each of the one or more hardware-based security resources to a scheduling node; and receiving an application deployment unit for execution on a compatible hardware-based security resource of the one or more hardware-based security resources.

Another example (e.g., example 36) relates to a previously described example (e.g., example 35), further comprising providing updated hardware-based security resource data for each of the hardware-based security resources.

Another example (e.g., example 37) relates to a previously described example (e.g., one of the examples 35-36), further comprising determining whether the application deployment unit can execute in the compatible hardware-based security resource; and notifying the scheduling node when the application deployment unit cannot execute in the compatible hardware-based security resource.

Another example (e.g., example 38) relates to a previously described example (e.g., one of the examples 35-37), wherein each hardware-based security resource is a trusted execution environment (TEE).

Another example (e.g., example 39) relates to a previously described example (e.g., one of the examples 35-38), relates to a previously described example (e.g., example 2), wherein the cluster comprises a plurality of types of TEEs.

Another example (e.g., example 40) relates to a previously described example (e.g., one of the examples 35-39), wherein the application deployment unit is at least one of: a pod; and a container.

Another example (e.g., example 41) relates to a previously described example (e.g., examples 35-40), wherein security resource data includes the type of hardware-based security resource, a trusted memory size, a number of devices, and a number of supported keys.

Another example (e.g., example 42) relates to a previously described example (e.g., examples 35-41), further comprising notifying the scheduling node when the worker apparatus is added to the cluster and/or when the worker apparatus leaves the cluster.

Another example (e.g., example 43) relates to a previously described example (e.g., one of the examples 35-42), further comprising providing the security resource data after receiving the application deployment unit.

Another example (e.g., example 44) relates to a previously described example (e.g., one of the examples 35-43), further comprising executing the application deployment unit on the hardware-based security resource.

An example (e.g., example 45) relates to a method for a system and/or a cluster comprising a scheduler apparatus and/or a scheduler node performing the method according to a previously described example (e.g., one of the examples 23-34) and a worker apparatus and/or worker node according to a previously described example (e.g., one of the examples 35-44).

An example (example 46) relates to a non-transitory, computer-readable medium comprising program code that, when the program code is executed on a processor, a computer, or a programmable hardware component, causes the processor, computer, or programmable hardware component to perform the method of a previously described example (e.g., one of the examples 23-44).

The description and drawings merely illustrate the principles of the disclosure. Furthermore, all examples recited herein are principally intended expressly to be only for pedagogical purposes to aid the reader in understanding the principles of the disclosure and the concepts contributed by the inventor(s) to furthering the art. All statements herein reciting principles, aspects, and examples of the disclosure, as well as specific examples thereof, are intended to encompass equivalents thereof.

As used herein, the term “module” refers to logic that may be implemented in a hardware component or device, software or firmware running on a processing unit, or a combination thereof, to perform one or more operations consistent with the present disclosure. Software and firmware may be embodied as instructions and/or data stored on non-transitory computer-readable storage media. As used herein, the term “circuitry” can comprise, singly or in any combination, non-programmable (hardwired) circuitry, programmable circuitry such as processing units, state machine circuitry, and/or firmware that stores instructions executable by programmable circuitry. Modules described herein may, collectively or individually, be embodied as circuitry that forms a part of a computing system. Thus, any of the modules may be implemented as circuitry. A computing system referred to as being programmed to perform a method may be programmed to perform the method via software, hardware, firmware, or combinations thereof.

Any of the disclosed methods (or a portion thereof) may be implemented as computer-executable instructions or a computer program product (e.g., machine-readable instructions, program code, etc.). Such instructions can cause a computing system or one or more processing units capable of executing computer-executable instructions to perform any of the disclosed methods. As used herein, the term “computer” refers to any computing system or device described or mentioned herein. Thus, the term “computer-executable instruction” refers to instructions that may be executed by any computing system or device described or mentioned herein.

The computer-executable instructions may be part of, for example, an operating system of the computing system, an application stored locally to the computing system, or a remote application accessible to the computing system (e.g., via a web browser). Any of the methods described herein may be performed by computer-executable instructions performed by a single computing system or by one or more networked computing systems operating in a network environment. Computer-executable instructions and updates to the computer-executable instructions may be downloaded to a computing system from a remote server.

Further, it is to be understood that implementation of the disclosed technologies is not limited to any specific computer language or program. For instance, the disclosed technologies may be implemented by software written in C++, C#, Java, Perl, Python, JavaScript, Adobe Flash, C#, assembly language, or any other programming language. Likewise, the disclosed technologies are not limited to any particular computer system or type of hardware.

Furthermore, any of the software-based examples (comprising, for example, computer-executable instructions for causing a computer to perform any of the disclosed methods) may be uploaded, downloaded, or remotely accessed through a suitable communication means. Such suitable communication means include, for example, the Internet, the World Wide Web, an intranet, cable (including fiber optic cable), magnetic communications, electromagnetic communications (including RF, microwave, ultrasonic, and infrared communications), electronic communications, or other such communication means.

The disclosed methods, apparatuses, and systems are not to be construed as limiting in any way. Instead, the present disclosure is directed toward all novel and nonobvious features and aspects of the various disclosed examples, alone and in various combinations and sub-combinations with one another. The disclosed methods, apparatuses, and systems are not limited to any specific aspect, feature, or combination thereof, nor do the disclosed examples require that any one or more specific advantages be present, or problems be solved.