DYNAMICALLY TUNING HYPERPARAMETERS DURING ML MODEL TRAINING

A method of automatically tuning hyperparameters includes receiving a hyperparameter tuning strategy. Upon determining that one or more computing resources exceed their corresponding predetermined quota, the hyperparameter tuning strategy is rejected. Upon determining that the one or more computing resources do not exceed their corresponding predetermined quota, a machine learning model training is run with a hyperparameter point. Upon determining that one or more predetermined computing resource usage limits are exceeded for the hyperparameter point, the running of the machine learning model training is terminated for the hyperparameter point and the process returns to running the machine learning model training with a new hyperparameter point. Upon determining that training the machine learning model is complete, training results are collected and computing resource utilization metrics are determined. A correlation of the hyperparameters to the computing resource utilization is determined from the completed training of the machine learning model.

BACKGROUND

Technical Field

The present disclosure generally relates to computing devices, and more particularly, to an efficient use of resources of computing devices in developing a machine learning model.

Description of the Related Art

In modern machine learning, hyperparameter tuning involves choosing a set of optimal hyperparameters for a learning algorithm. A hyperparameter is a parameter whose value is used to control the learning process. The same kind of machine learning model can include different constraints, weights, or learning rates to generalize different data patterns. These measures are called hyperparameters, and are tuned so that the model can optimally solve the machine learning problem. Hyperparameter tuning finds a tuple of hyperparameters that yields an optimal model which minimizes a predefined loss function on given independent data.

Many extensions of hyperparameter tuning are used to accommodate specific applications and problem domains, properties of machine learning models, and even characteristics of training datasets. One aspect that is largely unaddressed is computing resource usage during the search of hyperparameter space.

SUMMARY

According to various exemplary embodiments, a computing device, a non-transitory computer readable storage medium, and a method are provided to carry out a method of automatically tuning hyperparameters. Upon determining that one or more computing resources exceed their corresponding predetermined quota for the received hyperparameter tuning strategy, the hyperparameter tuning strategy is rejected and the process returns to receiving a hyperparameter tuning strategy. Upon determining that the one or more computing resources do not exceed their corresponding predetermined quota for the received hyperparameter tuning strategy, a machine learning model training is run with a hyperparameter point of the hyperparameter tuning strategy. Upon determining that one or more predetermined computing resource usage limits are exceeded for the hyperparameter point, the running of the machine learning model training for the hyperparameter point is terminated. The process returns to running the machine learning model training with a new hyperparameter point on the hyperparameter tuning strategy. Upon determining that training the machine learning model is complete, training results are collected and computing resource utilization metrics are determined. From the completed training of the machine learning model, a correlation of the hyperparameters to the computing resource utilization is determined.

In one embodiment, the hyperparameter point of the hyperparameter turning strategy is selected randomly.

In one embodiment, the inference of the correlation of the hyperparameters to the computing resource utilization is additionally based on one or more recorded previously successful completed training of machine learning models.

In one embodiment, the inference of the correlation of the hyperparameters to the computing resource utilization is by way of a probe run.

In one embodiment, the probe run generates a multi-dimensional grid of hyperparameters of permissible hyperparameter combinations.

In one embodiment, the hyperparameter tuning strategy is based on a grid search.

In one embodiment, the hyperparameter tuning strategy is based on a random search.

In one embodiment, running a machine learning model training includes dividing a training data into predetermined batches and iteratively running each training data.

In one embodiment, determining that the one or more computing resources exceed their corresponding predetermined quota for the received hyperparameter tuning strategy is based on each hyperparameter point being within an admissible region of a geometric convex hull in a hyperparameter space of the hyperparameter tuning strategy.

In one embodiment, the correlation of the hyperparameters to the computing resource utilization is inferred by measuring an increasing or decreasing trend of computing resource utilization metrics when a hyperparameter is changed in magnitude.

DETAILED DESCRIPTION

Overview

A typical machine learning model, such as deep learning, may initiate training by specifying parameters. Usually, most of the parameters are specified using guess work. For example, there may be an initial determination for the batch size, number of epochs, network layers, timeout, etc. Often, parameter values specified may not be suitable for the dataset being trained on, as the initial parameters may not accommodate all scenarios. Valuable time can be wasted before problems with the specified parameter values supplied can be identified. For example, too large a batch size can be taxing on the memory, as well as the central processing unit (CPU) and the graphics processing unit (GPU) resources. Memory transfers between GPU and CPU are a source of reducing the speed of training a machine learning model. Eventually, an out of memory condition may result in a core dump, wasting training time and the computational resources consumed to arrive at this state. Indeed, once the machine learning training starts, there is often no practical way to modify the parameters if needed. Consequently, the training may be stopped and restarted with a new set of parameters.

Another challenge faced by known systems is not knowing when to stop the training. For example, continued training to satisfy an epoch requirement, even when the model accuracy is not improving, presents a waste of computational resources. Often, a timeout is not specified for training—and even when it is specified, the training may be stopped abruptly without achieving a desired machine learning model accuracy. Also, a determination is not made whether the machine learning model accuracy is near the training goal, to facilitate additional training time to be added (i.e., delta time).

Accordingly, a salient aspect missing in the consideration of hyperparameter tuning is computing resource usage during the search of hyperparameter space. In many machine learning models, different hyperparameters determine the CPU, GPU, memory and sometimes storage IOPS (input/output operations per second) utilization. Take Convolutional Neural Network (CNN) based models as an example. In this regard, reference is made toFIG.1, which provides a conceptual block diagram100of an example CNN. The CNN100includes an input layer, one or more hidden layers and an output layer. The more hidden layers a network has, the more neurons it has and the more weights can be learned to make such a model more powerful. In addition, the size of each layer can be configured to capture more complex input data features. Increasing the number of layers or their sizes leads to larger memory usage, as well as higher CPU and GPU utilization. The hyperparameters are often tuned to achieve the best model performance. However, often limited by computing resource availability, not all desired hyperparameter combinations can be accommodated, often leading to model training execution crashing without explanation and ultimately to be manually investigated for cause and restarted by either relying on a bigger machine or tweaking the hyperparameter search to constrain resource usage. It is also costly to employ large and powerful computers, virtual machines, or a networked cluster.

The teachings herein provide a method and system to monitor computing resource usage as hyperparameter tuning progresses. The method terminates a model training execution run early if a hyperparameter combination leads to excessive resource usage, which may be a computing resource budget set by developers or administrators. The method learns from previously successful execution runs and early terminated runs to establish an admissible hyperparameter combination region, which is used to later determine whether a new combination of hyperparameters could be accepted or rejected. By virtue of the teachings herein, there is a technical effect of a reduced time to train a machine learning (ML) model by providing real-time system and model accuracy monitoring information to optimize usage of system resources, as well as conserving valuable computational resources. The techniques described herein may be implemented in a number of ways. Example implementations are provided below with reference to the following figures.

Example Architecture

FIG.2is an example block diagram of a distributed data processing system200in which aspects of the present disclosure may be implemented. Processing system200may relate to a single computer server or a cluster of computer servers in which one or more embodiments of the present disclosure can be implemented. The system200may include one or more servers, such as servers206and208, which are interconnected via a network208. Each server, such as server206, may include one or more central processing unit (CPU)201, main memory202(e.g., volatile memory), and one or more storage devices204(e.g., non-volatile memory). For example, code or instructions implementing the processes of the illustrative embodiments are executed by the CPU201and located temporarily in the main memory202. The one or more storage devices204can be used to store the instructions as well as a machine learning model that used training data to be processed by the system200. In other embodiments, one or more of these computing platforms may be implemented by virtual computing devices in the form of virtual machines or software containers that are hosted in the cloud, thereby providing an elastic architecture for processing and storage. The cloud is discussed in more detail later.

To better appreciate the features of the present disclosure, it may be helpful to describe by way of contrast. To that end, reference now is made toFIG.3, which shows a process for a known hyperparameter search and its iteration loop to learn batched training data. At block301, a combination of hyperparameters (p1, p2, p3, . . . , pn) is received as an input to a model training run. At block302, the model training configuration takes the received hyperparameters and initiates the relevant data structures and algorithms.

At block304, a model training process (which is usually iterative) may be performed by dividing training data into batches to learn and update model internal coefficients. The iteration can be skipped if the entire training data fits into main memory and can be consumed directly. At block306, after model training is completed, a cross validation or holdout data is used to evaluate the performance, usually measured in accuracy. This combination of hyperparameters is then associated with the measured performance. The program control then returns to block301to start on another combination of hyperparameters.

In contrast, the teachings herein provide a more advanced process by including additional components that facilitate a reduction in computational overhead while improving the speed of model training. In this regardFIG.4provides an example process400involving computing resource checking and usage statistics collection to facilitate an early termination when resource utilization exceeds predetermined limits and/or when a desired model accuracy is obtained, consistent with an illustrative embodiment.

At block401, a combination of hyperparameters (p1, p2, p3, pn) is received as an input to a model training run. At block402, the available computing resources are determined. For example, the computing resource determination may be against both completed and terminated hyperparameter combinations in the past. An example of hyperparameters (e.g., degrees of freedom) and their impact on computing resource usage is provided below:training_data_batch_size: 10, positively correlated with GPU memoryconvolutional_neural_network_layers: 10, positively correlated with GPU memorymaximum_number of training_steps: 1500, positively correlated with time durationconvolutional_neural_network_learning_rate: 0.001, negatively correlated with time duration

In various embodiments, correlations can be received from domain knowledge, user annotation, or experimental evidence through short probe runs with a small number of training steps such as 5 to 10 steps. For example, one or multiple hyper parameters can be tweaked, while monitoring resources. An estimation is performed, through extrapolation, if such set of hyperparameters may fit the available resources. A multi-dimensional grid of permissible combinations is incrementally built. In one embodiment, a correlation of the hyperparameters to the computing resource utilization is inferred by measuring an increasing or decreasing trend of utilization metrics when a hyperparameter is changed in magnitude.

A resource checking logic may elect to reject the proposed combination or accept and forward it. The checking logic uses an admissible region to determine the action, described in more detail later. If the identified computing resources are not adequate for the combination of hyperparameters, they are rejected. However, upon determining that the computational resources are sufficient, the process continues with block404, where the model training initialization takes the received hyperparameters and initiates the relevant data structures and algorithms. Data batching may be by random grouping by the batch size specified as one of the hyperparameters. Batch size impacts model learning convergence rate and cannot be too small nor too large, and thus needs to be explored for the best value. For example, a convolutional neural network (CNN) can be set up with a desired size and number of hidden layers, based on the received hyperparameters.

At block406, a model training process (which may be iterative) is performed by dividing training data into batches through random grouping to learn and update model internal coefficients, such as weights in convolutional neural network or centroid locations in K-Nearest Neighbor clustering. The iteration can be skipped if the entire training data fits into main memory and can be consumed directly. Significantly, after every model training iteration406, computing resource utilization metrics (e.g., statistics), such as maximum and mean main memory usage, maximum and mean GPU memory usage, maximum and mean CPU utilization, maximum and mean GPU utilization, etc., are collected at block408. In this way, an intelligent decision can be made whether to terminate or continue to the next training iteration. If resource usage exceeds the allowed limit, the process terminates the current training run and logs the hyperparameter combination and its early termination. However, if at block408it is determined that the computing resource usage is operating within predetermined limits, the iteration continues until the current run is completed (i.e., all allowed hyperparameter combinations have been evaluated or a user-specified model accuracy threshold has been reached). If it is determined that the iteration is to be terminated, the402receives notification of completed runs and their hyperparameter combinations. At block402, a tally of hyperparameter combinations and their outcome are kept, to update the decision logic, illustrated and described inFIG.8.

At block410, after model training is completed, a cross validation or holdout data is used to evaluate the performance (e.g., model accuracy measured by prediction errors, forecasting errors, classification errors, etc. in the problem context). This combination of hyperparameters is then associated with the measured performance. The process then returns to block401to start on another combination of hyperparameters, sometimes referred to herein as a hyperparameter point of a hyperparameter tuning strategy.

FIG.5illustrates two hyperparameter tuning strategies, consistent with an illustrative embodiment. More specifically, a grid search500and a random search502are illustrated, respectively. For example, two hyperparameters (p1, p2) are depicted with their combinations represented by corresponding solid black dots (i.e., hyperparameter point). In grid search500, hyperparameter search is expected to visit all dots in the area represented by each picture. In random search502, hyperparameter combinations or dots are randomly picked. Accordingly, the best combination is found by chance, at possibly shorter tuning time as compared to hyperparameter grid search500.

For example, a grid search500, sometimes referred to as a parameter sweep, provides an exhaustive search through a (e.g., manually) specified subset of the hyperparameter space of a learning algorithm. A grid search algorithm is guided by a performance metric, typically measured by cross-validation on the training set or evaluation on a hold-out validation set. In one embodiment, GridSearchCV can be used for a hyperparameter tuning strategy to exhaustively search over specified parameter values for an estimator, as provided by way of example hereinbelow:class sklearn.model_selection.GridSearchCV(estimator, param_grid, *, scoring=None, n_jobs=None, refit=True, cv=None, verbose=0, pre_dispatch=″2*n_jobs, error_score=nan, return_train_score=False)param_grid: dict or list of dictionariesDictionary with parameters names (str) as keys and lists of parameter settings to try as values, or a list of such dictionaries, in which case the grids spanned by each dictionary in the list are explored. This enables searching over any sequence of parameter settings.

The random search502, replaces the exhaustive enumeration of all combinations by selecting them randomly. Such approach can be applied to the discrete setting described above, as well as generalized to continuous and mixed spaces. It can outperform grid search, especially when only a small number of hyperparameters affects the final performance of the machine learning algorithm.

FIG.6depicts a grid search600and a random search602, respectively, having computing resource usage limits, consistent with an illustrative embodiment. More specifically,FIG.6depicts an admissible region (e.g., envelope), or hyperparameter combinations (e.g., points) allowed by a computing resource budget, marked as shaded regions inFIG.6. For example, for a restricted grid search600and restricted random search602, (p1, p2) combinations in shaded areas do not exceed computing resource usage limits, while combinations in non-shaded areas are not allowed (i.e., identified as exceeding allocated computing resources). An admissible region may be conceptualized as a geometric convex hull shape in the hyperparameter space or a trained binary classifier to flag a combination as pass or fail. An admissible region (e.g., envelope) represents the frontier of hyperparameter combinations to keep model training under limited and/or shared resources. For example, increasing deep learning model depth may increase GPU memory utilization. Increasing training batch size may increase CPU-GPU transfer data rate. WhileFIG.6depicts a two-dimensional grid to facilitate the present discussion, it will be understood that a three-dimensional grid is within the scope of the present teachings.

FIG.7shows a process700to monitor resource usage and record combinations of hyperparameters, consistent with an illustrative embodiment. At block701a search grid is received. At block702, a new training execution run is initialized with a randomly picked grid point. At block704, computing resource usage is monitored to measure various computing resources, such as CPU, GPU, memory utilization, etc. In one embodiment, Python language packages psutil and nvidia-ml-py may be used, although other system resource monitoring tools could be used as well.

At block706, aggregated usage is checked to determine if any predetermined limits are exceeded. If exceeded, the run is terminated and the hyperparameter combination (e.g., hyperparameter point of a hyperparameter tuning strategy) is logged as terminated. If the predetermined limits are not exceeded, the run continues to completion and logged as completed. The process700ofFIG.7may be executed multiple times to establish an admissible region in a hyperparameter tuning strategy.

FIG.8provides a possible continuation of the process ofFIG.7, consistent with an illustrative embodiment. In one embodiment, process800ofFIG.8is performed after multiple combinations of hyperparameters are labeled as completed or terminated, represented by block802, which is substantially similar to block708ofFIG.7. At block804, a mapping of the admissible region bordered by completed combinations is provided.

At block804, when the number of hyperparameters is small, as depicted on a 2D plot602inFIG.6, the admissible region is the largest convex hull whose vertices are coordinates of completed combinations and there are no terminated combinations inside the polygon identifying the admissible region of operation for hyperparameter parameter combinations.

When the number of hyperparameters is large (e.g., 10 or greater), constructing a convex hull can be more computationally complex. An alternate implementation is to train a binary classifier, such as (without limitation) a logistic regression or decision tree, to be the admissible region of operation. The input to the classifier may be the tabulated rows of hyperparameter combinations with completed (e.g., pass) or terminated (e.g., fail) as classification labels. In one embodiment, both the convex hull and the binary classifier, representing the admissible region, can be updated with new information.

At block806, the decision logic is created by checking if a proposed hyperparameter combination is inside the convex hull or its classification label is positively flagged as pass.

FIG.9shows a process900to check if a new hyperparameter combination is in the admissible region before starting a new model training run, consistent with an illustrative embodiment. For example, at block902, a new hyperparameter combination is received. At block904, the decision logic created in block806ofFIG.8can be used to determine whether the new hyperparameter combination should be admitted or rejected (i.e., whether or not it exceeds the allocated computational resources). For example, if the operating point is outside of a convex region of a known multi-dimensional grid, the run is rejected. If rejected, the process returns to block902. However, if admissible, the process continues with block906, where machine learning model training is started. After a new training run starts, resource monitoring may observe utilization beyond limits and terminate the run early. In one embodiment, if the new set of hyperparameters is outside the convex region of a known multi-dimensional grid of allowable parameters, a recommendation may be provided to scale back one or more hyperparameters known to reduce resource usage. In one embodiment, if an operating point cannot be determined, probe runs are initiated to determine a set of allowable hyperparameters. Infeasible parameters can be rejected to skip violating training runs, thereby avoiding any early termination in model training.

FIG.10shows a process1000that combines completed and terminated hyperparameters to update an admissible region for the decision logic, consistent with an illustrative embodiment. Process1000synthesizes various features discussed herein to provide a more comprehensive restricted hyperparameter search.

At block1002, a proposed hyperparameter combination is received. For example, the hyperparameter combination may be provided by user defined strategy such as grid search or random search. At block1004, an admissible decision logic is used to check if the hyperparameter combination lies in an admissible region, as described through the process discussed in the context ofFIG.9.

At block1006, model training commences and its running time execution in the server is monitored at block1008. At block1010, a determination of computing resources is made. If a computing resource usage limit is exceeded, the iteration is terminated. However, if computing resource limits are not exceeded, the model training iterations are allowed to continue by loading the next batch of data, until the maximum number of training steps is reached. At block1012, the hyperparameters used and whether or not they were completed or terminated are recorded and the process continues with block1004with an updated admissible region. In various embodiments, the teachings herein can be adapted to running a container such as Docker or a networked cluster with multiple computing nodes such as a Kubernetes cluster.

FIG.11shows a process1100to probe the directions of resource usage as one hyperparameter increases or decreases, consistent with an illustrative embodiment. At block1102, a model training hyperparameter search grid is received. At block1104, in contrast to process700ofFIG.7, instead of selecting a combination randomly as in block802, only a single hyperparameter value is changed to compare to previous combinations1102. The changed value may be derived as increasing or decreasing a stepped value to its neighbor on the search grid. The hyperparameter chosen at each time could be random or scanned in a fixed order given by a user. At block1106, computing resource usage is monitored to measure various computing resources, such as CPU, GPU, memory utilization, etc. In one embodiment, Python language packages psutil and nvidia-ml-py may be used, although other system resource monitoring tools could be used as well.

At block1108, aggregated usage is checked to determine if any predetermined limits are exceeded. If exceeded, the run is terminated and the hyperparameter combination (i.e., hyperparameter point) is logged as terminated. If the predetermined limits are not exceeded, the run continues to completion at block1110and logged as completed.

By way of example, consider a 2-hyperparameter search. Increasing p1 while holding p2 constant could lead to increased usage for some resources and decreased usage for others. If at some p1_value, say p1_limit, resource usage already exceeds an allowed threshold, then any p1>=p1_limit, assuming other hyperparameters are the same, should not be accepted.

In one embodiment, there could be a user annotated directional hint, such as the increase of p1 leads to increase in CPU, GPU but no increase in memory usage and the increase of p2 leads to increase in memory usage but no impact in CPU utilization. The annotated directional hints can be included in block1104to probe the boundaries of the admission region. The process1100ofFIG.11may be executed multiple times to establish an admissible region.

Example Computer Platform

As discussed above, functions relating to automatically tuning hyperparameters used in machine learning based on monitored computing resources, and other functions discussed herein, can be performed with the use of one or more computing devices connected for data communication via wireless or wired communication, as shown inFIG.2.FIG.12is a functional block diagram illustration of a particularly configured computer hardware platform that can be used to implement the computing device discussed in the context ofFIG.2.

The computer platform1200may include a central processing unit (CPU)1204, a hard disk drive (HDD)1206, random access memory (RAM) and/or read only memory (ROM)1208, a keyboard1210, a mouse1212, a display1214, and a communication interface1216, which are connected to a system bus1202.

In one embodiment, the HDD1206, has capabilities that include storing a program that can execute various processes, such as the hyperparameter engine1240, in a manner described herein. The hyperparameter engine1240may have various modules configured to perform different functions. For example, there may be a hyperparameter search engine operative to receive a combination of hyperparameters (p1, p2, p3, pn) as an input to a model training run. There may be a computing resource checking engine1244operative to determine the available computing resources as well as determine the computing resources used during the training. There may be a machine learning model training module1246operative to train a model without exceeding allocated and/or available computational resources. There may be a model training iteration module1248operative to continue the iterative model training until a predetermined threshold accuracy is achieved.

Example Cloud Platform

As discussed above, functions relating to automatically tuning hyperparameters used in machine learning based on monitored computing resources, may include a distributed computing and/or storage architecture, as in a cloud. It is to be understood that although this disclosure includes a detailed description on cloud computing, implementation of the teachings recited herein are not limited to a cloud computing environment. Rather, embodiments of the present disclosure are capable of being implemented in conjunction with any other type of computing environment now known or later developed.

Characteristics are as Follows:

Service Models are as Follows:

Deployment Models are as Follows:

Referring now toFIG.13, an illustrative cloud computing environment1300is depicted. As shown, cloud computing environment1300includes one or more cloud computing nodes1310with which local computing devices used by cloud consumers, such as, for example, personal digital assistant (PDA) or cellular telephone1354A, desktop computer1354B, laptop computer1354C, and/or automobile computer system1354N may communicate. Nodes1310may communicate with one another. They may be grouped (not shown) physically or virtually, in one or more networks, such as Private, Community, Public, or Hybrid clouds as described hereinabove, or a combination thereof. This allows cloud computing environment1350to offer infrastructure, platforms and/or software as services for which a cloud consumer does not need to maintain resources on a local computing device. It is understood that the types of computing devices1354A-N shown inFIG.13are intended to be illustrative only and that computing nodes1310and cloud computing environment1350can communicate with any type of computerized device over any type of network and/or network addressable connection (e.g., using a web browser).

Referring now toFIG.14, a set of functional abstraction layers provided by cloud computing environment1350(FIG.13) is shown. It should be understood in advance that the components, layers, and functions shown inFIG.14are intended to be illustrative only and embodiments of the disclosure are not limited thereto. As depicted, the following layers and corresponding functions are provided:

Hardware and software layer1460includes hardware and software components. Examples of hardware components include: mainframes1461; RISC (Reduced Instruction Set Computer) architecture based servers1462; servers1463; blade servers1464; storage devices1465; and networks and networking components1466. In some embodiments, software components include network application server software1467and database software1468.

Virtualization layer1470provides an abstraction layer from which the following examples of virtual entities may be provided: virtual servers1471; virtual storage1472; virtual networks1473, including virtual private networks; virtual applications and operating systems1474; and virtual clients1475.

In one example, management layer1480may provide the functions described below. Resource provisioning1481provides dynamic procurement of computing resources and other resources that are utilized to perform tasks within the cloud computing environment. Metering and Pricing1482provide cost tracking as resources are utilized within the cloud computing environment, and billing or invoicing for consumption of these resources. In one example, these resources may include application software licenses. Security provides identity verification for cloud consumers and tasks, as well as protection for data and other resources. User portal1483provides access to the cloud computing environment for consumers and system administrators. Service level management1484provides cloud computing resource allocation and management such that required service levels are met. Service Level Agreement (SLA) planning and fulfillment1485provide pre-arrangement for, and procurement of, cloud computing resources for which a future requirement is anticipated in accordance with an SLA.

Workloads layer1490provides examples of functionality for which the cloud computing environment may be utilized. Examples of workloads and functions which may be provided from this layer include: mapping and navigation1491; software development and lifecycle management1492; virtual classroom education delivery1493; data analytics processing1494; transaction processing1495; and Translation Engine1496, as discussed herein.

CONCLUSION

Aspects of the present disclosure are described herein with reference to a flowchart illustration and/or block diagram of a method, apparatus (systems), and computer program products according to embodiments of the present disclosure. The processes discussed herein are each illustrated as a collection of blocks representing a sequence of operations that can be implemented in hardware, software, or a combination thereof. In the context of software, the blocks represent computer-executable instructions that, when executed by one or more processors, perform the recited operations. Generally, computer-executable instructions may include routines, programs, objects, components, data structures, and the like that perform particular functions or implement particular abstract data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described blocks can be combined in any order and/or performed in parallel to implement the process.