Patent Description:
To continue improving the user experience and/or providing an optimal and reliable user experience, an entity providing the cloud-based service may deploy changes and/or upgrades to different resource units. The upgrades may include updated code and/or other mechanisms configured to maintain, correct, add, and/or remove functionality associated with the service provided, as well as individual resource units. In addition, due to the dynamic and unpredictable nature of many cloud-based applications, cloud-based service providers may deploy resource unit upgrades extremely frequently. In various examples, a resource unit upgrade can include certificates, compiler updates, operating system version updates, security updates, updates for utilizing new hardware, and so forth.

Unfortunately, in typical solutions, resource unit upgrades are broadly deployed without safety mechanisms in place. For example, an upgrade may be deployed to the entire breadth of a server farm (e.g., <NUM>% of the resource units in a server farm) without first testing and validating that the upgrade functions properly. In addition, upgrades that target low-level infrastructure such as the resource units discussed above, are highly risky and non-backwards compatible. In a specific example, a resource unit upgrade may replace an operating system at a resource unit with a newer version of the operating system. Accordingly, if an unforeseen fault arises that requires rolling back the upgrade, the affected resource units must be taken offline and manually reconfigured. As will be discussed in further detail below, deploying upgrades in this way exposes the upgrade to a practically unlimited fault domain and can lead to catastrophic results if the upgrade causes a problem at the resource units.

Thus, deploying upgrades in this manner can result in widespread failures across the cloud platform which can lead to a degraded user experience and/or a loss of functionality within the cloud-based service. For instance, as a cloud-based service grows in size and complexity, managing the deployment of various upgrades can become overly burdensome and impractical. Furthermore, broadly deploying unvalidated upgrades can exacerbate the difficulty of addressing problems during deployment which can ultimately have a greater negative effect on the service. For instance, recovering from failures at the full scale of a large cloud platform can require extended downtime, significant commitment of engineering resources, and unacceptable disruption to critical applications.

<CIT> relates to device analytics. Specifically, resource deployment at an organization including one or more devices, receiving telemetry data from the one or more devices associated with the organization, generating a set of deployment rings for the deployment of the software resource at the one or more devices of the organization, and deploying the software resource within the organization based on the set of deployment rings.

The disclosed techniques improve the efficiency and functionality of cloud-based platforms and services by providing mechanisms for small scale deployment of upgrades and associated variants to resource units. Generally described, a system selects a set of resource units at which to deploy one or more variants of an upgrade to validate the variants of the upgrade before proceeding to deploy the upgrade to the entirety of the cloud-based platform.

As described, an upgrade may include updated code and/or other mechanisms configured to maintain, correct, add, and/or remove functionality associated with the service provided. The service can include cloud-based applications that provide access to internal sites, documents, lists, libraries and the like from diverse user devices. In addition, the service can be constructed using various server farms composed of individual resource units such as virtual machines, physical machines, network devices, and containers. In various examples, the server farms may be different systems configured within the same and/or different datacenters. In another example, the server farms may be different networks configured for the same and/or different geographic locations (e.g., datacenters).

As mentioned above, existing solutions deploy upgrades very frequently and at scale without proper safety mechanisms or validation. While deploying in this manner can allow for rapid adoption of the upgrade, unforeseen problems that potentially arise due to the upgrade can cause severe problems that are significantly exacerbated by the scale of various cloud platforms. This is especially true for upgrades that target low-level resource units. As discussed previously, such upgrades often include high-risk changes that are not backwards compatible. In this way, many resource unit upgrades prevent graceful recovery in the event of a failure thereby requiring unacceptable service downtime and significant engineering effort. Thus, there is a need for cloud-based service providers to validate and deploy resource unit upgrades at a smaller scale to enable efficient testing and validation and ensure maximal availability and reliability of the cloud-based service.

In addition, due to the diversity of resource units utilized by a cloud platform, a particular resource unit upgrade may require several variants where each variant comprises a different feature set. For instance, a resource unit upgrade may be a security update in which a variant includes a security feature that targets a certain hardware configuration. Accordingly, other variants of the security update may include different features that target other hardware configurations. However, typical solutions for upgrade deployments lack the ability to select and deploy multiple variants of the same resource unit upgrade, instead requiring separate deployments for each variant.

In contrast to existing solutions, the use of small-scale upgrade deployment as described herein enables efficient testing and validation of an upgrade as well as several variants of the same upgrade. Typically, upgrades are deployed at the scale of server farms which can represent a wide fault domain that may especially hamper validation and recovery efforts for non-backward compatible upgrades. By reducing the initial scale of deployment, the system can mitigate the negative effects of problems that may arise during testing of an upgrade and greatly improve the efficiency of experimentation and debugging of upgrade variants.

To deploy an upgrade, the system receives several variants of the upgrade and selects a set of resource units to receive the upgrade variants based on various criteria such as a hardware configuration of the resource units, upgrade type, the number of variants, and so forth. In addition, the upgrade variants can be received from a feature group which can be any entity that is responsible for the upgrade and upgrade variants in question, such as a single engineer or a group of developers.

In addition, the system is configured to collect and analyze telemetry data from the selected set of resource units to enable the feature group to view and assess the effectiveness of the variants in a live environment. Telemetry data can also define usage information of an upgrade variant at a resource unit, e.g., any new errors occurring in a software service because of the upgrade event deployment. The telemetry data can include reliability signals describing performance of a particular upgrade variant.

By analyzing the telemetry data, the system detects that one or several upgrade variants have caused problems arising at the selected set of resource units. For instance, the system may determine that a certain compiler update variant caused an unexpected runtime error for a particular configuration of a virtual machine. As will be discussed in more detail below, the system takes various actions in response to detecting the problem including removing the offending variant from deployment, and optionally modifying the offending variant to resolve the technical issue, and/or notifying the feature group of the problem caused by the offending variant. Once the upgrade variants have been tested and validated at the initial set of resource units, deployment then proceeds at a wider scale to provide the upgrade variants to the broader cloud platform.

As mentioned above, and in greater detail below, the disclosed techniques enable small-scale deploying of infrastructure upgrades and upgrade variants for cloud-based services. In this way, cloud-based service providers can rapidly test upgrade variants, nimbly recover from any potential problems, and minimize any the negative impact of those problems on the cloud-based service. Features and technical benefits other than those explicitly described above will be apparent from a reading of the following Detailed Description and a review of the associated drawings. This Summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

The techniques described herein provide systems for optimizing the use of computing resources and for improving the operation of upgrade deployment to various resource units that enable server farms to provide a cloud-based service by the introduction of small-scale deployment and upgrade variants. As mentioned above, resource units can include physical and/or virtual resource units such as virtual machines, physical machines, network devices, and containers. In addition, an upgrade can include updated code and/or other mechanisms configured to maintain, correct, add, remove or otherwise change functionality associated with the service provided, as well as individual resource units. An upgrade may also be referred to herein as a change. In one example, the service can include cloud-based applications that provide access to internal sites, documents, lists, libraries and the like from diverse user devices around the world.

The disclosed techniques address several technical problems associated with upgrade deployment for resource units (which may also be referred to herein as low-level infrastructure objects). For example, the disclosed system addresses inefficiencies and problems associated with existing upgrade deployment systems that deploy upgrades at the scale of server farms without safety mechanisms to accommodate the necessary frequency of resource unit upgrade deployments. As discussed above, however, this can expose large portions of a cloud platform to potential problems since upgrades that target resource units can be very risky and not backwards compatible. In this way, if a deployed upgrade causes a problem or fault, entire server farms may need to be taken offline to restore functionality resulting in extensive downtime and unacceptable disruptions to critical applications.

In contrast to existing solutions and as further discussed below, the disclosed techniques provide mechanisms for small-scale deployment of resource unit upgrades (e.g., a set of individual resource units as opposed to whole server farms). Deploying resource unit upgrades in this manner enables several technical benefits such as simultaneous deployment of several upgrade variants, as well as efficient validation and testing of the upgrade variants prior to wide-scale deployment to the cloud platform. In this way, a feature group can continue to frequently deploy resource unit upgrades while minimizing the potential impact of problems that arise due to the upgrade variants. In a specific example, a feature group wishes to deploy an upgrade for virtual machines to allow compatibility with a new hardware configuration (e.g., a cloud service provider has changed vendors for server hardware). Rather than immediately deploying the upgrade to entire server farms, the feature group provides the upgrade to the system which automatically selects a subset of the available resource units at which to deploy the upgrade. As will be discussed in more detail below, the selection can be based on a diverse set of criteria to ensure a suitable subset of resource units is chosen. The system then collects and analyzes telemetry data from the selected set of resource units and determines if any problems have occurred. If a problem is detected, the system takes several courses of action to resolve the problem which is also discussed below. Thus, the system can efficiently validate resource unit upgrades and additionally conserve computing resources through small-scale deployment.

In another example of the technical benefit of the present disclosure, the capability to deploy multiple variants of an upgrade enables a feature group to easily experiment with different feature sets and debug. A feature group has several implementations of a certain upgrade with that includes variants with differing feature sets. To assess the effectiveness of various implementations the system is configured to deploy a group of upgrade variants to a set of resource units and another group of upgrade variants to another set of resource units. As will be discussed below, the system analyzes telemetry data from the sets of resource units using various performance criteria to compare different upgrade variants.

In still another example of the technical benefit of the present disclosure, the techniques described here can improve the security of cloud platforms and cloud-based services. As discussed above, small-scale resource upgrade deployment enables the disclosed system to minimize the potential impact of problems that arise at various resource units due to upgrade variants. In this way, the disclosed system can enhance security by preventing downtime of the cloud platform thereby removing potential vulnerabilities that can be exploited by potential attackers.

Various examples, scenarios, and aspects that enable consistent and stable deployment of upgrade events through small-scale deployment of upgrade variants and monitoring of telemetry data are described below with reference to <FIG>.

<FIG> illustrates an example environment <NUM> in which a system <NUM> is configured to deploy an upgrade <NUM> that includes one or more upgrade variants 104A-104D. As mentioned above and discussed further herein, system <NUM> deploys upgrade <NUM> to a plurality of resource units 106A -106N (may be referred to herein as resource units <NUM>) that make up a cloud platform. It should be understood that an individual one of the plurality of resource units 106A can comprise a single resource unit or several resource units. For instance, resource unit 106A can be a single virtual machine or a group of virtual machines. In addition, it should be understood that while upgrade <NUM> as illustrated contains four upgrade variants 104A-104D, an upgrade <NUM> can contain any number of upgrade variants.

As discussed above, resource units <NUM> can include various types of infrastructure objects such as virtual machines, physical machines, network devices, containers and the like that are under control of an entity providing a cloud-based service and/or operating the system <NUM> configured to deploy the upgrade <NUM> and to monitor the health and performance of the upgrade <NUM>. To this end, each of the resource units 106A-106N provides at least a part of a service to end-users, customers, tenants, clients, etc. In addition, resource units <NUM> can be used to construct portions of a cloud platform (e.g., server farms) and configured to provide various cloud-based services such as access to hosted applications, computation resources, databases, and the like. Resource units <NUM> may also be located within the same and/or different datacenters configured at different geographic locations.

A feature group <NUM> can submit an upgrade <NUM>, comprising a plurality of upgrade variants 104A-104D, to the system <NUM> to be deployed. The feature group <NUM> can be any entity that is responsible for deploying the upgrade <NUM> to the resource units 106A - 106N such as one or more developers, a technician, a system administrator, and so forth. Upon receiving the upgrade <NUM>, the system <NUM> can utilize upgrade analysis module <NUM> to determine various upgrade characteristics <NUM> such as an upgrade type (e.g., an operating system update, a compiler update, etc.), a targeted hardware configuration, a targeted resource unit type, the number of upgrade variants 104A-104D within upgrade <NUM>, and so forth. Alternatively, upgrade characteristics <NUM> may be provided to the system <NUM> by feature group <NUM> as one or more predefined characteristics. In another example, upgrade characteristics <NUM> can be a combination of predefined characteristics and characteristics that are automatically determined by upgrade analysis module <NUM>.

Upgrade analysis module <NUM> can subsequently provide upgrade characteristics <NUM> to a resource unit selection module <NUM>. Accordingly, resource unit selection module <NUM> uses upgrade characteristics <NUM> as criteria for selecting the resource units 106A, from the plurality of resource units 106A-106N, as the set to which the upgrade variants 104A-104D is deployed. In a specific example, upgrade <NUM> may be an operating system update that targets virtual machines (e.g., hundreds or thousands of virtual machines). Accordingly, resource unit selection module <NUM> can select a set of resource units 106A that are virtual machines and are executing the targeted version of the operating system. As mentioned above, resource unit(s) 106A can be a single resource unit or several resource units. For the purposes of the present discussion, resource units 106A can be considered as a plurality of resource units. In addition, individual resource units within resource units 106A can receive some or all of the upgrade variants 104A-104D. For instance, feature group <NUM> may specify that a portion of the selected set of resource units 106A are to receive all upgrade variants 104A-104D while others only receive a single upgrade variant 104A. In addition, the size of the selected set of resource units can be based on the number of upgrade variants 104A-104D in a particular upgrade <NUM>. For instance, an upgrade <NUM> having only a few upgrade variants (e.g., five) may only require a small set of resource units 106A to deploy (e.g., ten to twenty). Conversely, an upgrade <NUM> having many upgrade variants (e.g., a dozen) may require a comparatively larger set of resource units 106A (e.g., a hundred).

The system <NUM> is further configured to automatically collect and analyze telemetry data <NUM> from resource units 106A using telemetry analysis module <NUM>. Telemetry data <NUM> can describe many aspects of the execution of upgrade <NUM> at resource units 106A such as resource usage, performance metrics, and any errors that may have occurred. As will be further detailed below, analysis of telemetry data <NUM> enables the system <NUM> to assess and compare performance levels for various upgrade variants 104A-104D. In addition, telemetry analysis module <NUM> can include various performance criteria <NUM> that can be utilized to determine a level of performance for an upgrade variant 104A. In various examples, this can include assigning a one or more numerical scores to quantify the performance of an upgrade variant 104A. In addition, telemetry analysis module <NUM> can compare levels of performance for an upgrade variant 104A against one or more performance thresholds <NUM>. For example, feature group <NUM> may define an acceptable level of network latency for a given resource unit 106A.

As mentioned, analysis of telemetry data <NUM> also enables telemetry analysis module <NUM> to detect problems at resource units 106A that have occurred in response to execution of upgrade variants 104A-104D and identify upgrade variants that are causing the problems. Problems at resource units 106A can include unresponsive resource units, application crashes, degraded performance, high latency, abnormal resource usage, and the like. In a specific example, upgrade variant 104A, as part of an operating system update, may implement a new feature to improve network speeds for virtual machines. However, upon collecting telemetry data <NUM>, telemetry analysis module <NUM> may detect unexpectedly high resource usage in resource units 106A that received upgrade variant 104A.

In response to detecting problems at the resource units 106A based on telemetry data <NUM>, the system <NUM> takes several courses of action to resolve various issues. For instance, telemetry analysis module <NUM>, upon detecting a problem at resource units 106A, can generate one or more action recommendations <NUM> to notify feature group <NUM> of detected problems. Action recommendation <NUM> can also include a proposed course of action for feature group <NUM> to address a detected problem. In a specific example, telemetry analysis module <NUM> can notify feature group <NUM> that the networking feature implemented by upgrade variant 104A is causing extremely high resource usage. By providing specific information on various problems, the system <NUM> can enable feature group <NUM> to focus engineering effort to efficiently address issues. Telemetry analysis module <NUM> can accordingly propose that feature group <NUM> remove the problematic upgrade variant 104A from deployment. Alternatively, the system <NUM> can automatically execute proposed actions such as removing an upgrade variant 104A and subsequently notify feature group <NUM> of the action taken.

As illustrated in <FIG>, telemetry analysis module <NUM> has detected that problems have occurred at resource units 106A in response to executing upgrade variant 104C. Accordingly, the system <NUM> removes upgrade variant 104C (referred to in <FIG> as removed upgrade variant <NUM>) from active deployment at resource units 106A and generates an action recommendation <NUM> to notify feature group <NUM> of the problem and actions taken by system <NUM>. As discussed above, action recommendation <NUM> can include specific information regarding the detected problems derived from telemetry data <NUM> such as various metrics, logs, traces, and the like. Alternatively, telemetry analysis module <NUM> may be configured to notify feature group <NUM> of a problem and refrain from automatically removing upgrade variant 104C from active deployment. In this way, feature group <NUM> can elect to continue observing upgrade variant 104C and fully assess the extent of potential issues.

In a specific example, removed upgrade variant <NUM> may have caused high resource consumption at resource units 106A that executed removed upgrade variant <NUM>. However, feature group <NUM> may have anticipated this behavior and wish to measure the impact of removed upgrade variant <NUM> on the performance of resource units 106A. Feature group <NUM> can configure the system <NUM> to prevent automatic removal of upgrade variant 104C from deployment. As will be discussed in greater detail below, affording flexibility to feature group <NUM> as to which upgrade variants 104A-104D are deployed enables feature group <NUM> to compare various upgrade variants 104A-104D and assess the effectiveness of upgrade variants 104A-104D for various situations. In this way, system <NUM>, through small-scale deployment of upgrades <NUM>, can enable more efficient testing and debugging of features for upgrade variants 104A-104D.

In another example of the capabilities of telemetry analysis module <NUM>, system <NUM> may resolve an issue by modifying an upgrade variant 104A-104D that is causing problems at resource units 106A as illustrated in <FIG>. For instance, as in the previous example illustrated in <FIG>, telemetry analysis module <NUM> may detect that upgrade variant 104C is causing excessive resource consumption at resource units 106A. In response, system <NUM> can generate modified upgrade variant <NUM> to replace upgrade variant 104C and reduce resource consumption and potentially prevent failure of resource units 106A. As will be discussed further below, modified upgrade variant <NUM> can be generated in a variety of ways such as by disabling certain features using feature flags, generating error tags to place limits on modified upgrade variant <NUM>, and so forth. Alternatively, like the above example in <FIG>, telemetry analysis module <NUM> may generate an action recommendation <NUM> to notify feature group <NUM> of problems at resource units 106A without modifying the upgrade variant 104C. In this example, feature group <NUM> can provide system <NUM> with a modified upgrade variant <NUM> to replace upgrade variant 104C. It should be understood that the modified upgrade variant <NUM> provided by feature group <NUM> can differ from modified upgrade variant <NUM> that is automatically generated by the system <NUM>. For example, modified upgrade variant <NUM> as provided by feature group <NUM> may contain more complex revisions to source code to correct errors.

Proceeding now to <FIG>, based on the action recommendation <NUM>, feature group <NUM> has elected to resolve various problems by providing a modified upgrade variant <NUM> to replace upgrade variant 104C in upgrade <NUM>. Additional validation, such as the techniques shown and discussed in <FIG>, may indicate no additional problems at resource units 106A. Accordingly, upgrade <NUM> proceeds to more wide scale deployment and target all applicable resource units <NUM> across various server farms, as shown in <FIG> via the arrow to each of resource units 106A-106N. By employing small-scale deployment to enable rapid validation of upgrade variants 104A-104D, system <NUM> can safely deploy upgrade <NUM> to all resource units <NUM>. As discussed with respect to the examples illustrated in <FIG>, the techniques described herein enable a feature group <NUM> to deploy a resource unit upgrade <NUM> at a small scale (e.g., individual resource units <NUM>). In this way, the system <NUM> can greatly reduce the fault domain of an upgrade <NUM> thereby minimizing the potential impact of potential problems. This is especially crucial for upgrades <NUM> that target low-level infrastructure objects such as the above-mentioned resource units <NUM> as these upgrades <NUM> tend to be highly risky and non-backwards compatible. Furthermore, in the event a problem does arise at a resource unit <NUM>, significantly less engineering effort is required to resolve the problem and restore functionality to affected resource units <NUM>.

Turning now to <FIG>, additional aspects of a system <NUM> for selecting and deploying a plurality of upgrade variants 104A-104D are shown and described. More specifically, <FIG> illustrates features that enable efficient experimentation and comparison of differing upgrade variants 104A-104D. As in previously discussed examples, system <NUM> has received an upgrade <NUM> having upgrade variants 104A-104D from feature group <NUM>. Feature group <NUM> provides a deployment configuration <NUM> to system <NUM> to cause deployment of multiple upgrades <NUM> having differing sets of upgrade variants 104A-104D referred to here as test upgrades 304A and 304B (may also be referred to herein as test upgrades <NUM>). As shown in <FIG>, test upgrade 304A includes upgrade variants 104A, 104B, and 104D. Upgrade variant 104C is omitted from test upgrade A 304A as indicated by the shading. Conversely, test upgrade 304B includes upgrade variants 104A, 104C, and 104D with upgrade variant 104B omitted. Thus, test upgrade 304A may implement a first feature set with upgrade variants 104A, 104B, and 104D while test upgrade 304B implements a second feature set using upgrade variants 104A, 104C, and 104D. As further illustrated in <FIG>, resource unit selection module <NUM> has chosen to deploy test upgrade 304A to resource units 106A and test upgrade 304B to resource units 106C. Deployment configuration <NUM> may contain additional settings provided by feature group <NUM>. For instance, feature group <NUM> may wish to deploy test upgrades <NUM> during a specific period of time to ensure maximal traffic to the test upgrades <NUM>. As previously discussed, resource units 106A can be selected based on upgrade characteristics <NUM>. In addition, the size of the selected set of resource units 106A can also be determined based on the number of upgrade variants 106A-106D that are selected for deployment.

As discussed above, resource unit selection module <NUM> automatically selects an appropriate set of resource units at which to deploy an upgrade <NUM> based on upgrade characteristics <NUM>. In the present example, to accurately compare the effectiveness of upgrade variants 104A-104D, resource unit selection module <NUM> may select resource units 106A, 106C that share similar or identical characteristics. For instance, feature group <NUM> may wish to assess different test upgrades <NUM> that target network devices to implement a new networking protocol. Since the upgrade characteristics <NUM> of test upgrades 304A and 304B are similar, resource unit selection module <NUM> can accordingly select resource units 106A and 106C that meet those upgrade characteristics <NUM>. Alternatively, system <NUM> may be configured to select one set of resource units 106A at which to deploy both test upgrades 304A and 304B. Stated another way, system <NUM> can evaluate test upgrades <NUM> in parallel by selecting multiple resource units <NUM> or sequentially by selecting a single set of resource units 106A.

Following deployment of the test upgrades <NUM> to the sets of resource units 106A and 106C respectively, the system <NUM> collects telemetry data 306A and 306B (may also be referred to herein as telemetry data <NUM>) for each corresponding test upgrade <NUM> as shown in <FIG>. As discussed above, telemetry data <NUM> can include various types of data such as metrics, logs, traces, and the so forth. Upon receiving telemetry data <NUM>, telemetry analysis module <NUM> can utilize performance criteria <NUM> and performance thresholds <NUM> to analyze the various types of data in telemetry data <NUM>. In a specific example, a performance criterion <NUM> may measure network latency for a given resource unit <NUM> executing a test upgrade <NUM>. In this example, a performance threshold <NUM> can specify an acceptable network latency of eighty milliseconds and an optimal network latency of twenty milliseconds. In this way, telemetry analysis module <NUM> can introduce additional granularity when evaluating the performance of upgrade variants <NUM>.

Based on the analysis of the telemetry data <NUM>, telemetry analysis module <NUM> can generate performance data <NUM>. Performance data <NUM> can include performance scores for each test upgrade <NUM>. In various examples, performance scores can be a numerical score that is generated from a comparison of the telemetry data <NUM> against the performance criteria <NUM> and performance thresholds <NUM>. Continuing with the network latency example above, telemetry analysis module <NUM> may detect that the network latency for resource units 106A that are executing test upgrade 304A exceeds eighty milliseconds. Conversely, telemetry analysis module <NUM> may detect that the network latency for resource units 106C executing test upgrade 304B falls below twenty milliseconds. Accordingly, telemetry analysis module <NUM> may assign a high performance score to test upgrade 304B and a low performance score to test upgrade 304A in performance data <NUM>. Alternatively, telemetry analysis module <NUM> may detect that an upgrade variant 104B does not satisfy a performance threshold. In response, the system <NUM> may automatically remove or otherwise disable upgrade variant 104B from deployment. For instance, if a measured network latency for upgrade variant 104B exceeds eighty milliseconds, system <NUM> may remove upgrade 104B from deployment.

It should be understood that, for the sake of simplicity, only one metric is mentioned in the above example. However, a performance score can represent several performance metrics rather than merely a single metric. For instance, a performance score can be calculated based on an aggregation of metric scores such as resource usage, power consumption, network latency, and so forth. In addition, individual metric scores may be optionally weighted to emphasize or deemphasize their influence in determining performance scores. In various examples, telemetry analysis module <NUM> may also utilize a machine learning model to determine performance criteria <NUM>, performance thresholds <NUM>, as well as the various weights that correspond to individual metric scores. For instance, in the above-mentioned example, telemetry analysis module <NUM> may determine that network latency is the most important metric and thus weight network latency more heavily compared to other metrics such as power consumption. Over time, telemetry analysis module <NUM> can refine the weightings for metric scores to suit various types of upgrades <NUM>. Furthermore, telemetry analysis module <NUM> can rank test upgrades <NUM> based on the performance scores in performance data <NUM>. Alternatively, telemetry analysis module <NUM> can rank individual upgrade variants <NUM> to expose finer details for each test upgrade <NUM>. Upgrade rankings and performance scores can subsequently be provided to feature group <NUM> as performance data <NUM> to grant detailed insights on the effectiveness of individual upgrade variants <NUM>. Continuing with the networking protocol example discussed above, performance data <NUM> may indicate to feature group <NUM> that test upgrade 304B received a higher performance score than test upgrade 304A. As shown in <FIG>, after considering the experiment results provided by performance data <NUM>, feature group <NUM> have elected to omit upgrade variant 104B from final upgrade <NUM> in favor of upgrade variant 104C. In combination with the validation techniques shown and discussed in <FIG>, feature group <NUM> can safely deploy final upgrade <NUM> to the full plurality of applicable resource units <NUM>. In the example discussed above, applicable resource units <NUM> can be all network devices in a particular server farm, geographic region, or even the entire cloud platform. In this way, feature group <NUM> can quickly and efficiently compare and contrast upgrade variants <NUM>. In a specific example, nimble small-scale testing as illustrated in <FIG> can contribute to rapid mitigation of various errors. Consider an enterprise that utilizes a cloud-based service to host an important database. Utilizing the small-scale testing and validation techniques discussed herein enables a service provider to quickly determine a root cause for any erroneous behavior and implement a fix while minimizing disruptions to database operations.

Turning now to <FIG>, aspects of an individual upgrade variant 104A are shown and described. As discussed above, an upgrade variant 104A can implement a certain set of features <NUM> where different upgrade variants <NUM> implement different sets of features <NUM>. In addition, individual features <NUM> can be enabled or disabled by a corresponding feature flag <NUM>. For instance, in the examples shown in <FIG>, a test upgrade 304A may utilize feature flags <NUM> to disable a certain upgrade variant 104C rather than omit upgrade variant 104C entirely. In this way, if experimentation indicates that an upgrade variant 104C is more performant than another upgrade 104B, feature flags <NUM> can be configured to enable upgrade variant 104C and disable upgrade variant 104B. Thus, system <NUM> can reduce network traffic by merely communicating updated configurations for feature flags <NUM> as opposed to deploying a full final upgrade <NUM>. Upgrade variant 104A may additionally include metadata <NUM> that include various details pertaining to upgrade variant 104A. For instance, metadata <NUM> can specify an upgrade type <NUM> (e.g., an operating system update, a security update, etc.), resource unit target <NUM> such as virtual machines and network devices, a configuration target <NUM> such as a specific operating system version, and a variant identifier <NUM>. Inclusion of metadata <NUM> in upgrade variant 104A can enable several features such as generation of upgrade characteristics <NUM> for resource unit selection module <NUM>. In addition, metadata <NUM> can include error tags <NUM> to associate problems that occur with the upgrade variant 104A that caused the problem. Including error tags <NUM> and a variant identifier <NUM>, in upgrade variant 104A can allow for more efficient generation of telemetry data <NUM> which may lead to more rapid testing of upgrade variants 104A.

Proceeding to <FIG>, an example graphical user interface (GUI) <NUM> illustrates how telemetry data <NUM> and analysis can be presented to a feature group <NUM>. As illustrated, feature group GUI <NUM> can present details of an active deployment for a set of upgrade variants 104A-104D such as an upgrade type (e.g., "Operating System Version Update"), a targeted resource unit <NUM> ("Object: Virtual Machines") as well as a target configuration ("OS Version <NUM>. Feature group GUI can additionally provide selectable elements <NUM>-<NUM> that correspond to individual upgrade variants 104A-104D which a user may select to view additional details and analysis for a particular upgrade variant 104A. It should be understood that the system can receive the selection of an element <NUM>-<NUM> by way of any suitable user input including but not limited to a touch-sensitive display, a pointing device such as a mouse or stylus and the like.

For example, as shown in <FIG>, a user has selected element <NUM> to view additional details for upgrade variant 104A. In response, selectable element <NUM> has expanded to inform the user that no problems have been detected. The expanded element <NUM> can also present performance data <NUM> based on analysis of telemetry data <NUM> such as a performance score ("<NUM>") and performance ranking ("<NUM>"). In addition, the expanded element <NUM> can inform the user of the various metrics that were used to calculate the performance score.

In another example, <FIG> illustrates expanded element <NUM> alerting feature group <NUM> to a problem that has occurred at resource units 106A. In this example, one or more virtual machines that executed upgrade variant 104C have become unresponsive. Expanded element <NUM> can also provide an error code ("ClientConnectionFailure") to assist feature group <NUM> in determining a root cause of the error. In this example, the virtual machines that executed upgrade variant 104C are failing to connect to a network. Accordingly, expanded element includes an action recommendation <NUM> proposing that feature group <NUM> disable upgrade variant 104C.

Turning now to <FIG>, aspects of a routine for enabling small-scale deployment of resource units upgrade variants 104A-104D are shown and described. For ease of understanding, the processes discussed in this disclosure are delineated as separate operations represented as independent blocks. However, these separately delineated operations should not be construed as necessarily order dependent in their performance. The order in which the process is described is not intended to be construed as a limitation, and any number of the described process blocks may be combined in any order to implement the process or an alternate process. Moreover, it is also possible that one or more of the provided operations is modified or omitted.

The particular implementation of the technologies disclosed herein is a matter of choice dependent on the performance and other requirements of a computing device. Accordingly, the logical operations described herein are referred to variously as states, operations, structural devices, acts, or modules. These states, operations, structural devices, acts, and modules can be implemented in hardware, software, firmware, in special-purpose digital logic, and any combination thereof. It should be appreciated that more or fewer operations can be performed than shown in the figures and described herein. These operations can also be performed in a different order than those described herein.

It also should be understood that the illustrated methods can end at any time and need not be performed in their entireties. Some or all operations of the methods, and/or substantially equivalent operations, can be performed by execution of computer-readable instructions included on a computer-storage media, as defined below. The term "computer-readable instructions," and variants thereof, as used in the description and claims, is used expansively herein to include routines, applications, application modules, program modules, programs, components, data structures, algorithms, and the like. Computer-readable instructions can be implemented on various system configurations, including single-processor or multiprocessor systems, minicomputers, mainframe computers, personal computers, hand-held computing devices, microprocessor-based, programmable consumer electronics, combinations thereof, and the like.

Thus, it should be appreciated that the logical operations described herein are implemented (<NUM>) as a sequence of computer implemented acts or program modules running on a computing system and/or (<NUM>) as interconnected machine logic circuits or circuit modules within the computing system. The implementation is a matter of choice dependent on the performance and other requirements of the computing system. Accordingly, the logical operations described herein are referred to variously as states, operations, structural devices, acts, or modules. These operations, structural devices, acts, and modules may be implemented in software, in firmware, in special purpose digital logic, and any combination thereof.

For example, the operations of the routine <NUM> are described herein as being implemented, at least in part, by modules running the features disclosed herein can be a dynamically linked library (DLL), a statically linked library, functionality produced by an application programing interface (API), a compiled program, an interpreted program, a script or any other executable set of instructions. Data can be stored in a data structure in one or more memory components. Data can be retrieved from the data structure by addressing links or references to the data structure.

Although the following illustration refers to the components of the figures, it should be appreciated that the operations of the routine <NUM> may be also implemented in many other ways. For example, the routine <NUM> may be implemented, at least in part, by a processor of another remote computer or a local circuit. In addition, one or more of the operations of the routine <NUM> may alternatively or additionally be implemented, at least in part, by a chipset working alone or in conjunction with other software modules. In the example described below, one or more modules of a computing system can receive and/or process the data disclosed herein. Any service, circuit or application suitable for providing the techniques disclosed herein can be used in operations described herein. With reference to <FIG>, routine <NUM> begins at operation <NUM> where a system receives a plurality of upgrade variants from a feature group for deployment to resource units. As discussed above, resource units can include various low-level infrastructure objects such as virtual machines, physical machines, network devices, containers and the like.

Next, at operation <NUM>, the system selects a set of resource units from a plurality of resource units. At operation <NUM>, the system collects telemetry data from the set of resource units that received the upgrade variants.

Proceeding to operation <NUM>, the system then analyzes the telemetry data to detect whether problems have occurred at the selected set of resource units. As discussed above, a problem can include unresponsive resource units, application crashes, degraded performance, high latency, abnormal resource usage, and the like.

Next, at operation <NUM>, the system can identify upgrade variants that caused the problems.

At subsequent operation <NUM>, the system can modify the upgrade to isolate the identified variants. As discussed above, isolating variants can include removing the variant from deployment.

Finally at subsequent operation <NUM>, the system can deploy the upgrade variants, excluding the identified variant, to the plurality of resource units.

<FIG> shows additional details of an example computer architecture <NUM> for a device, such as a computer or a server configured as part of the system <NUM>, capable of executing computer instructions (e.g., a module or a program component described herein). The computer architecture <NUM> illustrated in <FIG> includes processing unit(s) <NUM>, a system memory <NUM>, including a random-access memory <NUM> ("RAM") and a read-only memory ("ROM") <NUM>, and a system bus <NUM> that couples the memory <NUM> to the processing unit(s) <NUM>.

Processing unit(s), such as processing unit(s) <NUM>, can represent, for example, a CPU-type processing unit, a GPU-type processing unit, a field-programmable gate array (FPGA), another class of digital signal processor (DSP), or other hardware logic components that may, in some instances, be driven by a CPU. For example, and without limitation, illustrative types of hardware logic components that can be used include Application-Specific Integrated Circuits (ASICs), Application-Specific Standard Products (ASSPs), System-on-a-Chip Systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc..

A basic input/output system containing the basic routines that help to transfer information between elements within the computer architecture <NUM>, such as during startup, is stored in the ROM <NUM>. The computer architecture <NUM> further includes a mass storage device <NUM> for storing an operating system <NUM>, application(s) <NUM>, modules <NUM>, and other data described herein.

The mass storage device <NUM> is connected to processing unit(s) <NUM> through a mass storage controller connected to the bus <NUM>. The mass storage device <NUM> and its associated computer-readable media provide non-volatile storage for the computer architecture <NUM>. Although the description of computer-readable media contained herein refers to a mass storage device, it should be appreciated by those skilled in the art that computer-readable media can be any available computer-readable storage media or communication media that can be accessed by the computer architecture <NUM>.

Computer-readable media can include computer-readable storage media and/or communication media. Computer-readable storage media can include one or more of volatile memory, nonvolatile memory, and/or other persistent and/or auxiliary computer storage media, removable and non-removable computer storage media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data. Thus, computer storage media includes tangible and/or physical forms of media included in a device and/or hardware component that is part of a device or external to a device, including but not limited to random access memory (RAM), static random-access memory (SRAM), dynamic random-access memory (DRAM), phase change memory (PCM), read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, compact disc read-only memory (CD-ROM), digital versatile disks (DVDs), optical cards or other optical storage media, magnetic cassettes, magnetic tape, magnetic disk storage, magnetic cards or other magnetic storage devices or media, solid-state memory devices, storage arrays, network attached storage, storage area networks, hosted computer storage or any other storage memory, storage device, and/or storage medium that can be used to store and maintain information for access by a computing device.

In contrast to computer-readable storage media, communication media can embody computer-readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave, or other transmission mechanism. As defined herein, computer storage media does not include communication media. That is, computer-readable storage media does not include communications media consisting solely of a modulated data signal, a carrier wave, or a propagated signal, per se.

According to various configurations, the computer architecture <NUM> may operate in a networked environment using logical connections to remote computers through the network <NUM>. The computer architecture <NUM> may connect to the network <NUM> through a network interface unit <NUM> connected to the bus <NUM>. The computer architecture <NUM> also may include an input/output controller <NUM> for receiving and processing input from a number of other devices, including a keyboard, mouse, touch, or electronic stylus or pen. Similarly, the input/output controller <NUM> may provide output to a display screen, a printer, or other type of output device.

It should be appreciated that the software components described herein may, when loaded into the processing unit(s) <NUM> and executed, transform the processing unit(s) <NUM> and the overall computer architecture <NUM> from a general-purpose computing system into a special-purpose computing system customized to facilitate the functionality presented herein. The processing unit(s) <NUM> may be constructed from any number of transistors or other discrete circuit elements, which may individually or collectively assume any number of states. More specifically, the processing unit(s) <NUM> may operate as a finite-state machine, in response to executable instructions contained within the software modules disclosed herein. These computer-executable instructions may transform the processing unit(s) <NUM> by specifying how the processing unit(s) <NUM> transition between states, thereby transforming the transistors or other discrete hardware elements constituting the processing unit(s) <NUM>.

<FIG> depicts an illustrative distributed computing environment <NUM> capable of executing the software components described herein. Thus, the distributed computing environment <NUM> illustrated in <FIG> can be utilized to execute any aspects of the software components presented herein. For example, the distributed computing environment <NUM> can be utilized to execute aspects of the software components described herein.

Accordingly, the distributed computing environment <NUM> can include a computing environment <NUM> operating on, in communication with, or as part of the network <NUM>. The network <NUM> can include various access networks. One or more client devices 806A-806N (hereinafter referred to collectively and/or generically as "clients <NUM>" and also referred to herein as computing devices <NUM>) can communicate with the computing environment <NUM> via the network <NUM>. In one illustrated configuration, the clients <NUM> include a computing device 806A such as a laptop computer, a desktop computer, or other computing device; a slate or tablet computing device ("tablet computing device") 806B; a mobile computing device 806C such as a mobile telephone, a smart phone, or other mobile computing device; a server computer 806D; and/or other devices 806N. It should be understood that any number of clients <NUM> can communicate with the computing environment <NUM>.

In various examples, the computing environment <NUM> includes servers <NUM>, data storage <NUM>, and one or more network interfaces <NUM>. The servers <NUM> can host various services, virtual machines, portals, and/or other resources. In the illustrated configuration, the servers <NUM> host virtual machines <NUM>, Web portals <NUM>, mailbox services <NUM>, storage services <NUM>, and/or, social networking services <NUM>. As shown in <FIG> the servers <NUM> also can host other services, applications, portals, and/or other resources ("other resources") <NUM>.

As mentioned above, the computing environment <NUM> can include the data storage <NUM>. According to various implementations, the functionality of the data storage <NUM> is provided by one or more databases operating on, or in communication with, the network <NUM>. The functionality of the data storage <NUM> also can be provided by one or more servers configured to host data for the computing environment <NUM>. The data storage <NUM> can include, host, or provide one or more real or virtual datastores 826A-826N (hereinafter referred to collectively and/or generically as "datastores <NUM>"). The datastores <NUM> are configured to host data used or created by the servers <NUM> and/or other data. That is, the datastores <NUM> also can host or store web page documents, word documents, presentation documents, data structures, algorithms for execution by a recommendation engine, and/or other data utilized by any application program. Aspects of the datastores <NUM> may be associated with a service for storing files.

The computing environment <NUM> can communicate with, or be accessed by, the network interfaces <NUM>. The network interfaces <NUM> can include various types of network hardware and software for supporting communications between two or more computing devices including, but not limited to, the computing devices and the servers. It should be appreciated that the network interfaces <NUM> also may be utilized to connect to other types of networks and/or computer systems.

It should be understood that the distributed computing environment <NUM> described herein can provide any aspects of the software elements described herein with any number of virtual computing resources and/or other distributed computing functionality that can be configured to execute any aspects of the software components disclosed herein. According to various implementations of the concepts and technologies disclosed herein, the distributed computing environment <NUM> provides the software functionality described herein as a service to the computing devices. It should be understood that the computing devices can include real or virtual machines including, but not limited to, server computers, web servers, personal computers, mobile computing devices, smart phones, and/or other devices. As such, various configurations of the concepts and technologies disclosed herein enable any device configured to access the distributed computing environment <NUM> to utilize the functionality described herein for providing the techniques disclosed herein, among other aspects.

It should be appreciated that any reference to "first," "second," etc. elements within the Summary and/or Detailed Description is not intended to and should not be construed to necessarily correspond to any reference of "first," "second," etc. elements of the claims. Rather, any use of "first" and "second" within the Summary, Detailed Description, and/or claims may be used to distinguish between two different instances of the same element (e.g., two different variants, two different upgrades, etc.).

Claim 1:
A method (<NUM>) for deploying an upgrade (<NUM>) to a network computing environment that comprises a plurality of resource units (106A-106N), the method comprising:
receiving (<NUM>) a plurality of variants (104A-104D) of the upgrade (<NUM>), wherein one or more features of each variant (104A) of the plurality of variants (104A-104D) is different from other variants (104B) of the plurality of variants (104A-104D);
selecting (<NUM>) a set of resource units (106A) from the plurality of resource units (106A-106N);
deploying (<NUM>) the plurality of variants (104A-104D) of the upgrade (<NUM>) to the selected set of resource units (106A), wherein deploying the plurality of variants (104A-104D) comprises deploying multiple test upgrades (<NUM>), each test upgrade having a differing set of variants (104A-104D), and wherein each test upgrade is deployed to a different subset of resource units from the set of resource units (106A);
collecting (<NUM>), by one or more processing units, telemetry data (<NUM>) from the plurality of variants (104A-104D) at the set of resource units (106A);
analyzing (<NUM>) the telemetry data (<NUM>) to detect that a problem has occurred at the set of resource units (106A) in response to execution of the plurality of variants of the upgrade (104A-104D);
identifying (<NUM>), based at least in part on the analysis (<NUM>), a variant (104C) that caused the problem that occurred at the set of resource units (106A);
modifying (<NUM>) the upgrade (<NUM>) to isolate the identified variant (104C); and
deploying (<NUM>) the plurality of variants of the upgrade (<NUM>), excluding the identified variant (104C), to the plurality of resource units (106A-106N).