Cloud computing benchmarking

Cloud computing benchmarking is performed wherein the resource usage of a measuring benchmarking application is compensated for as to not impact measurement. The measurements are of a cloud instance's benchmarking indicia which may include performance, functions and characteristics of the cloud instance. The benchmarking indicia use scalable measures as to allow the use of arithmetic operations such as those used in statistical functions. The benchmarking application is dispatched along with a configuration file and is controlled from a central controller to specified cloud instances. The dispatched benchmarking application takes measurements of the cloud instance based on the configuration file. The benchmarking application then stores the measurements in a results file for return back to the central controller. At the central controller, results files from one or more benchmarking applications are stored in a data store for comparative and statistical analysis.

BACKGROUND

Enterprises and other companies may reduce information technology (“IT”) costs by externalizing hardware computing costs, hardware maintenance and administration costs, and software costs. One option to externalize IT costs is by purchasing cloud computing processing and hosting from a third party cloud computing provider. Cloud computing providers purchase and maintain computer servers typically in server farms, and act as a utility company by reselling their computing capacity to customers. Some customers may be value added resellers (“VARs”), that are software companies who host their software applications on computing capacity from cloud providers. These VARs then make money by selling access to their software applications to customers. In this way, cloud computing providers directly externalize hardware computing costs and hardware maintenance costs, and indirectly externalize software costs by providing a hosting platform for VARs.

Cloud computing providers typically add infrastructure services, that provide common services for the cloud provider. Some infrastructure services are operating system-like services that control allocation of services of the cloud. For example, physical servers in server farms are typically disaggregated and resold in unitary blocks of service in the form of processing power, memory, and storage. Specifically, a unitary block is some unit to inform a customer of the volume of computing capacity purchased from a cloud provider. Consider a customer that purchases a unitary block of denoted, for example, one “virtual processor”. That customer may in fact be purchasing processing power where the virtual process is provided by different cores on a processor, different processors on the same physical server, or potential processing cores on different physical servers. The unitary block measuring computer service is proffered by the vendor, rather than a third party operating at arm's length.

Other infrastructure services provide services that support the cloud provider business model. For example, cloud providers typically provide different billing options based on metering a customer's usage on the cloud. A billing infrastructure is an example of an infrastructure service that supports the cloud provider business model. However, metering, service level agreements, and ultimately billing are often provided in terms of a vendor's chosen unitary measure.

Accordingly, customers are obliged to independently verify vendor claims about the unitary measure, or alternatively simply take the vendor at their word. Thus customers are faced with evaluating cloud provider claims without a ready point of reference.

Verification of claims about unitary services is not trivial. Cloud providers use infrastructure services as competitive differentiators to attract customers and VARs. For example, yet other infrastructure services provide abstractions that facilitate application development and hosting on the cloud. Well known examples include Platform-as-a-Service (“PAAS”), Infrastructure-as-a-Service (“IAAS”) and Software-as-a-Service (“SAAS”) hosting and development infrastructure.

Thus additionally, customers who seek to compare cloud providers are faced with evaluating different hardware configurations, different software configurations, and different infrastructure services, often without transparency to the operation of different cloud providers.

DETAILED DESCRIPTION

Cloud Computing and Benchmarking

Measurement and Benchmarking

The present disclosure describes benchmarking from the perspective of benchmarking cloud computing. Before discussing benchmarking cloud computing, the present disclosure will describe some preliminaries regarding benchmarking.

Benchmarking is the selection of one or more indicia that are used to compare one item to another or one item to an idealized version of that item. In the case of computer science, common comparative indicia may include software performance, hardware performance, overall system performance. For example volume of data processed, number of faults, and memory usage may be candidate metrics for benchmarking software performance. A particular software implementation may be compared to a competing implementation. Alternatively, the software implementation might be compared to the theoretical optimum values of those metrics. Regardless of what metrics are chosen, the aggregating of those chosen metrics constitutes benchmarking.

Since the indicia chosen to constitute a benchmark are used for comparisons, the indicia chosen are to be based on a measure. A measure is sometimes called a distance function that is a value based on a comparison. Measure can be categorized by their behavior upon comparing measure values, called measurements, against each other. Measures may come in the following four categories.

i. Different Categories

Indicia may be placed in different categories. Here, the indicia indicates what kind of item, something is. It does not indicate whether something is better or worse than another item. Rather it simply indicates that it is different and should be treated and/or evaluated differently. For example, a cloud infrastructure service might be classified as PAAS, IAAS, or SAAS. None of the three options are necessarily better or worse, rather just in different categories.

ii. Ordered Categories

Indicia may be placed in ordered categories. Here, the categories have a clear order as to which categories is more desirable. Typically the categories are ordered in monotonically increasing order, such as from worst to best. For example, customer satisfaction with a cloud vendor might be classified from “bad”, “average”, “good” and “excellent.” Therefore, a cloud vendor classified as “excellent” might be considered better than another classified as “average.” However, there is no indication of degree of how much better an “excellent” vendor is over another that is merely “average.”

iii. Additive Categories

Indicia may be additive. Additive indicia allow multiple measurements to be aggregated into a single measurement, where order is preserved. For example, number of processors on a server for parallel processing is additive. Two processors generally are able to do more processing than one processor. However, two processors are not necessarily able to do twice as much processing as one processor, due to communications overhead and/or the possibility of the processors being heterogeneous. So additive indicia do not scale.

Indicia may be scalable. Not only are scalable indicia additive, scalable indicia support all arithmetic operations including multiplication and division. For example, megaflops per second (“MFLOPS”) is an indicium that is a scalable measure. A processor that can perform 2,500 MFLOPS is two and half times as powerful as a processor that can perform 1,000 MFLOPS.

Additive and scalable measures are sometimes called metrics, because the distance function comprising the measure satisfies the mathematical properties of separation, coincidence, symmetry and the triangle inequality. Regarding the latter, a measure satisfies the triangle inequality if the measurements between A and C is greater than or equal to the measurement between A and B added to the measurement between B and C. Expressed mathematically, F(x, y) satisfies the triangle inequality if:
F(A,C)≤F(A,B)+F(B,C).

Metrics provide the basis for performing statistical functions, many of which are based on arithmetic operations. Accordingly, metrics are desirable measures, because they enable statistical techniques to be brought to bear during analysis. For example, consider the function for a standard deviation:

The standard deviation function is comprised of square roots and exponents which use multiplication, summations which use addition, averages which use division, and the like. Thus the standard deviation function is mathematically and statistically meaningful where a metric is used as a measurement.

Goals in Benchmarking Cloud Computing

Turning to the application of benchmarking to cloud computing, there are several potential cloud provider evaluation goals that are driven by business operations. The evaluation goals may include a potential business decisions to:move to an alternative cloud provider;evaluate a service design of a cloud provider;verify continuity of service from a cloud provider over time;verify consistency of service over different service/geographic zone for a cloud provider;verify a cloud provider can support a migration to that cloud provider;enable service/price comparisons between different cloud providers;verify terms of a service level agreement are satisfied;evaluate performance times hibernation and re-instantiation by services of a cloud provider;performance; andevaluate and validate service change management in a cloud provider.

These evaluation goals may be achieved by identifying and selecting indicia to comprise a benchmark. The indicia may support simple difference comparisons, between one or more systems. Alternatively, the indicia may provide the basis to define a measure in terms of one or more normalized units to make baseline measurements. Defining a normalized unit that supports a metric enables bringing not only direct comparisons, but also statistical techniques to support a comprehensive evaluation.

The selected indicia are chosen on the basis of either being an indicium of a cloud provider's performance, functionality, or characteristics, known collectively as a PFC. Performance indicia are artifacts that indicate how a cloud provider performs under a work load, for example processor usage percentage. Functionality includes computing features that are available from the cloud provider, for example a maximum of 4 GB memory available to a virtual server instance. Characteristics differentiate categories for cloud providers, such as type of billing model. The selected indicia may be measured with varying frequency. In some situations, a single measurement may be made over the lifetime of a benchmarking cycle. In others, multiple measurements are made either periodically, according to a predetermined schedule, or upon detecting an event or condition.

Cloud computing benchmarks may comprise indicia that allow for the aggregation of measurements over time. Specific indicia may be selected to continuously, periodically, or at selected intervals measure and track the overall performance capability over time. This enables the development of complex algorithms which may include for example the overall performance capabilities across systems; the impact of multiple demands on a system; impact to the system's capabilities; and their respective trend over time. A specific benchmark may be to capture the processor maximum performance over time, to capture the network throughput over time and to combine these measures based on a workload demand to generate a predictive model of what the maximum processor capability is given a variable network throughput. While this benchmark example outlines two indicia, by definition, the overall performance capability will be impacted by all of the demand on the cloud provider. Thus, the measurement of indicia is enhanced by the temporal view that enables adaptive and predictive modeling based on customer defined indicia.

Potential indicia include indicia in the following categories.

The compute category covers information about the physical and/or virtual processor cores used by servers in a cloud provider. In general, computing processors are known as computing processing units (“CPUs”). The following table lists potential indicia in the compute category.

The memory category covers information about the physical and/or virtual (swap) random access memory (“RAM”) used by servers in a cloud provider. The following table lists potential indicia in the memory category.

TABLE 2Memory IndiciaUpdateIndiciaDescriptionFrequencyPFC Testtotal RAMHow much RAM is allocatedonceFunctionalityto the server(Validation Test)total swapHow much disk space isonceFunctionalityallocated for swap space(Validation Test)allocatedHow much of the system'sfrequentPerformancememorymemory is currently in use(Stress Test)page faultsNumber of times that afrequentFunctionalityprocess requested something(Validation Test)from RAM but it had to beretrieved from swapmemoryTotal/Allocated/free statisticfrequentPerformanceusagefor RAM and swap(Stress Test)top 5processes using the mostfrequentFunctionalitymemorymemory(Validation Test)hogsqueue sizeAmount of RAM devoted tofrequentFunctionalitydata for processes that are not(Validation Test)currently active

The disk category covers information about the storage media available via the operating system or disk drives used by servers in a cloud provider. The following table lists potential indicia in the disk category.

TABLE 3Disk IndiciaUpdateIndiciaDescriptionFrequencyPFC Testtotal capacityHow much disk space isonceFunctionality(per fileallocated to the server(Validationsystem)Test)used capacityHow much disk space is usedfrequentFunctionality(per fileby the system(Validationsystem)Test)disk writes/secHow many disk writes can be/frequentPerformancehave been performed in a(Stress Test)seconddisk reads/secHow many disk reads can be/frequentPerformancehave been performed in a(Stress Test)secondpermissionscheck permissions to ensurefrequentFunctionalitythat applications have the(Validationproper amount of permissionsTest)to act and that permissions forcritical files have not changedIOWAIT timeProcesses that cannot actfrequentPerformance(input/outputbecause they are waiting for(Stress Test)wait time)disk read/write

iv. Operating System

The operating system (“OS”) category covers information about the operating system used by servers in a cloud provider. The following table lists potential indicia in the operating system category.

TABLE 4Operating System IndiciaUpdateIndiciaDescriptionFrequencyPFC TestsVersionWhat OS Version isonceFunctionalityrunning on the system(ValidationTest)kernel parametersAny changes in kernelfrequentFunctionalityparameters(ValidationTest)scrape the bootInformation gatheredfrequentFunctionalityscreenfrom the console logs(Validationduring system bootTest)check syslog forCheck the console logsdailyFunctionalityerrorsand other system logs for(ValidationerrorsTest)context switchingHow much time havefrequentPerformancetime (to go fromprocesses spent switching(Stress Test)user to kernelfrom user application tomode)OS kernel modenumber of runningCount of runningfrequentPerformanceprocessesprocesses(Stress Test)zombie processesChild processes that didfrequentFunctionalitynot terminate when the(Validationparent process terminatedTest)

The network category covers information about the server's connection to its local area network (“LAN”) and to the Internet for servers in a cloud provider. The following table lists potential indicia in the network category.

The database (“DB”) category covers information about a structured query language (“SQL”) or noSQL database management system (“DBMS”) application running on servers in a cloud provider. The following table lists potential indicia in the database category.

TABLE 6Database IndiciaUpdateIndiciaDescriptionFrequencyPFC TestsDatabaseType and Version of theonceFunctionalityversionrunning database system(ValidationTest)DB writesTime to write a transaction offrequentPerformancelocalknown size to the DB on the(Stress Test)localhostDB writesTime to write a transaction offrequentPerformanceover IPknown size from a known(Stress Test)external host to the DB on thelocalhostDB readsTime to read a transaction offrequentPerformancelocalknown size from the DB on(Stress Test)the localhostDB reads overTime to read a transaction offrequentPerformanceIPknown size to a known(Stress Test)external host from the DB onthe localhostDBTime to perform a knownfrequentPerformancecalculationmath calculation within the(Stress Test)databasegrowth rate ofCheck the current size of thefrequentFunctionalitythe DB dataDB fdes, including raw(Validationfilesdatafde/partition size, rowTest)count, etc.

vii. Cloud Provider

The cloud category covers information about the cloud provider in which the server is instantiated. In some cases, the indicia may be in terms of a normalized work load unit. The following table lists potential indicia in the cloud provider category.

TABLE 7Cloud IndiciaUpdateIndiciaDescriptionFrequencyPFC TestsLoad unitDetect when a load unitfrequentFunctionalitymeasurementsmeasurement check is(Validationfrom serverdelayed or missing from aTest)stoppedgiven serverrespondingprovisioningTime to create a new serverfrequentPerformancespeed CPUinstance of a given size in a(Stress Test)given availability zone (e.g.by creating a tailored area ofmutual interest (AMI) toprovision identical machinesand report back aboutprovisioning time)ProvisioningTime to create new storagefrequentPerformancespeed Storage(Stress Test)migrate serverTime to create a snapshot andfrequentPerformanceto anotherclone the instance of a server(Stress Test)datacenterin a different availability zoneclusterInformation about otherfrequentFunctionalityinformationservers related to this one,(Validationlike server farms, databaseTest)clusters, application rings
Cloud Computing Benchmarking Issues

Selection of indicia for a benchmark may be driven by the consumer of the benchmark. A basis for a benchmark to be accepted by a consumer is that the consumer trusts the measurement. There are several factors that may affect the trust of a measurement.

i. The Observation Problem aka Heisenberg

The act of observing a system will affect a system. When a measurement consumes computing resources as to affect the observable accuracy of a measurement, the measurement will not be trusted. This problem is also known as the “Heisenberg” problem. In the case of cloud computing, a benchmarking application running within a cloud instance will use processing, memory, and network resources. In particular, since cloud communications are typically geographically disparate, network latency during measurement may have a significant adverse impact on measurement accuracy. Furthermore, cloud infrastructure services often have sophisticated “adaptive” algorithms that modify resource allocation based on their own observations. In such situations, it is very possible that a benchmarking application may become deadlocked.

One approach is to guarantee performance overhead of a benchmarking application to be less than some level of load/processing core overhead. Measurements would be compared only on like systems. For example a Windows™ based platform would not necessarily be compared to a Linux platform. Also, memory and network overhead could be managed by carefully controlling collected data is transferred. For example, benchmark data may be cached on a local disk drive and will transfer upon an event trigger such as meeting a predetermined threshold to limit disk load. Since data transfer potentially creates network load, data may be transferred upon receiving a transfer command from a remote central controller.

Another approach may be to understand the statistical behavior of the system to be benchmarked. If an accurate statistics model is developed, then a statistically small amount of benchmarking data may be collected, and the measurement projected by extrapolation based on the statistics model. For example, a workload over time model may be developed where an initial measurement is made at the beginning of benchmarking. Since the initial measurement theoretically occurs before any additional workload, that initial measurement may be used as a theoretical processing maximum to compare subsequent measurements against.

Statistical models may be comprised where a cloud provider has infrastructure services that are adaptive. For example, a measurement at time T0may not be comparable at time Tnif the cloud provider silently reconfigured between the two times. However, properly designed normalized unit should continue to be a normalized unit. Thus even if measurements may not be consistently comparable, the performance changes may be detected over time. Thus the adaptations of the cloud infrastructure and the triggers for those adaptations may be detected, and the benchmarking application may be configured to avoid those triggers or to compensate.

Yet another approach is to limit benchmarking under predetermined conditions. Some conditions are detected prior to benchmarking, and other conditions are detected during benchmarking. Regarding the former, given that the benchmarking application can negatively impact its environment, the central controller may have an “emergency stop” button customer that halts at least some of the benchmarking on at least some cloud provider instances under test. For example, a configuration file received by the benchmarking application may contain a “permit to run” flag. Before starting benchmarking, the benchmarking application may poll the central controller for the most recent configuration file. If there have been no changes the benchmarking application may receive a message indicating that the configuration file has not changed along with a set “permit to run” flag, and that the benchmarking application is permitted to start benchmarking. In this case, the benchmarking application will use the present configuration file and commence benchmarking. If the “permit to run” flag is not set, then the benchmarking application will not commence testing. In case where the benchmarking application cannot communicate with the central controller, the benchmarking application may default to not benchmarking and will assume the “permit to run” flag is not set. Regarding the detecting of conditions during benchmarking, the benchmarking application may gather at least some environment data for the cloud provider instance under test. If the benchmarking application detects that the environment data satisfies some predetermined condition, such as some or all of the current environment data being in excess of a predetermined level, then the benchmarking application may prevent benchmarking from starting.

Note that the benchmarking application under operation would only effect performance data collection, if at all. Thus functionality and characteristic data may continue to be collected without compromising the cloud performance instance under test.

ii. Meaningful Statistics

Books have been written about how to characterize statistics. For some, the risk is that the consumer is overly credulous when confronted with statistics, and may conflate the reception of statistics with a full analysis in making a business decision. For others, the risk is that the consumer has been exposed to shoddy statistical analysis, and may be overly suspicious of all statistics. Benchmarking trustworthiness may be based on some of the following factors: the results are verifiable, the methodology is transparent and verifiably accurate, and the methodology is repeatable.

Consumer trust may be engendered by methodology transparency. For example, reporting may clearly indicate that a statistically significant amount of data has not yet been collected when reporting a benchmark. One way to ensure statistical significance is to take an initial measurement at the beginning of benchmarking and to track frequency/periodicity and timing of data sampling. Alternatively, reporting may indicate a confidence level, potentially calculated by the sampling frequency/periodicity and timing data. In this way, the consumer's desire for immediate data may be balanced against potential inaccuracies.

In addition to transparency, benchmarking may be performed by trusted third parties. Past benchmarks have been “gamed” by vendors, where the vendor implemented features specifically to optimize benchmark reports, without commensurate genuine improvements. While vendors may continue to game benchmarks, having a trusted third party owning the benchmarking infrastructure allows that third party to independently verify results, and modify the benchmarks as vendor gaming is detected.

Benchmarking is ideally repeatable. In other words, the performance reported by a benchmark should be similar to a separate test under similar test conditions. In general, samplings of indicia or benchmarking may be time/stamped. Accordingly, arbitrary time sets may be compared to each other in order to determine whether the benchmarking results were repeatable.

Benchmarking data and performance data are inherently sensitive. Cloud providers and VARs will not like poor performance results to be publicized. Furthermore, the integrity of the benchmarking system has to be protected from hackers, lest the collected results be compromised.

Security is to be balanced against processing overhead giving rise to a Heisenberg observation problem. For example, cryptography key exchange with remote key servers gives rise to network load. Such measurements may render at least network measurements inaccurate. However, sensitive data is ideally encrypted. Encryption overhead may be minimized by selectively encrypting only the most sensitive data and/or by encrypting portions of the data.

By way of an example, a benchmarking application may include a configuration file that may define the behavior of that benchmarking application. Therefore, the configuration file is to be delivered securely so that it is not a point of insertion for rogue instructions that would put the benchmarking operation at risk. The configuration file may be encrypted and/or make use of message digests to detect tampering. Hash algorithms and/or security certificates may be used to allow the benchmarking application to validate the configuration file prior to any benchmarking. For example, a configuration file may be identified as work only with a specified target cloud provider instance identifier, a version identifier, a time stamp, and a security identifier. The benchmarking application may be configured to only load and/or execute the configuration file only if some predetermined subset of these identifiers, or if all of these identifiers are validated and authorized.

Since the benchmarking application has not begun benchmarking prior to receiving and validating the configuration file, any network load from accessing key servers is not measured, and therefore will not cause a Heisenberg observation problem.

Note that the security of benchmarking is not the same as testing the security of the cloud provider. However, security testing of the cloud provider may be a function of the benchmarking application. Part of benchmarking applications capabilities may be to adapt its measurements based on an understanding of the relationship between both latency and security service checks. An initial benchmark measurement and can be validated across a number of clouds to identify the difference between the latency for a non-secure transaction and the latency for a security impacted latency for secure transactions. This difference may then be factored into the ongoing tests to confirm consistent performance.

Context of Cloud Computing Benchmarking

FIG. 1is an exemplary context diagram for a cloud computing benchmarking infrastructure100.

The cloud computing benchmarking infrastructure100may comprise a central controller102. The central controller102may be local or remote to the cloud provider. For example, where the central controller102may be guaranteed to be in the same server cluster as the cloud provider instance under test, it may be desirable to host the central controller102locally as to reduce network latency. However, the central controller102may be located on a remote computer to provide a single point of control where multiple cloud provider instances are to be tested.

Central controller102may comprise a controller application104a data store108to store benchmarks, benchmarking results, configuration files, and other related data for cloud computing benchmarking. For example, in addition to storing benchmarking results and collected raw indicia data, the central controller102may perform comparative reporting and statistics, or other automated analysis, and store that analysis on data store108.

The cloud computing benchmarking infrastructure100may benchmark enterprise servers110on a local area network (“LAN”). Alternatively, cloud computing benchmarking infrastructure100may benchmark one or more clouds112,114. Note that clouds112,114need not be the same type of cloud. For example, cloud112may be a PAAS infrastructure and cloud114may be a SAAS infrastructure. Communications connections between the central controller102and enterprise servers110and clouds112and114may be effected via network connections116,118,120respectively.

Network connections116,118,120may be used to send/install a benchmarking application122on enterprise servers110and/or clouds112,114.

Once benchmarking application122is installed, the benchmarking application122may request a configuration file124indicating which PFC are to be collected may be sent to enterprise servers110and/or clouds112from central controller102. Accordingly, the benchmarking application122may operate on a pull basis. Alternatively, central controller102may push a configuration file124to enterprise servers110and/or clouds112.

Periodically, benchmarking application122may send benchmarking data results126back to the central controller102for storage in data store108. The sending may be based on a predetermined condition being detected, such as benchmarking completing. Alternatively, the central controller102may affirmatively request some or all of the benchmarking data results126.

The central controller102may affirmatively send commands130to the benchmarking application122. For example, it may send a “permit to run” flag set to “on” or “off” In the latter case, the benchmarking application may stop upon reception of command130.

Exemplary Hardware Platform for Cloud Computing Benchmarking

FIG. 2illustrates one possible embodiment of a hardware environment200for cloud computing benchmarking.

Client device202is any computing device. A client device202may have a processor204and a memory206. Client device202's memory206is any computer-readable media which may store several programs including an application208and/or an operating system210.

To participate in a communications environment, client device202may have a network interface212. The network interface212may be one or more network interfaces including Ethernet, Wi-Fi, or any number of other physical and data link standard interfaces. In the case where the programming language transformations are to be done on a single machine, the network interface212is optional.

Client device202may use the network interface212to communicate to remote storage214. Remote storage214may include network aware storage (“NAS”) or may be removable storage such as a thumb drive or memory stick.

Client device202may communicate to a server216. Server216is any computing device that may participate in a network. Client network interface212may ultimate connect to server216via server network interface218. Server network interface218may be one or more network interfaces as described with respect to client network interface212.

Server216also has a processor220and memory222. As per the preceding discussion regarding client device202, memory222is any computer-readable media including both computer storage media and communication media.

In particular, memory222stores software which may include an application224and/or an operating system226. Memory222may also store applications224that may include a database management system. Accordingly, server216may include data store228. Data store228may be configured as a relational database, an object-oriented database, and/or a columnar database, or any configuration to support policy storage.

Server216need not be on site or operated by the client enterprise. Server216may be hosted in a cloud230. Cloud230may represent a plurality of disaggregated servers which provide virtual web application server232functionality and virtual database234functionality. Cloud230services232,234may be made accessible via cloud infrastructure236. Cloud infrastructure236not only provides access to cloud services232,234but also billing services. Cloud infrastructure236may provide additional service abstractions such as Platform as a Service (“PAAS”), Infrastructure as a Service (“IAAS”), and Software as a Service (“SAAS”).

Exemplary Architecture for Cloud Computing Benchmarking

FIG. 3is an exemplary detailed system diagram of the example operation of a cloud computing benchmarking infrastructure300.FIG. 3expands on the high level system diagram ofFIG. 1.FIG. 4illustrates a flowchart400of the example operation of cloud computing benchmarking infrastructure300.

Central controller302comprises a computer304hosting a controller application (not shown) and data store306. In the present example, central controller302is to benchmark enterprise server308on a LAN, Cloud A310and Cloud B312.

Clouds A and B310,312may include disaggregated application servers314and disaggregated data storage316either exposed via a file system or database management system. Cloud A310and Cloud B312each expose cloud functionality through their respective infrastructure services318and320.

Central controller302may communicate with enterprise server308, Cloud A310, or Cloud B312via communications connections322,324,326respectively. Over communications connections322,324,326, executables, configuration files, results, commands, and generally arbitrary data328,330,332may be transmitted and received without loss of generality.

In block402ofFIG. 4, the central controller302will initially select one or more cloud provider instances to benchmark. Upon selection, the central controller302identifies the network addresses of the selected cloud provider instances, and dispatches benchmarking applications334,336,338.

While dispatching benchmarking applications334,336,338, in406ofFIG. 4, the central controller302creates data entries in data store306to store and/or index anticipated received results from the dispatched benchmarking applications334,336,338.

Upon arrival, benchmarking applications334,336,338will instantiate. In block408ofFIG. 4, central controller302will dispatch configuration file340,342,344. Specifically, after instantiation, benchmarking applications334,336,338will first determine whether there is configuration file to load. If no configuration file is available, the benchmarking applications334,336,338affirmatively poll central controller302for a configuration file. Central controller302generates configuration files by identifying relevant PFCs for the respective platform. Candidate PFCs are described with respect to Tables 1-7 above.

The configuration file340,342,344provides for separation data and metadata, which enable versioning. This enables for measurements based on a data point to be collected and tied to a particular version and a particular set of applicable predictive models. For each new version, the benchmarking application334,336,338may then validate data for backwards compatibility, and adapts the metadata based on usability. At this point the metadata is assigned and maintained by the central controller102and serialized such that the configuration file340,342,344carries the metadata tag through benchmarking operations to ensure that the data sets are collected and stored with the metadata version for tracking, auditability and certification.

The data is also keyed and/or serialized to a given cloud provider instance where its respective benchmarking application334,336,338is executing, since cloud provider instances are both temporal in location and existence. Several services are activated by benchmarking measurements over time. An example of such a service will be for a cloud provider to use the benchmarking measurements to move workloads between cloud provider instances as to ensure minimize impact to the overall workload. Another example may be the ability to enable hibernation of cloud instances, such as development and test instances, that are only needed sporadically, but may be restarted quickly while ensuring that the restarted instances meet the same benchmarking measurements before. Over time, the benchmarking measurements may enable analyzing service performance trends across interruptions in service,

Additionally, tracking metadata and the cloud computing instance, enables cross correlation of benchmarking measurements both within the same cloud provider and between different cloud providers. For example, two very different customers may select a similar application profile comprised of one or more PFCs and/or indicia. Comparison is only possible if the PCFs and/or indicia are of a common specific test methodology and serialized for analysis against consistent benchmarking algorithms.

The benchmarking applications334,336,338will perform several checks prior to initiating benchmarking. First the benchmarking applications334,336,338authenticate and validate the configuration files340,342,344as described previously. The benchmarking applications334,336,338will then affirmatively poll for a new version from the central controller302. If there is a new version, then the new version is retrieved. Otherwise, a command indicating that the benchmarking is “permitted to run” is dispatched by the central controller302. Furthermore, the benchmarking applications334,336,338will determine if its local environment has sufficient capacity to perform benchmarking. The benchmarking may be in the form of measuring known PFCs. If there is sufficient capacity, then the benchmarking applications334,336,338may instantiate other executables or scripts (not shown) to aid in benchmarking.

Benchmarking applications334,336,338then make an initial PFC and time stamp measurement. This initial PFC measurement provides a baseline for comparing future measurements. During the benchmarking cycle, the benchmarking applications334,336,338may periodically or upon detecting an event take PFC measurements. The measurements are persisted to local storage. When the central controller302requests the results, or when a predetermined condition is satisfied, the benchmarking applications334,336,338transmit at least some of the persisted measurements as results346,348,350back to central control302for storage in data store306.

In block410ofFIG. 4, when central controller302receives results, it may perform store the raw results, or otherwise perform some precalculations of the raw data prior to storing in data store306.

Proceeding to block412ofFIG. 4, benchmarking applications334,336,338eventually detect a condition to stop benchmarking. One condition is that the benchmarking is complete. Another condition is that the benchmarking applications334,336,338have lost communications with central controller302. Yet another condition is the detection that capacity PFCs the local environment benchmarking applications334,336,338exceed a predetermined threshold. Finally, another condition is the reception of a negative “permit to run” flag or a command from the central controller302to cease execution. Upon detecting any of the conditions, in block414ofFIG. 4, benchmarking applications334,336,338stop benchmarking. Optionally, in block416, central control302may verify that the benchmarking applications334,336,338have stopped benchmarking.

CONCLUSION