System, Method and Apparatus for Determining Virtual Machine Performance

A method for a computing apparatus for evaluating a virtualized application is provided. Accordingly, performance characteristics for a virtual machine as well as predetermined criteria based on resource requirements for a proposed virtualized application can be received through the network interface of the computing apparatus. The method also allows for determining, at the computing apparatus, whether the predetermined criteria are satisfied based on the performance characteristics and transmitting instructions to cause the proposed virtualized application to be set up within the virtual machine if the predetermined criteria are satisfied. Once the virtualized application is set up, it can be monitored and if the performance characteristics do not satisfy the virtualized application's performance requirements, the virtualized application can be caused to be moved to a different virtual machine. Performance characteristics of a virtual machine can also be determined on the basis of a parking database.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Referring now toFIG. 1an embodiment of a system for adaptive virtualization is shown at100. System100includes a virtualized application105. The virtualized application105can be any system that can be implemented in a virtual machine (VM). To illustrate system100, an example embodiment will be used where the system being virtualized is assumed to be a voice over IP (VoIP) controller system such as an IP-PBX. It is to be understood that the present example embodiment is for illustration purposes only, and is not a limitation on scope. In other embodiments and variations, virtualized application105can be a database system, a customer relations management systems, a web server or other systems that will now occur to a person of skill in the art that can be implemented as part of a VM.

Continuing withFIG. 1, for a specific virtualized application105to be implemented, the operator of the implemented virtualized application105can develop an operator model110for determining the specific configuration of the virtualized application105needed by the operator. The operator model110, for example, can be developed on the basis of an estimate of the amount, type and level of services required. Accordingly, in an embodiment, the operator model110can be a table of time based estimates of configuration factors for specific time periods during which the system is in use. Configuration factors can be factors such as usage behaviors, use of specific applications and features, maximum size, load and other factors that can be used to specify requirements for a specific system. Configuration factors can include, for example, a maximum size for the system, the features requested, feature options for specific features and the estimated usage for the features and their options.

Continuing with the example embodiment for an IP-PBX operator, an example operator model110can include configuration factors for the number of users of specific types such as the number of automatic call center distribution (ACD) agents, the number of office phones and others, as well as a required quality and quantity of service.

In some embodiments, operator model110can be implemented on any computing apparatus, such as a server, a desktop computer, a virtual machine or any other computing apparatus, individual or distributed, that will now occur to a person of skill in the art. Operator model110can be generated through various operations, both manual and automated, including by rule based model generators and others that will now occur to a person of skill in the art.

Continuing withFIG. 1, a vendor model115is a model provided by a system vendor. The purpose of the vendor model115is to provide estimates for the resources required for specific system configurations under conditions of varying load or capacity needs. Accordingly, a vendor model115is a model that can generate estimates of the server and other resource requirements to meet various possible capacity needs and other service requirements for a specific configuration of the system vended by a vendor. Vendor model115is typically supplied by the vendor along with the vended system to be virtualized.

In some embodiments, vendor model115can be implemented on any computing apparatus, such as a server, a desktop computer, a virtual machine or any other apparatus, individual or distributed, that will now occur to a person of skill in the art. In variations, vendor model115can be resident on computing apparatus operated by the vendor and accessed through a network. In other variations, vendor model115can be resident on computing apparatus operated by the operator, and accessed through a network. Other variations of how the vendor model115can be vended and operated will now occur to a person of skill in the art.

The estimated resources can include, for example, number of servers, computing power, mass storage, short-term storage or local memory, network capacity and network latency. The estimated resources can be specified precisely. For example, it can be specified that 100 megabytes of mass storage is needed. In variations, estimated resources can be specified with varying levels of granularity. For example, mass storage may be specified with a coarse level of granularity such as small, medium and large. Alternatively, the estimated resources can be specified with a finer granularity, such as in increments of 100 million instructions per second (MIPs) for computing power. Other resources that can be estimated and mechanisms of specifying such estimates will now occur to a person of skill in the art and are within scope.

In an embodiment, configuration factors specifying a specific system can be used in cooperation with the vendor model115to estimate resources for that specific system. For example, vendor model115can be used, in cooperation with the operator model110, to estimate required virtualization services needed from a cloud service provider. Accordingly, the operator model110can be used to provide at least some of the configuration factors to vendor model115. In a variation, the operator model110can be used to index into the vendor model115to determine the required resources. In a further variation, the operator model110can contain a translation element that can convert the contents of the operator model110into the syntax and semantics required for communicating with the vendor model115to supply data. In yet a further variation, a table or rule-set or such similar mechanisms can be used to perform the conversion. In other embodiments, at least some of the configuration factors can be derived by the vendor model115based on data obtained from operator model110, which can include some other configuration factors. In further embodiments, operator model110can be linked to multiple different vendor models provided either by the same vendor or by different vendors. Accordingly, different resource estimates generated by each vendor model115can be used in choosing amongst vendors, or different systems provided by the same vendor or both.

In the present example embodiment where the vendor is a vendor of an IP-PBX system, the vendor model identifies the resource requirements for a specific IP-PBX system based on the configuration factors for that specific system as specified, at least in part, by the operator model110and other factors negotiated. In the present example embodiment, configuration factors include a maximum size for the system which is the maximum number of system users. The configuration factors for the example embodiment further include features requested which are the specific user types such as ACD agents, secretarial stations, office phones, and others. The example embodiment configuration factors further include feature options for specific user types such as unified messaging, voice mail and other options specified for each user. The example embodiment configuration factors additionally include an estimated usage for user types such as the number of calls per hour per user. Additional example configuration factors can be based on quality and service levels required such as factors needed for obtaining a certain media quality.

The estimation performed by vendor model115can be multi-dimensional with resource requirements dependent on each of the configuration factors as dimensions. In an embodiment, estimates can be based on the clustering of these dimensions or configuration factors. Accordingly, for any specific system configuration, an estimate of the resources required can be generated on the basis of the grouping together of related configuration factors to get a means for generating estimates. For the example embodiment, the configuration factors or dimensions can include the number of ACD agents with their selected feature capabilities, the number of office users with unified messaging, and others that can be can be clustered.

In an embodiment, various configurations of features can represent individual dimensions in a multi-dimensional space. For example, a first axis can represent various feature types and a second axis can represent maximum usage for that feature type. Thus, in this example, each feature cluster will have its own two dimensions: the type of feature and maximum possible usage of such a feature. To illustrate, the feature cluster could be ACD agents with the dimension of a feature set and maximum size of ACD operation. Continuing with the example, a third axis can represent the resources required for a currently offered load for a feature cluster of a specified maximum size or load. Accordingly, in this example, the third axis represents the current resource usage for a specific configuration of a system of a certain maximum size under various conditions of load. Once many clusters for many features are considered, this representation becomes a highly dimensional matrix of resource estimates.

In an embodiment, a vendor model115can be generated from a set of rules supplied by the vendor. Tables representing vendor model115can be physically instantiated at system start up or table values may be generated on demand during the operation of the vended system. The tables may be highly dimensional and very large and so some embodiments may instantiate the tables only for the configurations and load or needed capacity ranges that are expected to be applicable for specific systems as opposed to all possible systems.

Continuing withFIG. 1, a cloud service provider, can use a cloud model120, to identify a capacity to allocate to a virtual machine125based on resource requirements specified. Typically, virtual machine125is set up on within a private data center of the cloud service provider. The private data center of the cloud service provider can include servers, networks and storage units, as well as other components which will now occur to a person of skill in the art. Capacity allocation is accomplished by allocating a portion of the capacity of its data center to the virtual machine such that the allocated capacity closely matches the capacity identified by cloud model120. Accordingly, virtual machine125can be executing at a single data center or in a mirrored form across a plurality of data centers.

In one embodiment, the cloud service provider can supply the operator with the required addresses and authorization keys and other appropriate access items necessary to access the virtual machine125. In other embodiments, other methods of setting up a virtual machine125can be used. Using the supplied access items operator can place the virtualized application105within the virtual machine125. Once the virtualized application105is set up within the allocated virtual machine125, the operator can accordingly begin to operate and offer services to its own internal and/or external users. In variations, the virtual machine can be set up in data centers or servers that are operated by other entities such as the operator itself.

In an embodiment, resource estimates generated by vendor model115can be used in cooperation with cloud model120to identify capacity to allocate. For example, cloud model120can be used, in cooperation with the vendor model115, to identify and allocate the required virtualized resources. In this example, the vendor model115can be used to provide at least part of the inputs to cloud model120. In a variation, the vendor model115can contain a translation element that will convert the resource estimates into the syntax and semantics required by a cloud model120. In a further variation, a table or rule-set or such similar mechanisms can be used to perform the conversion. In other embodiments, at least a part of the resource estimates can be derived by the cloud model120based on the data obtained from the vendor model115.

In some embodiments, cloud model120can be implemented on any computing apparatus, such as a server, a desktop computer, a virtual machine or any other apparatus, individual or distributed, that will now occur to a person of skill in the art. In variations, cloud model120can be resident on computing apparatus operated by the cloud service provider, and accessed through a network. In other variations, cloud model120can be resident on computing apparatus operated by the operator or the vendor, and accessed through a network. Other variations of operating and accessing cloud model120will now occur to a person of skill in the art.

Continuing withFIG. 1, an adaptation module is indicated at130. The adaptation module130can include policies which indicate when and how often resource updates should be requested for virtual machine125. The number and timing of updates can be represented by update policies. Update Policies can be set by the operator in part based on capacity factors such as cloud service restrictions, delays, costs and other capacity factors which will now occur to a person of skill in the art. Update policies, can additionally be based on change requirements of the virtualized application105. For example, to maximize efficiency of resource use, updates to allocated capacity for virtual machine125can be requested frequently. Alternatively, with systems where resource changes are costly, or where there is a large delay between request and allocation of capacity, the requests for update can be carried out infrequently.

Update policies can also be based on schedules that reflect estimated resource and capacity requirements of a virtualized application105. For example, update policies can be based on historical data which can then be used as a way of anticipating application-specific capacity need or load behavior during certain time of the day. Accordingly, a system operator may know when changes in load or capacity needs for virtualized application105occur. For example, daily backup operation in the middle of the night require more input/output and networking capacity to ensure backup operation is completed in timely manner with minimal impact to the normal operation of the application. As a further example, at the start of the day, majority of call center agents may login to the call center server within a short period of time, thus increasing the volume of calls handled. Update policies can be set up to predict the need for additional capacity needs during the start of the day to support larger agents login requests. In another example, the schedule for resource and capacity requirement changes can be learned based on historical capacity need or load statistics. In further examples, the schedules can be self-adjusting. Update policies can be set up to reflect these learned and self-adjusting schedules. It will now occur to a person of skill in the art that there are other means of predicting capacity changes on which update policies can be based. These are within scope.

Adaptation module130can be implemented on any computing apparatus, such as a server, a desktop computer, a virtual machine or any other apparatus, individual or distributed, that will now occur to a person of skill in the art. In variations, adaptation module130can be resident on computing apparatus operated by the vendor and accessed through a network. In other variations, adaptation module130can be resident on computing apparatus operated by the operator, and accessed through a network. Other variations of how the adaptation module130can be operated will now occur to a person of skill in the art.

Continuing withFIG. 1, a learning module135is shown which can take indications of resource usage, or performance indicators, from various components of system100, interpret these and modify the existing vendor model to generate an updated vendor model140. The updated vendor model140can in turn be used to generate updated resource estimates to reflect the current state of the running system. Accordingly, the learning module135functions by taking performance indications from various components of system100and transforming these into updated resources requirements for a specific service level that is stored in an updated vendor model140. The updated vendor model140can be stored in the same format as that of vendor model115described previously. Moreover, the updated vendor model140can be written to a new store. Accordingly, at system initialization, vendor model115obtained from the vendor is stored permanently but the vendor model115is also be copied to another store as the updated vendor model140, as shown inFIG. 1. Thus, the updated vendor model140that is used in the initial resource request is a copy of the vendor model115. Subsequently, updated vendor model140becomes the repository for the learned resource requirements and used for capacity allocation update requests and vendor model115is not changed. In a variation, the updated vendor model140can overwrite the original vendor model115. In this variation, the original vendor model is not preserved, and hence system100cannot be reset to a state where learning is undone.

In one embodiment, the updated vendor model140can be provided to a vendor. The vendor can refine the vendor model115default values based on updated model or models it received from one or more operators.

Learning module135can be implemented on any computing apparatus, such as a server, a desktop computer, a virtual machine or any other apparatus, individual or distributed, that will now occur to a person of skill in the art. In variations, learning module135can be resident on computing apparatus operated by the vendor and accessed through a network. In other variations, learning module135can be resident on computing apparatus operated by the operator, and accessed through a network. Other variations of how the learning module135can be operated will now occur to a person of skill in the art. In an embodiment, as shown inFIG. 1, learning module135and adaptation module130can be implemented as a single adaptation system150.

Performance indicators can originate from one or more various sources and can be accessed through various application programming interfaces (APIs), for example. These sources include the virtualized application105, the virtual machine125and the cloud provider's management system145. The cloud service provider can have an internal management system145that can determine the performance levels of the virtualized resource that it provides to the system operator. So, for example, cloud management system145can monitor CPU usage, memory usage, network usage, network latency and so on. The cloud provider can provide an API to allow an operator to access this information. One example of where the services provided by the cloud management system145are useful is the case where virtualized application105is exhibiting adequate performance but the reserved resource at the cloud provider is being underutilized. The operator can then reduce the requested resource level to achieve economy without sacrificing the service it requires.

The virtual machine125is another source of performance indicators. Virtual machine125can provide an indication of its internal monitoring system's detections such as whether the virtualized application105residing within virtual machine125has adequate or inadequate resource supply. Using this capability virtual machine125can provide a specific indicator indicating whether or not the virtual machine125is overloaded or exceeds the allocated capacity. Virtual machine125can also provide another indicator that indicates that virtual machine is approaching overload. The cloud service provider may provide an API to access this type of information from the virtual machine125that has been assigned to the operator.

The virtualized application105can also be equipped with a variety of performance indicators that are application-specific performance indicators. These may be indicators of the quality of the services provided to users of virtualized application105or internal metrics on the operation of the virtualized application105. The vendor or the operator can include various APIs that allow accessing the performance indicators from the virtual application105. Continuing with the present example embodiment where virtualized application105is an IP-PBX system, an indicator indicating call quality can be provided. Such an indicator could provide indication of issues regarding missing and faulty packets, jitter and other quality factors affecting call quality. Other indicators could also be provided regarding the time required to offer dial tone in response to a service request or time required to cut through a connection during a specific operation of a specific feature such as call transfer. Specific internal metrics can also be provided regarding the length and stability in length of queues in media jitter buffers, throughput in transcoders, delays in acquiring specific signal processing functions such as conference units, and others that will now occur to a person of skill in the art. These indicators collectively and individually may be taken as indications of various resource usage levels such as memory allocation, network capacity, CPU capacity and others that will now occur to a person of skill in the art.

As a further illustrative example, for a virtualized web server the application-specific performance indicators can include time to transmit a web page with all its content, time to create dynamic content and others. As a further example, for a virtualized email server, application-specific performance indicators can include time to deliver mailbox contents to a client application or browser, time to process a new incoming email (including actions like scanning for viruses, black-list and white-list searches on sender) and others. These actions can involve significant communication with other servers and external web services.

Referring now toFIG. 2another embodiment of a system for determining virtual machine performance is indicated at200. System200includes a virtual machine125, set up, for example, based on requirements specified by a customer, a vendor and a cloud model as described above. Virtual machine125is intended to include a virtualized application that will be in communication with one or more computing apparatuses210, located, for example, at one or more of the operator's sites or locations that can be co-located with cloud service provider's sites, or remote from cloud service provider's sites. The communication can be through a network235. Accordingly, at least one computing apparatus210(computing apparatuses210-1,210-2and210-3) is connected, via network235, to virtual machine125. Collectively, computing apparatuses210-1,210-2and210-3are referred to as computing apparatuses210, and generically as computing apparatus210. This nomenclature is used elsewhere herein.

Computing apparatuses210can be based on any suitable computing environment, and the type is not particularly limited so long as each computing apparatus210is capable of receiving data from virtual machine125and transmitting data to virtual machine125. In a present embodiment, computing apparatuses210can be configured to at least execute a client application and simulators that can interact with virtual machine125. For example, computing apparatus210can be configured to perform the functions of many different systems such a distant server (210-1), an operator management system (210-2) or a simulated data traffic system (210-3) as shown inFIG. 2. Computer apparatuses210can be operated by an operator, a system vendor, a cloud service provider or other parties. Other configuration for computer apparatus210will now occur to a person of sill in the art, and are within scope.

Certain apparatus factors can affect the performance of a virtualized application. For example, when a distant server210-1, distant from the virtual machine125, is utilized as part of the system set up by the operator, the latency of the connection between the virtual machine125and the distant server210-1can be important to the performance of the operator's system. For example, in the present example, where the virtualized application is to be an IP-PBX, user telephones and other devices at the operator site as well as the operator management system210-2interact with virtual machine125. The latency, error and bandwidth properties of the communications with the virtual machine125can affect the perceived performance of the virtualized IP-PBX. For example, there could be a faulty or overloaded router located in network235that affects only some locations in the network. The router may be causing many retransmissions or lost packets, which is not reflected in latency metrics but could greatly impact application-specific performance metrics. In some variations, latency can depend on the traffic load generated by the applications executing at a computing apparatus210, such as a load simulator230.

Computing apparatuses210can be based on any type of computing environment, such as a tablet, a smart phone, a desktop, a server or any other computing platform that is known in the art. Each computing apparatus210includes at least one processor connected to a non-transitory computer readable storage medium such as a memory. Memory can be any suitable combination of volatile (e.g. Random Access Memory (“RAM”)) and non-volatile (e.g. read only memory (“ROM”), Electrically Erasable Programmable Read Only Memory (“EEPROM”), flash memory, magnetic computer storage device, or optical disc) memory. In one embodiment, memory includes both a non-volatile memory for persistent storage computer-readable instructions and other data, and a volatile memory for short-term storage of such computer-readable instructions and other data during the execution of the computer-readable instructions. Other types of computer readable storage medium external to computing apparatus210are also contemplated, such as secure digital (SD) cards and variants thereof. Other examples of external computer readable storage media include compact discs (CD-ROM, CD-RW) and digital video discs (DVD).

A computing apparatus210can also include one or more input devices connected to at least one processor. Such input devices are configured to receive input and provide data representative of such input to the processor. Input devices can include, for example, a keypad, a pointing device and a scanner. A pointing device can be implemented as a computer mouse, track ball, track wheel, touchscreen or any suitable combination thereof. In some examples, computing apparatus210can include additional input devices in the form of one or more additional buttons, sensors, microphones and the like. More generally, any suitable combination of the above-mentioned input devices can be incorporated into computing apparatus210. The input devices can be integral to a device or can be an add-on option which can be operably connected to the device as needed.

A computing apparatus210can further include one or more output devices. The output devices of computing apparatus210can include a display. When the pointing device includes a touchscreen, the touchscreen can be integrated with the display. Each computing apparatus210can also include additional output devices such as printers. The output devices can be integral to a device or can be an add-on option which can be operably connected to the device as needed.

Each computing apparatus210can also include one or more communications interfaces connected to the processor. The communications interface allows computing apparatus210to communicate with other computing devices, for example via network125, as well as peripherals such as headsets, handheld scanners, and other input and output devices that will now occur to a person of skill in the art. At least one of the communications interfaces can be selected for compatibility with network125. The communication interfaces may be wireless radios such as Bluetooth, cellular radio such as LTE, near field communications (NFC) or Wi-Fi, or they may be wired such as USB, serial or parallel ports.

Network235can comprise any network capable of linking virtual machine125with computer apparatuses210and can include any suitable combination of wired and/or wireless networks, including but not limited to a Wide Area Network (WAN) such as the Internet, a Local Area Network (LAN), Personal Area Network (PAN), cell phone networks, Wi-Fi networks, and the like.

Continuing with system200, a proxy module215and proposed virtualized application220are provided at virtual machine125as two virtual machine based test modules. An additional test module, a load simulator230, is provided at the data traffic system210-3.

Proxy module215can include one or more test routines that, when executed, can stress capacity needs or loads, and evaluate performance based on certain resource requirements. At least some of these resource requirements can be negotiated with a cloud service provider. Other resource requirements can be application-specific resource requirements based on predetermined application-specific performance-metric targets and would typically be excluded from the resources specified to the cloud service provider. Other resources can be those resources outside the control of the service provider. Other examples of resources will now occur to a person of skill in the art and are within scope.

Proposed virtualized application220is part or whole of the vended system which will form, when fully installed the virtualized application. Accordingly, in one embodiment, proposed virtualized application is equivalent to the intended virtualized application that will be operated by the operator. In other embodiments, proposed virtualized application220can include only portions of the intended virtualized application to be provided. In yet other embodiments, proposed vended system can be simulators for simulating all or portions of the intended virtualized application to be provided. Other variations of the relationship between proposed virtualized application220and the intended virtualized application will now occur to a person of skill in the art and are within scope. In one embodiment, proposed virtualized application220is capable of testing application-specific resource requirements by providing application-specific metrics for evaluation of the intended virtualized application.

The load simulator module230can generate a simulated customer load. The simulated customer load can be varied along multiple parameters such as volume and type so as to be able to evaluate the performance of the intended virtualized application in meeting operator requirements. In an alternative embodiment, the virtual machine125, along with the test modules can be set up on an operator's own physical servers or data centers. The physical configuration of the physical servers can be such that they can be used to test the identified resource requirements. Other type, location and combination of test modules will now occur to a person of skill in the art and are within scope.

Referring now toFIG. 3, a method for selecting virtual machine performance is indicated at300. In order to assist in the explanation of the method, it will be assumed that method300is operated using system100as shown inFIG. 1, and system200as shown inFIG. 2. Additionally, the following discussion of method300leads to further understanding of system100and system200. However, it is to be understood that systems100and200, and method300can be varied, and need not work exactly as discussed herein in conjunction with each other, and that such variations are within scope.

Referring now toFIG. 3, a virtual machine is set up at305. To set up a virtual machine125, for example, an operator can use its own operator model in combination with a vendor model to generate required resource estimates, which can then be used to negotiate with a cloud service provider. Once the resource requirements are agreed upon, the cloud service provider can then create the required virtual machine125, allocate it capacity on the basis of the negotiated resource requirements and make the virtual machine125available to the operator for use.

The negotiation between the operator and the cloud service provider for the creation of a virtual machine125will, in part, contain identified resource requirements that are identified, for example, by the operator and vendor models and that need to be maintained to obtain a predetermined performance for the virtualized application that will be implemented by the operator. However, in some embodiments, resource requirements specified to the cloud service provider do not contain excluded resource requirements, for example those that can be associated with the specific virtualized application. Excluded resource requirements can form a portion of the identified resource requirements that are not specified to the cloud service provided for negotiations. There may be various reasons for the exclusion of some requirements from those specified to the cloud service provider. Some of the requirements, for example, may be unspecified due to inadequacies in the semantics of the communication used between the operator and the cloud service provider. Certain configurations or capabilities of equipment that are not contained in the semantics of the communications is an illustration of the limitations introduced by the semantics. As a further example, other requirements may be requirements that are not normally maintained or guaranteed by the cloud service provider. Speed of data transfer between separate virtual machines or virtualized applications provided by the same cloud service provider are examples of resources that the cloud service provider may not guarantee or maintain. As an additional example, other requirements may be outside of control of the cloud service provider such as the latency of an external network that is used to connect a virtual machine to an external computing apparatus210as part of the user utilization of a virtualized application, for example. Additionally, the bandwidth and latency of network215used to connect virtual machine125to the computing apparatuses210may be an important requirement that is not a specified requirement. Additional requirements can be requirements that are application-specific. For example, the degree of compatibility between the specific CPU family and version of the cloud provider's hardware and the compiler and options used when compiling the kernel and the virtualized application can all be considerations when identifying application-specific resource requirements. As a specific example, the compiler can use machine instructions that are only emulated on some CPUs but fully implemented on others. Other examples of considerations that affect application-specific resource requirements include: speed and bandwidth of RAM memory access; speed of various internal buses; floating point performance of the specific CPU family and version; basic input/output system (BIOS) version; firmware version of internal and external attached devices; outside the server hardware, the network topology around virtual machine125, as well as computing apparatuses210; speed of domain name system (DNS) servers used (different locations in the cloud may use different DNS servers): and speed of authentication servers used by virtual machine that may be part of the cloud provider's infrastructure (e.g., domain controllers, Remote Authentication Dial in User Service (RADIUS) servers, and others).

Continuing with method300, at310the virtual machine125that is set up is evaluated. In many cases an operator will wish to evaluate, prior to setting up or operating a virtualized application, whether virtual machine125meets at least some of the identified requirements whether those requirements are specified to the cloud service provider or excluded from the requirements specified to the cloud service provider. To perform the evaluations, in one embodiment test modules such as proxy module215, load simulator230and proposed virtualized application220can be used.

To perform evaluation of virtual machine125, in one embodiment, proxy module215can be caused to execute test routines that can test one or more resource requirements including those based on application-specific requirements to generate performance characteristics. In some embodiments, performance characteristics can include performance indicators. In other embodiments, performance characteristics can include those that are specific to the virtualized application. In yet other embodiments performance characteristics can be generated by means other than test routines. These and other variations that will now occur to a person of skill in the art are within scope.

The resource requirements can be tested, and performance characteristics generated, for example, by causing the capacity need or load of the intended virtualized application to approach its estimated maximums, while measuring whether the resources stay within the identified requirements. In one embodiment, these test routines can be executed in cooperation with the proposed virtualized application220. The tested resource requirements can be both those that are specified to the cloud service provider when negotiating the capacity allocation for virtual machine125, and also those that were not specified, such as those that are outside the control of the cloud service provider. The tested resources can also include those resources that are application-specific and on which application based performance metrics are based. For example, proxy module125can be caused to execute test routines that can test the resources associated with network connections between the virtual machine125and one or more of the computer apparatuses210such as a distant server210-1or the data traffic system210-3which includes the load simulator module230for simulating a customer site with simulated traffic. One or more of the test routines can be executed serially, in parallel or in combination for various resources under various capacity needs or loads. Moreover, a test routine may be executed either manually or automatically such as programmatically. Instructions such as scripts or other programmatic methods to execute a test routine can be included locally, as part of the proxy module215, or received through network215from one or more of the computer apparatuses210. Proxy module215may also be instructed to run test routines that test resources such as available memory, internal bandwidth and latency for messaging and other resources contained within the virtual machine itself.

Referring back toFIG. 2, to evaluate a virtual machine125an operator, in addition to executing test routines of the proxy module215, can also utilize capacity or load and service level testing performed by the load simulator230which can simulate user loads and cause generation of performance characteristics. In one embodiment these tests can be performed in cooperation with the proxy module215and/or the proposed virtualized application220. Additional computer apparatuses210located elsewhere in the network may be used to participate in these tests as well by triggering one or more operations of the proposed virtualized application220. Evaluation of whether resource requirements are met for the proposed virtualized application220can be done under various test conditions of capacity needs or loads and feature mixes. For example, in the present example embodiment where the intended virtualized application is an IP-PBX, media packets with errors and delays can be measured as indicators for determining whether the performance of acoustic resources satisfy the identified acoustic resource requirements. Moreover, acoustic requirements can be an example of excluded resources requirements, namely resource requirements that are excluded from those specified to the cloud service provided; in this example, the cloud service provider would not have the means to monitor acoustic resource requirements. In addition, evaluation process can also measure delay to dial tone, excessive latency on media connections, and errors with feature operations and others that will now occur to a person of skill in the art. Accordingly, in some embodiments, specific testing can be performed for application-specific resource requirements of specific proposed applications. In other embodiments, generic testing can be performed in addition with or as an alternative to specific testing. Generic testing can measure multiple conditions of capacity needs, one of more of which can be suited to the specific proposed virtualized application220under consideration. This can include making measurements that indicate user evaluation of the performance of the proposed virtualized application220for one or more resources.

In some embodiments, results of previously performed test measurements can be used to evaluate a virtual machine125. A parking database can be provided that stores previously measured performance characteristics of virtual machines as set up at different locations within the private operations of a cloud service provider and with different cloud service providers or both. Performance characteristics can indicate resource levels available under different capacity needs or loads. The parking database can be prepared by performing tests based on various test modules at a variety of locations with various configurations of external servers, class or type of proposed virtualized applications. Accordingly, the parking database contains performance characteristics of a number of alternatives for any proposed virtualized application.

The parking database can be used as a method of evaluating the suitability of a location or a cloud service provider for use with a proposed virtualized application220without the need to perform tests before setting up the intended virtualized application. Therefore instead of evaluation being performed for a proposed system as described above, the database may be consulted and a virtual machine with suitable performance and/or cost may be selected.

Resource requirements for the intended virtualized application can be identified and the suitability of a variety of virtual machines evaluated on the bases of the performance database in order to decide as to where the instantiation can be created such that it will satisfy the identified resource requirements. In a variation, the parking database can contain data for types or classes of proposed virtualized application. In another variation, the parking database, can also contain an estimated cost for the system.

The Parking database can be made accessible either locally or on a networked computer apparatus210. User interfaces can be provided through which the details contained within the parking database can be viewed and edited. Creation of the database may be triggered either manually or automatically such as programmatically and can be based on a schedule. Locations for the database may be stored in a file that can be accessed and updated manually or automatically. Other methods of creating and updating a parking database will now occur to a person of skill in the art and is within scope.

Continuing with method300, at315satisfaction of identified resource requirements is determined on the bases of the evaluation results. Determination of whether the resource requirements are satisfied can be carried out based on required standards for latency, bandwidth, errors etc. under different conditions capacity needs or loads. The determination can be based on a predetermined satisfaction criteria. Accordingly, the determination can be based on comparing some or all of the predetermined satisfaction criteria based on identified resource requirements with measured performance characteristics. The resource requirements on which the criteria is based can be grouped or prioritized for the purposes of the determination. These and other variations will now occur to a person of skill in the art and are within scope.

The determination can be carried out by one or more of the test modules or at one or more of the computer apparatuses210such as at an operator management system on in a combination of both. The determination can be performed manually or automatically, such as programmatically. The determination can be based on one or more or of the identified resource requirements.

Where determination is made at315that the capacity allocation for virtual machine125does not satisfy at least some of the identified resource requirements for proposed virtualized application220, virtual machine125can be modified or moved (step320). The action to modify or move virtual machine125is taken so as to enable satisfying more or all of the resource requirements. For example, in one variation, the capacity allocation for virtual machine125can be altered to remedy any deficiencies identified in testing. In other variations, a different virtual machine can be set up either with the same cloud service provider or a different cloud service requirement to address the deficiencies identified by testing. In an embodiment, the parking database can be consulted for the suitability of a location or searched to identify a new location that is more satisfactory for resource requirements. These and other variations of addressing deficiencies identified by testing are within scope.

Continuing with method300, where virtual machine125is modified or moved, the modified or moved virtual machine can be subject to validation itself by repeating310for the modified or moved virtual machine. By re-performing310, testing can be used to evaluate a proposed move or modification before the virtualized application is actually set up. Additionally or alternatively, the parking database can be consulted for the suitability of the new location. Once the newly modified or relocated virtual machine is evaluated, a determination is made whether measured performance characteristics satisfy the predetermined criteria that is based on one or more of the identified resource characteristics. In the present example, it will be assumed that the predetermined criteria are satisfied, and method300advances to325.

Continuing with method300, at325a virtualized application is set up. The cloud service provider can supply the operator with the required addresses and authorization keys and other appropriate access items necessary to access the virtual machine125. Using the supplied access items, the operator can place the virtualized application205within the virtual machine125, alongside proxy module215of system400as shown inFIG. 4. Once the virtualized application205is set up within the allocated virtual machine125, the operator can accordingly begin to operate and offer services to its own internal and/or external users. Based on the evaluation done, it is expected that virtualized application205can satisfy all or a portion of the identified requirements so as to provided acceptable performance within the limits of the expected capacity needs and feature mix.

Referring back toFIG. 3, at step330performance of virtualized application205is monitored. As shown inFIG. 4, proxy module215can be used as part of the active management of the virtualized application205during its operation. Proxy module215can be used to determine whether inadequate or excess resources allocated to virtualized application205resulting either in unacceptable performance or cost inefficiency by providing additional performance characteristics. In such a case, it may be desired to move the virtualized application to a modified or relocated virtual machine that allows the provision of resources that better satisfies the satisfaction criteria for virtualized application205. Any proposed move may then be evaluated using either or both the proxy and parking technologies described above.

In a variation, performance indicators can be received from virtualized application205, and the determination of whether resource requirements can be met can be based on a comparison of the performance indicators with the predetermined criteria.

Adaptation system150ofFIG. 1can also be given the capability to trigger virtual machine moves. The adaptation system150can be provided with a schedule of expected service load and based on that schedule can trigger a move to a more suitable location at appropriate times such as a predetermined amount of time prior to significant increase in capacity needs or load. As well, the adaptation system150can determine that the adapted load that it has created is significantly below the initially reserved resource allocation for the virtual machine125and trigger a move to more suitable locations. Additionally, the adaptation system can determine that resources are inadequate at a specific location to meet resource requirements for the virtual machine125current load. Accordingly, it can then automatically trigger a move of the virtual machine125. The triggered moves can use the proxy and or parking technology to carry out the move to a more suitable location.

The movement of the virtual machine125can also be triggered manually. A user interface can be provided with which the current performance and/or resource usage of a virtual machine125can be viewed and the move of virtual machine125triggered manually. Alternatively, a system operator may know, by some means, that a change in load or capacity needs for virtual machine125is imminent, for example as a result of a scheduled backup or a large report being run. The system operator may then trigger a move or schedule the move to happen at an appropriate time. The system operator may be given the capability of creating a script that will automatically trigger moves based on a schedule and/or on measured system performance and resource usage.

The movement of virtual machine125can also be based on schedules that reflect estimated resource and capacity requirements of a virtualized application105. For example, schedules can be based on historical data which can then be used as a way of anticipating application-specific capacity need or load behavior during certain time of the day. Accordingly, a system operator may know when changes in load or capacity needs for virtualized application105occur. For example, daily backup operation in the middle of the night require more input/output and networking capacity to ensure backup operation is completed in timely manner with minimal impact to the normal operation of the application. As a further example, at the start of the day, majority of call center agents may login to the call center server within a short period of time, thus increasing the volume of calls handled. Update policies can be set up to predict the need for additional capacity needs during the start of the day to support larger agents login requests. In a further example, large reports may need to be generated at specific times. In another example, schedule changes can be learned based on historical capacity need or load statistics. In further examples, the schedules can be self-adjusting. It will now occur to a person of skill in the art that there are other means of predicting capacity changes on which schedules can be based. These are within scope.

Method300can be used for reasons of cost as well as system performance. So300can be used from time to time, either periodically, or randomly or based on a schedule to evaluate the current costs for a suitable virtual machine form multiple providers. Moves can then be triggered to a location with the most appropriate performance and cost.

The parking database information for multiple suppliers can be made available so that it can be supplied to cloud providers as a part of negotiation process. Cloud service providers can be given access to suitably altered or otherwise anonymized information so that comparative cost and performance information of other cloud service providers can be made available. This may be used as part of a negotiating process for new installations or to resolve issues with existing installations. This information can be made available from a web server. The web server may require suitable authentication procedures to be carried out for access.

In further variations, updated vendor model140can be used for system verification and initial generation of the vendor model by vendors. Thus a vendor can take a system under development, place simulated capacity requirements on it and determine the values required for its databases.