Cluster computing service assurance apparatus and method

Apparatuses, methods and storage medium associated with cluster computing are disclosed herein. In embodiments, a server of a computing cluster may include memory. input/output resources, and one or more processors to operate one of a plurality of application slaves of an application master; wherein the other application slaves are operated on other servers, which, together with the server, are members of the computing cluster. The server may further include a service assurance manager agent to manage allocation of the one or more processors, the memory and the input/output resources to the application slave, to assure compliance with a node level service level agreement, derived from an application level service level agreement, to contribute to proximate assurance of compliance with the application level service agreement; wherein the application level service agreement specifies the aggregate service level to be jointly provided by the application master and slaves. Other embodiments may be described or claimed.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application is a national phase entry under 35 U.S.C. § 371 of International Application No. PCT/CN2015/075051, filed Mar. 25, 2015, entitled “CLUSTER COMPUTING SERVICE ASSURANCE APPARATUS AND METHOD”, which designated, among the various States, the United States of America. The Specification of the PCT/CN2015/075051 Application is hereby fully incorporated by reference.

TECHNICAL FIELD

The present disclosure relates to the field of computing. More particularly, the present disclosure relates to cluster computing service assurance.

BACKGROUND

In many real life computer cluster deployment (e.g., big data cluster), there can be multiple application tasks running in parallel without any physical or logical isolation due to overall hardware resources shortage/scarcity or cluster sharing usages. The application tasks consume and even compete for the same set of underlying hardware resources, e.g., central processing unit (CPU), memory, and input/output (I/O) resources in each of the computer nodes employed. Due to dynamic needs of the application during its lifecycle, resource allocation to the various instances of the application may result in resource imbalance and unsatisfied service level agreement and thus poor user experience.

For example, existing resource scheduling solutions like Hadoop YARN for big data cluster deployments allocate resources for requesting application according to their static resource allocation configurations instead of their real-time resource need and usage that can be very dynamic during various stages of the application lifecycle. Static resource allocation is far from satisfactory. Some other solutions provide operating system (including virtual machines) or computer node level resource isolation to guarantee the service level agreement (SLA) of the systems or applications of interest. However, this level of service assurance granularity is too coarse and does not address the scenarios of sharing the resources among applications within the same cluster or even the same node.

DETAILED DESCRIPTION

Apparatuses, methods and storage medium associated with cluster computing are disclosed herein. In embodiments, a server of a computing cluster may include memory and input/output resources; and one or more processors coupled with the memory and the input/output resources, to operate one of a plurality of application slaves of an application master; wherein the other application slaves are operated on other servers, which, together with the server, are members of the computing cluster. The server may further include a service assurance manager agent to manage allocation of the one or more processors, the memory and the input/output resources to the application slave, to assure compliance with a node level service level agreement, derived from an application level service level agreement, to contribute to proximate assurance of compliance with the application level service agreement; wherein the application level service agreement specifies the aggregate service level to be jointly provided by the application master and slaves.

Aspects of the disclosure are disclosed in the accompanying description. Alternate embodiments of the present disclosure and their equivalents may be devised without parting from the spirit or scope of the present disclosure. It should be noted that like elements disclosed below are indicated by like reference numbers in the drawings.

Various operations may be described as multiple discrete actions or operations in turn, in a manner that is most helpful in understanding the claimed subject matter.

“The terms “master” and “slave” as used herein are for their technical meaning. The terms “application master” and “application slaves” are synonymous with “application parent” and “application children.” The term “service assurance manager master” is synonymous with “service assurance manager broker.”

Referring now toFIG. 1, wherein a block diagram illustrating a cluster computing environment incorporated with the node level service assurance technique of the present disclosure, according to various embodiments, is shown. As illustrated, computing cluster100may include a number of computing nodes102. Each computing node102may include various hardware resources105, such as CPU112, memory114and input/output resources115, and at least one slave of each of a plurality of applications, e.g,104aand104b, operated thereon. Each application may include an application master103a/103bspawning application slaves104a/104bin the plurality of computing nodes102. Each application slave104a/104bmay have a number of processes/tasks110a/110b. Further, each computing node102may include a service assurance manager (SAM) agent108to dynamically manage allocation of CPU112, memory114and I/O115resources to application slaves104a/104b, to assure compliance with respective node level service level agreements (SLA)118. Each SLA118may be derived from a corresponding application level service level agreement117that specifies the aggregate service level to be jointly provided by the application master and slaves103aand104aor103band104bon the various computing nodes102. Resultantly, respective compliance to the derived SLA118of an application slave at each computing node102may contribute to proximate assurance of compliance with the overall application level service agreement117.

For example, a computing cluster hosting an application for processing transactions may have a SLA of 100,000 transactions per second. In embodiments, four (4) substantially similar computing nodes, in terms of capability and capacity, may be employed to host four (4) application slaves spawned by the application master. The application level SLA may be proximately assured by assuring compliance of each computing node with a derived SLA of 25,000 transactions per sec. In another example, three (3) dissimilar computing nodes, in terms of capability and capacity, may be employed to host three (3) instances of the application. The application level SLA may be proximately assured by assuring compliance of respective derived SLA of 50,000 transactions per second, 35,000 transactions per second, and 25,000 transactions per second (totaling more than 100,000 transactions per second). The respective derived SLA of 50,000) transactions per second, 35,000 transactions per second, and 25,000 transactions per second may roughly correspond to the relative processing power of the three (3) dissimilar computing nodes.

In embodiments, various SAM agents108correspondingly located in various computing nodes102may be coordinated by a SAM master107. In embodiments, SAM master107may be disposed on its own computing node102, or share a computing node102with a SAM agent108, as illustrated. In embodiments, SAM master107may include a configuration interface (not shown) to facilitate configuration of SAM master107with the application level SLA117of the applications. In embodiments, SAM master107may derive the node level SLA118, and provide the node level SLA118to the various SAM agent108respectively disposed on computing nodes102. In embodiments, SAM master107may provide the node level SLA118to the various SAM agent108via communications119. In embodiments, SAM master107may derive the node level SLA118, based at least in part on computing node usage information, obtained from application master103a/103b. In embodiments, SAM master107may obtain the computing node usage information from application master103a/103bthrough interactions121.

In embodiments, SAM agent108may assure compliance with a node level service level agreement (SLA)118, via interactions120to understand the needs of application instances104a/104b, and communications122to dynamically regulate allocation of CPU112, memory114and I/O115resources to application slaves104a/104b. In embodiments, SAM108may also assure compliance with a node level SLA118, via interactions120to reduce the resource needs of some application slaves104a/104b, in favor of other application slaves. In embodiments, reduction of resource needs may include preempting some application slaves, reducing or halting their processes/tasks110a/110b. In embodiments, halting processes/tasks110a/110bof application slaves104a/104bmay be coordinated with halting spawning of application slaves104a/104bby application masters103/103b. In embodiments, SAM master107and SAM agents may coordinate halting of application masters103a/103band halting of application slave processes/tasks110a/110bvia interactions119. And SAM amster107may halt spawning of application slaves104a/104bby application masters103a/103bvia interactions121.

In embodiments, SAM master and/or agents107and/or108may be implemented in hardware. For example, SAM master and/or agents107and/or108may be implemented via an Application Specific Integrated Circuit (ASIC) or a field programmable circuit, such as Field Programmable Gate Arrays (FPGAs) programmed with the operating logic described herein. In alternate embodiments, SAM master and/or agents107and/or108may be implemented in hardware. For example, SAM master and/or agents107and/or108may be implemented in assembler instructions of the underlying processor, or in C or higher level language that can be complied into the instruction set architecture of the underlying processor. In still other embodiments, SAM master and agents107and/or108may be implemented in a hardware/software combination.

CPU112may be any one of a number of single or multi-core processors known in the art. Memory114may be any one of a number of volatile or non-volatile, electrical, magnetic, and/or optical memory known in the art. I/O resources115may include any one of a number of I/O devices/interfaces, such as, but not limited to, wired or wireless networking interfaces, serial and/or parallel I/O interfaces, and so forth. While for ease of understanding, only CPU112. Memory114and I/O115are illustrated, hardware resources105may include other hardware resources, such as, but not limited to, any one of a number of co-processors, graphics processing units (GPU), math co-processors, digital signal processors (DSP), and so forth.

Further, in addition to hardware resources105, each computing node102may include various firmware/software resources, e.g., various operating system/hypervisor services, such as, but not limited to, scheduler, memory manager, process/task manager, and so forth.

Similarly, each collection of application master and slaves103a/103band104a/104bmay be any one of a number of applications known in the art, in particular, big data applications that typically involve simple or complex computations that involve a large amount of data with multiple application slaves instantiated and operated on multiple servers of a computing cluster. Examples of big data applications may include, but not limited to,Consumer product companies and retail organizations' applications monitoring social media like Facebook and Twitter to get a view into customer behavior, preferences, and product perception.Manufacturers' applications monitoring social networks to detect aftermarket support issues before a warranty failure become publicly detrimental.Financial Service companies' applications using data mined from customer interactions to divide their users into finely tuned segments, to create more relevant and sophisticated offers.Advertising and marketing agencies' applications tracking social media to understand responsiveness to campaigns, promotions, and other advertising mediums.Insurance companies' applications using Big Data analysis to see which home insurance applications can be immediately processed, and which ones need a validating in-person visit from an agent.Web-based businesses' applications developing information products that combine data gathered from customers to offer more appealing recommendations and more successful coupon programs.Sports teams' applications using data for tracking ticket sales and even for tracking team strategies.

Thus, except for SAM master and agent107and108, each computing node102may be any one of a number cluster computing node known in the art. The constitution and operations of SAM master and agent107and108will be further described with references toFIGS. 2-5. Before further describing SAM master and agent107and108and other aspects of the present disclosure, it should be noted that why for ease of illustration,FIG. 1depict one slave each for two applications, application slave104aand application slave104b, the present disclosure is not so limited. As will be readily understood from the description to follow, the present disclosure may be practiced with SAM master and agent107and108proximately assuring compliance with SLA117for one or more applications, via assurance of SLA118of any number of application slaves104a/104boperating on any number of computing nodes102. Further, in embodiments, the functionality and operations of SAM master107may be assumed by one SAM agent108, or shared among a plurality or all of the SAM agents108.

Referring now toFIG. 2, wherein a block diagram illustrating communications/interactions between SAM master107, SAM agent108, application master103a/b, and an application instance104a/104b, according to the disclosed embodiments, is shown. As illustrated, application master and slaves103a/103band104a/104b(more specifically, the application in general) may be configured with an interface (not shown) that enables SAM master107and SAM agent108to query application masters103a/103band application slaves104a/104bfor various information. In particular, the interface may be configured to enable SAM master107to query an application master103a/103bfor computer node usage, and enable SAM agent108to periodically query application slaves104a/104bon its CPU, memory, I/O et al resource needs202. For examples, amount of CPU cycle times, amount of memory space, and/or amount of I/O bandwidths desired by application slaves104a/104b(which may vary over time during operation, depending on the workloads at various points in time or various stages of the application).

In embodiments, application slaves104a/104b(more specifically, the application in general) may be configured with an interface (not shown) that enables SAM agent108to periodically query application slaves104a/104bon various performance metrics204, to independently determine resource needs of application slaves104a/104b. Examples of performance metrics204may include, but are not limited to, wait time for CPU cycle, number of cache misses, frequency and/or time incurred for memory swaps, wait time and/or transmission bit rates for I/O channels, and so forth.

In embodiments, application slaves104a/104b(more specifically, the application in general) may be configured with an interface (not shown) that enables SAM agent108to dynamically throttle the resource needs of some of the application slaves104a/104b(for the benefit of other application slaves on computing node102). Throttling command206bmay include, but are not limited to, command to instruct application instance104a/104bto pre-empt one or more of its child processes/tasks110a/110b, decreasing or increasing the number of child processes/tasks110a/110b, and/or pausing or resume one or more of the child processes/tasks110a/110b.

Similarly, in embodiments, application masters103a/103b(more specifically, the application in general) may be configured with an interface (not shown) that enables SAM master107to dynamically throttle206athe spawning of application slaves104a/104bby some of application masters104a/104b(for the benefit of other application masters and slaves on computing nodes102).

In embodiments, interactions119between SAM master107and agents108, as described earlier, may include node level SLA118provided to SAM agents108by SAM master107. Additionally, interactions119may also include reporting208of compliance from SAM agents108to SAM master107, including earlier described halting of processes/tasks110a/110b, to enable SAM master107to similarly halt spawning of application slaves104a/104bby the affected application masters103a/103b.

FIG. 3illustrates communications between the service assurance manager agent and various computing resources, according to the disclosed embodiments. As illustrated, in embodiments, various hardware resources105, e.g., CPU112, memory112and I/O resources115may be configured with interfaces (not shown) to enable SAM agent108to query302the various hardware resources for status information. For examples, CPU112may be queried for its idle and/or busy times, memory114may be queried for its allocated and/or unallocated space, latencies, and I/O resources may queried for its availability or unavailability status. In alternate embodiments, computing node102may include hardware monitor314, and SAM agent108may query302hardware monitor314for the various hardware resource status information.

FIGS. 4-5illustrate an example process for assuring service level agreement of an application via node level service assurance, according to the disclosed embodiments. As illustrated, the process may include process400for configuring SAM agents and interacting with application masters as depicted inFIG. 4, and process500for assuring service level for application slaves distributed on a plurality of computing nodes of a computing cluster, via node level service assurance.

As shown inFIG. 4, process400may include operations at block402-408. The operations may be performed by e.g., the earlier described SAM master107ofFIGS. 1-2. In alternate embodiments, process400may be practiced with additional operations, or with some of the operations omitted, combined and/or re-ordered.

Process400may start at block402. At block402, an application level SLA may be obtained/provided. Next, at block404, computing node usage may be obtained, e.g., from the application master. Then at block406, the node level SLA may be derived. As earlier described, the node level SLA may be derived through decomposition of the application level SLA in view of the capacity/capability of the computing nodes used.

Next, process400may wait at block406for reporting from the SAM agents. On receipt of the reports, a determination may be made on whether processes/tasks of application slaves of an application master are being halted. If processes/tasks of application slaves of an application master are not being halted, process400may return to block406, and proceed therefrom as earlier described. On the other hand, processes/tasks of application slaves of an application master are being halted, process400may proceed to block408. At block408, throttle commands may be sent to the application master to halt further spawning of application slaves by the application master. Therefore, process400may proceed back to block406, and proceed therefrom.

As shown inFIG. 5, process500may include operations at block502-508. The operations may be performed by e.g., the earlier described SAM agent108ofFIGS. 1-3. In alternate embodiments, process500may be practiced with additional operations, or with some of the operations omitted, combined and/or re-ordered.

As shown, process500may start at block502. At block502, application instances on a computer node may be checked for resource needs. As described earlier, the checking may be performed directly by querying the application instances on their needs, or indirectly by querying the application instances on various performance metrics, and infer the needs from the performance metrics. If no new needs are identified, process500may remain at block502, until new needs are identified.

On identification of needs, process500may proceed to block504. At block504, resource allocation to be adjusted and/or resource needs to be reduced may be selected. For example, allocation or re-allocation of CPU resources, memory resources and/or I/O resources may be considered in view of the needs identified in block502. In some situations, allocation of CPU, memory, and I/O resources may be selected from free unallocated resources. In other situations, allocation of CPU, memory, and I/O resources may require de-allocation of some of these resources from other application instances to free and make those resources selected for allocation. However, in situations where allocation and/or de-allocation of hardware resources are not selected (not viable), reduction of resource needs of some of the application instances may be selected.

On a determination to readjust resource allocation, process500may proceed to block506. At block506, allocation of hardware resources to the application instances on a computer node may be adjusted. On the other hand, on a determination to reduce resource needs of some of the application instances, process500may proceed to block508. At block508, the application instances may be asked to curtail their resource needs, e.g., by reducing and/or temporarily halting one or more their processes/tasks. At a later point in time, the application instances may be informed that additional processes/tasks may be spawned and/or re-started.

From block506or508, process500may return to block502, and proceed there from as earlier described.

FIG. 6illustrates an example computer system that may be suitable for use to practice selected aspects of the present disclosure. As shown, computer600may include one or more processors or processor cores602, and system memory604. For the purpose of this application, including the claims, the terms “processor” and “processor cores” may be considered synonymous, unless the context clearly requires othervwise. Additionally, computer600may include mass storage devices606(such as diskette, hard drive, compact disc read only memory (CD-ROM) and so forth), input/output devices608(such as display, keyboard, cursor control and so forth) and communication interfaces610(such as network interface cards, modems and so forth). The elements may be coupled to each other via system bus612, which may represent one or more buses. In the case of multiple buses, they may be bridged by one or more bus bridges (not shown).

Each of these elements may perform its conventional functions known in the art. In particular, system memory604and mass storage devices606may be employed to store a working copy and a permanent copy of the programming instructions implementing the operations associated with SAM108ofFIG. 1and/or processes200,300and/or400ofFIGS. 2, 3, and/or4, earlier described, collectively referred to as computational logic622. The various elements may be implemented by assembler instructions supported by processor(s)602or high-level languages, such as, for example, C, that can be compiled into such instructions.

The number, capability and/or capacity of these elements610-612may vary, depending on whether computer600is used as a mobile device, a stationary device or a server. When use as mobile device, the capability and/or capacity of these elements610-612may vary, depending on whether the mobile device is a smartphone, a computing tablet, an ultrabook or a laptop. Otherwise, the constitutions of elements610-612are known, and accordingly will not be further described.

As will be appreciated by one skilled in the art, the present disclosure may be embodied as methods or computer program products. Accordingly, the present disclosure, in addition to being embodied in hardware as earlier described, may take the form of an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to as a “circuit,” “module” or “system.” Furthermore, the present disclosure may take the form of a computer program product embodied in any tangible or non-transitory medium of expression having computer-usable program code embodied in the medium.FIG. 7illustrates an example computer-readable non-transitory storage medium that may be suitable for use to store instructions that cause an apparatus, in response to execution of the instructions by the apparatus, to practice selected aspects of the present disclosure. As shown, non-transitory computer-readable storage medium702may include a number of programming instructions704. Programming instructions704may be configured to enable a device, e.g., computer600, in response to execution of the programming instructions, to perform operations associated with SAM master and agent107and108ofFIG. 1and/or processes200,300,400and/or500ofFIGS. 2, 3, 4 and/or 5. In alternate embodiments, programming instructions704may be disposed on multiple computer-readable non-transitory storage media702instead. In alternate embodiments, programming instructions704may be disposed on computer-readable transitory storage media702, such as, signals.

Referring back toFIG. 6, for one embodiment, at least one of processors602may be packaged together with memory having computational logic622(in lieu of storing on memory604and storage606). For one embodiment, at least one of processors602may be packaged together with memory having computational logic622to form a System in Package (SiP). For one embodiment, at least one of processors602may be integrated on the same die with memory having computational logic622. For one embodiment, at least one of processors602may be packaged together with memory having computational logic622to form a System on Chip (SoC). For at least one embodiment, the SoC may be utilized in, e.g., but not limited to, a smartphone or computing tablet.

Thus various example embodiments of the present disclosure have been described including, but are not limited to:

Example 1 may be a server for cluster computing comprising memory and input/output resources; one or more processors coupled with the memory and the input/output resources, to operate one of a plurality of application slaves of an application master; wherein the other application slaves are operated on other servers, which, together with the server, are members of a computing cluster. The server may further comprise a service assurance manager agent to manage allocation of the one or more processors, the memory and the input/output resources to the application slave, to assure compliance with a node level service level agreement, derived from an application level service level agreement, to contribute in proximate assurance of compliance with the application level service agreement; wherein the application level service agreement specifies the aggregate service level to be provided by the application master and slaves.

Example 2 may be example 1, wherein the service assurance manager agent may receive the node level service level agreement from a service assurance manager master.

Example 3 may be example 1, wherein the service assurance manager agent may query the application slave for processor, memory or input/output resource needs.

Example 4 may be example 1, wherein the service assurance manager agent may query the application slave on one or more performance metrics.

Example 5 may be any one of examples 1-4, wherein the service assurance manager agent may select one or more of the one or more processors, memory or input/output resources for resource allocation or de-allocation adjustments, or reduce resource needs of other application slaves on the server, wherein the other application slaves are associated with other application masters.

Example 6 may be example 5, wherein the service assurance manager agent may provide one or more throttling commands to the application slave or to another application slave on the server, wherein the other application slave is associated with another application master.

Example 7 may be example 6, wherein throttling commands may comprise a command to preempt a process of the other application slave, a command to decrease a number of processes of the other application slave, or a command to pause the other application slave.

Example 8 may be example 7, wherein the throttling commands may further comprise a command to increase a number of processes of the other application slave, or a command to resume the other application slave.

Example 9 may be example 6, wherein the service assurance manager agent may provide one or more throttling commands to the other application master, through a service assurance manager master of the service assurance manager agent.

Example 10 may be example 5, wherein the service assurance manager agent may query the one or more processors, the memory or the L/O resources for status or resource availability.

Example 11 may be example 5, wherein the service assurance manager agent may provide one or more allocation commands to the one or more processors, the memory or the I/O resources to allocate additional resources of the one or more processors, the memory or the I/O resources to the application slave.

Example 12 may be example 5, wherein the service assurance manager agent may provide one or more de-allocation commands to the one or more processors, the memory or the I/O resources to de-allocate resources of the one or more processors, the memory or the I/O resources previously allocated to another application slave on the server, wherein the other application slave is associated with another application master.

Example 13 may be a method for managing cluster computing, comprising: operating, by a computing node of a computing cluster, at least one of a plurality of application slaves of an application master, in conjunction with other computing nodes of the computing cluster operating the other application slaves; and managing, by the computing node, with an service assurance manager agent, allocation of one or more processors, memory and input/output resources of the computing node to the application slave, to assure compliance with a node level service level agreement, derived from an application level service level agreement, to contribute to proximate assurance of compliance with the application level service agreement; wherein the application level service agreement specifies the aggregate service level to be jointly provided by the application master and slaves.

Example 14 may be example 13, wherein managing may comprise receiving, with the service assurance manager agent, the node level service level agreement from a service assurance manager master.

Example 15 may be example 13, wherein managing may further comprise the service assurance manager agent querying the application slave for processor, memory or input/output resource needs.

Example 16 may be example 13, wherein managing may further comprise the service assurance manager agent querying the application slave on one or more performance metrics.

Example 17 may be any one of examples 13-16, wherein managing may further comprise the service assurance manager agent selecting one or more of the one or more processors, memory or input/output resources for resource allocation or de-allocation adjustments, or reducing resource needs of other application slaves on the server, wherein the other application slaves are associated with other application masters.

Example 18 may be example 17, wherein managing may further comprise the service assurance manager agent providing one or more throttling commands to the application slave or to another application slave on the server, wherein the other application slave is associated with another application master.

Example 19 may be example 18, wherein throttling commands may comprise a command to preempt a process of the other application slave, a command to decrease a number of processes of the other application slave, or a command to pause the other application slave.

Example 20 may be example 19, wherein the throttling commands may further comprise a command to increase a number of processes of the other application slave, or a command to resume the other application slave.

Example 21 may be example 18, wherein managing may further comprise the service assurance manager agent providing one or more throttling commands to the other application master, through a service assurance manager master of the service assurance manager agent.

Example 22 may be example 18, wherein managing may further comprise the service assurance manager agent querying the one or more processors, the memory or the I/O resources for status or resource availability.

Example 23 may be example 17, wherein managing may further comprise the service assurance manager agent providing one or more allocation commands to the one or more processors, the memory or the I/O resources to allocate additional resources of the one or more processors, the memory or the I/O resources to the application slave.

Example 24 may be example 17, wherein managing may further comprise the service assurance manager agent providing one or more de-allocation commands to the one or more processors, the memory or the I/O resources to de-allocate resources of the one or more processors, the memory or the I/O resources previously allocated to another application slave on the server, wherein the other application slave is associated with another application master.

Example 25 may be one or more computer-readable media comprising instructions to cause a computing node of a computing cluster, in response to execution of the instructions by the computing node, to operate a service assurance manager agent to: manage allocation of one or more processors, memory and input/output resources of the computing node to one of a plurality of application slaves of an application master, operated on the computing node, to assure compliance with a node level service level agreement, derived from an application level service level agreement of the application master, to contribute to proximate assurance of compliance with the application level service agreement; wherein the application level service agreement specifies the aggregate service level to be jointly provided by the application master and slaves, wherein the other application slaves are operated on other computing nodes of the computing cluster.

Example 26 may be example 25, wherein the service assurance manager agent may receive the node level service level agreement from a service assurance manager master.

Example 27 may be example 25, wherein the service assurance manager agent may query the application slave for processor, memory or input/output resource needs.

Example 28 may be example 25, wherein the service assurance manager agent may query the application slave on one or more performance metrics.

Example 29 may be any one of examples 25-28, wherein the service assurance manager agent may select one or more of the one or more processors, memory or input/output resources for resource allocation or de-allocation adjustments, or reduce resource needs of other application slaves on the server, wherein the other application slaves are associated with other application masters.

Example 30 may be example 29, wherein the service assurance manager agent may provide one or more throttling commands to the application slave or to another application slave on the server, wherein the other application slave is associated with another application master.

Example 31 may be example 30, wherein throttling commands comprise a command to preempt a process of the other application slave, a command to decrease a number of processes of the other application slave, or a command to pause the other application slave.

Example 32 may be example 31, wherein the throttling commands further comprise a command to increase a number of processes of the other application slave, or a command to resume the other application slave.

Example 33 may be example 30, wherein the service assurance manager agent may provide one or more throttling commands to the other application master, through a service assurance manager master of the service assurance manager agent.

Example 34 may be example 29, wherein the service assurance manager agent may query the one or more processors, the memory or the I/O resources for status or resource availability.

Example 35 may be example 29, wherein the service assurance manager agent may provide one or more allocation commands to the one or more processors, the memory or the I/O resources to allocate additional resources of the one or more processors, the memory or the I/O resources to the application slave.

Example 36 may be example 29, wherein the service assurance manager agent may provide one or more de-allocation commands to the one or more processors, the memory or the I/O resources to de-allocate resources of the one or more processors, the memory or the I/O resources previously allocated to another application slave on the server, wherein the other application slave is associated with another application master.

Example 37 may be a server for cluster computing, comprising: means for operating, by a computing node of a computing cluster, at least one of a plurality of application slaves of an application master, in conjunction with other computing nodes of the computing cluster operating the other application slaves; and means for managing, by the computing node, allocation of one or more processors, memory and input/output resources of the computing node to the application slave, to assure compliance with a node level service level agreement, derived from an application level service level agreement, to contribute to proximate assurance of compliance with the application level service agreement; wherein the application level service agreement specifies the aggregate service level to be jointly provided by the application master and slaves.

Example 38 may be example 37, wherein means for managing may comprise means for receiving the node level service level agreement from a service assurance manager master.

Example 39 may be example 37, wherein means for managing may further comprise means for querying the application slave for processor, memory or input/output resource needs.

Example 40 may be example 37, wherein means for managing may further comprise means for querying the application slave on one or more performance metrics.

Example 41 may be example 37-40, wherein means for managing may further comprise means for selecting one or more of the one or more processors, memory or input/output resources for resource allocation or de-allocation adjustments, or reducing resource needs of other application slaves on the server, wherein the other application slaves are associated with other application masters.

Example 42 may be example 41, wherein means for managing may further comprise means for providing one or more throttling commands to the application slave or to another application slave on the server, wherein the other application slave is associated with another application master.

Example 43 may be example 42, wherein throttling commands comprise a command to preempt a process of the other application slave, a command to decrease a number of processes of the other application slave, or a command to pause the other application slave.

Example 44 may be example 43, wherein the throttling commands further comprise a command to increase a number of processes of the other application slave, or a command to resume the other application slave.

Example 45 may be example 42, wherein means for managing may further comprise means for providing one or more throttling commands to the other application master, through a service assurance manager master of the service assurance manager agent.

Example 46 may be example 42, wherein means for managing may further comprise means for querying the one or more processors, the memory or the I/O resources for status or resource availability.

Example 47 may be example 41, wherein means for managing may further comprise means for providing one or more allocation commands to the one or more processors, the memory or the I/O resources to allocate additional resources of the one or more processors, the memory or the I/O resources to the application slave.

Example 48 may be example 41, wherein means for managing may further comprise means for providing one or more de-allocation commands to the one or more processors, the memory or the I/O resources to de-allocate resources of the one or more processors, the memory or the I/O resources previously allocated to another application slave on the server, wherein the other application slave is associated with another application master.