Abstract:
In one embodiment there is shown a method for improving multi-node processing, the method operable in a system having multi-node resources distributed across a network at various network nodes. The method of the embodiment comprises measuring application workload response time at each node in the system; communicating the measured application workload response time from each node where measurements are taken to a central point in the system; and from the central point, adjusting the resources available at each node so as to optimize the overall response time and throughput of work processed by the system.

Description:
FIELD OF THE INVENTION 
     The following description relates to workload balancing and more specifically to systems and methods for improving multi-node application processing. 
     DESCRIPTION OF RELATED ART 
     In many situations it is required that a computing task, or set of tasks is performed at different nodes of a distributed network. In an attempt to manage the response time of a computing task it is important to be able to add (or possibly subtract) resources at a point where bottlenecks exist. Simply identifying the response times at various nodes in the network and adding resources at a “slow” or bottleneck node, can, under some situations, actually increase the overall response time. 
     BRIEF SUMMARY OF THE INVENTION 
     In one embodiment there is shown a method for improving multi-node processing, the method operable in a system having multi-node resources distributed across a network at various network nodes. The method of the embodiment comprises measuring application workload response time at each node in the system; communicating the measured application workload response time from each node where measurements are taken to a central point in the system; and from the central point, adjusting the resources available at each node so as to optimize the overall response time and throughput of work processed by the system. 
     In a further embodiment there is shown a multi-node processing system comprising a plurality of resources running at different nodes; a network interconnecting the nodes; a resource manager for each node; a data gathering point common to the nodes; and a communication link between each resource manager and the data gathering point such that the data gathering point can monitor response time for each node, thereby controlling resources at any node found to be a bottleneck node. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  shows one embodiment of a multi-node distributed processing system; 
         FIG. 2  shows one embodiment of a system for collecting data from the nodes of the processing system of  FIG. 1 ; 
         FIG. 3  shows one embodiment of a flow chart for controlling workload balancing among nodes of a multi-node processing system; and 
         FIG. 4  shows one embodiment of a flow chart for determining which, if any, node is causing a response delay. 
     
    
    
     DETAILED DESCRIPTION 
       FIG. 1  shows one embodiment of multi-node distributed processing system  10  where different nodes in a computer system, such as nodes  102 - 105 , are interconnected by a communication network, such as network  11 A- 11 N. Network  11 A- 11 N could be portions of a single network, or different networks and can be wireline, wireless or a combination thereof. At node  102 , there are several instances,  102 A- 102 N, of web servers that handle various clients  101 A- 101 N. Each web server can handle a number of clients, each directed to one of a number of applications located on an application server  103 A- 103 N. At node  103 , there is shown three applications, (Sales, HR and Customer Support) each having multiple instances (sets) of the application handled by that particular server. These three applications are representative examples only and any number and/or types of applications can be substituted therefore. Each application set ( 103 A,  103 B,  103 N) can have different application types and applications that are likely to cause system slow down should be positioned in more than one set. Likewise, the concepts discussed herein could be used with any number of tiers as well. 
     In operation, a first client  101 A that desires certain data pertaining to sales sends a message to the system. In such a situation, client  101 A is connected through network  11 A to node  102  and is directed to web server  102 B by load balancer  150 . Web server  102 B, in turn, sends a message via network  11 B to load balancer  151  to select an available (sales) application. Load balancer  151 , in turn, selects a (sales) application in system (tier)  103 B at node  103 . The sales application then is connected, if desired, to database server  114  (app  1 ) at node  104  via network  11 C so as to obtain information from storage  115  via network  11 N at node  105 . 
     This system operation goes on for each request from a client  101 A- 101 N. Over a period of time, the workloads on the various instances of an application could get out of balance, thereby affecting the overall response time and throughput of the system. In this context, response time is defined as the time it takes an instruction from a client to be completed and the results returned to the client. One example of the problem (at the application node) would be when seven clients require access to application server (sales)  103 A. Assuming there are only six instances of sales applications, for example, the time for response at node  103  would go up and could possibly exceed the expected or desired response time of the system. Note that many other scenarios could arise at any of the nodes that could result in a higher than desired response time. For example, a web server (or portion thereof) could go down, or storage  115  could become slow to respond. 
     At each node it is possible to measure the response time from each application instance at each tier. This makes it possible to ensure that the response time of the node application is consistent between different instances of each clustered workload as is further consistent with the response time expected for the particular task. It is also possible, under control of resource managers  121 - 125  to determine the overall response time of the entire system, as will be discussed hereinafter. Each node can have, if desired, access to other applications, such as shown at servers  132 ,  133 , and  134 . 
       FIG. 2  shows an example of system  20  in which the response time measured from the various applications at each node in the network path being communicated over links  201 - 205  to common gathering point  21 . These communication links can be wireline or wireless and can be part of network  11 A- 11 N, or can be separate therefrom. Note that common gathering point  21  can be physically separated from the nodes or can be, if desired, part of a node. By collecting the response time from each node ( 102 - 105 ,  FIG. 1 ), or from those nodes known to be potential problem nodes, at common gathering point  21 , for example, under control of processor  22  and memory  23 , the node causing a delay (bottleneck) in the end-to-end response time can be determined. In one embodiment, the code for controlling the operations discussed herein could reside on media running at processor  22 . It should be appreciated that system  20  includes computer usable storage medium for at least partially controlling the operation of a computer (e.g., nodes  101 - 105 ). Examples of computer usable storage medium include, among other things, one or more electronic storage devices associated with computer systems, such as a database server  104 , an application server  103 , a web server  102  and so on, where the one or more electronic storage devices would have instructions stored thereon for various embodiments described herein. Other examples of computer usable storage medium include, among other things, a compact disk (CD) with instructions stored thereon that can be installed on one or more electronic storage devices associated with a database server  104 , an application server  103 , or a web server  102 . 
     When a bottleneck is determined, gathering point  21  issues directives to the workload management tools located at the node determined to be responding slower than anticipated (target node) instructing the target node to allocate more resources to the component that is experiencing performance degradation. These other resources can come from other applications that are on the node, for example, in a different tier. The system also has the ability to activate temporary capacity for this purpose. In this case, it would be possible for each node to have only one workload (application) as long as there was temporary capacity that could be applied if needed. Also, in some situations it might be possible to increase (perhaps temporarily) the speed of an application. 
     A counter-productive scenario could occur if each node were allowed to only analyze itself without regard to what is going on at other nodes. For example, the problem could occur if the measure of response time at the application server was slow, but the real reason was that the database was having problems. If the system added resources to the application server, it is possible that the result would be to just send more work to the database which would slow down even more. 
       FIG. 3  shows an operational flow diagram  30  for one embodiment where process  301  selects which nodes to monitor (this process is optional in this example), and process  302  samples end-to-end response time from, for example, client  101  to storage  105  (in the example shown in  FIG. 1 ). Note that any number of nodes could be monitored and that a sub-set of all nodes could be monitored. Also, note that the number of nodes monitored can be changed (for example, by the system administrator, or otherwise) from time to time, if desired. 
     Process  303  (which could be optional) determines whether the response time for the selected nodes is within the anticipated time bounds. This time can be a fixed time, a statistically determined time or a variable time, as desired. Optional process  308  adjusts the acceptable time depending on the number and type of nodes and/or other factors. The acceptable response times can be set differently for each tier, if desired. 
     When process  303  determines that the response time is unacceptable, process  304  reads the individual node process times and process  305  (for example, by the process shown in  FIG. 4 ) determines which node (the target node) is causing the throughput delay, i.e., is responding in the time anticipated. In this embodiment, the system does not react to a delay at one node until all the downstream nodes are checked for proper operation. 
     Process  306  sends a message to the target node (resource manager) to request additional resources be allocated so as to ease the problem. 
     Note that while the nodes are each shown (e.g., in  FIG. 1 ) with multiple instances of the same application, different applications at a node can also be accommodated, either by sending node response times as a whole to the central gathering point or by grouping the application types and sending response times for each group. Likewise, the control, gathering point can control individual group resources or can control the node resources as a whole. In the later case, the node will have individual group controllers to add resources as needed. 
       FIG. 4  shows a flow chart of one embodiment of a system and method for determining which, if any, node is causing a response delay. Since all nodes are monitored, flow chart determines which node is most likely to be the node in trouble. If a node is “slow”, it could be slow because that node is in trouble, or because one (or more) nodes “below” that node in the chain is slow. Thus, if storage  115  is not responding, all nodes ( 102 - 104 ) will also appear slow. 
     Following the logic of flow chart  40 , process  401  determines if node  102  is slower than anticipated. If not, then the problem, if any, is at client node  101  (process  410 ). If node  102  is slow and node  103  is not (as determined by process  402 ) then node  102  is the root cause of the problem (process  411 ). If node  103  is slow and node  104  is not (as determined by process  403 ) then node  103  is the problem (process  412 ). If node  104  is slow and node  105  is not (as determined by process  404 ) then node  104  is the problem (process  413 ). If node  105  is slow, as determined by process  404 , then the problem must be at node  105  (process  414 ) assuming no further lower nodes. Note that problems could exist at several nodes, but by starting at the end of the chain and working up to the top, the problems are cleared node by node.