Abstract:
The invention relates to a system and a method for tracking and executing a job comprising a series of tasks. Each task may be executed on a separate computing device. The method comprises having a workload manager to identify an initial schedule of implementation for the job; having agents to selectively control execution of the tasks; and utilizing a publish/subscription interface between the workload manager and the agents to isolate the communications of the workload manager from the agents. The workload manager and the agents each subscribe and schedule execution of and reporting of the tasks through the publish/subscription interface.

Description:
RELATED APPLICATION  
       [0001]     This application is related to U.S. Provisional application No. 60/614,493 filed on Oct. 1, 2004. 
     
    
     FIELD OF THE INVENTION  
       [0002]     The present invention relates to a system and method managing, scheduling controlling and monitoring execution of jobs by a job scheduler.  
       BACKGROUND  
       [0003]     Job scheduling systems provide a centralized system in hardware and software for processing large scale tasks. Typically, large scale tasks are broken down into several smaller tasks which are individually, sequentially and coincidentally executed according to a master task schedule to complete the task. For example, for a large company, a job scheduling system may be used to process its payroll payment. The payroll task may be broken down into the following processes: (i) access the company database for a list of the employees; (ii) execute a payroll program to identify salary payments to be made; (iii) execute a deposit program to make electronic bank deposits of the salary payments to the bank accounts of the employees; (iv) execute a report generator to print and send pay stubs to the employees. The payroll payment may be automated to occur at regular intervals. A job scheduling system utilizes a series of agents generally operating on computers to perform the smaller tasks. A workload manager controls each of the agents. The manager is connected to the agents using a communication network configured in a “star” pattern, with the workload manager at the center and each of the agents on a ray of the star.  
         [0004]     In a network based system, agents operate on separate computers and each of the computers communicate to a central computer running the workload manager. IP is a commonly used communication protocol. The workload manager needs to track each agent for its status and job completion. When an agent has a fault, e.g. its communication link is broken, the workload manager must be able to recognize the fault and take corrective action, if possible. For example, upon the detection of a fault in an agent, a backup agent on a different computer may be brought in to take the place of the agent. With large systems having many tasks, it is a non-trivial exercise for the workload manager to track and manage the operation of all of the agents.  
         [0005]     With a “star” network configuration, if the agents are allowed to move, there is a need to keep accurate data about the IP addresses of the agents. In practice it is a labor intensive task.  
         [0006]     Failover of agents (i.e. providing backup for agents) is difficult to achieve because communication between agents and the workload manager is restricted to one instance of an agent and the workload manager. Further, it is difficult to maintain “shadow” agents and to re-assign schedules to different agents after a schedule has been created.  
         [0007]     Under heavy load conditions, the workload manager may be overloaded with events. If it cannot process job events, they are queued and processing is delayed. This reduces overall productivity and reduces utilization of enterprise tools.  
         [0008]     Also, the “star” architecture creates a performance bottleneck and a single point of failure. If the manager is down, no workload can be executed at all. Jobs that have to run at the time of failure will be delayed.  
         [0009]     There is a need for a system and method which addresses deficiencies in the prior art.  
       SUMMARY  
       [0010]     In a first aspect, a method for tracking and executing a job comprising a series of tasks is provided. Each task may be executed on a separate computing device. The method comprises having a workload manager to identify an initial schedule of implementation for the job; having agents to selectively control execution of the tasks; and utilizing a publish/subscription interface between the workload manager and the agents to isolate the communications of the workload manager from the agents. The workload manager and the agents each subscribe and schedule execution of and reporting of the tasks through the publish/subscription interface.  
         [0011]     In a second aspect, a method for tracking and executing a job comprising a series of tasks is provided. Each task may be executed on a separate computing device. The method comprises having a workload manager to identify an initial schedule of implementation for the job; having agents to selectively control execution of the tasks; and utilizing a publish/subscription interface between the workload manager and the agents to isolate the communications of the workload manager from the plurality of agents. The workload manager provides a schedule for the job to the publish/subscription service. Also the agents manage execution of their associated tasks by subscribing to the service for the schedule and extracting the associated tasks from the schedule after it is provided thereto.  
         [0012]     In a third aspect, a system implementing any of the above noted aspects is provided.  
         [0013]     In other aspects various combinations of sets and subsets of the above aspects are provided. 
     
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0014]     The foregoing and other aspects of the invention will become more apparent from the following description of specific embodiments thereof and the accompanying drawings which illustrate, by way of example only, the principles of the invention. In the drawings, where like elements feature like reference numerals (and wherein individual elements bear unique alphabetical suffixes):  
         [0015]      FIG. 1  illustrates a block diagram of job scheduler system providing an embodiment of the invention;  
         [0016]      FIG. 2  illustrates an exemplary job provided to the system of  FIG. 1  for scheduling and processing;  
         [0017]      FIG. 3  illustrates a components of a workload manager of the job scheduler system of  FIG. 1 ;  
         [0018]      FIGS. 4-8  illustrate aspects of the job scheduler of  FIG. 1  while processing the exemplary job of  FIG. 2 ; and  
         [0019]      FIGS. 9-12  illustrate aspects of another embodiment of the job scheduler of  FIG. 1  while processing a part of the exemplary job of  FIG. 2 . 
     
    
     DETAILED DESCRIPTION  
       [0020]     The description which follows, and the embodiments described therein, are provided by way of illustration of an example, or examples, of particular embodiments of the principles of the present invention. These examples are provided for the purposes of explanation, and not limitation, of those principles and of the invention. In the description, which follows, like parts are marked throughout the specification and the drawings with the same respective reference numerals.  
         [0021]     In  FIG. 1 , system  100  illustrates an embodiment and comprises a workload manager (WM)  102 , publish/subscription system (PSS)  104  and agents  106 . WM  102  communicates with PSS  104  via bidirectional communication link  108 . Each agent  106  communicates with PSS  104  via a bidirectional communication link  110 . In one embodiment each of WM  102 , PSS  104  and agents  106  is a software program and each operates on a separate computer. Each of WM  102 , PSS  104  and agents  106  have access to internal and external data and databases (not shown). Communication between elements in system  100  may utilize messages created by a sender and sent over the respective links containing commands and data which can be extracted and acted upon by the recipient.  
         [0022]     PSS utilizes a publish/subscribe paradigm, where information producers submit data (as publications) to it and information consumers indicate their interests by submitting subscriptions. On receiving a publication, the system matches the data with the set of clients&#39; subscriptions, notifying the interested parties. This system provides decoupling of publishers and subscribers. This decoupling is useful for job scheduling, as the individual job monitors/controllers, or agents, are lightweight and potentially quite numerous. The address-free automatic content-based routing relieves WM  102  of the burden of maintaining connection state information and routing messages to and from all the agents.  
         [0023]     In the embodiment, PSS  104  is a server for one or more clients. There are at least two types of clients. The first type is a publisher (producer) of events. This is typically WM  102 . The second type is a subscriber to (consumer of) events. These are typically agents  106 . Subscribing clients send subscriptions to PSS  104 . Subscriptions can be in the form of messages and contain any task names, status for tasks, any operating, function, geographic or other pertinent characteristic of the client. For example, the subscription can identify the network location of the client, the operating system resident on the client, the list of agent applications resident on the client, status information, information about its immediate neighbours, etc. Publications relate to a status of a task and its task name. For example, publications can relate to a failure of a task or a success of a task and can be published in a publish/subscribe system. Publishing client sends publications to PSS  104 . PSS  104  matches information extracted from publications to information extracted from subscriptions. When a match is found, the publication is forwarded to all subscribers of the publication. PSS  104  maintains a status list for tasks sent by publishers to it. The status list tracks the tasks and the responsible agent (subscriber) for the tasks.  
         [0024]     Referring to  FIG. 2 , exemplary job  200  is shown which is executed by system  100 . Job  200  comprises tasks performed by agents A 1 , A 2  and A 3 . At the top of job  200 , Agent A 1  performs its task T 1  then tasks T 2  and T 3  are performed separately and simultaneously by agents A 2  and A 3 . Upon successful completion of the tasks T 2  and T 3 , a further task T 4  is performed by agent A 1 . Subsequently, upon successful completion of task T 4 , task T 5  is performed by agent A 2  and upon successful completion of task T 5 , final task T 6  is performed by agent A 3 . WM  102  and PSS  104  control activation of agents  104  for job  200 .  
         [0025]     In  FIG. 3 , further detail on WM  102  is provided. WM  102  comprises scheduler  302  and manager  304 . Scheduler  302  provides a database and scheduling routine to store tasks and defines a schedule and process for executing the stored tasks. Manager  304  evaluates conditions that have to be satisfied for execution of an element in the schedule and provides execution information in the form of publication information to PSS  104 .  
         [0026]     In defining a schedule for a job, such as job  200  ( FIG. 2 ), a job graph associated with the job is defined and then converted into a series of subscriptions and publications. This conversion may be done using known publication/subscription conversion techniques known in the art.  
         [0027]     Referring to  FIGS. 1-8  a description of an exemplary execution of job  200  is provided for the above noted configuration of WM  102 . Referring to  FIG. 4 , at the first step, agents  106  each subscribe to PSS  104  for commands to execute. The software at each agent  106  identifies which command it needs to be triggered to activate. Meanwhile, scheduler  302  has a process schedule relating to job  200 . The process schedule defines the relationships among the defined publications and subscriptions for job  200 . Job  200  is typically activated by an activation trigger (such as a timer signal). Upon receiving the activation trigger, scheduler  302  creates a schedule relating to job  200  and sends it to manager  304 .  
         [0028]     When manager  304  receives the schedule, it starts execution of the schedule by publishing to PSS  104  a command to execute Task T 1 . PSS reviews a status list for tasks and can determine that agent A 1  has subscribed to Task T 1 . As such, PSS  104  sends a command to agent A 1  to execute Task T 1 .  
         [0029]     When agent A 1  receives this command, it sends a command to its target to execute task T 1 . In the example, task T 1  is successfully completed by the target and sends agent A 1  a “DONE” message. Subsequently, Agent A 1  receives the “DONE TASK T 1 ” message from the target. In response to receiving that message, it publishes a “DONE TASK T 1 ” message to PSS  104 . PSS  104  receives the message and checks it against its status list. As there is a match in the list for Task T 1  and sends a “DONE TASK T 1 ” message to manager  304 . Next, manager  304  marks task T 1  as completed and accordingly PSS  104  unsubscribes task T 1  from A 1 .  
         [0030]     For the next step, manager  304  consults the schedule and publishes commands to A 2  and A 3  to execute tasks T 2  and T 3 . At the same time manager  304  subscribes to a status list in PSS  104  to receive messages from agents A 2  and A 3 . PSS  104  sends messages execute tasks T 2  and T 3  to agents A 2  and A 3 . Agents A 2  and A 3  receive the messages and send command to their respective targets to execute. In the example, target of agent A 2  fails and sends failure indication to it. Meanwhile, target of agent A 3  successfully completes and send a success message to agent A 3 . Upon receipt of the failure message, agent A 2  publishes a corresponding failure message to PSS  104 . As manager  304  has subscribed to messages from agent A 2 , PSS  104  sends a failure message to manager  304 . Meanwhile, agent A 3  publishes a success message to PSS  104  and PSS  104  sends a pass message to manager  304 . Upon receipt of the each message, manager  304  marks task T 2  for agent A 2  as failed and task T 3  for agent A 3  as completed.  
         [0031]     As task T 3  has failed, in job  200 , task T 4  cannot be started. As such, the processing of job  200  ends without full completion. In other task arrangements, failure of some tasks may be tolerated.  
         [0032]     Referring to  FIGS. 1, 2  and  9 - 12 , in another embodiment, functionality of manager  304  is ported into each agent  106 . Agents A 1 , A 2  and A 3  subscribe to PSS  104  for schedules that contain their names. Meanwhile, scheduler  302  creates a schedule with assignment of tasks for agents A 1 , A 2  and A 3 .  
         [0033]     For the next step in processing job  200 , scheduler  302  publishes the schedule for job  200  with PSS  104 ′ and PSS  104 ′ sends schedule to agents A 1 , A 2  and A 3 , as they are identified in the schedule.  
         [0034]     Next, agents A 1 ′, A 2 ′ and A 3 ′ each parse the received schedule and select tasks that identify them as the active agent.  
         [0035]     In the next step, agents A 1 ′, A 2 ′ and A 3 ′ process their respective tasks. The agents identify external, internal and environmental conditions required by their respective tasks. Agent A 1 ′ is used for tasks T 1  and T 4 . For task T 1 , agent  1  identifies that all conditions are met and accordingly, sends a command to its target to execute task T 1 . For task T 4 , agent A 1 ′ cannot execute it until successful completion of both tasks T 2  and T 3 . As such, it subscribes to the status list for tasks T 2  and T 3 . Meanwhile, agent A 2  identifies external condition for tasks T 2  (which depends on the successful completion of task T 1 ) and task T 5  (which depends on the successful completion of task T 4 ). As there are outstanding conditions not yet met, neither task can be executed and subscribes to the status list for tasks T 1  and T 4 . Agent A 3 ′ identifies external conditions of task T 3  (which depends on successful completion of task T 1 ) and task T 6  (which depends on the successful completion of task T 5 ). As there are outstanding conditions not yet met, neither task can be executed and agent A 3 ′ subscribes to the status list for tasks T 3  and T 5 .  
         [0036]     In operation, once task T 1  completes execution, it sends a success indication to agent A 1 ′. Agent A 1 ′ sends a success status message to PSS  104 ′, which then identifies a match with any subscriptions wanted for agents A 2 ′ and A 3 ′. Accordingly, PSS  104 ′ sends status messages to agents A 2 ′ and A 3 ′. Agents A 2 ′ and A 3 ′ receive the status messages and both identify that conditions for T 1  are met. As such, each agent A 2 ′ and A 3 ′ sends a command to execute respective tasks T 2  and T 3 .  
         [0037]     Presuming successful completion of tasks T 2  and T 3 , each task sends a success indication to their agents and each agent sends success status message to PSS  104 ′. PSS  104 ′ then identifies a match with the subscription for agent A 1 ′. Accordingly, PSS  104 ′ sends status messages to agent A 1 ′. Agent A 1 ′ receives the status messages and identifies that conditions for T 4  are met. As such, agent A 1 ′ sends a command to execute task T 4 .  
         [0038]     Presuming successful completion of task T 4 , it sends a success indication to agent A 1  which then sends success status message to PSS  104 ′. PSS  104 ′ then identifies a match with the subscription for agent A 2 . Accordingly, PSS  104 ′ sends status messages to agent A 2 ′. Agent A 2 ′ receives the status messages and identifies that conditions for T 5  are met. As such, agent A 2 ′ sends a command to execute task T 5 .  
         [0039]     Finally, presuming successful completion of task T 5 , it sends a success indication to agent A 2 ′ which then sends success status message to PSS  104 ′. PSS  104 ′ then identifies a match with the subscription for agent A 3 ′. Accordingly, PSS  104 ′ sends status messages to agent A 3 ′. Agent A 3 ′ receives the status messages and identifies that conditions for T 6  are met. As such, agent A 2 ′ sends a command to execute task T 6 . Upon completion of task T 6 , agent A 2 ′ sends a status message to PSS  104 ′, which then recognizes that job  200  is completed and sends a status message to scheduler  302 ′.  
         [0040]     It will be appreciated that the order of scheduling, subscribing and reporting messages to and from PSS  104 ′ and PSS  104  may be provided in any order, as long as the continuity of the scheduled job is maintained.  
         [0041]     It will be further appreciated that other lists and messaging techniques may be implemented in either PSS  104  or its clients using techniques known in the art to achieve necessary synchronization of activation of tasks by the connected clients.  
         [0042]     Although the invention has been described with reference to certain specific embodiments, various modifications thereof will be apparent to those skilled in the art without departing from the scope of the invention as outlined in the claims appended hereto.