Abstract:
A multiple task wait system and associated method allow a client application to wait for multiple tasks to be successfully or conditionally implemented before running subsequent tasks. Two mechanisms can be used to accomplish this multiple wait process: The first mechanism uses a multi-wait grouping process that is visible to the client, and the second mechanism uses a graphical representation to identify the tasks to be completed. The multi-wait grouping process allows a client to group a related set of tasks together for both control and documentation purposes. The client can add as many tasks as the resources of the computer allow to a group while defining the data flows and control flows between the tasks in the group using various graphical tools. The multi-wait system allows the client to define the constraints and conditions for a set of tasks to be considered complete, and further allows the system to define the constraints and conditions for considering all the tasks within the group to be completed. By utilizing the group concept, the system can selectively control the tasks to be included in the completion decision based on predefined rules.

Description:
FIELD OF THE INVENTION 
   The present invention generally relates to the field of data processing and particularly to a method and system for multi-scalar data processing, managing multiple tasks in one or more job threads. More specifically, this invention describes a system and a method that implement a design to wait for multiple tasks to complete prior to implementing a new task in a data warehouse environment. The system can use a logical grouping implementation or a graphical representation for control and documentation purposes. 
   BACKGROUND OF THE INVENTION 
   In a data warehouse environment, a task is the smallest entity that can be run as a single unit. Clients in the data warehouse environment often need to run Extraction/Transformation/Loading (ETL) tasks in one or more job threads. Each job thread usually contains a group of related tasks that must be completed before the next group of tasks can be started, so that the waiting tasks in the next group will not run multiple times. 
   A task is responsible for data extraction, transformation, and loading to the data warehouse. Clients usually have a group of tasks to perform on a regular basis. As an example, a client (or client application) may have three regions to account for and manage: Western, Eastern, and Central regions. The data for each region is distributed into databases. The client wishes to build a data warehouse for the transaction data collected during the day. Data must be pulled from each region, mapped, and processed. The results are then stored in the data warehouse. If the results of the Western region transactions depend on those of the Eastern region, the Eastern region data must be processed first and a dependency must be built into the processing instructions. 
   The client needs to carefully plan and manually link these related tasks in a group or across related groups with the required run conditions to keep from loading the data warehouse with redundant data. Grouping of tasks becomes a very tedious, manual task for the client, which is rendered even more complicated when groups of tasks are scheduled to run with multiple overlapped schedules. 
   Data warehouse clients need a relatively easy way to use a graphical client interface that allows them to group related tasks with predetermined run conditions and to run the grouped tasks automatically without running the same tasks multiple times. The need for such a system has heretofore remained unsatisfied. 
   SUMMARY OF THE INVENTION 
   The multiple task wait system and method of the present invention satisfy this need by allowing the client (or client application) to wait for multiple tasks (also referred to herein as “multiple wait”) to be successfully or conditionally implemented before running the next task (or tasks). Two mechanisms can be used to accomplish the multiple wait process: The first mechanism uses a multi-wait grouping process that is visible to the client, and the second mechanism uses a graphical representation to identify the tasks to be completed. 
   The multi-wait grouping process allows a client to group a related set of tasks together for both control and documentation purposes. The client can add as many tasks as the resources of the computer allow to a group while defining the data flows and control flows between the tasks in the group using various graphical tools. With the control flows, the client can link tasks together such that when one or more tasks are completed (on success, failure, or unconditionally), a follow-on task can be started. The tasks linked via control flows can be linked (i.e., drawn, connected and extended) to other tasks within this group or in a different multi-wait group. The multi-wait groups also provide a mechanism to allow a set of requirements (e.g., schedules for tasks to start or notifications when tasks are completed) to be added to all the tasks within that group. 
   Utilizing the multi-wait group concept for multiple wait allows the client to define the constraints and conditions for a set of tasks to be considered complete. The multi-wait group concept also allows the present system and associated method to define the constraints and conditions for considering all the tasks within the group to be completed. By utilizing the group concept, the present system and method can selectively control the tasks to be included in the completion decision based on predefined rules embedded in the system. 

   
     BRIEF DESCRIPTION OF THE DRAWINGS 
     The various features of the present invention and the manner of attaining them will be described in greater detail with reference to the following description, claims, and drawings, wherein reference numerals are reused, where appropriate, to indicate a correspondence between the referenced items, and wherein: 
       FIG. 1  is a schematic illustration of an exemplary operating environment in which a multiple task wait system of the present invention can be used; 
       FIG. 2  is an exemplary graphical illustration of a multiple wait relationship processed by the multiple task wait system of  FIG. 1 ; 
       FIG. 3  illustrates a first example showing two groups, Group 0  and Group 1 , wherein Group 0  contains a plurality of cascade tree tasks and branch tree nodes; 
       FIG. 4  illustrates a second example showing two groups, Group 0  and Group 1 , wherein Group 0  contains a root transient task; and 
       FIG. 5  is a process flow chart illustrating an exit method for the multiple task wait system of  FIG. 1 . 
   

   DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS 
   The following definitions and explanations provide background information pertaining to the technical field of the present invention, and are intended to facilitate the understanding of the present invention without limiting its scope: 
   Instance: In object-oriented technology, a member of a class; for example, “Lassie” is an instance of the class “dog.” When an instance is created, the initial values of its instance variables are assigned. 
   Internet: A collection of interconnected public and private computer networks that are linked together with routers by a set of standards protocols to form a global, distributed network. 
   SQL: Structured Query Language, a standardized query language for requesting information from a database. 
     FIG. 1  portrays the overall environment in which a multiple task wait system  10  of the present invention may be used in a data warehouse environment. System  10  includes a software or computer program product that is typically embedded within or installed on a host server  15 . Alternatively, the system  10  can be saved on a suitable computer usable medium such as a diskette, a CD, a hard drive, or like storage devices. While system  10  will be described in connection with a data warehouse environment or the World Wide Web (WWW), the system  10  can be used with a stand-alone database of documents or other text sources that may be derived from the WWW or other sources using wireless or cable connections. 
   The cloud-like communication network  20  is comprised of communication lines and switches connecting servers such as servers  25 ,  27 , to gateways such as gateway  30 . The servers  25 ,  27  and the gateway  30  provide the communication access to the WWW Internet. Clients, such as remote Internet clients are represented by a variety of computers such as computers  35 ,  37 ,  39 , and can query the host server  15  for the desired information. 
   The host server  15  is connected to the network  20  via a communications link such as a telephone, cable, or satellite link. The servers  25 ,  27  can be connected via high speed Internet network lines  44 ,  46  to other computers and gateways. The servers  25 ,  27  provide access to stored information such as hypertext or web documents indicated generally at  50 ,  55 , and  60 . The hypertext documents  50 ,  55 ,  60  most likely include embedded hypertext link to other locally stored pages, and hypertext links  70 ,  72 ,  74 ,  76  to other webs sites or documents  55 ,  60  that are stored by various web servers such as the server  27 . 
   System  10  includes an editor that graphically displays task flow relationships between tasks. An exemplary series of tasks as displayed by system  10  is illustrated in  FIG. 2 . As a data warehouse performs a series of tasks, Task 1   200  is completed first, and then both Task 2   205  and Task 3   210  are completed before Task 4   215  is started. 
   The order of completion for Task 2   205  and Task 3   210  may not be significant. However, both tasks must be completed before Task 4   215  can be started. Task 1   200  is referred to herein as a predecessor task while Task 2   205  and Task 3   210  are referred to as successor tasks. If, as illustrated in  FIG. 2  the tasks Task 1   200 , Task 2   205 , and Task 3   210  are related, the client can group them into a predecessor Group 220  and a successor Group 225  where the successor Group 225  has to wait for the predecessor Group 220  to complete. The client introduces a joint link from the predecessor Group 220  to the successor Group 225 . The joint link is a graphical instruction allowing the client to instruct the data warehouse that the successor Group 225  has to wait for the predecessor Group 220  to complete. As used herein, completion is based on any one or more of the following three conditions: success, failure, or unconditional. 
   System  10  provides two different techniques for creating a multiple wait using a graphical interface. With the first technique, the client draws a task flow line from Task 2   205  to Task 4   215 . Next, the client draws a line from Task 3   210  to the task flow line previously drawn from Task 2   205  to Task 4   215 . System  10  places a “connector” where the task flow lines join and the lines are connected. With the second technique, the client first draws the task flow line from Task 2   205  to Task 4   215  and then draws a task flow line from Task 3   210  to Task 4   215 . The client then selects both task flow lines, brings up a popup menu for the lines provided by system  10  and selects a menu choice that will combine the lines. 
   To start the implementation of the successor Group 225 , system  10  must be able to determine that the predecessor Group 220  has completed, and to determine the status of that completion. To this end, system  10  uses “exit criteria” to evaluate the group control flow condition. The exit process and related criteria will be described later in more detail. A group may contain multiple tasks in different run modes. A scheduled group is considered enabled so that only the tasks in the ready-to-run mode will be executed. 
   A group may have multiple job threads of tasks, wherein each job thread may have multiple branches. The task control flow condition can be on success, on failure or on completion (i.e., unconditional). Multiple task control flow conditions can be defined between two tasks in a given job thread. System  10  may not execute some of the tasks in the job thread branches if task control flow conditions are not met. The exit criteria must be able to dynamically determine if the task flow conditions occur, and to handle those tasks that do not meet the task flow conditions. A task in the job thread can also have multiple predecessor tasks or multiple task control flow conditions. Consequently, the exit criteria are able to count all the predecessor tasks before running the successor tasks to guarantee a meaningful run. 
   Furthermore, clients can schedule a group to run at a certain interval or specific time. This schedule for the group will apply to all the tasks in the group. Therefore, the root tasks of all the job threads in the group must observe this group schedule. However, these root tasks may have predecessor tasks originating from other groups. The successor tasks of the terminal tasks in the current group may also reside in another group. The exit criteria must be able to determine the root tasks and terminal tasks for a given group, in order to calculate the final completion status of the predecessor group, i.e.,  220 . The terminal task is the last task in each job thread, and it is the last executed task that does not have a successor task to trigger in the same job thread, This final completion status will then be used to determine if the successor group, i.e.,  225 , should be executed. The execution of the successor group can be conditional, on success or on failure, or unconditional completion. 
   Exit criteria requirements can be illustrated by examining two examples of possible grouping of the job threads.  FIG. 3  shows a first example  300 , illustrating a group, Group 0   315 , with multiple cascade task trees and branch tree nodes. Joint links between tasks are represented by arrows such as link  310 . Group 0   315  includes two job threads: job thread  320  and job thread  325 . Job thread  320  includes four tasks; Task 1   305 , Task 2   330 , Task 3   335 , and Task 7   340 . Job thread  325  includes two tasks, Task 5   335  and Task 6   350 . Job threads  320  and  325  are grouped into Group 0   315 , but are not connected. The successor Group 2   365  contains Task 8   355  that has a dependency on the completion of both of job threads  320 ,  325  in the predecessor Group 0   315 . 
   System  10  manages the execution of the various tasks through a set-up phase, behavior definition, and exit criteria. The setup phase for Group 0   315  includes the following steps:
         System  10  schedules and enables Group 0   315  and promotes all tasks to ready-to-run mode;   Group 0   315  uses Task 1   305 , Task 2   330 , Task 5   345 , Task 6   350 , and Task 7   340  to determine exit criteria;   System  10  defines the root tasks of Group 0   315  as Task 1   305  and Task 5   345 ; and   System  10  determines the terminal tasks for Group 0   315  as Task 5   335 , Task 6   350 , Task 7   340 , Task 1   305  and Task 2   330 . The terminal task is not predetermined, as the last executed task in a job thread is considered to be the terminal task. As an example, if Task 1   305  fails, Task 1   305  is considered to be the terminal task in job thread  320 ; if Task 1   305  and Task 2   330  are executed successfully, Task 2   330  is considered to be the terminal task.       

   System  10  defines the behavior of Group 0   315  as follows:
         Task 1   305  will run once by the cascade link of Task 0   370  on completion, but the status of Task 0   370  will not influence the exit status of Group 0  because Task 0  is not physically defined in Group 0 .   Task 4   360  will be run once by the cascade link of Task 3   335  on completion; but the status of Task 4  will not influence the exit status of Group 0  because Task 4  is not physically defined in Group 0 .   Task 7   340  will always be run if Task 1   305  is completed successfully and Task 2   330  fails. Task 2   330  will always run if Task 1   305  is completed successfully.   Task 6   350  will always be run if Task 5   345  is successful; and   Task 3   335  will be run only if Task 1   305  is successful and Task 2   330  fails.       

   Two criteria for Group 2   330  to run upon the successful completion of Group 0   315 , for the example  300  of  FIG. 3 , are as follows:
         Task 6   350  and Task 2   330  complete successfully; or   Task 6   350 , Task 7   340 , Task 1   305  is completed successfully, and Task 2   330  fails.       

   Referring now to  FIG. 4 , it illustrates another example  400 . For illustration purposes, the structure of the task flow is similar to example  300  of  FIG. 3 , except that Task 1  is a transient step to Task 2   420 . As described earlier, system  10  manages the set-up phase, defines the behavior of the process, and determines the exit criteria based on client input. However, Task 1   405  in example  400  is a transient task to Task 2   420  because of the data dependency between Task 1   405  and Task 2   420 , and is shown in a dashed line. 
   System  10  extracts and transforms the data in Table 0   410  to create Table 1   415 . Table 1   415  exists only as long as needed to complete Task 2   420 . While in the example  FIG. 3  each task has a task-to-task relationship, there is no predefined task-to-task relationship between Task 1   405  and Task 2   420  and therefore Task 1   405  can never directly run Task 2   420 . On the contrary, Task 1   405  can trigger Task 8   490  on failure because there is a task-to-task relationship defined between them, but there is no data dependency between these two tasks. 
   Since the data generated by Task 1   405  is not persistent and the generated data is used by Task 2   420 , Task 1   405  cannot be used as a root task in the job thread containing Task  2   420 . The requirements for a task to be a root task are as follows:
         1. The task must not have a predecessor task within the same group;   2. The task must be physically defined to the current group;   3. The task must be in a ready-to-run mode; and   4. The task must be a task that produces persistent warehouse data or has data dependency from the predecessor task.
 
Task 1   405  will not run until TASK 0   480  is complete. However, Task 1   405  resides in Group 0   425  and TASK 0   480  resides in Group 1   426 . Consequently, pursuant to the rules above, TASK 0   480  is not a root task for job thread containing Task 8   440 .
       

   The setup phase for Group 0   425  includes the following steps:
         System  10  schedules and enables Group 0   425  and promotes all tasks to the ready-to-run mode;   System  10  defines Task 5   450 , Task 1   405 , and Task 2   420  as the root tasks;   System  10  determines that Task 8   440  is not a root step because there is no data dependency between Task 1   405  and Task 8   440 , and thus, Task 8   440  does not consider Task 1   405  as a transient step; and   The terminal tasks in each job thread are determined dynamically, for example, system  10  may determines at execution time that the terminal tasks as Task 6   445 , Task 7   435 , Task 3   430 , and Task 8   440 .       

   System  10  defines the behavior of Group 0   425  as follows:
         For Task 2   420  to be run, system  10  executes Task 1   405  before Task 2   420  in order to generate the transient Table 1   415  with current data for Task 2   420  to use, but is not used to determine the exit status of Group 0   425 .   Task 4   455  will be run once by the cascade link of Task 3   430  on completion.   Task 7   435  will always be run if Task 1   405  is completed successfully.   Task 6   445  will always be run.   Task 8   440  will be run if Task 1   405  fails.   Task 3   430  will be run only if Task 1   405  is successful and Task 2   420  fails.       

   The criteria for Group  2  to run upon successful completion of Group  0 , for the example  400  of  FIG. 4  for Group 2   460  to start, are as follows:
         Task 6   445  and Task 7   435  and Task 1   405  complete successfully or   Task 6   445  and Task 7   435  and Task 1   405  complete successfully, and Task 2   420  fails but Task 3   430  is completed successfully   Task 6   445  and Task 7   435  complete successfully, and Task 1   405  fails but Task 8   440  completed successfully.       

   Once system  10  has defined the root tasks, each root task represents the beginning of a job thread. In  FIG. 4 , Task 2   420 , Task 5   450 , and Task 1   405  are root tasks. System  10  begins monitoring the execution of these initial job threads. Each job thread can branch and multiply. As an example, Task 2   420  branches into two tasks: Task 3   430  and Task 7   435 , creating more job threads to monitor and to manage in Group 0   425 . 
   System  10  manages each branch because as soon as a job thread terminates, the last task executed in each job thread is used to calculate the final group execution status (or completion status). In example  400 , job thread  465  contains Task 5   450  and Task 6   445 . When Task 5   450  is completed, system  10  executes Task 6   445 . At the successful completion of Task 6   445 , the job thread execution status is successful for job thread  465 . 
   Job thread  470  contains Task 2   420  and Task 7   435 . When Task 1   405  and Task 2   420  are completed, system  10  executes Task 7   435 . At the successful completion of Task 7   435 , the job thread execution status is successful for job thread  470 . 
   A task is not included in more than 1 job thread and it is run once. Therefore, if a task has two or more successor tasks, the first successor task will remain in the same job thread, and the second (and the other successor tasks) will start a new job thread (or threads). Job thread  475  includes Task 3   430 . Since Task 3   430  is executed on failure of Task 2   420  and successful completion of Task 1   405 , job thread  475  is successful only if Task 3   430  completes successfully. 
   Job thread 4   480  includes Task 1   405  and Task 8 . If Task 1   405  is completed successfully, or if Task 1   405  fails with Task  8  completed successfully, then Group 0   425  is also completed successfully. For the purposes of illustration in these examples, system  10  considers an entire group as having one or more job threads fail. 
   Task 2   420  has an ON COMPLETION branch and another ON FAILURE branch. Task 2   420  is included in job thread  420 . Task 2   420  will only be executed once, and the branch will take place only after the completion of the Task 2   420 . The definition of failure for a group can be customized by the client. 
   The performance of system  10  is also described by method  500  of  FIG. 5 . In operation, and with further reference to  FIG. 4 , system  10  first identifies at step  502 , all the job threads in each group, i.e., Group 0   425  of example  400 , so that the eligible tasks in the same thread can start execution. A group may have multiple job threads. Each job thread may contain one or more tasks. For performance reasons, some data warehouse clients choose to have a temporary task to produce transient (i.e., non persistent, temporary, or staging) data that will be deleted when it is not needed. 
   Once all the job threads are identified at step  502 , system  10  identifies, at step  503 , the root task in each job thread as the next task to run in each job thread. At step  504 , system  10  loops through all the job threads that do have not a completion status. If all the job threads are completed, system  10  calculates the final group exit status. The exit criteria for a group do not simply use the beginning task of each job stream as the root task because the beginning task may have a predecessor task physically defined outside of the current group or it may not be persistent. 
   At step  510 , system runs the next task in the current job thread. For example, system  10  will run Task 5   450  and check the status of that task. Based on the completion status of Task 5   450 , system  10  will check the link relationship. Once the predecessor task such as Task 5   450  is completed, system  10  follows the task link to Task 6   445 . These root tasks in the same group will carry the same run schedule specified by the user for the group. The schedule associated with each individual task is not applicable when system  10  runs the group. 
   System  10  then checks, at step  515 , the number of links the current task has; i.e., whether the current task has a successor task. In the case of Task 5   450 , one link exists to Task 6   445 , and system  10  checks to determine if the current task completion status matches the required link condition at step  525 . In this example, system  10  inquires if Task 5   450  has completed. If so, system  10  continues to step  510  and executes the successor step, step 6   445 , and repeats steps  515  through  510  as long as each task completes successfully until the job thread is completed, in this case, job thread  465 . 
   If system  10  determines, at step  515 , that there is more than one link for the current task being run, it proceeds to step  516 , where for the first control flow link, it proceeds to decision step  525  as explained earlier. Otherwise, for any subsequent control flow link, system  10  identifies the successor task as the current task in the new job thread and proceeds to step  520 . 
   At step  520 , system  10  adds a new job thread to the current group for the additional control flow link and proceeds to step  504 . For example, Task 2   420  has two links. System  10  adds job thread  475  to Group 0   425  at step  520 . Job thread  470  contains Task 2   420  and Task 7   435  while job thread  475  contains only Task 3   430 . The task execution of job thread  470  then continues looping through steps  525 ,  530 ,  515 , as described earlier. 
   When system  10  determines at step  515  that no more links exist for the current task (value=0), it proceeds to step  535  where it records the completion status of the current task as the completion status of the current job thread, and then proceeds to step  504 . Each set in the group may have multiple branch nodes causing more threads of execution to be created and managed. When all these threads of execution are completed, the group is completed. 
   System  10  calculates the exit status of the group based on the last executed task in each thread. For simplicity, if one or more of the last executed task fails, the group fails. The client may further define the rules for calculating the final group return status. The last executed task in a thread of execution must satisfy the following conditions:
         1. It has no successor task that is ready-to-run and is physically defined in the current group; and   2. it has no matching condition to cascade down to the successor task that is ready-to-run and is physically defined in the current group.
 
When a task fails, it may have a cascade on failure link to the successor task. If the successor task is the last task in the thread of execution and it completes successfully, system  10  considers this thread of execution successful.
       

   It is to be understood that the specific embodiments of the invention that have been described are merely illustrative of certain application of the principle of the present invention. Numerous modifications may be made to the wait for multiple tasks before running the next task invention described herein without departing from the spirit and scope of the present invention. Moreover, while the present invention is described for illustration purpose only in relation to mapping data from one database to another, it should be clear that the invention is applicable as well to any collection of data or databases accessible either through an internet, intranet connection, or any other link.