Abstract:
A batch job multiplex processing method which solves the problem that a system which performs multiplex processing including parallel processing on plural nodes cannot cope with a sudden increase in the volume of data to be batch-processed using a predetermined value of multiplicity, for example, in securities trading in which the number of transactions may suddenly increase on a particular day. The method dynamically determines the value of multiplicity of processing including parallel processing in execution of a batch job on plural nodes. More specifically, in the method, multiplicity is determined depending on the node status (node performance and workload) and the status of an input file for the batch job.

Description:
[0001]    The present application claims priority from Japanese application serial No. 2009-172674, filed on (Jul. 24, 2009), the content of which is hereby incorporated by reference into this application. 
       FIELD OF THE INVENTION 
       [0002]    The present invention relates to a technique for effective batch processing. More particularly, it relates to a technique which determines the optimum processing multiplicity in parallel execution of batch jobs using plural nodes for high speed batch processing of large volumes of account data. 
       BACKGROUND OF THE INVENTION 
       [0003]    JP-A-2008-226181 proposes a technique for execution of batch jobs. In this technique, script data about job nets which defines the order of execution of jobs is received and a request for allocation of resource nodes for execution of the job nets is issued on a per-job-net basis in accordance with the script data so that resource nodes are allocated to each job net in response to the allocation request. 
       SUMMARY OF THE INVENTION 
       [0004]    In batch processing, there are cases where the volume of data to be processed suddenly increases. For example, in the securities industry, it is necessary to cope with various situations: for example, all accounts for month-end reinvestment of investment trusts must be processed on a particular day; the number of stock transactions suddenly increases in some economic climate; and when there are many initial public offerings (IPO) in a short time, an increasing number of transactions must be dealt with, resulting in a longer batch processing time. Consequently, the volume of batch processing jobs largely varies day by day and sometimes a longer time must be taken for batch processing. This is likely to lead to a delay in the start of next day&#39;s online service and a shorter online service time for customers. Also, such lengthened batch processing may affect processing time for another job which is executed on the same node simultaneously, again resulting in a delay in the start of online service related to that job. Therefore, daily batch processing time should be constant even when the volume of data to be processed varies from day to day. 
         [0005]    In order to address the above problem, the present invention dynamically determines multiplicity of processing including parallel processing in execution of a batch job on plural nodes. More specifically, the invention provides a system which flexibly determines execution multiplicity and execution nodes to shorten batch processing time by effective utilization of resources. Processing time can be made (almost) constant regardless of the number of batch jobs by batch processing in a scale-out manner on a particular day when the number of batch jobs increases. This eliminates the possibility that a long time is taken to batch-process large volumes of data on a particular day and a delay in the start of next-day online service occurs. 
         [0006]    There are many types of batch processes: some batch processes require CPU resources and others require disk resources. In the present invention, the user can specify parameters for each job group and choose one of two methods for determining execution multiplicity so that the user can determine execution multiplicity by the more suitable method for the type of jobs and the location of input data to shorten batch processing time more effectively. 
         [0007]    According to the present invention, batch processing is performed more efficiently. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0008]      FIG. 1  shows a system configuration according to a preferred embodiment of the invention; 
           [0009]      FIG. 2  shows the content of a node management table on a job management node; 
           [0010]      FIG. 3  shows the content of a sub job management table on the job management node; 
           [0011]      FIG. 4  shows the content of a job management table on the job management node; 
           [0012]      FIG. 5  shows the content of a data location information table on the job management node; 
           [0013]      FIG. 6  shows the content of a job group execution condition table on the job management node; 
           [0014]      FIG. 7  shows the content of a job group execution node group table on the job management node; 
           [0015]      FIG. 8  shows a job execution flow according to the preferred embodiment of the invention; 
           [0016]      FIG. 9  shows the first half of a flow of multiplicity determination by a sub job synchronization method; 
           [0017]      FIG. 10  shows the second half of the flow of multiplicity determination by the sub job synchronization method; and 
           [0018]      FIG. 11  shows a flow of multiplicity determination by a sub job parallel method. 
       
    
    
     DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
       [0019]    Next, the preferred embodiment of the present invention will be described in detail referring to the accompanying drawings. The embodiment explained below is illustrative and the invention is not limited thereto. 
         [0020]    For the sake of simplicity, the explanation given below is based on the assumption that the number of CPU cores  204  is allocated to one batch process, but the system does not depend on the number of physical CPU cores and processing multiplicity (number of CPU cores  204 ) can be freely set for each node  201 . Even when plural threads such as multi-threads or hyper-threads are used, processing multiplicity can also be freely set depending on the situation. 
         [0021]      FIG. 1  shows a system configuration according to the preferred embodiment of the invention. This system includes a client node  101 , a job management node  102 , and job execution nodes  103  to  105 . These components are interconnected in a way that they can communicate with each other. The user can access the system through the client node  101  to set parameters. Specifically, the user can set minimum multiplicity  242 , maximum multiplicity  243 , a start key  244  and an end key  245  for object data to be processed, and an execution option  246  for a job group execution condition table  110 . Here, it does not matter what kind of means the user uses to set these parameters through the client node  101 . 
         [0022]    Next, the flow of processing steps in this embodiment will be described referring to the flowcharts ( FIGS. 8 to 11 ). 
         [0023]    First, prior to starting job execution, parameter values have been entered for a node management table  109 , a job management table  108 , a data location information table  112 , job group execution condition table  110 , and a job group/execution node group table  114  on the job management node  102 . Here, the type, entry method and location of parameters do not matter. 
         [0024]    When a job group start condition (for example, timed start) is met, a job management section  106  starts a job group (Step  301 ). Job group start conditions here are the same as conventional job start conditions and there are various types of job start conditions: for example, timed start, log/event monitoring, preceding job, file creation, and manual function. In this embodiment, it does not matter what type of start condition is adopted. 
         [0025]    As a start condition is met and the job group is started, the job management section  106  of the job management node  102  acquires the minimum multiplicity  242 , maximum multiplicity  243 , object data start key  244  and end key  245  for object data to be processed, and execution option  246  for the job group from the job group execution condition table  110  (Step  302 ). 
         [0026]    Next, the job management section  106  acquires information on the node group  252  corresponding to the started job group  251  from the job group/execution node group table  114  (Step  303 ). 
         [0027]    Next, the job management section  106  sends the minimum multiplicity  242 , maximum multiplicity  243 , object data start key  244  and end key  245  for object data to be processed for the job group, and information on the execution node group  252  to a node multiplicity calculating section  107 , and the node multiplicity calculating section  107  calculates multiplicity in job execution (Step  304 ). According to the execution option  246  sent by the job management section  106 , the node multiplicity calculating section  107  decides whether the multiplicity for the job group is determined by the sub job synchronization method or sub job parallel method (Step  305 ). 
         [0028]    Next, how multiplicity in job execution is determined in the sub job synchronization method and the sub job parallel method will be explained. 
         [0029]    First, the process of determining multiplicity by the sub job synchronization method is explained. In this method, processing multiplicity is determined depending on the workload on the CPU of each of the job execution nodes  103  to  105  in order to optimize multiplicity in execution of jobs. In this method, temporary multiplicity is first determined and then final multiplicity is determined based on the temporary multiplicity. Temporary multiplicity is multiplicity with which the largest number of cores among free cores are occupied (used), provided that it is within the range between minimum multiplicity  242  and maximum multiplicity  243  in the job group execution condition table  110 . In calculating final multiplicity based on the temporary multiplicity, the performances of the job execution nodes  103  to  105  are taken into consideration for the most effective use of the CPU resources. The determination of temporary multiplicity before the determination of final multiplicity makes it possible to find optimum multiplicity without calculating processing performances with different multiplicities, leading to reduction in multiplicity calculation time. 
         [0030]    As the node multiplicity calculating section  107  of the job management node  102  starts calculation (Step  314 ), comparison is made between maximum multiplicity  243  in the job group execution condition table  110  and the total number of free cores  206  in the node management table  109  (Step  315 ). As a result of comparison, if it is found that the total number of free cores  206  is not smaller than maximum multiplicity  243 , as many free cores as expressed by the maximum multiplicity are occupied with preference given to nodes with higher performance ratios in the node management table  109 . In this case, the total number of free cores  206  is taken as temporary multiplicity (Step  316 ). 
         [0031]    If the maximum multiplicity  243  is larger than the total number of free cores  206 , comparison is made between minimum multiplicity  242  in the job group execution condition table  110  and the total number of free cores  206  in the node management table  109  (Step  318 ). As a result of comparison, if it is found that the minimum multiplicity  242  is not larger than the total number of free cores  206 , the free cores are occupied and the number of free cores  206  is taken as temporary multiplicity (Step  317 ). If the minimum multiplicity  242  is larger than the total number of free cores  206 , the free cores  206  are occupied, provided that multiplicity value 1 is allocated to one node for as many nodes as expressed by the minimum multiplicity with preference given to nodes with higher performance ratios in the node management table  201  (Step  320 ). In this case, the value of temporary multiplicity is equal to the value of minimum multiplicity. 
         [0032]    If the number of free cores is zero, the node multiplicity calculating section  107  allocates CPUs in accordance with the CPU allocation method selected for each node in the node management table  201  (Step  321 ). If “OTHER NODE” is selected for the CPU allocation method, allocation is made to other nodes (Step  321 ). If “QUEUING” is selected for the CPU allocation method, the system waits until the number of free cores becomes 1 or more (Step  320 ). In this case, without affecting the execution of jobs occupying the CPUs at that time, the system waits until a preceding job releases a CPU and the CPU becomes free. 
         [0033]    At this stage, the node multiplicity calculating section  107  determines temporary multiplicity (Step  322 ). Once the temporary multiplicity has been determined, the node multiplicity calculating section  107  starts processing to determine (final) multiplicity based on the temporary multiplicity. 
         [0034]    First, the system decides whether the temporary multiplicity is equal to maximum multiplicity  243  (Step  323 ). If the temporary multiplicity is not equal to the maximum multiplicity  243 , throughput is calculated using temporary multiplicity+1 as multiplicity (Step  325 ). This throughput is an index representing the processing performance of each node as calculated from a performance ratio  203  and the number of CPU cores  204  in the node management table  201 . A job is processed by a higher-throughput node in a shorter time than by a lower-throughput node. 
         [0035]    If the total number of free cores is smaller than the number of jobs, the number of free cores/the number of jobs is calculated and the calculation result is taken as throughput (Step  324 ). 
         [0036]    After throughput calculation, comparison is made between throughput with temporary multiplicity and throughput with temporary multiplicity+1 (Step  326 ). If throughput with temporary multiplicity+1 is higher, using temporary multiplicity+1 as temporary multiplicity and again the system decides whether the temporary multiplicity is equal to the maximum multiplicity (Step  323 ). By repeating these steps, the system determines to which level below the maximum multiplicity the value of temporary multiplicity should be increased. 
         [0037]    Using a similar algorithm, the system determines to which level above the minimum multiplicity the value of temporary multiplicity should be decreased. In this case, comparison is made between throughput with temporary multiplicity and throughput with temporary multiplicity−1 (Step  330 ). If throughput with temporary multiplicity−1 is higher, temporary multiplicity−1 (temporary multiplicity minus 1) is taken as temporary multiplicity (Step  329 ). 
         [0038]    By adjusting the value of temporary multiplicity in accordance with the above algorithm, multiplicity corresponding to the highest throughput is calculated and determined as (final) multiplicity (Step  331 ). Here, multiplicity corresponding to the “second highest” throughput may be chosen instead of multiplicity corresponding to the “highest” throughput. 
         [0039]    After multiplicity has been determined as mentioned above, the node multiplicity calculating section  107  sends multiplicity information to the job management section  106 . 
         [0040]    Thus, the sub job synchronization method provides a system in which processing multiplicity is calculated depending on how the job execution nodes  103  to  105  are being used, so that jobs are executed with optimum multiplicity. 
         [0041]    Next, the process of determining multiplicity by the sub job parallel method is explained. This method provides a system which recognizes a node in which an input file for a job is located and executes the job on that node to minimize communication workload. Here, it does not matter how and where the input file is located. 
         [0042]    As the node multiplicity calculating section  107  starts multiplicity calculation in accordance with the sub job parallel method, the system refers to a data location information table  112  and acquires the number of divisions of the input file for the job to be executed (Step  332 ). This number of divisions is the multiplicity for the job to be executed (Step  333 ). Here, the node which executes a job should be the node on which the data to be processed for the job is located. For example, on a node in which key #1 to #100 files are located, the job for processing the key #1 to #100 files is executed. 
         [0043]    In the sub job parallel method, on a node in which a file to be processed is located, a job for processing the file is executed. This eliminates the need for processing a file located in another node, reducing the communication workload in job execution. 
         [0044]    Once multiplicity has been determined, the job management section  106  acquires information on execution of each sub job from the node multiplicity calculating section  107  and creates a sub job management table  113  (Step  308 ). 
         [0045]    The job execution command input section  111  of the job management node  102  sends a job execution command to the job execution nodes  103  to  105  with reference to the sub job management table  202  (Step  309 ). As the job execution nodes  103  to  105  receive the execution command, they execute jobs in accordance with the received job execution command (Step  310 ). 
         [0046]    After the jobs have been executed, the job management section  106  updates execution status information on each sub job in the sub job management table  202  (Step  311 ).