Abstract:
The present invention discloses a solution that automatically decomposes a batch process into multiple units of work without changing code of a preexisting batch application. In the solution, the batch application is first analyzed to identify a set of processing segments or units of work, where each unit of work is of a size that minimizes interference with other data accesses. Once each unit of work is defined, these units can execute one at a time. A syncpoint can be established for each unit before it is executed, which locks the records included in the unit. After the unit of work executes, the record lock can be released and a new syncpoint can be established for the next unit. If an execution problem occurs, execution for the unit of work can be terminated and chances can be restored to the syncpoint.

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
       [0001]    This application claims the benefit of British Patent Application No. 0623237.5 filed 22 Nov. 2006 and which is hereby incorporated by reference. 
     
    
     BACKGROUND 
       [0002]    1. Field of the Invention 
         [0003]    The present invention relates to the field of data processing and database technologies, and, more specifically, to issuing syncpoints during the execution of a batch application. 
         [0004]    2. Description of the Related Art 
         [0005]    In very complex data processing systems such as global banking networks run on mainframe systems, a transaction processing system (e.g., Customer Information Control System (CICS) from IBM) manages the interface between application programs and database records. If an application wishes to access a record (i.e., a customer&#39;s bank account balance stored on a database), then the transaction processing system will mediate the transaction. The transaction processing system will recall the record from the database, and will place a lock on the record so that no other application can access or update that specific record. Any read or write data requests relating to the record are then processed. Once the application program has finished with the record, a syncpoint is issued to the transaction processing system which results in the lock being removed from the record. No other application can access the record while the lock exists. 
         [0006]    In addition to application programs accessing records via a transaction processing system, batch applications are also used to update records. A batch application executes a set of updates which can include very large sets that can consume computing resources of transaction processing systems for an execution period. In the context of a banking system, a batch application may relate to a series of over the counter transactions that need to be applied to the computerized records representing the various bank account details of the customers. 
         [0007]    Conventional transaction processing systems impose a lock on all batch records while the batch is processing. Locking the records to prevent other applications from accessing or manipulating the locked records permits the systems to “roll-back” batch manipulations whenever a batch execution fails or whenever an authorized administrator wishes to cancel or reverse an active batch process for any reason. The roll-back removes all changes occurring after the syncpoint so that a data processing system state and record content is the same as it was when the syncpoint was established. 
         [0008]    In other words, before a batch application is started, a syncpoint is established. Once a batch application is started, each update is processed in turn. This involves accessing each record to be updated, locking the record and then performing the necessary update. Once all of the records referred to in a batch application have been processed, then a new syncpoint can be issued and all of the locked records can be unlocked and therefore made available to other applications. If a problem occurs during the batch, the pre-batch syncpoint can be used to reverse changes resulting from the aborted batch execution. 
         [0009]    Historically, batch processes were executed at times when real-time updates were disallowed. For example, real-time updates occurred during workday hours and batch processes were executed at night when no real-time accesses occurred. Consequently, batch applications did not interfere with other processing since a system could be taken “offline” during a time in which real-time accesses are blocked. All batch applications could be executed at this time. However, with the globalization of markets, integration of databases with remote computing systems, data center consolidation, Web database access capabilities for users (i.e., online banking), remote workplace software tools (e.g., CITRIX, PCANYWHERE, etc.), flexible work hours, and other factors have minimized or eliminated acceptable times for placing a system offline. That is, today&#39;s database systems often need to be available for real-time accesses twenty four hours a day. Any downtime can result in significant loss of service to key customers, and/or a loss of revenue to a data processing system owner. 
         [0010]    Businesses wish to achieve this capability without expensive changes to their data processing infrastructure. For example, businesses do not wish to replace or re-code batch processing applications which are working properly, yet which have an unfortunate side effect of preventing database access when executing. At the same time, businesses are compelled to provide competitive services which can include 24/7 Internet access to customer records and/or data availability to business partners and their computing systems at all times. 
         [0011]    No solutions without significant drawbacks currently exist for handling batch processing while providing increased data availability. For example, data processing centers have implemented partial solutions that are mainly based on tools that minimize the impact of the unavailability of data: either by careful scheduling or by limiting either the scope (i.e., number of data sets included in a batch) of batch related “outages” or their duration. The inadequate solutions that do exist often limit the ability of users to stay current and adopt new technologies since the partial solutions impose limitations that conflict with implementation requirements of emerging technologies, system upgrades, and new data processing or data interfacing techniques. To date, no solution has attempted to eliminate batch related downtime or outages entirely. 
       SUMMARY OF THE INVENTION 
       [0012]    The present invention discloses a solution that automatically decomposes a batch process into multiple units of work without changing code of a pre-existing batch application. In the solution, the batch application is first analyzed to identify a set of processing segments or units of work, where each unit of work is of a size that minimizes interference with other data accesses. That is, each unit of work is designed so that a set of records involved in the unit of work will be locked for an acceptably low period of time. Configurable parameters and/or automated system monitoring tools can be used to define the acceptable period of record locking, which in turn is used to define a set of records included in each unit of work. Once each unit of work is defined, these units can execute one at a time. A syncpoint can be established for each unit before it is executed, which locks the records included in the unit. After the unit of work executes, the record lock can be released and a new syncpoint can be established for the next unit. If an execution problem occurs, execution for the unit of work can be terminated and changes can be “rolled back” to the syncpoint. Thus, changes made by the terminated execution of the unit can be discarded and restored to a pre-execution state. 
         [0013]    The present invention can be implemented in accordance with numerous aspects consistent with the material presented herein. For example, one aspect of the present invention can include a method of increasing database record availability during a batch process execution. In the method, a batch process of a batch application can be identified. The batch process can be associated with a batch set of database records which are accessed by the batch process as the batch process executes. The batch process can be analyzed to determine multiple units-of-work. Each unit-of-work can be associated with a unit set of the database records, wherein the unit set can be a smaller subset of the batch set. A next one of the units-of-work can be determined. A syncpoint can be established for the determined unit-of-work. Records in the unit set associated with the determined unit-of-work can be locked so that applications other than the batch application are unable to modify the records while locked. The records in the unit can be batch processed in accordance with programmatic instructions of the batch application. Upon success of the processing step, the records in the unit set can be unlocked. The method steps can be repeated until each unit-of-work is processed. 
         [0014]    Another aspect of the present invention can include a method of operating a data processing system. The method can execute a batch application. The batch application can read one or more inputs from one or more data files, can perform updates on one or more records according to the inputs, and can issue a new syncpoint when each of the updates are completed. The method can monitor the inputs read from the data file. A predefined algorithm can operate based upon the monitored inputs. Output of the predefined algorithm can cause new syncpoints to be periodically issued during the execution of the batch application. Issuing a new syncpoint can commit record updates of a previous partial batch processing step. User intervention can occur whenever syndication feed updates are obtained by the feed reader. 
         [0015]    Still another aspect of the present invention can include a data processing system that includes a processing function arranged to execute a batch application. This processing function can execute to reading one or more inputs from one or more data files, to perform updates on one or more records according to the inputs, and to issue a syncpoint when said updates are completed. The processing function can monitor the inputs to operate a predefined algorithm based upon the monitored inputs. A syncpoint can be periodically issued during the execution of the batch application according to an output of the predefined algorithm. 
         [0016]    It should be noted that various aspects of the invention can be implemented as a program for controlling computing equipment to implement the functions described herein, or as a program for enabling computing equipment to perform processes corresponding to the steps disclosed herein. This program may be provided by storing the program in a magnetic disk, an optical disk, a semiconductor memory, or any other recording medium. The program can also be provided as a digitally encoded signal conveyed via a carrier wave. The described program can be a single program or can be implemented as multiple subprograms, each of which interact within a single computing device or interact in a distributed fashion across a network space. 
         [0017]    It should also be noted that the methods detailed herein can also be methods performed at least in part by a service agent and/or a machine manipulated by a service agent in response to a service request. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0018]    There are shown in the drawings, embodiments which are presently preferred, it being understood, however, that the invention is not limited to the precise arrangements and instrumentalities shown. 
           [0019]      FIG. 1  is a schematic diagram of a data processing system that decomposes a single batch execution into multiple units of work, each having its own syncpoint. 
           [0020]      FIG. 2  is a further schematic diagram of the data processing system in accordance with an embodiment of the inventive arrangements disclosed herein. 
           [0021]      FIG. 3  is a flow diagram of a method of operating a data processing system that decomposes a single batch execution into multiple units of work, each having its own syncpoint. 
           [0022]      FIG. 4  is a schematic diagram of embodiment of the data processing system in accordance with an embodiment of the inventive arrangements disclosed herein. 
       
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
       [0023]    The invention provides a solution permitting transparent file sharing for existing batch applications. The solution allows transaction processing systems (e.g., Customer information Control System (CICS) and batch applications) to make updates to the same records (e.g., Virtual Storage Access Method (VSAM) files) at the same time. The solution provides a mechanism to break up the existing batch applications into multiple units-of-work by issuing syncpoints on behalf of the batch application for each unit-of-work. Each unit-of-work can be processed individually which locks records for that unit only during unit processing. When processing errors occur, the system can restore itself back to the syncpoint associated with the unit, thus restoring system information to a pre-processing state. Thus, unlike traditional batch processes that lock large sets of records during a batch process, the present solution only locks small sets of records at a time. The solution is able to execute batch functions of unmodified batch applications without requiring a processing system to enter an online state. 
         [0024]    More specifically, many batch application systems read input from one or more very large sequential files. The batch application processes the data from a set of these input records and then performs an update or multiple updates to a set of records. Appropriately, at that point a syncpoint would be issued because a set of logically connected updates has been performed. The data processing system can base the positioning of syncpoints on information in the records in the sequential input files that are input to the batch application. For example, in one situation each record in a sequential input file will start a new unit of work. Alternatively, batches of records in the input file are processed as a single unit of work in the batch application. A user would have the capability of influencing which records in the input stream would signal that a new unit of work is starting. 
         [0025]    Another way in which the system may decide on the issuing of syncpoints is that there might be, in a particular application, a certain fixed number of input records that would be processed as a single unit of work. If that number were one, then the system would know that each record read from the input stream would be starting a new unit of work. Not only would each update be starting a new unit of work, it would be terminating the previous unit of work. Thus this would be an appropriate place to take a syncpoint. 
         [0026]    In a different system, each unit of work would process, for example, four input records. The data processing system would then count the input records and when the fifth input record is read the system would know that a new unit of work was starting and the old unit of work was finished and the system would issue a syncpoint. In other cases, certain data in an input record might signify the start of a new unit of work. for example, when byte  1  of a record contains ‘S’, then that might indicate the start of a new unit of work. Thus every time the batch application reads a record with an ‘S’ in the first byte, the system would note that a new unit of work was starting and the old unit of work had finished and a syncpoint would be issued. The information in the record would already exist but the system provides an interface for the user to tell the system which information to look for in a record update. By these techniques the data processing would be able to issue syncpoints at appropriate points on behalf of the batch application and there would therefore be recoverable data records without any need to change the batch application. 
         [0027]      FIG. 1  shows a data processing system  10  which includes a computer  12  which can be a mainframe computer that provides a processing function. The mainframe  12  can include a batch application  14  that can have its own address space within the mainframe  12 . The mainframe  12  can also include a transaction processing system  16 . The transaction processing system  16  can be any hardware/software/firmware configured to process database records. In one embodiment, the transaction processing system  16  can be a CICS region within the mainframe  12 . 
         [0028]    When the batch application  14  is being executed, the application  14  communicates with one or more data files  18  and applies updates to records  20  being stored by databases  22 . The execution of the batch application  14  comprises reading one or more inputs from the data files  18 , performing updates on one or more records  20  according to the inputs read from the data files  18 , and ultimately issuing a syncpoint when the updates are completed. As each record  20  is accessed by the batch application  14  during the execution of the batch, then that record  20  is locked until the completion of the batch application  14  where the syncpoint releases the locks on all of the records  20  accessed by the batch application  14 . In contrast, system  10  permits the records  20  to be updated while region  16  remains online. System  10  can operate without modifying batch application  14 . 
         [0029]    In the known conventional arrangement, while the batch application  14  is being executed, the CICS region  16  is not part of the batch process and is either taken offline while the batch application  14  is executing or will be unable to access any records  20  that have had locks placed on them by the batch application  14 . 
         [0030]      FIG. 2  shows the data processing system  10  of  FIG. 1  adapted according to an embodiment of the invention. An application program  24  is written into the same address space as the batch application  14  (without any amendment needed to the batch application  14 ). This application program  24  is configured to monitor the inputs read from the data files  18  and operates a predefined algorithm  26  based upon the monitored inputs. 
         [0031]    The predefined algorithm  26  can be simple and, for example, can be arranged to generate an output following a fixed number of inputs being read from a data file  18 . The data processing system  10  of  FIG. 2  is arranged periodically to issue a syncpoint during the execution of the batch application  14 , according to the output of the predefined algorithm  26 . This is achieved by intercepting data requests from the batch application  14  to the records  20  and routing the intercepted data requests through the transaction processing system  16  (the CICS region). This intercepting of the data requests from the batch application  14  to the records  20  can comprise intercepting internal communications within the batch application  14 . In effect, the application program  24  monitors calls within the batch application  14 , and internal communications to the output interface of the batch application  14  are intercepted by the program  24  and rerouted through the CICS region  16 . 
         [0032]    In this configuration, the step of periodically issuing a syncpoint during the execution of the batch application  14 , is carried out by the transaction processing system  16 . The system of  FIG. 2  is arranged, effectively, to break up the batch defined by the data files  18  into a series of much smaller units of work. A syncpoint is issued after each of these smaller units of work is completed. This ensures that large numbers of records  20  are not held by locks and the CICS region  16  can mediate access to records  20  required by other application programs. If CICS receives a data request from another application in respect of a record  20  for which a lock is being applied for the current unit of work that the batch application  14  is processing, then this is handled in the normal manner and CICS will hold that data request until the lock is released. 
         [0033]    The algorithm  26  is monitoring the input to the batch application  14  and is defining a break up of the inputs into separate units of work. An output is generated that triggers the issuing of a syncpoint when a unit of work completes, according to the rules of the algorithm  26 . The predefined algorithm  26  can be operated to generate an output following detection of a data flag in an input being read from a data file  18 . This data flag can be selected by a user, and can be as simple as looking for a stated character at a specific bit position in an input received from the file  18 . 
         [0034]      FIG. 3  summarizes the methodology used in the data processing system  10  of  FIG. 2 . The method of operating the data processing system  10  comprises the step S 1  of executing the batch application  14 , the step S 2  of monitoring the inputs read from the data files  18 , the step S 3  of operating the predefined algorithm  26  based upon the monitored inputs, and finally the step S 4  of periodically issuing a syncpoint during the execution of the batch application  14 , according to an output of the predefined algorithm  26 . 
         [0035]      FIG. 4  shows another embodiment of the data processing system  10 , which uses an application program  28  “CICS Anytime” to mediate between the batch application  14  and the transaction processing system  16 , a standard implementation of the CICS. The batch application  14  includes a shared Virtual Storage Access Method (VSAM) file request/response unit which interfaces with CICS Anytime. CICS Anytime is responsible for the issuing of syncpoints, and for restarting (auto and manual) the system and request mapping. CICS Anytime is connected to a restart database  30 , a checkpoint database  32 , a Multiple Virtual Storage Resource Recovery Services (MVS RRS) unit  34  and an external CICS interface (EXCI) unit  36 . CICS connects to the shared records stored as VSAM datasets in a database  38 . 
         [0036]    The primary objective of CICS Anytime is to eliminate completely CICS application outages caused by the inability of CICS to share its VSAM file data. CICS Anytime enables non-CICS programs to access VSAM file data through CICS, so that CICS appears to treat each batch application  14  as just another transactional user. In return for getting access to CICS&#39;s file data on behalf of batch applications, CICS Anytime has to ensure that the client&#39;s non-CICS applications are operating as if they were well-designed transactional applications. This requires short duration units of work, changes being hardened when committed, changes able to be backed out upon failure of units of work, the proper handling of error conditions, coordinated with CICS handling, and able to restart from a recent point in time and back out failures, following any catastrophic failure. 
         [0037]    CICS Anytime has to be applicable to existing batch applications  14  without requiring them to be modified. Many existing batch applications, by their structure, are unable to operate as well-designed transactional applications. If they were run as is, many batch applications  14  would hold CICS locks for the duration of the batch job, which could be a very long time. This would result in CICS online transactions suffering long delays due to waits for locks, or being aborted due to timeouts or, in the worst case, deadlocks. 
         [0038]    Rather than requiring compliance of non-CICS client programs, such as the batch application  14 , through redesign, CICS Anytime provides for these applications the infrastructure to achieve compliance. In practice, CICS Anytime takes control by intercepting VSAM requests that the batch applications make, and creating granularity, both in the stream of data requests in normal operation and in the sequence of events in restart and recovery in the event of job or system failure. 
         [0039]    This role is crucial when working with unchanged, legacy, batch applications  14 . These aged applications  14  are usually unsophisticated, with simple error handling and little, if any, recovery and restart capability. CICS Anytime addresses these problems by issuing syncpoints on behalf of the batch application  14 , and tracks and manages status and restart processing in the event of any failure. 
         [0040]    In summary, the functions of CICS Anytime include intercepting and redirecting file requests from z/OS batch applications, where the files are being managed by CICS, dynamically splitting up large batch jobs, presenting them to CICS as a series of small units of work, and working with MVS Resource Recovery Services and CICS Recovery Manager to coordinate commitment or rollback of file changes. CICS Anytime also automatically inserts syncpoints at appropriate places in the job stream, with user defined syncpoint frequency by, for example, time or number of updates and can dynamically split up large batch jobs into checkpoint restartable units with user defined checkpoint frequency by, for example, time or number of updates. CICS Anytime can handle communication failures such that they are transparent to the batch applications and do not require a restart, where possible, can ensure back-out of all changes to all shared files to last successful syncpoint, and can provide restart from the last successful checkpoint. 
         [0041]    The use of the system of  FIG. 4  with the CICS Anytime eliminates the batch window for virtually all batch applications without requiring any batch application code changes, with minimal job control language (JCL) or procedure changes and without having any significant negative impact on CICS transaction response time, service levels or availability. 
         [0042]    CICS Anytime operates without having a significant effect on batch application performance. In particular, elapsed time remains acceptable. Depending on the nature of the application there are likely to be user requirements to have certain batch jobs complete within a certain time frame. For example, check clearing has to be completed by a statutory time every day so banks can settle financial positions between themselves. The current quantification of acceptable is no more than a doubling of batch job elapsed time. In this embodiment, the backing up of data files before and after the batch job are no longer required. 
         [0043]    The system of  FIG. 4  has a mechanism to provide basic file data sharing between multiple MVS address spaces. This is provided by the reuse of CICS file sharing, based on MRO Function Shipping and the CICS mirror transactions. Although this is currently limited to sharing between CICS Address Spaces, the EXCI capability, which provides Distributed Program Link (DPL) between a non-CICS MVS address space and a CICS Server address Space, can be used. The CICS Anytime solution is based on enhancements to EXCI to support shipping of File Control requests. 
         [0044]    With a basic file sharing mechanism, the system is able to provide a way to intercept application file requests without requiring any program changes to the batch application  14 . This function also maps the VSAM request issued by the batch application  14  to a CICS format that can be shipped by EXCI to the CICS system that owns the files. This must all be done in a way that is transparent to the existing batch application  14 , such that it appears to the batch application  14  that the batch application  14  is still accessing a non-shared file under exclusive control. 
         [0045]    This means that, for example, new errors that could arise due to the fact that there are new components being used must be handled within CICS Anytime and not exposed to the batch application  14  in any way. Locking is fundamental to a shared environment but is totally absent from the exclusive control environment in which the batch application  14  thinks it is running. Therefore locking, timeouts, and deadly embrace situations all need to be catered for transparency with respect to the batch application  14 . 
         [0046]    With the capabilities provided above, the system provides a solution which provides transparent file sharing for existing batch jobs. However, with no other changes, the existing batch job would run as a single long running unit-of-work (UOW). This means that the batch job would potentially hold thousands of locks in CICS and the online CICS regions would grind to a halt and potentially fail. 
         [0047]    Therefore, there is provided a mechanism to break the existing batch job up into multiple UOWs by issuing syncpoints on behalf of the batch application  14 . CICS Anytime has to issue the syncpoints at appropriate places and the syncpoints are handled by MVS Resource Recovery Services unit  34  which acts as the syncpoint coordinator. 
         [0048]    The system also needs to provide checkpoint restart and positional recovery. The capabilities described above (of issuing periodic syncpoints) result in a break up of the batch job and stop it having a significant impact on online CICS transaction response times. However, should the batch job fail, there needs to be a mechanism which will allow the system to be restarted either automatically or manually from the point of failure. What is possible is the ability to create restart checkpoints at periodic intervals, not too often (less frequently than the syncpoints mentioned above) as they are expensive to create (Elapsed Time, CPU and I/O) and then to restart from the most recent checkpoint. 
         [0049]    The system is running transactionally and therefore CICS will back-out of any uncommitted units of work to the most recent syncpoint in cases of failure. What is provided is the ability for the batch job which is restarted at the most recent checkpoint to “catch-up” with the state of the data as backed out to the most recent syncpoint. This is achieved by use of the Restart Dataset which is created by CICS Anytime and contains a record of all the VSAM requests and responses since the most recent checkpoint. 
         [0050]    The present invention may be realized in hardware, software or a combination of hardware and software. The present invention may be realized in a centralized fashion in one computer system or in a distributed fashion where different elements are spread across several interconnected computer systems. Any kind of computer system or other apparatus adapted for a carrying out methods described herein is suited. A typical combination of hardware and software may be a general purpose computer system with a computer program that, when being loaded and executed, controls the computer system such that it carries out the methods described herein. 
         [0051]    The present invention also may be embedded in a computer program product, which comprises all the features enabling the implementation of the methods described herein, and which when loaded in a computer system is able to carry out these methods. Computer program in the present context means any expression, in any language, code or notation, of a set of instructions intended to cause a system having an information processing capability to perform a particular function either directly or after either or both of the following: a) conversion to another language, code or notation; b) reproduction in a different material form. 
         [0052]    This invention may be embodied in other forms without departing from the spirit or essential attributes thereof. Accordingly, reference should be made to the following claims, rather than foregoing the specification, as indicating the scope of the invention.