Data summarization recovery

Embodiments of the invention provide systems and methods for recovering a failed data summarization. According to one embodiment, recovering a failed instance can comprise processing existing summarization instances identified as instances for which a new data summarization instance needs to wait. Upon a completion or a timeout of each of the instances identified as instances for which the new data summarization instance needs to wait, an exclusive lock can be acquired on a table storing scope information for the plurality of data summarization instances. One or more existing data summarization instances that match the new data summarization instance or that have an overlapping scope with the new data summarization instance can be processed, remaining tasks to be performed by the new data summarization instance can be defined, the exclusive lock can be released, and the remaining tasks to be performed by the new data summarization instance can be performed.

BACKGROUND OF THE INVENTION

Embodiments of the present invention relate generally to methods and systems for providing summarizations of data and transactions related to one or more applications and more particularly to providing summarizations of data and transactions including recovery of failed summarization instances.

A common business need is to summarize data that exists in a system. This can be accomplished using a summarization program. A summarization program can involve any number of functional data transformations that bring data from an existing state in a system to a state that is ready or closer to being ready to be viewed by an end user, (e.g., in a report). A report may be any entity that displays summarized data to an end user, for example in a performance reporting dashboard of a user interface. In bringing summary data into such a dashboard, any number of users can be running the summarization program at any given time. Besides the architecture for efficiently dealing with and processing all those requests, there are additional problems to be solved when things do not go as expected. Dealing with source system data corruption issues, environment issues up to and including plug-pulls, and even some code bugs, can lead to very complex scenarios from which the summarization program must seamlessly recover.

For example, the problem remains of how to sort out the existing state when any number of failed summarization instances can exist, and any number of users can concurrently be attempting to clean up those failed summarization instances, as well as any number of users concurrently running summarization normally (not attempting to recover from previous failure). The problem is further complicated when considerations such as security and system resource limitations are introduced (e.g. there may be practical or imposed limits on the total number of failed or concurrently running summarization instances that can exist at any one time). The parameters passed to an instance of summarization can determine some number of elements that make up the scope of the summarization instance. Depending on how users choose to pass parameters to summarization instances, and whether imposed data security allows access to some or all of the elements defined by the parameters, then summarization instances can have scope that overlaps with the scope of other summarization instances.

In previous solutions to these problems, if a failed or running instance of summarization exists on a given scope, then any new attempted instance that overlaps with the scope of the existing instance may not be allowed to run and may be forced to raise an error. However it can be a very frustrating user experience to try to predict when it is safe to run summarization. Also in previous solutions, when failures occur, summarization instances can be left to be cleaned up by some external method or user influence. However again, the user can become frustrated when special actions are needed to clean up failed summarization instances. If the scope of summarization that a user has submitted overlaps with the scope of a running summarization instance, then the newly submitted instance can wait forever or can timeout and fail. However, there are several related scenarios in which useful data can still be summarized for the user but previous solutions do not address these scenarios. Hence, there is a need for improved methods and systems for providing summarizations of data and transactions related to an application such as a project management application.

BRIEF SUMMARY OF THE INVENTION

Embodiments of the invention provide systems and methods for providing summarizations of data and transactions related to one or more applications in a transaction processing system that can include recovery of summarization instances so that a more complete summarization can be provided. According to one embodiment, a method for recovering a failed data summarization instance of a plurality of data summarization instances can comprise processing one or more existing data summarization instances of the plurality of data summarization instances identified as an instance for which a new data summarization instance needs to wait. Upon a completion or a timeout of each of the one or more existing data summarization instances identified as instances for which the new data summarization instance needs to wait, an exclusive lock can be acquired on a table storing scope information for the plurality of data summarization instances. One or more existing data summarization instances of the plurality of data summarization instances that match the new data summarization instance or that have an overlapping scope with the new data summarization instance can be processed, remaining tasks to be performed by the new data summarization instance can be defined, the exclusive lock can be released, and the remaining tasks to be performed by the new data summarization instance can be performed.

Processing existing data summarization instances identified as data summarization instances for which the new data summarization instance needs to wait can comprise setting a timeout limit for each data summarization instance identified as data summarization instances for which the new data summarization instance needs to wait. Periodically a status of each data summarization instance identified as data summarization instances for which the new data summarization instance needs to wait can be polled and any data summarization instance identified as data summarization instances for which the new data summarization instance needs to wait that has exceeded the timeout limit can be expired. Polling a status of each data summarization instance identified as data summarization instances for which the new data summarization instance needs to wait and expiring any data summarization instance identified as data summarization instances for which the new data summarization instance needs to wait that has exceeded the timeout limit can be repeated until all data summarization instance identified as data summarization instances for which the new data summarization instance needs to wait have completed or expired.

Processing existing data summarization instances that match the new data summarization instance can comprise determining whether an existing data summarization instance that matches the new data summarization instance is running. In response to determining that the matching existing data summarization instance is not running, the new data summarization instance can be rolled back, the exclusive lock can be released, and the new data summarization instance can pick up the matching existing data summarization instance. In response to determining that the matching existing data summarization instance is running, a further determination can be made as to whether the matching existing data summarization instance is a bulk data summarization instance. In response to determining the running matching existing data summarization instance is not a bulk data summarization instance, the new data summarization instance can be rolled back, the exclusive lock can be released, the running matching existing data summarization instance can be allowed to finish or timeout, the timeout causing an error message to be issued and further processing of the new data summarization instance to be ended, and the new data summarization instance can be restarted. In response to determining the running matching existing data summarization instance is a bulk data summarization instance, an error message can be issued and further processing of the new data summarization instance can be ended.

Processing existing data summarization instances that have an overlapping scope with the new data summarization instance can comprise determining whether the new data summarization instance that has an overlapping scope with existing data summarization instances is a bulk summarization instance. In response to determining the new data summarization instance that has an overlapping scope with existing data summarization instances is a bulk summarization instance, an error message can be issued and further processing of the new data summarization instance can be ended. In response to determining that the new data summarization instance that has an overlapping scope with existing summarization instances is not a bulk summarization instance, a list of existing data summarization instances to resubmit and a list of existing data summarization instances to wait for can be created. Any existing data summarization instances to resubmit or any existing data summarization instances to wait for that are newly added to the lists, any incremental or re-extract existing data summarization instances, and any online existing data summarization instances can be processed. Processing any existing data summarization instances to resubmit or any existing data summarization instances to wait for that are newly added to the lists can comprise deleting overlapping scope from the new data summarization instance. Processing any incremental or re-extract existing data summarization instances can comprise rolling back the new data summarization instance, releasing the exclusive lock, and resubmitting any non-bulk existing data summarization instances in the list. Processing any online existing data summarization instances can comprise determining whether the new data summarization instance is an online instance and in response to determining that the new data summarization instance is not an online instance, rolling back the new data summarization instance, releasing the exclusive lock, and restarting data summarization. Processing the new data summarization instance can further comprise in response to determining the new data summarization instance is an online instance, determining whether the new data summarization instance overlaps with non-online instances. In response to determining that the new data summarization instance does not overlap with non-online instances rolling back the new data summarization instance, releasing the exclusive lock, and resubmitting the new data summarization instance, and in response to determining that the new data summarization instance overlaps with non-online instances, issuing an error message and ending further processing of the new data summarization instance.

DETAILED DESCRIPTION OF THE INVENTION

Embodiments of the invention provide systems and methods for providing summarizations of data and transactions related to one or more applications in a transaction processing system that can include recovery of summarization instances so that a more complete summarization can be provided. More specifically, embodiments provide a summarization program adapted to take actions to get whatever data it can into the dashboard or report, with minimal input from users, while at the same time recording what did not get into the dashboard and why it did not. So for example, a newly submitted summarization instance can attempt to resubmit new and overlapping summarization instances, can wait for running and/or previously submitted overlapping instances of summarization, and can skip the scope of summarization instances that fail too many times, and notify the user of actions taken. Various additional details of embodiments of the present invention will be described below with reference to the figures.

FIG. 1is a block diagram illustrating components of an exemplary operating environment in which various embodiments of the present invention may be implemented. The system100can include one or more user computers105,110, which may be used to operate a client, whether a dedicate application, web browser, etc. The user computers105,110can be general purpose personal computers (including, merely by way of example, personal computers and/or laptop computers running various versions of Microsoft Corp.'s Windows and/or Apple Corp.'s Macintosh operating systems) and/or workstation computers running any of a variety of commercially-available UNIX or UNIX-like operating systems (including without limitation, the variety of GNU/Linux operating systems). These user computers105,110may also have any of a variety of applications, including one or more development systems, database client and/or server applications, and web browser applications. Alternatively, the user computers105,110may be any other electronic device, such as a thin-client computer, Internet-enabled mobile telephone, and/or personal digital assistant, capable of communicating via a network (e.g., the network115described below) and/or displaying and navigating web pages or other types of electronic documents. Although the exemplary system100is shown with two user computers, any number of user computers may be supported.

The system may also include one or more server computers120,125,130which can be general purpose computers and/or specialized server computers (including, merely by way of example, PC servers, UNIX servers, mid-range servers, mainframe computers rack-mounted servers, etc.). One or more of the servers (e.g.,130) may be dedicated to running applications, such as a business application, a web server, application server, etc. Such servers may be used to process requests from user computers105,110. The applications can also include any number of applications for controlling access to resources of the servers120,125,130.

In some embodiments, an application server may create web pages dynamically for displaying on an end-user (client) system. The web pages created by the web application server may be forwarded to a user computer105via a web server. Similarly, the web server can receive web page requests and/or input data from a user computer and can forward the web page requests and/or input data to an application and/or a database server. Those skilled in the art will recognize that the functions described with respect to various types of servers may be performed by a single server and/or a plurality of specialized servers, depending on implementation-specific needs and parameters.

The system100may also include one or more databases135. The database(s)135may reside in a variety of locations. By way of example, a database135may reside on a storage medium local to (and/or resident in) one or more of the computers105,110,115,125,130. Alternatively, it may be remote from any or all of the computers105,110,115,125,130, and/or in communication (e.g., via the network120) with one or more of these. In a particular set of embodiments, the database135may reside in a storage-area network (“SAN”) familiar to those skilled in the art. Similarly, any necessary files for performing the functions attributed to the computers105,110,115,125,130may be stored locally on the respective computer and/or remotely, as appropriate. In one set of embodiments, the database135may be a relational database, such as Oracle 10 g, that is adapted to store, update, and retrieve data in response to SQL-formatted commands.

FIG. 2illustrates an exemplary computer system200, in which various embodiments of the present invention may be implemented. The system200may be used to implement any of the computer systems described above. The computer system200is shown comprising hardware elements that may be electrically coupled via a bus255. The hardware elements may include one or more central processing units (CPUs)205, one or more input devices210(e.g., a mouse, a keyboard, etc.), and one or more output devices215(e.g., a display device, a printer, etc.). The computer system200may also include one or more storage device220. By way of example, storage device(s)220may be disk drives, optical storage devices, solid-state storage device such as a random access memory (“RAM”) and/or a read-only memory (“ROM”), which can be programmable, flash-updateable and/or the like.

The computer system200may additionally include a computer-readable storage media reader225a, a communications system230(e.g., a modem, a network card (wireless or wired), an infra-red communication device, etc.), and working memory240, which may include RAM and ROM devices as described above. In some embodiments, the computer system200may also include a processing acceleration unit235, which can include a DSP, a special-purpose processor and/or the like.

The computer-readable storage media reader225acan further be connected to a computer-readable storage medium225b, together (and, optionally, in combination with storage device(s)220) comprehensively representing remote, local, fixed, and/or removable storage devices plus storage media for temporarily and/or more permanently containing computer-readable information. The communications system230may permit data to be exchanged with the network220and/or any other computer described above with respect to the system200.

The computer system200may also comprise software elements, shown as being currently located within a working memory240, including an operating system245and/or other code250, such as an application program (which may be a client application, web browser, mid-tier application, RDBMS, etc.). It should be appreciated that alternate embodiments of a computer system200may have numerous variations from that described above. For example, customized hardware might also be used and/or particular elements might be implemented in hardware, software (including portable software, such as applets), or both. Further, connection to other computing devices such as network input/output devices may be employed. Software of computer system200may include code250for implementing embodiments of the present invention as described herein.

FIG. 3is a block diagram illustrating, at a high-level, functional components of a system for providing data summarization recovery according to one embodiment of the present invention. In this example, the system300can include one or more applications305for processing data and/or transactions including but not limited to any of a variety of different enterprise applications. These applications305can store and maintain the data and transactions in one or more repositories310such as a relational database (RDBMS) and/or a multidimensional database (MDBMS). A summarization module315or program can access the repositories312and314and generate a body of summarization instances320that include data transformations that bring the data stored in the repositories310to a state to be viewed by an end user, (e.g., in a report330presented by a reporting module). The report330may be any form that displays summarized data to an end user, for example through a reporting dashboard of a user interface for the application305.

This example illustrates additional details of the body of summarization instances320introduced above that maybe generated by the summarization module315and used by the reporting module325to generate one or more reports330. As illustrated here, the body of summarization instances320can include any number of instances335, each having a corresponding instance identifier. Other information for each instance and related thereto based on the instance identifier can include a defined summarization scope340

Generally speaking, an instance of summarization can contain any number of database transactions to be executed in serial and parallel, each satisfying some portion of a set of functional data transformations. Consider a particular summarization instance which can have InstanceID “i.” This instance of summarization can be one instance among all existing summarization instances320. At any time, multiple users can attempt to instantiate summarization so, for example, to bring their data into the reporting dashboard. These attempts can add to the body of summarization instances320.

For the instance of summarization i, in becoming an instance of summarization it has a particular scope340defined, which can be a list of elements unique among all instances of summarization and is used to identify the data in the source system on which to execute the functional data transformations. The instance of summarization i then proceeds to summarize data for its elements of “Scope”, for example “A”, “B” and “C” in340. If instance of summarization i is completed, then the state for summarization instance i can be removed in which case it would no longer be an instance of summarization in the body of summarization instances320.

According to embodiments described herein, the summarization module315and/or the reporting module325can be adapted to recover summarization instances by taking actions to get whatever data it can into the dashboard or report with minimal input from users, while at the same time recording what did not get into the dashboard and why it did not. So for example, a newly submitted summarization instance can attempt to resubmit existing overlapping summarization instances, can wait for running and/or previously submitted overlapping instances of summarization, and can skip the scope of summarization instances that fail too many times or that run too long, and can notify the user of actions taken. In describing the recovery of summarizations, various terms will be used herein for which the following descriptions are provided.

Existing instance (i.e., Bulk, Incremental, Reextract, Online, etc.): Can be considered to be a summarization instance that exists in the body of summarization instances320at the time a newly submitted summarization instance acquires an exclusive lock on the scope of the instances. It can either be currently running, or failed. Note that the last database transaction of a summarization instance can clean its state. Therefore, a completed summarization instance cannot be an existing or failed summarization instance since by design it will no longer have any database state. Also note, a newly submitted summarization instance does not become an existing summarization instance until it commits its state to the database310in its first database transaction.

New instance (i.e., Bulk, Incremental, Reextract, Online, etc.): Can be considered to be any newly submitted summarization instance320that has acquired the exclusive lock but has not yet committed its state and released the exclusive lock.

Params: Can be the parameters passed to a newly submitted summarization instance (e.g. Summarization Method (e.g. Bulk, Incremental, Reextract, Online), From Project, To Project, Business Unit, etc.).

Identical/Different Params: Whether or not two summarization instances have exactly the same parameters.

Scope: Can be considered to be the result of the translation of the parameters of a summarization instance into a list of elements that can be compared with the scope of another summarization instance to determine if the two summarization instances are attempting to work on some or all of the same data.

Overlapping/Non-Overlapping: Can be determined for two instances based on whether or not at least one element in the scope of one summarization instance is the same as at least one element in the scope of another summarization instance. If such an element exists, then the two summarization instances can be considered to be overlapping, else they can be considered to be non-overlapping.

Running/Failed: Can be determined for an instance based on whether or not an existing summarization instance has a thread currently executing its code. If the thread exists, then the existing summarization instance can be considered to be running, else it can be considered to be failed (recall that a completed summarization instance cannot be an existing summarization instance).

Run: In this situation, the newly submitted summarization instance can finalize its scope, commit to release the exclusive lock (hence becoming an existing summarization instance) and continue its work.

Pick up: In this situation, the newly submitted summarization instance can assume the scope of the failed summarization instance, commit to release the exclusive lock (hence becoming an existing summarization instance), and pick up where the failed instance had left off, continuing with the next incomplete database transaction of that failed summarization instance.

Fail: In this situation, the newly submitted summarization instance encountered one of the situations in which it is not desirable to continue running. The newly submitted summarization instance can report the reason for failure and can report the existing summarization instance(s) because of which it will not continue. The new instance can rollback, releasing the exclusive lock, and cease execution (hence never becoming an existing summarization instance). Note that since the first transaction of the newly submitted summarization instance is rolled back, the newly submitted summarization instance will not have stored any state in the database, but if determining the scope of summarization is a sufficiently expensive operation, the scope can be cached for that particular set of parameters for future use as needed.

Skip: In this situation, the newly submitted summarization instance can remove from its scope the elements that overlap with an existing summarization instance it has decided to skip. Any skipped scope can be reported to the user.

Resubmit: In this situation, the newly submitted summarization instance can spawn another newly submitted summarization instance with the same parameters to an existing summarization instance. Note that this can cause the spawned newly submitted summarization instance to “Pick up” an existing summarization instance.

Wait: In this situation, the newly submitted summarization instance can wait for an existing summarization instance to complete. If the existing summarization instance completes, the newly submitted summarization instance can continue. If the existing summarization instance fails or a timeout is reached then the scope of the existing summarization instance can be skipped so that the newly submitted summarization instance can continue on the remaining scope. Note that the newly submitted summarization instance can release the exclusive lock while it is waiting and then revalidate its scope after it re-obtains the exclusive lock after a suitable waiting period. When the scope cannot be skipped, for example as in a special ONLINE situation detailed herein, the newly submitted summarization instance may fail if the existing summarization instance fails or does not complete before a timeout.

According to one embodiment, particular rules defining how summarization should behave in various recovery scenarios are designed to improve user experience. Based on the definitions above, TABLE 1 illustrates different combinations of events that can be defined by such rules.

For example, based on column 4, row 3 in Table 1: for a newly submitted instance of bulk summarization, if it encounters an existing running instance of Bulk summarization with identical parameters (hence overlapping scope), then the newly submitted instance of bulk summarization fails with an appropriate message. Whenever a newly submitted summarization instance is affected by an existing summarization instance, then it can output whatever existing summarization instances affected it and how it handled those existing summarization instances. It should be understood that Table 1 illustrates one particular implementation and that the exact behaviors described in Table 1 can be adapted to a particular implementation's requirements. Additionally, it should be understood that the processes for recovering a summarization instance need not be implemented or defined in a set of rules. Rather, the processes for recovering a summarization instance can be implemented as a set of algorithms such as will be described below.

Regardless of the exact implementation, embodiments of the present invention can be accessible to users in various ways, such as with a project manager who may not have concern for existing summarization instances and just wants to see data in the dashboard. Implementing summarization according to embodiments of the present invention can help ensure that any data that can be shown will be shown and that the current state of relevant errors will be shown to the project manager as needed. Also, a system administrator can query the state of existing summarization instances and determine what course of action, including what types of summarization instances to re-submit, to easily clean up different subsets of issues.

A project manager will typically run summarization without regard for existing summarization instances. For this reason, the normal use-cases for the project manager (the incremental and online methods of summarization), can reduce the potential for designed failure. However, it will still be useful for the project manager to have a list of relevant existing summarization instances for analysis. The security permissions of a user, including the project manager, can also restrict the scope of summarization on which the project manager is allowed to run summarization. The concept of an element of scope is defined at the design time of a summarization program. Therefore, it should be considered whether a failed summarization instance on some element of scope may hinder a user from running summarization on a different set of parameters such that the user may not understand the conflict. The scope can be designed to minimize such conflicts.

A system administrator has more concern for the underlying mechanics of the recovery processes. The system administrator can take advantage of the recovery processes and can, for example, submit a request with a range of parameters to attempt to clean up many failed summarization instances or submit a request with a specific set of parameters to pinpoint the cleanup. There can be a way to specify a scope and run the recovery methods without picking up an existing summarization instance since the recovery methods occur before a newly submitted summarization instance becomes an existing summarization instance. Furthermore, tools can be created to help the system administrator. One tool might present a list of summarization instances that have failed as a result of the system administrator's actions such as while running BULK summarization. Another tool might suggest summarization parameters to efficiently cover the scope of existing failed instances and identify chains of overlapping instances and dependencies and suggest a recovery plan (recall that the state of each existing summarization instance is stored in the database). Another tool might allow particular summarization runs to be selected and “rolled back”; they could also be selected by defining scope in the same way scope is defined for a summarization instance. Depending on whether an implementation has a wrapper around the summarization program, it may also be advantageous to catch errors and return a warning state to the wrapper. Additionally or alternatively, a tool to purge all data and start from scratch can be built into the summarization program architecture and/or a tool can also be devised to purge data for a particular summarization scope. Another tool might allow the system administrator to view the scope of a failed summarization instance and see if it is possible to recover just a portion of the scope (if, for example, the error is not with all of the scope). Another tool might allow the system administrator to cancel a long queue of newly submitted summarization instances before they become existing summarization instances.

Another role, a developer, can determine scenarios in which the recovery methods can be used not only in the development process, but also during patching and bug fixes similar to the scenarios used by the system administrator.

FIG. 4is a flowchart illustrating a process for providing data summarization recovery according to one embodiment of the present invention. In this example, recovering a failed data summarization instance of a plurality of data summarization instances can begin with determining405whether a newly submitted summarization should wait for any previous summarization instances. In response to determining405that there are summarization instances to wait for, one or more existing data summarization instances of the plurality of data summarization instances identified as an instance(s) for which the newly submitted data summarization instance needs to wait410may or may not complete before the newly submitted summarization instance decides it must timeout on waiting for the existing data summarization instance(s). An exemplary process for handling these summarization instances is described in greater detail below with reference toFIG. 5.

In response to determining405that there are summarization instances to wait for or upon a completion or a timeout of each of the one or more existing data summarization instances identified as instances for which the newly submitted data summarization instance needs to wait, an exclusive lock can be acquired415on a table storing scope information for the plurality of data summarization instances. One or more existing data summarization instances of the plurality of data summarization instances that exactly match the newly submitted data summarization instance can then be processed420. An exemplary process for processing existing data summarization instances of the plurality of data summarization instances that exactly match the newly submitted data summarization instance will be described in greater detail below with reference toFIG. 6. In some cases, a determination425can be made as to whether there is sufficient room in memory for a new summarization instance. If a determination425is made that there is not sufficient room, the recovery process may stop430and an error message can be issued.

If a determination425is made that there is sufficient room for the new instance, a scope of the new summarization instance can be determined435and the new summarization instance can be written440to the database or other memory. One or more existing data summarization instances that have an overlapping scope with a new data summarization instance can then be processed445. An exemplary process for processing existing data summarization instances that have an overlapping scope with the new data summarization instance will be described in greater detail below with reference toFIG. 7. A determination450can then be made as to whether the new summarization instance is a recovery-only instance. In response to determining450the new summarization instance is a recovery-only instance, the new summarization instance can be rolled back455, the exclusive lock can be released, and the recovery-only process may stop.

In response to determining450that the new summarization instance is not a recovery-only instance, a further determination460can be made as to whether scope remains in the new summarization instance to be processed. In response to determining460no scope remains in the new summarization instance to be processed, the new summarization instance may stop465and an error message can be issued.

In response to determining460scope does remain in the new summarization instance to be processed, the parameters and scope for the newly submitted summarization instance can be recorded470into the in database. Remaining tasks to be performed by the new data summarization instance can be used to define475a course of action for the new summarization instance. These tasks or course of action can be saved480to the repository or other memory and then committed and the exclusive lock can be released485. The remaining tasks defining the course of action for the new summarization instance can then be performed490.

FIG. 5is a flowchart illustrating additional details of an exemplary process for processing existing summarization instances according to one embodiment of the present invention. In this example, processing existing data summarization instances identified as data summarization instances for which the new data summarization instance needs to wait can comprise setting505a timeout limit for each data summarization instance identified as a data summarization instance for which the new data summarization instance needs to wait. A status of each data summarization instance identified as a data summarization instance for which the new data summarization instance needs to wait can be periodically polled510. Any data summarization instance identified as a data summarization instance for which the new data summarization instance needs to wait that has not yet completed515and the has exceeded520the timeout limit can be expired525. Polling510a status of each data summarization instance identified as a data summarization instance for which the new data summarization instance needs to wait and expiring525any data summarization instance identified as a data summarization instance for which the new data summarization instance needs to wait that has not yet completed515and the has exceeded520the timeout limit can be repeated until all data summarization instances identified as data summarization instances for which the new data summarization instance needs to wait have completed or expired.

FIG. 6is a flowchart illustrating additional details of an exemplary process for processing matching summarization instances according to one embodiment of the present invention. In this example, processing existing data summarization instances that match the new data summarization instance can comprise determining605whether there are existing data summarization instances that match the new data summarization instance and determining610whether a matching instance is running. In response to determining610that the matching existing data summarization instance is not running, the new data summarization instance can be rolled back615and the exclusive lock released and the new data summarization instance can be picked up620using the matching existing data summarization instance, so that the thread of the new data summarization instance can become associated with the state of the matching existing data summarization instance, at which time the matching existing data summarization instance can change from being a failed existing data summarization instance to being a running existing data summarization instance.

In response to determining610that the matching existing data summarization instance is running, a determination625can be made as to whether the matching existing data summarization instance is a bulk data summarization instance. In response to determining625the running matching existing data summarization instance is a bulk data summarization instance, an error message can be issued and further processing of the new data summarization instance can be ended630. In response to determining625the running matching existing data summarization instance is not a bulk data summarization instance, processing can continue with rolling back635the new data summarization instance and releasing the exclusive lock, waiting640for the running matching existing data summarization instance to finish or timeout, the timeout causing an error message to be issued and further processing of the new data summarization instance to be ended, and restarting645the new data summarization instance.

FIG. 7is a flowchart illustrating additional details of an exemplary process for processing overlapping summarization instances according to one embodiment of the present invention. In this example, processing existing data summarization instances that have an overlapping scope with the new data summarization instance can comprise determining705whether the new data summarization instance that has an overlapping scope with existing data summarization instances is a bulk summarization instance. In response to determining705that the new data summarization instance that has an overlapping scope with existing data summarization instances is a bulk summarization instance, an error message can be issued and further processing of the new data summarization instance can be ended710.

In response to determining705that the new data summarization instance that has an overlapping scope with existing data summarization instances is not a bulk summarization instance, processing can continue with creating715a list of existing data summarization instances to resubmit, creating720a list of existing data summarization instances to wait for, processing any existing data summarization instances to resubmit or any existing data summarization instances to wait for that are newly added to the lists, processing any incremental or re-extract existing data summarization instances, and processing any online existing data summarization instances.

More specifically, processing any existing data summarization instances to resubmit or any existing data summarization instances to wait for that are newly added to the lists can comprise determining725whether there are any existing data summarization instances to resubmit or any existing data summarization instances to wait for that are newly added to the lists. In response to determining725that there are existing data summarization instances to resubmit or any existing data summarization instances to wait for that are newly added to the lists, and that the lists have previously been created for the new data summarization instance, then the overlapping scope can be deleted780from the new data summarization instance.

In response to determining725that there are existing data summarization instances to resubmit or any existing data summarization instances to wait for that are newly added to the lists, and that the lists have not previously been created for the new data summarization instance, then a determination730can be made as to whether the new data summarization instance is an incremental or re-extract instance. In response to determining730that the new data summarization instance is an incremental or re-extract instance, the new data summarization instance can be rolled back735and the exclusive lock can be released, and any non-bulk existing failed data summarization instances in the lists can be resubmitted740and data summarization for the new data summarization instance can be restarted775.

In response to determining730that the new data summarization instance is not an incremental or re-extract instance, a further determination745can be made as to whether the new data summarization instance is not an online instance. In response to determining745that the new data summarization instance is not an online instance, the new data summarization instance can be rolled back770and the exclusive lock released and data summarization for the new data summarization instance can be restarted775.

In response to determining745that the new data summarization instance is an online instance, a further determination750can be made as to whether the new online data summarization instance does not overlap with a non-online data summarization instance. In response to determining750that the new online data summarization instance does not overlap with a non-online instance, processing can continue with rolling back760the new online data summarization instance, and the exclusive lock can be released, to resubmit any existing failed online data summarization instances765and data summarization for the new data summarization instance can be restarted775. In response to determining750that the new online data summarization instance overlaps with a non-online data summarization instance, an error message can be issued and further processing of the new data summarization instance can be ended755.

Stated another way, one possible algorithm for implementing summarization recovery such as outlined above with reference toFIGS. 4-7can be implemented as follows:

Label: <<restart_summarization>>If 405 there are existing summarization instances for which the newly submittedsummarization instance needs to wait (as determined by a previous pass through thealgorithm)410 Set 505 a suitable timeout duration and poll 510 the status of theoverlapping instances until they are all complete 515 or until the timeout 520is reachedEnd ifAcquire 415 exclusive lock on the database table that is used to store summarizationscope (so that only one summarization instance can continue)Loop 420 through all existing summarization instances to see if the parameters of thenewly submitted summarization instance match those of any existing summarization instanceIf 605 the parameters match exactly to an existing summarization instanceIf 610 that summarization instance is runningIf 625 that summarization instance is not a Bulk instanceRollback 635 (release the exclusive lock)Wait 640 for that summarization instance to complete(potentially timing out and raising an error)Goto 645 <<restart_summarization>>Else if 625 that summarization instance is a Bulk instanceFail 630 (we do not want to wait for a long running Bulkprocess)End ifElse if 610 that summarization instance is not runningRollback 615 (release the exclusive lock)Pick up 620 work where that summarization instance has left off(this includes recording summarization state such that if anotherprocess polls the status of this existing summarization instance,it will be checking the process of the newly submittedsummarization instance which has picked up the existingsummarization instance)Execution passes out of this algorithm and onto the course ofaction of the existing summarization instanceEnd ifEnd ifEnd loopIf 425 there are practical or imposed limits on the number of summarization instancesthat can exist at any one time and there is no room for the newly submitted summarizationinstanceFail 430 (all failures and actions should have suitable user notifications)End ifUsing the parameters from the newly submitted summarization instance, determine 435 thescope of summarization (i.e. the list of elements that can be compared with othersummarization instances) and write 440 it to the databaseIf 442 the scope of the newly submitted summarization instance overlaps with any existingsummarization instances process 445 overlapping instances (note that their scope has beencommitted to the database and cannot be changed while the exclusive lock exists)If 705 the newly submitted summarization instance is a Bulk instanceFail 710 (We do not want to break up the scope of a Bulk instance since itdefeats the purpose of doing a large volume run all at once. We do notwant a Bulk instance to wait for other summarization instances, since wewant to know immediately if a Bulk instance is waiting rather thansummarizing data.)End ifLoop through the overlapping instances to determine whether we need to resubmitthem and/or wait for them to completeCreate 715 a list of instances that we need to resubmit (are failed)Create 720 a list of instances for which we need to wait (are running)(Note: these lists are used upon returning to <<restart_summarization>>)End loopIf 725 the newly submitted summarization instance has not previously attempted toresubmit and/or wait for existing instancesIf 730 the newly submitted summarization instance is an Incremental orReextract instanceRollback 735 (release exclusive lock)Resubmit 740 all non-Bulk failed summarization instances (We donot want to resubmit a Bulk instance because we want Bulksubmissions to be direct user actions.)Else if 745 the newly submitted summarization instance is an OnlineinstanceIf 750 some of the overlapping instances are not Online instancesFail 755 (User expectation can be that Online instanceswill always complete quickly, so we cannot wait for longerrunning Incremental, Reextract and Bulk instances tocomplete. Furthermore, Online scope can be defined by afunctional batch of data rather than parameters defining astrict list of scope elements, so it may not makefunctional sense to skip any scope.)End ifRollback 760 (release exclusive lock)Resubmit 765 all Online failed summarization instancesEnd ifRollback 770 (release exclusive lock)Goto 775 <<restart_summarization>>Else if 725 the newly submitted summarization instance has already attempted toresubmit and/or wait for existing instancesDelete 780 all overlapping scope from the scope of the newly submittedsummarization instance (and notify the user)End if(Note that if control gets this far, then the scope for the newly submitted summarizationinstance is finalized.)If 450 the newly submitted summarization instance is a recovery only instanceRollback 455 (release exclusive lock)End execution (since the recovery methods have been executed)End ifIf 460 no scope remains for the newly submitted summarization instanceFail 465 (Raise suitable message to inform the user that there is no scoperemaining on which to summarize data. Note that the notification would also havebeen raised about what scope has been skipped.)End ifWrite 470 all summarization state to the databaseDetermine 475 the course of action for the summarization instance (that is, the stepsthat the summarization instance needs to execute) and write 480 it to the databaseCommit 485 (release exclusive lock and the newly submitted summarization instance becomesan existing summarization instance)Continue 490 onto the course of action for the summarization instance