Communication graph tracking of multi-system operations in heterogeneous database systems

Techniques a provided for communication graph tracking in multi system operation in heterogeneous database systems, including retrieving two or more database commit logs from two or more corresponding heterogeneous databases. For each of the commit logs, a subset of information is extracted, including a correlation identifier, and event stream entries are produced based on that information and are stored. The event stream entries are then retrieved and event database entries are created based on the event stream entries. A request for a communication graph related to a particular correlation identifier is received and the event database is searched for matching entries. Those entries are used to generate a displayable communication graph for the particular correlation identifier, which is then provided in response to the request.

FIELD OF THE INVENTION

The present invention relates to tracking operations involving multiple computer systems and, more specifically, communication graph tracking of multi-system operations in heterogeneous database systems.

BACKGROUND

Many real-world computer-implemented operations involve making changes in multiple computer systems. Such operations are referred to herein as “multi-system operations”. Many of these multi-system operations will cause the execution of multiple database events across multiple databases, and those databases can be heterogeneous. For example, a single transaction may be associated with changes to several databases, and those databases may span multiple brands of databases. An issue with such system is that it is difficult to track what database changes are associated with what multi-system operation, and further, it is yet more difficult to consolidate those database changes and display them in a coherent form. Displaying the transactions together can be critical for debugging, accounting, auditing, etc.

As another example, the online purchase of an airline ticket may involve adjusting seat availability in an airline's database system, as well as adjusting account balance in one or more bank database systems. Since the changes made by multi-system operations are logically all part of the same high-level operation, it is preferable that either all of the changes be made permanent, or that none of the changes are made permanent. When only a subset of the changes required by a multi-system operation are made permanent, the systems are left in an inconsistent state (e.g. the customer is charged for an airline ticket, but no seat is reserved for the customer).

When the database systems that are involved in a multi-system operation are tightly coupled, inconsistent states can be avoided using a technique referred to as two-phase commit. In two-phase commit, all systems wait until all other systems are “prepared” to make their changes permanent (“commit”). At that point, all systems commit in unison. If any system cannot achieve the “prepared” state, then all systems “roll back” to remove all changes made relative to the multi-system operation.

While the two-phase commit protocol guarantees that a multi-system operation is treated as an atomic unit (either entirely commits, or entirely fails), it does so at the expense of performance. For example, when the systems involved in the multi-system operation are not tightly coupled, the inter-system interactions required by the two-phase commit protocol may impose unacceptable delays. Further, one or more of the systems involved in the multi-system operation may not support the two-phase commit protocol.

To avoid having resources locked while waiting for other systems involved in a two-phase commit operation, techniques have been developed for avoiding two-phase commit in multi-system operations. One such technique is described in U.S. Pat. No. 7,904,434 entitled “Framework for handling business transactions”. In the technique described therein, operational constraints set forth conditions that must be satisfied before an update is allowed to proceed. If an attempt is made to update a particular value that has changes that may be undone, then the database server determines a plurality of “possible result values” for the particular value. If the possible result values satisfy the operational constraint conditions, then the update is allowed to proceed.

Unfortunately, formulating such operation constraint conditions may not be straightforward or even possible for some types of multi-system operations. Further, it is possible that some systems involved in the multi-system operations do not support such operational constraints. Consequently, there is a need to perform multi-system operations in a way that does not impose the delays of two-phase commit, and that does not require the programming of operational constraint conditions.

Techniques herein address these issues.

DETAILED DESCRIPTION

In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. It will be apparent, however, that the present invention may be practiced without these specific details.

In other instances, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring the present invention.

General Overview

Correlating all of the transactions associated with a particular multi-system operation can be time and resource consuming, if not impossible, especially when the multi-system operation causes operations or transactions on multiple heterogeneous databases. If one were to query each of those databases after conclusion of the multi-system operation in order to discover what operations or transactions were associated with that operation, then that would burden and slow down the various systems and databases in the multiple systems.

Generally, the techniques herein include a streaming module receiving database commit logs for multiple, heterogeneous databases. The commit logs represent the database commits of the various heterogeneous databases that were performed on behalf of service applications. Each operation will have associated with it a correlation identifier. The correlation identifier will be the same for all actions that are part of the same multi-system operation, even if each database transaction has a different transaction identifier.

In some embodiments, the streaming module will extract information from the commit logs including at least a correlation identifier. The extracted information may also include information about the type of database commit that was performed. The streaming module produces event stream entries based on the extracted subset of information. These event stream entries are stored in the event stream (sometimes called a “time stream”) associated with the corresponding database. Each of the heterogeneous database may have its own corresponding event stream. The streaming module may repeat these steps and continue to build up the event stream.

In some embodiments, the reconciliation system will take the information from the event stream and create corresponding event database entry for each entry in the event stream. The event stream database is a different database from the heterogeneous databases. The event database entries will include the correlation identifier from the corresponding event stream entry. The reconciliation system may run continuously, adding event database entries for each new event stream entry, or may run periodically, adding new event database entries for any new event stream entries.

In some embodiments, a communication graph control receives a request (e.g., from a user device) for a communication graph related to a particular correlation identifier. The communication graph control will then search the event database for the particular correlation identifier. It will then generate a communication graph based on the events that come back from the search on the event database for the particular correlation identifier. This displayable communication graph is then sent back in response to the request, and, in some embodiments, displayed on the screen of the requester.

Operational Overview

FIG. 9is a flowchart depicting a process900for communication graph tracking of multi-system operations in heterogeneous database systems. Generally, process900proceeds by receiving910two or more database commit logs. The database commit logs can be from two or more heterogeneous databases. A streaming module (e.g., streaming module1010ofFIG. 10) will produce and store920event streams based on at least a subset of the two or more commit logs. The entries in the event stream may include at least a correlation identifier in addition to other metadata. As indicated by the dotted line between920and910, after producing and storing920the event stream entry, the streaming module may continue to receive910new and/or updated database commit logs. A reconciliation system (e.g., reconciliation system1030ofFIG. 10) may retrieve930event streams that have been produced by the streaming module. Retrieving930the event streams may happen asynchronously with the creation of the event stream. For example, the streaming module may continue to receive910database commit logs, and produce and store920event stream entries based on those commit logs at any given rate or based on any particular schedule (e.g., every second, minute, or based on detection of modification of a commit log). Separate from that rate or schedule of producing and storing920the event stream entries, the reconciliation system may retrieve930the event streams (e.g., every second, minute, or upon detection of new event stream entries). After retrieving930the event streams, the reconciliation system will create940event stream database entries based on entries in the retrieved event streams. The database in which the event database entries are stored may be separate from the databases from which commit logs were received910. As indicated between by the dotted line between940and930, the reconciliation system may continue to retrieve more and/or updated event streams and create940event stream database entries based on those new and/or updated event streams. A communication graph control system will receive950a request for a communication graph for a particular correlation identifier. This request may be received, for example, from a user device (e.g., user device1020). The communication graph control will search the event database for entries that match the particular correlation identifier that was received. Based on the results of this search960, the communication graph control will generate970a displayable graph of entries that match the correlation identifier. This displayable graph is then provided980in response to the request. For example, the displayable graph may be sent back to the user device that requested the communication graph. In some embodiments, not depicted inFIG. 9, a user account can interact with the displayable graph once it is displayed. For example, using an interface a user may be able to click on nodes in the graph to “drill down” into that node. This may allow a user to view, for example, what a particular database row looked like before and after a particular commit.

Returning to the beginning of process900, a streaming module receives910database commit logs from two or more heterogeneous databases. A database commit log will often be representative of individual transactions within the database. Each “commit” is the application of a single transaction to the relevant database. The commit log as a record of those transactions and may be used to keep track of what has happened with the database. In some scenarios, commit logs may be used, for example, in disaster recovery.

Generally, commits are written to the log before being applied. So, transactions that are in process but not completed when an error occurs or a database server goes down can be recovered and reapplied by checking the database commit log. In some embodiments, each record in the commit log has its own unique sequential log entry number and/or other timestamp information. Database commit logs may also record other metadata such as table name, mutation type (delete, create, update), transactions identifiers and other metadata such as a correlation identifier for a multi-system operation.

Based on the database commit log, the streaming module can produce and store920the event streams based on a subset of the information in the commit logs. For example, the streaming module may produce an event stream entry that includes, for example, a database name and/or type, the name of the table being modified, the mutation type (delete, create, or update), etc. Additional information may include the values of columns before and after the mutation in question. For example, the entry may show the columns that were updated before they were updated. The event stream entry may also include the columns after the mutation. As a particular example, in some embodiments, if a modification was made to an address for a particular user, then the “before” portion of the event stream entry which show the address before the mutation and the “after” portion of the mutation of the event stream entry would show the address after the mutation. The event stream entry may also include the correlation identifier for the multi-system operation. Correlation IDs for multi-system operation are discussed extensively elsewhere here in. An event stream entry may also include a transaction identifier unique to the transaction for that database and/or timestamp information. Other information in the database commit logs may not be included in the event stream entry. For example, if a database commit log includes information about the amount of storage left in the particular database, that information may or may not be included in the event stream entry, depending on the embodiment.

The event stream entries that are produced can then be stored920in an event stream. The event stream can be a file and/or other storable data that can be accessed by a reconciliation system (e.g., reconciliation system1030ofFIG. 10). After storing920the event stream entry, process900may proceed, optionally, by receiving910more and/or updated database commit logs. For example, as new commits are happening on each of the heterogeneous databases, the commit logs for those databases may be updated. The streaming module may access those updated commit logs and extract the updates to those commit logs in order to produce and store920more events stream entries.

This may happen at a fixed interval, such as every one, two, ten, or one hundred seconds, etc. This can also happen based on detecting that a database commit log has been updated. For example, if the streaming module detects that the commit log has a modified date more recent than its most recent production and storage920of event streams, then it may access that database commit log in order to produce and store920more events stream entries based on the updates to the commit log.

A reconciliation system may retrieve930the event streams that were produced and stored920by the streaming module. Retrieving930these event streams may include accessing the stream in its stored form as a file and/or other stored data. The reconciliation system will create940event database entries based on the event stream entries in the events stream. The event database entry may include all or some of the metadata in the event stream entry. For example, in some embodiments, the event database entry may include, from the event stream entry, a database name and/or type a table name, a mutation type (delete, create, or update) an indication of the appearance of the mutated columns before mutation, and the mutated columns after mutation. Event database entry may also include the correlation identifier identifying the multi-system operation. The event database entry may also optionally include a timestamp associated with the original commit in the commit log.

In some embodiments, the event database into which the event database entries are stored940may be a database that is separate from the heterogeneous databases from much the commit logs are received910. In some embodiments, as indicated by the dotted line between creating940and retrieving930, event database entries may be created940based on retrieved930events streams on a periodic basis. For example, each time an event stream is produced and stored920, the reconciliation system may detect that based on the modification of the event stream and retrieve930the event stream in order to create940event stream event database entries. In some embodiments, the reconciliation system may operate in a periodic manner retrieving930and creating event, creating940event database entries every one, ten, one hundred second, etc.

A communication graph control may receive950a request for communication graph for particular correlation identifier. That request may have been received from a user device by the communication graph control. That user device may, for example, be request a graph of the database transactions associated with a particular multi-system operation. The displayable graph of the multi-system operation may allow the user device to display what happened at particular databases across all of the heterogeneous databases affected by the multi-system operation. For example, referring toFIG. 10, the multi-system operation for corresponding to the correlation identifier sent in the request may be associated with a transaction that caused the execution of multiple database transactions in multiple heterogeneous databases1040-1042by multiple service applications1060. The request for the displayable communication graph may have been received from a user device1020and sent to the communication graph control1050, which receives950the request. As discussed below, the communication graph control1050may query event database1042in order to find all of the event database entries related to the particular correlation identifier. The communication graph control1050can create a displayable communication graph to send back to user device1020.

Returning to process900, the communication with graph control can search960the event database for entries matching a particular correlation identifier. This correlation identifier may be the one received950from a user device. Based on the results of this query to the event database, numerous event database entries may be returned. Each of these database entries may include information related to the particular event associated with the event database entry, such as what happened to the related database entry in the commit log of the corresponding heterogenous database.

The query results from the search960are used to generate970a displayable graph of entries that match the correlation identifier. The displayable graph of entries may be formed in any appropriate format and in any appropriate way. For example, the earliest entry associated with the correlation identifier may be shown on the far left of the graph as the one that spawned the other actions taken. For example, a particular multi-system operation may have invoked a first service application1060. That service application may have caused a database commit in a database1040and may have caused a second service application1060to cause a database commit in a second database1041. Databases1040and1041may be of different types, stored in and/or executing on different servers, etc. as discussed elsewhere here in.

The graph displayed may include the first service application1060on the left and an arrow indicating the database transaction for the first database1040and then to the right a database transaction for the second page of the database1041. Additional examples of displayable communication control graphs are discussed elsewhere here in (see, e.g.,FIG. 11). The communication graph control may provide980the displayable graph in response to the request. The communication graph in any appropriate form, such as a markup language.

Not depicted inFIG. 9, the displayable graph may include selectable nodes. The selectable nodes may provide detail associated with the particular database commit(s) associated with that node. For example, selecting or “clicking on” a node may provide a markup or additional information about the associated database commit. As discussed herein, in some embodiments, this additional information may be from the database commit log that was transformed into the event stream and then stored in the event database as an entry. This information may include the database name and/or type a table name, a mutation type, the appearance of the associated columns before and after the mutation, and possibly transaction identifiers and/or timestamp information.

Providing this information in a displayable form may help system developers and administrators understand the cascading actions (e.g., among heterogeneous databases) taken for a multi-system operation. This can be very beneficial for debugging systems and/or finding places for additional efficiencies and systems.

Example System for Communication Graph Tracking of Multi-System Operations in Heterogeneous Database Systems.

FIG. 10is a block diagram depicting an example system1000for communication graph tracking of multi-system operations in heterogeneous database systems. System1000includes numerous modules and systems communicating via a network1090many of these modules and systems are depicted as running separately from each other, and in some embodiments that is the case. In some embodiments however, two or more of these systems or modules may run on the same computing devices and/or in the same co-location facility.

System1000includes a streaming module1010coupled to network1090. Streaming module1010may access heterogeneous databases1040,1041in order to retrieve database commit logs associated with those databases from their module may extract all or a subset of the information from the database commit logs in order to create an event stream for each of those databases. The created event stream may be stored locally on streaming module1010and/or in other storage, not depicted in system1000.

System1000also includes a reconciliation system1030. The reconciliation system1030may retrieve the event stream entries from the event stream created by the streaming module1010. From there, the reconciliation system1030may create event database entries to store in an event database1042. Event database1042may be accessible over network1090. In some embodiments, event database1042may be connected directly to the reconciliation system1030and/or the communication graph control1050. Numerous service applications1060are connected to the network1090. These service applications1060may perform tasks that require commits to the heterogeneous databases1040and1041.

User devices1020and1021are connected to the network1090. These user devices1020,1021may request displayable communication graphs from the communication graph control1050and, upon receipt of the communication graph, display that communication graph. Communication graph control1050is connected to network1090. The communication graph control may receive requests from user devices1020and1021and produce communication graphs for particular correlation identifiers in response to those requests.

Each of the depicted modules and systems, as noted above, may execute on one or more computing devices. Examples of computing devices and other appropriate hardware are discussed elsewhere here in.

Example Communication Graphs

FIG. 11depicts an example displayable communication graph1100. Each of the nodes1110-1183in the communication graph1100may represent an event database entry. The graph1100is shown as being displayed from top to bottom, but may be displayed from left to right or in any other appropriate organization. In the example graph1100, the top of the graph represents the earliest events and the bottom of the graph represents the later events.

For example, when a multi-system operation begins a first event1110may occur. Thereafter, a second event1120may occur. Event1120may be followed by two parallel events1130and1131. The subsequent events1140-1183may occur after the events1130to1131as depicted inFIG. 11. For example, the last events to occur in the multi-system operation may be1180-1183. As discussed above, in some embodiments, selecting or clicking on one of the nodes (e.g., when the graph is displayed on a user device) may show details of the event database entry for that node.

For example, if event node1131is selected, information1190for the selected node1131may be displayed. That information may include any of the information from the event database entry, such as table name correlation identifier, and/or the appearance of the relevant rose before and after the database transaction.

General Overview of Dynamically Enriching and Propagating a Correlation Context

Techniques are provided for executing multi-system operations based on information stored in the commit logs of database systems. Specifically, assume that a particular multi-system operation requires a first set of changes in a first database system and a second set of changes in a second database system. According to one embodiment, the first changes are made by a first transaction in the first database system. When that first transaction commits, information about the first set of changes is stored in the commit logs of the first database system. The commit logs of the first database system are scanned and converted into an event stream.

The event stream is analyzed and, based on the analysis, a second transaction is executed in the second database system to perform the second set of changes. When the second transaction commits, information about the second set of changes is stored in the commit logs of the second database system. At a later time, the commit logs of both the first and second database systems may be read and analyzed to determine whether all changes required by the particular multi-system operation were performed successfully in all systems involved in the particular multi-system operation. Because the first transaction, the second transaction, and the correctness determination are made asynchronously, no synchronization delays are imposed on the individual transactions that collectively constitute the larger multi-system operation.

Operational Overview for Dynamically Enriching and Propagating a Correlation Context

An overview of how multi-system operations may be performed based on the commit logs of database servers shall be given with reference toFIGS. 1 and 2A.FIG. 1is a flowchart for performing a multi-system operation, according to an embodiment.FIG. 2Ais a block diagram of computing devices used to perform the multi-system operation, according to an embodiment. For the purpose of explanation, a multi-system operation shall be described that involves two services. The programs that provided the two services are generally illustrated as first service application202and second service application204. In the case where the multi-system operation is booking an airlines flight, first service application202may be the airline's reservation system, and second service application204may be a banking system that manages the account that is to be used to pay for the airfare.

Performance of the multi-system operation begins at step100, at which a request to perform a multi-system operation (MSO1) is received. For the purpose of illustration, it shall be assumed that the request is in the form of an HTTP request made by a client machine200, and that the request is received over the Internet by a web server that is associated with the first service application202that provides the first service (e.g. reserving a seat on a flight). It shall be further assumed that the multi-system operation requires first changes to be made in a first database system206associated with the first service (e.g. decrementing the number of seats available on the flight), and second changes to be made in a second database system228associated with a second service (e.g. reducing the account balance to reflect the ticket purchase).

At step102, the first service application202generates a correlation identifier (CI1) for the multi-system operation. At step104, the first service application202sends commands to database server208to cause the first database system206to make the first changes as part of a first database transaction (TX1). In addition to the first changes, as part of the same first transaction TX1, the first service application202also causes entries to be added to a first multi-system operation table (MSOT1) that is managed by the first database server208. The entries that are added to MSOT1include the correlation identifier CI1and metadata related to the multi-system operation MSO1. The metadata included in the MSOT1entries shall be described in greater detail hereafter.

When the first transaction TX1commits, commit records for the first transaction are stored in a first commit log212of the first database system206. Consequently, upon commit of the first transaction TX1, all changes made by the first transaction (including both those made to the multi-system operation table MSOT1and those made to other tables (e.g. table210)) will be reflected in commit records that are all assigned the same transaction identifier TX1-id. Specifically, the commit records for the first transaction TX1will contain the before and after values of all data items changed by the first transaction TX1. As mentioned above, the changes made by the first transaction TX1include storing the correlation identifier CI1in the multi-system operation table MSOT1. Consequently, the after-update values of the commit records of the first transaction TX1will include the correlation identifier CI1, so those commit records may easily be identified based on the correlation identifier.

At this point, the two database systems206and228are in an inconsistent state relative to the multi-system operation. Specifically, the changes that reduce the number of available seats that have been committed to database216, but database222has not yet been changed to reduce the account balance to pay for the airfare. Rather than have client machine200send a separate request to the second service application204to cause the second service application204to perform its part of the multi-system operation, the second service application204performs its part of the multi-system operation based on an event stream produced from the commit log212of database system206.

Specifically, at step106, a streaming module214reads the commit log212of the first database system206, and generates a first event stream217therefrom. At step108, a second service application204for the second service reads and analyzes the event stream (either directly or after the stream has been stored as an event log218) to identify any multi-system operations that involve the second service. In the present example, the event stream will include one or more event records, associated with the correlation id CI1, for the multi-system operation MSO1. The second service application204recognizes MSO1as a type of operation that involves the second service. Consequently, at step110the second service application sends commands to the second database system228to perform a second transaction TX2that makes the second changes to complete the multi-system operation MSO1. The specific commands sent by the second service application204are based on the metadata obtained from the event records associated with CI1, which themselves were obtained from the commit records of the first database system206. For the purpose of illustration, it shall be assumed that those changes include adjusting an account balance, which is stored in a row of table224, to indicate that funds were used to buy airfare.

In addition to the second changes, as part of the same second transaction TX2, the second service application204also causes entries to be added to a second multi-system operation table (MSOT2) that is managed by a second database server220. Similar to the entries added to MSOT1, the entries that are added to MSOT2include the correlation identifier CI1and metadata related to the multi-system operation MSO1.

When the second transaction TX2commits, commit records for the second transaction are stored in a second commit log226of the second database system228. Consequently, upon commit of the second transaction TX2, all changes made by the second transaction (including both those made to the multi-system operation table MSOT2and those made to other tables) will be reflected in commit records that are all assigned the same transaction identifier TX2-id. Specifically, the commit records for the second transaction TX2will contain the before and after values of all data items changed by the second transaction TX2. The after-update values of the commit records for the changes to the multi-system operation table MSOT2include the correlation identifier CI1, so those commit records may easily be identified based on the correlation identifier.

Upon commit of the second transaction TX2, the databases216and222involved in the multi-system operation MSO1are consistent. Specifically, the committed changes of table210reflect the reservation was made, and the committed changes of table224reflect the account balance reduction due to the airfare purchase. Further, the commit records in both commit logs212and226that relate to MSO1are associated with the same correlation identifier (i.e. CI1). As shall be explained in greater detail hereafter, the fact all systems have associated the same correlation identifier (CI1) with the changes made for the multi-system operation allows a reconciliation system to determine whether the multi-system operation completed successfully.

In the present example, only two services were involved in the multi-system operation MSO1. Consequently, once TX1has been committed in the first database system206and TX2has been committed in the second database system228, the multi-system operation MSO1has been completed. However, a multi-system operation may involve any number of systems. For example,FIG. 2Bis a block diagram that illustrates how the event stream from one system's commit log can be fed to any number of additional systems (e.g. systems2-N), causing those additional systems to perform their part of any multi-system operations that are reflected in that event stream.

Multi-system operations are not complete until the necessary changes have been committed in each of the systems involved in the multi-system operations. Unfortunately, it is possible that some of the systems are unable to complete their respective portion of the multi-system operation. Thus, techniques shall be described hereafter for determining whether a multi-system operation completed successfully.

Populating the Commit Logs

As explained above, the commit logs of a transaction involved in a multi-system operation are populated (“injected”) with information required for other systems to determine what they must do as part of that multi-system operation. To ensure that the commits logs will have the requisite information, this information is inserted into a multi-system operation table (also referred to as the BizOP table) as part of the same database transaction that is making the changes required by the multi-system operation. The term “correlation context” refers to the information each system involved in the multi-system operation stores/propagates as part of the multi-system operation. As shall be explained hereafter, the correlation context may include many attributes in addition to the correlation identifier.

During the processing of the commit logs, an event stream is generated. For each event represented in the event stream, there is information about the event that may be referred to as the “event payload”. The event payload is controlled by the application itself, because the application of each service that determines what information should be stored in the BIZ-OP table (and committed as part of the same database transaction that is making changes to the other tables used by the service). For example, the event payload for event created by the update to the loan app table (in the creation of a new loan application) should have enough information for person service to extract the person data for the corresponding update that the person application needs to make to the person app table.

Streaming Modules

InFIG. 2b,streaming module214generates an event stream217based on information from the commit log212associated with the first service application202. Each system involved in a multi-system operation may have such a streaming module. The streaming modules produce “event records”. In one embodiment, the event records are ultimately stored in a reconciliation repository. In one embodiment, the reconciliation repository includes a database that can be used for a variety of purposes, including the generation of graphs, and learning what “normal behavior” is for each type of multi-system operation. Once the normal behavior is determined, each new execution of a multi-system operation may be compared to the normal behavior to detect anomalies.

Child Events

In one embodiment, information about all events are stored in an event table, and information about which events have been consumed, and by whom, are stored in a consumption table. On the consumer side, an event can have more than one consumer, so more than one system can process the same event. Thus, a one-to-many relationship is possible between events and consumers.

A single high-level business operation may spawn multiple child operations. For example, when borrower does a payment on a loan, there may be multiple investors who invested the loan. The high-level business operation will correspond, for example, to payment OF 100 dollars which caused the creation of one entry in a row of the BIZOP table. That one row would indicate a business operation payment. However, assume that there are 10 investors in that loan, that payment operation may trigger 10 child business operations, each of which is payment of one of the ten investors. However, a parent operation and all child operations are treated as one business operation, with a single correlation identifier even if the operation touches 10 tables. That is, the child operations inherit the correlation ID of the parent.

Heartbeat Events

According to one embodiment, artificial “heartbeat events” are inserted into a heartbeat table every minute. Heartbeat events do not correspond to any actual multi-system operation. However, the generation of artificial heartbeat events at a predictable schedule allows the system to check its own health. Specifically, if the heartbeat events cease to appear in the event stream that is generated for a particular service, it is likely that something has gone wrong with the service and corrective measures may be initiated.

Correlation Context

The correlation identifier is the metadata that ties together the records of the events that are all part of the same multi-system operation. However, according to one embodiment, the correlation identifier is merely one of many pieces of metadata, for the multi-system operation, that are intentionally injected into the commit logs (by inserting the metadata into an MSO table as part of the transaction that is making the other changes for the MSO).

According to one embodiment, the correlation context includes a history of actions taken as part of the multi-system operation associated with a given correlation identifier (i.e. who called where and when). The history may indicate a sequence number, the source of each call, the name of the service that was called. It may also include the caller ID, the client IP and the additional information that identifies who was the user who actually logged in. Additional details on the content of the correlation context is given hereafter.

Content of the Correlation Context

As explained herein, the correlation context is used in asynchronous “eventual consistency” multi-system operations in which event records derived from the commit log of one system are used to drive additional operations (which are all part of the same multi-system operation) in other systems. For example, a loan system may get a payment. This payment could be processed in system A, and then two days later the record of the event (including the correlation context) can be propagated to system B to cause system B to perform something as part of the same multi-system operation. The correlation context may be used in multi-system operations that include multiple asynchronous transactions in multiple systems. The time that passes between two systems performing their respective parts of the same multi-system operation can be seconds, minutes, hours, or even days.

In one embodiment, when a service receives an initial HTTP call (or an internal system process initiates a multi-system operation), a correlation context is created with just two characteristics. An example of an internally-initiated multi-system operation would be when a scheduled process obtains a payment from a client account. It is not an HTTP but rather an internally-initiated call.

For the purpose of explanation, an example shall be given in which the multi-system operation is initiated by an HTTP call to a service application. In response, the service application may create a correlation context with the following fields:corrId (correlation identifier)corrSourceId (correlation identifier's origination application)seq (sequence number)callerID (caller ID of the consumer)clientID (provided by the UI services, e.g. browser id)userID (UI login user id)client IP (source client IP address)

In JSON format, a particular instance of a correlation context may appear as:

In addition to the fields already discussed herein, the correlation context may include additional fields. Such additional fields may include, for example, module names of the modules, within any given service, that performed work that was part of the multi-system operation.

Propagation of the Correlation Context

As mentioned above, the correlation context of a multi-system operation may be injected into the commit logs of database systems that perform the work required by the multi-system operation. The actual content, format and use of the correlation context may be established in such a way as to improve the performance of the computer systems involved in the multi-system operation. According to one embodiment, the correlation context gets propagated, or sent around in multiple communication channels. Specifically, the applications involved in a multi-system operation may communicate with different protocols. Such protocols include, but are not limited to, TCP, HTTP, messages, events, etc. Further, the correlation context may be contained in or transferred inter-system with files produced by one system that are imported into another system.

The correlation context is propagated internally within an application, across applications, and propagated to external tools involved in the multi-system operation. Consequently, the work done by such external tools can be analyzed and, using the correlation context, what happened external to an application can be related back to what happened within the application.

Dynamically Enriching the Correlation Context

According to one embodiment, the correlation context that is sent between the systems involved in a multi-system operation is not static. Rather, while the correlation identifier remains constant, other metadata within the correlation context may be enriched on the fly. For example, the systems involved in the multi-system operation may add more attributes as tasks are performed as part of the multi-system operation to which the correlation context corresponds.

For example, in one embodiment, the correlation context may start out with only two attributes: the correlation identifier (uniquely identifying the multi-system operation) and an aggregate root identifier (uniquely identifying the highest-level object of the service that was first called to initiate the multi-system operation). When the correlation context is passed to an entity that is able to identify the type of the business operation being performed, an additional attribute “operation type identifier” may be added to the correlation context. In one embodiment, the operation type identifier is a label given to the high-level business operation that is being performed by the service. Within one correlation (identified by a single correlation ID) it is possible to perform multiple business operations of different types. The operation type identifier may be accompanied by a sub-type identifier, which would typically be information that is inserted by the application itself.

When a system involved in the multi-system operation performs some work for the operation as part of a transaction, the transaction identifier for that transaction may be added to the correlation context. Similarly, when a particular application performs work relating to the multi-system operation, an application identifier for the application may be added to the correlation context. Thus, the attributes reflected in the correlation context at any given time may indicate where, within the business flow of that type of business operation, the particular multi-system operation is currently at.

During the life of a multi-system operation, the correlation context associated with the operation can expand (attributes added), or contract (attributes removed), thereby reflecting the current state of the multi-system operation. This correlation context of a multi-system operation is available whenever these operations involve a database (the context will be persisted as part of database operation, for example). If the multi-system operation involves some kind of message queue, then the correlation context becomes part of those messaging events.

With respect to the dynamic enrichment of the correlation context, the correlation context is used to track multiple communications that occurred as part of the multi-system operation. For example, if service does multiple calls to other services, back and forth, back and forth, the correlation context can be used to keep track of those calls. The system can use the correlation context to determine that two entities involved in the multi-system operation communicated only four times, three times, or two times, etc.

In the example correlation context illustrated above, the sequence number is 4. When the correlation context is first created the sequence would be 0. The sequence number is incremented in response to each call made as part of the same multi-system operation, that occurs after the correlation context is created.

Referring toFIG. 7, it is a block diagram that illustrates the content of the correlation context in a multi-system operation that is initiated by a call to an “Origination App”, where the origination app calls a service2and the service2calls services3and4. In addition to those calls, the origination app, service2and service4also pass the correlation context in interactions with an external tool (Splunk).

In one embodiment, at the time the correlation context is created, sequence number is going to be zero, the caller ID is going to be null, the client IP is going to be the client IP of the system that is initiating the call. Then, when another call is made as part of the same multi-system operation, the correlation context will change: The sequence is going to be one, incrementing the sequence. In addition, the correlation context will be updated: the caller ID is going to be the previous caller (where it started from), and the client ID is updated to show the current client ID. In one embodiment, the correlation context is updated to reflect any form of inter-system communication that is made as part of the multi-system operation, and not just HTTP calls.

When a call reaches the service, the service can detect who made the call, do authentication authorization, and then inject that user who actually requested this activity in the service. As mentioned above, this and other metadata about the multi-system operation can be “injected” by inserting the data into a multi-system operation table (a.k.a. BizOp table) and then committing those inserts as part of the same transaction that made, in that same system, the other changes required by the multi-system operation.

In response to the initial call, the service application generates this a unique ID for the multi-system operation (the correlation ID). In addition to the correlation ID, the service also stores in the correlation context information that identifies itself (the service that was originally called). In the example given above, the originator is the “auto-funnel” service. Initially, the “caller” attribute may be empty. If the initial call came from a UI, then the system may be able to obtain a “client ID” to put in the correlation context. The “client ID” may, for example, identify a particular web browser as the “client ID” when the call is initiated through a browser. “Client IP” indicates the IP address used by the client to send the message that initiated the request. Once a service (e.g. auto funnel) makes the call downstream, it becomes a caller for downstream call.

The “client IP” is the IP address of the client. Significantly, even though a person using the service does not change, the client IP may change (based on the firewall, etc.). However, Client ID is like a cookie ID, so that type system can keep track that of the same person coming back within a day and logging in again with a different IP, because that person is still going to have the same Client ID.

With respect to the client ID, the system may initially drop a cookie, with some random ID, and the system can then use that client ID to identify that it is the same person coming back each time, even though it could be from a new IP address

With respect to the User ID, this dynamically changes. It is the ID how the calling entity is authorized against this particular service that was called. If a user calls a first service, and the first service calls a second service, then for the first call the User ID would reflect the user authorized to use the first service, and then for the second call the User ID would reflect how the first service authorized to use the second service.

In one embodiment, the newly-created correlation context does not have any sequence number attribute. The attribute is added to the correlation context, with the value “1”, when the initially-called service makes a call. However, in an alternative embodiment, each new correlation context has a sequence attribute, and the value of the attribute is set to “0”.

If the initially-called service does something (e.g. stores records to a database, or makes a call, or makes a message, sends a message), the system adds metadata about that action to the correlation context.

As shall be describe hereafter, this correlation context is piggybacked onto every message sent by the service, and committed with any changes that are persisted to a database system. Those systems that receive the correlation context, they also piggyback it on any interactions they have as part of the same multi-system operation. In addition, they supplement it and enrich it with tags containing their own information.

As an example, service A receives an HTTP call and performs operations as part of a multi-system operation. Service A creates a correlation ID, and adds tags to the correlation context. Then service A makes a call to some other service B, as part of the same multi-system operation. It piggybacks the correlation context onto that call. Service B would then enrich the correlation context by adding its own tags. For example, service B will change (increment) the sequence number.

If service B makes another call, service B would put itself (service B) as the new caller ID. The caller ID is whoever initiated the call in question. The current version of the correlation context will have the caller ID of the most recent call initiator in the multi-system operation. For a call from service one to service two, the correlation context would say caller ID=service one. Once service two calls service three, the correlation context is changed so that caller ID=service two.

The correlation identifier does not change as the multi-system operation progresses. However, the correlation context does. Each system records the correlation context as it existed when that system did its part of the multi-system operation. Consequently, when the records of all systems are collected (for example, by a reconciliation system), a complete history of the multi-system operation may be constructed. The history may include, for example, a call graph (who called who, and when) as well as the actual changes made within each system as part of the multi-system operation.

Because each system records the correlation context at the time the system performs its part of the multi-system operation, later systems can overwrite existing information in the correlation context with their own information without the prior information being lost. Thus, why the correlation context is continuously changing, the size of the correlation context is not necessarily continuously increasing.

In one embodiment, at each call, the sequence ID is incremented, the caller ID usually gets updated (overwritten). The Client ID may stay the same if it's there. The User ID gets updated all the time (It depends, again, how you authorized against that service). Thus, when service one calls service two, how service one is authorized against service two will be the new user ID.

According to one embodiment, any communication that a system (involved in the multi-system operation) does with any other system, regardless of the protocol, the correlation context (including the correlation ID) is piggybacked in those communications. Each of the other systems is designed to store the correlation context in such a way that when everything is pulled together again, they could make the calling graph.

Subscribing to Operation Types

In the scenario illustrated inFIG. 2B, each of systems2-N analyzes all of the events in event stream217(either directly or by reading event log218) to find events that would require actions to be taken by their respective service applications. Rather than have each service application examine all such events, the event stream271(or event log218) may be managed by a publish/subscribe system, where streaming module124“publishes” event records to the event log218, and each of the service applications subscribes to the event records that are associated with operation types that involve the service. For example, system2may be for a financial transaction service, and thus may only subscribe to event types that involve financial transactions. On the other hand, system3may be for a service that manages employee records, and thus may only subscribe to event types that involve employee records. Because each service application would only consume event records for those event types to which it subscribes, the analysis performed by systems2-N is simplified and inter-system security is improved.

As explained above, using the techniques described herein, the changes made in each system involved in a multi-system operation are not made atomically relative to the changes made in each other system involved in the same multi-system operation. Thus, it is possible to have situations in which one or more of the systems involved in a multi-system operation are unable to complete their part of the operation. Consequently, it is desirable to have a mechanism to determine, after the fact, whether all portions of a multi-system operation have been successfully completed.

Referring toFIG. 3, it is a block diagram of a reconciliation system310according to an embodiment. Reconciliation system310is configured to receive the event streams from any number of streaming modules (e.g. streaming modules1,2, N), each of which is associated with separate service. Rather than receive the event records directly from the streams, reconciliation system310may obtain the event records from event logs that store the output of the streams. Further, as mentioned above, each service may actually generate several event streams in parallel, rather than the one-stream-per-service scenario shown inFIG. 3.

Reconciliation system310groups the event records obtained from the event streams based on correlation ID. Thus, each group will correspond to a distinct correlation ID, and will contain all events, from all systems, that have that same correlation ID. Once all event records for a given correlation ID have been identified, reconciliation system310determines whether everything that needed to happen for the multi-system operation associated with that correlation ID did actually happen for that correlation ID. If everything that needed to happen did happen, then reconciliation system310treats that multi-system operation as successful. Otherwise, reconciliation system310treats that multi-system operation as having failed, and remedial measures are taken.

According to one embodiment, the determination of whether a multi-system operation has succeeded is only performed when the multi-system operation is “ripe” for analysis. A multi-system operation may not be ripe for analysis immediately after the correlation ID for the multi-system operation is first seen by reconciliation system310. For example, assume that a particular type of multi-system operation (OP-TYPE-X) requires changes in both database system1and database system2. Assume further that, under normal conditions, the event records for an occurrence of OP-TYPE-X are not produced by streaming module2until two hours after the event records for the same occurrence of OP-TYPE-X are produced by streaming module1. Under these conditions, it would not be desirable for reconciliation system310to conclude that a particular occurrence of OP-TYPE-X has failed simply because reconciliation system310sees an event record with a particular correlation ID for an OP-TYPE-X operation from streaming module1and no event records with that correlation ID from streaming module2. Rather, under these circumstances, the validation analysis for the OP-TYPE-X operation is only ripe after two hours have elapsed since reconciliation system310has seen a correlation ID for an occurrence of OP-TYPE-X from streaming module1. At that point, if reconciliation system310has still not seen any event with that correlation ID from streaming module2, reconciliation system310may determine that the operation failed.

In one embodiment, machine learning is employed to generate the rules for determining whether a particular type of transaction has completed successfully. In an embodiment that employs machine learning to generate the rules for post-facto reconciliation, the machine learning system learns the rules based on the logs from previously executed operations of the same type. For example, based on historic records, the machine learning system may know that other business operations of the same type end up with multiple entries in multiple databases of multiple services, and it knows how much time for all these to happen. If a thousand prior executions of the business operation follow the expected pattern, the machine learning system knows what that particular type of multi-system operation supposed to look like. Deviations from that pattern may be flagged as anomalies.

Building Communication Graphs Based on the Correlation Context of a Multi-System Operation

According to one embodiment, the presence of the correlation context in the logs/events/messages of each system involved in a multi-system operation enables an analyzer system to use the data from the correlation context of the multi-system operation to build a communication graph that illustrates the communications that occurred as a result of execution of the multi-system operation.

Such communication graphs are useful for tracing, debugging and troubleshooting. However, in one embodiment the communication graphs are also use for analysis and other purposes. Thus, a communication graph can be used to see what kind of activity is happening in the system, and based on the graph, it may be evident what triggered a loan, and that the loan creation impacted five other services external to the loan app service itself In one embodiment, an analysis system builds a “live flow graph” which can be used both for debugging and for discovery (auto-discovery and/or delay discovery for activity in the systems).

Using Communication Graphs for Discovery and Anomaly Detection

As mentioned above, one use of communication graphs is to detect anomalies. An anomaly may be indicated, for example, when two occurrences of the same type of business operation result in significantly different graphs. The dissimilarity of the graphs indicates that something has changed in how the system performs that particular type of multi-system operation. The difference may be cause by a new module being added to the business flow (in which case the divergence in the graphs does not represent a problem) or because there has been anomalous behavior that must be addressed. When the same system with the same version of applications is performing the same type of business operation, the anomaly detection system should see most of the same type of activities graph. When a new version of an application is deployed, the graph may change because some new components are being used. That is, the graph now reflects that something happened that had never been executed.

Because the context changes, depending on what the flow actually is, an anomaly detection system can detect what the flow actually is and compare that to what the flow has been in the past, what it is currently expected to be. Based on automatically determined expectations, the anomaly detection system can use the expectations as a reconciliation tool to decide when something strange happens, or didn't happen and should have happened.

Referring toFIG. 5, it is a block diagram that illustrates how communication graphs of prior instances of a particular type of multi-system operation may be used to detect whether a current instance of the particular type of multi-system operation is anomalous. Referring toFIG. 5, communication graphs1-N are created based on the information obtained from the logs of systems1-4. As explained above, each of those communication graphs corresponds to a distinct correlation identifier, and is built from the correlation context information, associated with that correlation identifier, that is stored in the logs of systems1-4. Those N communication graphs are fed as a training set into machine learning engine500. Once trained, machine leaning engine500uses the model produced by the training to determine what features (expected communication graph features502) the communication graph of that particular type of multi-system operation is expected to exhibit.

One the expected communication graph features502have been derived based on the historical communication graphs1-N for that particular type of operation, new instances of that particular type of operation may be analyzed to determine whether they conform to those expectations. For example, assume that expected communication graph features502are for an operation type X. When a new instance of operation type X is executed, a communication graph N+1 may be generated for the new instance based on the correlation context information in the relevant system logs. An anomaly detection system504may compare the new communication graph N+1 against the expected communication graph features502of operation type X to generate an anomaly detection result506. The anomaly detection result506may simply indicate whether the new instance of the operation type X qualifies as an anomaly, or may be a more sophisticated indication of whether the new instance deviates from expectations and, if so, how much and in what ways.

Learning “Normal” Activity of a Multi-System Operation

As explained above, communication graphs of prior executions of a particular type of multi-system operation can be used to determine what activity is “normal” for that particular type of multi-system operation. In one embodiment, the “normal” activity includes both which actions occurred (e.g. system A called system B) but also the timing of those actions. For example, based on the event information stored for prior executions of operation type X, a machine learning system may learn the average time that it takes for each action to happen.

Once the actions and timings of “normal” activity are known, rules may be established for determining what type of activity for that particular type of multi-system operation is anomalous. The rules may relate to activities (e.g. a multi-system operation is anomalous if more than 5 records are created in system B), to timing (e.g. a record should be created in system B within 5 minutes of changes being committed in system A), or to both.

Post-facto reconciliation and anomaly detection involve using injected/logged information about a multi-system operation to determine whether the operation completed successfully in all systems, and/or whether it exhibited behavior that significantly deviated from the expectations for that particular type of multi-system operation. However, rather than mine the logs for errors after the fact, the same pattern information/rules may be used to pre-emptively flag potential problems.

For example, assume that it has been determined (through machine learning or human observation) that for operation type X, action B should happen in system B within five minutes of action A happening in system A. According to one embodiment, these expectations can be used to pre-emptively communicate expectations to the relevant systems. For example, in response to detecting that action A happened in system A, a pre-emptive message may be sent to system B to indicate that action B should happen within five minutes of the occurrence of action A. If system B does not take action B within five minutes, then system B may generate an alert that the multi-system operation in question is deviating from expectations.

The rules used to pre-emptively detect anomalies may be generated by training a machine learning engine, or may simply be manually formulated based on a human analysis of past operation. For example, the system may track the timing at which things normally happen during a particular type of multi-system operation. Such timing information may indicate how quickly the system should see ‘X’ when ‘Y’ happens, when X and Y are operations performed by different services as part of the same multi-system operation.

Based on an expectation model for a particular type of multi-system operation (created manually or by a trained machine learning engine) it is possible to know how much time normally lapses between events involved in that type of multi-system operation. For example, since each one of the INSERTS in the BizOp table will also have the create date, it is possible to know how long it takes on the average between the time that (a) a loan app is created by the loan app service and (b) the person is inserted into the person table in the database used by the person service. Based on this knowledge, events that occur in one system involved in a multi-system operation may trigger the sending of expectation messages to one or more other systems involved in the multi-system transaction. If the other system does not perform the act identified in the expectation message in the time period specified in the expectation message, an alert is generated indicating a problem may have occurred.

FIG. 6is a block diagram illustrating the system ofFIG. 2Awith the addition of a pre-emptive warning system600. Pre-emptive warning system600reads the pattern that a particular type of operation (operation type X) is expected to follow. As mentioned above, this pattern information may be derived by training a machine learning engine based on correlation context information from previously-executed multi-system operation of the same type. Typically, the pattern would include expected timing information, such as “action B should happen in system B within 5 minutes of A happening in system A”.

Based on the pattern information for operation type X, pre-emptive warning system600monitors the event log218to find any events relating to operation type X. In response to detecting an event that indicates that action A occurred in database system206for an operation of type X with correlation identifier X, pre-emptive warning system600may send an expectation message to second service application204. The message may indicate that the second service application204should perform an action B for the multi-system operation with correlation identifier X within five minutes. If second service application204does not perform action B for the multi-system operation with correlation identifier X within 5 minutes, an alert may be raised.

Automated Fixes

As explained above, the multi-system operations performed using the techniques described herein involve transactions that are asynchronously committed relative to each other. Consequently, when an anomaly is detected, the fact that some of the system may have committed their changes makes it impossible to simply “rolled back” the entire multi-system operation. Often, the solution is to execute a compensation transaction that fixes the inconsistency. For example, if system A inserted the data for a loan, but the person record did not show up in system B according to the expected timing, the problem may be fixed by manually executing a compensation transaction that inserts the person record in system B.

After automating the detection of anomalies, and storing a history of how the anomalies were fixed, rules can be formulated to automatically fix them. For example, for a multi-system operation where something went wrong, the system can capture the data that is in a stream and use that information to automatically fix it later, instead of manually entering any fix.

According to one embodiment, in addition to detecting anomalies, the system also tracks the manual fixes that were made to correct the anomalies. Both the anomalies and the manual fixes made to correct the anomalies are fed to a machine learning engine. Once trained in this manner, the machine learning engine generates a “fix suggestion” that regarding how to fix a newly identified anomaly. To avoid erroneous fixes, the fix suggestion may simply be presented to a user, rather than applied automatically. After reviewing the fix suggestion and determining that it is correct, the user may simply select a control to cause the fix suggestion to be executed.

In one embodiment, the automated fix system may track the number and/or percentage of times that the user has executed the automatically presented fix suggestion. In cases where the fix suggestion for a particular anomaly is always or nearly always executed, the user may be presented with an option to cause the automated fix for the particular type of anomaly to be executed automatically in the future.

Referring toFIG. 8, it is a block diagram of a system for automating the generation of suggested fixes. Initially, machine learning engine806is trained with log information for prior anomalous executions of a particular type of operation (e.g. operation type X). In conjunction with the log information of the anomalous executions, the machine learning engine806is fed the corresponding fixes802of each of the anomalous executions. Once trained, the log information for a newly identified anomalous execution may be fed to machine learning engine806, and machine learning engine806generates a suggested fix.

In the illustrated embodiment, the suggested fix is provided to a fix application system810that may either present the suggested fix808to a user for verification, or automatically apply the fix. In situations where the suggested fix808is presented to a user, the user may accept the fix or make modifications to it prior to instructing fix application system810to execute the fix. In cases where the user makes changes to the fix prior to applying it, the modified fix, along with the log information for the newly identified anomalous execution, may be fed back into machine learning engine806for further training.

Aggregate Root Identifiers

In domain driven design, a separate service is provided for each “domain”, and each domain has a highest-level object that it manages. The highest-level object managed by a service is referred to as the aggregate root of the service. For example, the aggregate root of a loan service may be a loan object, while the aggregate root of a person service is a person object. Each instance of an aggregate root will have a unique identifier within its applicable service. Thus, within the loan service, each loan will have a unique identifier. Since “loan” is the aggregate root for the loan service, the unique identifier for a loan is an aggregate root identifier.

In one embodiment, the aggregate root ID associated with each multi-system operation is inserted, as part of the correlation context metadata, into the BIZ-OP table (e.g. MSOT1). As shall be described in greater detail hereafter, the presence of the aggregate root identifier in the event metadata of multi-system operations allows parallelization, and the life-time monitoring of aggregate objects.

Parallelizing the Event Streams

In the embodiments described above, each commit log is read by a single streaming module produces a single event stream. However, to improve performance and take full advantage of multi-processing systems, a single streaming module may generate multiple event streams in parallel, or multiple streaming modules may concurrently generate event streams from the same commit log. According to one embodiment, the only constraint on the parallelization of event stream generation is that events that relate to the same aggregate root be in the same stream and, within the stream, be in the sequence in which the events occurred.

Specifically, if each commit log were mined a single thread, and everything were processed in order, eventually it is not going to scale. Therefore, in one embodiment, when mining a commit log, the commit log is processed in parallel by any number of threads so long as all records associated with any given aggregate root ID are processed in the same thread.

Discovering New Business Operations

In some cases, the operations performed relative to a particular correlation identifier may not be associated with any existing business operation type. That is, there may be event records where the “operation type” attribute is empty.

According to an embodiment, event records that do not have any operation identifier are fed as input to a machine learning engine to identify operations that exhibit similar patterns. Once a set of operations that exhibit the same pattern is identified, a user may determine whether all of those operations represent the same type of high-level business operation. In situations where those operations represent the same type of high-level business operation, a new business operation label may be created for that type of business operation, and the applications that perform that operation may be modified to include that label in the correlation context when performing that type of operation.

In the preceding discussions, event records are grouped based on correlation identifier, where all events with the same correlation identifier are known to have been part of the same multi-system operation. Typically, those records will also have the same aggregate root identifier. As mention above, the aggregate root identifier for a given multi-system operation identifies the specific highest-level object, involved in the multi-system operation, of the first service called to initiate the multi-system operation.

For example, to initiate a loan application, a client may make an HTTP call to a loan application service. In response, the loan application service may generate an identifier for the new loan. That identifier will be included in correlation context of all further operations performed as part of the loan application process. In addition, that same loan identifier will be included in the correlation context of all future operations involving that loan, even though those future operations will have different correlation identifiers. For example, during the life-cycle of each loan, the loan may be involved in several operations: loan application, loan funding, loan issuance. For each loan, each of those operations may have its own correlation identifier, but all event records would have the aggregate root identifier of the loan in question. The set of all event records that are associated with a given high-level object is referred to herein as the “life-cycle record-set” of the high-level object.

Discovering Life Cycle Patterns of a High-Level Object

According to one embodiment, a life-cycle analysis system includes a machine learning system that is trained with the life-cycle record-set of multiple instances of the same type of high-level object. For example, assume that a loan system has handled 10 thousand loans from beginning to completion. The life-cycle record-set of each of those 10 thousand loans may be fed to a machine learning engine to cause the machine learning engine to learn the common patterns for the normal life-cycle of a loan.

Once trained to recognize the normal life-cycle of a high-level item, the lifeOcycle record-set of not-yet-completed items may be fed to the system to detect anomalies in a manner similar to detecting anomalies with individual multi-system operations. For example, if the normal life-cycle pattern of a loan indicates that loans are funded within a certain amount of time of the loan application, the life-cycle analysis system may identify a particular loan as an anomaly if twice that amount of time has occurred and the loan remains unfunded. Thus, the same event data that is used to track multi-system operations can be used to track the larger life-cycle patterns, and then detect anomalous life-cycle situations.

Not only can the detected life-cycle patterns be used to identify anomalous cases, but it may also be used to improve “normal” behavior. For example, it may become apparent from the life-cycle analysis that the normal loan is taking too long at a particular state in the process. Based on the event data, it may be possible to identify exactly where the delay is occurring.

Post-Facto Annotation of Business Operations

In one embodiment, every application will be annotating their API transactions with the appropriate correlation context. Initially, each application may indicate, for example, a business operation tag, and a business operation subtype, and an aggregate root. Unfortunately, legacy applications were not designed to store correlation contexts.

To include legacy data in the training sets used by the machine learning engines, it would be helpful to retroactively add the appropriate annotations to the records of already-executed operations. Specifically, the system finds out the patterns of the database changes, database fields, database changes to and from values, and aggregates the records using the correlation ID.

Based on the patterns, a user may identify what type of database operation was being performed, and cause the legacy data to be annotated accordingly. For example, the machine learning engine may be trained to identify the activity pattern exhibited by a loan application operation. Legacy data operation data may be compared against the pattern to identify the operations that are likely to have been loan applications, but were not annotated as such. The event records for those operations may be annotated to identify the appropriate type and subtype of the operation in question. If the legacy data does not identify an aggregate root, a user can manually annotate the data with appropriate aggregate root ID.

Further, the annotation may be added on-the-fly for data being generated by legacy applications that do not perform the annotation themselves. As an example of on-the-fly annotation, assume that a legacy application X does not inject annotations to reflect the correlation context of the operations it performs. Based on the learned pattern for a known type of business operation, a rule may be: whenever application X performs an operation that changes table X, and field Y, if there is no business operation type identifier associated with the event data, then label this transaction as part of a business operation type X. After such retro-active annotation, the data from the legacy operation can be fed back into the system, causing the streaming modules to producing events that contain the desired contextual information. Thus, for the case of legacy apps that aren't built to automatically annotate their changes, the system monitors what the application is doing and automatically adds the appropriate annotations.

Once an applications records are automatically tagged/annotated, even though the application itself has not changed, the applications events may be used with all of the prediction and anomaly detection techniques that have been described herein.

In an alternative embodiment, rather than automatically annotating a legacy application's output retroactively, modules can be added to the communication channels between the applications to annotate the messages as they pass between systems.

Hardware Overview