SYSTEM AND METHOD FOR PROVIDING OUTBOX SERVICE

A method and system for providing outbox services separate from a source application in a distributed application system are disclosed. The method includes performing, by the source application, an operation; writing, onto an operation store, data associated with the operation performed by the source application; writing, onto an event store, an event corresponding to the data; receiving, by an outbox transporter and from the event store, the event; and publishing, to a target and by the outbox transporter, the event received by the outbox transport processor, in which the outbox transporter is separate from the source application.

TECHNICAL FIELD

This disclosure generally relates to a system and method for providing separate outbox services for removal of application dependencies and streamlining of event publications.

BACKGROUND

The developments described in this section are known to the inventors. However, unless otherwise indicated, it should not be assumed that any of the developments described in this section qualify as prior art merely by virtue of their inclusion in this section, or that those developments are known to a person of ordinary skill in the art.

In a distributed applications system, many separate applications work together over a network. In order to facilitate such a working relationship, a source application may publish an event for an operation performed to notify other applications, systems and/or services. Based on the event published by a messaging service provided on the respective source application, one or more downstream applications may perform corresponding operations. However, often, in such a distributed application system, a large number of hard dependencies may exist, such that application downtime may increase any time any of the dependencies fail. Accordingly, it may be desirable to remove as many hard dependencies in a critical data path to reduce application downtime and to improve resiliency. However, once hard dependencies are removed or separated, problems of transactionality that applications must support in the storing of critical business data, and forwarding of that data to downstream applications, systems and/or services are introduced. However, existing solutions may be subject to out of order publishing of events, leading to incorrect order of execution by downstream systems, applications and/or services. Further, when an application is restarted from a downed state, queued or buffered events may then polled and processed for publishing. However, a large quantity of events that were queued or buffered during downtime may cause a bottleneck in publishing of events, such that some of the events may eventually go unpublished as the current system may be unable to timely process the backlogged events, leading to obscure operations in the downstream systems, applications and/or services.

SUMMARY

According to an aspect of the present disclosure, a method for performing outbox services separate from a source application in a distributed application system is provided. The method includes performing, by the source application, an operation; writing, onto an operation store, data associated with the operation performed by the source application; writing, onto an event store, an event corresponding to the data; receiving, by an outbox transporter and from the event store, the event; and publishing, to a target and by the outbox transporter, the event received by the outbox transport processor, in which the outbox transporter is separate from the source application.

According to another aspect of the present disclosure, in response to receiving the event by the target, transmitting a query to an outbox recon processor, the query requesting a number of events written to the event store and a number of events published during a specified time period, in which the outbox recon processor is separate from the source application.

According to another aspect of the present disclosure, comparing the number of events written to the event store and the number of events published during the specified time period; and when the number of events published does not match with the number of events written for the specified time period, identifying one or more events that were not published and controlling the outbox transport processor to publish the one or more events, and controlling the outbox transport processor to publish the one or more events identified by the outbox recon processor.

According to yet another aspect of the present disclosure, controlling an outbox replay processor to replay one or more events stored in the event store to the target.

According to another aspect of the present disclosure, the event received by the outbox transport processor is streamed to the target in a sequential manner.

According to a further aspect of the present disclosure, the event is written to the event store in a sequential manner with time stamp information.

According to yet another aspect of the present disclosure, the event is published in a sequential manner according to the time stamp information associated with the event.

According to a further aspect of the present disclosure, the source application does not receive or read information from the event store.

According to another aspect of the present disclosure, the source application does not receive or read information from the operation store.

According to a further aspect of the present disclosure, the event store has a buffer for receiving a plurality of events to the event store, and the outbox transport processor sequentially publishes one event at a time in a stream.

According to another aspect of the present disclosure, a system for performing outbox services separate from a source application in a distributed application system is disclosed. The system includes the source application that performs an operation; an operation store that stores data associated with the operation performed by the source application; an event store that stores an event corresponding to the data stored in the operation store; an outbox transporter that receives, from the event store, the event; and publishes, to a target, the received event, in which the outbox transporter is separate from the source application.

According to a further aspect of the present disclosure, the system further includes an outbox recon processor. The outbox recon processor, in response to receiving the event by the target, the target transmits a query to the outbox recon processor, the query requesting a number of events written to the event store and a number of events published during a specified time period. The outbox recon processor is separate from the source application.

According to a further aspect of the present disclosure, the outbox recon processor compares the number of events written to the event store and the number of events published during the specified time period; and when the number of events published does not match with the number of events written for the specified time period, identifies one or more events that were not published and controls the outbox transport processor to publish the one or more events, and controls the outbox transport processor to publish the one or more events identified by the outbox recon processor.

According to a further aspect of the present disclosure, the system further includes an outbox replay processor that replays one or more events stored in the event store to the target.

According to a further aspect of the present disclosure, the event received by the outbox transport processor is streamed to the target in a sequential manner.

According to a further aspect of the present disclosure, the event is written to the event store in a sequential manner with time stamp information.

According to a further aspect of the present disclosure, the event is published in a sequential manner according to the time stamp information associated with the event.

According to a further aspect of the present disclosure, the source application does not receive or read information from the event store.

According to a further aspect of the present disclosure, the source application does not receive or read information from the operation store.

According to a further aspect of the present disclosure, the event store has a buffer for receiving a plurality of events to the event store, and the outbox transport processor sequentially publishes one event at a time in a stream.

DETAILED DESCRIPTION

FIG.1illustrates a computer system for implementing an outbox system in accordance with an exemplary embodiment.

The system100is generally shown and may include a computer system102, which is generally indicated. The computer system102may include a set of instructions that can be executed to cause the computer system102to perform any one or more of the methods or computer-based functions disclosed herein, either alone or in combination with the other described devices. The computer system102may operate as a standalone device or may be connected to other systems or peripheral devices. For example, the computer system102may include, or be included within, any one or more computers, servers, systems, communication networks or cloud environment. Even further, the instructions may be operative in such cloud-based computing environment.

Furthermore, the computer system102may include any additional devices, components, parts, peripherals, hardware, software or any combination thereof which are commonly known and understood as being included with or within a computer system, such as, but not limited to, a network interface114and an output device116. The network interface114may include, without limitation, a communication circuit, a transmitter or a receiver. The output device116may be, but is not limited to, a speaker, an audio out, a video out, a remote-control output, a printer, or any combination thereof.

In accordance with various embodiments of the present disclosure, the methods described herein may be implemented using a hardware computer system that executes software programs. Further, in an exemplary, non-limited embodiment, implementations can include distributed processing, component/object distributed processing, and an operation mode having parallel processing capabilities. Virtual computer system processing can be constructed to implement one or more of the methods or functionality as described herein, and a processor described herein may be used to support a virtual processing environment.

FIG.2illustrates an exemplary diagram of a network environment with an outbox system in accordance with an exemplary embodiment.

An outbox system202may be implemented with one or more computer systems similar to the computer system102as described with respect toFIG.1.

The outbox system202may store one or more applications that can include executable instructions that, when executed by the outbox system202, cause the outbox system202to perform actions, such as to execute, transmit, receive, or otherwise process network messages, for example, and to perform other actions described and illustrated below with reference to the figures. The application(s) may be implemented as modules or components of other applications. Further, the application(s) can be implemented as operating system extensions, modules, plugins, or the like.

Even further, the application(s) may be operative in a cloud-based computing environment or other networking environments. The application(s) may be executed within or as virtual machine(s) or virtual server(s) that may be managed in a cloud-based computing environment. Also, the application(s), and even the outbox system202itself, may be located in virtual server(s) running in a cloud-based computing environment rather than being tied to one or more specific physical network computing devices. Also, the application(s) may be running in one or more virtual machines (VMs) executing on the outbox system202. Additionally, in one or more embodiments of this technology, virtual machine(s) running on the outbox system202may be managed or supervised by a hypervisor.

In the network environment200ofFIG.2, the outbox system202is coupled to a plurality of server devices204(1)-204(n) that hosts a plurality of databases206(1)-206(n), and also to a plurality of client devices208(1)-208(n) via communication network(s)210. According to exemplary aspects, databases206(1)-206(n) may be configured to store data that relates to distributed ledgers, blockchains, user account identifiers, biller account identifiers, and payment provider identifiers. A communication interface of the outbox system202, such as the network interface114of the computer system102ofFIG.1, operatively couples and communicates between the outbox system202, the server devices204(1)-204(n), and/or the client devices208(1)-208(n), which are all coupled together by the communication network(s)210, although other types and/or numbers of communication networks or systems with other types and/or numbers of connections and/or configurations to other devices and/or elements may also be used.

The communication network(s)210may be the same or similar to the network122as described with respect toFIG.1, although the outbox system202, the server devices204(1)-204(n), and/or the client devices208(1)-208(n) may be coupled together via other topologies. Additionally, the network environment200may include other network devices such as one or more routers and/or switches, for example, which are well known in the art and thus will not be described herein.

The outbox system202may be a standalone device or integrated with one or more other devices or apparatuses, such as one or more of the server devices204(1)-204(n), for example. In one particular example, the outbox system202may be hosted by one of the server devices204(1)-204(n), and other arrangements are also possible. Moreover, one or more of the devices of the outbox system202may be in the same or a different communication network including one or more public, private, or cloud networks, for example.

The server devices204(1)-204(n) may be hardware or software or may represent a system with multiple servers in a pool, which may include internal or external networks. The server devices204(1)-204(n) hosts the databases206(1)-206(n) that are configured to store metadata sets, data quality rules, and newly generated data.

The plurality of client devices208(1)-208(n) may also be the same or similar to the computer system102or the computer device120as described with respect toFIG.1, including any features or combination of features described with respect thereto. Client device in this context refers to any computing device that interfaces to communications network(s)210to obtain resources from one or more server devices204(1)-204(n) or other client devices208(1)-208(n).

According to exemplary embodiments, the client devices208(1)-208(n) in this example may include any type of computing device that can facilitate the implementation of the outbox system202that may efficiently provide a platform for implementing a cloud native outbox system module, but the disclosure is not limited thereto.

One or more of the devices depicted in the network environment200, such as the outbox system202, the server devices204(1)-204(n), or the client devices208(1)-208(n), for example, may be configured to operate as virtual instances on the same physical machine. For example, one or more of the outbox system202, the server devices204(1)-204(n), or the client devices208(1)-208(n) may operate on the same physical device rather than as separate devices communicating through communication network(s)210. Additionally, there may be more or fewer outbox system202, server devices204(1)-204(n), or client devices208(1)-208(n) than illustrated inFIG.2. According to exemplary embodiments, the outbox system202may be configured to send code at run-time to remote server devices204(1)-204(n), but the disclosure is not limited thereto.

FIG.3illustrates a system diagram for implementing an outbox system in accordance with an exemplary embodiment.

As illustrated inFIG.3, the system300may include an outbox system302within which a group of API modules306is embedded, a server304, a database(s)312, a plurality of client devices308(1) . . .308(n), and a communication network310.

According to exemplary embodiments, the outbox system302including the API modules306may be connected to the server304, and the database(s)312via the communication network310. Although there is only one database has been illustrated, the disclosure is not limited thereto. Any number of databases may be utilized. The outbox system302may also be connected to the plurality of client devices308(1) . . .308(n) via the communication network310, but the disclosure is not limited thereto.

According to exemplary embodiment, the outbox system302is described and shown inFIG.3as including the API modules306, although it may include other rules, policies, modules, databases, or applications, for example. According to exemplary embodiments, the database(s)312may be embedded within the outbox system302. According to exemplary embodiments, the database(s)312may be configured to store configuration details data corresponding to a desired data to be fetched from one or more data sources, user information data etc., but the disclosure is not limited thereto.

According to exemplary embodiments, the API modules306may be configured to receive real-time feed of data or data at predetermined intervals from the plurality of client devices308(1) . . .308(n) via the communication network310.

The API modules306may be configured to implement a user interface (UI) platform that is configured to enable outbox system as a service for a desired data processing scheme. The UI platform may include an input interface layer and an output interface layer. The input interface layer may request preset input fields to be provided by a user in accordance with a selection of an automation template. The UI platform may receive user input, via the input interface layer, of configuration details data corresponding to a desired data to be fetched from one or more data sources. The user may specify, for example, data sources, parameters, destinations, rules, and the like. The UI platform may further fetch the desired data from said one or more data sources based on the configuration details data to be utilized for the desired data processing scheme, automatically implement a transformation algorithm on the desired data corresponding to the configuration details data and the desired data processing scheme to output a transformed data in a predefined format, and transmit, via the output interface layer, the transformed data to downstream applications or systems.

The plurality of client devices308(1) . . .308(n) are illustrated as being in communication with the outbox system302. In this regard, the plurality of client devices308(1) . . .308(n) may be “clients” of the outbox system302and are described herein as such. Nevertheless, it is to be known and understood that the plurality of client devices308(1) . . .308(n) need not necessarily be “clients” of the outbox system302, or any entity described in association therewith herein. Any additional or alternative relationship may exist between either or both of the plurality of client devices308(1) . . .308(n) and the outbox system302, or no relationship may exist.

The first client device308(1) may be, for example, a smart phone. Of course, the first client device308(1) may be any additional device described herein. The second client device308(n) may be, for example, a personal computer (PC). Of course, the second client device308(n) may also be any additional device described herein. According to exemplary embodiments, the server304may be the same or equivalent to the server device204as illustrated inFIG.2.

The process may be executed via the communication network310, which may comprise plural networks as described above. For example, in an exemplary embodiment, one or more of the plurality of client devices308(1) . . .308(n) may communicate with the outbox system302via broadband or cellular communication. Of course, these embodiments are merely exemplary and are not limiting or exhaustive.

The computing device301may be the same or similar to any one of the client devices208(1)-208(n) as described with respect toFIG.2, including any features or combination of features described with respect thereto. The outbox system302may be the same or similar to the outbox system202as described with respect toFIG.2, including any features or combination of features described with respect thereto.

FIG.4Aillustrates a desired method to be performed to remove hard dependencies in a distributed application environment in accordance with an exemplary embodiment.

In operation1, AppA stores data in an operation store. In operation2, AppA communicates a corresponding event to AppB so that AppB is aware of the storing of the data by AppA. In operation3, AppB communicates with AppA to inform AppA how many events were processed and published. In order to achieve the desired method outlined inFIG.4A, pattern-based solutions were proposed, which is exemplarily illustrated inFIG.4B. technology.

FIG.4Billustrates a pattern-based solution in accordance with a related technology.

In related technology, there may be two common patterns that applications implement to address these problems, namely, Store & Forward and Transactional Outbox patterns. These patterns have six main requirements for correctness: (1) a need to transactionally commit the business record with the record to forward; (2) a need to guarantee the delivery of forwarded record; (3) a need to recon the deltas of data delivered vs. not delivered for a given time; (4) a need to re-deliver data to a target technology; (5) need to forward like-records in-order; and (6) a need to scale for normal and failure scenarios.

However, due to vast amounts of permutations of source technologies to target technologies, this leads to brittle, often-times incorrect implementations, and point-to-point solutions that do not scale for Enterprise use when there are a large number of (micro) services and engineers which must implement the patterns correctly.

In operation1a, AppA stores data into an operation store. In operation1b, AppA stores a corresponding event in an event store. In operation2, AppA publishes the event to notify another application, such as AppB of the event. The pattern-based solutions may rely on publication of events and parallel storage. Generally, when a change has occurred in a distributed system, an event may be published to alert relevant parties of the occurred event. Also, the event of such change may be stored in parallel. However, when publishing events, the events may not necessarily be published in order. Accordingly, a subsequent operation may be performed before a prerequisite operation is performed due to a delayed publishing of an event that would have triggered the prerequisite operation.

In operation3a, a poller component of AppA polls the event store and publishes any missed events. According to exemplary aspects, when AppA fails, there may be events that were generated but not yet published during the downtime. In operation3b, the poller component of AppA then publishes any missed events to notify AppB of such events. In operation4a, the recon component of AppA will recon the events published while AppA was down. In operation4b, the recon events are then published to notify AppB of the reconned events.

However, polling of events may cause bottle necks whenever its system goes down, which in turn will slow down subsequent operations as downstream systems may be unaware of the events to trigger operation executions. In certain cases, if the downtime is significant, all of the events that failed to publish during the downtime will be polled all at once for publication leading to significant bottleneck. For example, if a system averages publication of 1 million events per hour, and the system is down for six hours, the system may be unable to capture and process 7 million events at the seventh hour when it goes back up. However, the application may be unable to publish such a large number of events, which will in turn lead to a large number of events that may not publish for notifying other applications of such events.

FIG.4Cillustrates a system flow diagram utilizing an outbox system in accordance with an exemplary embodiment.

As illustrated inFIG.4C, AppA stores data into an operation store in operation1a, and stores a corresponding event in an event store in operation1b. According to exemplary aspects, operations1aand1bmay be performed contemporaneously. However, aspects of the present disclosure not limited thereto, such that operation1ais performed prior to performance of the operation1b. Unlike the system flow illustrated inFIG.4B, AppA only transmits or writes to the operation store and the event store, and does not receive or read information from the operation store or the event store for publishing events and/or performing a recon operation.

In operation2, a database (DB) log tail is generated for the event store. According to exemplary aspects, DB log may include a sequential log of all of the events that were stored in the event store. The event store then streams the event that was stored in the event store to the outbox transporter in operation3a. In operation3b, the outbox transporter transmits the event to the App B. In operation4, the App B communicates with the outbox reconer via an API to obtain how many events were processed and published for a given period. In operation5, the outbox reconer communicates with the event store to determine how many events were processed and published for the given period. If the outbox reconer determines that a number of events processed corresponds or matches with a number of events published for the given period, no further action is performed or an indication of the matching numbers is communicated to the event store and/or the outbox transporter. On the other hand, if the outbox reconer determines that the number of events processed is greater than or does not match with the number of events published for the given period, the outbox reconer may identify the unpublished events and control the event store and/or the outbox transporter to publish the unpublished events so that none of the stored events are left unpublished. Further, according to exemplary aspects, each of the events may be associated with a time stamp to ensure sequential processing.

Based on such a configuration, chances of out of order publishing is reduced or eliminated by streamed event transmission. Likewise, at least since the events are streamlined to the downstream App, a risk of incurring polling bottlenecks upon resuming operation after downtime is reduced. Also, by leveraging outbox recon service, a check may be performed to ensure all of the events were processed and published to avoid a risk of having events unprocessed or unpublished.

FIG.5illustrates a system flow diagram in a distributed application environment in accordance with an exemplary embodiment.

According to exemplary aspects, an outbox solution or system that is built-once and reused-anywhere that can adapt to the many different permutations of source to target technologies is provided. Applications in need of transactionally Store & Forwarded data can configure and plug and play the outbox solution to cater to their need. The Outbox solution provides the following capabilities: (1) transactionality—business data and outbox data exist atomically; (2) forwardability—outbox data is guaranteed for delivery; (3) replayability—outbox data is replayable from any point in time; and (4) ordered processing—outbox data is forwarded in-order for the same business record. Although four capabilities are iterated for the Outbox solution, aspects of the present disclosure are not limited thereto, such that an Outbox solution may be implemented with less or more capabilities than the ones identified.

Although patterns may be beneficial for understanding and building, they may be impractical at scale to adapt to technology landscape and variability of technologies. Libraries may be good for re-use, however, libraries are language specific and is thus, impactful to the in-line processing of the application. Further, services may be great for re-use that's language agnostic, however, services are executed out-of-line processing of the application. In contrast to the above, the outbox solution as a service may be deployed to any environment to adapt to an application's need.

A network of systems that utilizes an outbox system or solution includes a group of source apps510, a group of sources520, and an outbox system530and a group of targets540.

The group of source apps510may include one or more applications that may generate an event to be forwarded for subsequent or corresponding processing. In an example, one or more of the applications (e.g., App1, App2App N) included in the group of source apps510may execute a code or deploy a patch. For example, one or more of the applications may include an event-driven, serverless computing platform application, an elastic Kubemetes service application, a deployment and management application that distributes application workloads across a Kubernetes cluster, and the like.

The source520includes a postgres/mySQL 521, dynamo database (DB)522, NoSQL DB management system523, operation store524and an event store525. Although five sources of data are described herein, aspects of the present disclosure are not limited thereto, such that more or less data sources may be present.

Once one or more of the applications included in the group of source apps510generates data, the generated data may be written onto the operation store524, and a corresponding event associated with the generated data may be written onto the event store525. In an example, events written onto the event store525may be sequentially written and/or with time stamp information to ensure that the events are similarly sequentially published by the outbox530. Further, the event store525may include a buffer to receive multitudes of events from one or more source applications included in the group of source apps510.

The outbox system530includes a transporter531, a reconer532, and a replayer533. According to exemplary aspects one or more of the transporter531, the reconer532, and the replayer533may be implemented by a processor or an integrated circuit.

According to exemplary aspects of the present disclosure, the outbox system530provides an externalized messaging service, which is separated and independent of the source application. Such a configuration may allow the outbox system530to continue publishing of events stored at the event store independent of operating status of the source application. Accordingly, even if the source application is in a downed state, the outbox system530may continue to publish events so that a bottleneck is avoided when the source application returns to an operating state. Further, at least since the publication, polling and reoconing operations are separately performed by the outbox530, memory and/or processing resources typically utilized by source apps may be redirected to perform application operations (e.g., running of code or patch), allowing for more application operations to be performed for more efficient processing. Accordingly, scaling of such source apps510may more easily or efficiently be achieved than previously.

Further, as exemplarily illustrated inFIGS.4B and4C, the transporter531is separated from the source application for independent processing of publication of events. The transporter531may sequentially receive or obtain events from the event store525, and stream the received events to one or more targets included in the group of targets540. At least since the events are retrieved or received from the event store525, rather than from a source app, even if the source app is to incur a downtime, the events stored in the event store525is streamed to the one or more targets included in the group of targets540. Accordingly, bottle neck caused by polling of events after a downtime may be reduced.

According to exemplary aspects and exemplarily illustrated inFIGS.4B and4C, the reconer532may also be separated from the source application for independent processing of recon operation. The reconer532may, in response to a communication signal from a target application, query the event store525to verify that a number of events received by the target application corresponds or matches with a number of events written to the event store525for a specified time period (e.g., a specific date, a particular time window on a specific date, a date range and the like). When the reconer532determines that the number of events processed via the event store525matches or corresponds to the number of events published for the specified time period, the reconer532may communicate with the event store525and/or the transporter531to indicate that all of the processed events have been published.

According to exemplary aspects, the replayer533may replay events to a destination different from a target application. In an example, the replayer533may select a destination of a different technology or same technology but a different topic. Accordingly, events may be replayed to other destination targets even when they may not be original intended targets.

The group of targets540may include one or more targets, such as App1, App2App N. Although applications are illustrated as being targets, aspects of the present disclosure are not limited thereto, such that targets may include a service, a data stream, a serverless event bus, and the like.

FIG.6illustrates a process flow for performing an outbox transportation operation in accordance with an exemplary embodiment.

In operation A1, a user may access a cloud management API to access a source application1on a cloud network for posting operation information. In operation A2b, the source application1may write the created/updated data into a source database and also write in parallel corresponding data and event into an outbox system. According to exemplary aspects, the database source may include an object store for storing of data and an event store for storing of events. The source database and the outbox system may communicate with one another.

In operation B1, the could management API may then perform a put operation for inserting the operation information into an identifier file. A source application2then performs operation B2ato write association of the operation information with the identifier file into the source database. In operation B2b, the source application2write corresponding data and event into the outbox system.

In operation C1a, the cloud management API may perform a put operation for performing an update request on a target application1for synchronizing operation information with an identifier of the user. In operation C2a, the target application1may perform a polling operation on the source application2to verify the number of events that were processed and published within a specified time frame (e.g., an hour, a day, or the like). In operation C1b, the events that were written into the outbox system are then published to an outbox stream. In operation C2b, the outstream streams the published event to an outbox transporter. In operation C3, the outbox transporter publishes the event to a target application2. In operation C4, the outbox transporter sends a message back to the outbox to inform the outbox system that the event has been published to the target application. The outbox system may then write an event to the outbox system indicating publishing the event to the target application2.

In operation D1, the cloud management API may request an outbox reconer to perform a get operation for obtaining a number of events that were processed and published within a specified time frame. For example, the request may specify to obtain a number of events that were processed and published by the outbox system for a specified time range. In operation D2, the outbox reconer performs a read operation of the outbox system for obtaining the requested information. For example, the outbox reconer may read that 100 events were processed and 100 events were published, indicating that all events were published. In another example, the outbox reconer may read that 100 events were processed but only 90 were published, indicating that 10 events were not published. In such a scenario the outbox reconer may identify the non-published events and publish such events in a sequential order. At least since the events may utilize database log tailing with time stamp information, a sequential order of events may be indicated. Accordingly, out of order publication may be avoided, which may be problematic in a pattern-based method as exemplarily illustrated inFIG.4B.

FIG.7illustrates a process flow for performing an outbox recon operation in accordance with an exemplary embodiment.

According to exemplary aspects, a primary key (pk) may be utilized to aggregate events for a certain record identifier. Further, a sort key (sk) may be utilized for identifying sequential order of the events for the respective primary key.

As exemplarily provided inFIG.7, the outbox reconer may access various source databases and outbox systems for aggregating all of the processed events and published events for a specified time period. Further, all of the events processed and published for may be specified for a particular product code. However, aspects of the present disclosure are not limited thereto, such that processed and published events may be identified for different criteria.

For example, recon criteria may specify a product code of 012 for the date of Jul. 9, 2022. As illustrated inFIG.7, product code 012 is indicated as being associated with a pk of6016. . . , which may be a record identifier, in the source database. In the outbox system, pk may tie in the identifier with the product code, which may also indicate an event identifier as the sk, along with time stamp information, status information and recon criteria information. The identifier may also be associated with a unique identifier of a shard within a stream, such as a shardid.

The outbox reconer may also access various streams, such as outbox stream with multiple shards (e.g., 5 shards) and a stream with multiple shards (e.g., 5 shards). The data stored in the outbox stream may indicate a payload associated with the shardid and the identifier.

Based on all of the accessed information, the outbox reconer may be able to aggregate relevant information corresponding to the recon criteria specified, such as status, pk, sk, payload and updated time stamp information.