Patent Description:
To ensure that a package is applied in full in the target system or not applied at all, the producer job can wait to receive a notification that the consumer job successfully committed the package before processing a new package. While the producer job waits for the notification from the consumer job, the source system and target system can lock their respective persistent storages to ensure data consistency between the source system and the target system. If a problem occurs anywhere between the source system and the target system, the source system can roll back the uncommitted package and try reconstructing it in the target system.

But this handshake process between the producer job and the consumer job can introduce a long wait time that reduces data throughput between the producer job and the consumer job. Moreover, this handshake process can limit the number of producer jobs and consumer jobs that can be executed in parallel because each job may need to be assigned to a separate region of data or else wait for its assigned region of data to be unlocked by another job.

Further, <CIT> contemplates the use of Change Data Capture (CDC) between a source database and a target database. To execute CDC, one or more computer processors execute the steps of obtaining a plurality of log records comprising information on transactions processed in the source database, grouping the plurality of log records into a plurality of groups based on a predetermined condition, and determining that the plurality of obtained log records is to be replicated from the source database to the target database in parallel in a unit of a group.

Thus, according to an aspect, the problem relates to accelerate data extraction from a source system to a target system.

This problem is solved by the subject-matter of the independent claims.

Provided herein are system, apparatus, device, method and/or computer program product embodiments, and/or combinations and sub-combinations thereof, for utilizing look-ahead-staging (LAS) to accelerate data extraction from a source system to a target system while guaranteeing data consistency and reproducibility.

An embodiment operates by receiving a data change for a data extraction from a producer job at a source system. The embodiment stores the data change in a staging area of a persistent storage together with a respective sequence identifier, wherein the staging area corresponds to the data extraction. The embodiment receives a request for a next package of data changes in the staging area from a consumer job at a target system. The embodiment generates the next package of data changes from the staging area. The embodiment transmits the next package to the consumer job. The embodiment receives a commit notification for the next package from the consumer job in response to the transmitting. The embodiment then removes the data changes in the next package from the staging area in response to receiving the commit notification for the next package.

The embodiment can solve at least four technological problems when performing a data extraction from a source system to a target system. First, the embodiment can increase data throughput by breaking the dependency between a producer job at the source system and a consumer job at the target system. For example, the producer job can store a data change for the data extraction in a LAS and immediately commit the data change to a persistent storage of the source system. This can allow the producer job to continue with the data extraction without waiting for a consumer job at the target system to actually commit the data change. Because both the producer job and the consumer job are decoupled, they no longer need to synchronize with each other and can scale independently. As a result, there can be an increase in data throughput between the producer job and the consumer job.

Second, the embodiment can improve load balancing among multiple producer jobs and multiple consumer jobs. For example, the LAS can generate a separate staging area in its persistent storage for each data extraction performed by a producer job. This can allow the producer job to store data changes from its respective data extraction in its respective staging area without blocking another producer job. As a result, multiple producer jobs can be executed in parallel, thereby increasing data throughput. Moreover, the LAS can enable switching to a different producer job when the staging area for the current producer job is full. Similarly, the LAS can enable switching to a different consumer job when the staging area for the current consumer job is empty. This can further increase data throughput between the source system and the target system.

Third, the embodiment can increase data throughput between the source system and the target system by reducing the amount of time the persistent storage of the source system is locked. For example, a producer job can store data changes in its respective staging area and then immediately commit those data changes to the persistent storage of the source system without waiting for those data changes to be committed at the target system. Similarly, a consumer job can retrieve a package of data changes from the LAS without blocking a producer job from processing more data changes for the consumer job. Finally, a consumer job can retrieve a package of data changes from the LAS without blocking another consumer job from retrieving a package of data changes from the LAS.

Fourth, the embodiment can increase data throughput between the source system and the target system by reducing a size of the data extraction. For example, a producer job can reduce the size of the data extraction by applying filters and or projections to the data changes of the data extraction prior to storing the data changes in the LAS. This can increase data throughput by decreasing the amount of time that the persistent storage of the LAS is locked. Moreover, this can increase data throughput by reducing a size of the data transmission to the target system.

<FIG> is a block diagram of a system <NUM> that utilizes LAS to accelerate data extraction from a source system to a target system while guaranteeing data consistency and reproducibility, according to some embodiments. System <NUM> can include source system <NUM>, LAS <NUM>, and target system <NUM>. As would be appreciated by a person of ordinary skill in the art, system <NUM> can include multiple source systems <NUM> and multiple target systems <NUM>.

Source system <NUM> can be a desktop computer, server, virtual machine, container, laptop, tablet, smartphone, or other device as would be appreciated by a person of ordinary skill in the art. Source system <NUM> can also be a software platform for cloud computing.

Source system <NUM> is communicatively coupled to persistent storage <NUM>. Persistent storage <NUM> can represent any storage device that retains data after power to the storage device is shut off. For example, persistent storage <NUM> can be a hard disk drive, solid-state drive, database, filesystem, object-store, or various other types of storage device as would be appreciated by a person of ordinary skill in the art.

LAS <NUM> can be a software module that is communicatively coupled to source system <NUM> and target system <NUM>. LAS <NUM> can be a hardware module, device, or system that is communicatively coupled to source system <NUM> and target system <NUM>.

LAS <NUM> can also be communicatively coupled to persistent storage <NUM>. Persistent storage <NUM> can represent any storage device that retains data after power to the storage device is shut off. For example, persistent storage <NUM> can be a hard disk drive, solid-state drive, database, filesystem, object-store, or various other types of storage device as would be appreciated by a person of ordinary skill in the art.

Target system <NUM> can be a desktop computer, server, virtual machine, container, laptop, tablet, smartphone, or other device as would be appreciated by a person of ordinary skill in the art. Target system <NUM> can also be a software platform for cloud computing.

Target system <NUM> can be communicatively coupled to persistent storage <NUM>. Persistent storage <NUM> can represent any storage device that retains data after power to the storage device is shut off. For example, persistent storage <NUM> can be a hard disk drive, solid-state drive, database, filesystem, object-store, or various other types of storage device as would be appreciated by a person of ordinary skill in the art.

Source system <NUM> can include one or more producer jobs. A producer job can be a software or hardware implemented process that collects data changes made to persistent storage <NUM> by source system <NUM>. A producer job can provide the data changes affecting persistent storage <NUM> to one or more consumer jobs in target system <NUM>. The producer job can provide data changes for a particular data extraction (e.g., data changes to a particular database table). A consumer job can be a software or hardware implemented process that receives the data changes from a producer job and attempts to commit the data changes to persistent storage <NUM> of target system <NUM>.

To accelerate a data extraction, a producer job can provide data changes for the data extraction to LAS <NUM>. LAS <NUM> can store the data changes to persistent storage <NUM>. The producer job can then immediately commit the data changes to persistent storage <NUM>.

After storing the data changes in persistent storage <NUM>, LAS <NUM> can construct a package from the data changes. A package (also referred to as a transaction) can represent a set of data changes that are either applied in full in target system <NUM> (also referred to as committed) or are not applied at all. LAS <NUM> can transmit the package to a consumer job in target system <NUM>. Thus, LAS <NUM> can break the dependency between source system <NUM> and target system <NUM>. Because LAS <NUM> decouples a producer job from a consumer job, the producer job and the consumer job may no longer need to synchronize and can scale independently, thereby potentially allowing maximum data throughput between source system <NUM> and target system <NUM> during data extraction.

To accelerate data extraction, LAS <NUM> can generate a separate staging area in persistent storage <NUM> for each data extraction. A staging area can be a log of data changes made to persistent storage <NUM> for a given data extraction. A producer job can append a data change for a data extraction to its corresponding staging area. LAS <NUM> can generate a package from the data changes in the staging area. LAS <NUM> can then transmit the package to a consumer job at target system <NUM> to apply the data changes to persistent storage <NUM>.

<FIG> is an example staging area <NUM> for a data extraction from a producer job at source system <NUM>, according to some embodiments. <FIG> is described with reference to <FIG>.

LAS <NUM> can generate staging area <NUM> for a data extraction from a producer job at source system <NUM>. For example, LAS <NUM> can generate staging area <NUM> from a schema for a database object (e.g., a database table). LAS <NUM> can generate staging area <NUM> by extending the schema for the database object with additional fields. For example, staging area <NUM> can include data change entries <NUM>, <NUM>, <NUM>, <NUM>, and <NUM>. A data change entry can include an underlying data change <NUM> to the database object, a sequence identifier <NUM>, an operation type <NUM>, and a package identifier <NUM>.

LAS <NUM> can set the size of staging area <NUM>. LAS <NUM> can calculate the size of staging area <NUM> based on throughput. For example, LAS <NUM> can calculate the size of staging area <NUM> based on a maximum package size, a number of active producers, and a number of active consumer jobs. The maximum package size, the number of active producers, and the number of active consumer jobs can be stored in control area <NUM> as maximum package size <NUM>, number of active producers <NUM>, and number of active consumers <NUM>, respectively, as described in <FIG>. LAS <NUM> can calculate the size of staging area <NUM> based on various other parameters as would be appreciated by a person of ordinary skill in the art.

A producer job can append a data change read from persistent storage <NUM> to staging area <NUM> via LAS <NUM>. LAS <NUM> can receive the data change at an input adapter (also referred to as LAS IN adapter). The input adapter can be an application programming interface (API) to LAS <NUM>. The producer job can utilize the input adapter to append the data change to staging area <NUM>, and therefore to store the data change in persistent storage <NUM>.

In some embodiments, a "push" producer job on source system <NUM> transmits the data change to LAS <NUM> in a synchronous call. For example, the "push" producer job can call the input adapter of LAS <NUM> to transmit the data change to LAS <NUM>, and therefore store the data change to persistent storage <NUM>. The "push" producer job can wait until the call succeeds or fails (e.g., receives a response from LAS <NUM>). If the call succeeds (e.g., LAS <NUM> stored the data change to persistent storage <NUM>), the "push" producer job commits the data change that was read from persistent storage <NUM>. In response to the committing, source system <NUM> releases any associated locks on persistent storage <NUM>, thereby reducing the duration of locking of persistent storage <NUM>. If the call fails (e.g., LAS <NUM> failed to store the data change to persistent storage <NUM>), the "push" producer job can retry calling LAS <NUM> to store the data change to persistent storage <NUM>. In the case of a "push," writing into LAS <NUM> can happen from inside the producer job. As would be appreciated by a person of ordinary skill in the art, a "push" producer job can be software that runs in its own execution environment (e.g., outside LAS <NUM>).

In some other embodiments, a "pull" producer job on source system <NUM> can be invoked from an external execution environment (e.g., in the same environment as LAS <NUM>). For example, the input adapter of LAS <NUM> can poll the "pull" producer job on source system <NUM> for new data. In response, the "pull" producer software can transmit a data change representing the new data to LAS <NUM>, and therefore store the data change to persistent storage <NUM>. If the call succeeds (e.g., LAS <NUM> stored the data change to persistent storage <NUM>), the "pull" producer job can commit the data change that was read from persistent storage <NUM>. In response to the committing, source system <NUM> can release any associated locks on persistent storage <NUM>, thereby reducing the duration of locking of persistent storage <NUM>. If the call fails (e.g., LAS <NUM> failed to store the data change to persistent storage <NUM>), the "pull" producer job can be re-invoked by the input adapter of LAS <NUM>. In the case of a "pull," writing into LAS <NUM> can happen from a process inside LAS <NUM>. As would be appreciated by a person of ordinary skill in the art, a "pull" producer job can be software in which its invocation occurs in an external execution environment.

In response to receiving the data change from the producer job, LAS <NUM> can generate a sequence identifier <NUM> indicating the order of the data change among the data changes for the data extraction. The consumer job can use the sequence identifier <NUM> to apply the data change to persistent storage <NUM> in the correct order.

LAS <NUM> can generate the sequence identifier <NUM> using a monotonically increasing number. LAS <NUM> can also generate the sequence identifier <NUM> using a timestamp of the data change. LAS <NUM> can also generate the sequence identifier <NUM> using various other techniques as would be appreciated by a person of ordinary skill in the art.

LAS <NUM> can generate an operation type <NUM> to indicate the type of change performed by the data change. For example, operation type <NUM> can indicate that the type of change is an insertion, update, or deletion. LAS <NUM> can generate the operation type <NUM> by analyzing the data change.

LAS <NUM> can assign a package identifier <NUM> to the data change. The package identifier <NUM> can indicate that the data change belongs to a particular package. LAS <NUM> can initially assign a package identifier <NUM> that indicates the data change is not yet assigned to any package. For example, LAS <NUM> can assign a package identifier <NUM> of <NUM> to the data change to indicate the data change is not assigned to any package. As would be appreciated by a person of ordinary skill in the art, LAS <NUM> can use a different value for package identifier <NUM> to indicate the data change is not assigned to any package.

LAS <NUM> can store the data change together with its respective sequence identifier <NUM>, operation type <NUM>, and package identifier <NUM> as a data change entry (e.g., data change entry <NUM>) in staging area <NUM>. LAS <NUM> can store data changes for the data extraction in order of receipt from the producer job.

Upon receipt of a data change at LAS <NUM>, LAS <NUM> can store the data change in staging area <NUM> of persistent storage <NUM>. If LAS <NUM> successfully stores the data change to persistent storage <NUM>, source system <NUM> can treat the data change as committed without risk of data loss. In other words, once LAS <NUM> stores the data change to persistent storage <NUM>, source system <NUM> can unlock the associated region of persistent storage <NUM> and read the next data change for transmission to LAS <NUM>. This can allow LAS <NUM> to begin processing another data change before a consumer job completely process the original data change. Thus, LAS <NUM> can asynchronously process data changes in order through one or more consumer jobs.

To accelerate data extraction, source system <NUM> can reduce a size of the data changes stored in staging area <NUM>, as well as a number of operations that may need to be performed by a consumer job. Source system <NUM> can reduce a number of rows that are pushed into a producer job, and if feasible into persistent storage <NUM>, by applying one or more filters. A filter can remove the loading of one or more unnecessary rows. For example, if source data in persistent storage <NUM> has a field for "year," and a query only applies to the current year, the producer job can skip loading the other years. Source system <NUM> can apply one or more filters where a producer job does not support filtering itself.

In some embodiments, a producer job can "push down" a filter by applying it in the producer job. In some other embodiments, the producer job can "push down" a filter into a source database on persistent storage <NUM> by incorporating it into a database query.

Source system <NUM> can also reduce number of columns that are pushed into a producer job, and if feasible into persistent storage <NUM>, by applying one or more projections. A projection can remove the loading of one or more unnecessary columns (also referred to as a fields).

In some embodiments, a producer job can "push down" a projection by applying it in the producer job. In some other embodiments, the producer job can "push down" a projection into a source database on persistent storage <NUM> by incorporating it into a database query.

To accelerate data extraction, LAS <NUM> can generate one or more packages from the staging area <NUM> for transmission to one or more consumer jobs at target system <NUM>. , A package (also referred to as a transaction) can represent a set of data changes that are either applied in full in target system <NUM> (also referred to as committed) or are not applied at all. LAS <NUM> can transmit the package to a consumer job in target system <NUM>. Thus, LAS <NUM> can break the dependency between source system <NUM> and target system <NUM>. Because LAS <NUM> decouples a producer job and a consumer job, the producer job and the consumer job may no longer need to synchronize and can scale independently, thereby potentially allowing maximum throughput during data extraction between source system <NUM> and target system <NUM>.

To reduce the amount of time persistent storage <NUM> is locked, LAS <NUM> can perform a mark and sweep process to generate a package for transmission to a consumer job on target system <NUM>. The mark and sweep process can involve LAS <NUM> marking data change entries in a staging area with a next package identifier. LAS <NUM> can then identify data change entries in the staging area assigned the next package identifier. LAS <NUM> can then generate a package for the next package identifier that includes the identified data change entries.

LAS <NUM> can perform the mark and sweep process using staging area <NUM> and a control area. The control area can track the data changes in staging area <NUM>, reduce pressure on staging area <NUM>, and coordinate the generation and transmission of one or more packages between a producer job on source system <NUM> and a consumer job on target system <NUM>.

<FIG> is an example control area <NUM> for accelerating a data extraction, according to some embodiments. <FIG> is described with reference to <FIG> and <FIG>.

Control area <NUM> can include various control flags. Control area <NUM> can include staging area is full <NUM>, data available <NUM>, maximum package size <NUM>, number of active producers <NUM>, number of active consumers <NUM>, and next package identifier <NUM>. Staging area is full <NUM> can indicate whether staging area <NUM> is currently full. A producer job can check staging area is full <NUM> to determine if it can append a data change to staging area <NUM>. A producer job can often determine if the staging area <NUM> is full much faster and computationally cheaper using staging area is full <NUM>. This is because checking a flag (e.g., staging area is full <NUM>) is often much faster and computationally cheaper than calculating whether staging area <NUM> is full each time.

Data available <NUM> can indicate whether one or more data changes are available in staging area <NUM> for package transmission to a consumer job at target system <NUM>. A consumer job can check data available <NUM> to determine if it can retrieve a package of data changes from LAS <NUM>.

Maximum package size <NUM> can indicate a maximum package size for a package. Maximum package size <NUM> can specify a maximum package size as a maximum number of data changes or a maximum data size. Maximum package size <NUM> can specify a maximum package size in various other ways as would be appreciated by a person of ordinary skill in the art.

Number of active producers <NUM> can indicate the number of active producer jobs interacting with LAS <NUM>. LAS <NUM> can use number of active producer <NUM> to perform load balancing.

Number of active consumers <NUM> can indicate the number of active consumer jobs interacting with LAS <NUM>. LAS <NUM> can use number of active consumers <NUM> to perform load balancing.

Control area <NUM> can include package entries <NUM>, <NUM>, <NUM>, <NUM>, and <NUM>. Each package entry can represent a package that was processed by a consumer job, is being processed by a consumer job, or will be processed by a consumer job. Each package can include a subscription identifier <NUM>, a package identifier <NUM>, a package status <NUM>, and data changes <NUM>. As would be appreciated by a person of ordinary skill in the art, each package can include various other types of fields.

Subscription identifier <NUM> can uniquely identify a subscription of packages. A subscription of packages can represent a group of logically related packages. For example, a subscription of packages can represent a set of packages associated with a particular user.

Package identifier <NUM> can identify a set of data changes in staging area <NUM> that are either applied in full in target system <NUM> or are not applied at all. Package identifier <NUM> can also uniquely identify a logical position of a package within a subscription that is to be processed by a consumer job and committed to target system <NUM>.

Package status <NUM> can represent a processing status of a package. A status of "committed" can indicate the corresponding package was committed by target system <NUM> to persistent storage <NUM>. A status of "in-progress" can indicate the corresponding package is currently being processed by a consumer job and has not yet been committed by target system <NUM> to persistent storage <NUM>. A status of "rolled back" can indicate the corresponding package failed to be committed by target system <NUM> to persistent storage <NUM> and therefore needs to be reconstructed. A status of "new" can indicate the corresponding package has not yet been processed by a consumer job. As would be appreciated by a person of ordinary skill in the art, a package can have various other statuses.

Data changes <NUM> can represent the actual set of data changes making up a package. For example, data changes <NUM> can include a set of data changes in staging area <NUM> having been marked with the corresponding package identifier <NUM>. Data changes <NUM> can represent a series of structured query language (SQL) statements. Data changes <NUM> can also contain the actual contents of the package. Data changes <NUM> can also contain references to the content of the package. As would be appreciated by a person of ordinary skill in the art, data changes <NUM> can contain various other types of data.

To accelerate a data extraction, a consumer job can determine if data changes are available in staging area <NUM> using control area <NUM>. The consumer job can determine if data changes are available using control area <NUM> via an output adapter of LAS <NUM> (also referred to as LAS OUT). The output adapter of LAS <NUM> can be an API.

The consumer job can determine if data changes are available by checking that data available <NUM> is set. Data available <NUM> can be a flag that indicates that data changes are available in staging area <NUM> for package transmission to the consumer job. As would be appreciated by a person of ordinary skill in the art, consumer job can determine if data changes are available using various other techniques.

In some embodiments, if data changes are unavailable for the current consumer job, LAS <NUM> can allow another consumer job to proceed with requesting a next package of data changes. The other consumer job can determine if data changes are available for it and then request a package of those data changes.

In some other embodiments, if data changes are unavailable for the current consumer job, LAS <NUM> can determine a work list of consumer jobs having data available for them. LAS <NUM> can allow each of the consumer jobs in this list to proceed with requesting a respective next package of data changes.

For example, in the case of multiple concurrent extractions to target system <NUM> that are generating data at a slow rate (e.g., during delta extraction where a table is not changed very often), LAS <NUM> can periodically get a work list of extractions. LAS <NUM> can then determine which extraction to execute from the work list. This can avoid the case of having one or more extractions run idle most of the time.

If data changes are available for the current consumer job, the consumer job can request a next package from LAS <NUM>. The consumer job can request the next package via LAS OUT. In response, LAS <NUM> can retrieve the first package that was "rolled back" after a failure to commit the respective package at target system <NUM> to persistent storage <NUM>. If there is no "rolled back" package, LAS <NUM> can generate a new package.

To generate a new package, LAS <NUM> can claim the next package identifier by atomically incrementing the next package identifier <NUM> in control area <NUM>. Next package identifier <NUM> can indicate the next package identifier for a newly generated package.

LAS <NUM> can then perform a marking process. LAS <NUM> can atomically mark data change entries in staging area <NUM> with the next package identifier. LAS <NUM> can mark one or more data change entries in staging area <NUM> such that their respective package identifiers <NUM> are set to the next package identifier. LAS <NUM> can mark one or more data changes entries in staging area <NUM> up to maximum package size <NUM>. Maximum package size <NUM> can specify a maximum package size as a maximum number of data changes or a maximum data size. Maximum package size <NUM> can specify a maximum package size in various other ways as would be appreciated by a person of ordinary skill in the art.

LAS <NUM> can block other consumer jobs from modifying staging area <NUM> during the marking process. This can prevent the other consumer jobs from marking the same data change entries. LAS <NUM> can perform the marking process using a single database operator.

After marking the data change entries in staging area <NUM>, LAS <NUM> can perform a sweep operation. LAS <NUM> can identify data change entries in staging area <NUM> assigned the next package identifier. LAS <NUM> can then generate a package for the next package identifier that includes the identified data change entries.

For example, in <FIG>, package entry <NUM> can represent a package including data change entries <NUM>, <NUM>, and <NUM> from staging area <NUM>. Similarly, package entry <NUM> can represent a package including data change entries <NUM> and <NUM> from staging area <NUM>.

LAS <NUM> can perform the sweeping process without blocking other consumer jobs. This is possible because the data change entries were already marked (e.g., claimed) in staging area <NUM>, and thus this alerts the other consumer jobs so that they avoid remarking these data change entries.

In some embodiments, LAS <NUM> can condense the identified data changes as part of the sweeping process. For example, LAS <NUM> can condense data changes for the same keys. LAS <NUM> can condense data changes by combining multiple data changes for the same record with a single data change. This can increase data throughput between source system <NUM> and target system <NUM> because of the reduced size of the generated package.

After performing the sweep process, LAS <NUM> can store the generated package as a package entry in control area <NUM>. LAS <NUM> can set the package status <NUM> of the generated package to "New. " A status of "new" can indicate that a consumer job has not yet started processing the corresponding package.

After performing the sweep process, LAS <NUM> can provide the generated package to a consumer job. For example, LAS <NUM> can provide the data change entries making up the generated package together with the corresponding package identifier <NUM>. As part of providing the generated package to the consumer job, LAS <NUM> can set the package status <NUM> for the generated package to "in progress. " A status of "in-progress" can indicate the corresponding package is currently being processed by the consumer job but has not yet been committed by target system <NUM> to persistent storage <NUM>.

In response to receiving a commit notification <NUM> for a package from target system <NUM>, LAS <NUM> can mark the corresponding package's status <NUM> as "committed" in control area <NUM>. LAS <NUM> can then delete the corresponding package from control area <NUM> of persistent storage <NUM>. LAS <NUM> can also delete the corresponding data change entries in staging area <NUM> of persistent storage <NUM>.

In some embodiments, where a package is attempted to be committed to multiple target systems <NUM>, LAS <NUM> can mark the corresponding package's status <NUM> as "committed" in control area <NUM> in response to receiving a commit notification <NUM> for the package from each target system <NUM>. LAS <NUM> can then delete the corresponding package from control area <NUM> of persistent storage <NUM> after receiving a commit notification <NUM> from each target system <NUM>. LAS <NUM> can also delete the corresponding data change entries in staging area <NUM> of persistent storage <NUM>.

In response to receiving a rollback notification <NUM> for a package from target system <NUM>, LAS <NUM> can automatically drop all packages subsequent to the rolled back package in control area <NUM>. This can prevent the creation of duplicate packages in persistent storage <NUM>. This can further prevent inconsistencies due to order-dependency in the packages.

In response to receiving the rollback notification <NUM> for the package from target system <NUM>, LAS <NUM> can also rollback and reconstruct all succeeding "in-progress" packages. In some embodiments, LAS <NUM> can identify the succeeding "in-progress" packages as those packages having a package status <NUM> of "in-progress" and a package identifier <NUM> having a time of receipt (e.g., a larger package identifier) that is later than the package being rolled back. In some other embodiments, LAS <NUM> can identify the succeeding "in-progress" packages as those packages having a package status <NUM> of "in-progress," a package identifier <NUM> having a time of receipt (e.g., a larger package identifier) that is later than the package being rolled back, and a subscription identifier <NUM> that is the same as the subscription identifier <NUM> of the package being rolled back.

LAS <NUM> can then generate new packages corresponding to all the succeeding "in-progress" packages. LAS <NUM> can change the package status <NUM> of each of the generated packages to "rolled back. " LAS <NUM> can then sequentially transmit each of the generated packages to a consumer job. LAS <NUM> can change the package status <NUM> of each of the generated packages to "in-progress" after transmitting the respective generated package to the consumer job.

LAS <NUM> can generate these new packages using a variation of the mark and sweep process. LAS <NUM> can generate these new packages in response to a consumer job requesting a next package. LAS <NUM> can retrieve the first package that was "rolled back" after a failure to commit the respective package at target system <NUM> to persistent storage <NUM>. In other words, LAS <NUM> can retrieve the package identifier <NUM> of the first "rolled back" package.

LAS <NUM> can skip the mark process for the first "rolled back" package because the data change entries in staging area <NUM> were previously marked with the corresponding package identifier <NUM>. LAS <NUM> can then perform the sweep process. LAS <NUM> can identify the data change entries in staging area <NUM> that are assigned the package identifier <NUM> of the "rolled back" package. LAS <NUM> can then generate a package for the "rolled back" package that includes the identified data change entries.

In addition to the mark and sweep process, LAS <NUM> can facilitate load balancing by controlling how often a producer job appends data changes to staging area <NUM>. In some embodiments, LAS <NUM> can append data changes for a data extraction from a producer job to staging area <NUM> until reaching a high-water mark of data in staging area <NUM>. This can prevent the ordered list of data changes from growing unbounded and can provide the producer job a chance to do other work.

In some other embodiments, LAS <NUM> can append data changes for a data extraction from a producer job to staging area <NUM> until staging area <NUM> contains a threshold number of data changes. For example, LAS <NUM> can append data changes to staging area <NUM> while the staging area <NUM> contains less than a threshold number of data changes.

In some other embodiments, LAS <NUM> can append data changes for a data extraction from a producer job to staging area <NUM> until staging area <NUM> contains a threshold level of data. In some other embodiments, LAS <NUM> can append data changes for a data extraction from a producer job to staging area <NUM> until reaching various other threshold values.

In some embodiments, LAS <NUM> can append data changes for a data extraction from a producer job to staging area <NUM> until reaching a global high-water mark of data. The global high-water mark of data can be a threshold level of data that is independent of whether the next received data change is part of the same logical set of packages (e.g., shares the same subscription identifier <NUM>).

In some other embodiments, LAS <NUM> can append data changes for a data extraction from a producer job to staging area <NUM> until reaching a subscription high-water mark of data. The subscription high-water mark of data can be a threshold level of data that is subscription identifier <NUM> specific.

In some embodiments, a producer job can determine whether it can append a data change to staging area <NUM> by checking staging area is full <NUM>. Staging area is full <NUM> can be a flag that represents whether staging area <NUM> is full. LAS <NUM> can ensure that staging area is full <NUM> reflects whether staging area <NUM> is full. This can enable the producer job to avoid having to calculate whether staging area <NUM> is full.

<FIG> is a flowchart for a method <NUM> for utilizing a LAS to accelerate data extraction from a source system to a target system while guaranteeing consistency and reproducibility, according to an embodiment. Method <NUM> can be performed by processing logic that can comprise hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, etc.), software (e.g., instructions executing on a processing device), or a combination thereof. It is to be appreciated that not all steps may be needed to perform the disclosure provided herein. Further, some of the steps may be performed simultaneously, or in a different order than shown in <FIG>, as will be understood by a person of ordinary skill in the art.

Method <NUM> shall be described with reference to <FIG>, <FIG>, and <FIG>. However, method <NUM> is not limited to that example embodiment.

In <NUM>, LAS <NUM> receives a data change for a data extraction from a producer job at source system <NUM> for appending to staging area <NUM>.

In some embodiments, LAS <NUM> can generate staging area <NUM> for the data extraction. LAS <NUM> can generate staging area <NUM> based on a schema of a database object (e.g., a database table) associated with the data extraction. LAS <NUM> can also generate staging area <NUM> based on a schema of a database object (e.g., a database table) associated with the data extraction and maximum package size <NUM>. LAS <NUM> can also generate staging area <NUM> based on a schema of a database object (e.g., a database table) associated with the data extraction, maximum package size <NUM>, and number of active consumers <NUM>. LAS <NUM> can also generate staging area <NUM> based on a schema of a database object (e.g., a database table) associated with the data extraction, maximum package size <NUM>, number of active producers <NUM>, and number of active consumers <NUM>.

In some embodiments, LAS <NUM> can determine whether staging area <NUM> is partially empty. LAS <NUM> can receive the data change for the data extraction from the producer job in response to determining that staging area <NUM> is partially empty.

In <NUM> LAS <NUM> stores the data change in staging area <NUM> of persistent storage <NUM> together with a respective sequence identifier <NUM>.

In some embodiments, LAS <NUM> can generate the respective sequence identifier <NUM> in response to receiving the data change. The respective sequence identifier <NUM> can identify an order of the data change in the data extraction.

In <NUM>, LAS <NUM> receives a request for a next package of data changes in staging area <NUM> from a consumer job at target system <NUM>. LAS <NUM> can receive the request via an output adapter of LAS <NUM> (also referred to as LAS OUT). The output adapter of LAS <NUM> can be an API.

In some embodiments, LAS <NUM> can determine whether staging area <NUM> contains data changes for a next package. LAS can determine whether staging area <NUM> contains data changes for a next package by checking whether data available <NUM> is set. LAS <NUM> can receive the request for a next package of data changes from the consumer job in response to determining that staging area <NUM> contains data changes for a next package.

In some embodiments, if data changes are unavailable for the consumer job, LAS <NUM> can allow another consumer job to proceed with requesting a next package of data changes. The other consumer job can determine if data changes are available for it and then request a package of those data changes. In some other embodiments, if data changes are unavailable for the consumer job, LAS <NUM> can determine a work list of consumer jobs having data available for them. LAS <NUM> can allow each of the consumer jobs in this list to proceed with requesting a respective next package of data changes.

In <NUM>, LAS <NUM> generates the next package of data changes from staging area <NUM>.

In some embodiments, if there is no "rolled back" package in control area <NUM>, LAS <NUM> can generate a new package. To generate the new package, LAS <NUM> can claim the next package identifier by atomically incrementing the next package identifier <NUM> in control area <NUM>. LAS <NUM> can then atomically mark data change entries in staging area <NUM> with the next package identifier. LAS <NUM> can mark one or more data change entries in staging area <NUM> such that their respective package identifiers <NUM> are set to the next package identifier.

After marking the data change entries in staging area <NUM>, LAS <NUM> can identify data change entries in staging area <NUM> assigned the next package identifier. LAS <NUM> can then generate a new package for the next package identifier that includes the identified data change entries. LAS <NUM> can store the generated package as a package entry in control area <NUM>. LAS <NUM> can set the package status <NUM> of the generated package to "New.

In some other embodiments, if there a "rolled back" package in control area <NUM>, LAS <NUM> can retrieve the first package that was "rolled back" after a failure to commit the respective package at target system <NUM> to persistent storage <NUM>. In other words, LAS <NUM> can retrieve the package identifier <NUM> of the first "rolled back" package.

In some embodiments, LAS <NUM> can set data available <NUM> in control area <NUM> after generating the next package.

In <NUM>, LAS <NUM> transmits the next package of data changes to the consumer job. LAS <NUM> can set the corresponding package status <NUM> of the next package to "in progress" after transmitting the next package to the consumer job.

In <NUM>, LAS <NUM> receives a commit notification <NUM> for the transmitted next package from target system <NUM>. LAS <NUM> can receive the commit notification <NUM> via an output adapter of LAS <NUM> (also referred to as LAS OUT). The output adapter of LAS <NUM> can be an API. LAS <NUM> can mark the transmitted package as "committed" in control area <NUM> in response to receiving the commit notification <NUM>.

In <NUM>, LAS <NUM> removes the data changes in the next package from staging area <NUM> in response to receiving the commit notification <NUM>. LAS <NUM> can also delete the corresponding package from control area <NUM>.

Various embodiments may be implemented, for example, using one or more well-known computer systems, such as computer system <NUM> shown in <FIG>. One or more computer systems <NUM> may be used, for example, to implement any of the embodiments discussed herein, as well as combinations and sub-combinations thereof. Computer system <NUM> can be used, for example, to implement method <NUM> of <FIG>.

Computer system <NUM> may include one or more processors (also called central processing units, or CPUs), such as a processor <NUM>. Processor <NUM> may be connected to a communication infrastructure or bus <NUM>.

Computer system <NUM> may also include user input/output device(s) <NUM>, such as monitors, keyboards, pointing devices, etc., which may communicate with communication infrastructure <NUM> through user input/output interface(s) <NUM>.

One or more of processors <NUM> may be a graphics processing unit (GPU). In an embodiment, a GPU may be a processor that is a specialized electronic circuit designed to process mathematically intensive applications. The GPU may have a parallel structure that is efficient for parallel processing of large blocks of data, such as mathematically intensive data common to computer graphics applications, images, videos, etc..

Computer system <NUM> may also include a main or primary memory <NUM>, such as random access memory (RAM). Main memory <NUM> may include one or more levels of cache. Main memory <NUM> may have stored therein control logic (i.e., computer software) and/or data.

Removable storage unit <NUM> may include a computer usable or readable storage device having stored thereon computer software (control logic) and/or data. Removable storage drive <NUM> may read from and/or write to removable storage unit <NUM>.

Secondary memory <NUM> may include other means, devices, components, instrumentalities or other approaches for allowing computer programs and/or other instructions and/or data to be accessed by computer system <NUM>. Such means, devices, components, instrumentalities or other approaches may include, for example, a removable storage unit <NUM> and an interface <NUM>. Examples of the removable storage unit <NUM> and the interface <NUM> may include a program cartridge and cartridge interface (such as that found in video game devices), a removable memory chip (such as an EPROM or PROM) and associated socket, a memory stick and USB port, a memory card and associated memory card slot, and/or any other removable storage unit and associated interface.

Communication interface <NUM> may enable computer system <NUM> to communicate and interact with any combination of external devices, external networks, external entities, etc. (individually and collectively referenced by reference number <NUM>). For example, communication interface <NUM> may allow computer system <NUM> to communicate with external or remote devices <NUM> over communications path <NUM>, which may be wired and/or wireless (or a combination thereof), and which may include any combination of LANs, WANs, the Internet, etc. Control logic and/or data may be transmitted to and from computer system <NUM> via communication path <NUM>.

Computer system <NUM> may also be any of a personal digital assistant (PDA), desktop workstation, laptop or notebook computer, netbook, tablet, smart phone, smart watch or other wearable, appliance, part of the Internet-of-Things, and/or embedded system, to name a few non-limiting examples, or any combination thereof.

Computer system <NUM> may be a client or server, accessing or hosting any applications and/or data through any delivery paradigm, including but not limited to remote or distributed cloud computing solutions; local or on-premises software ("on-premise" cloud-based solutions); "as a service" models (e.g., content as a service (CaaS), digital content as a service (DCaaS), software as a service (SaaS), managed software as a service (MSaaS), platform as a service (PaaS), desktop as a service (DaaS), framework as a service (FaaS), backend as a service (BaaS), mobile backend as a service (MBaaS), infrastructure as a service (IaaS), etc.); and/or a hybrid model including any combination of the foregoing examples or other services or delivery paradigms.

Any applicable data structures, file formats, and schemas in computer system <NUM> may be derived from standards including but not limited to JavaScript Object Notation (JSON), Extensible Markup Language (XML), Yet Another Markup Language (YAML), Extensible Hypertext Markup Language (XHTML), Wireless Markup Language (WML), MessagePack, XML User Interface Language (XUL), or any other functionally similar representations alone or in combination. Alternatively, proprietary data structures, formats or schemas may be used, either exclusively or in combination with known or open standards.

In some embodiments, a tangible, non-transitory apparatus or article of manufacture comprising a tangible, non-transitory computer useable or readable medium having control logic (software) stored thereon may also be referred to herein as a computer program product or program storage device. This includes, but is not limited to, computer system <NUM>, main memory <NUM>, secondary memory <NUM>, and removable storage units <NUM> and <NUM>, as well as tangible articles of manufacture embodying any combination of the foregoing. Such control logic, when executed by one or more data processing devices (such as computer system <NUM>), may cause such data processing devices to operate as described herein.

It is to be appreciated that the Detailed Description section, and not any other section, is intended to be used to interpret the claims. Other sections can set forth one or more but not all exemplary embodiments as contemplated by the inventor(s), and thus, are not intended to limit this disclosure or the appended claims in any way.

While this disclosure describes exemplary embodiments for exemplary fields and applications, it should be understood that the disclosure is not limited thereto. Other embodiments and modifications thereto are possible, and are within the scope of this disclosure. For example, and without limiting the generality of this paragraph, embodiments are not limited to the software, hardware, firmware, and/or entities illustrated in the figures and/or described herein. Further, embodiments (whether or not explicitly described herein) have significant utility to fields and applications beyond the examples described herein.

Alternate boundaries can be defined as long as the specified functions and relationships are appropriately performed.

Claim 1:
A computer implemented method for accelerating a data extraction from a source system (<NUM>) to a target system (<NUM>), comprising:
receiving, by at least one processor, a data change (<NUM>) for the data extraction from a producer job at the source system (<NUM>), the source system (<NUM>) being communicatively coupled to a first persistent storage (<NUM>);
storing, by the at least one processor, the data change (<NUM>) in a staging area (<NUM>) of a second persistent storage (<NUM>) together with a respective sequence identifier (<NUM>), wherein the staging area (<NUM>) corresponds to the data extraction;
transmitting, by the at least one processor, a response to the producer job that the storing the data change (<NUM>) succeeded causing the producer job committing the data change, wherein in response to the committing the data change by the producer job, the source system (<NUM>) releases an associated lock on the first persistent storage (<NUM>);
receiving, by the at least one processor, a request for a next package of data changes in the staging area (<NUM>) from a consumer job at the target system (<NUM>); generating, by the at least one processor, the next package from the staging area (<NUM>);
transmitting, by the at least one processor, the next package to the consumer job;
receiving, by the at least one processor, a commit notification (<NUM>) for the next package from the consumer job in response to the transmitting the next package; and
removing, by the at least one processor, the data changes in the next package from the staging area (<NUM>) in response to receiving the commit notification (<NUM>) for the next package.