Data reconciliation system

A method includes receiving a data reconciliation request requesting data reconciliation for a first dataset and a second dataset. The method also includes obtaining the first dataset including one or more dimensions each having a plurality of dimension members and obtaining the second dataset including one or more dimensions each having a plurality of dimension members. For each dimension of the first dataset, the method includes obtaining a bridge member that associates the respective dimension of the first dataset with a dimension of the second dataset. The method includes generating a first and second set of combination dimension members and refreshing the first and second set of combination dimension members using an execution delimiter value. The method includes generating, using the refreshed first and second set of combination dimension members, a third set of combination dimension members to generate a data reconciliation report.

TECHNICAL FIELD

This disclosure relates to data reconciliation.

BACKGROUND

Data reconciliation is a process for verifying data during or after, for example, data migration. That is, target data may be compared against original data to ensure that the data was properly migrated or in the case of system updates or changes, data that should have remained static did not change. In such a situation, data reconciliation procedures check for missing records or values, incorrect records or values, duplicated values or records, and various formatting issues. Conventional data reconciliation techniques are expensive, time consuming, error prone, or some combination thereof.

SUMMARY

One aspect of the disclosure provides a computer-implemented method that when executed on data processing hardware causes the data processing hardware to perform operations for data reconciliation. The operations include receiving a data reconciliation request requesting data reconciliation for a first dataset and a second dataset from a user device in communication with the data processing hardware. The operations also include obtaining the first dataset from a first data source. Here, the first dataset includes one or more dimensions each having a plurality of dimension members. The operations also include obtaining the second dataset from a second data source. Here, the second dataset includes one or more dimensions each having a plurality of dimension members. For each respective dimension of the first dataset, the operations also include obtaining a respective bridge member associating the respective dimension of the first dataset with a respective dimension of the second dataset. The operations also include generating a first set of combination dimension members using each pair of dimension members of the one or more dimensions of the first dataset and the respective bridge members. The operations also include generating a second set of combination dimension members using each pair of dimension members of the one or more dimensions of the second dataset and the respective bridge members. The operations also include refreshing the first set of combination dimension members and the second set of combination dimension members based on an execution delimiter value corresponding to a quantity of dimension members refreshed simultaneously. The operations also include generating a third set of combination dimension members using the refreshed first set of combination members and the refreshed second set of combination dimension members and generating a data reconciliation report from the third set of combination dimension members.

Implementations of the disclosure may include one or more of the following optional features. In some implementations, refreshing the first set of combination dimension members and the second set of combination dimension members based on the execution delimiter value further includes refreshing the first set of combination dimension members based on a first execution delimiter value and refreshing the second set of combination dimension members based on a second execution delimiter value. Refreshing the generating the first set of combination dimension members and the second set of combination dimension members based on the execution delimiter value may further include obtaining an updated dimension member from the plurality of dimension members of the first dataset from the first data source and obtaining an updated dimension member from the plurality of dimension members of the second dataset from the second data source.

In some examples, generating the third set of combination members using the refreshed first set of combination dimension members and the refreshed second set of combination dimension members includes suppressing a portion of the combination dimension members. Here, the portion of the combination dimension members may include combination dimension members associated with a zero or no data. In these examples, suppressing the portion of the combination dimension members includes zeroing a second portion of the combination dimension members based on a variance threshold.

In some implementations, generating the third set of combination dimension members using the refreshed first set of combination dimension members and the refreshed second set of combination dimension members includes inversing values associated with the refreshed second set of combination dimension members. In other implementations, generating the third set of combination dimension members using the refreshed first set of combination dimension members and the refreshed second set of combination dimension members includes combining the refreshed first set of combination dimension members and the refreshed second set of dimension members into a linear format. The operations may further include sending the data reconciliation report to the user device. Here receiving the data reconciliation report causes the user device to display the data reconciliation report via a graphical user interface.

Another aspect of the disclosure provides a system that includes data processing hardware and memory hardware storing instructions that when executed on the data processing hardware causes the data processing hardware to perform operations. The operations include receiving a data reconciliation request requesting data reconciliation for a first dataset and a second dataset from a user device in communication with the data processing hardware. The operations also include obtaining the first dataset from a first data source. Here, the first dataset includes one or more dimensions each having a plurality of dimension members. The operations also include obtaining the second dataset from a second data source. Here, the second dataset includes one or more dimensions each having a plurality of dimension members. For each respective dimension of the first dataset, the operations also include obtaining a respective bridge member associating the respective dimension of the first dataset with a respective dimension of the second dataset. The operations also include generating a first set of combination dimension members using each pair of dimension members of the one or more dimensions of the first dataset and the respective bridge members. The operations also include generating a second set of combination dimension members using each pair of dimension members of the one or more dimensions of the second dataset and the respective bridge members. The operations also include refreshing the first set of combination dimension members and the second set of combination dimension members based on an execution delimiter value corresponding to a quantity of dimension members refreshed simultaneously. The operations also include generating a third set of combination dimension members using the refreshed first set of combination members and the refreshed second set of combination dimension members and generating a data reconciliation report from the third set of combination dimension members.

Implementations of the disclosure may include one or more of the following optional features. In some implementations, refreshing the first set of combination dimension members and the second set of combination dimension members based on the execution delimiter value further includes refreshing the first set of combination dimension members based on a first execution delimiter value and refreshing the second set of combination dimension members based on a second execution delimiter value. Refreshing the generating the first set of combination dimension members and the second set of combination dimension members based on the execution delimiter value may further include obtaining an updated dimension member from the plurality of dimension members of the first dataset from the first data source and obtaining an updated dimension member from the plurality of dimension members of the second dataset from the second data source.

In some examples, generating the third set of combination members using the refreshed first set of combination dimension members and the refreshed second set of combination dimension members includes suppressing a portion of the combination dimension members. Here, the portion of the combination dimension members may include combination dimension members associated with a zero or no data. In these examples, suppressing the portion of the combination dimension members includes zeroing a second portion of the combination dimension members based on a variance threshold.

In some implementations, generating the third set of combination dimension members using the refreshed first set of combination dimension members and the refreshed second set of combination dimension members includes inversing values associated with the refreshed second set of combination dimension members. In other implementations, generating the third set of combination dimension members using the refreshed first set of combination dimension members and the refreshed second set of combination dimension members includes combining the refreshed first set of combination dimension members and the refreshed second set of dimension members into a linear format. The operations may further include sending the data reconciliation report to the user device. Here receiving the data reconciliation report causes the user device to display the data reconciliation report via a graphical user interface.

DETAILED DESCRIPTION

Data reconciliation is a process for verifying data during or after, for example, data migration. For example, target data is compared against original data to ensure that the data was properly migrated or in the case of system updates or changes, data that should have remained static did not change. In such a situation, data reconciliation procedures check for missing records or values, incorrect records or values, duplicated values or records, and various formatting issues. Conventional data reconciliation techniques are expensive, time consuming, error prone, or some combination thereof.

Implementations herein are directed toward data reconciliation controller that not only decreases time required to start and perform data reconciliation (e.g., of financial data or other values), but to provide a more rich and valuable information trail that is more visible faster, thus allowing a user to focus on fixing problems instead of finding them. The data reconciliation controller exposes errors in approaches to fixing issues. For example, when an issue is improperly fixed, the data reconciliation controller allows a user to both find the issue and revert to a proper solution in less time. The data reconciliation controller employs methods that ensures users focus on what is important earlier in the process (e.g., items that frequently cause delays) for not just top and bottom layers, but every layer in-between. The data reconciliation tool provides auditing tools that allows a user practical ways to enter and filter information that enable drawing conclusions faster while simultaneously creating and maintaining an auditing trail.

Referring now toFIG.1, in some implementations, an example system100includes a processing system140in communication with one or more user devices110via a network120. The processing system140may be a single computer, multiple computers, a user device, or a distributed system (e.g., a cloud computing environment) having fixed or scalable/elastic computing resources142(e.g., data processing hardware) and/or storage resources144(e.g., memory hardware). Data sources135may be overlain on the storage resources144by one or more of the users (e.g., the user device110) or the computing resources142. The data source135is configured to store one or more datasets130.

The processing system140is configured to receive a reconciliation request (e.g., data reconciliation request)102from the user device110associated with a respective user10. The user device110may correspond to any computing device, such as a desktop workstation, a laptop workstation, or a mobile device (i.e., a smart phone). The user device110includes computing resources112(e.g., data processing hardware), storage resources114(e.g., memory hardware), and/or a display116(e.g., graphical user interface (GUI)). In some examples, the reconciliation request102includes a storage location of a first dataset130,130astored at a first data source135,135aand a storage location of a second dataset130,130bstored at a second data source135,135b. Thus the reconciliation request requests data reconciliation for the first dataset130aand the second dataset130b. In other examples, the user10provides, via the user device110, the first and second datasets130a,130bdirectly to the data reconciliation controller200as part of the reconciliation request102.

The processing system140executes a data reconciliation controller200configured to reconcile the first and second datasets130a,130bbased on (i.e., in response to) receiving the reconciliation request102. Each data source135may be a relevant application (i.e., “app”), table, database, or any other source of the datasets130for reconciliation. As such, data source135may also interchangeably be referred to as “application135or app135” herein. In some examples, the datasets130represent financial information (e.g., financial account balances, debts, credits, etc.) or other numerical data (e.g., inventory counts). In yet other examples, the datasets130include string values (e.g., employee names, locations, descriptions, etc.). In some examples, each dataset130includes a number of dimensions132. Each dimension132may correspond with a column of data of the dataset130. For instance, a dimension132may correspond to a column of data representing “Accounts” or “Year” data of the dataset130. In some implementations, the data reconciliation controller200obtains or receives dimension alignment data. The dimension alignment data aligns the dimensions132of each dataset130, for example, by bridge members136. For instance, a first dimension132labeled as “Account” for a first dataset130may align with a second dimension132labeled as “Acct” for a second dataset130, as discussed in more detail below.

Moreover, each dimension132includes one or more members134(e.g., dimension members) of the dimension132. For example, the dimension “Account” may include a number of members (i.e., dimension members)134of different individuals or entities that have accounts for a server associated with the dataset130, data source135, or application. Here, each dimension member134represents a respective data value for the corresponding dimension132. Along with the dimensions132of each dataset130, the data reconciliation controller200obtains each dimension member134of the datasets130for reconciliation. Each dimension member134may correspond to a base member and/or a parent member. In some examples, one or more dimension members134include or are associated with one or more comments, annotations, and/or descriptions. For example, a user may provide a comment providing a brief description of one or more dimension members134. The comments, for example, include details for auditing notes, cube or system definitions, to notate responsible parties, incorporate different codes to assist with different ways of filtering data, member exceptions to one or more general rules, etc.

FIGS.2-10illustrate various flowcharts of exemplary arrangements of operations and various GUI views of the data reconciliation controller200performing data reconciliation. It is understood that the flowcharts illustrated inFIGS.2-10are exemplary only and are not meant to limit this disclosure in any way. For example, the flowcharts may repeat any of the operations any number of times, include more or fewer operations, and/or perform any of the operations in a different order without departing from the scope of this disclosure. Moreover, the various GUI views may be displayed on the GUI116of the user device110whereby the user is able to provide user input to modify the GUI view and/or data.

Referring now toFIG.2, the data reconciliation controller200includes a user login operation210, a reconciliation label (i.e., naming) module300, a dimension linking module400, an import module500, a bridging module600, a bridge execution module700, a transformation identifier800, a transformation execution module900, and a report execution module1000. The data reconciliation controller200is configured to reconcile one or more datasets130in response to receiving the reconciliation request102. Each dataset130is stored at a respective data source135, thus reconciliation of the datasets130may also by referred to as reconciliation of the data sources135(e.g., the datasets130stored at the respective data sources135).

In some implementations, the reconciliation request102may be manually sent by the user10via the user device110(FIG.1). In other implementations the reconciliation controller200generates the reconciliation request, thereby initiating the reconciliation process, based on detecting a trigger event. For example, the trigger event may be a timer such that reconciliation occurs at a fixed interval or a data modification event indicating a data update in one or more of the datasets130. The user may configure the trigger event to be any trigger such that the data reconciliation controller200performs reconciliation in response to detecting the trigger event without any direct user interaction.

Accordingly, the data reconciliation controller200obtains the datasets130(e.g., from respective data sources135) in response to receiving the reconciliation request102. For example, the reconciliation request102may provide a user name, password, or any other credentials necessary to access the datasets130from respective data sources135. In some examples, the data sources135are applications, and the data reconciliation controller200obtains a user name, a password, and a connection name for each application (e.g., from a user of the processing system140or a remote device in communication with the processing system140). Here, the data reconciliation controller200authenticates the obtained user name and password for the respective applications135before retrieving datasets130from the data sources135. The user10may manually provide some or all of the datasets130to the data reconciliation controller200via the GUI116(FIG.1) of the user device110another interface executing on the user device110in communication with the processing system140or executing on the processing system140itself.

Users may have read access or read/write access to the datasets130. For instance, a read access user may only read the datasets130and perform reconciliation while a user with read/write access (e.g., an administrator) may update data included in the datasets130. Accordingly, the data reconciliation controller200may grant or deny a data reconciliation request102based on whether the respective user has read access, read/write access, or no access to the requested datasets for reconciliation (i.e., neither read access nor read/write access).

Referring now toFIGS.3A and3B, the reconciliation label module300is configured to select data sources135for reconciliation. That is, an example reconciliation label module300(FIG.3A) first determines, at operation310, whether the first data source135aor the second data source135bcorresponds to a new data source based on the reconciliation request102. Based on determining that neither the first nor second data sources135a,135bcorrespond to a new data source, the reconciliation label module300selects the data sources (e.g., applications)135to reconcile and selects versions of the selected data sources135for reconciliation at operations320and330, respectively. For example, the reconciliation label module300may select the first data source135aand the second data source135b(and the corresponding data source135versions) based on user input included in the reconciliation request102(FIG.1).

On the other hand, based on determining that at least one (or both) of the first or second data sources135a,135bcorresponds to a new data source, the reconciliation label module300creates data source labels and reconciliation execution labels at operations340and350, respectively. Here, the user10may include the data source labels and the reconciliation execution labels as part of the reconciliation request102(FIG.1) to the data reconciliation controller200. For instance, the user10may specify that a new data source label is “app1” while the reconciliation execution label is “abc2.” After selecting the data sources135for reconciliation, the reconciliation label module300provides the reconciliation execution label including the data sources135and their corresponding versions to the dimension linking module400.

FIG.3Billustrates an example GUI view301depicting a graphical representation of the reconciliation label module300. In the example shown, the example GUI view300bdepicts a reconciliation execution label (e.g., run name)362for each row of the table. Each reconciliation execution label362, includes a description364(e.g., demo reconciliation run), and a corresponding status366. The status366may indicate which step of the reconciliation process the respective reconciliation is currently at. For instance, the status366may indicate a reporting, dimension update, or bridged status. Moreover, the example GUI view301may display metadata including, but not limited to, creation data368indicating a creating user (e.g., administrator or user) and a creation timestamp, modification data indicating a modifying user (e.g., administrator or user) and a modification timestamp, and an execution timestamp.

Referring now toFIGS.4A-4C, the dimension linking module400is configured to align dimensions132from the data sources135. An example dimension linking module400(FIG.4A) selects, at operation410, the reconciliation run for execution. That is, the dimensions linking module400selects the reconciliation execution label362created by the reconciliation label module300(FIG.3). At operation420, the dimension linking module400sets a variance threshold422. For example, the data reconciliation controller200may receive one or more variance thresholds422as part of the reconciliation request102. The variance threshold422determines an amount of margin available when performing data reconciliation.

For instance, when the variance threshold422is 0.05, the data reconciliation controller200may determine that values within 0.05 of each other are the same or values within 0.05 of zero are treated as a zero value (i.e., variance values under the variance threshold are converted to zeros and do not show any variance within the reconciliation). Simply put, the data reconciliation controller200determines that when a difference in values from the first dataset130aand the second dataset130bare within the variance threshold422no reconciliation of the values is needed. A user may configure the variance threshold422to be any value and, unless otherwise configured, the data reconciliation controller200may implement a default variance threshold. In some examples, the data reconciliation controller200determines the variance threshold422based on the data sources135, the datasets130, and/or other contextual information.

At operation430, the dimension linking module430selects an execution delimiter432for the first data source135aand the second data source135b. The execution delimiter432may be a semi-colon, colon, hash, or any other character. The execution delimiter432of the first and second data sources135a,135bmay be a same execution delimiter or a different execution delimiter (e.g., first execution delimiter value and second execution delimiter value). In some examples, the data reconciliation controller200receives or obtains the execution delimiter432from the data reconciliation request102. In other examples, the data reconciliation controller200generates the execution delimiter432based on the data sources135.

The execution delimiter (i.e., “line run rate”) determines a maximum number of combinations (e.g., rows or lines of a table) that are processed or refreshed simultaneously. Stated differently, the dimension linking module400refreshes the dimension members134(e.g., first set of combination dimension members) of the first dataset130aand the dimension members134(e.g., second set of combination dimension members) based on the execution delimiter value. Here, dimension linking module400refreshes the dimension members by obtaining an updated dimension member (e.g., updated data value) from the plurality of dimensions members134of the first dataset130afrom the first data source135aand obtains an updated dimension member from the plurality of dimension members134of the second dataset130bfrom the second data source135b. Although, in some scenarios, no data updates have occurred in the first and second datasets130a,130band the dimension linking module400does perform refreshing because there is no new data to update. As such, the use of dimension members134(or set of combination dimension members) herein may either refer to refreshed dimension members or non-refreshed dimension members134. In some instances, the dimension linking module400refreshes the first set of combination dimension members based on a first execution delimiter value and refreshes the second set of combination dimension members based on a second execution delimiter value different than the first execution delimiter value. In some scenarios, processing or refreshing all combinations simultaneously (e.g., thousands or millions of combinations or rows) causes failures (e.g., crashes, freezes, etc.) or other suboptimal behavior.

The execution delimiters432may include corresponding locations in the dataset130that “split” processing or refreshing of the combinations or rows at the execution delimiter locations into “batches,” thereby assuring stable execution. A user may configure the execution delimiter432to have any location such that the user may control a batch size for execution. Unless otherwise configured, the data reconciliation controller200may implement a default execution delimiter, for example, to have a default batch size. In some implementations, the data reconciliation controller200determines the execution delimiter432based on the data sources135, the datasets130, or other contextual information (e.g., available computational resources). Described in greater detail with reference toFIGS.4B-4D, the dimension linking module400aligns, at operation440, the dimensions132of the first and second data sources135and generates (i.e., creates) additional dimensions at operation450. Thereafter, the dimension linking module400, at operation460, saves and outputs the aligned first and second data sources135a,135bto the import module500.

FIGS.4B-4Dillustrate various example GUI views401each depicting a graphical representation of the dimension linking module400. As shown in example GUI view401,401a, the user may select the reconciliation execution label362for alignment and select a dataset130import type. For instance, the first and second datasets130a,130bmay be imported from their respective data sources135by file upload, direct connection, by user input from the reconciliation request, or some combination thereof. Continuing with the example, the user may also select, for each data source135(e.g., app1 and app2), the currency delimiter432, a currency symbol434for removal, and/or a removal currency delimiter436that the dimension linking module400removes from the data source135. In some implementations, the example GUI view401aalso allows the user to set the variance threshold422and indicate whether each data source135has a header438or no header.

Referring now toFIGS.4C and4D, in some implementations the dimension linking module400displays (e.g., via GUI116) alignments between data sources135. For instance, example GUI view401,401bshows alignment data for the first data source (e.g., app1)135aand example GUI view401,401cshows alignment data for the second data source (e.g., app2)135b. The GUI views401b,401cshow the alignment data separately, it is understood that the alignment data may be shown adjacently. That is, the GUI views401b,401cmay be shown side-by-side for simple comparison between the data sources135.

The alignment data shown by the dimension linking module400includes the data source135(e.g., app1 or app2), dimensions132, and an indicator442(e.g., yes or no) representing whether the respective dimension132is included in the respective data source135. In the example shown, the dimension132corresponding to “Year” is not included in the first and second data sources135a,135b, and thus, a top member446of the “Year” dimension is shown as “2022.” As such, the dimension member134for the dimension132of “Year” includes only the value “2022” in this example. Moreover, the dimension132corresponding to “Amount” is included in the first and second data sources135a,135b, and thus, a field444of ‘4’ is shown. The field444may configure a display position of the respective dimension132for reconciliation. That is, the field444may align the “Amount” dimension132between the data sources135. In some implementations, the user manually aligns and populates the alignment data or imports additional dimensions132. In other implementations, the dimension linking module400aligns and populates the alignment data based on the data sources135without any user input.

FIGS.5A and5Billustrate the import module500configured to bridge alignment data from the data sources135. In particular, an example import module500(FIG.5A) receives the data sources135(e.g., aligned data sources) from the dimension linking module400and generates, at operation510, a corresponding bridge member136for each dimension132of the respective dataset130. Bridge members136link the datasets130together for reconciliation purposes. In some implementations, the data reconciliation controller200generates a new data structure or table (FIG.5B) that includes a first set of combination dimension members of the first dataset130aand a second set of combination dimension members of the second dataset130b.

That is, for each dataset130, the data reconciliation controller200generates every combination (i.e., pair) of input dimension members134. For example, when a dataset130includes five dimensions132, with a first dimension132having 40 dimension members134, a second dimension132having 35 dimension members134, a third dimension132having 3 dimension members134, a fourth dimension132having 10 dimension members134, and a fifth dimension132having 32 dimension members134, the data reconciliation controller200generates 1,344,000 combinations for the dataset130(i.e., 40*35*3*10*32=1,344,000). Accordingly, the data reconciliation controller200generates a first set of combination dimension members (e.g., dimension members)134using each pair of dimension members134of the one or more dimensions132of the first dataset130aand generates a second set of combination dimension members (e.g., dimension members)134using each pair of dimension members134of the one or more dimensions of the second dataset130b. Thereafter, the data reconciliation controller200adds the bridge members136in connection with each combination dimension member134of the new data structure. That is, the data reconciliation controller200may place the respective bridge member136in association with each corresponding combination dimension member134of the new data structure.

After generating the combination dimension members and generating the data structures including the combinations and bridge members, the data reconciliation controller200refreshes or updates each set of combinations at a rate governed by the execution delimiter. The data reconciliation controller200repeats this process for each dataset130from each data source135(i.e., each application) and continues to split up refreshes (based on the execution delimiter) for larger datasets130. After the data has been refreshed at least once, the data reconciliation controller200may determine whether any of the data in the datasets130have changed prior to re-refreshing or updating the data. That is, the data reconciliation controller200, in some examples, only updates or refreshes data when a change in the data is detected (i.e., a change in the datasets130).

In some implementations, the data reconciliation controller200generates a suppressed values data structure (e.g., a table) that includes two or more tables that include all the data without suppressed data such as cells (i.e., column and row combinations) that are zero and/or no data (i.e., unadjusted suppressed values). In some examples, cells zeroed by the variance threshold are removed. As discussed above, the data reconciliation controller200may only generate the suppressed values data structure when a change has been detected in the data since the last execution of the data reconciliation controller200.

In some implementations, after generation of the suppressed values data structure, the data reconciliation controller200combines the two tables of the suppressed values data structure into a linear format. Next, the data reconciliation controller200may combine the adjustment data into a linear format. Next, the data reconciliation controller200or tool may combine some or all of the previously generated tables (i.e., the tables of the combinations and bridge members and the adjusted data tables into an all data table (i.e., in a linear format). The data reconciliation controller200, when generating the all data table, may inverse each quantity for the combinations from the second dataset200b(e.g., the value 100 becomes −100 and the value 14 becomes −14, etc.).

In some examples, the bridge members link dimensions132together in a one-to-one manner. For instance, when the first data source135ahas a dimension132titled “Accounts” and the second data source135bhas a dimension132titled “Accts” that maps to the dimension members134(e.g., data) in the Accounts dimension132of the first data source135a, the bridge member between Accounts and Accts is a one-to-one relationship. In other examples, the bridge members link dimensions132together in a many-to-one, a one-to-many, and/or a many-to-many relationship for the data sources135. For example, when the first data source135ahas a dimension titled “Accounts” and the second data source122bhas two dimensions titled “Accts1” and “Accts2” that splits the dimension members134into two separate dimensions132, the bridge member may bridge “Accounts” to both “Accts1” and “Accts2,” forming a one-to-many relationship between the first data source135aand the second data source135b. As such, the dimensions132may be divided in any number of ways between the data sources135, and the bridge members may be used to properly map the dimension members134from the first data source135ato the second data source135b.

In some scenarios, the import module500previously generated the bridge members136for a respective data source135whereby the import module500, at operation520, imports the previously generated bridge members136rather than generating the bridge members again. At operation530, the user may manually edit any of the bridge members136generated by the import module500. Moreover, the user may add comments, at operation540, for dimensions132or bridge members136such as an exception to one or more general rules.

FIG.5Billustrates an example GUI view501depicting a graphical representation displayed by the import module500. As shown, the graphical representation depicts the plurality of bridge members136,136a—e generated by the import module500for the first and second data sources135a,135b. Here, each bridge member136is in association with the first combination of dimension member134and/or the second combination dimension member134. In particular, the example shows a first bridge member136abetween the first and second application135a,135band a second and third bridge member136b,136cbetween the “Year” and “Period” dimensions132of the data sources135. Moreover, the example shows a fourth bridge member136dbetween an “Account” dimension132of the first data source135athat corresponds to an “Acct” dimension of the second data source135band a fifth bridge member136ebetween an “Entity” dimension132of the first data source135athat corresponds to an “Ent” dimension132of the second data source135b. Below each bridge member136is a combination dimension member134corresponding to the bridge member136.

Referring now toFIGS.6A-6C, in some implementations, the bridging module600is configured to synchronize source members and bridge members136. An example bridging module600(FIG.6A) generates, at operation610, a concatenation612of the bridging member136and a dimension member134(e.g., one of the combination dimension members) for each respective dimension132. As such, the bridging module600may import the bridge members136generated by the import module500(FIG.5) for each data source135and generate the concatenation612using the bridging members136and the corresponding dimension members134. Simply put, the bridging members136link dimensions between the data sources135and the concatenation612associates dimension members134(e.g., data values) of the dimension132for bridging member136. After, generating the concatenations, the user may edit/configure any of the generated concatenations. Using the final concatenations612, the bridging module600, at operation620, generates a synchronization map622that maps the source members (e.g., dimension members134) to the bridge members136and submits the synchronization maps622at operation630.

FIGS.6B and6C, illustrate the bridging module600displaying (e.g., via GUI116) a representation corresponding to the synchronization map622. For instance, example GUI view601,601ashows an example synchronization map for the first data source (e.g., app1)135aand example GUI view601,601bshows an example synchronization map for the second data source (e.g., app2)135b. The synchronization map includes the dimensions132, dimension members134, an inverse operation indicator602, and bridge members136. Here, each row corresponds to a concatenation612generated by the bridging module600. For example, the synchronization map622for the first data source135a(FIG.6B) shows the “Period” dimension132having a dimension member134(e.g., data value) of “600” and a bridge member136of “EXP” with the inverse operation indicator602indicating “NO.” Thus, the data reconciliation controller200will not perform an inverse operation for the “Period” dimension based on the inverse operation indicator602. Alternatively, in a scenario where the inverse operation indicator602indicates “YES,” the data reconciliation controller200performs the inverse operation for the “Period” dimension.

FIGS.7A and7Billustrate the bridge execution module700configured to generate a source file table722displaying the dimensions132of the first and second data sources135a,135bas headers with corresponding values below. In particular, an example bridge execution module700(FIG.7A) imports, at operation710, the first and second datasets130a,130bfrom the first and second data sources135a,135b, respectively. Each dataset130includes one or more dimensions132and each dimension132includes a plurality of dimension members134(e.g., data values). At operation720, the bridge execution module700generates the source file table722using the first and second datasets130a,130band the data synchronization maps622. Thereafter, the bridge execution module700exports the source file table722at operation730.

When the data reconciliation controller200imports the datasets130the data reconciliation controller200generates a bridge member136for each dimension member (e.g., for every dimension member in the combination of dimension members). However, if the bridge execution module700detects that one or more of the dimension members134do not include a bridge member136, a bridge kickout occurs indicating the missing bridge member. Responsive to the bridge kickout, the data reconciliation controller200and/or the user may update the bridge members (e.g., via the bridging module600).

FIG.7Billustrates an example GUI view701depicting a graphical representation of the source file table722generated by the bridge execution module700. In the example shown, the source file table722includes the data sources135(e.g., app1 and app2), the “Year” dimension132next to the bridging member136for the “Year” dimension132, and the “Account” or “Acct” dimension next to the bridging member for the account dimension132.

Referring now toFIGS.8A and8B, in some examples, the transformation identifier800is configured to identify required transformations for the source file table722(e.g., generated by the bridge execution module700) before reconciliation execution. In some implementations, one or more of the data sources135(e.g., the second data source135b) have a dimension132that does not directly map (i.e., via the bridge members136) to any corresponding dimension132of the other data source135(e.g., the first data source135a). For example, in some scenarios the second data source135bperforms additional processing on dimension members134of a dimension132from the first data source135athereby generating a new dimension132that is not present in the first data source135a. In this scenario, there is no direct mapping (e.g., no bridge member136) for the new dimension132of the second data source135bbecause the first data source135ahas no corresponding dimension132. Thus, the reconciliation controller200may generate an additional table to concatenate any number of columns to perform additional multi-dimensional mapping in situations where a dimension132does not exist in one data source135but is present in the other data source135.

In the example shown, at operation810, the transformation identifier800executes the source file table722to generate the bridge table. The bridge table identifies any transformations and displays the executed source file table at operation820. Here, identified transformations may indicate that one or more of the dimensions132and/or the bridge members136are unmapped. At operation830, the transformation identifier800determines whether the source file table has any unmapped dimensions132or missing dimensions. Based on determining that there are not any unmapped dimensions132(or bridge members136), the transformation identifier800submits and exports the table at operations840and850, respectively. Here, data reconciliation controller200may input a field number from the source file and enable the status as active.

On the other hand, based on determining that there are one or more unmapped dimensions132or missing dimensions132, the transformation identifier800updates the one or more unmapped dimensions at operation860. For instance, the data reconciliation controller200may prompt the user to manually update the unmapped dimensions. Alternatively, the data reconciliation controller200may update the unmapped dimensions or missing dimension directly, for example, entering a place holder value for the missing dimension. Moreover, if there is a missing source member the data reconciliation controller may enter a top member value for the missing value (e.g., 2022 for missing dimension132of “Year). After correcting the unmapped dimensions, transformation identifier800submits and exports the bridge table at operations840and850.

FIG.8Billustrates an example GUI view801depicting a graphical representation of the executed source file table generated by the transformation identifier800. Here, the executed source file table shows the dimensions132and concatenations612for each of the first and second data sources135a,135b. In the example shown, each dimension132of the data sources135is mapped to a corresponding dimension in the other of the data sources135. That is, in this example there are no unmapped dimensions132. However, if there were any unmapped dimensions132the executed source file table would display the unmapped dimensions132for correction. In particular, the data reconciliation controller200may re-execute using updated data to map the unmapped dimension132or the user may manually map the unmapped dimension132.

FIGS.9A and9Billustrate the transformation execution module900configured to generate a bridge synchronized table922displaying the dimensions132of the first and second data sources135a,135bas headers with corresponding values below. In particular, an example transformation execution module900(FIG.9A) receives the bridge table generated by the transformation identifier800at operation910. Using the bridge table, the transformation execution module900generates the bridge synchronized table922at operation920and exports the bridge synchronized table at operation930. Notably, operation920is similar to operation720(FIG.7) with the only difference being that, at operation920, the transformation execution module900uses the bridge table with the transformations. Simply put, operation720(FIG.7) executes before any transformations (e.g., unmapped dimensions) were corrected and operation920executes using the bridge table that includes any updates to correct the unmapped dimensions.

FIG.9Billustrates an example GUI view901depicting a graphical representation of the bridge synchronization table922generated by the bridge execution module transformation execution module900. In the example shown, the bridge synchronization table922includes the data sources135(e.g., app1 and app2), the “Year” dimension132next to the bridging member136for the “Year” dimension132, and the “Account” or “Acct” dimension next to the bridging member for the account dimension132. Notably, in this example there were no transformations (e.g., fromFIG.8B), and thus, the bridge synchronization table922is similar (or identical) to the source file table722shown inFIG.7B.

Referring now toFIGS.10A and10B, in some implementations, the report execution module1000is configured to a reconciliation report160. That is, using the all data table (e.g., bridge synchronization table922), the data reconciliation controller200generates one or more reconciliation reports160, such as a reconciliation bridge report. The report visually provides the reconciliation results to, for example, a user. For example, the report highlights or annotates data discrepancies between the first dataset130afrom the first data source135aand the second dataset130bfrom the second dataset130b. In some implementations, the data reconciliation controller200generates additional dimensional conversions (i.e., the data reconciliation controller200generates another opportunity to change or adjust bridge account data based on concatenation of existing dimensions to allow for reconciliations where a single dimension in one application gets split out into multiple dimensions in another application).

In particular, the report execution module1000generates the reconciliation report160at operation1010. Optionally, the user may edit and/or submit commits for the reconciliation report160at operations1020and1030, respectively. Thereafter, the report execution module1000exports the reconciliation report160at operation1040. For instance, the report execution module1000may export the reconciliation report160to the user device110associated with the reconciliation request102that initiated the reconciliation of the datasets130. In response to receiving the reconciliation report160, the user device110may display (e.g., via the GUI116) the reconciliation report160to the user.

FIG.10Billustrates an example GUI view1001depicting a graphical representation of the reconciliation report106generated by the report execution module1000. In the example shown, the reconciliation report160includes one or more bridge members136, comments1004, and a reconciliation value1002. In particular, the reconciliation report shows a first bridge member136between the “Account” and “Acct” dimension132of the first and second data sources135a,135band a second bridge member between the “Entity” and “Ent” dimension132of the first and second data sources135a,135b. Any comments1004input by the user during the reconciliation process are displayed in connection with the reconciliation report to notify the user of any special circumstances with a particular reconciliation occurrence. The reconciliation values1002graphically display to the user whether any reconciliation between the datasets130needs to occur. Simply put, the reconciliation values1002indicate whether any data is different, missing, etc. between the datasets130from the data sources135.

In some implementations, after generation of the suppressed values data structure, the data reconciliation controller200combines the two tables of the suppressed values data structure into a linear format. Next, the data reconciliation controller200may combine the adjustment data into a linear format. Next, the data reconciliation controller200may combine some or all of the previously generated tables (i.e., the tables of the combinations and bridge members and the adjusted data tables into an all data table (i.e., in a linear format). The data reconciliation controller200, when generating the all data table, may inverse each quantity for the combinations from the second dataset200b(e.g., the value 100 becomes −100 and the value 14 becomes −14, etc.).

FIG.11is a flowchart of an exemplary arrangement of operations for a method1100of performing data reconciliation. The method1100may be performed, for example, by the data reconciliation controller200operating at the processing system140. At operation1102, the method1100includes obtaining a first dataset130afrom a first data source135a. The first dataset130aincludes one or more dimensions132each dimension132having a plurality of dimension members134(e.g., data values). At operation1104, the method1100includes obtaining a second dataset130bfrom a second data source135b. Similarly, the second dataset130balso includes one or more dimensions132each dimension132having a plurality of dimension members134. Obtaining the first and second datasets130a,130bmay be in response to receiving, from a user device110in communication with the data processing hardware142(e.g., in communication via the network120), a data reconciliation request102requesting data reconciliation for a first dataset130aand a second dataset130b. The data reconciliation request102may be sent by a user10associated with the user device110or in response to a trigger event.

For each respective dimension132of the first dataset130a, the method1100, at operation1106, includes obtaining and/or generating a respective bridge member136. The respective bridge member136associates the respective dimension132of the first dataset130awith a respective dimension132of the second dataset130b. For instance, the a respective bridge member136may associate a dimension132of “Entity” of the first dataset130awith a corresponding dimension132of “Ent” of the second dataset. The bridge member136between these two dimensions132indicates to the data reconciliation controller200to perform reconciliation between the data value (e.g., dimension members134) of these two dimensions132. As such, each dimension member134included in these two dimensions132may be associated with the respective bridge member136.

At operation1108, the method1100includes generating a first set of combination dimension members134using each pair of dimension members134of the one or more dimensions132of the first dataset130aand the respective bridge members136. Similarly, at operation1110, the method1100includes generating a second set of combination dimension members134using each pair of dimension members134of the one or more dimensions132of the second dataset130band the respective bridge members136. For example, generating a set of combination dimension members134for a dataset130that includes five dimensions132, with a first dimension having 40 members, a second dimension having 35 members, a third dimension having 3 members, a fourth dimension having 10 members, and a fifth dimension having 32 members, the data reconciliation controller200generates 1,344,000 combinations (e.g., lines) for the dataset130(i.e., 40*35*3*10*32=1,344,000). Simply put, the set of combination dimension members134represents every combination of dimension members134from the dimensions132of the dataset130.

Optionally, the method1100may further include refreshing the first set of combination dimension members and the second set of combination dimension members based on an execution delimiter value corresponding to a quantity of dimension members refreshed simultaneously Advantageously, the execution delimiter value splits the rows into “batches” thereby assuring stable execution. Thus, when the method1100refreshes the first and second combination dimension members the method uses the refreshed first and second combination dimension members. On the other hand, when no refreshing occurs the method simply uses the first and second combination dimension members.

At operation1112, the method1100includes generating a third set of combination dimension members using the first set of combination dimension members134and the second set of combination dimension members134. Here, the third set of combination dimension members correlates the dimension members134between the first and second datasets130a,130b. For instance, the third set of combination dimension members may correspond to the all data table (e.g., either the source file table722(FIG.7) or bridge synchronized table922(FIG.9)). Thereafter, at operation1114, the method1100includes generating a data reconciliation report160from the third set of combination dimension members (e.g., the source file table722(FIG.7) or the bridge synchronized table922(FIG.9)). For example, when no transformations of the datasets130are required, the data reconciliation controller200generates the data reconciliation report160using the source file table722. Otherwise, where transformations are required for the datasets130, the data reconciliation controller200generates the reconciliation report using the bridge synchronized table922. Notably, the reconciliation report reconciles data between the first and second datasets130a,130bwhereby the data reconciliation controller200detects whether any data has been modified, added, or deleted between the first and second datasets130a,130b.

The computing device1200includes a processor1210, memory1220, a storage device1230, a high-speed interface/controller1240connecting to the memory1220and high-speed expansion ports1250, and a low speed interface/controller1260connecting to a low speed bus1270and a storage device1230. Each of the components1210,1220,1230,1240,1250, and1260, are interconnected using various busses, and may be mounted on a common motherboard or in other manners as appropriate. The processor1210can process instructions for execution within the computing device1200, including instructions stored in the memory1220or on the storage device1230to display graphical information for a graphical user interface (GUI) on an external input/output device, such as display1280coupled to high speed interface1240. In other implementations, multiple processors and/or multiple buses may be used, as appropriate, along with multiple memories and types of memory. Also, multiple computing devices1200may be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a multi-processor system).

The storage device1230is capable of providing mass storage for the computing device1200. In some implementations, the storage device1230is a computer-readable medium. In various different implementations, the storage device1230may be a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid state memory device, or an array of devices, including devices in a storage area network or other configurations. In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as the memory1220, the storage device1230, or memory on processor1210.

The high speed controller1240manages bandwidth-intensive operations for the computing device1200, while the low speed controller1260manages lower bandwidth-intensive operations. Such allocation of duties is exemplary only. In some implementations, the high-speed controller1240is coupled to the memory1220, the display1280(e.g., through a graphics processor or accelerator), and to the high-speed expansion ports1250, which may accept various expansion cards (not shown). In some implementations, the low-speed controller1260is coupled to the storage device1230and a low-speed expansion port1290. The low-speed expansion port1290, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), may be coupled to one or more input/output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g., through a network adapter.

The computing device1200may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a standard server1200aor multiple times in a group of such servers1200a, as a laptop computer1200b, or as part of a rack server system1200c.