Identifying entity mappings across data assets

Entity mappings that produce matching entities for a first data asset having attributes and a second data asset having attributes are generated by: generating entity mappings that produce matching entities for a first data asset having attributes with attribute values and a second data asset having attributes with attribute values by: matching the attribute values of the attributes of the first data asset with the attribute values of the attributes of the second data asset, using the matching attribute values to generate matching attribute pairs, and using the matching attribute pairs to identify entity mappings; computing an entity mapping score for each of the entity mappings based on a combination of factors; ranking the entity mappings based on each entity mapping score; and using some of the ranked entity mappings to determine whether a same real-world entity is described by the first data asset and the second data asset.

FIELD

Embodiments of the invention relate to identifying entity mappings across data assets.

BACKGROUND

Big data refers to very large data sets. In big data paradigms, there may be a need to integrate various data assets. The data assets may be structured, semi-structured, or unstructured. A structured data asset may be described as a set of attributes and corresponding values. This integration may be done using join, merge or union operations. The data sets may be stored in tables having rows and columns (“attributes”) in a Relational Database Management System (RDBMS). For performing these operations across data assets, entity mappings between them need to be obtained. The entity mappings describe the column values that should be compared to know whether the same real-world entity is described in the two data assets. Currently, such column mapping is done manually, which is not suitable for data discovery in big data paradigms.

Some systems identify foreign keys in relational tables. The primary key (PK) to foreign key (FK) relationship may involve a single column or multiple columns. These systems assume that at-least one key is a primary key and that the relationship has to be one-to-one (e.g., 100%) (i.e., each foreign key should be a primary key of the other data asset).

Some systems estimate individual column mappings using semantic similarities. However, these systems are not based on joins between the two datasets.

Some systems identify attribute pairs that may be used to link two tables, but such systems discover only single attribute mappings.

SUMMARY

Provided is a method for identifying entity mappings across data assets. The method comprises generating entity mappings that produce matching entities for a first data asset having attributes and a second data asset having attributes by: generating entity mappings that produce matching entities for a first data asset having attributes with attribute values and a second data asset having attributes with attribute values by: matching the attribute values of the attributes of the first data asset with the attribute values of the attributes of the second data asset, using the matching attribute values to generate matching attribute pairs, and using the matching attribute pairs to identify entity mappings; computing an entity mapping score for each of the entity mappings based on a combination of factors; ranking the entity mappings based on each entity mapping score; and using the ranked entity mappings to determine which of the entity mappings are to be used to determine whether a same real-world entity is described by the first data asset and the second data asset.

Provided is a computer program product for identifying entity mappings across data assets. The computer program product comprises a computer readable storage medium having program code embodied therewith, the program code executable by at least one processor to perform: generating entity mappings that produce matching entities for a first data asset having attributes and a second data asset having attributes by: generating entity mappings that produce matching entities for a first data asset having attributes with attribute values and a second data asset having attributes with attribute values by: matching the attribute values of the attributes of the first data asset with the attribute values of the attributes of the second data asset, using the matching attribute values to generate matching attribute pairs, and using the matching attribute pairs to identify entity mappings; computing an entity mapping score for each of the entity mappings based on a combination of factors; ranking the entity mappings based on each entity mapping score; and using the ranked entity mappings to determine which of the entity mappings are to be used to determine whether a same real-world entity is described by the first data asset and the second data asset.

Provided is a computer system for identifying entity mappings across data assets. The computer system comprises: one or more processors, one or more computer-readable memories and one or more computer-readable, tangible storage devices; and program instructions, stored on at least one of the one or more computer-readable, tangible storage devices for execution by at least one of the one or more processors via at least one of the one or more memories, to perform: generating entity mappings that produce matching entities for a first data asset having attributes and a second data asset having attributes by: generating entity mappings that produce matching entities for a first data asset having attributes with attribute values and a second data asset having attributes with attribute values by: matching the attribute values of the attributes of the first data asset with the attribute values of the attributes of the second data asset, using the matching attribute values to generate matching attribute pairs, and using the matching attribute pairs to identify entity mappings; computing an entity mapping score for each of the entity mappings based on a combination of factors; ranking the entity mappings based on each entity mapping score; and using the ranked entity mappings to determine which of the entity mappings are to be used to determine whether a same real-world entity is described by the first data asset and the second data asset.

Embodiments advantageously generate a first inverted index of entity identifier pairs for the first data asset; generate a second inverted index of entity identifier pairs for the second data asset; and use the first inverted index and the second inverted index to generate the matching attribute pairs based on matching attribute values that form the entity mappings.

Embodiments do not assume that at least one key is a primary key and that the relationship between foreign key and primary key between data assets is one-to-one. Moreover, unlike some prior art systems, embodiments advantageously discover multiple attribute mappings.

Moreover, embodiments allow for matching values fuzzily for the matching entities. This advantageously avoids a need for exact matching.

Also, embodiments compute the entity mapping score for each of the entity mappings by generating an entity mapping score for factors selected from: a number of attributes involved in an entity mapping, a cardinality of that individual entity mapping, support of that entity mapping, a probability of one to one matching for that entity mapping, a join utility measure for that entity mapping, and a probability of previous user selections for that entity mapping, and adding the entity mapping score for each of the factors to generate the entity mapping score for that entity mapping. Thus, embodiments advantageously take many factors into consideration.

In addition, embodiments allow for one of the first data asset and the second data asset to be semi-structured data having hierarchical data that is flattened. Embodiments also allow for one of the first data asset and the second data asset to be an unstructured data asset formed by a collection of documents and is modelled based one of a bag of words and annotated words. Thus, embodiments advantageously allow for data assets to be structured, semi-structured, and/or unstructured.

Furthermore, embodiments integrate the first data asset and the second data asset using ranked entity mappings by performing one of a join operation, a merge operation, and a union operation. That is, unlike some prior art systems, embodiments advantageously use join, merge, and union operations.

DETAILED DESCRIPTION

FIG. 1illustrates, in a block diagram, a computing environment in accordance with certain embodiments. InFIG. 1, a computing device100is coupled to data asset A150a, data asset b150b, . . . and data asset N150n. The ellipses are used to indicate that any number of data assets may be coupled to the computing device100. The data assets150a. . .150nmay store big data. The computing device100includes an entity mapping identifier110. The entity mapping identifier110discovers entity mappings between data assets. The entity mapping identifier110covers different data types and provides ranking for entity mappings.

Linking of multiple disparate data assets may be advantageously used for: identifying related data assets, provisioning of data for analytics, enriching data assets, and entity resolution. The data assets may be integrated using operations, such as join, merge or union. The entity mapping identifier110obtains entity mappings (also referred to as schema mappings) that are used for integrating the data assets.

An entity mapping may be described as a set of attributes that define a real-world object. A real-world object may be, for example, a person, a building, a location, a car, etc. That is, an entity mapping for a data asset A with columns a1, a2, . . . an, with regard to data asset B with columns b1, b2, . . . bm is a set of column mappings {ai:bj|ai⊂(a1,a2, . . . an), bj⊂(b1,b2, . . . bm)} and matches at least λ rows of A. With embodiments, the entity mapping is directional since the number of matching rows of A to B may be different from the number of matching rows of B to A. Entity resolution may be described as a process used to find whether two sets of attributes are describing the same real-world object. To know whether two data assets are related, entity resolution finds whether two data assets are describing the same or similar sets of real-world objects. For example, a data asset A entity may be represented by its attributes {a1, a2, a3, . . . , an}, while a data asset B entity may be represented by its attributes {b1, b2, b3, . . . , bm}. An entity mapping between two data assets A {a1, a2, a3, . . . , an} and B {b1, b2, b3, . . . , bm} may be represented as:
{(ai:bj|aiϵA,bjϵB)}

Entity mapping is identified between various types of data assets: structured to structured, structured to semi-structured, semi-structured to structured, semi-structured to semi-structured, unstructured to structured, and unstructured to semi-structured.

FIG. 2illustrates entity mapping between data assets in accordance with certain embodiments. In particular,FIG. 2illustrates that entity mapping may be a directional relationship between two data assets, data asset A and data asset B. A directional relationship implies that a first data asset may be related to a second data asset, but the second data asset may not be related to the first data asset.

FIG. 3illustrates, in a flowchart, operations to generate ranked entity mappings in accordance with certain embodiments. Control begins at block300with the entity mapping identifier110generating entity mappings that produce matching entities for a first data asset having attributes and a second data asset having attributes by: generating a first inverted index of entity identifier pairs for the first data asset with a value for each entity and attribute name pair; generating a second inverted index of entity identifier pairs for the second data asset with a value for each entity and attribute name pair; and using the first inverted index and the second inverted index to generate matching attribute pairs (which may be referred to as matching entity pairs) based on matching attribute values that form the entity mappings. In block302, the entity mapping identifier110computes an entity mapping score for each of the entity mappings based on a combination of factors. In block304, the entity mapping identifier110ranks (and optionally filters) the entity mappings based on each entity mapping score. In block306, the entity mapping identifier110uses the ranked entity mappings to determine which of the entity mappings are to be used to determine whether a same real-world entity is described by the first data asset and the second data asset.

In certain embodiments, depending on the entity mapping score, the entity mapping identifier110may filter out entity mappings having entity mapping scores less than a certain threshold and may rank the remaining entity mappings using the associated entity mapping scores.

FIG. 4illustrates, in a flowchart, operations to generate the entity mappings and entity mapping scores in accordance with certain embodiments. The operations ofFIG. 4give possible entity mappings along with their matching attribute pairs.

InFIG. 4, control begins in block400with the entity mapping identifier110generating an inverted index for data asset A (a first data asset) (Valued→{entity_idi1, attribute_namei1}) and in block402with the entity mapping identifier110generating an inverted index for data asset B (a second data asset) (Valuej2→{entity_idj2, attribute_namej2}). An inverted index for a particular entity or database row represented by an entity identifier (entity_idi) has a value Valueifor an attribute having an attribute_namei. Merely to enhance understanding,FIGS. 5A, 5B, 5C, 5D, and 5Eillustrate an example of the operations ofFIG. 4in accordance with certain embodiments. InFIG. 5A, data asset A500and data asset B510are illustrated.FIG. 5Billustrates an inverted index520for data asset A500, and an inverted index for data asset B530. InFIG. 5B, e11, e12, etc. are in different rows for data asset A500, and e21, e22, etc. are in different rows for data asset B530. (

In block404, the entity mapping identifier110performs attribute value matching to match attribute values of attributes of data asset A with attribute values of attributes of data asset B (e.g., attribute Name from data asset A500matches attribute SName of data asset B510).

In block406, the entity mapping identifier110uses the matching attribute values to generate matching attribute pairs. In particular, in block406, the entity mapping identifier110obtains a list of tuples, with each tuple including an entity identifier from data asset A, an attribute name from data asset A, an entity identifier from data asset B, and an attribute name from data asset B ({entity_idi1, attribute_namei1, entity_idj2, attribute_namej2}) if their corresponding values are matching.FIG. 5Cillustrate example output540for the processing of block406.

In block408, the entity mapping identifier110aggregates the output of block406to get the set of attribute pairs on which an entity identifier pair is matching {entity_idi1: entity_idj2, [attribute_namei1: attribute_namej2]}.FIG. 5Dillustrates example output550for the processing of block408.

In block410, the entity mapping identifier110uses the matching attribute pairs to identify entity mappings by aggregating the output of block408, to get the list of entity-id pairs which are matching on a set of attribute pairs to {[attribute_namei1: attribute_namej2], [entity_idi1: entity_idj2]}.FIG. 5Eillustrates example output560for the processing of block410.

The example output560also shows the number of entity pairs that are matching based on the set of attribute pairs.

There may be multiple entity mappings (i.e., a set of attribute name pairs as the output of block410, between a pair of data assets). All these mappings are such that based on each of these mappings a large number (more than certain threshold) of entities from the source data asset (data asset A) have at least one matching entity in the target data asset (data asset B). Determining which of the multiple entity mappings are useful depends on many factors, such as: the result of joining the data assets based on the entity mappings, the percentage of rows that join (support), and the final objective of joining the assets. With embodiments, the entity mapping identifier110ranks the entity mappings based on a combination of factors. The entity mappings describe the column values that may be compared to determine whether the same real-world entity is described in the two data assets. Once the entity mappings are ranked, they may be used in the ranked order described in the two data assets.

In an entity matching scenario, each row represents one entity. So the mapping between the two data assets should match one row (A1) in the first data asset (data asset A) with at least one row (B1) in the second data asset (data asset B). In such embodiments, the entity mapping identifier110assigns a higher entity mapping score to the entity mapping that causes a 1:1 mapping between the data assets. Both the data assets have a similar type of entities, e.g., both row A1and row B1are describing customers. In certain embodiments, for the general scenario of joins between two data assets, the fan out does not matter. That is, each row in the source data asset (A) may match with one or more rows in the target data asset (B). In this case, the entity mapping may be m:n rather than 1:1.

For each entity matching, the entity mapping identifier110also considers the number of additional attributes or columns which are not part of the matching. These columns provide value add (utility) when performing join/merge of entities. In cases in which the user knows the entity definition. An entity definition defines attributes of entities of an asset that may be used to identify the real-world entities (whereas entity mapping is across two data assets to know whether two entities are defining the same real-world entity) of one of the data assets (i.e., the list of attributes of one data asset which are part of entity mapping), the entity mapping identifier110ranks the entity mappings for that particular entity definition based on their support as follows: rank the mapping with larger support, higher (i.e., entity mappings with more matching rows are ranked higher).

In certain embodiments, the entity mapping identifier110maintains a history of user selections and weighs the support by probability of user selections (i.e., if the user has selected a particular entity mapping compared to other entity mappings, that entity mapping is ranked higher).

In certain embodiments, the entity mapping identifier110combines different factors, such as: a number of entity mapping attribute pairs, cardinality of entity mapping, join utility, support, probability of 1:1 or M:N mappings (based on scenario), and probability of past user selection, if any.

In certain embodiments, the entity mapping score f is a function of the following factors:a number of attributes involved in entity mapping (n)a cardinality of individual entity definition (i.e., joint cardinality of a set of attributes used for entity mapping, C1and C2, for the source and target data asset)support of matching (s, a number of entities or rows of the source data asset which are matching with one or more entities or rows of the target data asset)a probability of one to one (1:1) matching compared to (M:N) matching (p)a join utility (j) that accounts for obviousness of matches across data assets for different attribute cardinalities. A match on an attribute, such as ‘Gender’, that contains only a few distinct values, will find matching records from other assets, but may not add much value to the overall mapping (e.g., inverse of joint cardinality of attributes from the source data asset). A join utility may be a function of a number of columns besides entity definitions.a probability of previous user selections, if any (u, higher for entities which were selected earlier)

The value of the entity mapping score f may be derived as function(1):

With reference to the entity mapping score f, the ki's and α values are positive coefficients giving weights to individual factors. For example, in certain scenarios where one needs to add more and more attributes to the source asset, one should assign lower value to α; or when there is enough user history with similar use-cases one can assign higher value to k6. From earlier example ofFIGS. 5A-5E, assuming parameters k1=0.1, k2=0.15, k3=0.2, k4=0.1, k5=0.3, and k6=0.15, and values from example embodiment for variables n=1, C1=3, C2=3, s=3, α=1, p=1, j=1/3, and u=0, a mapping score f for mapping Age#Age would evaluate to:
f=0.1*1+0.15*3*3+0.2*3/4+0.1*1+0.3*1/3=1.8

In certain embodiments, a user may be presented the entity mapping results with descending values of the entity mapping score f.

Thus, computing the entity mapping score for each of the entity mappings includes generating an entity mapping score using one or more of: a number of attributes involved in an entity mapping of the entity mappings, a cardinality of that individual entity mapping, support of matching for that entity mapping, a probability of (1:1) matching for that entity mapping, a join utility for that entity mapping, and a probability of previous user selections for that entity mapping, and adding the entity mapping score for each element to generate the entity mapping score for that entity mapping.

Certain embodiments are advantageously directed to semi-structured data assets (e.g., eXtensible Markup Language (XML) and JavaScript Object Notation (JSON)). JavaScript is a registered trademark of Oracle Corporation in the United States and/or other countries.

In comparison to structured data assets, the entity mapping identifier110handles the following additional features for semi-structured data assets:each record (row) may be hierarchicaleach record may have different schema—with optional attributes.attributes may have multiple values (e.g., an array structure for JSON)

For semi-structured data assets, embodiments provide techniques of handling arrays and optional attributes as follows:

1) Flatten the hierarchical structures of these documents to handle the hierarchy. Optional attributes are converted into attributes that can store null values, as well. For example, phone_number: {cell: +1-234-567890, landline:+1-987-6543210} may be flattened as entity with two attributes phone_number.cell=+1-234-567890 and phone_number. Landline=+1-987-6543210.

2) Arrays may be handled by defining different equality operations for arrays. For example, embodiments may define two arrays to be equal if they have at least one value in common.

The entity mapping identifier110may employ a similar technique for semi-structured nested data as for structured data assets with some modifications. In certain embodiments, hierarchical schema for semi-structured data may be handled by flattening the data with hierarchical column names.

With embodiments, each record in semi-structured data may have a different schema. Thus, instead of having 1:1 attribute mappings, there may be 1 to many and/or many to 1 mappings. For example, an attribute phone number may have to be matched with a cell number, as well as, a home phone number. The entity mapping identifier110may also handle this by having a comprehensive entity mapping with support penalty for absent attributes while having 1:M (or M:1) mapping for some attributes ({(ai:bj|aiϵA, bjϵB)} with partial matching). For handling array attributes, the entity mapping identifier110may change the attribute value matching to include operators such as: in, subset, etc.

Certain embodiments advantageously apply to unstructured data assets. In certain embodiments, an unstructured data asset may be considered as a collection of documents. An entity definition for an unstructured data asset may be established with respect to another structured or semi-structured data asset. Any unstructured document may be modeled in two ways: 1) bag of words or 2) annotated words.

In the bag of words case, there may be one sides mapping, i.e., a list of columns/paths from other (structured/semi-structured) data asset that should match the words from the unstructured data assets.

In case of annotated documents, the unstructured data assets may be represented as structured/semi-structured data assets, and the techniques for structured and semi-structured data assets are applied. For example a sentence, “Mr. Wilson went to Chicago on May 14”, may be annotated as “Mr. Wilson <person-name> went to Chicago <city> on May 14 <date>”. This can be constructed as entity with three attributes person-name, city, and date with associated values.

With embodiments, a user may be given choices for joining multiple data assets associated with a score. With embodiments, an entity mapping score is defined as a combination of multiple factors discussed above with reference to function (1). Once the data assets are joined, various operations may be performed to characterize the matching rows, e.g., defining relationship contexts between data assets.

Embodiments extract entity mappings that produce matching entities between a pair of data assets. Embodiments compute a strength of the entity mapping that produces matching entities between a pair of data assets. Embodiments extract entity mappings that produce matching entities between a pair of data assets where values may match fuzzily. Here fuzzy matching may be described as matching two values approximately. For example, if two values are “International” and “International”, then these are not exact matches, but they may be said to be fuzzily matching, allowing, for example, some spelling mistakes.

Merely to enhance understanding of embodiments, some examples will be provided. However, embodiments are not limited to such examples.

In use case 1, a user may want to add to the factual information of the matching rows in data asset A from another data asset B. The user may choose an entity mapping (provided by the entity mapping identifier110) that minimizes duplicate sets of matching values. This biases towards an entity mapping with columns having more unique values. This closely resembles 1:1 matching. This works for data assets that host similar entity types e.g. both A and B both talk about a person. Vertically partitioned data (where an entity's attributes are partitioned and kept as different data assets) may also be merged in this process.

In use case 2, the user may want to integrate generalizations of a set of values (e.g. given street/city, the user wants to add state/zipcode) against each matching row. This biases towards choosing an entity definition with more duplicate values and a fewer number of columns. This closely resembles M:1 matching. This works for data assets which need not be similar. In use case 3, the user may want to keep the information that results in a high number of matching rows. This biases towards choosing an entity definition with very high support. This closely resembles M:N matching. This helps in integrating data which are horizontally partitioned.

With reference to computing probabilities of entity mappings, embodiments may want to compute the joint distribution arising from each entity definition identified for source data asset A. Consider the entity definition of concern is e1, which results in rows r1,r2, . . . rnof A to match against rows s1, s2, . . . skof target data asset B. The joint distribution θe1(A,B) in this case may be calculated using following formula (1):
Pr(r1,r2, . . . rn:s1,s2, . . . ,sk)[Given thatnrows fromAmatchedkrows fromB]
Pr(r1,r2, . . . ,rn|s1,s2, . . . ,sk)Pr(s1,s2, . . . ,sk)

Note that this value of θe1(A,B) reflects whether the join is close to 1:[1 or N] (higher value) or M:[1 or N] (lower value) from A's side.

For the three use cases discussed above, the entity mapping identifier110opts for three scores to rank entity definitions against each of the use cases. Using formula (2), the entity mapping identifier110calculates the joint distribution
θe1(A,B)=Pre1(r1|s1,s2, . . . ,sk)Pre1(r2|s1,s2, . . . ,sk) . . .Pre1(rn|s1,s2, . . . ,sk)Pre1(s1,s2, . . . ,sk)
Similarly, the reverse relation from assetBto assetAis calculated as θe1(B,A)

The strength of an entity definition e1is defined in terms of Fβscore with formula (3):

With use case 1: with β(e1,A)<1, the score increases with the entity definition containing unique values on both A, B (i.e. 1:1 case), so the entity mapping identifier110chooses β(e1,A)=Se1(A).

With use case 2: with β(e1,A)>=1, the score increases with the entity definition containing unique values on B (i.e., m:1 case), so the entity mapping identifier110chooses β(e1,A)=1+Se1(A)

Ranking for use case 3 is Se1(A)×Se1(B).

FIG. 6illustrates table T1600and table T2610in accordance with certain embodiments. With table T1600and table T2610, there is no PK-FK mapping, and the potential set of joinable columns are:

With table T1600and table T2610, The spurious matches are: #Students:#TestTakers. The match may be described as spurious because it holds only on a syntactical level (i.e., matching on values), but not on a semantical level (i.e., matching two tables on #Students against #TestTakers does not have practical significance, unlike matching on School Name or Location).

Therefore, m:1 gets more strength. Assume that addition of location removes the redundancy from the B side, leading to a perfect match. Then, the score is computed as follows:
Fβ(e1,A)=2*(1/2)=1

Therefore, 1:1 gets even more strength than m:1.

FIG. 8illustrates an example implementation800in accordance with certain embodiments.

Given input data assets S and T, both having a set of records with entityId as the key, and one or more fields represented using {attrName, attrValue}, at the first block denoted MR1, the entity mapping identifier110builds a reverse index with attrValue as the key, and one or more pairs of {entityId, attrName}.

Reverse indices from both these data assets are fed to block MR2, at which the entity mapping identifier110combines all pairs of {entityId, attrName} from both the data assets with matching attrValue and outputs a set of {entityIdS:entityIdT} against corresponding {attrNameS:attrNameT}, where entityIdS is a specific entityId from S, entityIdT is a specific Id (identifier) from T, attrNameS is a specific field from S under entity identifier entityIdS, attrNameT is a specific field from T under entity identifier entityIdT, and attrNameS as well as attrNameT have the same value in the corresponding data assets.

At the next block denoted R3, the entity mapping identifier110aggregates all such pairs with same key.

The next step, MR4, is directional in nature, and its outcome depends on whether the direction is from S to T, or from T to S. In the first case, the entity mapping identifier110outputs a list of entity identifiers from data asset S, whereas, in the latter case, the entity mapping identifier110outputs a list of entity identifiers from data asset T, aggregated against unique combinations of {attrNameS:attrNameT}.

FIG. 9illustrates processing800for creating an inverted index on attribute values of various entity attributes of a data asset S in accordance with certain embodiments. This is also illustrated inFIG. 7.

Given a data asset S with entityId as the key for a record, and a set of fields with their corresponding values, this phase builds a reverse index in which every field value (attrrValue) becomes a key, and the enttyId-attrName under which it was found are aggregated against it.

FIG. 10illustrates further processing900in accordance with certain embodiments. The output of this stage is list of possible entity definitions and support of each of them. After the processing of MR4, the following inputs are received from the user: 1. entity definition and 2. context attributes. The reverse index is built for data asset S as well as data asset T as perFIG. 10. These indices are then combined over the keys, and every value from data asset S under a key is combined with every value from T under the same key. This process is repeated for all the keys which are common on both the sides.

This results in entity identifiers from S and T (entityIdS and entityIdT, respectively) paired against the corresponding attribute names (attrNameS and attrNameT, respectively).

The attribute name pairs are then aggregated by entity identifier (ID) pairs. This process collects all attribute names under entityIdS and entityIdT which have matched on some value.

Finally, entity identifiers which match on similar combinations of attribute names are collected together. Each unique combination of attributes serves as an Entity Mapping.

FIG. 11illustrates further processing1100in accordance with certain embodiments. This optional processing is responsible for discovery of contexts which may horizontally partition the data based on attribute value filtering. Given the list of entity mappings discovered earlier, optionally a set of context attributes pre-defined by the user, and the original data of the data asset, the processing of MR5and the routine to identify context predicates, discovers predicates on a subset of context attributes with user-defined parameters.

FIG. 12illustrates, in a flowchart, operations to identify entity mappings that are to be used to determine whether a same real-world entity is described by the first data asset and the second data asset in accordance with certain embodiments. Control begins at block1200with the entity mapping identifier110generating entity mappings that produce matching entities for a first data asset having attributes and a second data asset having attributes by: generating entity mappings that produce matching entities for a first data asset having attributes with attribute values and a second data asset having attributes with attribute values by: matching the attribute values of the attributes of the first data asset with the attribute values of the attributes of the second data asset, using the matching attribute values to generate matching attribute pairs, and using the matching attribute pairs to identify entity mappings. In block1202, the entity mapping identifier110computes an entity mapping score for each of the entity mappings based on a combination of factors. In block1204, the entity mapping identifier110ranks the entity mappings based on each entity mapping score. In block1206, the entity mapping identifier110uses the ranked entity mappings to determine which of the entity mappings are to be used to determine whether a same real-world entity is described by the first data asset and the second data asset.

Embodiments provide extensions to semi-structured data assets. A semi-structured data asset is another form of structured data that does not follow traditional RDBMS like table format. Semi-structured data (e.g., XML or JSON data) has a schema associated with it. A semi-structured data asset may be converted into a structured data asset with an annotated XML schema decomposition” technique. First, the XML schema is shred into a set of tables, and then the data is inserted into the tables accordingly, while ensuring that the entity identifier is appropriately associated with each of the records in the tables. The same technique applied for structured data may be applied for semi-structured data assets now.

For semi-structured data asset to semi-structured data asset processing, given two data assets A and B, the entity mapping identifier110creates an attribute value dictionary for both DA and DB. Each entry in an attribute value dictionary is of the form {vj:ai, vj ⊂ Range(ai)}, where ai is a column of the data asset in concern. The entity mapping identifier110performs a cartesian product between DA and DB to collect similar values of the form, {v_j:a_i:r_m, b_k:s_n}, where r_m is the entity from A and s_n is an entity of B. The entity mapping identifier110computes support and confidence of all entity mappings for a pair of entity mapping r_m and s_n, of the form, {r_m,s_n:a_i_i1:b_k1, a_i2:b_k2. . . a_ix:b_kx}. Then, the entity mapping identifier110emits each entity mapping that satisfies a user given threshold.

For semi-structured data asset to structured data asset processing, given two data assets A and B, where data asset B is unstructured, the entity mapping identifier110creates an attribute value dictionary for DA. For every entity (document) of B, the entity mapping identifier110annotates the tokens with respect to DA. The entity mapping identifier110creates DB with the respective annotated values. The entity mapping identifier110performs a cartesian product between DA and DB to collect similar values of the form, {v_j:a_i:r_m, b_k:s_n}, where r_m is the entity from A and s_n is an entity of B. The entity mapping identifier110computes support and confidence of all entity mappings for a pair of entity mapping r_m and s_n, of the form, {r_m,s_n:a_i1:b_k1, a_i2:b_k2. . . a_ix:b_kx}. Then, the entity mapping identifier110emits each entity mapping that satisfies a user given threshold.

In certain embodiments the entity mapping identifier110applies entity resolution techniques to discover how two data sources may be related to each other by performing entity resolution on the data in those sources to reverse-engineer the matching references.

Thus, embodiments generate entity mappings that produce matching entities for a first data asset and a second data asset, compute an entity mapping score for the entity mappings, and rank the mappings based on the entity mapping scores.

With embodiments, generating the entity mappings includes generating a first inverted index of entity identifier pairs for the first data asset, generating a second inverted index of entity identifier pairs for the second data asset, and using the first inverted index and the second inverted index to get matching entity pairs based on a number of entity mappings, and calculate scores of various entity mappings. With embodiments, the first data asset and the second data asset are each one of structured data, unstructured data, and semi-structured data. In certain embodiments, the matching entities are ranked based on a combination of factors.

Also, embodiments integrate the first data asset and the second data asset using ranked matching entities by performing one of a join operation, a merge operation, and a union operation. Data integration may be described as the process of combining the data from one asset to one or more related data assets in order to generate a holistic view. The end result of the integration varies depending on the operation carried out to integrate the assets. The process depicted inFIG. 4is one embodiment that can generate join keys between two data assets based on which these assets can be integrated.

Data assets may be database tables (structured data assets), XML or JSON tree formatted data (semi-structured), or natural language textual data. The entity mapping may be used to know whether a common set of entities are being described in both the data assets. The entity mapping provides list of columns/tags/annotations that are compared to know whether there is any common entity. Unlike the traditional methods of schema mapping that work with data schema, embodiments use the data instances to identify the entity mappings and their supports.

Traditionally data mapping is done manually where some expert needs to know all the data assets and define schema mapping among them. With the volume and variety of big data, this may no longer be possible. In such scenarios, there is typically a large amount of data that is discovered, integrated, and processed for various analytics requirements. There are various techniques for data integration—join, entity merge, union, etc. For data discovery and integration, the user needs to understand the entity mapping between data assets to answer questions such as: which data assets should be used for a particular requirement, whether the data assets should be integrated, what method can be used for integration, etc. Embodiments answer these questions.

Embodiments provide automated data linking, multi-asset data profiling, scalable data linking and profiling, and profiling semi-structured and unstructured data assets.

With automated data linking, some techniques are manual where a subject matter expert maps columns in one data asset to that of another data asset, while other techniques identify primary key-foreign key relationships by assuming that mapping involves primary key of one of the data assets. Embodiments make no such assumption while providing method of finding entity mappings for data assets.

Some techniques profile data assets independently—cardinality, value distribution, patterns and data types, foreign key dependencies, etc., are profiled for a single data asset (and its columns). In comparison, embodiments perform data profiling over multiple data assets to figure out data overlap by detecting duplicates (Union), linkages (Complement), multiple representations of the same real-world entities, or to discover an entity to complement an existing one.

With scalable data linking and profiling, embodiments propose using distributed computing platform (e.g., a map-reduce platform) for scalable data linking and profiling.

With profiling semi-structured and unstructured data assets, embodiments are applicable for wide variety of data assets—structured, semi-structured and unstructured.

Referring now toFIG. 14, a schematic of an example of a cloud computing node is shown. Cloud computing node1410is only one example of a suitable cloud computing node and is not intended to suggest any limitation as to the scope of use or functionality of embodiments of the invention described herein. Regardless, cloud computing node1410is capable of being implemented and/or performing any of the functionality set forth hereinabove.

As shown inFIG. 14, computer system/server1412in cloud computing node1410is shown in the form of a general-purpose computing device. The components of computer system/server1412may include, but are not limited to, one or more processors or processing units1416, a system memory1428, and a bus1418that couples various system components including system memory1428to processor1416.

Computer system/server1412typically includes a variety of computer system readable media. Such media may be any available media that is accessible by computer system/server1412, and it includes both volatile and non-volatile media, removable and non-removable media.

System memory1428can include computer system readable media in the form of volatile memory, such as random access memory (RAM)1430and/or cache memory1432. Computer system/server1412may further include other removable/non-removable, volatile/non-volatile computer system storage media. By way of example only, storage system1434can be provided for reading from and writing to a non-removable, non-volatile magnetic media (not shown and typically called a “hard drive”). Although not shown, a magnetic disk drive for reading from and writing to a removable, non-volatile magnetic disk (e.g., a “floppy disk”), and an optical disk drive for reading from or writing to a removable, non-volatile optical disk such as a CD-ROM, DVD-ROM or other optical media can be provided. In such instances, each can be connected to bus1418by one or more data media interfaces. As will be further depicted and described below, memory1428may include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the invention.

Program/utility1440, having a set (at least one) of program modules1442, may be stored in memory1428by way of example, and not limitation, as well as an operating system, one or more application programs, other program modules, and program data. Each of the operating system, one or more application programs, other program modules, and program data or some combination thereof, may include an implementation of a networking environment. Program modules1442generally carry out the functions and/or methodologies of embodiments of the invention as described herein.

Computer system/server1412may also communicate with one or more external devices1414such as a keyboard, a pointing device, a display1424, etc.; one or more devices that enable a user to interact with computer system/server1412; and/or any devices (e.g., network card, modem, etc.) that enable computer system/server1412to communicate with one or more other computing devices. Such communication can occur via Input/Output (I/O) interfaces1422. Still yet, computer system/server1412can communicate with one or more networks such as a local area network (LAN), a general wide area network (WAN), and/or a public network (e.g., the Internet) via network adapter1420. As depicted, network adapter1420communicates with the other components of computer system/server1412via bus1418. It should be understood that although not shown, other hardware and/or software components could be used in conjunction with computer system/server1412. Examples, include, but are not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.

In certain embodiments, the computing device100has the architecture of computing node1410. In certain embodiments, the computing device100is part of a cloud environment. In certain alternative embodiments, the computing device100is not part of a cloud environment.

Cloud Embodiments

Characteristics are as follows:

Service Models are as follows:

Deployment Models are as follows:

Referring now toFIG. 14, illustrative cloud computing environment1450is depicted. As shown, cloud computing environment1450comprises one or more cloud computing nodes1010with which local computing devices used by cloud consumers, such as, for example, personal digital assistant (PDA) or cellular telephone1454A, desktop computer1454B, laptop computer1454C, and/or automobile computer system1454N may communicate. Nodes1010may communicate with one another. They may be grouped (not shown) physically or virtually, in one or more networks, such as Private, Community, Public, or Hybrid clouds as described hereinabove, or a combination thereof. This allows cloud computing environment1450to offer infrastructure, platforms and/or software as services for which a cloud consumer does not need to maintain resources on a local computing device. It is understood that the types of computing devices1454A-N shown inFIG. 14are intended to be illustrative only and that computing nodes1010and cloud computing environment1450can communicate with any type of computerized device over any type of network and/or network addressable connection (e.g., using a web browser).

Referring now toFIG. 15, a set of functional abstraction layers provided by cloud computing environment1450(FIG. 14) is shown. It should be understood in advance that the components, layers, and functions shown inFIG. 15are intended to be illustrative only and embodiments of the invention are not limited thereto. As depicted, the following layers and corresponding functions are provided:

Hardware and software layer1560includes hardware and software components. Examples of hardware components include: mainframes1561; RISC (Reduced Instruction Set Computer) architecture based servers1562; servers1563; blade servers1564; storage devices1565; and networks and networking components1566. In some embodiments, software components include network application server software1567and database software1568.

Virtualization layer1570provides an abstraction layer from which the following examples of virtual entities may be provided: virtual servers1571; virtual storage1572; virtual networks1573, including virtual private networks; virtual applications and operating systems1574; and virtual clients1575.

Workloads layer1590provides examples of functionality for which the cloud computing environment may be utilized. Examples of workloads and functions which may be provided from this layer include: mapping and navigation1591; software development and lifecycle management1592; virtual classroom education delivery1593; data analytics processing1594; transaction processing1595; and prior compare processing1596.

Thus, in certain embodiments, software or a program, implementing prior compare processing in accordance with embodiments described herein, is provided as a service in a cloud environment.

Additional Embodiment Details