Patent Description:
This disclosure relates to complex processing of metadata from diverse data sources to provide a unified metadata view, and to use the metadata to drive data analytics, including data quality and data lineage analyses.

The processing power, network bandwidth, available storage space, and other resources available to computing systems have increased exponentially in recent decades. Advances in computing systems, both local and cloud-based, have led to the capture, storage, and retention of immense quantities of information content in a globally distributed manner. It is a significant technical challenge to obtain a meaningful, consistent and normalized view of the metadata that describes the information content, as well as to perform meaningful analytics on the information content.

In United States Patent Application Publication <CIT> there is described a system for managing objects. The system may be configured to provide a set of objects in a repository with associated metadata including workflow metadata and an application program interface. The system may determine a first region of interest in a graphical user interface, the first region of interest corresponding to a first time range. The system may retrieve metadata for a first set of objects having workflow metadata indicating at least one event will occur within the first time range, create a first set of nodes representing the first set objects, display the first set of nodes based on a time characteristic of the workflow metadata associated with the at least one event, and perform an action on a corresponding object from the first set of objects.

<CIT> discusses methods, system, and apparatuses for generating and delivering analytic results for simple or highly complex problems for which data exists that software or similar automated means can analyze.

<CIT> discusses apparatus and methods for data lineage management operation procedures.

A closed-loop universal metadata architecture ("architecture") implements a universal metadata repository ("UMR") and performs data analytics, including determining and maintaining data lineage, and determining meaningful data quality metrics. The architecture automatically scans and captures metadata characterizing input data driven into any pre-defined enterprise workflows from any number and type of data sources, whether internal to an enterprise or external to the enterprise that hosts the workflows. The architecture may be programmatically used for data tracking, management, review, reporting and many other purposes.

The UMR is universal in the sense that it provides a single logical user interface view of input data to the workflows, including business and technical data in the form of, for example, a graph schema. The programmatic capabilities of the architecture implement flexible data lineage tracking, data quality determination, data gap filling, and discovery of additional data via, e.g., similarity detection. The UMR integrates any desired data profiles and similarity profiles across an entire enterprise platform. The architecture includes a feedback loop that, e.g., enforces business rules, re-scans the data sources, and updates the UMR on any scheduled or directed basis.

<FIG> and <FIG> provide an example context for the discussion below of the technical solutions in the architecture, including the UMR, data quality metrics, data lineage and reporting. The examples in <FIG> and <FIG> show possible implementations. In that respect, the technical solutions in the architecture are not limited in their application or implementation to the systems shown in <FIG> and <FIG>, or any of the other Figures. Instead, the technical solutions may be implemented via many other system implementations, architectures, and connectivities. For example, additional hardware and software implementations of the architecture, additional logic flows implemented in the architecture, and additional GUI interfaces defined and delivered by the architecture are described herein.

<FIG> shows network connected entities <NUM>, including the architecture <NUM>, and (with respect to the architecture <NUM>) both internal data sources <NUM> and external data sources <NUM>. The architecture <NUM> receives source metadata, e.g., the source metadata <NUM>, from the data sources, and returns feedback messages, e.g., the feedback message <NUM>, to the data sources via any number or type of public and private communication networks, e.g., the communication network <NUM>.

As described herein, the architecture <NUM> receives the source metadata, and on behalf of any given enterprise running any pre-defined workflows, analyzes and processes the source metadata, builds and maintains a UMR, determines data quality metrics, scores data quality metrics, builds and maintains data lineage, performs data lineage scoring, sends feedback to the data sources and provides a holistic view of the UMR in a graphical user interface (GUI). To that end, the architecture <NUM> includes communication interface circuitry <NUM> that is connect to the data sources <NUM> and <NUM>, as well as repository processing circuitry <NUM>, reporting circuitry <NUM>, and display circuitry <NUM>. The architecture <NUM> performs its analysis of input data directed into pre-defined workflow on the basis of metadata elements, e.g., the metadata elements <NUM> and <NUM>, received from any data source.

With regard to the obtaining the source metadata, the architecture <NUM> may facilitate metadata collection by implementing a vendor agnostic communication layer, with an exposed application programming interface (API) for import and export of data. The architecture <NUM> includes a repository of enrichment algorithms, including data quality and data lineage generation and analysis to enrich the metadata beyond what is available via vendor tools. Moreover, a computer based feedback loop may be present as part of the enrichment algorithms to automatically and dynamically obtain additional metadata from the data sources as deficiencies in the metadata are identified by the architecture <NUM>.

Expressed another way, the communication interface circuitry <NUM> retrieves source metadata from data sources <NUM>. The data sources <NUM> provide input data in various schemas to pre-defined workflows for any given enterprise, and the source metadata characterizes that input data to create a uniform schema in a universal metadata repository. The communication interface circuitry <NUM> provides the source metadata to the repository processing circuitry <NUM>. In turn, the repository processing circuitry <NUM> integrates the source metadata into the schema of the universal metadata repository. As part of the source metadata integration process, the repository processing circuitry <NUM> may identify key data frames within the source metadata from each data source <NUM>. The key data frames may be stored in the universal metadata repository to be representative of the entirety of the extracted source metadata. The repository processing circuitry <NUM> may also perform data analytics on the input data represented in the universal metadata repository, and execute a feedback loop responsive to the data analytics to deliver the feedback messages <NUM> to the data sources <NUM> to update and/or enrich the metadata present in the universal metadata repository.

The architecture <NUM> may perform a wide variety of data analytics, including static data analysis and dynamic data analysis. Static data analysis may include computer based review and analysis of the normalized metadata information ingested from the data sources. For example, static data analysis may include analysis of the a data lineage schema or structure, the data sources from which the metadata is extracted, and/or the collected metadata. In another example, static data analysis may involve comparison of newly obtain metadata with previously obtained metadata for completeness, trends, omissions and significant changes. In still another example, static analysis may be an analysis for gaps or omissions in the normalized metadata obtained from the data sources. The static data analysis may be performed by the computer based on computer based historical data comparison, rules, relationships, predetermined setpoints, predetermined thresholds, and/or any other mechanisms or process to identify shortfalls, omissions, or undesirable variations.

Dynamic data analysis may involve rules based data analysis, generation of data metrics and/or analysis of generated data metrics. As just one example, the architecture <NUM> may dynamically determine a data quality metric for the input data based on the source metadata. Data quality metrics generated by the architecture <NUM> may include <NUM>)computer based data quality source completeness analysis, <NUM>)computer based data accuracy analysis, <NUM>)computer based data precision analysis, <NUM>)computer based timeliness analysis, and <NUM>)computer based reasonableness analysis may be performed. In addition, duplication, veracity/integrity, data coverage, data variation, and other parameter analysis may be included in the computer based accuracy analysis.

Such computer based dynamic analysis may be based on predetermined values, predetermined thresholds, comparison with third party information, comparison with collateral information, consideration of service level agreements, historical data comparison or any other analysis capable of identifying possible issues. For example, data accuracy, precision, and veracity/precision analysis may be based on data source analysis, attribute analysis and the like. Timeliness analysis may be involve a time based analysis in which data from a certain time period or time value is compared, contrasted and otherwise analyzed with respect to data from another certain time period or time value. Reasonableness, duplication, data coverage and data variation analysis may be based on historical data analysis, corresponding data analysis, predetermined values or thresholds and other such considerations. Such computer based analysis may be rules based, based on statistical analysis, based on modeling, based on machine learning, based on artificial intelligence (Al), based on third party information, and/or based on any other parameters or conditions. Also, consideration and comparison of technical context and business context information, such as via an incidence graph may also be included as part of the dynamic data analysis.

As a result of the dynamic data analysis, a feedback message <NUM> may specify a data quality alert responsive to the data quality metric, with the data quality alert responsive to a data quality rule executed on the data quality metric (e.g., 'send an alert when the last data value does not fit the current trend'). In another example, a gap in metadata may be identified and included in a feedback message <NUM>.

The architecture <NUM> may also provide various reporting using the reporting circuitry <NUM> based on the data analytics performed. Reporting may be in the form of exportable files, viewable graphs, lists, tables and the like, and/or generation of databases or other data repositories. Reporting may be via the user interface circuitry <NUM>.

The user interface circuitry <NUM> may include one or more graphical user interfaces, displays, touch sensitive displays, voice or facial recognition inputs, buttons, switches, speakers, printers, and other peripheral elements or devices that allow human sensory perception of system operation. Additional examples include microphones, video and still image cameras, and any other input output devices. The user interface circuitry <NUM> may include hardware displays such as liquid crystal displays (LCD), Light Emitting Diode (LED) displays or any other form of image rendering hardware. In addition or alternatively, the user interface circuitry <NUM> may include transmission and/or image rendering hardware to enable image rendering and display on any hardware device or system whether local or remote from the architecture. For example, the user interface circuitry <NUM> may support web based interaction and display via a browser or other form of communication interface.

The user interface circuitry <NUM> may also provide sensory perception to a user of operation, functionality and control of the architecture <NUM>. For example, the user interface circuitry <NUM> may include the capability to provide a holistic view of the extracted source metadata represented in the universal metadata repository on a user interface, such as a display. The holistic view may be provided based on the key data frames. In an example, the holistic view may be depicted in an incidence schema or graph user interface view depicting relational aspects of the metadata represented in the universal metadata repository in a technical context. In addition, a business context may be extrapolated from the technical context. For example, the incidence schema view may illustrate relation of the key data frames to the data sources and to various attributes of the key data frames. In addition, relationships and context such as reference, relatedness, inheritance and hierarchy may be identified between data sources, technical context of the metadata, and business focus of a particular business or industry. For example, in an incidence graph user interface, attributes of the key data frames in the technical context may be depicted in a relationship, and/or in context to, nodes in a business context that is specific to a particular business or industry segment. The technical context may also be used in relationships to more than one business or industry by re-focusing extrapolation of the technical context into additional business context scenarios.

As another example, the static data analytics includes creating or updating a data lineage structure for the input data. The data lineage structure captures the lifecycle of input data and its consumption, which may be implemented and described in many different ways. In one respect the data lineage structure provides an incidence schema, incidence graph, or line of descent map, of the input data. The incidence graph may include lineage data fields that specify any combination of the following, as just a few examples: who affected the input data, what affected the input data, where the input data was affected, when the input data was affected, why the input data was affected, and how the input data was affected. In the case of data lineage, the feedback message may specify or include a data lineage alert responsive to the data lineage structure, with the data lineage alert responsive to a data lineage rule executed on the incidence schema, or line of descent map (e.g., 'send an alert when user X has modified the input data.

<FIG> shows another example implementation <NUM> of the architecture <NUM>. The example implementation <NUM> includes communication interfaces <NUM> of the communication interface circuitry <NUM>; system circuitry <NUM> of the repository processing circuitry <NUM>; input/output (I/O) interfaces <NUM> and display circuitry <NUM> of the user interface circuitry <NUM>. The display circuitry <NUM> may generate machine interfaces <NUM> locally or for remote display, such as in a web browser running on a local or remote machine. The machine interfaces <NUM> and the I/O interfaces <NUM> may include GUIs, touch sensitive displays, voice or facial recognition inputs, buttons, switches, speakers and any other user interface elements.

The communication interfaces <NUM> may include wireless transmitters and receivers ("transceivers") <NUM> and any antennas <NUM> used by the transmit and receive circuitry of the transceivers <NUM>. The transceivers <NUM> and antennas <NUM> may support WiFi network communications, for instance, under any version of IEEE <NUM>, e.g., <NUM>. 11b, g, n, or ac. The communication interfaces <NUM> may also include physical transceivers <NUM>. The physical transceivers <NUM> may provide physical layer interfaces for any of a wide range of communication protocols, such as any type of Ethernet, data over cable service interface specification (DOCSIS), digital subscriber line (DSL), Synchronous Optical Network (SONET), or other protocol.

The system circuitry <NUM> may include hardware, software, firmware, or other circuitry in any combination. The system circuitry <NUM> may be implemented, for example, with one or more systems on a chip (SoC), application specific integrated circuits (ASIC), microprocessors, discrete analog and digital circuits, and other circuitry. The system circuitry <NUM> is part of the implementation of any desired functionality in the architecture <NUM>. For example, the system circuitry <NUM> may include one or more instruction processors <NUM> and memories <NUM>. The memory <NUM> stores, for example, control instructions <NUM> and an operating system <NUM>. In one implementation, the processor <NUM> executes the control instructions <NUM> and the operating system <NUM> to carry out any desired functionality for the architecture <NUM>. The control parameters <NUM> provide and specify configuration and operating options for the control instructions <NUM>, operating system <NUM>, and other functionality of the architecture.

The architecture <NUM> may also include the enterprise data stores <NUM>. The enterprise data stores <NUM> may represent any number of data sources <NUM> and may include a UMR <NUM>, or any other enterprise data. The enterprise data stores <NUM> may be hosted on volume storage devices, e.g., hard disk drives (HDDs) and solid state disk drives (SDDs) and may adhere to a very wide range of data structures and data types. As examples, the data structures may include: SQL, no-SQL, object-oriented, and relational databases; unordered and ordered storage mechanisms; structured files; hash buckets; trees; and other structures. The data sources <NUM> may provide any type of input data to any number and type of enterprise workflows <NUM> defined within the enterprise. As just a few examples, the enterprise workflows <NUM> may include: human resources (HR) workflows that govern HR procedures such as hiring, reviews, and firing; banking workflows that create new client accounts, approve loans, or issue mortgages; manufacturing workflows that execute quality assurance procedures, process failure reports, and execute preventative maintenance; and customer service workflows that assign tasks to agents, processing claims, and resolve customer incident reports. Such enterprise workflows <NUM> may form the basis for the business context that is extrapolated from the technical context of the metadata as part of the holistic view of the universal metadata repository.

All of the workflows are driven by input data. The data sources <NUM> and <NUM> supply the input data to drive the workflows, and the source metadata characterizes the input data. Example input data includes, as just a few examples: database tables, columns, and fields; keyboard and mouse input; documents; graphs; metrics; files, such as Word, Excel, PowerPoint, PDF, Visio, CAD, Prezi files; application data objects, e.g., calendar entries, task lists, and email file folders; and other data types. Other examples of input data include attributes of the metadata such as, time stamps, data source ID, extraction tool used, and any other information that may be useful in characterizing the metadata in the holistic view.

The control instructions <NUM> include repository processing logic <NUM>, data analytics logic <NUM>, and machine interface generation logic <NUM>. The repository processing logic <NUM> integrates source metadata into the UMR <NUM>, and provides closed-loop feedback to the data sources responsive to data analyses driven by the UMR <NUM>. The data analytics logic <NUM> performs the data analyses, including determining data quality metrics, and building and maintaining the data lineage structure. The machine interface generation logic <NUM> may create and deliver a holistic view such as an incidence schema in the form of a linked interactive set of GUIs that facilitate interaction within the architecture <NUM>. The control instructions <NUM> may also, for example, be executed to identify workflows for analysis, determine data sources for those workflows, obtain source metadata from the data sources, obtain selections of data analyses to run, define closed-loop feedback rules, identify relations and inheritance among the source metadata, and allow operators to set configuration and preference parameters for the overall operation of the architecture <NUM>.

Among other aspects, the enterprise data stores <NUM>, repository processing logic <NUM>, data analytics logic <NUM>, machine interface generation logic <NUM> improve the functioning of the underlying computer hardware itself. That is, these features (among others described below) are specific improvements in the way that the underlying computer system operates. The improvements facilitate the generation of a universal holistic view of metadata across disparate data sources by integrating various different schemas within which the metadata is transformed to a single universal schema. Due to the universal schema present in the universal metadata repository, improvements in, for example, discovery of missing data (gap detection), and/or enrichment of the metadata by execution of data analytics across disparate data sources, may be performed to provide, among other things, closed-loop feedback that helps improve the execution of any defined workflow in the enterprise. The improved functioning of the underlying computer hardware itself achieves further technical benefits. For example, the architecture <NUM> automatically performs the complex processing needed to improve workflow performance, and thereby reduces manual intervention and reduces the possibility for human error. Still further, the architecture <NUM> facilitates a reduction in resource expenditure, including reduced storage volume accesses and processor-driven analytics due to, for example, data management using key data frames. Also, the architecture <NUM> operates with the key data frames in a distributed network environment to efficiently avoid duplicative data storage and large data transmission events among data sources distributed throughout a communication network. In addition, the architecture <NUM> may reduce or eliminates cumbersome and inexact manual tuning and analysis of the data sources and workflows, in favor of the centralized uniform schema metadata repository architecture stored in a distributed data storage system.

<FIG> shows an example of logic <NUM> that the architecture <NUM> may implement for analyzing data quality, e.g., as part of the data analytics logic <NUM>. The architecture <NUM> defines an enterprise rule definition framework (<NUM>), in which the architecture <NUM> creates enterprise rules (<NUM>) for its data. The architecture <NUM> maps enterprise rules to data quality rules (<NUM>). The data quality rules define quality metrics for the input data to the workflows. The enterprise rules may be specific to a business or industry.

The architecture <NUM> executes data profilers or data mappers to run profile scans on the data sources (<NUM>) to obtain the dataset of source metadata, such as technical metadata objects. The data profilers may be metadata scraping tools such as, for example, ATTIVIO, EMCIEN, CLOUDERA NAVIGATOR, SINEQUA or UNIFI, which include functionality to extract metadata from data sources. The scan results are parsed for data elements such as codes and scripts, and the scan profiles are captured (e.g. stored) in the UMR <NUM> (<NUM>). Next, the architecture <NUM> performs dynamic data analysis by executing the data quality rules, e.g., by traversing the structure of the UMR <NUM> and applying each data quality rule to the applicable field, table, column or other data element (<NUM>). As part of execution of the data quality rules, the architecture may dynamically perform data quality scoring (<NUM>) to obtain data quality metrics. The data quality scores generated by the data quality scoring may be stored as part of the data quality metrics in the UMR <NUM> in association with the respective data. Once all the data quality rules are applied and the data quality scoring is complete (<NUM>), the architecture <NUM> may conduct computer based acceptance testing (<NUM>). Acceptance testing may include issuing, as appropriate, feedback messages to the data sources. Acceptance testing may include consideration of data quality metrics and data lineage scores, as well as consideration of relations, inheritance, reference or any other data reconciliation, authentication and completeness parameters. Otherwise, if there are more, different, or new data quality rules to apply, the architecture <NUM> re-runs the profile scans using the same or different mappers to continue to test the data quality (<NUM>). In an example, the data quality scores may be used in the acceptance testing and to determine if more, different, or new data quality rules should be applied. The architecture <NUM> may alternatively, or in addition, conduct acceptance testing of the workflow, or take other actions once the scan profiles are parsed.

<FIG> shows an example of logic <NUM> that the architecture may implement for handling data lineage. The architecture <NUM> defines data lineage integration points in a business glossary (<NUM>) as key data frames and configures data lineage views in the business glossary (<NUM>) by integration of the technical context into the business context. The business glossary may be descriptive of the business related data used in one or more enterprises or organizations and may include business metadata. The business glossary may be maintained and provided with a data governance platform, such as COLLIBRA.

For unconfirmed relationships within the metadata in the universal metadata repository, the architecture <NUM> may configure lineage scanners, or mappers, to proceed with proposed relationships (<NUM>) based on the metadata, attributes and related information, and use a review tool to confirm the relationships to verify the dataflow (<NUM>). The lineage scanners may tools such as, for example, ALATION and CLOUDERA NAVIGATOR. The architecture <NUM> tags confirmed relationships in the UMR <NUM> (<NUM>). For confirmed relationships, the architecture <NUM> configures the lineage scanners to track the known lineage (<NUM>). In either case, the architecture <NUM> programmatically stores the data lineage in the UMR <NUM> (<NUM>).

The architecture <NUM> determines whether there are any exceptions (<NUM>), and if so, those exceptions are reviewed and the UMR <NUM> is updates to resolve the exceptions (<NUM>). When there are no exceptions, the architecture <NUM> conducts acceptance testing (<NUM>). In any case, the architecture <NUM> may selectively expand the data lineage, e.g., for troubleshooting purposes (<NUM>). When all the data is accounted for (<NUM>), the architecture <NUM> may conclude troubleshooting and mark the issue as resolved (<NUM>).

As mentioned above, the architecture <NUM> performs discovery of additional data via, e.g., similarity detection. <FIG> shows an example of logic <NUM> that the architecture <NUM> may implement for performing data discovery. The architecture <NUM> defines a data discovery scanner integration point in a business glossary (<NUM>), and configures data discovery scanners to infer similarity (<NUM>), using, e.g., pre-defined similarity rulesets. The architecture <NUM> runs data scanners on the data sources (<NUM>) and determine relationships between data sets based on the inferred similarities, which are recommended to data stewards (<NUM>). The determined relationships are programmatically stored in the UMR <NUM> (<NUM>).

The determined relationships may also be parsed for duplicates as part of the quality analysis (<NUM>). Where there are duplicates in the relationships, a snapshot of a dataset of source metadata from a data source in which the duplicates exist in relation to other datasets of source metadata may be generated to identify preferred data sources and a route of the relationships by walking through other use cases and the dataset (<NUM>). The architecture <NUM> may sample the data set of identified preferred data sources to determine if the data in the preferred data set is ok (<NUM>). If the data in the data set of the identified preferred data source is ok, access to the preferred data source may be requested (<NUM>), and the architecture <NUM> checks for whether the recommended relationships meets pre-defined acceptability criteria (<NUM>). If the relationship to the identified preferred data source is not acceptable, the architecture <NUM> may expand the search for relationships to other data sets, by, for example, identifying similar data in other data sets, or by identifying data sets that are connected by data flow to the identified preferred data source (<NUM>).

If the recommended relationships are not acceptable, then the architecture <NUM> modifies or drops the recommended relationships and re-runs the data scanners (<NUM>). When the recommended relationships are accepted (<NUM>), the architecture <NUM> may save those relationships, and continue to review the data sources into the future for new data relationships (<NUM>).

The architecture <NUM> addresses the technical challenges with managing data sources and summarizing their content. Many of these challenges arise due to the wide range of data sources, varying interfaces, varying accessibility standards, disparate data sources both internal and external to the enterprise, and/or variations in schemas of the extracted content. The architecture <NUM> provides a unified schema that provides a view of metadata, data lineage, and data quality for a system or set of systems, leading to much improved ability to track and maintain the information, while reducing errors. The architecture <NUM> understands the dichotomy of documents, data sources, source clusters, and source composition and defines a holistic model of metadata describing system state.

The architecture <NUM> also monitors and improves data quality. In that regard, the architecture <NUM> may define data quality metrics in terms of completeness, accuracy, precision, timeliness, reasonableness, or other factors. Additional data quality metrics include whether there is data duplication, the veracity/integrity of the data, and the coverage of the data. With regard to data lineage, the architecture <NUM> tracks aspects of data lineage including the 'who', 'what', 'when', 'where', 'why', and 'how' characteristics of the data. The data lineage may be discrete, probabilistic or both.

<FIG> shows an example of logic <NUM> that the architecture <NUM> may implement for performing mapping of business metadata to technical metadata objects. Business context descriptive of an organization may be included in business metadata, such as data usage and ownership, stored in a business glossary. The architecture <NUM> executes data profilers or data mappers to run profile scans on the data sources to obtain a profile of respective input data in the form of a dataset of source metadata, such as technical metadata objects (<NUM>). The technical metadata objects may be mapped to business terms included in the business glossary (<NUM>) using, for example, data analytics and automated inferences. In an example, data analytics may be performed by a scanner tool such as CAMBRIDGE SEMANTICS to create computer based inferences that may be used to map business terms to technical metadata objects by, for example, discovering semantic similarities and syntactic similarities between the business terms and selected technical metadata objects. The architecture may run a profile scan of the data source to obtain a recommendation for business metadata and similarities, such as semantic similarities and syntactic similarities (<NUM>). The technical metadata objects may be labeled with the mapping to the business metadata (<NUM>).

The architecture <NUM> may determine the accuracy of the mapping, such as using quality control functions (<NUM>). If the mapping is not accurate, such as not exceeding a predetermined threshold of accuracy, the architecture <NUM> may reassess the mappings (<NUM>) and then accept and label the new recommendations (<NUM>). If the mapping is accurate, the architecture may determine if a predetermined amount of data is labeled with mappings (<NUM>) If not, a manual process may be initiated by the architecture <NUM> to examine and assess the data (<NUM>). Otherwise, the process may end.

<FIG> is a block diagram illustrating an example of the functionality of the architecture <NUM>. The architecture <NUM> may include a front end layer <NUM>, an integration layer <NUM>, a back end layer <NUM>, a data processing circuitry <NUM> and a data storage circuitry <NUM>. In other examples, additional or fewer layers may be included to depict the described functionality. The front end layer may include a dashboard <NUM> for user interaction with the architecture <NUM> and an administrative user interface <NUM> to manage and maintain the architecture <NUM>.

The integration layer <NUM> of the architecture <NUM> may be a form of middleware providing an interface between the front end layer <NUM> and the back end layer <NUM>. The middleware may operate as a stateless, client-server, cacheable web-based communication protocol for networking. In an example implementation, the middleware may include an application program interface (API), such as a RESTful API.

The back end layer <NUM> of the architecture <NUM> may be included in the repository processing circuitry <NUM> (<FIG>). The back end layer <NUM> may perform computer implemented metadata collection and normalization across a data supply chain in a number of distinct phases, which may be described as selection, ingestion, data management, and data preparation. In addition, the architecture <NUM> may use the back end layer <NUM> to perform computer implemented conflict resolution during distinct phases of quality and reconciliation, and actionable management.

The selection phase may be performed with a universal metadata selection circuitry <NUM>. The universal metadata selection circuitry <NUM> may consume metadata information from the data sources, and determine where to get additional metadata information from among the various available data sources. Thus, the universal metadata selection circuitry <NUM> may consume workflow information or job tasks, from vendor solutions and resolve the data sources of interest. In addition, the universal metadata selection circuitry <NUM> may perform communication with the different data sources, such as by enabling querying of the data sources and/or the various profiling tools or data mappers used to scrape data from the different data sources. Functionality of the universal metadata selection circuitry <NUM> may include source profiling, parsing, archiving and retention. The universal metadata selection circuitry <NUM> may perform exploration of existing data sources resulting in simplified onboarding of data sources. Onboarding of data sources may include cataloging and indexing trusted data sources, as well as onboarding new data sources and configure transformation rules for metadata extracted from such new data sources.

The ingestion phase may be performed by a metadata ingestion circuitry <NUM>. The metadata ingestion circuitry <NUM> may normalize different metadata schemas from the data sources into a target format and object schema that is common across the universal metadata repository. For example, the metadata ingestion circuitry <NUM> may provide normalization of metadata through a series of connectors to an ingestible formation, such from a non JSON data to JSON data across the different schemas.

Since the format of the raw metadata in the different schemas may vary significantly, the metadata ingestion circuitry <NUM> may reconcile the various disparate formats into a common schema format in the universal metadata repository. Further, extracted source data may be archive loaded for auditability by the metadata ingestion circuitry <NUM>.

The back end layer <NUM> may also include a metadata conflict resolution circuitry <NUM> and a metadata schema enforcement circuitry <NUM>. The metadata conflict resolution circuitry <NUM> may perform metadata object matching and conflict resolution among data from different data sources. Accordingly, the metadata conflict resolution circuitry <NUM> may resolve any duplicated information by identification and deletion of repeated metadata within the universal metadata repository once the metadata from the different data sources has been normalized and duplication can be recognized. Thus, the metadata conflict resolution circuitry <NUM> may "clean" the data received in the universal metadata repository.

The metadata schema enforcement circuitry <NUM> may provide/handle harmonization of technical metadata across diverse metadata stores included in a data storage layer of the architecture <NUM> and/or data sources. For example, the metadata schema enforcement circuitry <NUM> may maintain metadata and schema alignment and perform computational work, such as last minute normalization, canonization check/data resolution and the like. In addition, the metadata schema enforcement circuitry <NUM> may perform storage and processing of data in structured and un-structured data formats in a UMR data store <NUM> included in the data storage layer of the architecture <NUM>. Also, the metadata schema enforcement circuitry <NUM> may catalog the data sources and perform mapping of the origin of the metadata within the attributes of the metadata. Also, the metadata schema enforcement circuitry <NUM> may detect specific datatypes, assign metadata tags, and route the metadata to predetermined queues resulting in accelerated data management with type-based patterns. Routing of the metadata to the predetermined queues includes routing the key data frames and mapping of the destination in which the metadata is stored by including such information in attributes associated with the metadata.

The metadata analytics circuitry <NUM> represents a repository of algorithms, links to algorithms, and links to tools that perform analytical operations on the UMR and source data to provide a more holistic view of the underlying data. The metadata analytics circuitry <NUM> may include a descriptive module <NUM> and a predictive module <NUM> to perform data quality metric analysis and data lineage development and analysis. Based on the data quality metrics analysis and the data lineage analysis, the metadata analytics circuitry <NUM> identifies key data frames among the dataset of source metadata from a data source.

The key data frames may be representative of a larger body of metadata such that the key data frames provided a sampled version of the metadata. In other words, instead of the architecture <NUM> duplicative storing the entirety of the metadata received from a data source, only the key data frames are stored. The key data frames may include attributes that point to the location (e.g. the data source) of the metadata being represented by a respective key data frame. Using the key data frames, data quality metrics and data lineage measurements are developed to provide a holistic view of the entirety of the metadata represented in the universal metadata repository.

The metadata analytics circuitry <NUM> may mine the normalized data from the various data sources for data probabilistic lineage metadata. In addition, the metadata analytics circuitry <NUM> may leverage the normalized data that has been quality reviewed through the metadata conflict resolution circuitry <NUM> and the metadata schema enforcement circuitry <NUM>. Through analysis of the collected and normalized metadata, the data quality metrics and the probabilistic lineage metadata, the key data frames are identified in the dataset of source metadata from each respective data source.

The metadata analytics circuitry <NUM> may also review the metadata represented in the universal metadata repository for gaps or missing parts of the metadata. Such gaps may be identified from computer based performance of data quality and data lineage using the descriptive module <NUM> and the predictive module <NUM>. When gaps in the metadata are discovered, the metadata analytics circuitry <NUM> may perform enrichment of the universal metadata repository by identifying a data source capable of at least partially filling the gap, extracting metadata information, and normalizing such metadata in order to add the newly extracted metadata to the universal metadata repository.

The event manager <NUM> may determines what jobs to schedule based on predefined and curated rules of interest. The event manager <NUM> may detect changes and acts as a scheduler for processing future tasks based on events of interest. The event manager <NUM> may monitor for triggering events using rules based logic. In addition, the event manager <NUM> may perform management of business rules to assign metadata tags and route loaded source data. Actionable management by the event manager <NUM> may include centralize management of data use and rules to provide role based user access and permissions, and centralized management of business rules and workflows.

Accordingly, the architecture <NUM> may be used to integrate all data profiles and similarity profiles across the entire platform. This approach also includes a feedback loop that enforces the business rules and re-runs the scans automatically to update the universal metadata repository. The architecture <NUM> may apply the rules defined on specific columns or fields. The event manager <NUM> may check to ensure the rules are applied and specifies any additional rules using a check rules match function. The architecture <NUM> may re-run the profiles and similarity scans to update the data quality metrics as per the newly applied rules.

<FIG> is a block diagram of an example of logic layers included within the architecture <NUM>. The various circuitry or engines present in the back end <NUM> may operate within these logic layers to perform the described functionality. The architecture <NUM> may include a data flow layer <NUM>, a data source layer <NUM>, a data lineage layer <NUM>, and a data field layer <NUM>. In other examples, fewer or greater numbers of layers may be used to illustrate the functionality described herein.

With reference to <FIG> and <FIG>, the universal metadata selection circuitry <NUM> may operate in the data flow layer <NUM> to capture transaction data <NUM> from data sources <NUM> in the data source layer <NUM>. The transaction data <NUM> may also be referred to as a dataset of source metadata from a data source. Capture of transaction flow data <NUM> may be performed with a tool, such as CLOUDERA NAVIGATOR. The transactions <NUM> are represented as a graphical flow as data moves through system (can be used to show prescriptive lineage for known processes). The universal metadata selection circuitry <NUM> may also map transaction source data sources and transaction destinations within the dataflow layer <NUM>.

The metadata ingestion circuitry <NUM> may operate in the data source layer <NUM>. The metadata ingestion circuitry <NUM> may also map transactions to the data sources <NUM> (push and pull) and catalog all of the data sources <NUM> in the data source layer <NUM>. Data sources <NUM> may be discovered from the transaction/process flow graph by the metadata ingestion circuitry <NUM> in the data source layer <NUM>. Also, metadata information may be represented in the data source layer <NUM> to a "table" or "document" level of data granularity.

Within the data lineage layer <NUM>, the metadata analytics circuitry <NUM> determines a data lineage timeline <NUM> that consists of key data frames <NUM>. Using the key data frames <NUM>, the architecture <NUM> tracks lineage at key points in time. In addition, the architecture <NUM> may provide historic, prescriptive, and probabilistic lineage. The key data frames <NUM> may be pointers to profiles containing the dataset of the source metadata of each respective data source <NUM>. In an example, the data set of source metadata from each data source (e.g. transaction) may be represented with one key data frame. However, in other examples, a data set may be represented by multiple key data frames <NUM>. The key data frames <NUM> are updated by time, event, signification change or other operational attributes, which are associated with the respective key data frames <NUM>. In an example, updates may be constrained by: desired granularity, space, or triggers from the event management <NUM>.

In <FIG>, a timeline flow of key data frames <NUM> are depicted by the data lineage timeline <NUM>. The depicted data lineage time line <NUM> illustrates a flow where each key data frame <NUM> points to at least one data profile, to indicate timing of when a dataset of source metadata was obtained from a respective data source <NUM>. The lineage timeline <NUM> may be created by, for example, time sampling to create a data profile at consistent predetermined time intervals. Alternatively, or in addition, the lineage timeline <NUM> may be based on an event being triggered. In the event trigger scenario the data profile transaction may be extracted based upon a rule (e.g. data upload, etc.) triggered by the event manager <NUM>. Alternatively, or in addition, the lineage timeline <NUM> may be generated based on, for example, change monitoring. For example, data profiling may occur when there is a significant change in data from a previous profile time (e.g. value drift, significant data upload, etc.) Occurrence of a significant change may be based on a predetermined set point or threshold which is compared at the respective data source <NUM> to previously received metadata parameters. The comparison may occur at a predetermined time, or frequency, or in response to an event.

The data field layer <NUM> may include field information within data source fields <NUM> of the key data frames <NUM>. The metadata analytics circuitry <NUM> may operate in the data field layer <NUM> to generate the data source fields <NUM> for the key data fields <NUM>. The data source fields <NUM> may include information such as: i. What fields are composed of and inter database relationships, ii. Field composition metrics (e.g. distribution, data types, and other attributes related to generation of a respective key data field) and/or iii. Field data quality metrics such as value composition, value coverage, heterogeneity/distinctness and the like. The data source fields <NUM> may be generated with a predetermined structure. In an example, each of the data source fields <NUM> may be generated in a basic JSON structure such as:
{
Profile:
Metadata:
ColumnStatistics:
DataQuality: { Fields, Rules}
}.

The metadata analytics circuitry <NUM> may also generate a data field composition in the form of a graph schema <NUM> of the lineage data to model the relational composition of the metadata represented by the key data frames <NUM>. The relational composition may be abstractly modeled with an object oriented approach to provide a visual illustration of the traits or aspects of the data sources, metadata, key data frames and the operational aspects of metadata collection and the relationships therebetween. In addition, time stamping, logic rules, attributes, definitions and the like may be used for inferencing of relatedness of the technical context to the business context. The depiction of the relational composition of the graph schema <NUM> may be based on data field statistics <NUM> and allow the architecture to traverse the graph schema <NUM> from top to bottom.

Referring again to <FIG>, the data processing circuitry <NUM> may perform real time processing of data. In addition to handling payload passing, the data processing circuitry <NUM> also defines the data workflow of the entire UMR system in that it handles all job executions and the passing of data between circuitry, jobs, engines, and modules.

The data storage circuitry <NUM> provides an interface to the universal metadata repository, which is a UMR data store <NUM> included in the architecture <NUM>, which includes both a graph store <NUM> and a document store <NUM>. The interfacing with the document store <NUM> may include interface with, for example, HADOOP, cloud storage, and NoSQL storage facilities, such as MONGO. Interfacing with the graph store <NUM> may include interfacing with, for example, NEO4i.

The graph store <NUM> holds information with regard to the relationship linkages between pieces of data that is ingested and the document store <NUM> contains the detailed information of data. The graph store <NUM> itself is comprised of a technical context and a business context which are two graphs that are loosely connected. The technical graph contains all information and relationships with regard to data, infrastructure, and transactions that occur on the data. The business context graph contains concepts, rules, reports, etc. that are used to understand and make business decisions. The two graphs are separated to serve the purpose of a strong technical underpinning where traversal is well known and a business graph that is specific to a use case, domain, client, or industry. The business graph serves as a knowledge piece that is exportable and reusable. For example, key aspects of a business graph may reused for differing domains, businesses, industries and the like. In addition, the business graph may be exportable and reusable by being bootstrapped with existing ontologies to kick start an engagement of the architecture <NUM> with a different business entity or industry. Conversely the business graph may be removed without effecting the underlying infrastructure representation in the technical manifestation (e.g. technical context) of the collected metadata.

<FIG> is an example of a incidence graph schema <NUM> for the architecture <NUM>. The incidence graph schema may represent the UMR by consolidating and presenting a shared view of all metadata across data sources, data types, data stores, and application uses of the data. The incidence graph schema <NUM> includes an example of a technical context graph <NUM> and a business context graph <NUM> which can depict normalized metadata from different data sources and information related thereto. The technical context graph <NUM> may depict Information about the applications, tools, and systems supporting a data management solution. The business context graph <NUM> may depict Information that describes the process, context and meaning about the data. The business context graph depiction may place emphasis on significance of data from a business point of view.

In <FIG>, the incidence graphs visually depict inheritance <NUM> as interconnecting dot-dash-dot lines, relations <NUM> as interconnecting solid lines, and reference <NUM> as dotted lines. The technical context graph <NUM> and the business context graph <NUM> may include representations of time stamps (A), nodes (B), Tools (C), Containers (D), Operation (E), Field (F), Data Source (G), Key Data Frame (H), Profile (I), Collection (J), Attribute (K), Data Rule (L), Definition (M), Role (N), User (O), and Report (P). In other examples, additional or fewer representations may be included in the incidence graph schema <NUM>.

In <FIG>, inheritance and relationships are only illustrated under one example, and one link for brevity. In addition, all vertices are of type Node (B) and inherit all the properties. Further, the user (O) inherits the Role (N), and key data frames inherit Timestamps (A). In general, the inheritance <NUM> included in the incidence graph schema <NUM> may indicate where various attributes and/or information is provided from (origin or source). The incidence graph schema <NUM> also illustrates relations <NUM>, such as "is a," "derived from," "instance of," and "consists of.

For example, a tool (C), such as EMCIEM or CLOUDERA NAVIGATOR MCM, may be represented and added data such as a timestamp (A) may be a relation <NUM> shown as an "instance of" the tool (C). In addition, the tool (C) may have one or more relations <NUM> to containers (D) which "is a" container of the tool, and an operation (E), which "is a" operation of the tool. The container (D) and the operation (E) may include relations <NUM> to fields (F) which "is a" field of the respective container (D) and the operation (E), and may be identified with a relation <NUM> to other fields (F) which are an "instance of" another field (F). In another example, a Collection (J), such as a table, may have a relation indicating the table "consists of" a number of different attributes (K), some of which may have a relation of being "derived from" a data rule (L).

In this way, the incidence graph may be used in different analysis scenarios to present information that would not otherwise be available due to the normalization of metadata received from different data sources, and due to the inheritance <NUM>, relation <NUM> and reference <NUM> represented therein. For example, the graph schema may be used to search for data received, for example, from a certain tool (C), and this same type of framework may be used to develop data lineage scores. For example, if the meta data includes a combination of manually entered data and tool obtained data, a data lineage score may be determined. The data lineage score may be based on "derived from" relations <NUM>. For example, if the incidence graph included ten nodes (B), and nine out of ten of the nodes (B) have a derived from link, than the data lineage score would be <NUM>. In another example, in a search for reliability of relationships between collections (J) of key data frames (H), a number of attributes (K) having a relation <NUM> of "instance of" may be used to determine a data lineage score. Thus, two collections (J) with a large number of attributes that are "instances of" each other will have a higher data lineage score.

The reference <NUM> may indicate that elements of one of the technical context graph or the business context graph is used in elements of the other of the technical context graph or the business context graph. The reference <NUM> may be created by providing reference to information that is technical context information or business context information. For example, as illustrated in <FIG> the collection (J) in the technical context graph <NUM> may include an indication that an origin of information included therein is an attribute (K) included in the business context graph <NUM>.

<FIG> is an example depicting a lineage data structure <NUM>. The lineage data structure <NUM> illustrates different collections <NUM> that depict relations <NUM> between different fields <NUM>. Also illustrated are calculators <NUM> to indicate whether the relations <NUM> between the fields include a computer based transformation, calculation or aggregation. In addition, illustrated are transformers <NUM>, which are representative of predetermined constants, formulas or mathematical functions.

<FIG> is an operational flow diagram illustrating example operation of the architecture. Referring to <FIG>, <FIG> and <FIG>, the universal metadata selection circuitry <NUM> may identify and select data sources <NUM> to be sources of metadata and provide transaction flow data <NUM> (<NUM>). The metadata ingestion circuitry <NUM> may ingest the metadata information (<NUM>) and reconcile the data by normalizing the information (<NUM>). The metadata conflict resolution circuitry <NUM> may review the normalized information for conflicting information (<NUM>) using, for example, machine learning and artificial intelligence (Al). The metadata conflict resolution circuitry <NUM> may also catalog all of the data sources from which metadata is obtained (<NUM>). User input regarding identification of additional data sources to add to the catalog may be received (<NUM>), and the operation may return to selecting metadata sources (<NUM>). In addition, the metadata conflict resolution circuitry <NUM> may map the transactions to the data sources (<NUM>) and map the transactions to the data destinations (<NUM>) as part of creating information for the lineage group structure. The catalog of data sources and the source and destination mapping may be stored in the universal metadata repository.

The metadata schema enforcement circuitry <NUM> may also perform schema reconciliation by matching mappers to different schemas to determine which pieces of data bind different schemas together (<NUM>). Schema reconciliation may also include mining data sources for probabilistic values and discrete values for completion of lineage relationships (<NUM>) within the incidence schema. In addition, schema reconciliation may include mining different data sources for data quality related information (<NUM>).

The metadata analytics circuitry <NUM> may identify the key data frames among the transactions for each data source to establish a timeline data flow (<NUM>). It is then determined if all the schemas received from the different data sources are properly identified to allow normalization of the metadata (<NUM>). If not, the metadata analytics circuitry <NUM> may choose a different mapper (<NUM>) and return to matching the different mapper the schema identified as not being properly identified (<NUM>).

Referring now to <FIG>, if all the schemas are properly identified the metadata analytics circuitry <NUM> may generate a timeline flow of the key data frames (<NUM>). The key data frames may be stored in the graph store and the document store with time stamps (<NUM>). The data source fields may be associated with the key data frames (<NUM>) by the metadata analytics circuitry <NUM>. Data field statistics may be stored in the graph store and document store (<NUM>). Data attributes may also be stored in the graph store and the document store (<NUM>). The metadata analytics circuitry <NUM> may generate the data lineage structure using the graph store and the document store (<NUM>). The metadata analytics circuitry <NUM> may leverage the descriptive module <NUM> and the predictive module <NUM> to perform static data analysis (<NUM>) of the normalized metadata across the different data sources, such as lineage data structure analysis and scoring. Based on the static analysis, the metadata analytics circuitry <NUM> identifies gaps in the normalized meta data (<NUM>). If there are gaps in the metadata, the metadata analytics circuitry <NUM> locates a data source(s) that can fill the gap (<NUM>). The metadata analytics circuitry <NUM> selects the metadata source(s) (<NUM>). Metadata information is extracted from the selected metadata data source(s) (<NUM>) and the operation returns to add the selected metadata sources to the catalog (<NUM>).

If no gaps in the data are identified, the metadata analytics circuitry <NUM> may perform dynamic data analysis (<NUM>), such as determining data quality metrics, identifying data quality, and performing data quality scoring. The metadata analytics circuitry <NUM> may determine if a holistic view of the incidence schema is present (<NUM>). In not, the metadata analytics circuitry <NUM> may find source(s) (<NUM>) select metadata source(s) (<NUM>), extract information (<NUM>), etc. If the holistic view is present, the event manager <NUM> may apply predetermined rules and logic that may be triggered on events, conditions, or happenings (<NUM>). The event manager <NUM> may monitor for a quality threshold (<NUM>). If a quality threshold has not been crossed, processing by the data processing circuitry <NUM> may be performed, data may be displayed (<NUM>), and data may be stored in the graph store <NUM> and the document store <NUM> as appropriate (<NUM>).

If a quality threshold has been breached, the event manager <NUM> may generate an alert (<NUM>), such as a data quality alert or a data lineage alert. The event manager <NUM> may identify one or more of the data sources that caused the quality threshold crossing (<NUM>) The event manager may generate a feedback message (<NUM>) and transmit the feedback message to the identified one or more data sources (<NUM>). The operation may then return to mapping the transaction sources (<NUM>).

The architecture <NUM> may perform computer based selection of data sources of interest from a myriad of workflows that might be of interest. Data is ingested from each of the data source(s) that align with a workflow. The ingested data may be normalized and stored in a universal metadata repository. If there is data missing (e.g. gaps) in the normalized information, selection of additional data may be performed. In addition, the architecture may prepare and enrich the normalized data. For example, data may be transformed, normalized, and selectively enriched. Data transformation, normalization and enrichment may be based on rules, machine learning, logic, artificial intelligence, modeling and other computer based analysis and functionality.

Analysis may be performed automatically, such as static analysis and dynamic analysis to automatically create unified data view(s). Static analysis may include lineage data structure generation, analysis and scoring, for example. Data lineage analysis may involve information surrounding what the composition of values can be performed. Data lineage data quality may also be analyzed and automatically enriched using, for example, artificial intelligence (Al) and/or machine learning (ML). Dynamic analysis may include data quality metric identification, generation and analysis. Data may be profiled and information regarding statistics, coverage, data quality rules may be determined automatically by the architecture using, for example, artificial intelligence (Al) and/or machine learning (ML).

The architecture may also provide living reports that may be generated for all aspects of the data represented in the universal metadata repository, including quality and lineage. The universal metadata repository may represent a central location that holds a holistic view that can be queried and automatically refreshes views, such as data quality and data lineage views. Within the universal metadata repository, AI/ML may be employed to resolve data conflicts and verify data integrity. The architecture may use the dynamic analysis to automatically repeat the process, resulting in measuring and grading on live data.

The methods, devices, architectures, processing, circuitry, circuitrys and logic described above may be implemented in many different ways and in many different combinations of hardware and software. For example, all or parts of the circuitrys and other implementations may be circuitry that includes an instruction processor, such as a Central Processing Unit (CPU), microcontroller, or a microprocessor; or as an Application Specific Integrated Circuit (ASIC), Programmable Logic Device (PLD), or Field Programmable Gate Array (FPGA); or as circuitry that includes discrete logic or other circuit components, including analog circuit components, digital circuit components or both; or any combination thereof. The circuitry may include discrete interconnected hardware components or may be combined on a single integrated circuit die, distributed among multiple integrated circuit dies, or implemented in a Multiple Chip Module (MCM) of multiple integrated circuit dies in a common package, as examples.

Accordingly, the circuitry may store or access instructions for execution, or may implement its functionality in hardware alone. The instructions may be stored in a tangible storage medium that is other than a transitory signal, such as a flash memory, a Random Access Memory (RAM), a Read Only Memory (ROM), an Erasable Programmable Read Only Memory (EPROM); or on a magnetic or optical disc, such as a Compact Disc Read Only Memory (CDROM), Hard Disk Drive (HDD), or other magnetic or optical disk; or in or on another machine-readable medium. A product, such as a computer program product, may include a storage medium and instructions stored in or on the medium, and the instructions when executed by the circuitry in a device may cause the device to implement any of the processing described above or illustrated in the drawings.

The implementations may be distributed. For instance, the circuitry may include multiple distinct system components, such as multiple processors and memories, and may span multiple distributed processing systems. Parameters, databases, and other data structures may be separately stored and managed, may be incorporated into a single memory or database, may be logically and physically organized in many different ways, and may be implemented in many different ways. Example implementations include linked lists, program variables, hash tables, arrays, records (e.g., database records), objects, and implicit storage mechanisms. Instructions may form parts (e.g., subroutines or other code sections) of a single program, may form multiple separate programs, may be distributed across multiple memories and processors, and may be implemented in many different ways. Example implementations include stand-alone programs, and as part of a library, such as a shared library like a Dynamic Link Library (DLL). The library, for example, may contain shared data and one or more shared programs that include instructions that perform any of the processing described above or illustrated in the drawings, when executed by the circuitry.

Claim 1:
A method comprising:
in a network connected hardware system:
identifying a pre-defined workflow for analysis;
selecting data sources (<NUM>, <NUM>) that provide input data to the pre-defined workflow;
with communication circuitry (<NUM>):
retrieving source metadata (<NUM>) from the data sources that characterizes the input data to the pre-defined workflow, wherein each of the data sources provides a dataset of the source metadata;
providing the source metadata to repository processing circuitry (<NUM>); and
with the repository processing circuitry:
integrating the source metadata into a universal metadata repository, comprising identifying key data frames in the source metadata, based on data quality metrics analysis and data lineage analysis, the key data frames being stored in the universal metadata repository to represent the source metadata, and routing the source metadata to predetermined queues, said routing comprising: routing the key data frames and including, in attributes associated with the metadata, an indication of where the metadata is stored;
performing data analytics on the input data driven by the universal metadata repository, wherein said performing data analytics comprises: performing static data analysis to update a data lineage structure for the source data, identifying gaps in the source metadata based on said static data analysis, and, in response to identifying gaps in the source metadata, discovering one or more additional data source and extracting additional metadata from the one or more additional data source to update the universal metadata repository; and
determining a data lineage timeline (<NUM>) that comprises key data frames (<NUM>), and depicts a timeline flow of the key data frames (<NUM>) such that a data lineage is tracked at key points in time; and
executing a feedback loop responsive to the data analytics to deliver a source data feedback message (<NUM>) to at least one of the data sources.