SYSTEMS AND METHODS FOR SELF-SERVICE DATA STRUCTURE DISCOVERY

Systems and methods for self-service data structure discovery are disclosed. In one embodiment, a method may include: (1) retrieving, by a data crawler, data from one or more data sources; (2) storing, by the data crawler, the data to a data catalog; (3) receiving, at a search dashboard, a query from a user, the query comprising a search field; (4) submitting, by the search dashboard and using a query tool, the query to a data catalog, wherein the query tool searches the search field by data structure, datatype, field journey, primary key details, and/or multi-value column detail across layers, data lineage controls in the data catalog; (5) receiving, at the search dashboard, a result of the query from the data catalog; and (6) presenting, by the search dashboard, the results.

BACKGROUND OF THE INVENTION

1. Field of the Invention

Embodiments are generally directed to systems and methods for self-service data structure discovery.

2. Description of the Related Art

Consumers seeking to understand the data structure of a database, data event contracts from publishers, or similar must request a time consuming, manual effort to look at code in code repositories, Avro schema, and/or batch file contracts. This is a slow process, at best.

SUMMARY OF THE INVENTION

Systems and methods for self-service data structure discovery are disclosed. In one embodiment, a method may include: (1) retrieving, by a data crawler, data from one or more data sources; (2) storing, by the data crawler, the data to a data catalog; (3) receiving, at a search dashboard, a query from a user, the query comprising a search field; (4) submitting, by the search dashboard and using a query tool, the query to a data catalog, wherein the query tool searches the search field by data structure, datatype, field journey, primary key details, and/or multi-value column detail across layers, data lineage controls in the data catalog; (5) receiving, at the search dashboard, a result of the query from the data catalog; and (6) presenting, by the search dashboard, the results.

In one embodiment, the data crawler retrieves the data periodically.

In one embodiment, the data sources may include one or more of a schema registry, a code repository, and an API store.

In one embodiment, the data may include at least one of event schema, database schema details, database tables, metadata, and API specifications.

In one embodiment, the method may also include identifying, by the data crawler or the query tool, data naming conventions based on relationships among the event schema, the database schema details, and the API specifications.

In one embodiment, the method may also include identifying, by the data crawler or the query tool, data type inconsistencies based on relationships among the event schema, the database schema details, and the API specifications.

In one embodiment, the query may include at least an application identifier, an event identifier, an API identifier, and a data identifier.

According to another embodiment, a system may include: a plurality of data sources; a data crawler computer program executed by a computer processor that may be configured to retrieve data from one or more of the plurality of data sources and to store the data to a data catalog; and an electronic device executing a search dashboard computer program and a query tool computer program, wherein the search dashboard computer program may be configured to receive a query from a user, the query comprising a search field, to submit, using the query tool computer program, the query to the data catalog, wherein the query tool computer program may be configured to search the search field by data structure, datatype, field journey, primary key details, and/or multi-value column detail across layers, data lineage controls in the data catalog, to receive a result of the query from the data catalog, and to present the result.

In one embodiment, the data crawler computer program may be further configured to retrieve the data periodically.

In one embodiment, the data sources may include one or more of a schema registry, a code repository, and an API store.

In one embodiment, the data may include at least one of event schema, database schema details, database tables, metadata, and API specifications.

In one embodiment, the data crawler computer program or the query tool computer program may be further configured to identify data naming conventions based on relationships among the event schema, the database schema details, and the API specifications.

In one embodiment, the data crawler computer program or the query tool computer program may be further configured to identify data type inconsistencies based on relationships among the event schema, the database schema details, and the API specifications.

In one embodiment, the query may include at least an application identifier, an event identifier, an API identifier, and a data identifier.

According to another embodiment, a non-transitory computer readable storage medium, may include instructions stored thereon, which when read and executed by one or more computer processors, cause the one or more computer processors to perform steps comprising: retrieving data from one or more data sources; storing the data to a data catalog; receiving a query from a user, the query comprising a search field, wherein the query may include at least an application identifier, an event identifier, an API identifier, and a data identifiers; submitting the query to a data catalog; searching the data catalog for the search field by data structure, datatype, field journey, primary key details, and/or multi-value column detail across layers, data lineage controls; receiving a result of the query from the data catalog; and presenting the results.

In one embodiment, the data may be retrieved periodically.

In one embodiment, the data sources may include one or more of a schema registry, a code repository, and an API store.

In one embodiment, the data may include at least one of event schema, database schema details, database tables, metadata, and API specifications.

In one embodiment, the non-transitory computer readable storage medium may also include instructions stored thereon, which when read and executed by one or more computer processors, cause the one or more computer processors to perform steps comprising identifying, by the data crawler or the query tool, data naming conventions based on relationships among the event schema, the database schema details, and the API specifications.

In one embodiment, the non-transitory computer readable storage medium may also include instructions stored thereon, which when read and executed by one or more computer processors, cause the one or more computer processors to perform steps comprising identifying data type inconsistencies based on relationships among the event schema, the database schema details, and the API specifications.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

Systems and methods for self-service data structure discovery are disclosed.

Embodiments may implement self-service data discovery engines that may dynamically fetch schema details from a database, tables (e.g., with primary keys information), columns and fields under multi-value column for an application identifier, etc. Embodiments may provide a search capability to find any specific data elements across table, column, multi-value column, etc. Embodiments may also identify the occurrence of specific data elements across multiple tables, application identifiers, etc. Embodiments may also integrate with other sources, including event registries from data publishers, Application Programming Interface (API) registry stores for consumers of the data, etc.

Referring toFIG.1, a self-service data structure discovery engine system is disclosed according to an embodiment. System100may include user electronic device110, such as a computer (e.g., workstation, desktop, laptop, notebook, tablet, etc.), a smart device, an Internet of Things appliance, etc., that may execute search dashboard112.

Search dashboard112may use or include query tool114to pull information from data catalog120. Data Catalog120may be a database/data repository for data gathered from different sources, such as application databases142, real time data exchange event schema and API registry stores144, etc.

Search dashboard112may provide an interface by which a user may enter search criteria for one or more of an application identifier, data exchange event identifier, API identifier, multiple detailed data identifiers, etc. For example, search dashboard112may provide an interface that provides a capability to search a field across various layers, data lineage controls, etc. by data structure, datatype, field journey, primary key details, multi-value column details, etc., and may provide access to data for ease of data governance and audits.

Search dashboard112may interface with data catalog120, which may maintain a catalog of data received from data crawler130. Data crawler130may crawl one or more schema registries140, one or more databases142, and one or more API stores144for data.

For example, data crawler130may periodically (e.g., hourly, daily, weekly, etc.) or on demand pull the information from different sources such as one or more schema registries140, one or more databases142, one or more API stores144, and store the data in data catalog120. Each source (e.g., one or more schema registries140, one or more databases142, one or more API stores144) may have a different plugin, and each plugin may provide an interface for REST API calls. For example, each database142may have a database related plugin to pull schema information, each schema registry140and each API store144may provide APIs to integrate and pull the information, etc.

Data Catalog120may organize the data it receives into a database or data platform. Examples of databases or data platforms may include AWS S3 buckets, Hive, Redshift, Oracle, etc.

One or more schema registries140may maintain schemas from one or more systems of record152, which may maintain the data. One or more systems of record152may stream data to messaging service162, such as Kafka. Examples of data that may be streamed include data exchange events such as customer transactions, balances, etc. Messaging service162may maintain batch files164.

Data from messaging service162may be provided to consumer apps172, which may be consumers of the data. The data may then flow to databases142.

APIs182may provide an access point to data. API store144may maintain APIs182.

FIG.2depicts a method for using self-service data structure discovery engine that is disclosed according to another embodiment.

In step205, a data crawler may periodically (e.g., hourly, daily, weekly, etc.) or as otherwise necessary and/or desired may retrieve data from one or more sources. Examples of sources may include schema registries, databases, code repositories, API stores, etc. For example, the data crawler may use HTTP/API calls to pull event schema from a schema registry, may pull database schema details from a database, may retrieve database tables and metadata from code repositories, and may use API calls to pull API specifications from an API store.

In step210, the data crawler may store the data in a data catalog store, or may update the data in the data catalog store with updated data.

In step215, a computer program, such as a query tool or the data crawler, may identify data naming conventions based on relationships among event schemas (e.g., Avro fields), the database schema details, and the API specifications (e.g., swagger fields). For example, datatype inconsistencies/variations, lineage patterns, duplicity, unknown/unwanted elements at different layers, etc. may also be identified. In one embodiment, machine learning and/or artificial intelligence and other design patterns may be used to establish the relationships.

In step220, a user may submit a query using a dashboard. For example, the user may submit a query. The query may include one or more of an application identifier, event identifier, API identifier, multiple data identifiers, etc. In one embodiment, the search dashboard may provide an interface that provides a capability to search a field across various layers, data lineage controls, etc. by data structure, datatype, field journey, primary key details, multi-value column details, etc.

In step225, a query tool may submit the query to the data in the data catalog store.

The data queries may depend on the database choice (and its native query language) for data catalog. For example, HIVEQL for Hive, SQL for Oracle, S3 Select/Redshift Spectrum for S3, etc. may be used. The query response provides the filtered data to be displayed on dashboard. There may be a default query executed to render the dashboard landing page with overview information.

In step230, the results of the query may be returned to the dashboard. In one embodiment, the data may be returned in a format, such as JSON, that makes it easy to render the data in the search dashboard.

In one embodiment, the results of the query may be used periodically to generate and trigger reports for data lineage, etc. For example, the results may be used as a source of information to determine if there is any deviation in the system design, duplication of data etc.

FIG.3depicts an exemplary computing system for implementing aspects of the present disclosure.FIG.3depicts exemplary computing device300. Computing device300may represent the system components described herein. Computing device300may include processor305that may be coupled to memory310. Memory310may include volatile memory. Processor305may execute computer-executable program code stored in memory310, such as software programs315. Software programs315may include one or more of the logical steps disclosed herein as a programmatic instruction, which may be executed by processor305. Memory310may also include data repository320, which may be nonvolatile memory for data persistence. Processor305and memory310may be coupled by bus330. Bus330may also be coupled to one or more network interface connectors340, such as wired network interface342or wireless network interface344. Computing device300may also have user interface components, such as a screen for displaying graphical user interfaces and receiving input from the user, a mouse, a keyboard and/or other input/output components (not shown).

Hereinafter, general aspects of implementation of the systems and methods of embodiments will be described.

In one embodiment, the processing machine may be a cloud-based processing machine, a physical processing machine, or combinations thereof.

As noted above, the processing machine used to implement embodiments may be a general-purpose computer. However, the processing machine described above may also utilize any of a wide variety of other technologies including a special purpose computer, a computer system including, for example, a microcomputer, mini-computer or mainframe, a programmed microprocessor, a micro-controller, a peripheral integrated circuit element, a CSIC (Customer Specific Integrated Circuit) or ASIC (Application Specific Integrated Circuit) or other integrated circuit, a logic circuit, a digital signal processor, a programmable logic device such as a FPGA (Field-Programmable Gate Array), PLD (Programmable Logic Device), PLA (Programmable Logic Array), or PAL (Programmable Array Logic), or any other device or arrangement of devices that is capable of implementing the steps of the processes disclosed herein.

The processing machine used to implement embodiments may utilize a suitable operating system.

To explain further, processing, as described above, is performed by various components and various memories. However, it is appreciated that the processing performed by two distinct components as described above, in accordance with a further embodiment, may be performed by a single component. Further, the processing performed by one distinct component as described above may be performed by two distinct components.

In a similar manner, the memory storage performed by two distinct memory portions as described above, in accordance with a further embodiment, may be performed by a single memory portion. Further, the memory storage performed by one distinct memory portion as described above may be performed by two memory portions.

Any suitable programming language may be used in accordance with the various embodiments. Also, the instructions and/or data used in the practice of embodiments may utilize any compression or encryption technique or algorithm, as may be desired. An encryption module might be used to encrypt data. Further, files or other data may be decrypted using a suitable decryption module, for example.

It will be readily understood by those persons skilled in the art that embodiments are susceptible to broad utility and application. Many embodiments and adaptations of the present invention other than those herein described, as well as many variations, modifications and equivalent arrangements, will be apparent from or reasonably suggested by the foregoing description thereof, without departing from the substance or scope.