Displaying data lineage using three dimensional virtual reality model

An example system includes a processor to receive data lineage including a plurality of levels, and a configuration. The processor is to also build a three dimensional (3D) virtual reality (VR) model including a first floor based on data lineage content corresponding to a first level of the plurality of levels and the configuration. The processor is to further display a view of the 3D VR model.

BACKGROUND

The present techniques relate to displaying a data lineage. More specifically, the techniques relate to displaying a data lineage using a three dimensional virtual reality model.

SUMMARY

According to an embodiment described herein, a system can include a processor to receive data lineage comprising a plurality of levels, and a configuration. The processor can also further build a three dimensional (3D) virtual reality (VR) model comprising a first floor based on data lineage content corresponding to a first level of the plurality of levels and the configuration. The processor can also display a view of the 3D VR model.

According to another embodiment described herein, a method can include receiving, via a processor, data lineage comprising a plurality of levels, and a configuration. The method can also further include building, via the processor, a three dimensional (3D) virtual reality (VR) model comprising a first floor based on data lineage corresponding to a first level of the plurality of levels and the configuration. The method can also include displaying, via the processor, a view of the 3D VR model.

According to another embodiment described herein, a computer program product for generating three dimensional virtual reality models can include computer-readable storage medium having program code embodied therewith. The computer readable storage medium is not a transitory signal per se. The program code is executable by a processor to cause the processor to receive data lineage comprising a plurality of levels, and a configuration. The program code can also cause the processor to generate data lineage content based on the data lineage. The program code can also cause the processor to build a three dimensional (3D) virtual reality (VR) model comprising a first floor based on data lineage content corresponding to a first level of the plurality of levels and the configuration. The program code can also cause the processor to display a view of the 3D VR model.

DETAILED DESCRIPTION

Extract, Transform, Load (ETL) systems can move and change data between different systems in enterprise and data lineage systems and can display how the data was moved and transformed from one source to another. When there are issues in the data in one source, data lineage tools can be used in order to find the cause for a particular issue.

However, since there may be multiple hierarchical data assets and a corresponding history of data movement in each level, diagnosing issues in 2d views of data lineage may be both difficult and time consuming. As used herein, data assets may include databases, schemas, tables, fields, etc. For example when a user encounters a wrong value in a column in a report and wants to track the cause of the issue, the user may apply data lineage in order to understand where the data came from and identify the root cause for a value being wrong. Currently, data lineage analysis may involve the use of two dimensional lineage graphs for each asset level in order to investigate the cause of wrong data in a particular column, which can be a difficult, time consuming, and tedious process.

According to embodiments of the present techniques a processor may receive data lineage including a plurality of levels and a configuration. For example, the levels may correspond to different data asset levels such as databases, schemas, tables, fields, etc., of a data asset. In some examples, the configuration may include default settings for a visual representation of the levels. The processor may build a three dimensional (3D) virtual reality (VR) model including a first floor based on data lineage content corresponding to a first level of the plurality of levels and the configuration. For example, the first level may be a set of databases. The processor may also display a view of the 3D VR model. The processor may further update the view of the VR model in response to detecting a request for existing data from the first level. The processor may then fetch additional data from the first level in response to detecting a request for additional data from the first level. The processor may then build additional floors including a second floor in the VR model based on a second level of the plurality of levels in response to detecting a request for a change in context. In some examples, the processor may prefetch data from the data lineage content based on a predicted request. For example, the processor can predict a request based on a history of past requests. The processor may then fetch data that may not be shown, but may probably be shown soon, in order to improve user experience. Thus, the present techniques may enable users to view multiple aspects of data lineage at the same time and within context. For example, users may be able to see a hierarchy and a history, among other aspects of the data lineage simultaneously. The present techniques may combine different levels of data hierarchy and enable users to navigate easily and quickly between different views of the data lineage. For example, the view present to a user may be updated as the context is changed by walking through the 3D VR model. Moreover, the data lineage content used to present the view may be retrieved as the view is being updated. Thus, the techniques described herein may be more efficient than a system using a 3D VR model including all data lineage content. Moreover, settings of the configuration can be changed based on a particular scenario. For example, the scenario may be a temporal issue in the data lineage. Thus, the techniques may enable a customized way of viewing a particular data lineage based on a suspected temporal issue. In addition, the present techniques may also use a default configuration to provide operability without any prerequisite configuration. Thus, the overhead and resources that might otherwise be used to preconfigure such a system may be conserved. In some examples, the techniques can thus be used to diagnose temporal issues in data lineage using a 3D VR model. For example, the 3D VR model may enable contextualized data lineage history inspection and a deeper inspection into data asset hierarchy with actual data. For example, the actual data may be values of different elements of the data assets. The techniques described herein thus enable a user to see a data lineage and its history and hierarchy in a quick and clear manner and adapt a view of the issue according to a suspected root cause of a temporal issue. In some examples, the root cause of a data lineage issue can be identified using the techniques described herein and modified accordingly to resolve the temporal issue.

In some scenarios, the techniques described herein may be implemented in a cloud computing environment. As discussed in more detail below in reference to at leastFIGS. 3, 4, and 5, a computing device configured to display data lineage using a three dimensional virtual reality model may be implemented in a cloud computing environment. It is understood in advance that although this disclosure may include a description on cloud computing, implementation of the teachings recited herein are not limited to a cloud computing environment. Rather, embodiments of the present invention are capable of being implemented in conjunction with any other type of computing environment now known or later developed.

Characteristics are as follows:

Service Models are as follows:

Deployment Models are as follows:

FIG. 1is a block diagram of an example system that can display data lineage using a three dimensional virtual reality model. The system is generally referred to using the reference number100and can be implemented at least in part using the computing device300ofFIG. 3below.

The example system100includes a data lineage server102, an administrator client104, and a user client106. The data lineage server102is coupled to the administrator client104and the user client106via connections108,110, respectively. In some examples, the user client106may include a virtual reality (VR) display, such as VR glasses or VR goggles, or the like. The data lineage server102further includes a configuration engine112, a metadata manager114, a virtual reality (VR) engine116, and a data lineage transformer118.

In the example system100, an administrator client104may receive one or more settings from an administrator and send the settings to the data lineage server102. For example, the settings may configure how each data asset, type of data, and each period in the history of the data lineage is to appear in the three dimensional virtual reality model. The settings may also include how navigation is to be performed between different views in the three dimensional virtual reality model. For example, the settings may specify actions that may cause movement between different views. In some examples, the settings may per user settings, per lineage type settings, etc. For example, per user settings may include styles of presenting the data and a set of actions for transforming views for a particular user. In some examples, per lineage type settings may include styles of presenting the data and a set of actions for transforming views for a particular data lineage type. For example, the data lineage type may be a one-to-one lineage type or a one-to-many lineage type.

As shown inFIG. 1, the data lineage server102may receive settings from the administrator client104. The configuration engine112may then be used to configure how each asset and data type appears in the three dimensional virtual reality model based on the received settings. In some examples, the configuration engine112may be used to customize a view for each user based on an area and an intensity of data and its hierarchy and history. For example, a higher Intensity of data with many assets in each level may result in a model that may look different than a lower intensity of data with few assets in each level. An area of data, as used herein, refers to a particular data asset type. For example, a database area may have one view and files area may have a different view. In some examples, the configuration engine112may also have default settings to use in the case that the data lineage server102does not receive any settings from an administrator client102. For example, the default settings may include a default configuration of how to map data and how navigation in the 3D VR model is to be performed. The configuration engine112may also be used to configure how different views may be navigated and which actions cause movement between views based on the received setting or the default settings. For example, the configuration engine112can be used to configure how a user moves in the 3D VR model in order to adapt the 3D VR model to a specific user with a specific scenario. For example, the scenario may be a temporal issue in the data lineage. The configuration may be sent to the data lineage transformer118.

The metadata manager114can receive a data lineage and provide data lineage content. In some examples, the data lineage may include a history of the data assets. For example, the data lineage content may include information about data flows between data assets. In some examples, the data lineage may be provided by the user client106. In some examples, the metadata manager114can provide the data lineage transformer118with data lineage content corresponding to one or more levels of data lineage in response to receiving a request from the data lineage transformer118for the data lineage content.

The VR engine116can programmatically and dynamically build a three dimensional virtual reality model based on the data lineage content and the configuration. For example, the VR engine116can dynamically build the 3D VR model while part of the 3D VR model is being used. In some examples, the VR engine116can include any suitable 3D modeling software that supports dynamic building of 3D VR models.

The data lineage transformer118can receive the configuration from the configuration engine112and monitor actions performed by the user client106in the 3D VR model. In some examples, the data lineage transformer118can detect an action from the user client106. For example, the action may be a user request to see data lineage of an asset or between specific assets. The data lineage transformer118can pull the current information form the metadata manager114and can present the information to the user client106as virtual reality model according to the configuration received from the configuration engine112. Thus, the data lineage transformer118may operate as a bridge between the user client106and the VR engine116that transforms actions taken in the 3D VR model and transforms the actions into requests for additional data from the data lineage. For example, the actions may be user movement, rotation, or other change in point of view. The data lineage transformer118may then present the additional data to the VR engine116to be presented as 3D VR elements in the 3D VR model at the user client110.

In some examples, the user client106can be used to navigate between the assets and see how the data flows between the assets. For example, the user client106can be used to request a change of a current view of a current flow and have a view of a history of flows between the assets. The data lineage transformer118can pull the history of the flow between the assets from a metadata repository and present the history via the user client106in a manner that may enable a user to see and compare the assets. In some examples, the data lineage transformer118can pull data from the metadata repository iteratively. For example, data may be pulled from the metadata repository as it is to be used.

In some examples, the user client106may thus be used to dive into a hierarchy of assets that the data was flowing such that relevant data is pulled from the metadata repository and interpreted by the data lineage transformer118into the virtual reality model. For example, a first view may present data flows between databases. The user client106may then be used to then dive deeper into schema flows and table views, etc. In some examples, a view may present different levels at the same time. In some examples, for each level of data assets and history range, the user client106can be used to inspect the actual data that is flowing between the assets as grouped by type, and then the view can be updated to present each actual transaction separately. The user client106may thus be used to navigate a path towards a cause of some data lineage issue. For example, the issue may be a temporal issue in the data lineage. In some examples, the user client106can be used to mark a way in the virtual way in order to report that path for the cause of the issue. The user client106can then be used to save a state in the virtual reality model so that a user may be able to come back to the saved state later. The user client106may thus be used to save set of virtual actions performed as a set in order to save time in future investigations. For example, the set of virtual actions and saved states may be sent to the configuration engine112and saved for later use.

In one example, the 3D VR model generated by the VR engine116may be a representation of a user in a transparent building with many floors corresponding to levels in the data lineage. In some examples, the floors may represent a hierarchy of assets. For example, the first floor may represent a database's fields, the second floor may represent a database's tables, and a third floor may represent a database's schemas. Each floor may also have many rooms and hallways. In some examples, each hallway may represent a flow of data between elements at a specific level. For example, the flow of data may be from a data column, to a data stage job field, to a report column. In some examples, while the user client106is used to navigate this hallway, the view may present these elements as items such as doors to rooms that may contain data for each corresponding element. In some examples, the user client106can be used to open the door and see more details about the corresponding element. In some examples, the user client106can be used to turn left between the doors and view an older history of the data lineage or turn right to return to a newer data lineage history. In some examples, the user client106can be used to jump to the next floor in order to look at data lineage in other level of objects. For example, the objects can include a table, report, job, etc. The user client106may then be used to walk in a hallway representing a second level of data lineage with the same convention of looking on history by navigating to left and right.

In some examples, the user client116can be used to walk in a path with different levels in the same path. For example, user could virtually walk between levels and see objects in different levels one after another. In some examples, the user may see database field first and then database schema, and then database table, etc. In some examples, while on the same floor, a user may see objects of the same type. In some examples, a user can wear VR glasses that can allow the user client116to tune to an asset level for each item in the data lineage. When the user moves back and forth the data remains. In some examples, the user client106can be used to look up and view items in upper levels of an asset. For example, if the asset is a column, then the user client116can be used to view the table, schema, database, etc. Likewise, the user client106can be used to look down to view lower levels. For example, if the asset is a database, then the lower levels may include a schema or table column. In some examples, the user client106may also be used to view a history of an element as a hallway, where every door is a history asset, instead of jumping one by one between history assets.

Thus, for example, a user may encounter a wrong value in a column in a data report and may thus want to track the cause of the issue in order to fix the underlying cause. The techniques described herein may thus be used in order to understand where the data is coming from and what the root cause is for the wrong value in the data report. In some examples, the user can use the techniques described herein to browse through views of a generated VR model and see that the current data is coming from some database, which may appear issue free. In some examples, the user may then look into in the history of flow via the user client106and see that the data was previously coming from the same database. In some examples, the user can then browse deeper into the actual field in the database that the data was taken from and may see that until recently that field was updated from some Job, but the Job changed to another Job. In addition, when the user looks at the actual value, the user may immediately visually see a current value is in Euros (EUR), while the value was previously US dollars (USD). The user can thus conclude that the change to the job caused the issue and fix that job. In some examples, the user may then track the fixed job to see if the change fixes the issue.

It is to be understood that the block diagram ofFIG. 1is not intended to indicate that the system100is to include all of the components shown inFIG. 1. Rather, the system100can include fewer or additional components not illustrated inFIG. 1(e.g., additional clients, servers, connections, engines, etc.).

FIG. 2is a process flow diagram of an example method that can display data lineage using a three dimensional virtual reality model. The method200can be implemented with any suitable computing device, such as the computing device300ofFIG. 3. For example, the method can be implemented via the processor302of computing device300.

At block202, the processor receives data lineage including a plurality of levels, and a configuration. In some examples, the data lineage may include a history of data assets. For example, the plurality of levels may include one or more data flows between data assets. In some examples, the configuration may include default or customized settings for displaying the data lineage. For example, the settings may be per user settings or per data lineage settings.

At block204, the processor builds a three dimensional (3D) virtual reality (VR) model including a first floor based on data lineage corresponding to a first level of the plurality of levels and the configuration. For example, the processor can execute the data lineage to generate data lineage content used to build the 3D VR model. In some examples, the data lineage content may include specific values of one or more data assets. For example, the first floor may represent the data lineage in the database field level, a second floor may represent the database table level, a third floor may represent the lineage at the database schema level, and a fourth floor represents the lineage at the database level. In some examples, any suitable virtual reality software development kit may be used to build the 3D VR model.

At block206, the processor displays a view of the 3D VR model. For example, the view may present data lineage content according to the configuration. In some examples, the view may present a first floor of a virtual building with a number of hallways representing data flow.

At block208, the processor updates the view of the 3D VR model in response to detecting a request for existing data from the first level. For example, the existing data may be a value of a data asset that has been already calculated but not yet presented. In some examples, the data asset may be located behind a door in a hallway. In some examples, upon opening the door from the hallway or moving into the room, the values of data assets within the room can be presented.

At block210, the processor fetches additional data from the first level in response to detecting a request for additional data from the first level. For example, the additional data may be historical data that is requested by looking to the left. In some examples, the processor can execute data lineage to generate data lineage content corresponding to the historical data and present the data in an updated view.

At block212, the processor builds additional floors including a second floor in the 3D VR model in response to detecting a request for a change in context. For example, the processor may build the second floor based on a second level of the plurality of levels. For example, the second floor may represent one or more data flows between data assets at a different level of the data lineage. For example, the level may be different databases, schemas, tables, fields, etc., of a data asset.

At block214, the processor prefetches data from the data lineage based on a predicted request. For example, the processor may save and store previous data request and predict future requests based on the stored data requests. Thus, the data may be readily available to be used when a request for the prefetched data is received. The processor may then request data corresponding to assets that are not presented based on previous information or any suitable heuristics in order to render better the VR model and provide a better user experience.

The process flow diagram ofFIG. 2is not intended to indicate that the operations of the method200are to be executed in any particular order, or that all of the operations of the method200are to be included in every case. Additionally, the method200can include any suitable number of additional operations. For example, the method may include detecting a root cause of a temporal issue and modifying a value of a data element corresponding to the detected root cause. The processor may then track the modified data element to determine whether the temporal issue is resolved.

With reference now toFIG. 3, an example computing device can display data lineage using a three dimensional virtual reality model. The computing device300may be for example, a server, a network device, desktop computer, laptop computer, tablet computer, or smartphone. In some examples, computing device300may be a cloud computing node. Computing device300may be described in the general context of computer system executable instructions, such as program modules, being executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, and so on that perform particular tasks or implement particular abstract data types. Computing device300may be practiced in distributed cloud computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media including memory storage devices.

The computing device300may include a processor302that is to execute stored instructions, a memory device304to provide temporary memory space for operations of said instructions during operation. The processor can be a single-core processor, multi-core processor, computing cluster, or any number of other configurations. The memory304can include random access memory (RAM), read only memory, flash memory, or any other suitable memory systems.

The processor302may be connected through a system interconnect306(e.g., PCI®, PCI-Express®, etc.) to an input/output (I/O) device interface308adapted to connect the computing device300to one or more I/O devices310. The I/O devices310may include, for example, a keyboard and a pointing device, wherein the pointing device may include a touchpad or a touchscreen, among others. The I/O devices310may be built-in components of the computing device300, or may be devices that are externally connected to the computing device300.

The processor302may also be linked through the system interconnect306to a display interface312adapted to connect the computing device300to a display device314. The display device314may include a display screen that is a built-in component of the computing device300. The display device314may also include a computer monitor, television, or projector, among others, that is externally connected to the computing device300. In addition, a network interface controller (NIC)316may be adapted to connect the computing device300through the system interconnect306to the network318. In some embodiments, the NIC316can transmit data using any suitable interface or protocol, such as the internet small computer system interface, among others. The network318may be a cellular network, a radio network, a wide area network (WAN), a local area network (LAN), or the Internet, among others. An external computing device320may connect to the computing device300through the network318. In some examples, external computing device320may be a remote client320. In some examples, external computing device320may be a cloud computing node.

The processor302may also be linked through the system interconnect306to a storage device322that can include a hard drive, an optical drive, a USB flash drive, an array of drives, or any combinations thereof. In some examples, the storage device may include a receiver module324, a builder module326, a view displayer module328, a view updater module330, a fetcher module332, and a prefetcher module334. The receiver module324can receive data lineage for a plurality of levels and a configuration. In some examples, the receiver module324can generate data lineage content based on the data lineage. The builder module326can then build a three dimensional (3D) virtual reality (VR) model including a first floor based on data lineage content corresponding to a first level of the plurality of levels and the configuration. The view displayer module328can display a view of the 3D VR model. The view updater module330can update the view of the VR model in response to detecting a request for existing data from the first level. The fetcher module332can fetch additional data from the first level in response to detecting a request for additional data from the first level. In some examples, the builder module326can also build a second floor in the VR model based on a second level of the plurality of levels in response to detecting a request for a change in context. The prefetcher module334can prefetch data from the data lineage content based on a predicted request. In some examples, the prefetcher module334can predict a request for data based on one or more previous requests. The prefetcher module334can then request data corresponding to assets that are not yet presented based on previous information or any suitable heuristics in order to render better the VR model and to provide a better user experience.

It is to be understood that the block diagram ofFIG. 3is not intended to indicate that the computing device300is to include all of the components shown inFIG. 3. Rather, the computing device300can include fewer or additional components not illustrated inFIG. 3(e.g., additional memory components, embedded controllers, modules, additional network interfaces, etc.). Furthermore, any of the functionalities of the receiver module324, the builder module326, the view displayer module328, the view updater module330, the fetcher module332, and the prefetcher module334, may be partially, or entirely, implemented in hardware and/or in the processor302. For example, the functionality may be implemented with an application specific integrated circuit, logic implemented in an embedded controller, or in logic implemented in the processor302, among others. In some embodiments, the functionalities of the receiver module324, the builder module326, the view displayer module328, the view updater module330, the fetcher module332, and the prefetcher module334, can be implemented with logic, wherein the logic, as referred to herein, can include any suitable hardware (e.g., a processor, among others), software (e.g., an application, among others), firmware, or any suitable combination of hardware, software, and firmware.

Referring now toFIG. 4, an illustrative cloud computing environment400is depicted. As shown, cloud computing environment400comprises one or more cloud computing nodes402with which local computing devices used by cloud consumers, such as, for example, personal digital assistant (PDA) or cellular telephone404A, desktop computer404B, laptop computer404C, and/or automobile computer system404N may communicate. Nodes402may communicate with one another. They may be grouped (not shown) physically or virtually, in one or more networks, such as Private, Community, Public, or Hybrid clouds as described hereinabove, or a combination thereof. This allows cloud computing environment400to offer infrastructure, platforms and/or software as services for which a cloud consumer does not need to maintain resources on a local computing device. It is understood that the types of computing devices404A-N shown inFIG. 4are intended to be illustrative only and that computing nodes402and cloud computing environment400can communicate with any type of computerized device over any type of network and/or network addressable connection (e.g., using a web browser).

Virtualization layer502provides an abstraction layer from which the following examples of virtual entities may be provided: virtual servers; virtual storage; virtual networks, including virtual private networks; virtual applications and operating systems; and virtual clients. In one example, management layer504may provide the functions described below. Resource provisioning provides dynamic procurement of computing resources and other resources that are utilized to perform tasks within the cloud computing environment. Metering and Pricing provide cost tracking as resources are utilized within the cloud computing environment, and billing or invoicing for consumption of these resources. In one example, these resources may comprise application software licenses. Security provides identity verification for cloud consumers and tasks, as well as protection for data and other resources. User portal provides access to the cloud computing environment for consumers and system administrators. Service level management provides cloud computing resource allocation and management such that required service levels are met. Service Level Agreement (SLA) planning and fulfillment provide pre-arrangement for, and procurement of, cloud computing resources for which a future requirement is anticipated in accordance with an SLA.

The present techniques may be a system, a method or computer program product. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present invention.

Referring now toFIG. 6, a block diagram is depicted of an example tangible, non-transitory computer-readable medium600that can display data lineage using a three dimensional virtual reality model. The tangible, non-transitory, computer-readable medium600may be accessed by a processor602over a computer interconnect604. Furthermore, the tangible, non-transitory, computer-readable medium600may include code to direct the processor602to perform the operations of the method200ofFIG. 2above.

The various software components discussed herein may be stored on the tangible, non-transitory, computer-readable medium600, as indicated inFIG. 6. For example, a receiver module606includes code to receive data lineage including a plurality of levels, and a configuration. In some examples, the data lineage may include a history of data assets. In some examples, the receiver module606may also include code to generate data lineage content based on the data lineage. A builder module608includes code to build a three dimensional (3D) virtual reality (VR) model including a first floor based on data lineage content corresponding to a first level of the plurality of levels and the configuration. For example, the configuration may include per user settings or per data lineage settings related to how data lineage content is to be presented. The builder module608also includes code to build a second floor in the VR model based on a second level of the plurality of levels in response to detecting a request for a change in context. A view displayer module610includes code to display a view of the 3D VR model. A view updater module612includes code to update the view of the VR model in response to detecting a request for existing data from the first level. A fetcher module614includes code to fetch additional data from the first level in response to detecting a request for additional data from the first level. A prefetcher module616includes code to prefetch data from the data lineage content based on a predicted request. In some examples, the prefetcher module616can include code to predict a request for data based on one or more previous requests. It is to be understood that any number of additional software components not shown inFIG. 6may be included within the tangible, non-transitory, computer-readable medium600, depending on the particular application.