Automated data enrichment and signal detection for exploring dataset values

One or more datasets are received by a data wrangling module and wrangled into a form that is computationally actionable by a user. At least some data from the one or more datasets are enriched by one or more data enrichment modules to generate an enriched form of at least some data corresponding to the one or more datasets that is computationally actionable by the user. The one or more datasets and the enriched form of the at least some data are processed by a signal detection module to identify relationships, anomalies, and/or patterns within the one or more datasets.

TECHNICAL FIELD

The subject matter described herein relates to techniques for automated data enrichment and signal detection which can be used to efficiently evaluate almost any type of dataset generated by any industry.

BACKGROUND

According to various estimates, 2.5 quintillion bytes of data are created every day. For instance, in every minute of every day, Apple® users download more than 51,000 apps, Netflix® subscribers stream more than 77,000 hours of video, Twitter® users send more than 350,000 tweets, and Facebook® users “like” more than 4 million posts. Each of these actions by users generates data. The data can be transactional data, i.e. data with a timestamp which has been generated as the result of a transaction conducted by the user, or otherwise generated by activity, outside of any formal transaction.

Even with all of the advancements in big data technologies, it can often be very difficult to understand and evaluate, let alone use, this data. Traditional data evaluation methodologies often rely on a complete understanding of the data being evaluated as well as a specific objective for the evaluation. What is needed is a system and method for evaluating datasets across multiple objectives covering the breadth of a business with limited understanding of the data being evaluated, to provide data-based evidence to the business which informs the creation of various types of actionable analytics.

SUMMARY

Systems and methods are presented herein, which allow for evaluation of datasets across multiple objectives covering the breadth of a business as well as with limited understanding of the data being evaluated. These systems and methods provide data-based evidence to the business which enables the creation of various types of analytics, much of which can be actionable.

In one aspect, a system for exploring value of one or more datasets from one or more data sources is provided. The system includes a data wrangling module connected with the one or more data sources to receive the one or more datasets. The data wrangling module is configured to wrangle the one or more datasets into a form that is computationally actionable by a user. The system further includes one or more data enrichment modules connected with the data wrangling module and/or at least one of the one or more data sources. Each of the one or more data enrichment modules is configured to enrich at least some data from the one or more datasets to generate an enriched form of at least some data corresponding to the one or more datasets that is computationally actionable by the user. The system further includes a signal detection module connected with the data wrangling module, the data enrichment module, and/or the one or more data sources. The signal detection module is configured to process the one or more datasets and the enriched form of at least some data, to identify relationships, anomalies, and/or patterns within the one or more datasets.

In other aspects, a method of exploring value of one or more datasets from one or more data sources is provided. The method includes receiving, by a data wrangling module connected with the one or more data sources, the one or more datasets, the receiving being configured to wrangle the one or more datasets into a form that is computationally actionable by a user. The method further includes enriching, by one or more data enrichment modules connected with the data wrangling module and/or at least one of the one or more data sources, at least some data from the one or more datasets to generate an enriched form of at least some data corresponding to the one or more datasets that is computationally actionable by the user. The method further includes processing, by a signal detection module connected with the data wrangling module, the data enrichment module, and/or the one or more data sources, the signal detection module being configured to process the one or more datasets and the enriched form of at least some data, to identify relationships, anomalies, and/or patterns within the one or more datasets.

DETAILED DESCRIPTION

This document describes systems and methods which allows for evaluation of datasets for multiple objectives covering a breadth of businesses as well as with limited understanding of the data being evaluated. In some implementations, the systems and methods provide data-based evidence to the business which informs the creation of various types of actionable analytics.

The systems and methods described herein utilize technology that is configured for receiving and consuming different types of data, potentially from multiple and usually different sources, wrangling the data, enriching the data and then automatically detecting signals of potential interest. The data wrangling, enrichment and the signal detection algorithms are each flexible, to allow for a user to define the configurations of each. This configuration is done through a user interface, such as, for example, a pre-configured application in FICO® Xpress Insight® to allow easy access to the platform as well as visual interactions with the data.

FIG. 1illustrates a system100for exploring value of one or more datasets from one or more data sources102. The system includes a data wrangling module104connected with the one or more data sources102to receive the one or more datasets, the data wrangling module being configured to wrangle the one or more datasets into a form that is computationally actionable by a user;

The data wrangling module104includes a number of sub-modules or sub-processes configured for cleaning the data, joining and/or merging different datasets, aggregating the data at desired or user-configurable levels, and combining and/or splitting fields of the data appropriately, etc. All of this is done in a visual manner, such as, for example, through the data wrangler104in the FICO® Analytics WorkBench®, to make it easy for the user to interact with the data and manipulate it as necessary. The system100can have a separate data wrangling module104as part of the platform, or can integrate with an external data wrangling tool.

The system100includes one or more data enrichment modules106connected with the data wrangling module104and/or at least one of the one or more data sources102. Each of the one or more data enrichment modules106are configured to enrich at least some data from the one or more datasets to generate an enriched form of at least some data corresponding to the one or more datasets that is computationally actionable by the user. Each data enrichment module106includes one or more sub-modules or sub-processes to enrich different types of data. Examples of different sub-modules for data enrichment include a transactional data enrichment sub-module, a network data enrichment sub-module, and an unstructured data enrichment sub-module.

The transaction data enrichment sub-module enables creation of a large number of characteristics through comprehensive pre-defined templates. The transaction data enrichment sub-module allows users to configure events and metrics of interest on the transaction. It also allows users to select the different time intervals as well as templates which are relevant for the use case.

The network data enrichment sub-module enables detection of networks across datasets using the data available. These networks, depending on the data available, could be as simple as connecting individuals that share a phone number/email or individuals who transact with each other, or can be complex such as connecting medical providers who share more than a certain number of patients. The network data enrichment sub-module, through a user interface, allows users to configure networks by specifying linking fields and attributes as well other network features such as link strengths, etc. Once the networks have been detected, characteristics that summarize the information contained within these networks can be configured and generated in the module to enrich the original data.

The unstructured data enrichment sub-module can handle different types of unstructured data. For example, an unstructured text data module can help find keywords and concepts of relevance across unstructured text, and then generate characteristics based on those to enrich the one or more data sources102. These characteristics can include, without limitation, a frequency of the keywords, a rate of the appearance of the keywords, etc.

In various implementations, a system100can include one or more of the several different data enrichment modules106described above, to handle still further different data types, such as a dataset related to a time series, audio data, etc. Furthermore, a system100can include data enrichment modules106that are associated with specific types of enrichment techniques, such as a time series, or libraries, such as a credit card fraud characteristic library, as examples.

The different data enrichment modules106of the system can run in parallel, generating independent datasets. Alternatively, an enriched dataset from one of the data enrichment modules (or sub-modules) can be used as an input to one or more other data enrichment modules106. For example, output from the transaction data enrichment sub-module can be used within the network data enrichment sub-module to aggregate the transaction-based characteristics across the network, or an audio-oriented data enrichment module may be configured to transcribe audio data to text, and then leverage the unstructured text data enrichment module for characteristics generation.

The enriched dataset is then processed for signal detection by a signal detection module108of the system100. The signal detection module108runs calculations that are optimized for a particular purpose to identify any of relationships, anomalies, monotonic and/or non-monotonic patterns within characteristics (or combinations thereof) against each of the multiple objectives. In some exemplary implementations, a base signal detection formula that allows for detection of different types of signals with minimal or no tweaks can be expressed generally as:

strength of relationship=log⁢{P⁡(xm,yn)P⁡(xm)*P⁡(yn)}

The above formula identifies relationships when values are sorted in decreasing order, but identifies outliers when values are sorted in increasing order. In addition, positive values signify a characteristic value occurring more frequently than random with an objective value, whereas negative values identify characteristic values that occur rarely with an objective value as compared to random.

A simple curve fitting over the strength also identifies monotonic and non-monotonic patterns in the data. In addition, different metrics over the different characteristics and objective values also inform the signal detection process. In an exemplary implementation, a curve fitting over simple average of the characteristics for each objective value can also be used to identify patterns within the data.

The system100further includes a user interface which includes a visualization layer110configured for analyzing different types of signals detected. The visualization layer110creates the graphs and visual graphics, which enables users to observe the signals in one or more of multiple views, i.e. through tables, graphs, and/or maps for geography-based characteristics, etc. It allows users to interact with the signals through multiple functionalities such as filters, sorting options, drill downs, etc.

The modular design of the system allows it to be flexible and the different modules can be used at different points of the process as necessary. For example, in some scenarios it may be useful to do some part of the enrichment before data wrangling.

The system100preferably leverages a big data platform that facilitates working with large datasets in keeping with current requirements. For example, the underlying raw data can be stored in a distributed file system and the calculations done via parallel processing. However, the innovative design with the different modules kept loosely integrated with each other, as separate applications, still capable of communicating with one another, within the same software platform, will allow the incorporation of the latest technologies quickly as individual modules can be swapped out for whatever is the best in class, which can mean open source, commercially available, or custom-built. Data can be consumed and outputted independently at any module, and hence the users can choose to use the full platform or individual modules while performing the other tasks in external tools.

FIG. 2is a flowchart of a method200of exploring value of one or more datasets from one or more data sources. At202, a data wrangling module connected with the one or more data sources receives the one or more datasets. At204, the data wrangling module is configured to wrangle the one or more datasets into a form that is computationally actionable by a user. At206, one or more data enrichment modules connected with the data wrangling module and/or at least one of the one or more data sources enriches at least some data from the one or more datasets to generate an enriched form of at least some data corresponding to the one or more datasets that is computationally actionable by the user. At208, a signal detection module connected with the data wrangling module, the data enrichment module, and/or the one or more data sources processes the one or more datasets and the enriched form of at least some data, to identify relationships, anomalies, and/or patterns within the one or more datasets.