System and method transforming source data into output data in big data environments

A system may receive a request to derive an output variable from a source variable. The request may include proposed logic to derive the output variable from the source variable. The system may then compare the proposed logic to existing logic to determine the proposed logic is new. In response to the proposed logic being new, the system may generate transformation code configured to execute the proposed logic. The system may further schedule the transformation code for execution at a predetermined time, and then execute the transformation code to generate data for the output variable.

FIELD

The present disclosure relates to systems and methods for transforming input data into the desired output variables for ingestion in big data storage formats.

BACKGROUND

Large data sets may exist in various sizes and organizational structures. With big data comprising data sets as large as ever, the volume of data collected incident to the increased popularity of online and electronic transactions continues to grow. For example, billions of records (also referred to as rows) and hundreds of thousands of columns worth of data may populate a single table. The large volume of data may be collected in a raw, unstructured, and undescriptive format in some instances. However, traditional relational databases may not be capable of sufficiently handling the size of the tables that big data creates.

As a result, the massive amounts of data in big data sets may be stored in numerous different data storage formats in various locations to service diverse application parameters and use case parameters. Data variables resulting from complex data transformations (e.g., model scores, risk metrics, etc.) may be central to deriving valuable insight from data driven operation pipelines. Many of the various data storage formats use transformations to convert input data into output variables. These transformations are typically hard coded into systems. As a result, retroactively determining the evolution of individual variables may be difficult, as retracing the layers of transformations for a given variable may be difficult and time consuming. Some of the output data may also contain and/or be derived from personally identifying information. Access to such data may be restricted and layers of derivation may make tracking such data difficult. Furthermore, duplicative output data is frequently generated. Duplicative output data may be generated using processing and storage resources, but the duplicative data may be difficult to detect and prevent.

SUMMARY

A system, method, and computer readable medium (collectively, the “system”) is disclosed for managing data transformation and derivation in a big data environment. The system may receive a request to derive an output variable from a source variable. The request may include proposed logic to derive the output variable from the source variable. The system may then compare the proposed logic to existing logic to determine the proposed logic is new. In response to the proposed logic being new, the system may generate transformation code configured to execute the proposed logic. The system may further schedule the transformation code for execution at a predetermined time, and then execute the transformation code to generate data for the output variable.

In various embodiments, the system may generate metadata that describes the proposed logic, and it may also look up metadata that describes the existing logic in a metadata store. The system may then compare the metadata of the proposed logic to the metadata of the existing logic. In various embodiments, the request to derive the output variable may include a requested execution time to execute the transformation code. The system may also store metadata describing the proposed logic in a metadata store in response to the proposed logic being new. The transformation platform may run on a distributed file system. The system may further store the data generated for the output variable in a big data management system. At least one of the output variable, a location of the output variable, or a copy of the output variable may be returned in response to executing the transformation code.

DETAILED DESCRIPTION

As used herein, “big data” may refer to partially or fully structured, semi-structured, or unstructured data sets including hundreds of thousands of columns and records. A big data set may be compiled, for example, from a history of purchase transactions over time, from web registrations, from social media, from records of charge (ROC), from summaries of charges (SOC), from internal data, and/or from other suitable sources. Big data sets may be compiled with or without descriptive metadata such as column types, counts, percentiles, and/or other interpretive-aid data points. The big data sets may be stored in various big-data storage formats containing millions of records (i.e., rows) and numerous variables (i.e., columns) for each record.

The present disclosure provides a system, method, and computer program product for defining, scheduling, and executing complex data transformations within a big data environment. The lineage (i.e., the origination and transformation history) of output variables as well as other complex metadata related to scheduling and programmatic flow is automatically captured by the system. This allows transformation logic in the form of code to be reused, and for holistic governance of the final utility of the transformation derived variables (e.g., access control reporting for personally identifying information).

With reference toFIG. 1, a distributed file system (DFS)100is shown, in accordance with various embodiments. DFS100comprises a distributed computing cluster102configured for parallel processing and storage. Distributed computing cluster102may comprise a plurality of nodes104in electronic communication with each of the other nodes, as well as a control node106. Processing tasks may be split among the nodes of distributed computing cluster102to improve throughput and enhance storage capacity. Distributed computing cluster may be, for example, a Hadoop® cluster configured to process and store big data sets with some of nodes104comprising a distributed storage system and some of nodes104comprising a distributed processing system. In that regard, distributed computing cluster102may be configured to support a Hadoop® distributed file system (HDFS) as specified by the Apache Software Foundation at http://hadoop.apache.org/docs/.

In various embodiments, nodes104, control node106, and client110may comprise any devices capable of receiving and/or processing an electronic message via network112and/or network114. For example, nodes104may take the form of a computer or processor, or a set of computers/processors, such as a system of rack-mounted servers. However, other types of computing units or systems may be used, including laptops, notebooks, hand held computers, personal digital assistants, cellular phones, smart phones (e.g., iPhone®, BlackBerry®, Android®, etc.) tablets, wearables (e.g., smart watches and smart glasses), or any other device capable of receiving data over the network.

In various embodiments, client110may submit requests to control node106. Control node106may distribute the tasks among nodes104for processing to complete the job intelligently. Control node106may thus limit network traffic and enhance the speed at which incoming data is processed. In that regard, client110may be a separate machine from distributed computing cluster102in electronic communication with distributed computing cluster102via network112. A network may be any suitable electronic link capable of carrying communication between two or more computing devices. For example, network112may be local area network using TCP/IP communication or wide area network using communication over the Internet. Nodes104and control node106may similarly be in communication with one another over network114. Network114may be an internal network isolated from the Internet and client110, or, network114may comprise an external connection to enable direct electronic communication with client110and the internet.

A network may be unsecure. Thus, communication over the network may utilize data encryption. Encryption may be performed by way of any of the techniques now available in the art or which may become available—e.g., Twofish, RSA, El Gamal, Schorr signature, DSA, PGP, PKI, GPG (GnuPG), and symmetric and asymmetric cryptography systems.

In various embodiments, DFS100may process hundreds of thousands of records from a single data source. DFS100may also ingest data from hundreds of data sources. The data may be processed through data transformations to generate output variables from input variables. In that regard, input variables may be mapped to output variables by applying data transformations to the input variables and intermediate variables generated from the input values. Nodes104may process the data in parallel to expedite the processing. Furthermore, the transformation and intake of data as disclosed below may be carried out in memory on nodes104. For example, in response to receiving a source data file of 100,000 records, a system with 100 nodes104may distribute the task of processing 1,000 records to each node104for batch processing. Each node104may then process the stream of 1,000 records while maintaining the resultant data in memory until the batch is complete for batch processing jobs. The results may be written, augmented, logged, and written to disk for subsequent retrieval. The results may be written to disks using various big data storage formats.

With reference toFIG. 2, an exemplary architecture of a big data management system (BDMS)200is shown, in accordance with various embodiments. BDMS200may be similar to or identical to DFS100ofFIG. 1, for example. DFS202may serve as the physical storage medium for the various data storage formats201of DFS202. A non-relational database204may be maintained on DFS202. For example, non-relational database204may comprise an HBase™ storage format that provides random, real time read and/or write access to data, as described and made available by the Apache Software Foundation at http://hbase.apache.org/.

In various embodiments, a search platform206may be maintained on DFS202. Search platform206may provide distributed indexing and load balancing to support fast and reliable search results. For example, search platform206may comprise a Solr® search platform as described and made available by the Apache Software Foundation at http://lucene.apache.org/solr/.

In various embodiments, a data warehouse214such as Hive® may be maintained on DFS202. The data warehouse214may support data summarization, query, and analysis of warehoused data. For example, data warehouse214may be a Hive® data warehouse built on Hadoop® infrastructure. A data analysis framework210may also be built on DFS202to provide data analysis tools on the distributed system. Data analysis framework210may include an analysis runtime environment and an interface syntax such similar to those offered in the Pig platform as described and made available by the Apache Software Foundation at https://pig.apache.org/.

In various embodiments, a cluster computing engine212for high-speed, large-scale data processing may also be built on DFS202. For example, cluster computing engine212may comprise an Apache Spark™ computing framework running on DFS202. DFS202may further support a MapReduce layer216for processing big data sets in a parallel, distributed manner to produce records for data storage formats201. For example, MapReduce layer216may be a Hadoop® MapReduce framework distributed with the Hadoop® HDFS as specified by the Apache Software Foundation at http://hadoop.apache.org/docs/. The cluster computing engine212and MapReduce layer216may ingest data for processing, transformation, and storage in data storage formats201using the distributed processing and storage capabilities of DFS202.

In various embodiments, DFS202may also support a table and storage management layer208such as, for example, an HCatalog installation. Table and storage management layer208may provide an interface for reading and writing data for multiple related storage formats. Continuing with the above example, an HCatalog installation may provide an interface for one or more of the interrelated technologies described above such as, for example, Hive®, Pig, Spark®, and Hadoop® MapReduce.

In various embodiments, DFS202may also include various other data storage formats218. Other data storage formats218may have various interface languages with varying syntax to read and/or write data. In fact, each of the above disclosed storage formats may vary in query syntax and interface techniques. Virtualized database structure220may provide a uniform, integrated user experience by offering users a single interface point for the various different data storage formats201maintained on DFS202. Virtualized database structure220may be a software and/or hardware layer that makes the underlying data storage formats201transparent to client222by providing variables on request. Client222may request and access data by requesting variables from virtualized database structure220. Virtualized database structure220may then access the variables using the various interfaces of the various data storage formats201and return the variables to client222.

In various embodiments, the data stored using various of the above disclosed data storage formats201may be stored across data storage formats201and accessed at a single point through virtualized database structure220. The variables accessible through virtualized database structure220may be similar to a column in a table of a traditional RDBMS. That is, the variables identify data fields available in the various data storage formats201.

In various embodiments, variables may be stored in a single one of the data storage formats201or replicated across numerous data storage formats201to support different access characteristics. Virtualized database structure220may comprise a catalog of the various variables available in the various data storage formats201. The cataloged variables enable BDMS200to identify and locate variables stored across different data storage formats201on DFS202. Variables may be stored in at least one storage format on DFS202and may be replicated to multiple storage formats on DFS202. The catalog of virtualized database structure220may thus track the location of a variable available in multiple storage formats.

The variables may be cataloged as they are ingested and stored using data storage formats201. The catalog may track the location of variables by identifying the storage format, the table, and/or the variable name for each variable available through virtualized database structure220. The catalog may also include metadata describing what the variables are and where the variables came from such as data type, original source variables, timestamp, access restrictions, sensitivity of the data, and/or other descriptive metadata. For example, internal data and/or personally identifying information (PII) may be flagged as sensitive data subject to access restrictions by metadata corresponding to the variable containing the internal data and/or PII. Metadata may be copied from the data storage formats201or generated separately for virtualized database structure220.

In various embodiments, virtualized database structure220may provide a single, unified, and virtualized data storage format that catalogues accessible variables and provides a single access point for records stored on data storage formats201. Client222(which may operate using similar hardware and software to client110ofFIG. 1) may access data stored in various data storage formats201via the virtualized database structure220. In that regard, virtualized database structure220may be a single access point for data stored across the various data storage formats201on DFS202.

In various embodiments, virtualized database structure220may store and maintain the catalog of variables including locations and descriptive metadata, but virtualized database structure220may not store the actual data contained in each variable. The data that fills the variables may be stored on DFS202using data storage formats201. Virtualized database structure220may enable read and write access to the data stored in data storage formats201without a client system having knowledge of the underlying data storage formats201.

The data stored in data storage formats201may be generated and/or ingested by applying a series of transformations to input data using DFS100. The transformations may comprise a series of logical steps to alter some or all of the source data. With reference toFIG. 3, a flow chart300for transforming source data302into output310is shown, in accordance with various embodiments. Source data302may comprise a one or more raw data files such as, for example, a delimited flat file, an XML file, a database file, a table, or any other structured, semi-structured or unstructured data format. Source data302may include a plurality of records with each record containing data. The data in the records may be separated into fields with each field being a source variable. Source data may have transformations304applied in the form of logic306.

In various embodiments, logic306may be a series of ordered processing steps to modify source data and generate intermediate variable values and/or output variable values. For example, logic306may include data formatting steps such as stripping white space and truncating numbers to a predetermined length. Logic306may also include evaluation steps that execute an action against the data or generate a transformed value308for an intermediate variable or output variable in response to evaluation of a logical statement against a value of the source variable. Transformed values308may be augmented and written into an output310such as a load file for loading into a big data storage format. For example, logical steps may identify and copy a zip code from a complete mailing address and write the zip code value into a zip code variable.

With reference toFIG. 4, a logic map400is shown in a graphical form depicting transformations304applied to source data302at a variable (e.g., column of a table) level, in accordance with various embodiments. A user may request an output variable by using a graphical tool to generate logic maps for the output variable. A user may also write a program that interfaces with a BDMS200to read and write data according to the transformations.

As shown in logic map400, source variable 1 is mapped directly to output 1 by a transformation410. The transformation410may modify the data in source variable 1 or preserve the original data in source variable 1 for writing into an output file. Thus, output 1 may originate from source variable 1 and transformation410.

In various embodiments, output variables408may originate from multiple source variables402. For example, as illustrated, source variable 2 is transformed into derived variable 1, source variable 3 is transformed into derived variable 2, derived variable 1 and derived variable 2 are both transformed into derived variable 5, and derived variable 5 is transformed into output 2. Thus, output variables408are derived from source variables402and derived variables404by applying transformations. The source variables402, derived variables404, and transformations410applied to the source variables402may be used to compare the origin of output variables and determine whether the output variables are duplicative of existing output variables.

With reference toFIG. 5, system500for transforming data in a big data environment is shown, in accordance with various embodiments. System500may facilitate user502requests for output variables, output tables, and/or output files by generating the requested output. User502may submit a request to transformation platform504. The request may be in the form of a text query and/or a submission from a graphical tool. The request may contain proposed logic for transforming source data into output data.

In various embodiments, transformation platform504may be a software and/or hardware system configured to perform logic evaluation505, code generation506, and schedule and execute508the resulting code. In response to receiving a request for an output variable from user502, transformation platform504of system500may evaluate the proposed logic that the user submitted to generate the output variable. Logic evaluation505may include comparing the proposed logic to existing logic to determine whether the logic is duplicative.

In various embodiments, the logic may be prepared for comparison in a deterministic manner to enable one to one comparison between logic. For example, transformation platform504may generate metadata describing the requested transformation to derive an output variable. Transformation platform504may access metadata describing existing transformations in metadata store514. Storing metadata describing transformations may enable logic comparison without manually evaluating each existing transformation in response to each request for a new output variable. The metadata for the requested transformation may be compared to the metadata describing existing transformations.

In various embodiments, transformation platform504may deny the request for a new transformation in response to the system detecting that a data transformation exists with the existing logic of the existing data transformation matching the proposed logic. Instead, transformation platform504may return the location of the existing transformation results, a copy of the existing transformation results, and/or the actual existing transformation results. In that regard, transformation platform504may reduce processing and storage space allotted to duplicative transformation tasks.

In various embodiments, transformation platform504may move on to code generation506, if the proposed transformation passes logic evaluation505. Transformation platform504may dynamically generate code in response to the proposed logic and/or transformation being new (i.e., not matching the existing logic or existing transformations). The machine generated code produced by transformation platform504may be an executable code segment that processes source data to produce the requested output in response to execution. Transformation platform504may automatically generate the code to perform the proposed logical steps received from user502and produce the requested output variable.

In various embodiments, after code generation506, transformation platform504may schedule and execute508the automatically generated code. Transformation platform504may receive the code and determine when the code can and/or should run. Transformation platform504may analyze existing transformation tasks that are scheduled and available processing power on DFS100to execute the task to determine when code should execute. User502may submit a desired execution schedule with the request for an output. For example, the user may specify that the output variable should be run daily, weekly, monthly, hourly, one time when available, one time immediately, etc. Output data510is then produced by execution of the dynamically generated code.

In various embodiments, output data510may be stored in a one or more data storage formats of BDMS200, as disclosed above. Transformation platform504may also generate metadata for storage in metadata store514. The metadata may describe the newly generated output data510and/or the transformation used to generate the output data510. Put another way, metadata may describe what the new output variables are and where the output variables came from. For example, metadata may include a data type, original source variables, logic used to generate the variables, timestamp, access restrictions, sensitivity of the data, and/or other descriptive metadata. The metadata may be used in logic evaluation505, for example, to identify duplicative transformations and output variables. The metadata may also be used by BDMS200as disclosed above to locate data in various data storage formats.

In various embodiments, existing data516may also be processed for analysis518resulting in metadata generation and updates. In that regard, metadata store514may be maintained to keep metadata up-to-date and accurate. Metadata store514may thus be a central location to reference metadata for transformations and existing data. BDMS200may use metadata store514to identify and locate existing data as disclosed in greater detail above.

With reference toFIG. 6, a process600for managing data transformations using transformation platform504is shown, in accordance with various embodiments. Transformation platform504may receive a request to derive an output variable (Block602). The request may include parameters such as proposed logic for application to a source variable and a proposed execution time and/or frequency. The logic may be used to derive output data for an output variable.

In various embodiments, transformation platform504may compare the proposed logic to existing logic (Block604). In order to complete the comparison, transformation platform504may generate metadata describing the proposed logic or the proposed transformation. The metadata describing the proposed logic may be compared to metadata describing existing logic. Transformation platform504may deny the request for a newly derived output variable in response to the proposed logic matching existing logic. Transformation platform504may then return the existing output variable, the location of the existing output variable, or a copy of the existing output variable rather than a newly derived output variable as requested.

In various embodiments, transformation platform504may also generate transformation code in response to the proposed logic being new (Block606). The dynamically generated code may include machine executable code that applies the proposed logic to a source variable to generate the requested output variable. Transformation platform504may schedule the transformation code for execution at a predetermined time (Block608). The predetermined time may include a requested time or frequency such as, for example, daily, weekly, monthly, hourly, once at a set time, repeatedly at a set time, one time when available, one time immediately, and/or when a source data refresh occurs. The predetermined time may also be determined by transformation platform by analyzing existing scheduled tasks and identifying the next suitable time for code execution.

In various embodiments, transformation platform504may then execute the transformation code to generate data for the output variable at the predetermined time (Block610). The data generated may result from applying the proposed logic from the request for a derived output variable to the identified source variable. Metadata describing the executed logic may be stored in a metadata store for comparison to later requests for newly derived output variables. The resultant data may be stored in a supported data storage format of BDMS200. Transformation platform504may thus reduce duplicative processing and storage on DFS100while optimizing the execution schedule to balance user requests with available system resources. In that regard, transformation platform504tends to increase efficiency of data ingestion into BDMS200.

The systems and methods herein may generate output data for ingestion into a wide variety of data storage formats (e.g., Hive®, Solr®, Hbase) having different support processing approaches (e.g., batch, real-time, process). The transformation platform may manage the output data available by evaluating logic to limit duplicative output data. In that regard, the transformation platform may limit duplicative data storage and reduce duplicative data processing on a DFS. The system may also dynamically generate code automatically in response to a request for an output variable. A user may have no programming capabilities and still access the desired data in an efficient manner. Furthermore, the scheduling and execution of generated code may be automatically organized in an efficient and responsive manner by the transformation platform. In that regard, the transformation platform may automate the entire data generation process in response to a user request for an output. Transformation platform504may also populate a metadata store for more efficient operation of a BDMS storing the underlying data.

As used herein, “match” or “associated with” or similar phrases may include an identical match, a partial match, meeting certain criteria, matching a subset of data, a correlation, satisfying certain criteria, a correspondence, an association, an algorithmic relationship and/or the like.

The present system or any part(s) or function(s) thereof may be implemented using hardware, software or a combination thereof and may be implemented in one or more computer systems or other processing systems. However, the manipulations performed by embodiments were often referred to in terms, such as matching or selecting, which are commonly associated with mental operations performed by a human operator. No such capability of a human operator is necessary, or desirable in most cases, in any of the operations described herein. Rather, the operations may be machine operations. Useful machines for performing the various embodiments include general purpose digital computers or similar devices.

Computer system also includes a main memory, such as for example random access memory (RAM), and may also include a secondary memory. The secondary memory may include, for example, a hard disk drive and/or a removable storage drive, representing a floppy disk drive, a magnetic tape drive, an optical disk drive, etc. The removable storage drive reads from and/or writes to a removable storage unit in a well-known manner. Removable storage unit represents a floppy disk, magnetic tape, optical disk, etc. which is read by and written to by removable storage drive. As will be appreciated, the removable storage unit includes a computer usable storage medium having stored therein computer software and/or data.

In various embodiments, secondary memory may include other similar devices for allowing computer programs or other instructions to be loaded into computer system. Such devices may include, for example, a removable storage unit and an interface. Examples of such may include a program cartridge and cartridge interface (such as that found in video game devices), a removable memory chip (such as an erasable programmable read only memory (EPROM), or programmable read only memory (PROM)) and associated socket, and other removable storage units and interfaces, which allow software and data to be transferred from the removable storage unit to computer system.

Computer system may also include a communications interface. Communications interface allows software and data to be transferred between computer system and external devices. Examples of communications interface may include a modem, a network interface (such as an Ethernet account), a communications port, a Personal Computer Memory Account International Association (PCMCIA) slot and account, etc. Software and data transferred via communications interface are in the form of signals which may be electronic, electromagnetic, optical or other signals capable of being received by communications interface. These signals are provided to communications interface via a communications path (e.g., channel). This channel carries signals and may be implemented using wire, cable, fiber optics, a telephone line, a cellular link, a radio frequency (RF) link, wireless and other communications channels.

The terms “computer program medium” and “computer usable medium” and “computer readable medium” are used to generally refer to media such as removable storage drive and a hard disk installed in hard disk drive. These computer program products provide software to computer system.

In various embodiments, software may be stored in a computer program product and loaded into computer system using removable storage drive, hard disk drive or communications interface. The control logic (software), when executed by the processor, causes the processor to perform the functions of various embodiments as described herein. In various embodiments, hardware components such as application specific integrated circuits (ASICs). Implementation of the hardware state machine so as to perform the functions described herein will be apparent to persons skilled in the relevant art(s).

Any databases discussed herein may include relational, nonrelational, hierarchical, graphical, or object-oriented structure and/or any other database configurations including various big data products available from the Apache Software Foundation as described above. Common database products that may be used to implement the databases include DB2 by IBM® (Armonk, N.Y.), various database products available from ORACLE® Corporation (Redwood Shores, Calif.), MICROSOFT® Access® or MICROSOFT® SQL Server® by MICROSOFT® Corporation (Redmond, Wash.), MySQL by MySQL AB (Uppsala, Sweden), or any other suitable database product. Moreover, the databases may be organized in any suitable manner, for example, as data tables or lookup tables. Each record may be a single file, a series of files, a linked series of data fields or any other data structure. Association of certain data may be accomplished through any desired data association technique such as those known or practiced in the art. For example, the association may be accomplished either manually or automatically. Automatic association techniques may include, for example, a database search, a database merge, GREP, AGREP, SQL, using a key field in the tables to speed searches, sequential searches through all the tables and files, sorting records in the file according to a known order to simplify lookup, and/or the like. The association step may be accomplished by a database merge function, for example, using a “key field” in pre-selected databases or data sectors. Various database tuning steps are contemplated to optimize database performance. For example, frequently used files such as indexes may be placed on separate file systems to reduce In/Out (“I/O”) bottlenecks.

Phrases and terms similar to “internal data” may include any data a credit issuer possesses or acquires pertaining to a particular consumer. Internal data may be gathered before, during, or after a relationship between the credit issuer and the transaction account holder (e.g., the consumer or buyer). Such data may include consumer demographic data. Consumer demographic data includes any data pertaining to a consumer. Consumer demographic data may include consumer name, address, telephone number, email address, employer and social security number. Consumer transactional data is any data pertaining to the particular transactions in which a consumer engages during any given time period. Consumer transactional data may include, for example, transaction amount, transaction time, transaction vendor/merchant, and transaction vendor/merchant location.

Although the disclosure includes a method, it is contemplated that it may be embodied as computer program instructions on a tangible computer-readable carrier, such as a magnetic or optical memory or a magnetic or optical disk. All structural, chemical, and functional equivalents to the elements of the above-described exemplary embodiments that are known to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be encompassed by the present claims. Moreover, it is not necessary for a device or method to address each and every problem sought to be solved by the present disclosure, for it to be encompassed by the present claims.