User interface data sample transformer

An approach for transforming a large dataset using user interface-based transformations applied to a sample of the dataset is disclosed. The sample of the large dataset has the same or similar format as the large dataset. A user can quickly apply transformations to the sample dataset using UI-based instructions. The UI-based instructions can be used to create a transformation job that can be configured to run on a backed database, such as a distributed database, to apply the transformations to the large dataset.

TECHNICAL FIELD

The present disclosure generally relates to the technical field of special-purpose machines that facilitate data manipulation and validation including computerized variants of such special-purpose machines and improvements to such variants, and to the technologies by which such special-purpose machines become improved compared to other special-purpose machines that facilitate data manipulation and validation. In particular, the present disclosure addresses systems and methods for user interface data sample based transformations of data.

BACKGROUND

In recent years, extremely large amounts of data have been generated by network-connected systems and users. The collected data may contain patterns that show malicious online behavior, e.g., behavior by malware or hackers, potential terrorism activities, potential sources of food poisoning, or even the best bike routes for a morning commute. Conventional data analysis tools have been unable to parse the extremely large amounts of data in human-understandable ways, thus the patterns remain hidden, e.g., signals lost in noise. Worse yet, much of the extremely large amounts of data is in an unstructured form which conventional data analysis tools cannot parse. Users attempting to add structure to the data encounter various types of errors, including program freezing and crashing. As is evident, there is a demand for improved approaches for structuring and analyzing extremely large sets of data.

DETAILED DESCRIPTION

In various example embodiments, raw data can be imported and transformed using a sample portion of the raw data. The raw data may be unstructured or structured data. The transformations may define structure for the raw data, change pre-existing structure (e.g., schema) of the data, add or remove portions of the data, modify the data values, or modify data types assigned to data values in the raw data. To apply transformations, a sample portion of the raw data is displayed in a UI with a control menu. The control menu includes one or more transformation elements (e.g., buttons, drop-downs, fields) that are configured to apply transformations to the raw data. The transformations are applied to the sample portions of the data in real time or near real time, without applying the transformations to the raw data not in the sample. In this way, by applying each transformation only to the sample displayed in the UI, the user can see the changes applied to the sample and judge whether the transformations were applied properly and further determine whether additional transformations are need to further transform the raw data into structured data.

Once the user determines that no more transformations are necessary, the transformations are recorded as a transformation job that can be applied to the rest of the raw data (e.g., the raw data not included in the displayed sample set) stored in a backend database. The transformations on the rest of the raw data transform the raw data into a structured form per the transformation job recorded from the sample dataset transformations.

When newer raw data (e.g., raw data in the same raw unstructured format) is received, the transformation job is automatically applied to the new raw data, and stored with the structured data in the database backend. The newer raw data may comprise entirely new values in raw format or updates to the data already transformed and stored in a backend database. In some embodiments, the transformations specify types of validations to occur when transforming the data (e.g., exclude data outside a defined range of values, make sure a given column contains only integers). If, during the transformations, an error occurs due to one or more validations failing, an error message is generated; the user can ignore the error message, correct the error manually, or create a new transformation task to address future errors of the same type.

In this way, a user can effectuate transformations to arbitrarily large datasets (e.g., trillions of rows, thousands of columns) through a fast and responsive UI-based approach that shows the results of the transformations in real time and uses a transformation job to transform raw data into a structured form ready for analysis.

With reference toFIG. 1, an example embodiment of a high-level client-server-based network architecture100is shown. A network-based data analysis system104provides server-side functionality via a network102(e.g., the Internet or wide area network (WAN)) to one or more client devices106and108. In some implementations, a data architect user (e.g., user196) interacts with the network-based data analysis system104using the client device106, and an analyst user (e.g., user198) interacts with the network-based data analysis system104using client device108. The data visualizer application128is an application to import, transform, and visualize data. For example, user196can use the data visualizer application128to import raw data and transform it for storage and later analysis. Further, user198can use the data visualizer application128to view the data transformed per user196. In some embodiments, the data visualizer application128is run as local software executed by processors of the client device (e.g., client device106and client device108). In some embodiments, the data visualizer application128is run from a web client (e.g., a browser) as a cloud service that works with application server118to provide cloud services (e.g., cloud-based data analysis).

In various implementations, the client devices106and108each comprise a computing device that includes at least a display and communication capabilities that provide access to the network-based data analysis system104via the network102. The client device can be implemented as, but is not limited to, a remote device, work station, Internet appliance, hand-held device, wireless device, portable device, wearable computer, cellular or mobile phone, Personal Digital Assistant (PDA), smart phone, tablet, ultrabook, netbook, laptop, desktop, multi-processor system, microprocessor-based or programmable consumer electronic, game consoles, set-top box, network Personal Computer (PC), mini-computer, and so forth.

In some embodiments, the data visualizer application128accesses the various systems of the network-based data analysis system104via the web interface supported by a web server122. Similarly, in some embodiments, the data visualizer application128can initiate tasks to be performed programmatically (e.g., automatically) without user input. In those example embodiments, the data visualizer application128can interface to perform the programmatic tasks through an Application Program Interface (API) server114located on the server side (e.g., within network-based data analysis system104).

Users (e.g., the user196and198) comprise a person, a machine, or other means of interacting with the client devices (e.g., client device106and108). In some example embodiments, the user is not part of the network architecture100, but interacts with the network architecture100via the client devices106and108. For instance, the user196provides input (e.g., touch screen input or alphanumeric input) to the client device106and the input is communicated to the network-based data analysis system104via the network102. In this instance, the network-based data analysis system104, in response to receiving the input from the user196, communicates information from application server118to the client device106via the network102to be presented to the user196. In this way, according to some example embodiments, users can interact with the network-based data analysis system104using their respective client devices.

As illustrated in the example embodiment ofFIG. 1, the API server114and the web server122are coupled to, and provide programmatic and web interfaces respectively to, one or more application server118. The application server118can host a UI sample transformer124configured to receive raw data, and perform transformations on a sample of the raw data to record as a transformation job. As described in further detail below, the UI sample transformer124may create a sample of the raw data for display on data visualizer application128for transformation job generation. The portion of the raw data not included in the sample is stored in a database system (e.g., database backend), such as database system120. In some example embodiments, the raw data not in the sample can be distributed across data stores122A-N, which are configured to work as distributed data stores for a distributed database system.

In some example embodiments, the database system120is implemented as an Apache Hadoop-based system, which may implement Hadoop techniques (e.g., MapReduce) on Hadoop Distributed File System (HDFS) datastores, such as data stores122A-N. It is appreciated that Hadoop and HDFS are mere examples of the database system120and features and file implementations may be modified. For example, in some embodiments, the data stores122A-N are HDFS formatted files which can be transformed using Apache Spark functionality that is integrated into UI sample transformer124.

FIG. 2illustrates a block diagram showing components provided within the UI sample transformer124, according to some example embodiments. As is understood by skilled artisans in the relevant computer and Internet-related arts, each functional component (e.g., engine, module, or database) illustrated inFIG. 2may be implemented using hardware (e.g., a processor of a machine) or a combination of logic (e.g., executable software instructions) and hardware (e.g., memory and processor of a machine) for executing the logic. Furthermore, the various functional components depicted inFIG. 2may reside on a single machine (e.g., a server) or may be distributed across several machines in various arrangements such as cloud-based architectures. Moreover, any two or more of these components may be combined into a single component (e.g., a single module), and the functions described herein for a single component may be subdivided among multiple modules.

As illustrated inFIG. 2, the UI sample transformer124comprises multiple engines that implement data transformation of raw data into structured data, according to some example embodiments. The components themselves are communicatively coupled (e.g., via appropriate interfaces) to each other and to various data sources, so as to allow information to be passed between the applications or so as to allow the applications to share and access common data. Although inFIG. 2components, such as the transformation engine205, are displayed within the UI sample transformer124on the server side, in some embodiments, one or more components of the UI sample transformer124may be integrated into a client-side program (e.g., data visualizer application128) to improve responsiveness. To this end, the UI sample transformer124comprises an interface engine200, a transformation engine205, a record engine210, a database engine215, an analysis engine220, and a validation engine225.

The interface engine200manages generating and displaying user interfaces on the client devices106and108using the data visualizer application128. In particular, the interface engine200generates a UI display of a sample dataset of data to be imported and control elements that can be manipulated by the user to effectuate changes to the displayed sample dataset. The transformation logic is provided by transformation engine205, which is configured to receive specific transformation commands from the UI, apply the transformation commands to the sample dataset, and pass the resultant transformed data to the interface engine200, which then transmits the resultant transformed data to the client device for display by the data visualizer application128. How the transformations are applied and example types of transformations are discussed in further detail below, with reference toFIGS. 6A-6E.

In some example embodiments, the transformation engine205is located in the data visualizer application128and transformations are implemented by the client-side transformation engine205using a client side programming language (e.g., browser-executable code type, browser executed JavaScript, code executed locally by client device106), which allow the user to quickly see the changes he/she is making to the sample dataset in real time or near real time, without waiting for the transformations to be applied to the full raw dataset, which may be many petabytes in size.

The record engine210is configured to record the applied transformations (e.g., types of transformations applied, and order of transformations applied) to the sample dataset. As with the transformation engine205, in some embodiments, the record engine210is integrated into the data visualizer application128to record client-side transformations applied to the sample dataset. Upon a build command being selected, the record engine210uses the selected transformations to generate a transformation job, which is then transmitted to the UI sample transformer124. The UI sample transformer124then applies the transformation job to the rest of the raw data stored in the database system120.

In some embodiments, the record engine210is configured to generate the transformation job into a database-executable code type that executes across a distributed data storage system. In according to some example embodiments, the browser-executable code type cannot be natively run on the database as it is configured as client-side script (e.g., JavaScript) that can be used to quickly apply transformations to the sample dataset. Similarly, according to some example embodiments, the database-executable code type cannot be natively run on the browser because the database-executable code type is code configured for functional programming (e.g., MapReduce) on a database backend, not a client side browser.

As an example, assume a transformation to the sample dataset involves locating a delimiter value and deleting values that appear before the delimiter value (e.g., if the data is “firstName;lastName”, the transformation would identify the delimiter “;” and delete the value before the delimiter, which is the “firstName” value). The transformation engine205may apply the process to the sample dataset directly, locating the specified delimiter and removing values that appear before the delimiter, and display the results directly in the display of the client device. In contrast, upon the build command being selected, the record engine210records the transformation as a task that may be applied in each node that manages each datastore (e.g., datastore122A, datastore122B). For example, the record engine210may record the task as part of a mapper code in a MapReduce sequence that can be applied across all the data stores concurrently (e.g., in parallel). Alternatively, according to some example embodiments, the record engine210records the task as part of an Apache Spark job to be performed by Spark workers across all data stores concurrently (e.g., in parallel).

The database engine215is configured to receive the transformation job from the record engine210and apply the transformations to the raw data in the data stores122A-N in database system120. As discussed, the database engine215may be implemented using different types of database systems (e.g., Apache Hadoop and HDFS, Oracle RDMS) and the record engine210transforms the code applied to the sample dataset (which is configured to only apply the transformation to the small displayed sample dataset) into code that can be applied at very large scales by the database engine215.

The validation engine225manages validation logic for the transformations applied to the raw data. As new raw data is received, the validation engine225retrieves the transformation job that was created by the record engine210and instructs the database engine215to apply the transformation job to the new raw data to transform the new raw data into new structured data, to be added to the originally transformed data stored in the data stores122A-N. The process of transforming new raw data into new structured data can be performed automatically by the UI sample transformer124(e.g., via validation engine225) without requiring the user to redo the transformations on the sample dataset to create the transformation job. If an error is encountered while transforming the new raw data, the validation engine225generates an error for the user to address. To address the error, the user may correct the faulty values in the new raw data, the user can choose to ignore the error, or the user can create a new transformation task to be included as part of the transformation job so that future errors are avoided.

In this way, an architect user (e.g., user196) can quickly set up a distributed workflow that automatically transforms raw data into structured data for analysis, and further ensure that new raw data is automatically structured and added to the previous data. Other users, such as user198, can analyze the structured data using the data visualizer application128. Because the potentially large set of transformed data is handled on the backend (e.g., across data stores122A-122N), the analyst user198can quickly apply filters to the data to hone the data down to understandable results. To this end, the analysis engine220is configured to generate filtered commands that the database engine215can use to retrieve filtered data from data stores122A-N. Further, because new data is automatically transformed using the pre-configured transformation job, the analyst user198can simply use a refresh command to check whether new data has been added to the data stores122A-N, instead of rerunning a transformation job on the entire dataset.

FIG. 3illustrates a flow diagram for a method300of transforming large sets of data using the UI sample dataset-based approached, according to some example embodiments. At operation305, the UI sample transformer124receives raw data (e.g., an input dataset) to be transformed. In some example embodiments, the raw data is in non-validated form in that further changes are required to make the data valid or parsable by the data visualizer application128. For example, the raw data may be in unstructured form (e.g., lists without delimiters, images). As a further example, the raw data may have some structure, such as columns, but the user still desires to transform the data to a desired structure so that that the data can be parsed and analyzed. The database engine215stores the raw data in the database system120and partitions off a sample of the raw data to be displayed by the interface engine200.

At operation310, the transformation engine205receives one or more transformations from the user (e.g., user196). In response, the transformation engine205applies the received transformations to the sample dataset, and displays the result on the data visualizer application128. At operation315, the UI sample transformer124receives the build command from the user through the user interface. At operation320, the record engine210, in response to receiving the build command, generates a transformation job that includes the one or more transformations received at operation310. In some embodiments, the record engine210records the transformations by translating the transformations from commands to be applied to the sample dataset (e.g., command to be run on a single table) into commands that run on at a large scale on database system120, e.g., distributed database commands. At operation325, the database engine215applies the transformation job to the raw dataset to transform the raw dataset into a structured format. For instance, the transformation job applies each of the transformations performed on the sample dataset to the raw dataset, thereby transforming the raw dataset into a structured dataset.

FIG. 4shows a flow diagram for a method400of transforming new raw data and validations, according to some example embodiments. Validations are performed to ensure newer data is transformed by the transformation job properly (e.g., so that the newly received data can be added to the already transformed structured data in data stores122A-N). An example validation includes checking that certain types of data are in certain forms (e.g., check that a given column contains only string characters). A further example of a validation is checking whether values are within a given range (e.g., checking that the values in a given column are between a minimum and maximum value, checking that the values of a given column are within some standard deviation value of the total values in the column).

At operation405, the UI sample transformer124receives new raw data to be transformed. The database engine215automatically transfers (e.g., upon receipt by the UI sample transformer124) the new data to the database system120for storage in data stores122A-N. Because the raw data is not yet structured, the newer raw data is stored in a staging partition in the data stores122A-N.

In the example ofFIG. 4, the new raw data is in the same or similar form as the original raw data for which the transformation job was created. In some embodiments, the new raw data is assumed to be in the same form because the data was uploaded from the same source (e.g., user uploads more data to the transformation job project). In some embodiments, the user196determines that the newer data is in the same or similar form as the original raw data and, accordingly, the user196chooses the same transformation job (e.g., the transformation job created to transform the original raw data) for application to the newer data. In some example embodiments, the UI sample transformer124creates a project session for each transformation job, and if a user (e.g., user196) uploads the data to the project session, the UI sample transformer124automatically applies the transformation job for that project session.

In some embodiments, the user (e.g., user196) manually uploads the new raw data, and then manually selects the transformation job to be applied to the new raw data. For example, the user may visually ascertain that the new raw data is in the same unstructured format as the original raw data (e.g., the raw data received in operation305, inFIG. 3) and accordingly select the same transformation job (e.g., the transformation job created at operation320, ofFIG. 3).

At operation410, the database engine215applies the transformation job to the new raw data stored in the staging partition of the data stores122A-N. At operation420, if the database engine215encounters an error when applying the transformation job to the new raw data, the error is passed to the validation engine225for operation425. For example, if a transformation to be applied is configured to identify a semi-colon as a delimiter, and a given value does not have a delimiter, the database engine215determines that validation has failed at operation420because there is an error in the data (e.g., missing delimiter). At operation425, the validation engine225receives the error (e.g., error data received from database engine215) and generates an error message for the user (e.g., user196) to manage the error. In some example embodiments, the validation error is due to failure of a transformation task. For example, if a transformation task specifies that a given column is to have its values transformed from an integer data type to floating point data type, and the column contains strings, then the transformation task may fail as the database engine215may not be configured to transform strings to floating point data types.

To address a validation error, in some embodiments, the database engine215ignores the error and the values that caused the error are left in uncorrected form in the newer transformed dataset. In some embodiments, the user corrects the values that caused the error (e.g., by deleting a stray delimiter in the new raw data that caused an error). In some embodiments, particularly those where the error is widespread throughout the newer raw data, the transformation engine205receives from the user (e.g., user196) a new transformation task to be included in the transformation job to address the error, as illustrated at operation430. Once the error is handled (e.g., by correcting the error or creating a new transformation) the transformation job is again reapplied to the newer raw data at operation410.

At operation435, if the database engine215does not encounter errors when applying the transformation job to the new raw data, the new raw data is thereby transformed into new structured data, and is added to the partition that stores the originally transformed raw data in data stores122A-N.

Once the data is transformed into structured data and stored in database system120, the data visualizer application128allows users (e.g., user198) to quickly retrieve, filter, and analyze the information. Furthermore, in contrast to past approaches, because new raw data is automatically transformed using the transformation job, the analyst user (e.g., user198) does not have to run a full transformation job his/herself to analyze the latest data.

FIG. 5shows a flow diagram for a method500of analyzing structured data transformed using the approaches disclosed herein, according to some example embodiments. At operation505, the analysis engine220receives an analysis request from an analyst user (e.g., user198). The analysis request may be a request to filter out portions of the structured data (e.g., return data only matching certain ranges) and/or visualize the structured data using a data visualization graph (e.g., social network graph, histogram).

At operation510, the database engine215receives the analysis request and applies operations of the analysis request to the structured data. For example, if the analysis request of operation505requests only rows having a value between a minimum and maximum, the database engine215formulates a query configured to run on database system120and retrieves the matching rows from the structured data. The database engine215then transmits the matching rows to the analysis engine220for further visualization or other operations specified in the analysis request. At operation515, the analysis engine220displays the requested analysis results to the user through a display of the data visualizer application128.

As an illustrative example, and strictly as a non-limiting example, assume that the new raw data and all of operations ofFIG. 4occurred between operations515and520ofFIG. 5. That is, assume that after viewing the requested analysis data, newer data is received and transformed using the transformation job, and further that the transformed data is stored in the distributed database system120. Continuing, further assume that at operation520, the user (e.g., user198) wants to refresh the data to get the latest data for analysis. Conventionally, the user would have to run the transformation job on the newly received data, or wait for other users with expertise to transform the data. However, using the approach here, the transformation job was quickly created using the sample-based approach. That is, through verifying that the transformations produce the desired structured data using a sample dataset, automatically applying the transformations at-scale on the back end to transform the entire large dataset, and constantly transforming newly received data using the sample-dataset-created transformation job, users of the data visualizer application128can transform and analyze data in an efficient, accurate way.

At operation520, the analysis engine220receives an update request from the analyst user (e.g., user198). The update request is a type a refresh requests configured to check whether any new data has been added to the data being analyzed (e.g., the transformed data stored in data stores122A-N). At operation525, the database engine215retrieves data matching the operations of the analysis request. At operation530, the analysis engine220display the requested data using one or more graphical data visualizations (e.g., network graph, point plot, histogram).

FIGS. 6A-6Edepict example user interfaces for the UI sample transformer124, according to some embodiments. AlthoughFIGS. 6A-6Edepict specific example user interfaces and user interface elements, these are merely non-limiting examples; many other alternate user interfaces and user interface elements can be generated by UI sample transformer124and data visualizer application128. It will be noted that alternate presentations of the displays ofFIGS. 6A-6Ecan include additional information, graphics, options, and so forth. Alternatively, other presentations can include less information, or provide abridged information for easy use by the user.

FIG. 6Ashows a graphical user interface600for transforming data according to some example embodiments. The user interface600includes a control menu602with display objects604a-e(e.g., buttons, drop-downs, fields) that are selectable by a user (e.g., user196, user198) for uploading raw data, applying transformations, selecting filters and graphical visualizations, and other operations discussed herein. For instance, display object604acan be configured as a data upload tool that allows a user (e.g., user196) to select raw data for upload to the application server118and UI sample transformer124. As discussed above, a sample dataset606of the raw data that represents the unstructured form of the data to be uploaded (e.g., the sample data is subset of the raw data that is stratified to accurately represent the raw dataset) is displayed within a portion of user interface600. The user (e.g., user196) can use transformation display objects604band604cto perform different transformations on the sample dataset606. Though only two display objects are displayed as transformation display objects inFIG. 6A-E, it is appreciated that in some example embodiments, more transformation display objects can be included in control menu602, in different menus and areas within user interface600, or as pop-up menus that appear upon selecting or visually manipulating data values within sample dataset606. Display object(s)604dcan be options for graphical visualizations to be applied to the sample dataset606and/or the transformed full dataset. The transformations selected by the user (e.g., user196) are displayed in the transformation area616. When a user (e.g., user196) has completed transformations of the sample dataset606, he/she may select the build display object610e, which triggers the record engine210to generate a transformation job from each of the applied transformations.

FIG. 6Bshows the result of a first sample transformation on the sample dataset606through the user interface600, according to some example embodiments. In the example shown inFIG. 6B, the user (e.g., user196) defined that each row in the top row of the sample dataset606is a header for the column of values below each top row value (e.g., the “name” value is a header for a column of name values for each of the rows or entries below the top row). Consequently, transformation engine205identifies the sample dataset606as a table with columns having values set by the top row values. The first transformation is shown as a first transformation task in the transformation area616.

FIG. 6Cshows the result of a second sample transformation on the sample dataset606through the user interface600, according to some example embodiments. In the example shown inFIG. 6C, the user (e.g., user196) combined two columns, the height column (“HT”) and the weight column (“WT)” into a single column, with the below values to be separated by a semi-colon delimiter (“;”). Consequently, as illustrated, the two columns are combined into a single column with the corresponding column values per row separated by the semi-colon delimiter. The second transformation is shown as a second transformation task in the transformation area616.

FIG. 6Dshows the result of a third sample transformation on the sample dataset606through the user interface600, according to some example embodiments. In the example shown inFIG. 6D, the user (e.g., user196) removed rows that have the value of “NL” in the “Country” column. Consequently, as shown inFIG. 6D, the second row (which contained data for the person “H. Lorentz”) has been removed, as that entry has “NL” in the country column. The third transformation is shown as a third transformation task in the transformation area616.

FIG. 6Eshows the result of a fourth sample transformation on the sample dataset606through the user interface600, according to some example embodiments. In the example shown inFIG. 6E, the user (e.g., user196) used a find and replace transformation to find and replace any value in the column “Country” that matches “GB” and replace the value with the value “UK”. Consequently, as shown inFIG. 6E, the first, third, seventh, and eight columns have their column values replaced per the transformation. The fourth transformation is shown as a fourth transformation task (e.g., validation transformation) in the transformation area616.

After the user196is finished transforming the sample dataset606, the user196selects the build display object610e. In response to the build display object610ebeing selected, the record engine210identifies each of the transformations tasks (e.g., validation transformations) applied to the sample dataset606and generates a transformation job in code that is configured to run on the backend, at scale (e.g., runnable in parallel across data stores122A-N). The record engine210then passes the transformation job code to the database engine215, which applies the transformation job to raw data in the database system120to transform the raw data to structured data that matches the changes made to the sample dataset606.

FIG. 7shows a network interaction diagram700showing network interactions for UI sample dataset-based transformations to large sets of data, according to some embodiments. As illustrated, the computing entities include the client device106, which runs the data visualizer application128, which communicates over network102(represented by a vertical dashed line) to the application server118, which hosts the UI sample transformer124, and which further issues instructions to the database system124over a network (represented by an additional vertical dashed line).

At operation705, using the client device106, the user196uploads the raw dataset to the application server118. At operation710, the UI sample transformer124(e.g., the database engine215) stores the uploaded raw data to the network-based data analysis system104. At operation715, the network-based data analysis system104receives the raw dataset from the UI sample transformer124and stores it in a database, e.g., in distributed form across data stores122A-N.

At operation750, the database engine215generates a sample dataset of the uploaded raw data for UI-based transformations. According to some example embodiments, the sample dataset should be small enough to maintain responsiveness in a UI on client device106. For example, the sample dataset may comprise all of the columns (e.g., schema) for a given dataset but only a small number of rows (e.g., less than 100). In this way, the transformations applied to the sample dataset will yield the same results when applied to the large raw dataset because the sample dataset accurately reflects the schema structure of the raw dataset, but only over a few rows.

At operation755, the client device106displays the sample dataset, as illustrated inFIG. 6A. At operation760, the user196applies one or more transformations to the dataset, as illustrated inFIGS. 6B-6E. At operation765, in response to the user196selecting the build display object610e, the record engine210generates a transformation job configured to run on the network-based data analysis system104. At operation770, the database engine215receives the transformation job code and applies the transformation job to the raw data in the network-based data analysis system104. For example, the network-based data analysis system104receives instructions from the database system120and applies the transformations on the raw data across the data stores122A-N in parallel.

The performance of certain of the operations may be distributed among the processors, not only residing within a single machine, but deployed across a number of machines. In some example embodiments, the processors or processor-implemented modules can be located in a single geographic location (e.g., within a home environment, an office environment, or a server farm). In other example embodiments, the processors or processor-implemented modules are distributed across a number of geographic locations. The modules, methods, applications and so forth described in conjunction withFIGS. 1-7are implemented in some embodiments in the context of a machine and an associated software architecture. The sections below describe representative software architecture and machine (e.g., hardware) architecture that are suitable for use with the disclosed embodiments.

FIG. 8is a block diagram illustrating components of a machine800, according to some example embodiments, able to read instructions from a machine-readable medium (e.g., a machine-readable storage medium) and perform any one or more of the methodologies discussed herein. Specifically,FIG. 8shows a diagrammatic representation of the machine800in the example form of a computer system, within which instructions816(e.g., software, a program, an application, an applet, an app, or other executable code) for causing the machine800to perform any one or more of the methodologies discussed herein can be executed. For example, the instructions816can cause the machine800to execute the flow diagrams ofFIGS. 3-5and network interaction diagram ofFIG. 7. Additionally, or alternatively, the instructions816can implement the interface engine200, transformation engine205, record engine210, database engine215, analysis engine220, and validation engine225ofFIG. 2, and so forth. The instructions816transform the general, non-programmed machine into a particular machine800programmed to carry out the described and illustrated functions in the manner described. In alternative embodiments, the machine800operates as a standalone device or can be coupled (e.g., networked) to other machines. In a networked deployment, the machine800may operate in the capacity of a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine800can comprise, but not be limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a cellular telephone, a smart phone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web appliance, a network router, a network switch, a network bridge, or any machine capable of executing the instructions816, sequentially or otherwise, that specify actions to be taken by the machine800. Further, while only a single machine800is illustrated, the term “machine” shall also be taken to include a collection of machines800that individually or jointly execute the instructions816to perform any one or more of the methodologies discussed herein.

The machine800can include processors810, memory/storage830, and I/O components850, which can be configured to communicate with each other such as via a bus802. In an example embodiment, the processors810(e.g., a Central Processing Unit (CPU), a Reduced Instruction Set Computing (RISC) processor, a Complex Instruction Set Computing (CISC) processor, a Graphics Processing Unit (GPU), a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Radio-Frequency Integrated Circuit (RFIC), another processor, or any suitable combination thereof) can include, for example, processor812and processor814that may execute instructions816. The term “processor” is intended to include multi-core processor that may comprise two or more independent processors (sometimes referred to as “cores”) that can execute instructions816contemporaneously. AlthoughFIG. 8shows multiple processors810, the machine800may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiples cores, or any combination thereof.

The memory/storage830can include a memory832, such as a main memory, or other memory storage, and a storage unit836, both accessible to the processors810such as via the bus802. The storage unit836and memory832store the instructions816embodying any one or more of the methodologies or functions described herein. The instructions816can also reside, completely or partially, within the memory832, within the storage unit836, within at least one of the processors810(e.g., within the processor's cache memory), or any suitable combination thereof, during execution thereof by the machine800. Accordingly, the memory832, the storage unit836, and the memory of the processors810are examples of machine-readable media.

The I/O components850can include a wide variety of components to receive input, provide output, produce output, transmit information, exchange information, capture measurements, and so on. The specific I/O components850that are included in a particular machine will depend on the type of machine. For example, portable machines such as mobile phones will likely include a touch input device or other such input mechanisms, while a headless server machine will likely not include such a touch input device. It will be appreciated that the I/O components850can include many other components that are not shown inFIG. 8. The I/O components850are grouped according to functionality merely for simplifying the following discussion, and the grouping is in no way limiting. In various example embodiments, the I/O components850can include output components852and input components854. The output components852can include visual components (e.g., a display such as a plasma display panel (PDP), a light emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)), acoustic components (e.g., speakers), haptic components (e.g., a vibratory motor, resistance mechanisms), other signal generators, and so forth. The input components854can include alphanumeric input components (e.g., a keyboard, a touch screen configured to receive alphanumeric input, a photo-optical keyboard, or other alphanumeric input components), point based input components (e.g., a mouse, a touchpad, a trackball, a joystick, a motion sensor, or other pointing instruments), tactile input components (e.g., a physical button, a touch screen that provides location and force of touches or touch gestures, or other tactile input components), audio input components (e.g., a microphone), and the like.

In further example embodiments, the I/O components850can include biometric components856, motion components858, environmental components860, or position components862among a wide array of other components. For example, the biometric components856can include components to detect expressions (e.g., hand expressions, facial expressions, vocal expressions, body gestures, or eye tracking), measure biosignals (e.g., blood pressure, heart rate, body temperature, perspiration, or brain waves), identify a person (e.g., voice identification, retinal identification, facial identification, fingerprint identification, or electroencephalogram based identification), and the like. The motion components858can include acceleration sensor components (e.g., an accelerometer), gravitation sensor components, rotation sensor components (e.g., a gyroscope), and so forth. The environmental components860can include, for example, illumination sensor components (e.g., a photometer), temperature sensor components (e.g., one or more thermometers that detect ambient temperature), humidity sensor components, pressure sensor components (e.g., a barometer), acoustic sensor components (e.g., one or more microphones that detect background noise), proximity sensor components (e.g., infrared sensors that detect nearby objects), gas sensor components (e.g., machine olfaction detection sensors, gas detection sensors to detect concentrations of hazardous gases for safety or to measure pollutants in the atmosphere), or other components that may provide indications, measurements, or signals corresponding to a surrounding physical environment. The position components862can include location sensor components (e.g., a Global Positioning System (GPS) receiver component), altitude sensor components (e.g., altimeters or barometers that detect air pressure from which altitude may be derived), orientation sensor components (e.g., magnetometers), and the like.

Communication can be implemented using a wide variety of technologies. The I/O components850may include communication components864operable to couple the machine800to a network880or devices870via a coupling882and a coupling872, respectively. For example, the communication components864include a network interface component or other suitable device to interface with the network880. In further examples, communication components864include wired communication components, wireless communication components, cellular communication components, Near Field Communication (NFC) components, BLUETOOTH® components (e.g., BLUETOOTH® Low Energy), WI-FI® components, and other communication components to provide communication via other modalities. The devices870may be another machine or any of a wide variety of peripheral devices (e.g., a peripheral device coupled via a Universal Serial Bus (USB)).

The instructions816can be transmitted or received over the network880using a transmission medium via a network interface device (e.g., a network interface component included in the communication components864) and utilizing any one of a number of well-known transfer protocols (e.g., Hypertext Transfer Protocol (HTTP)). Similarly, the instructions816can be transmitted or received using a transmission medium via the coupling872(e.g., a peer-to-peer coupling) to devices870. The term “transmission medium” shall be taken to include any intangible medium that is capable of storing, encoding, or carrying the instructions816for execution by the machine800, and includes digital or analog communications signals or other intangible medium to facilitate communication of such software.