Data engine integration and data refinement

Data engine integration and data refinement are described. The actions include receiving, by a data refinement engine, a request for data. The actions include determining a first amount of processing to be performed by the data refinement engine and a second amount of processing to be performed by one or more processors of a data source that include a plurality of data nodes. The actions include transmitting, by the data refinement engine, code to the plurality of data nodes of instructions associated with the second amount of processing. The actions include receiving, by the data refinement engine and from the plurality of data nodes, unprocessed first data and processed second data. The actions include processing, by the data refinement engine, the unprocessed first data. The actions include, in response to the request for data, transmitting, by the data refinement engine, the processed first data and the processed second data.

FIELD

This description relates to data engine integration and data refinement.

BACKGROUND

Computer systems are used to manage and store data in a structure known as a database. As such, computers can be used to analyze data and generate reports based on the analysis results. For instance, computer systems can filter data and calculate metric values based on the filtered data, ultimately providing a report including the calculated metric values. A database is an organized repository of data. There are various ways in which the data can be organized. Schemas are used to describe the different organizations of data.

Computers systems have two types of physical data storage—disk (e.g., hard drive) storage and Random Access Memory (RAM) storage. Typically, computer systems have more disk storage than RAM, but it can often take longer (e.g., in the range of 100-1,000 times longer) to read data from the disk than from RAM. This can result in noticeable performance degradation.

SUMMARY

In general, one innovative aspect of the subject matter described in this specification can be embodied in methods that include the actions of receiving, by a data refinement engine, a request for data; determining, by the data refinement engine, a first amount of processing to be performed by the data refinement engine and a second amount of processing to be performed by one or more processors of a data source that include a plurality of data nodes; transmitting, by the data refinement engine, code to the plurality of data nodes of instructions associated with the second amount of processing; receiving, by the data refinement engine and from the plurality of data nodes, unprocessed first data and processed second data; and processing, by the data refinement engine, the unprocessed first data; in response to the request for data, transmitting, by the data refinement engine, the processed first data and the processed second data.

These and other embodiments can each optionally include one or more of the following features. The first amount of processing and the second amount of processing includes filtering, aggregation, wrangling, searching, data mining, text analytics, on demand loading, incremental refreshing, streaming, data blending, complex ETL workflows, or multi-sourcing. The data refinement engine receives the request for data request from a dashboard application. The action of determining, by the data refinement engine, a first amount of processing to be performed by the data refinement engine and a second amount of processing to be performed by one or more processors of a data source that include a plurality of data nodes includes generating a query tree that includes query tasks for processing; and determining that the first amount of processing includes a first portion of the query tasks and the second amount of processing includes a second portion of the query tasks.

The action of determining, by the data refinement engine, a first amount of processing to be performed by the data refinement engine and a second amount of processing to be performed by one or more processors of a data source that include a plurality of data nodes includes determining the first amount of processing and the second amount of processing to balance a processing load of the data refinement engine and a processing load of the plurality of data nodes. The actions include after transmitting, by the data refinement engine, code to the plurality of data nodes of instructions associated with the second amount of processing, receiving, by the data refinement engine and from the plurality of data nodes, heartbeats and execution status updates. The action of transmitting, by the data refinement engine, code to the plurality of data nodes of instructions associated with the second amount of processing includes identifying a particular data node of the plurality of data nodes that stores a portion of the unprocessed second data; and transmitting, to the particular data node of the plurality of data nodes, the code to perform a portion of the second amount of processing on the portion of the unprocessed second data.

Other embodiments of this aspect include corresponding systems, apparatus, and computer programs recorded on computer storage devices, each configured to perform the operations of the methods.

DETAILED DESCRIPTION

Techniques are described for an in-memory engine that receives a request for analytics or visualization of data stored on a big data engine that is separate from the in-memory engine. The in-memory engine queries the big data engine for the requested data and requests the big data engine to perform any needed processing to arrive at the requested data prior to sending to the in-memory engine. The in-memory engine receives the requested data, stores it in an in-memory cache, presents the requested data, and processes further analytics on the requested data based on subsequent requests.

In some implementations, an in-memory engine and a big data engine collaborate to provide insights from raw data to a business analyst. In these implementations, the collaboration may include performing data cleaning/preparation/ETL, data enrichment/predictive analysis/text analytics, and visual data discovery. Further, in these implementations, data may be processed locally in a big data platform, the processed data may be cached at an in-memory layer, and data may be viewed using dashboards. In the following disclosure,FIGS. 2-8and the associated descriptions relate to data storage technology that may be used with data engine integration. Thereafter, data engine integration features that may be implemented using the systems described with reference toFIGS. 2-8are discussed in more detail with reference toFIGS. 1 and 9-13.

In some implementations, the big data engine may include a data refinement engine that provides data refinement capabilities for local files, as well as a big data file system. The data refinement engine has the ability to push the processing into the big data file system. In the following disclosure,FIG. 14and the associated description relates to the data refinement engine, andFIGS. 15-54and the associated descriptions relate to user interfaces for the data refinement engine.

In some implementations, a database system integrates with an outside source for refining data (e.g., Open Refine). In these implementations, the database system may integrate the web graphical user interface (GUI) of a data refinement source (e.g., Open Refine) into the database system web server, but keep an independent data refinement source (e.g., Open Refine) server along with the database system web server. In these implementations, the database system keeps most of the data refinement source (e.g., Open Refine) GUI intact and automatically directs the “refined data” file to data import.

In some examples, the database system may have a web GUI based on an independent design of data wrangling features, but invoke web commands of the data refinement source (e.g., Open Refine) server to achieve the desired effect. In other examples, the database system may extract and reuse individual action modules out of the data refinement source (e.g., Open Refine) server.

FIG. 1illustrates example integration with a data engine. As shown, a system10includes a dashboard layer11, an in-memory layer12, a data engine layer13, a data storage layer14, and a data acquisition layer15. The dashboard layer11provides an interface for data analysis and review by a data analyst. The dashboard layer11receives user input related to desired data analysis/viewing and produces output that is responsive to the received user input and that presents the data desired by the analyst. The dashboard layer11may generate a dashboard that is dynamic and flexible. The dashboard layer11interacts with the in-memory layer12to request data needed to service user requests, receive data from the in-memory layer12based on requests, and use the received data to generate dashboard output.

The in-memory layer12may include an embedded in-memory, column-oriented, distributed, analytic data store that is coupled to the dashboard layer11and that provides rapid query response and analytic processing. In the in-memory layer12, data is partitioned in various nodes on memory components and processed in parallel using load balancing and failover techniques. The in-memory layer12receives requests from the dashboard layer11and services the requests to the extent the in-memory layer12has the data needed to satisfy the requests. The in-memory layer12may send data to the dashboard layer11without processing or after performing analytics on the data. The in-memory layer12interacts with the data engine layer13to gather data needed to satisfy requests from the dashboard layer11or to otherwise populate its data store.

The data engine layer13performs various data analytics operations on data. For instance, the data engine layer13may perform filtering, aggregation, wrangling, searching, data mining, text analytics, on demand loading, incremental refreshing, streaming, data blending, complex ETL workflows, and multi-sourcing. Data wrangling will be described in more detail below. The data engine layer13may receive requests from the in-memory layer12to provide data that represents analytics performed on raw data. The data engine layer13may access the raw data from the data storage layer14and perform the needed analytics and/or may request the data storage layer14to perform the analytics and provide the data after the analytics have been performed. Performing the analytics at the data storage layer14may save time because all of the data does not need to be transmitted to the data engine layer13for the analytics to be performed.

The data storage layer14may include one or more sources of large volume data. For example, the data storage layer14may include a Hadoop Distributed File System (HDFS), a column-oriented database management system that runs on top of HDFS (Hbase), SQL on Hadoop, a web service, a search server, a relational database management system (RDBMS), streaming sources, a NoSQL database, or any other type of large volume data source. The data storage layer14may store data and may be able to send data to the data engine layer13either as raw, unprocessed data or processed data that includes results of performing analytics on the raw data.

The data acquisition layer15acquires data that is stored in the data storage layer14. The data acquisition layer15may use any types of data acquisition techniques to acquire data.

FIG. 2shows an example conceptual diagram of a computer system100that may be used in the system10. For example, computer system100can be implemented on or more computers (or nodes). As shown, computer system100can be conceptually represented as having two data storage areas, a hard disk104and a memory108. The computer system100includes a dashboard application130. Dashboard application130can include an interface (as described in detail below) for displaying grids and graphs based on underlying data to a user.

For example, memory108can be a random access memory or a flash memory. In some implementations, memory108allows data items to be read and written in a substantially similar amount of time regardless of an order in which the data items are access. In this regard, memory108can be different from, for example, hard disk104where the time to read and write data items can vary significant depending on the physical locations of the data items in the recording medium and because of, e.g., mechanical limitations such as media rotation speeds and arm movement delays.

Memory108includes an in-memory data store112. For example, the in-memory data store can be partitioned into one or more data sub sets116a-c. For example, one or more data sub sets116a-ccan include partitions (e.g. a portion) of one or more tables within data store112. Although three data sub sets116a-care shown and described here, there can be fewer or more (perhaps several more) than the three data sub sets116a-c. Each data sub set116a-cis associated with one or more processing units120a-c. Although three processing units120a-care shown and described here, there can be fewer or more (perhaps several more) than the three processing units120a-c. In some examples, a processing unit120acan be associated with more than one data sub set116a-c.

For example, processing units120a-ccan be one or more processor cores of a multi-core processor. For examples, multi-core processors can have two cores (dual-core CPUs, for example AMD Phenom II X2 and Intel Core Duo), four cores (quad-core CPUs, for example AMD Phenom II X4, Intel's i5 and i7 processors), six cores (hexa-core CPUs, for example AMD Phenom II X6 and Intel Core i7 Extreme Edition 980X), eight cores (octo-core CPUs, for example Intel Xeon E7-2820 and AMD FX-8350), ten cores (for example, Intel Xeon E7-2850), or more. In some implementations, a multi-core processor implements multiprocessing in a single physical package.

In some implementations, the computer system100can be implemented across multiple nodes. For example, a first processing unit120acan each be a processor core of a multi-core processor in a first node, and a second processing unit120bcan be a processor core of a multi-core processor in a second, different, node. In some implementations, while processing unit120ais physically located in a first node (e.g. a first processor core in the first node), processing units120band120ccan be physically located in a second, different node (e.g. second, different processor cores in the second node). In some implementations, data sub set116acan be physically located in the first node, and data sub sets116band116c, corresponding respectively to each of processing units120band120c, can be physically located in the second, different node. Although a few example combinations of processor cores and partitioned data sets are described here, a person of ordinary skill in the art would understand that any number of combinations of processor cores and partitioned data sets, spread out over a single node or multiple nodes, are possible.

In some examples, one or more database transactions can be processed in the form of one or more queries124a-cto the in-memory analytic data store112. For example, a high level database transaction can be divided into the multiple queries124a-c. In some examples, the number of queries124a-ccan be as high as a number of parallel processing units120a-cthat are available to process the queries124a-cin parallel. As shown, the queries124a-ccan be processed in parallel by the respective processing units120a-c. For example, query124amay require the summation of a column of data (e.g., numbers) residing in a portion of the data sub set116a. For example, the column of data relates to sales made by a customer over a period of time. This summation operation can be handled by respective processing unit120a. Substantially at the same time, a different (but perhaps related) operation, e.g. retrieving transaction dates for the sales fields being processed through the summation operation, can be handled by processing unit120boperating on data sub set116b. The results from respective queries124aand124bcan be sent back to a query engine (see e.g.FIG. 3described in further detail below) to assemble the information for, e.g., final display.

For example, computer systems implementing the techniques described herein (e.g. computer system100ofFIG. 2) uses information about an application and/or design aspects of a dashboard application130to generate queries124a-cto the in-memory data store. For example, dashboard application130can include a dashboard interface, as described in detail below, in which two or more grids (e.g. tables of data) are based on same or similar content. In some implementations, the computer system100can cause a single combined query (e.g., only query124a) or parallel queries (e.g., queries124a-c) to be executed on the in-memory data store for the two or more grids. In some implementations, dashboard application130can have two visualizations representing, e.g. sales trends over time through both a line chart and a grid of data. In the computer system100, the data needed for the two visualizations can be the same and so can be based on a either a single query or multiple parallel queries to in-memory analytic data store112. In some examples, dashboard application130can include two visualizations (not shown) based on selecting key performance indicators (KPIs) from a same set of underlying data in in-memory data store112. Because the underlying data involved is the same, the visualizations can be executed together—i.e. a single query or multiple parallel queries can be executed together. In some implementations, dashboard application130can include visualizations that are based on same or similar filtering criteria, and as such queries corresponding to these visualizations can be combined into a single query and/or executed together.

In some implementations, a data service engine128can receive data from multiple high volume data storage systems and load the received data into in-memory data store112. In some examples, data service engine128can perform parallel data loading into data store112through parallel processes128a-c. For example, processes128a-ccan load data from a corresponding data sources (not shown) into respective in-memory data store sub sets116a-cin parallel. In some implementations, the loaded data can be all of the market intelligence data needed to generate output for an end application, e.g., a dashboard/visualization engine as described in further detail below.

The in-memory analytic data store112can enable bigger data volume given its partitioned and parallel processing structure. For instance, current in-memory technologies are limited to two billion rows. By dividing datasets into partitions (e.g., data store sub sets116a-c), each partition or sub set116a-ccan have up to two billion rows, which increases the overall data volume. The partitioning can be performed on a single node or over multiple nodes as described below. For single node partitioning, data partitions are distributed across multiple cores on a single machine and grids/views are processed in parallel across all cores on a single multi-processor node. For multiple node partitioning, data partitions are distributed within and across multiple nodes (e.g., machines) and queries processed in parallel within and across multiple nodes.

In some implementations, the in-memory analytic data store112can provide broader analytic functionality. For instance, current in-memory cubes do not support full filter and metric functionality. In current in-memory cubes, “single pass” queries can be executed on underlying data. As such, complex business questions, such as, returning regional sales for those customers that bought certain widgets worth more than a predetermined number, could not be run on the data. The in-memory analytic data store112, however, extends to “multi-pass” analytics with multiple levels of aggregation and/or filtering. For example, computer system100can process metrics having conditions. In some examples, computer system100can also set qualification filters on the data.

In some implementations, the computer system ofFIG. 2can be implemented on a single node. Referring toFIG. 3, an example architecture of a single node200is shown. Node200can be a server implementing an in-memory analytic data store280. Node200can include an application tier215, a data management tier220, and a data service engine290. Application tier215includes an application programming interface (API) layer230and an intelligence dashboard/visualization engine240. For example, API layer230includes specifications describing how components in data management tier220can interact with other components, e.g., one or more web services250. For example, API layer230interfaces with web services250to receive data from one or more other applications (e.g., market intelligence data) and/or to provide collaborative functionality with the one or more other applications (e.g., receive user input from the one or more other applications and provide data to be output by the one or more other applications).

Dashboard/visualization engine240interacts with one or more of web applications, mobile applications, and documents260to receive user input and provide user output. For instance, dashboard/visualization engine240can generate a user interface400as shown inFIG. 5. For example, dashboard/visualization engine240can interact with a web or mobile application to output the user interface400on a user's device, e.g. a handheld device. Dashboard/visualization engine240also can output user interface400as a document or file that a user device is able to store and display. Application tier210can be a tightly-coupled with globally optimized query execution across multiple visualizations in single dashboard. Application tier210can also include a “data-only” JSON REST API and can provide super-fast search-style selectors.

Data management tier220can include a query execution engine270and an in-memory data store280. Query execution engine270receives queries (similar to queries124a-cdescribed in connection withFIG. 2) from application tier210and processes the received queries on data stored in in-memory data store280. Query execution engine270can access data from the in-memory data store280, perform analytics on the accessed data, and provide, to the application tier215, the accessed data and/or the results of the performed analytics. In some implementations, query execution engine270can divide a database transaction into a plurality of queries for processing on the respective data partitions.

In-memory data store280can be partitioned as shown. In some implementations, in-memory data store280can be partitioned to include, e.g., multiple partitioned tables225a-cand one or more shared tables235that are stored in-memory. In some implementations, while each of the partitioned tables225a-cis associated with a corresponding processor core, shared tables235can be accessed by multiple processor cores at substantially the same time. For example, the in-memory data store280can include a customer transactions table that can be partitioned such that each of the partitioned tables225a-chas one million customer transaction entries. In some implementations, a shared table can be a customer table that is shared among two or more processor cores.

Query execution engine270is configured to process queries to multiple partitioned tables225a-cand one or more shared tables235in parallel to provide fast data retrieval and enable a larger volume of data to be stored in-memory. For example, partition tables225a-ccan include a plurality of customer transaction records. Data management tier220can be a high-performance in-memory data tier that performs distributed in-memory analytics on the customer transaction records.

As explained above, data management tier220can have data partitioned across multiple processing cores and can perform parallel execution of queries across all cores according to a partition logic. In some implementations, a partition attribute can be defined to couple the processing cores to the respective data partition table e.g., any one of partition tables225a-c. For example, if a partition table225acontains customer transaction information, a customer transaction attribute such as a transaction identification code (“ID”) can be used as a partition attribute. In this regard, in some examples, the transaction ID can be processed through a hash function and sent to partition tables225a-cto determine which partition225a-chas the corresponding transaction information. In some implementations, while multiple customers can be located in a partition table225a, a customer located on partition table225acan remain on that partition table225aindefinitely (e.g., until the customer record is reallocated elsewhere).

Data service engine290can receive data from multiple high volume data storage systems and load the received data into the in-memory data store280in the data management tier220. The data service engine290can perform parallel data loading into the in-memory data store280from multiple data sources. The loaded data can be all of the market intelligence data accessible to generate output through the dashboard/visualization engine240. For example, data service engine290loaded information can be based on one or more of information contained on files, the cloud, a relational database management system (RDMBS), information from Apache Hadoop (an open source software framework for large scale storage and processing of data), multidimensional expressions (MDX), search query results, stream, and sampling information.

In some implementations, any arbitrary schema can be loaded into the in-memory analytic data store. In some implementations, the in-memory analytic data store280can be loaded with multiple star schemas and not just a single star schema. A star schema organizes a database such that business process data is separated into facts, which hold measurable, quantitative data about a business, and dimensions which are descriptive attributes related to the facts. For example, facts can include sales price, sale quantity, and time, distance, speed, and weight measurements. Related dimension attribute can include product models, product colors, product sizes, geographic locations, and salesperson names. In one star schema, the data is organize such that the fact table is typically located at the center of the star schema with the dimension table surrounding the fact table. Thus, multiple star schemas can each have a facts table at its center and a plurality of associated dimensional tables surrounding the facts tables.

In some implementations, fact tables at multiple levels can be loaded into the in-memory analytic data store. As an illustration, a first star schema can include sales transactions information, including customer information, transaction detail at a timestamp level, and store of purchase information. A second star schema can include store inventory information, such as products information, sales associates' information, and purchase information at a weekly inventory level. A third star schema can include corporate-level pricing data. Thus, each star schema represents data at a different level of granularity and detail. In some implementations, the in-memory analytic data store280can be loaded with all such star schemas.

FIG. 4illustrates an example system300with multiple nodes310,320. The system300includes master nodes310, further delineated as master nodes310a-c, and worker nodes320, further delineated as worker nodes320a-d. AlthoughFIG. 4illustrates three master nodes310a-cand four worker nodes320a-d, the system300can include more (perhaps, many more) or fewer master nodes310a-cand worker nodes320a-d.

As shown, each of the master nodes310a-cincludes an API layer325, a dashboard and/or visualization engine330, a query execution engine335, and an administration engine340. The API layer, dashboard/visualization engine330, and query execution engine335can be similar to the API layer230, the dashboard/visualization engine240, and the query execution engine270described above with respect toFIG. 3, except with for query execution engine270operating over multiple, different worker nodes320a-d. Administration engine340handles administration functionality for the corresponding master node310, including user security, multi-tenant administration, versioning, and process monitoring. Each of master nodes310a-ccan be operated on a separate machine.

As shown, each of the worker nodes320a-dincludes a corresponding in-memory analytic data store345a-d, each of which can be similar to the in-memory data store280described above with respect toFIG. 3. Each of worker nodes320a-dcan perform load balancing and failover operations for its own in-memory analytic data store nodes and/or across all of the worker nodes320. In this regard, in some implementations, a status of a node is monitored. If, for example, a node (or a core within the node) fails or the load on a node (or a core within the node) exceeds a predetermined maximum, its load is immediately redistributed across the remaining nodes (or cores). For example, if an abnormal condition state is detected with respect to one or more nodes (or cores in the nodes), a failover can be effected to another one of the plurality of nodes (or processor cores) to ensure continued operation.

Each of the worker nodes320a-dcan receive data from multiple large volume data sources and load the received data in parallel as described above. For example, each worker node320a-dcan be in communication with one or more corresponding data sources355a-d. AlthoughFIG. 4illustrates a one-to-one correspondence between worker nodes320a-dand data sources355a-d, it should be understood that any variation of relationships between the worker nodes320-a-dand data sources355a-dis possible. For example, a single data source, e.g., data source355a(say, a Hadoop system), can provide data to all four worker nodes320a-d. The data sources355a-dcan include high volume data storage systems. Accordingly, a data services engine (e.g. data service engine290ofFIG. 3) can load data from the data sources355a-din parallel into the in-memory data stores345a-d. In some implementations, the loaded data can be all of the market intelligence data needed to generate output through a dashboard/visualization engine.

In some implementations, the raw data from one or more information sources, e.g., a Hadoop system, can be processed before being loaded (e.g. via data service engine290ofFIG. 3) to an in-memory analytic data store. An example implementation of an interface for such processing is described in U.S. provisional Application No. 61/932,099, filed Jan. 27, 2014.

The system300can be configured differently depending on the type of application and the amount of data needed to support the application. For instance, for a market intelligence application that uses 2.2 billion rows, the system300can have a memory footprint of 59 GB and can have a hardware configuration of a single server with 32 cores and 1 TB of RAM. For a social media application that uses 2.8 billion rows, the system300can have a memory footprint of 100 GB and can have a hardware configuration of a single server with 40 cores and 1 TB of RAM. For an e-commerce application that uses 3.8 billion rows, the system300can have a memory footprint of 500 GB and can have a hardware configuration of a single server with 80 cores and 1 TB of RAM. For a social media application that uses 80 billion rows, the system300can have a memory footprint of 5-6 TB and can have a hardware configuration of 100 worker nodes, each with 16 cores and 144 GB of RAM, which results in a total of 1600 cores and 14 TB of RAM.

The system300can be configured to support use case characteristics with data volume in the 100's of GB to 1 TB range. In these cases, the system300can provide fast response time, as all executions are against in-memory datasets and datasets and queries are partition-friendly. The system300can serve mostly external-facing applications, although some applications can be internal. The data volume that can be handled by system300may not be limited to a particular size, such as 1 TB. In fact, depending on the available nodes in system300, a variety of data volumes can be serviced by system300.

FIG. 5illustrates an example user interface400of an intelligence dashboard. As shown, interface400comprises a plurality of control objects410-440. For example, control objects include grids (e.g. data displayed in table format), graphs, text fields, shapes, etc. that users can use to navigate through the data presented through interface400. Interface400can be powered by the in-memory analytic data store described throughout this disclosure (e.g., in-memory analytic data store112ofFIG. 2). In this regard, in some implementations, the analytic data store powers an extensive market intelligence network that provides the data shown in user interface400. For example, computer systems implementing the techniques described herein (e.g. computer system100ofFIG. 2) uses information about an application and/or design aspects of dashboard400to generate queries to the in-memory data store.

For example, all of the market intelligence data used by and displayed through the intelligence dashboard interface400can be loaded into the in-memory analytic data store. In this example, user interface400receives user input defining filter criteria410related to the market intelligence information a user seeks. Filter criteria410can include demographics data or any other type of data as shown in interface400or otherwise available to filter the market intelligence data stored in the in-memory analytic data store. For example, the user may be able to filter the data by gender, age, relationship status, education level, income bracket, urbanicity, etc. A query execution engine (e.g. query execution engine270ofFIG. 3) can receive the user input defining filter criteria410, and execute queries (e.g. queries124a-cofFIG. 2) to access the market intelligence data stored in an in-memory analytic data store. In some examples, the query execution engine can receive the accessed data (e.g. data from the in-memory analytic data that complies with the filter criteria410), perform analytics on the accessed data, and output the results of the analytics to user interface400.

As shown inFIG. 5, the user interface400specifies the demographic data used to generate the dashboard output broken down into various categories420(e.g. as shown in charts418a-c) and outputs ranked lists of interests422-a-efor people that fall within the demographic profile440defined by the filter criteria410. For example, the categories420can include what percentage of the relevant population is married, has attended college, or lives in an urban area. Other types of output and data visualization can be used. In addition, the user interface400can receive additional user input to refine or change the filter criteria410or the results sought and the user interface400can dynamically update in short order given the in-memory data retrieval and processing that occurs responsive to the additional user input.

By way of example,FIG. 6shows a user interface500of an intelligence dashboard also powered by an analytical in-memory data store (e.g., in-memory analytic data store112ofFIG. 2). Interface500displays a customer report505to, e.g., a sales associate in a retail store. In an example, the sales associate can view the customer report505on a store computer.

In some examples, graphical user interface500includes customer portion520that displays information indicative of customers who are, e.g. in a particular geographic location (say, the retail store). Customer portion520displays customer information520a-520h, with each item of customer information520a-520hrepresenting a customer. A user can select customer information520a-520hby, for example, using a mouse to click on, or using a touch screen display to touch, a desired item of customer information520a-520h. When an item of customer information520a-520his selected, interface500displays information pertaining to the selected customer. In the interface500ofFIG. 6, a viewer of graphical user interface500, e.g., the sales associate, has opted to view information associated with the customer represented by customer information520a.

A query execution engine (e.g. query execution engine270ofFIG. 3) can receive the user input, e.g., selection of customer information520a-520h, and execute queries (e.g. queries124a-cofFIG. 2) to access the market intelligence data stored in an in-memory analytic data store. In some examples, the query execution engine can receive the accessed data (e.g. data from the in-memory analytic data that complies with the filter criteria410), perform analytics on the accessed data, and output the results of the analytics to user interface500.

As shown, interface500includes past purchases link502, selection of which causes interface500to display information indicative of past purchases of the customer that is selected via customer portion520. Interface500also includes suggested items link, selection of which causes interface500to display suggestions information504indicative of suggested items that a particular customer (e.g., the customer selected from customer portion520) may be interested in and want to purchase. Suggestions information504can based on analyzing data that is retrieved from an in-memory analytic data store. For example, suggestions information504can be based on customers' past purchases. Interface500includes shopping bag link506, selection of which causes graphical user interface500to display items that a particular customer wishes to purchase. Interface500includes profile link508, selection of which causes interface500to be updated to display a customer profile of a particular customer (e.g., the customer selected via currently present customer portion520).

Interface500includes top occasions portion510that displays information (e.g., a graph) indicative of the top occasions for which a particular customer (e.g., customer520a) has purchased merchandise. Information for top occasions portion510can be generated based on analytics performed on market intelligence data contained in an in-memory data store. In this example, top occasions portion510is generated by tracking past purchases of the customer and then categorizing the types of purchase (e.g., based on various occasions). In another example, top occasions portion510displays information indicative of the top occasions for a group of customers, e.g., rather than displaying the top occasions for a particular customer.

Interface500also displays top categories information512, e.g., information specifying top categories of goods that have been purchased by a particular customer and/or by a group of customers at a retail store. Information for top categories portion510can be generated based on analytics performed on market intelligence data pertaining to the particular customer and/or the group of customers contained in an in-memory data store. In some implementations, interface500can include basket analysis portion514—for display of information indicative of types of goods that are currently in an electronic shopping cart of a customer.

Graphical user interface500also includes spending history portion516to display information indicative of how much money a particular customer (e.g., the customer selected in portion520) has spent with the retailer over a period of time. Information for spending history portion516can be generated based on analytics performed on market intelligence data pertaining to the particular customer contained in an in-memory data store. Spending history portion516can include a timeline516a, e.g., a representation of the period of time over which spending is tracked. Spending history portion516also includes information516bthat specifies an average amount of money a particular customer has spent with the retailer over a period of time. Interface500also includes portion518for display of information indicative of past purchases and/or transactions of a particular customer.

FIGS. 7 and 8illustrate example topologies for applications leveraging an in-memory, distributed, analytic data store. InFIG. 7, an example topology600includes an Internet Protocol (IP) load balancer610, multiple web server nodes620, multiple in-memory analytic data store nodes630, and a data staging area640. The IP load balancer610receives user requests over the Internet and balances the user requests across the web server nodes620. The web server nodes620process the user requests and access data needed to serve the user requests from the multiple in-memory analytic data store nodes630. Each web server node can use the operating system RHEL 6.2, can have a 12 core Intel Xeon @ 2.24 GHz central processing unit, and can have 32 GB of RAM.

The multiple in-memory analytic data store nodes630store data in a partitioned manner and perform parallel processing of the partitioned data. The multiple in-memory analytic data store nodes630are clustered for load balancing and failover and serve queries/requests from the web server nodes620. The multiple in-memory analytic data store nodes630communicate with one another to perform synchronization of changes made to the dataset. Each in-memory analytic data store node can use the operating system RHEL 6.2, can have a 32 core Intel Xeon @ 2.24 GHz central processing unit, and can have 1 TB of RAM. The full dataset can be replicated on each server.

The data staging area640accesses data to be loaded into the in-memory analytic data store nodes630. The data staging area640stages the data in a manner that enables parallel loading of the data into the in-memory analytic data store nodes630.

InFIG. 8, an example topology700includes an IP load balancer510, multiple web server nodes720, multiple in-memory analytic data store nodes730, and a relational database management system (RDBMS)740. The IP load balancer710receives user requests over the Internet and balances the user requests across the web server nodes720. The web server nodes720process the user requests and access data needed to serve the user requests from the multiple in-memory analytic data store nodes730. Each web server node can use the operating system Windows Server 2003 Enterprise x64 Edition (SP2), can have a Quad Core Intel Xeon L5520 @ 2.27 GHz central processing unit, and can have 6 GB of RAM.

The multiple in-memory analytic data store nodes730store data in a partitioned manner and perform parallel processing of the partitioned data. The multiple in-memory analytic data store nodes730are clustered for load balancing and failover and serve queries/requests from the web server nodes720. The multiple in-memory analytic data store nodes730communicate with one another to perform synchronization of changes made to the dataset. Each in-memory analytic data store node can be a model Sun Fire X4800 M2 server, can use the operating system RHEL 6.1, can have an 80 core Intel Xeon @ 2.40 GHz with hyper threading central processing unit, and can have 1 TB of RAM. The full dataset can be replicated on each server.

The RDBMS740stores data to be loaded into the in-memory analytic data store nodes730. In some implementations, the RDBMS740loads data into the in-memory analytic data store nodes730in parallel.

FIG. 9illustrates an example user/execution workflow. In the example user/execution workflow, a web graphical user interface (GUI)910receives user input related to data analysis and presentation requests and communicates with an in-memory master node920to receive the data needed to provide output for the analysis and presentation requests. The in-memory master node920includes a server922with an HDFS browser924and a data query engine926. The server922communicates with a metadata storage unit930to receive metadata related to the data analysis and presentation requests. The received metadata may be needed to service the requests itself or may be retrieved to enable the server922to determine the best way to service the requests. The server922determines whether analytics needed to service the analysis and presentation requests needs to be performed by the data query engine926and communicates the determinations to the data query engine926. The server922also uses the HDFS browser924to arrange a connection with a data storage system940. The data storage system940includes a name node942and multiple, data nodes944and946. The name node942is the centerpiece of an HDFS file system by keeping the directory tree of all files in the file system and tracking where across the cluster the file data is kept. The data nodes944and946store the data on the data storage system940and each include a data execution engine that is capable of performing analytics on the stored data. Although two data nodes944and946are shown for brevity, more (perhaps, many more) data nodes may be included in the data storage system940.

The HDFS browser924communicates with the name node942and the data nodes944and946to retrieve data needed to service the data analysis and presentation requests. The data query engine926also communicates with the name node942and the data execution engines on the data nodes944and946to perform the queries and necessary analytics on the data needed to service the data analysis and presentation requests. For analytics that the data execution engines are able to perform, the data query engine926requests that the analytics be performed prior to the data being sent. For analytics that the data execution engines are unable to perform, the data query engine926requests raw data and performs the analytics on the raw data. The server922receives the analyzed data needed to service the data analysis and presentation requests from the HDFS browser924and/or the data query engine926and provides the data to the web GUI910for output. By causing performance of at least a portion of the analytics at the data storage system940, data retrieval and analytics may have increased speed given that all of the raw data does not need to be communicated to the data query engine926.

FIG. 10illustrates example data flow in data fetching. As shown, data nodes1010access data on a big data engine and perform analytics using an execution engine. After performing the analytics, the data nodes1010send the results of the analytics to an in-memory data store1020for storage. The in-memory data store1020caches the analyzed data and can service queries on the analyzed data, as well as perform additional analytics on the analyzed data. Using these techniques for data fetching may enable fetching of data from a bid data engine (e.g., HDFS) at a speed of 20-30 MB/s on ten nodes.

A big data storage and processing platform may include a Hadoop big data system. Business intelligence products may be natively integrated with a Hadoop system. These technologies may combine the high performance of in-memory technology with the massive storage capability of the Hadoop system.

In implementations that integrate in-memory technology with a big data system, raw data in the big data system, such as Hadoop, needs to be discovered, cleaned, filtered, aggregated, and loaded into the in-memory database before conducting the in-memory analysis. In these implementations, the big data engine may conduct at least some of these operations on the raw data in an efficient way.

In some examples, a user interface for users to browse the raw data may be stored in a big data system. Through the user interface, including, for example, the user interfaces described below, users may define different data transformation rules to clean the data and perform data wrangling operation. Also, through the same user interface, users may pick the relevant columns from the relevant tables, apply filtering conditions, and define the aggregation and other functions to define the final tables (e.g., OLAP tables) for the data analysis. The in-memory system may provide the engine to automatically generate and execute queries to conduct the transformation in a distributed way.

In addition to the data importing (to the in-memory system) functionality, the big data engine also may support direct data access where the in-memory server may issue an SQL query to the big data engine directly. The query may be limited to a single table only or may span multiple tables.

FIG. 11illustrates an example big data engine architecture1100. The Big Data Engine1110is composed of one Big Data Query Engine (BDQE) paired with multiple (e.g., many) Big Data Execution Engines (BDEE). As shown, Big Data Engine1110may include multiple instance of a single BDQE paired with multiple BDEEs.

One BDQE is paired with one in-memory (e.g., PRIME) master server. It receives Big Data Requests from the in-memory (e.g., PRIME) master node. The main function of BDQE is twofold. First, it is responsible to compile and generate an optimized query tree. Secondly, it plays the role of the coordinator of all BDEEs. It dispatches the query tasks to all the execution engines so that loads are balanced and data are processed locally on each big data (e.g., Hadoop) data node. BDQE is a standalone server process that may be sitting on any machine. It could coexist with the in-memory (e.g., PRIME) master server; sit on a big data (e.g., Hadoop) node or any machine outside a Hadoop cluster.

BDEEs are deployed on the big data (e.g., Hadoop) data nodes with one BDEE process sitting on each big data (e.g., Hadoop) data node. The role of the execution engines is to execute the query task they receive from the query engine. The execution results are streamed to the in-memory (e.g., PRIME) slave nodes. The BDEEs also update the BDQE with heartbeats and execution status. As shown, the Big Data Engine1110may interact with multiple in-memory clusters1120and1130simultaneously.

FIG. 12illustrates an example big data query engine1200. In this example, BDQE is a standalone server process. The in-memory (e.g., PRIME) master connects to BDQE through a DB Role that is configured with the IP address of the BDQE machine and a port. Big Data Requests are structured SQL statements. The request is packed into ProtoBuf binary format and sent to BDQE through the connection. The BDQE may support two types of Big Data queries: Data Import (DI) and Direct Data Access (DDA). Both types may carry filter and aggregation to limit the data being imported and processed. Queries are queued inside BDQE. A thread pool picks up queries to process from the queue.

For each Big Data Request, the BDQE thread compiles the request into a query tree. The query tree is further optimized to obtain optimal performance. Each query tree is a sequence of processing steps that the execution engine takes and executes. It starts with a “load” query where the system reads the raw big data (e.g., Hadoop) data. It ends with a “store” step where the system streams the data to the in-memory (e.g., PRIME) slave nodes.

A query tree tells the BDEE what to do with the data and in what order since data is distributed in a big data (e.g., Hadoop) cluster. Another major job the BDQE performs is to dispatch splits (the block of data distributed on the big data (e.g., Hadoop) data nodes). By default each data block is replicated three times. Accordingly, the same block could be found on three different nodes. BDQE's find the one to dispatch so that overall the load is balanced. The BDQE is configured to be aware of the location of the big data (e.g., Hadoop) name node. With each Big Data Request, the BDQE obtains the data splits from the big data (e.g., Hadoop) name node. The BDQE relies on the coordinator to find the best data node to dispatch these splits.

The coordinator inside each BDQE receives heartbeats and task process status on a regular basis. That gives BDQE the information about the health and load of each BDEE. The query dispatch is based on this information.

FIG. 13illustrates an example pipelined execution engine architecture1300. The Big Data Execution Engine (BDEE) is another server process deployed on each big data (e.g., Hadoop) data node. It receives the query and a data split from the BDQE. The query and the split combined are called a sub task. BDEE employs a pipelined architecture1300. Four processing stations may be responsible for the following four data operations: Data Loading, Data Wrangling, Data Process and Data Streaming. The output of one processing station is the input of another. By separating I/O related processing steps into separate processing stations, I/O intensive work and data crunching tasks are in separated threads. In this regard, the CPU resource is better utilized.

Each Processing Station has its own subtask queue. Each also contains a pool of threads that pick subtasks from the queue. The output from a Processing Station is entered into the subtask queue of the subsequent Processing Station.

The output of the BDEEs is streamed to the in-memory (e.g., PRIME) slave nodes through a Data Transmitter. The transmitter client is responsible to perform the data partition. The partition logic is passed from the in-memory (e.g., PRIME) master and carried with the query to the BDEEs. The transmitter client maintains one bucket for each partition. When the result is received from BDEE row by row, the data is put into the corresponding bucket based on the partition rules. When a bucket is full the bucket is transmitted to the in-memory (e.g., PRIME) corresponding slave node.

Another role the data transmitter plays is to support streaming aggregation. To support aggregation when data is distributed, the transmitter performs aggregations in two steps. At the first step, the BDEE calculates the aggregations on each data node. Note that the local aggregation result still needs to be aggregated one more time at the global level. The second (global) aggregation is conducted by the transmitter. For instance, the second (global) aggregation is conducted at the receiver end of the transmitter. The aggregation group-by key is the same as the partition key. As a result, each partition naturally contains all the partially aggregated data within each group-by group. The receiver performs the final aggregation. Not all aggregations can be done in this fashion. Sum, Count, Max and Min fall in this category. Average is a typical example where the calculation occurs in two separate steps (e.g., the average of the local average is not the global average). To increase the speed of average calculation, the system calculates the Sum and Count separately at each node and then calculates the global average using the calculated Sum and Count at each. This technique broadens the aggregation functions that the system supports. In addition to the data transmitting function, the transmitter also supports data partition and streaming aggregation to increase the speed of how data is processed and loaded to the in-memory system.

FIG. 14illustrates an example architecture1400of a system that performs data wrangling. The system with the architecture1400shown inFIG. 14may be used to generate and display the user interfaces shown in following figures. The example architecture1400may be implemented in the wrangling module of the data engine layer13fromFIG. 1. The architecture1400provides a user the tools for manipulating data before the system analyses the data. For example, changing abbreviations to their full versions such as changing NY to New York or formatting times and dates to a common format.

FIG. 15illustrates an example user interface1500for selecting a data source. As shown, the user interface1500has a variety of sources a user may select to import data. In this example, the user selects to import data from a file.

FIG. 16illustrates an example user interface1600for selecting file data to import. The user interface1600is displayed in response to the user selecting to import data from a file in the user interface1500. The user interface1600allows users to upload from a machine, a uniform resource locator (URL), or from a Clipboard. In this example, the user selects the browse control with the “From My Computer/Network” button selected.

FIG. 17illustrates an example user interface1700for selecting a file from a computer. The user interface1700is displayed in response to the user selecting the browse control with the “From My Computer/Network” button selected in the user interface1600. The user interface1700allows users to select a file from a machine and the file upload options list all the File types that a data refinement source (e.g., OpenRefine) supports (e.g., Excel, CSV, text, XML, JSON, et al.).

FIG. 18illustrates an example user interface1800that displays data from a selected file. The user interface1800is displayed in response to the user selecting a file in the user interface1700. As shown, after uploading the file, the first step is to parse the file. After parsing, a user may either click on Refine or jump to the Mappings stage. In this example, the user selects the refine control to refine the data parsed from the file.

FIG. 19illustrates an example user interface1900that displays a refine data interface. The user interface1900is displayed in response to the user selecting the refine control in the user interface1800. As shown, the user interface1900has three options: List of Transformations and the corresponding UI, Sequence of Steps which is the data refinement (e.g., Open Refine) script, and Suggestions. The Suggestions is part of the intuitive UI where the system suggests to the user some of the transformation functions based on what they click in the preview of data below.

FIG. 20illustrates an example user interface2000that displays a refine data interface with a sequence built. The user interface2000is displayed in response to the user performing some transformations in the user interface1900. In this example, the system has received some transformations performed by the user and has built the sequence of steps shown based on the transformations. The user has the option to undo or redo the sequence steps by using the vertical slider shown. From this point, the user clicks on the mappings control.

FIG. 21illustrates an example user interface2100that displays a mappings interface. The user interface2100is displayed in response to the user selecting the mappings control in the user interface2000. In the previous interfaces, the system was dealing with columns. In the user interface2100, the system maps those columns to attributes and metrics. As shown, the system displays a preview of the mapping at the top of the user interface2100. The system may display inline mapping (e.g., the first row displays the attribute versus metrics).

FIG. 22illustrates an example user interface2200for selecting a type of data to import. As shown, the user interface2200enables a user to import data to and/or from a dashboard, a document, or a report. In this example, the user selects to import data from a document.

FIG. 23illustrates an example user interface2300for selecting a data source. As shown, the user interface2300has a variety of sources a user may select to import data. In this example, the user selects to import data from a file.

FIG. 24illustrates an example user interface2400for selecting file data to import. The user interface2400is displayed in response to the user selecting to import data from a file in the user interface2300. The user interface2400allows users to upload from a machine, a uniform resource locator (URL), or from a Clipboard. In this example, the user selects an Xtab file to import.

FIG. 25illustrates an example user interface2500that displays data from a selected Xtab file. The user interface2500is displayed in response to the user selecting an Xtab file. As shown, after uploading the file, the first step is to parse the file. The user interface2500has an option to click on the Xtab option on the “Parse” step. Any file can be an Xtab file. In this example, the user selects the Xtab option on the “Parse” step.

FIG. 26illustrates an example user interface2600related to figuring an Xtab layout. The user interface2600is displayed in response to the user selecting the Xtab option in the user interface2500. The user interface2600shows that the system is figuring the Xtab layout. For instance, when the user clicks on the Xtab option, the system may send the file from the data refinement (e.g., Open Refine) source to the database server (e.g., I-server).

FIG. 27illustrates an example user interface2700resulting from selection of the Xtab option. The user interface2700is displayed in response to figuring the Xtab layout in the user interface2600completing. The user interface2700is changed to reflect columns and not attribute\metrics so that it feels natural to other transformations. In this example, the user selects the refine control to refine the data parsed.

FIG. 28illustrates an example user interface2800that displays a refine data interface. The user interface2800is displayed in response to the user selecting the refine control in the user interface2700. As shown, the user interface2800has three options: List of Transformations and the corresponding UI, Sequence of Steps which is the data refinement (e.g., Open Refine) script, and Suggestions. The Suggestions is part of the intuitive UI where the system suggests to the user some of the transformation functions based on what they click in the preview of data below.

FIG. 29illustrates an example user interface2900that displays a refine data interface with a sequence built. The user interface2900is displayed in response to the user performing some transformations in the user interface2800. In this example, the system has received some transformations performed by the user and has built the sequence of steps shown based on the transformations. The user has the option to undo or redo the sequence steps by using the vertical slider shown. From this point, the user clicks on the mappings control.

FIG. 30illustrates an example user interface3000that displays a mappings interface. The user interface3000is displayed in response to the user selecting the mappings control in the user interface2900. In the previous interfaces, the system was dealing with columns. In the user interface3000, the system maps those columns to attributes and metrics. As shown, the system displays a preview of the mapping at the top of the user interface3000. The system may display inline mapping (e.g., the first row displays the attribute versus metrics).

FIG. 31illustrates an example user interface3100for selecting a type of data to import. As shown, the user interface3100enables a user to import data to and/or from a dashboard, a document, or a report. In this example, the user selects to import data from a document.

FIG. 32illustrates an example user interface3200for selecting a data source. As shown, the user interface3200has a variety of sources a user may select to import data. In this example, the user selects to import data from a database.

FIG. 33illustrates an example user interface3300for identifying database data to import. The user interface3300is displayed in response to the user selecting to import data from a database in the user interface3200. The user interface3300allows a user to build a query. At this point, the system just displays the “columns” generated from the query and not the mappings. The user may click on the refine control or choose to skip to mappings using the mappings control. In this example, the user clicks on the refine control.

FIG. 34illustrates an example user interface3400that displays a refine data interface. The user interface3400is displayed in response to the user selecting the refine control in the user interface3300. As shown, the user interface3400has three options: List of Transformations and the corresponding UI, Sequence of Steps which is the data refinement (e.g., Open Refine) script, and Suggestions. The Suggestions is part of the intuitive UI where the system suggests to the user some of the transformation functions based on what they click in the preview of data below. The system may send the data from the database server (e.g., I-server) to the data refinement (e.g., Open Refine) source in response to the user selecting the refine control in the user interface3300.

FIG. 35illustrates an example user interface3500that displays a refine data interface with a sequence built. The user interface3500is displayed in response to the user performing some transformations in the user interface3400. In this example, the system has received some transformations performed by the user and has built the sequence of steps shown based on the transformations. The user has the option to updo or redo the sequence steps by using the vertical slider shown. From this point, the user clicks on the mappings control.

FIG. 36illustrates an example user interface3600that displays a mappings interface. The user interface3600is displayed in response to the user selecting the mappings control in the user interface3500. In the user interface3600, the system maps the columns to attributes and metrics. As shown, the system displays a preview of the mapping at the top of the user interface3600.

FIG. 37illustrates an example user interface3700for selecting a type of data to import. The user interface3700is displayed in response to completion of the mappings in the user interface3600. As shown, the user interface3700enables a user to import data to and/or from a dashboard, a document, or a report and continue refinement and other data wrangling operations.

FIG. 38illustrates an example user interface3800that displays a refine data interface. The user interface3800may be displayed for any data refinement action on imported data. As shown, the user interface3800enables a user to refine data by selecting a data wrangling operation in a “Refine you data” portion of the user interface3800. In this example, the user has selected the “Split” operation to split a column into multiple columns.

FIG. 39illustrates an example user interface3900that displays a refine data interface. The user interface3900may be displayed for any data refinement action on imported data and may be displayed subsequent to the user interface3800. As shown, in the user interface3900, the preview dialogue is also clickable and there are three regions that a user can select: Row, Column, Cell (either the whole cell or partial data). Based on what user clicks, the system provides some suggestions. The suggestions are not intended to solve all data cleansing operations, but just help the user to intuitively understand and use the existing functions. In this example, the system has received user selection of row “1.”, the column “year”, and cell data “18 309,255”.

FIG. 40illustrates an example user interface4000that displays a refine data interface. The user interface4000may be displayed for any data refinement action on imported data and may be displayed subsequent to the user interface3900. As shown, the system has received user selection of some text in a cell and has determined and provided suggestions based on the selection. The corresponding suggestions may be to extract that data or split the column.

FIG. 41illustrates an example user interface4100that displays a refine data interface. The user interface4100may be displayed for any data refinement action on imported data and may be displayed subsequent to the user interface4000. As shown, the system has detected a mouse hover over the suggestion Split after “ ”. Based on detection of the mouse hover over the suggestion Split after “ ”, the system attempts to indicate to the user what would happen if that suggestion was adopted. As shown in the user interface4100, the system highlights the text that will be split after “ ”. The user interface4100may provide other options that the user may select. For example, the user interface4100may provide an option perform actions such as to review event logs, review the event logs to determine that a user added a product to an online shopping cart, but didn't purchase the item, limit a data set to these types of customers, select a subset of these customers, and then perform further analysis on the subset of customers.

FIG. 42illustrates an example user interface4200that displays a refine data interface with a sequence built. The user interface4200may be displayed for any data refinement action on imported data and may be displayed subsequent to the user interface4100. In this example, the system has received some operations either by manual user input or using the suggestions and has built the sequence of steps shown based on the operations. The user has the option to undo or redo the sequence steps by dragging the vertical slider shown.

FIG. 43illustrates an example user interface4300for splitting a column. The user interface4300may be displayed for any split refinement action on imported data. In this example, the user interface4300includes a menu for splitting a column and indicates operations performed based on various split options being selected.

FIG. 44illustrates an example user interface4400for splitting a column by first occurrence of a separator. The user interface4400may be displayed for any split by first occurrence of a separator refinement action on imported data. In this example, the user interface4400illustrates creation of two columns and deletion of the original column.

FIG. 45illustrates an example user interface4500for splitting a column by last occurrence of a separator. The user interface4500may be displayed for any split by last occurrence of a separator refinement action on imported data. In this example, the user interface4500illustrates creation of two columns and deletion of the original column.

FIG. 46illustrates an example user interface4600that displays a refine data interface. The user interface4600may be displayed for any data refinement action on imported data. As shown, in the user interface4600, the preview dialogue is clickable and there are three regions that a user can select: Row, Column, Cell (either the whole cell or partial data). Based on what user clicks, the system provides some suggestions. The suggestions are not intended to solve all data cleansing operations, but just help the user to intuitively understand and use the existing functions. In this example, the system has received user selection of text from a cell while the split refine option was selected. In this regard, the system provides some suggestions for splitting the column based on the selected text.

FIG. 47illustrates an example user interface4700that displays a refine data interface. The user interface4700may be displayed for any data refinement action on imported data and may be displayed subsequent to the user interface4600. As shown, the system has detected a mouse hover over the suggestion Split by Fixed Length. Based on detection of the mouse hover over the suggestion Split by Fixed Length, the system attempts to indicate to the user what would happen if that suggestion was adopted. As shown in the user interface4700, the system draws a vertical line through the column showing how the column would be split.

FIG. 48illustrates an example user interface4800for splitting a column by fixed length. The user interface4800may be displayed for any split by fixed length refinement action on imported data. In this example, the user interface4800illustrates examples of splitting by fixed lengths.

FIG. 49illustrates an example user interface4900that displays a refine data interface. The user interface4900may be displayed for any data refinement action on imported data. As shown, in the user interface4900, the preview dialogue is clickable and there are three regions that a user can select: Row, Column, Cell (either the whole cell or partial data). Based on what user clicks, the system provides some suggestions. The suggestions are not intended to solve all data cleansing operations, but just help the user to intuitively understand and use the existing functions. In this example, the system has received user selection of text from a cell while the split refine option was selected. In this regard, the system provides some suggestions for splitting the column based on the selected text.

FIG. 50illustrates an example user interface5000that displays a refine data interface. The user interface5000may be displayed for any data refinement action on imported data and may be displayed subsequent to the user interface4900. As shown, the system has detected a mouse hover over the suggestion Split before “ ”. Based on detection of the mouse hover over the suggestion Split before “ ”, the system attempts to indicate to the user what would happen if that suggestion was adopted. As shown in the user interface5000, the system highlights data showing how the column would be split.

FIG. 51illustrates an example user interface5100for splitting a column before a character or string. The user interface5100may be displayed for any split before a character or string refinement action on imported data. In this example, the user interface5100illustrates examples of splitting before a character or string.

FIG. 52illustrates an example user interface5200that displays a refine data interface. The user interface5200may be displayed for any data refinement action on imported data. As shown, in the user interface5200, the preview dialogue is clickable and there are three regions that a user can select: Row, Column, Cell (either the whole cell or partial data). Based on what user clicks, the system provides some suggestions. The suggestions are not intended to solve all data cleansing operations, but just help the user to intuitively understand and use the existing functions. In this example, the system has received user selection of text from a cell while the split refine option was selected. In this regard, the system provides some suggestions for splitting the column based on the selected text.

FIG. 53illustrates an example user interface5300that displays a refine data interface. The user interface5300may be displayed for any data refinement action on imported data and may be displayed subsequent to the user interface5200. As shown, the system has detected a mouse hover over the suggestion Split after “ ”. Based on detection of the mouse hover over the suggestion Split after “ ”, the system attempts to indicate to the user what would happen if that suggestion was adopted. As shown in the user interface5300, the system highlights data showing how the column would be split.

FIG. 54illustrates an example user interface5400for splitting a column after a character or string. The user interface5400may be displayed for any split after a character or string refinement action on imported data. In this example, the user interface5400illustrates examples of splitting after a character or string.

In some implementations, the user may select an option to have the system that stores the original data perform some of the processing. For example, in a Big Data file system, such as Hadoop, the user may instruct the file system to perform some of the processing. The user may place code for the processing in nodes of the file system and request that the file system execute the code. In some implementations, the user may provide an SQL script to the file system. The SQL script may need to be translated.

In some implementations, the system may be able to identify an appropriate amount of data to refine based on the subsequent steps performed on the data. The system may be able to look ahead to the subsequent steps and adjust the amount data being refined so that the system executes the subsequent steps more efficiently. For example, a system may determine to increase the size of strings that it is extracting from files in preparation for subsequent processing that may require longer strings despite receiving instructions to extract smaller strings.

In some implementations, the system may read data from multiple sources and the user may not necessarily be required to have knowledge of where the data is located. The user may provide the instructions for the type of data refinement, or wrangling, for the system to preform, and the system identifies the location to retrieve the data from based on the status of the various file systems.