Parallel query processing

Systems and methods are disclosed for distributing an in-memory data store over a plurality of independent data partitions. For example, the method includes associating each of the plurality of independent data partitions with at least one of a plurality of processing units such that one or more data sets in a corresponding each of the plurality of independent data partitions are processed by the at least one of the plurality of processing units. A query execution engine is provided for causing the plurality of processing units to execute, in parallel, a series of queries to the plurality of independent data partitions.

FIELD

This description relates to in-memory data processing.

BACKGROUND

Computer systems are used to manage and store data in a structure known as a database. As such, computers can be used to analyze data and generate reports based on the analysis results. For instance, computer systems can filter data and calculate metric values based on the filtered data, ultimately providing a report including the calculated metric values. A database is an organized repository of data. There are various ways in which the data can be organized. Schemas are used to describe the different organizations of data.

Computers systems have two types of physical data storage—disk (e.g., hard drive) storage and Random Access Memory (RAM) storage. Typically, computer systems have more disk storage than RAM, but it can often take longer (e.g., in the range of 100-1,000 times longer) to read data from the disk than from RAM. This can result in noticeable performance degradation.

SUMMARY

In general, one innovative aspect of the subject matter described in this specification can be embodied in methods that include the actions of distributing an in-memory data store over a plurality of independent data partitions, associating each of the plurality of independent data partitions with at least one of a plurality of processing units such that one or more data sets in a corresponding each of the plurality of independent data partitions are processed by the at least one of the plurality of processing units, and providing a query execution engine for causing the plurality of processing units to execute, in parallel, a series of queries to the plurality of independent data partitions.

These and other embodiments can each optionally include one or more of the following features. One or more of the plurality of processing units comprise individual processor cores of a multi-core processor. The distributing the in-memory data store over the plurality of independent data partitions comprises partitioning one or more tables of the in-memory data store across the plurality of independent data partitions as the plurality of independent data partitions. The actions include sharing one or more tables of the in-memory data store over the plurality of independent data partitions. The actions include presenting within a dashboard a visualization of the results from the processing of the series of queries to the in-memory data store. The actions further include generating the series of queries based on one or more visualizations to be displayed in a dashboard application. The plurality of independent data partitions are distributed over a plurality of nodes, each node comprising one or more of the plurality of processing units.

The actions include monitoring a processing status of the plurality of processing units, and balancing a processing load across the plurality of processing units in response to the monitored processing status. The actions include detecting one of a failure state or an abnormal condition state of the at least one of the plurality of processing units, and in response to the detected state, effecting a failover to another one of the plurality of processing units to ensure continued processing of series of queries. The actions include receiving a transaction to be processed on the in-memory data store, providing a first plurality of master nodes for deriving, from the received transaction, the series of queries to the in-memory data store, and a second plurality of worker nodes, wherein at least one of the second plurality of worker nodes comprises the plurality of independent data partitions across which the in-memory data store is distributed. The actions include, based on a user input, initiating the transaction at a selected one of a) an application programming interface layer, b) a dashboard engine, or c) both the application programming interface layer and the dashboard engine.

The actions include loading all data associated with a complete application into the in-memory data store. The actions include loading two or more star schemas into the in-memory data store. The actions include loading into the in-memory data store two or more fact tables. The fact tables include fact tables with varying levels of granularity. The actions include loading into the in-memory data store one or more sets of many-to-many relationship tables. The actions include providing, via the dashboard, a user interface for receiving user input defining one or more filters related to first desired information, and generating the series of queries in response to the user input defining the one or more filters related to the first desired information. The first desired information comprises market intelligence data. The actions include loading into the in-memory data store all market intelligence data used by the dashboard.

In general, another innovative aspect of the subject matter described in this specification can be embodied in methods that include the actions of receiving one or more transactions to be carried out on an in-memory data store, each transaction comprising one or more queries to respective independent data partitions within the in-memory data store, executing, in parallel, by processing units associated with corresponding ones of independent data partitions, the one or more queries, and presenting, within a dashboard associated with the in-memory data store, a visualization of the results from the processing of the series of queries to the in-memory data store.

Other embodiments of this aspect include corresponding systems, apparatus, and computer programs recorded on computer storage devices, each configured to perform the operations of the methods.

DETAILED DESCRIPTION

With the advent of in-memory processing, data can be loaded into the RAM (or sometimes, flash memory) instead of disk storage. This allows for faster query execution when dealing with large amounts of data.

In some implementations, an in-memory analytic database or data store can be distributed over a plurality of independent data partitions. Each of the plurality of independent data partitions can be associated with at least one of a plurality of processing units (e.g., processor cores of a multi-core computer processor). This way, one or more data sets of each of the plurality of independent data partitions are processed by at least one of the plurality of processor cores. For example, a data store or database transaction can include one or more queries performed on the in-memory data store. A series of such queries involving the in-memory data store can be processed in parallel across the plurality of processor cores.

A data store transaction can include, for example, a single logical operation on the data stored in the in-memory data store. For example, a data store transaction can include a transfer of funds from one bank account stored in a first data store at one location to another bank account stored in a second, different data store that is at another geographical location. The data store transaction can proceed as either a single query on the first data store, or a series of queries to multiple tables within the first data store to carry out the data store transaction. Similarly, the transaction on the second data store can proceed in the form of one or more update or write queries to the second data store.

In some examples, the in-memory analytic data store can provide broad schema support. In these examples, any arbitrary schema can be loaded into the in-memory analytic data store. For instance, the in-memory analytic data store can support multiple and varied dimensional schemas. Dimensional schemas are a type of database schemas that are used for storing and processing historical data. Dimensional schemas can be characterized in terms of facts, measures and dimensions. Facts describe data that can be analyzed, such as sales numbers or cost dollars. For example, facts can be aggregated. Measures are calculations based on facts. On the other hand, dimensions identify and categorize data. For example, dimensions describe data that can be used to generate groups and filters.

A typical relational schema can include definitions for tables within the database, fields of the tables, relationships between the tables, views involving the tables, indexes, functions, and other elements. A dimensional schema, on the other hand, is a type of database schema in which facts or measures that quantify a business are stored separately from descriptive elements (or dimensions) that describe the business. A dimensional schema can be further categorized as a star or snowflake schema, where the objects of the star or snowflake schema are database tables. In some implementations, the dimensional schema can be a single table.

A star schema consists of one or more fact tables referencing a plurality of dimension tables. In general, a fact table consists of the measurements, metrics or facts of a business process. The fact table is typically located at the center of the star schema with the dimension table surrounding the fact table. Dimension tables contain descriptive attributes (i.e. fields). For example, the descriptive attributes are typically textual fields. In a snowflake schema the tables are arranged such that the overall appearance resembles a snowflake shape. In particular, the snowflake schema is represented by centralized fact tables that are connected to multiple dimension tables.

Current in-memory cubes are often restricted to a single star schema, which makes the in-memory cubes closer to a cached result set based report. In some implementations, the in-memory analytic data store can be extended to receive data arranged in a variety of schemas. For example, the in-memory analytic data store can be loaded with multiple star schemas. For example, the in-memory analytic data store can be loaded with multiple fact tables and/or fact tables with varying grains, many-to-many (M-M) relationship tables, and entity-relation (ER) model semantics. In this manner, the in-memory analytic data store is made to resemble raw data that is staged in-memory.

With respect to ER model semantics, data that is loaded into the in-memory analytic data store is abstracted into business dimensions and measures. In some implementations, users can interact with the abstracted data to answer business questions using, e.g., a web based interface. In some examples, the web based interface can be used to build dashboards.

In some examples, the techniques described herein can be used for implementing an embedded in-memory, column-oriented, distributed, analytic data store that is coupled with a dashboard and/or visualization engine and that provides rapid query response and analytic processing. Data is partitioned in various nodes on memory components and processed in parallel using load balancing and failover techniques.

In some implementations, the in-memory data store described herein includes a column-oriented, distributed, analytic data store with a tightly-coupled dashboard and visualization engine for rapid application development. In this regard, the architecture implementing the in-memory analytic data store uses information about an application and/or design aspects of a dashboard to generate queries to the in-memory data store. In some implementations, the column-oriented, distributed, analytic data store can be embedded in memory components to provide fast responsiveness and data retrieval. With this structure, the analytic engine can be used to build fast-response analytical applications for various platforms, such as web applications and/or mobile device applications.

The in-memory analytic data store can be designed with a focus on read-only analytics, and may not be an ACID (Atomicity, Consistency, Isolation, Durability) compliant general purpose database. ACID refers to database properties that provide an assurance that transactions being processed on the database are being processed reliably. In some implementations, the analytic data store can thus handle any type of raw data. The analytic data store can provide fast query response time because the data is stored in-memory. In some implementations, the analytic data store can have data partitioned and processed in parallel. In some implementations, the data for an entire application can be loaded into memory and processed from memory.

FIG. 1shows an example conceptual diagram of a computer system100. For example, computer system100can be implemented on or more computers (or nodes). As shown, computer system100can be conceptually represented as having two data storage areas, a hard disk104and a memory108. The computer system100includes a dashboard application130. Dashboard application130can include an interface (as described in detail below) for displaying, to a user, grids and graphs based on underlying data.

For example, memory108can be a random access memory or a flash memory. In some implementations, memory108allows data items to be read and written in a substantially similar amount of time regardless of an order in which the data items are accessed. In this regard, memory108can be different from, for example, hard disk104where the time to read and write data items can vary significant depending on the physical locations of the data items in the recording medium and because of mechanical limitations such as media rotation speeds and arm movement delays.

Memory108includes an in-memory data store112. For example, the in-memory data store can be partitioned into one or more data sub sets116a-c. In this example, one or more data sub sets116a-ccan include partitions (e.g., a portion) of one or more tables within data store112. Although three data sub sets116a-care shown and described here, there can be fewer or more (perhaps many more) than the three data sub sets116a-c. Each data sub set116a-cis associated with one or more processing units120a-c. Although three processing units120a-care shown and described here, there can be fewer or more (perhaps many more) than the three processing units120a-c. In some examples, a processing unit120acan be associated with more than one data sub set116a-c.

For example, processing units120a-ccan be one or more processor cores of a multi-core processor. For examples, multi-core processors can have two cores (dual-core CPUs, for example AMD Phenom II X2 and Intel Core Duo), four cores (quad-core CPUs, for example AMD Phenom II X4, Intel's i5 and i7 processors), six cores (hexa-core CPUs, for example AMD Phenom II X6 and Intel Core i7 Extreme Edition 980X), eight cores (octo-core CPUs, for example Intel Xeon E7-2820 and AMD FX-8350), ten cores (for example, Intel Xeon E7-2850), or more. In some implementations, a multi-core processor implements multiprocessing in a single physical package.

In some implementations, the computer system100can be implemented across multiple nodes. For example, a first processing unit120acan each be a processor core of a multi-core processor in a first node, and a second processing unit120bcan be a processor core of a multi-core processor in a second, different, node. In some implementations, while processing unit120ais physically located in a first node (e.g., a first processor core in the first node), processing units120band120ccan be physically located in a second, different node (e.g., second, different processor cores in the second node). In some implementations, data sub set116acan be physically located in the first node, and data sub sets116band116c, corresponding respectively to each of processing units120band120c, can be physically located in the second, different node. Although a few example combinations of processor cores and partitioned data sets are described here, any number of combinations of processor cores and partitioned data sets, spread out over a single node or multiple nodes, may be used.

In some examples, one or more database transactions can be processed in the form of one or more queries124a-cto the in-memory analytic data store112. For example, a high level database transaction can be divided into the multiple queries124a-c. In some examples, the number of queries124a-ccan be as high as a number of parallel processing units120a-cthat are available to process the queries124a-cin parallel. As shown, the queries124a-ccan be processed in parallel by the respective processing units120a-c. For example, query124amay require the summation of a column of data (e.g., numbers) residing in a portion of the data sub set116a. For example, the column of data relates to sales made by a customer over a period of time. This summation operation can be handled by respective processing unit120a.

Substantially at the same time, a different (but perhaps related) operation, e.g., retrieving transaction dates for the sales fields being processed through the summation operation, can be handled by processing unit120boperating on data sub set116b. The results from respective queries124aand124bcan be sent back to a query engine (see, e.g.,FIG. 2described in further detail below) to assemble the information for display or other operation, such as storage or further analysis.

For example, computer systems implementing the techniques described herein (e.g., computer system100ofFIG. 1) uses information about an application and/or design aspects of a dashboard application130to generate queries124a-cto the in-memory data store. For example, dashboard application130can include a dashboard interface, as described in detail below, in which two or more grids (e.g., tables of data) are based on same or similar content. In some implementations, the computer system100can cause a single combined query (e.g., only query124a) or parallel queries (e.g., queries124a-c) to be executed on the in-memory data store for the two or more grids.

In an example, dashboard application130can have two visualizations representing sales trends over time through both a line chart and a grid of data. In the computer system100, the data needed for the two example visualizations can be the same and so can be accessed based on either a single query or multiple parallel queries to in-memory analytic data store112. In some examples, dashboard application130can include two or more visualizations based on selecting key performance indicators (KPIs) from a same set of underlying data in in-memory data store112. Because the underlying data involved is the same, the visualizations can be executed together—e.g., a single query or multiple parallel queries can be executed together. In some implementations, dashboard application130can include visualizations that are based on same or similar filtering criteria, and as such queries corresponding to these visualizations can be combined into a single query and/or executed together.

In some implementations, a data service engine128can receive data from multiple high volume data storage systems and load the received data into in-memory data store112. In some examples, data service engine128can perform parallel data loading into data store112through parallel processes128a-c. For example, processes128a-ccan load data from a corresponding data sources into respective in-memory data store sub sets116a-cin parallel. In some implementations, the loaded data can be all of the market intelligence data needed to generate output for an end user. For examples, the output can be displayed to the end user through an application interface, e.g., a dashboard/visualization engine as described in further detail below.

The in-memory analytic data store112can enable bigger data volume given its partitioned and parallel processing structure. For instance, certain in-memory technologies may be limited to two billion rows. By dividing datasets into partitions (e.g., data store sub sets116a-c), each partition or sub set116a-ccan have up to two billion rows, which increases the overall data volume. The partitioning can be performed on a single node or over multiple nodes as described below. For single node partitioning, data partitions are distributed across multiple cores on a single machine and grids/views are processed in parallel across all cores on a single multi-processor node. For multiple node partitioning, data partitions are distributed within and across multiple nodes (e.g., machines) and queries processed in parallel within and across multiple nodes.

In some implementations, the in-memory analytic data store112can provide broader analytic functionality. For instance, current in-memory cubes do not support full filter and metric functionality. In current in-memory cubes, “single pass” queries can be executed on underlying data. As such, complex business questions, such as, returning regional sales for those customers that bought certain widgets worth more than a predetermined number, could not be run on the data. The in-memory analytic data store112, however, extends to “multi-pass” analytics with multiple levels of aggregation and/or filtering. For example, computer system100can process metrics having conditions. In some examples, computer system100can also set qualification filters on the data.

In some implementations, the computer system ofFIG. 1can be implemented on a single node. Referring toFIG. 2, an example architecture of a single node200is shown. Node200can be a server implementing an in-memory analytic data store280. Node200can include an application tier215, a data management tier220, and a data service engine290. Application tier215includes an application programming interface (API) layer230and an intelligence dashboard/visualization engine240. For example, API layer230includes specifications describing how components in data management tier220can interact with other components, e.g., one or more web services250. For example, API layer230interfaces with web services250to receive data from one or more other applications (e.g., market intelligence data) and/or to provide collaborative functionality with the one or more other applications (e.g., receive user input from the one or more other applications and provide data to be output by the one or more other applications).

Dashboard/visualization engine240interacts with one or more of web applications, mobile applications, and documents260to receive user input and provide user output. For instance, dashboard/visualization engine240can generate a user interface400as shown inFIG. 4. For example, dashboard/visualization engine240can interact with a web or mobile application to output the user interface400on a user's device, e.g., a handheld device. Dashboard/visualization engine240also can output user interface400as a document or file that a user device is able to store and display. Application tier210can be a tightly-coupled with globally optimized query execution across multiple visualizations in single dashboard. Application tier210can also include a “data-only” JSON REST API and can provide super-fast search-style selectors.

Data management tier220can include a query execution engine270and an in-memory data store280. Query execution engine270receives queries (similar to queries124a-cdescribed in connection withFIG. 1) from application tier210and processes the received queries on data stored in in-memory data store280. Query execution engine270can access data from the in-memory data store280, perform analytics on the accessed data, and provide, to the application tier215, the accessed data and/or the results of the performed analytics. In some implementations, query execution engine270can divide a database transaction into a plurality of queries for processing on the respective data partitions.

In-memory data store280can be partitioned as shown. In some implementations, in-memory data store280can be partitioned to include, e.g., multiple partitioned tables225a-cand one or more shared tables235that are stored in-memory. In some implementations, while each of the partitioned tables225a-cis associated with a corresponding processor core, shared tables235can be accessed by multiple processor cores at substantially the same time. For example, the in-memory data store280can include a customer transactions table that can be partitioned such that each of the partitioned tables225a-chas one million customer transaction entries. In some implementations, a shared table can be a customer table that is shared among two or more processor cores.

Query execution engine270is configured to process queries to multiple partitioned tables225a-cand one or more shared tables235in parallel to provide fast data retrieval and enable a larger volume of data to be stored in-memory. For example, partition tables225a-ccan include a plurality of customer transaction records. Data management tier220can be a high-performance in-memory data tier that performs distributed in-memory analytics on the customer transaction records.

As explained above, data management tier220can have data partitioned across multiple processing cores and can perform parallel execution of queries across all cores according to a partition logic. In some implementations, a partition attribute can be defined to couple the processing cores to the respective data partition table e.g., any one of partition tables225a-c. For example, if a partition table225acontains customer transaction information, a customer transaction attribute such as a transaction identification code (“ID”) can be used as a partition attribute. In this regard, in some examples, the transaction ID can be processed through a hash function and sent to partition tables225a-cto determine which partition225a-chas the corresponding transaction information. In some implementations, while multiple customers can be located in a partition table225a, a customer located on partition table225acan remain on that partition table225aindefinitely (e.g., until the customer record is reallocated elsewhere).

Data service engine290can receive data from multiple high volume data storage systems and load the received data into the in-memory data store280in the data management tier220. The data service engine290can perform parallel data loading into the in-memory data store280from multiple data sources. The loaded data can be all of the market intelligence data accessible to generate output through the dashboard/visualization engine240. For example, data service engine290loaded information can be based on one or more of information contained on files, the cloud, a relational database management system (RDMBS), information from Apache Hadoop (an open source software framework for large scale storage and processing of data), multidimensional expressions (MDX), search query results, stream, and sampling information.

In some implementations, any arbitrary schema can be loaded into the in-memory analytic data store. In some implementations, the in-memory analytic data store280can be loaded with multiple star schemas and not just a single star schema. A star schema organizes a database such that business process data is separated into facts, which hold measurable, quantitative data about a business, and dimensions which are descriptive attributes related to the facts. For example, facts can include sales price, sale quantity, and time, distance, speed, and weight measurements. Related dimension attribute can include product models, product colors, product sizes, geographic locations, and salesperson names. In one star schema, the data is organize such that the fact table is typically located at the center of the star schema with the dimension table surrounding the fact table. Thus, multiple star schemas can each have a facts table at its center and a plurality of associated dimensional tables surrounding the facts tables.

In some implementations, fact tables at multiple levels can be loaded into the in-memory analytic data store. As an illustration, a first star schema can include sales transactions information, including customer information, transaction detail at a timestamp level, and store of purchase information. A second star schema can include store inventory information, such as products information, sales associates' information, and purchase information at a weekly inventory level. A third star schema can include corporate-level pricing data. Thus, each star schema represents data at a different level of granularity and detail. In some implementations, the in-memory analytic data store280can be loaded with all such star schemas.

FIG. 3illustrates an example system300with multiple nodes310,320. The system300includes master nodes310, further delineated as master nodes310a-c, and worker nodes320, further delineated as worker nodes320a-d. AlthoughFIG. 3illustrates three master nodes310a-cand four worker nodes320a-d, the system300can include more (perhaps, many more) or fewer master nodes310a-cand worker nodes320a-d.

As shown, each of the master nodes310a-cincludes an API layer325, a dashboard and/or visualization engine330, a query execution engine335, and an administration engine340. The API layer, dashboard/visualization engine330, and query execution engine335can be similar to the API layer230, the dashboard/visualization engine240, and the query execution engine270described above with respect toFIG. 2, except with for query execution engine270operating over multiple, different worker nodes320a-d. Administration engine340handles administration functionality for the corresponding master node310, including user security, multi-tenant administration, versioning, and process monitoring. Each of master nodes310a-ccan be operated on a separate machine.

As shown, each of the worker nodes320a-dincludes a corresponding in-memory analytic data store345a-d, each of which can be similar to the in-memory data store280described above with respect toFIG. 2. Each of worker nodes320a-dcan perform load balancing and failover operations for its own in-memory analytic data store nodes and/or across all of the worker nodes320. In this regard, in some implementations, a status of a node is monitored. If, for example, a node (or a core within the node) fails or the load on a node (or a core within the node) exceeds a predetermined maximum, its load is immediately redistributed across the remaining nodes (or cores). For example, if an abnormal condition state is detected with respect to one or more nodes (or cores in the nodes), a failover can be effected to another one of the plurality of nodes (or processor cores) to ensure continued operation.

Each of the worker nodes320a-dcan receive data from multiple large volume data sources and load the received data in parallel as described above. For example, each worker node320a-dcan be in communication with one or more corresponding data sources355a-d. AlthoughFIG. 3illustrates a one-to-one correspondence between worker nodes320a-dand data sources355a-d, it should be understood that any variation of relationships between the worker nodes320-a-dand data sources355a-dis possible. For example, a single data source, e.g., data source355a(say, a Hadoop system), can provide data to all four worker nodes320a-d. The data sources355a-dcan include high volume data storage systems. Accordingly, a data services engine (e.g., data service engine290ofFIG. 2) can load data from the data sources355a-din parallel into the in-memory data stores345a-d. In some implementations, the loaded data can be all of the market intelligence data needed to generate output through a dashboard/visualization engine.

In some implementations, the raw data from one or more information sources, e.g., a Hadoop system, can be processed before being loaded (e.g., via data service engine290ofFIG. 2) to an in-memory analytic data store. An example implementation of an interface for such processing is described in U.S. provisional Application No. 61/932,099, filed Jan. 27, 2014.

The system300can be configured differently depending on the type of application and the amount of data needed to support the application. For instance, for a market intelligence application that uses 2.2 billion rows, the system300can have a memory footprint of 59 GB and can have a hardware configuration of a single server with 32 cores and 1 TB of RAM. For a social media application that uses 2.8 billion rows, the system300can have a memory footprint of 100 GB and can have a hardware configuration of a single server with 40 cores and 1 TB of RAM. For an e-commerce application that uses 3.8 billion rows, the system300can have a memory footprint of 500 GB and can have a hardware configuration of a single server with 80 cores and 1 TB of RAM. For a social media application that employs massively parallel processing techniques and uses 80 billion rows, the system300can have a memory footprint of 5-6 TB and can have a hardware configuration of 100 worker nodes, each with 16 cores and 144 GB of RAM, which results in a total of 1600 cores and 14 TB of RAM.

The system300can be configured to support use case characteristics with data volume in the 100's of GB to 1 TB range. In these cases, the system300can provide fast response time, as all executions are against in-memory datasets and datasets and queries are partition-friendly. The system300can serve external-facing applications, although some applications can be internal. The data volume that can be handled by system300may not be limited to a particular size, such as 1 TB. In fact, depending on the available nodes in system300, a variety of data volumes can be serviced by system300.

FIG. 4illustrates an example user interface400of an intelligence dashboard. As shown, interface400comprises a plurality of control objects410-440. For example, control objects include grids (e.g., data displayed in table format), graphs, text fields, shapes, etc. that users can use to navigate through the data presented through interface400. Interface400can be powered by the in-memory analytic data store described throughout this disclosure (e.g., in-memory analytic data store112ofFIG. 1). In this regard, in some implementations, the analytic data store powers an extensive market intelligence network that provides the data shown in user interface400. For example, computer systems implementing the techniques described herein (e.g., computer system100ofFIG. 1) uses information about an application and/or design aspects of dashboard400to generate queries to the in-memory data store.

For example, all of the market intelligence data used by and displayed through the intelligence dashboard interface400can be loaded into the in-memory analytic data store. In this example, user interface400receives user input defining filter criteria410related to the market intelligence information a user seeks. Filter criteria410can include demographics data or any other type of data as shown in interface400or otherwise available to filter the market intelligence data stored in the in-memory analytic data store. For example, the user may be able to filter the data by gender, age, relationship status, education level, income bracket, urbanicity, etc. A query execution engine (e.g., query execution engine270ofFIG. 2) can receive the user input defining filter criteria410, and execute queries (e.g., queries124a-cofFIG. 1) to access the market intelligence data stored in an in-memory analytic data store. In some examples, the query execution engine can receive the accessed data (e.g., data from the in-memory analytic data that complies with the filter criteria410), perform analytics on the accessed data, and output the results of the analytics to user interface400.

As shown inFIG. 4, the user interface400specifies the demographic data used to generate the dashboard output broken down into various categories420(e.g., as shown in charts418a-c) and outputs ranked lists of interests422-a-efor people that fall within the demographic profile440defined by the filter criteria410. For example, the categories420can include what percentage of the relevant population is married, has attended college, or lives in an urban area. Other types of output and data visualization can be used. In addition, the user interface400can receive additional user input to refine or change the filter criteria410or the results sought and the user interface400can dynamically update in short order given the in-memory data retrieval and processing that occurs responsive to the additional user input.

By way of example,FIG. 5shows a user interface500of an intelligence dashboard also powered by an analytical in-memory data store (e.g., in-memory analytic data store112ofFIG. 1). Interface500displays a customer report505to, e.g., a sales associate in a retail store. In an example, the sales associate can view the customer report505on a store computer.

In some examples, graphical user interface500includes customer portion520that displays information indicative of customers who are, e.g., in a particular geographic location (say, the retail store). Customer portion520displays customer information520a-520h, with each item of customer information520a-520hrepresenting a customer. A user can select customer information520a-520hby, for example, using a mouse to click on, or using a touch screen display to touch, a desired item of customer information520a-520h. When an item of customer information520a-520his selected, interface500displays information pertaining to the selected customer. In the interface500ofFIG. 5, a viewer of graphical user interface500, e.g., the sales associate, has opted to view information associated with the customer represented by customer information520a.

A query execution engine (e.g., query execution engine270ofFIG. 2) can receive the user input, e.g., selection of customer information520a-520h, and execute queries (e.g., queries124a-cofFIG. 1) to access the market intelligence data stored in an in-memory analytic data store. In some examples, the query execution engine can receive the accessed data (e.g., data from the in-memory analytic data that complies with the filter criteria410), perform analytics on the accessed data, and output the results of the analytics to user interface500.

As shown, interface500includes past purchases link502, selection of which causes interface500to display information indicative of past purchases of the customer that is selected via customer portion520. Interface500also includes suggested items link, selection of which causes interface500to display suggestions information504indicative of suggested items that a particular customer (e.g., the customer selected from customer portion520) may be interested in and want to purchase. Suggestions information504can based on analyzing data that is retrieved from an in-memory analytic data store. For example, suggestions information504can be based on customers' past purchases. Interface500includes shopping bag link506, selection of which causes graphical user interface500to display items that a particular customer wishes to purchase. Interface500includes profile link508, selection of which causes interface500to be updated to display a customer profile of a particular customer (e.g., the customer selected via currently present customer portion520).

Interface500includes top occasions portion510that displays information (e.g., a graph) indicative of the top occasions for which a particular customer (e.g., customer520a) has purchased merchandise. Information for top occasions portion510can be generated based on analytics performed on market intelligence data contained in an in-memory data store. In this example, top occasions portion510is generated by tracking past purchases of the customer and then categorizing the types of purchase (e.g., based on various occasions). In another example, top occasions portion510displays information indicative of the top occasions for a group of customers, e.g., rather than displaying the top occasions for a particular customer.

Interface500also displays top categories information512, e.g., information specifying top categories of goods that have been purchased by a particular customer and/or by a group of customers at a retail store. Information for top categories portion510can be generated based on analytics performed on market intelligence data pertaining to the particular customer and/or the group of customers contained in an in-memory data store. In some implementations, interface500can include basket analysis portion514—for display of information indicative of types of goods that are currently in an electronic shopping cart of a customer.

Graphical user interface500also includes spending history portion516to display information indicative of how much money a particular customer (e.g., the customer selected in portion520) has spent with the retailer over a period of time. Information for spending history portion516can be generated based on analytics performed on market intelligence data pertaining to the particular customer contained in an in-memory data store. Spending history portion516can include a timeline516a, e.g., a representation of the period of time over which spending is tracked. Spending history portion516also includes information516bthat specifies an average amount of money a particular customer has spent with the retailer over a period of time. Interface500also includes portion518for display of information indicative of past purchases and/or transactions of a particular customer.

FIGS. 6 and 7illustrate example topologies for applications leveraging an in-memory, distributed, analytic data store. InFIG. 5, an example topology600includes an Internet Protocol (IP) load balancer610, multiple web server nodes620, multiple in-memory analytic data store nodes630, and a data staging area640. The IP load balancer610receives user requests over the Internet and balances the user requests across the web server nodes620. The web server nodes620process the user requests and access data needed to serve the user requests from the multiple in-memory analytic data store nodes630. Each web server node can use the operating system RHEL 6.2, can have a 12 core Intel Xeon @ 2.24 GHz central processing unit, and can have 32 GB of RAM.

The multiple in-memory analytic data store nodes630store data in a partitioned manner and perform parallel processing of the partitioned data. The multiple in-memory analytic data store nodes630are clustered for load balancing and failover and serve queries/requests from the web server nodes620. The multiple in-memory analytic data store nodes630communicate with one another to perform synchronization of changes made to the dataset. Each in-memory analytic data store node can use the operating system RHEL 6.2, can have a 32 core Intel Xeon @ 2.24 GHz central processing unit, and can have 1 TB of RAM. The full dataset can be replicated on each server.

The data staging area640accesses data to be loaded into the in-memory analytic data store nodes630. The data staging area640stages the data in a manner that enables parallel loading of the data into the in-memory analytic data store nodes630.

InFIG. 7, an example topology700includes an IP load balancer510, multiple web server nodes720, multiple in-memory analytic data store nodes730, and a relational database management system (RDBMS)740. The IP load balancer710receives user requests over the Internet and balances the user requests across the web server nodes720. The web server nodes720process the user requests and access data needed to serve the user requests from the multiple in-memory analytic data store nodes730. Each web server node can use the operating system Windows Server 2003 Enterprise x64 Edition (SP2), can have a Quad Core Intel Xeon L5520 @ 2.27 GHz central processing unit, and can have 6 GB of RAM.

The multiple in-memory analytic data store nodes730store data in a partitioned manner and perform parallel processing of the partitioned data. The multiple in-memory analytic data store nodes730are clustered for load balancing and failover and serve queries/requests from the web server nodes720. The multiple in-memory analytic data store nodes730communicate with one another to perform synchronization of changes made to the dataset. Each in-memory analytic data store node can be a model Sun Fire X4800 M2 server, can use the operating system RHEL 6.1, can have an 80 core Intel Xeon @ 2.40 GHz with hyper threading central processing unit, and can have 1 TB of RAM. The full dataset can be replicated on each server.

The RDBMS740stores data to be loaded into the in-memory analytic data store nodes730. In some implementations, the RDBMS740loads data into the in-memory analytic data store nodes730in parallel.