Patent Description:
Analytics applications generally access large datasets to perform analytic operations. When a user wishes to perform an operation on a dataset, the user identifies where the dataset is stored and the analytics application sends a query to the server storing the dataset. The server computer system storing the dataset executes the query against the dataset and returns the requested information to the analytics application.

Depending on the type of query, executing the query against the dataset can be extremely inefficient. For example, if a user's query requests information on only a small subset of rows of a database, executing the query directly against the database requires the server computer system to check each row to determine if the row satisfies the query. Additionally, if the database is subject to row-based access controls, the data a user is allowed to access may be incredibly sparse, thereby causing the database to perform the query and then remove the rows that the user is not allowed to access.

In order to increase the efficiency of the system, a server computer system may use an index of the database. When a query contains a filtering condition or row-based access controls, the database can identify the requested rows through the index. The server computer system may then use the row identifiers to access the rows stored in the database. While the use of an index is more efficient than directly searching the database for each query, performing a query against the index followed by a query against the database may be inefficient. That inefficiency is increased if the database is stored using one application, like APACHE PARQUET, while the index is created and stored by another application, like APACHE LUCENE.

One solution is to use the index to recreate rows of the database when the index is searched. In some scenarios, recreating the rows from the index may be faster than searching the database and only returning a small number of rows. In other situations, such as when all values in a single column are being used, using the index to recreate rows in the database is less effective.

Generally, the user is responsible for identifying the target of a search query. This means that a user must know where the dataset is stored before an analysis request can be sent to the server computer system. Additionally, the user has no way of indicating to the server computer system that the server computer should use the index, not use the index, and/or rebuild rows from the index.

Thus, there is a need for a system that dynamically selects a backing store for responding to a query based on a semantic analysis of the query.

Reference is made to <CIT>, which presents mirroring, in memory, data from disk to improve query performance. Further reference is made to <CIT>, which presents a data store access permission system with interleaved application of deferred access control filters. Further reference is made to <NPL>, which presents a simple indexing strategy and comparison of different reconstruction algorithms. The paper then proposes a new mirroring scheme, termed fractured mirrors, using both NSM and DSM models. Further refrene is made to <NPL>, wherein a hybrid filtering solution for main memory databases is disclosed Further reference is made to <CIT> (<NUM>-<NUM>-<NUM>) that discloses a solution of searching rows of a partitioned store based upon a given query and performing an optimization based upon the search.

The appended claims may serve to summarize the disclosure.

In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. It will be apparent, however, that embodiments may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring the present disclosure.

Embodiments are described in sections below according to the following outline:.

The invention of the present application is defined by the independent claims. In an embodiment, a server computer system stores one or more first datasets in a first data repository and one or more second datasets in a second data repository. The two data repositories may include a columnar data store and an index data repository. When a request to perform an analysis on a particular dataset is received, the server computer system determines whether the particular dataset is stored in each data repository. If the particular dataset is stored in both data repositories, the server computer system selects a data repository based on an attribute of the query, such as a size of the dataset being queried, whether the query is subject to access controls, whether the query includes an aggregation, and/or whether the query includes a filter condition.

In an embodiment, a method comprises storing, in a first data repository, one or more first datasets; storing, in a second data repository, one or more second datasets; receiving a request to perform an analysis on a particular dataset; determining that the particular dataset isstored in both the first data repository and the second data repository; selecting the second data repository based, at least in part, on the attribute of the request; responding to the request with data from the particular dataset stored in the second data repository.

<FIG> depicts a schematic diagram of a distributed computing system for performance of searches for resources.

In an embodiment, a distributed computing system comprises a server computer ("server") <NUM> coupled via network <NUM> to a host computing device <NUM>. The distributed computing environment can be within one or more data center, virtual computing facility or other hosting facilities connected to a network such as, for example, the Internet or other network; other embodiments can be within a networked home, office, or campus.

Network <NUM> broadly represents a combination of one or more local area networks, wide area networks, and/or internetworks and may include the public internet. The network <NUM> can connect multiple hosts <NUM> together within the distributed computing environment. Network <NUM> can be composed of multiple sub-networks connected together. For example, the network <NUM> can be an Internet Protocol Version <NUM>-based and/or an Internet Protocol Version <NUM>-based wired or wireless network or a combination of multiple such networks.

Host computing device <NUM> broadly represents one or many computers and the number of hosts <NUM> in the system at a given time may vary in different embodiments and any number may be used. In an embodiment, a host <NUM> can be a single computing device such as, for example, the computing device <NUM> described below with respect to <FIG>. Alternatively, a host <NUM> can be a single virtual computer instance that executes on a computing device facilitated by a virtualization layer interposed between the virtual computer instance and the computing device. The virtualization layer can be a virtual machine monitor such as, for example, virtual machine monitor <NUM> described below with respect to <FIG>.

Regardless if a single computing device or a single virtual computer instance, a host <NUM> can be configured with an operating system such as, for example, operating system <NUM> described below with respect to <FIG>. The operating system of a host <NUM> can manage low- level aspects of the host's operation including managing execution of processes, memory allocation, file input and output (I/O), and device I/O. Furthermore, the operating system of host <NUM> may manage presentation systems such as a windowed graphical user interface (GUI) and driving a computer display device such as a flat screen display or CRT. A host <NUM> may also be configured with a container system (e.g. the DOCKER container system) for running services within containers on the host's operating system.

Services that execute as processes on hosts in the distributed computing environment may be configured using the distributed configuration platform described herein or in <CIT>.

In an embodiment, host <NUM> comprises an application instance <NUM> which allows one or more services to be deployed on host <NUM> from one or more server computing devices, such as server <NUM>. In an embodiment, application instance <NUM> comprises resource request instructions <NUM>, a front-end graphical interface <NUM> and a command-line interface (CLI) <NUM> that may interoperate to provide the functions that are described further herein. In an embodiment, the front-end interface <NUM> and CLI <NUM> are programmed or configured to interact with one or more server-side functional units of server computer <NUM> as further described. Host computing device <NUM> also manages one or more resource requests <NUM> using processes that are further described in other sections.

A service that application instance <NUM> facilitates or hosts can be a single instance of a software product or software application installed on at least one of the hosts <NUM>. For example, a service might be a database server instance, a web server instance, or any other instance of a software product or a software application installed on one or more of the hosts <NUM>. Multiple different services may be installed on the hosts <NUM> including multiple different services on the same host <NUM>. For example, a service may be installed on multiple of the hosts <NUM> in a distributed, clustered, load balanced, or failover computing arrangement.

In an embodiment, server <NUM> comprises data repository <NUM> comprising data index <NUM> and access control lists <NUM>, data repository <NUM> comprising columnar data store <NUM> and data files <NUM>, and query analysis instructions <NUM> as further described. In an embodiment, query analysis instructions <NUM>, when executed by one or more processors, are programmed or configured to cause server computer system <NUM> to receive electronic digital messages that define database operations according to a structured query language, parse the messages to determine the operations, analyze the query to identify an attribute of the query, select a datastore based on the identified attribute, and execute the operations against the selected datastore. Typically, execution of the operations causes instructing one or more worker processes to execute builds of derived datasets, based on raw datasets, with data repository <NUM> and/or data repository <NUM>. In one implementation, query analysis instructions <NUM> comprises an executable instance of ELASTICSEARCH and/or a software layer in conjunction with ELASTICSEARCH.

In an embodiment data repository <NUM> comprises a distributed data storage system which stores one or more resources. The resources may comprise one or more of individual documents, folders, JSON blobs, rows in a database, user comments or flags, media files, build artifacts, and/or any other digital resource. Data repository <NUM> comprises data index <NUM> includes an index of datasets and/or other resources from data repository <NUM>. Data index <NUM> may be a distributed index for more efficient searches. Access control lists <NUM> comprise one or more access control policies to be applied to data repository <NUM>. For example, access control lists <NUM> may implement row-level access controls on data in a database. Access control lists may identify access controls for individual user identifications, role identifications, and/or user group identifications.

In an embodiment data repository <NUM> comprises a distributed data storage system which stores one or more resources. The resources may comprise one or more of individual documents, folders, JSON blobs, rows in a database, user comments or flags, media files, build artifacts, and/or any other digital resource. Data repository <NUM> comprises columnar data store <NUM> which stores data from one or more resources and makes the data available to analytics applications. Data repository <NUM> may additionally comprise data files <NUM>. Data files <NUM> may comprise one or more flat files that are used as a source of data stored in columnar data store <NUM>.

In an embodiment, data index <NUM> comprises one or more views of data stored in columnar data store <NUM>. Data repository <NUM> may additionally store index metadata which identifies one or more sets of data in data repository <NUM> that are additionally stored in data index <NUM>. The index metadata may also include data indicating which types of data are subject to access controls stored in access control lists <NUM>.

In an embodiment, query analysis instructions <NUM> comprises a set of one or more pages of main memory, such as RAM, in the server <NUM> into which executable instructions have been loaded and which when executed cause the server to perform the functions or operations that are described herein with reference to those modules. For example, the query analysis instructions <NUM> may comprise a set of pages in RAM that contain instructions which when executed cause performing the query analysis functions that are described herein.

The instructions may be in machine executable code in the instruction set of a CPU and may have been compiled based upon source code written in JAVA, C, C++, OBJECTIVE-C, or any other human-readable programming language or environment, alone or in combination with scripts in JAVASCRIPT, other scripting languages and other programming source text.

The term "pages" is intended to refer broadly to any region within main memory and the specific terminology used in a system may vary depending on the memory architecture or processor architecture.

In another embodiment, query analysis instructions <NUM> also may represent one or more files or projects of source code that are digitally stored in a mass storage device such as non-volatile RAM or disk storage, in the server <NUM> or a separate repository system, which when compiled or interpreted cause generating executable instructions which when executed cause the server <NUM> to perform the functions or operations that are described herein with reference to those modules. In other words, the drawing figure may represent the manner in which programmers or software developers organize and arrange source code for later compilation into an executable, or interpretation into bytecode or the equivalent, for execution by the server <NUM>.

<FIG> depicts an example method for selecting a backing store for responding to a request to perform an analysis on a particular dataset based on an attribute of the request. <FIG> is described in terms of a first data repository and a second data repository for the purpose of giving a clear example. While the description below may describe the first data repository as a columnar data store and the second data repository as a distributed index, in other embodiments, the first data repository is a distributed index and the second data repository is a columnar data store. Thus, a person of skill in the art would understand that either type of data repository may be selected in step <NUM> based on an attribute of a request.

Each of <FIG>, <FIG> and any other flow diagram or process description in this disclosure is intended to represent an algorithm or plan for programming computer programs at the same level of detail that skilled persons in this field use to communicate among themselves about the architecture, process or data flow, arrangement, ordering and content of similar computer programs of similar types. That is, the flow diagrams and process descriptions are presented at a level of detail that skilled persons in this field have determined to be sufficient to communicate between themselves the baseline information that is adequate to permit the coding,
completion, testing and deployment of computer programs that will execute and operate to provide the functions that are described herein.

At step <NUM>, one or more first datasets are stored in a first data repository. The first data repository may include one or more databases. As an example, one or more flat files may be read into a columnar data store, such as APACHE PARQUET. A dataset, as used herein, refers to a set of data to which a query may be directed. Thus, the first data repository may store datasets for different data types, customers, subsets of users in a customer group, and/or purposes for one or more users. For example, a first dataset may include sales data for a first business while a second dataset includes inventory data for the first business and a third dataset includes employee data for the first business. Each dataset may be queried individually using a structured query language which identifies the dataset.

At step <NUM>, one or more second datasets are stored in a second data repository. The second data repository may include one or more indexes of a dataset stored in the first data repository. For example, rows from a datable may be indexed based on the columns. The index may be distributed across a plurality of nodes such that a request for data may be federated across the plurality of nodes. An example of a distributed index which may include views of data stored in a database is ELASTICSEARCH.

In an embodiment, the second data repository stores views of one or more datasets from the first data repository. Additionally, one or more data repositories stored in the first data repository may not have a corresponding index in the second data repository. Thus, the datasets stored in the first data repository may differ from datasets stored in the second data repository. The second data repository may store index metadata that identifies which datasets stored in the first data repository are indexed in the second data repository. Additionally or/alternatively, a separate document may be stored in a separate location which identifies which datasets from the first data repository are indexed in the second data repository.

In an embodiment, the second data repository additionally includes a layer on top of one or more of the distributed indices which provides access controls for the distributed indices. For example, an access control layer may provide row-based access controls for rows of data stored in the first data repository and indexed in the second data repository. In an embodiment, the use of the access control layer may be optional for data indexed in the second data repository. The second data repository may store index metadata which identifies which datasets stored in the first data repository are subject to access controls in the second data repository. Additionally or/alternatively, a separate document may be stored in a separate location which identifies which datasets stored in the first data repository are subject to access controls in the second data repository.

At step <NUM>, a request to perform an analysis on a particular dataset is received. For example, host computing device <NUM> creates via user input, or a programmatic call from application instance <NUM> or an external system or program, a resource request <NUM> and transmits the resource request to the resource request instructions. The host computing device <NUM> may then transmit the resource request to server computer system <NUM>. Creation of the resource request <NUM> may be performed using a front-end graphical user interface which may be programmed to provide forms, GUI widgets, or other facilities to specify what datasets are requested. Additionally or alternatively, the resource request <NUM> may be generated by an analytics program in response to a request by a user for analytics on a particular dataset. The resource request <NUM> may conform to a structured query language for accessing one or more data repositories.

At step <NUM>, a determination is made that the particular dataset is stored in both the first data repository and the second data repository. For example, the server computer system <NUM> may parse a resource request in a structured query language to identify a dataset stored in the first data repository. The server computer may use one or more files or data stores to determine whether a view of the identified dataset is stored in the second data repository. For example, server computer system <NUM> may identify, in index metadata stored in the second data repository, data indicating that a view of the particular dataset is stored in the second data repository. While in an embodiment, the dataset is assumed to be stored in the first data repository, in other embodiments, the server computer system <NUM> may additionally utilize metadata and/or one or more mappings to determine whether the data is stored in the first data repository.

At step <NUM>, the second data repository is selected based, at least in part, on an attribute of the request. For example, the server computer system <NUM> may execute the query analysis instructions <NUM> to perform a semantic analysis of the query. The server computer system <NUM> may determine, from the semantic analysis of the query, a type of query and/or a target dataset for executing the query. The server computer system <NUM> may additionally obtain additional information regarding the target dataset for the query. Descriptions of the types of attributes used to select a data repository are described further herein in the Query Analysis section.

At step <NUM>, data from the particular dataset stored in the second data repository is used to respond to the request. For example, based on the semantic query analysis, the server computer <NUM> may determine that the more practical option is to execute the query against the columnar data store and return values, columns, and/or rows from the columnar data store in response to the query. Alternatively, the server computer <NUM> may determine that the more practical option is to request data from the distributed indexes. In embodiments where the query requests one or more rows, the server computer <NUM> may use the indices to rebuild the one or more rows and return the rebuilt rows in response to the query.

<FIG> depicts one example for performing an analysis of a query to determine which data repository to use as a backing store for responding to the query. By selecting a backing store based on an attribute of the query, the server computer <NUM> is able to identify the best data store for responding to a query on the fly. In embodiments where the query requests data subject to access controls or which would be most efficient to be accessed from the index, the server computer <NUM> may select the data repository containing the access controls and the index. In situations where the query requests data which would be most efficient to be accessed from a columnar data store, the server computer <NUM> may select the columnar data store for responding to the query.

<FIG> depicts a method of analyzing a query to identify a backing store for responding to the query. The individual elements of <NUM>-<NUM> in <FIG> may be used on their own in conjunction with element <NUM> to select a backing store, in conjunction with any of the other elements and element <NUM>, or in conjunction with all of the other elements. For example, in an embodiment a server computer system may identify a backing data store based only on elements <NUM> and <NUM>. In a separate embodiment, the server computer system may select a backing store based on elements <NUM>, <NUM>, <NUM>, and <NUM>.

In <FIG>, one or more datasets are stored in a columnar data store. An index data repository comprises a view of one or more datasets that are stored in the columnar data store. At step <NUM>, the server computer system receives a query using any of the methods described herein where the query identifies a dataset. At step <NUM>, the server computer system determines whether a view of the dataset is stored in the index data repository. For example, the index data repository may store index metadata identifying the datasets stored in the index data repository. The server computer system may request the index metadata from the index data repository to determine whether the dataset is identified in the index metadata. If a view of the dataset is not stored in the index data repository, the process moves to step <NUM> and the server computer system executes the query against the columnar data store. If a view of the dataset is stored in the index data repository, the process proceeds to step <NUM>.

At step <NUM>, the server computer system determines whether the dataset is subject to access controls which are stored in the index data repository. For example, the index metadata may include data identifying whether each dataset stored in the index data repository is subject to access controls. If the dataset is subject to access controls, the process continues to step <NUM> and the server computer system executes the query using the index data repository. If the dataset is not subject to access controls, the process continues with step <NUM>. By performing the analysis of the query to determine whether the dataset is subject to access controls, the server computer system can store a plurality of datasets for one or more users where a subset of the plurality of datasets are subject to access controls while another subset of the plurality of datasets are not subject to access controls. Thus, the use of access controls on one dataset would not require access control data to be stored for each stored dataset.

Additionally or alternatively, the server computer may use the index metadata to identify the access controls and request a pre-filter from the index data repository which identifies the resources in the columnar data store that the user may access. The server computer may then use the access controls as an additional filter to be used in conjunction with the query when the query is executed. Thus, step <NUM> may still be used to identify the access controls without being dispositive in determining which data repository is accessed.

At step <NUM>, the server computer system determines whether the dataset contains more than a threshold number of rows. For example, the server computer system may store a row threshold value identifying a threshold number of rows in a dataset. The server computer system may additionally store a value for each dataset stored in the columnar data store and/or in the index data repository indicating a number of rows in the dataset. If, for the requested dataset, the number of rows exceeds the row threshold value, the process continues with step <NUM>. If the row threshold value exceeds the number of rows in the dataset, the process continues to step <NUM> and the server computer system executes the query using the columnar data store. Thus, the server computer system essentially determines that the number of rows in the dataset is low enough that using the index data repository would not significantly increase performance of the server computer system.

At step <NUM>, the server computer system determines whether the query includes a column aggregation. A column aggregation, as described herein, refers to any use of all of the values in a particular column. For example, a column aggregation may include a summation of values in a column, an average of values in a column, a maximum or minimum value of values in a column, and/or a range of values in a column. The server computer system may use the semantic analysis of the query to determine whether the query includes any column aggregations. If the query includes a column aggregation, the process continues with step <NUM> and the server computer system executes the query using the columnar data store. If the query does not include a column aggregation, the process continues with step <NUM>.

At step <NUM>, the server computer system determines whether the query includes a filtering condition. A filtering condition, as described herein, refers to a condition that filters rows such that less than all of the rows may be used to respond to the query. For example, a filtering condition may include partial string matching, value searches, and/or row filters. The server computer system may use the semantic analysis of the query to determine whether the query includes any filtering conditions. If the query includes a filtering condition, the process continues with step <NUM> and the server computer system executes the query using the index data repository. If the query does not include a filtering condition, the process continues with step <NUM> and the server computer system executes the query using the columnar data store.

In an embodiment, the filtering condition may include filters generated by access controls as opposed to the access controls being dispositive. For example, the server computer system may determine that the dataset is subject to access controls. In response, the server computer system may identify, based, at least in part, on the requestor and the access control lists, one or more rows that user may access. The server computer system may augment the query to include a filter based on the access control list, such that aggregations may still be performed, but the server computer system does not return full rows of data that the requestor does not have access to. If, at step <NUM>, a filter based on access controls is identified, the server computer system may continue with step <NUM> and execute the query using the data repository.

In an embodiment, the server computer system may further determine whether a query with a filtering condition would return a small number of rows or a large number of rows before selecting a backing store for executing the query. For example, the server computer system may first use the index to identify a number of rows that satisfy the query. If the number of rows that satisfy the query are greater than a stored row threshold value, such as fifty rows, the server computer system may request the rows from the columnar data store. If the number of rows that satisfy the query are lower than the stored row threshold value, the server computer system may use the index to rebuild the rows.

In an embodiment, the server computer system uses a proportional value as the row threshold value. For example, if the row threshold value is set at ten percent, then the server computer system may respond to a query that would return less than ten percent of the rows of the dataset with rows rebuilt from the index data repository. Alternatively, if a query would return more than ten percent of the rows of the dataset, the server computer system may respond to the query with rows from the columnar data store.

In an embodiment, a subset of the datasets stored in the columnar data store are indexed in the index data repository. The server computer system may dynamically identify datasets to index into the index data repository based on the size of the datasets, the usage of the datasets, and/or other needs of the system.

In an embodiment, the server computer system stores a row threshold value identifying a threshold number of rows of a dataset, such that a dataset with less than the threshold number of rows is not stored in the index data repository. By limiting the datasets stored in the index data repository based on a number of rows, the server computer system may increase the efficiency of responding to queries while optimizing the storage space for the datasets. Thus, a dataset which is less likely to need an index is less likely to be stored in the index data repository.

In an embodiment, the server computer system identifies datasets to index in the index data repository based on usage. For example, in the method of <FIG>, at step <NUM>, the server computer system determines whether a view of the dataset is stored in the index data repository. If a view of the dataset is not stored in the data repository, the server computer system uses the dataset stored in the columnar data store. In an embodiment, in addition to using the dataset stored in the columnar data store, the server computer system increments a value indicating a number of times a query was executed against the dataset. The server computer system may additionally store a query threshold value. If the incremented value for a dataset exceeds the query threshold value, the dataset may be indexed into the index data repository.

In an embodiment, the server computer identifies datasets to index in the index data repository based on usage for a particular period of time. For example, the server computer may reset the incremented values for each dataset periodically, such as each twenty-four hours. Thus, a dataset may be indexed in the data repository if an incremented value indicating a number of uses of the dataset exceeds a threshold value within a twenty-four hour period. By using an incrementing value which resets after a particular period of time, the server computer system is able to optimize storage of datasets based on peak usage.

In an embodiment, the server computer only increments a value for a dataset if a query against the dataset would have been executed against the index data repository had the dataset been stored in both the columnar data store and the index data repository. For example, if a query against the dataset includes an aggregation of values in a column, the server computer system may not increment the value as the query would have been executed against the columnar data store. In contrast, if the query does not include an aggregation but does include a filtering condition, the server computer system may increment the value as the query would have accessed the index data repository instead of the columnar datastore.

By only incrementing the value when a query to the database would have been executed using the index data repository, the server computer system is able optimize storage of datasets based on likely usage. Thus, if a user group performs a large number of aggregation queries against a dataset, the server computer system may continue to only store the dataset in the columnar data store as adding the dataset to the index data repository may have a relatively small effect on performance.

For example, <FIG> is a block diagram that illustrates a computer system <NUM> upon which an embodiment may be implemented.

<FIG> is a block diagram of a basic software system <NUM> that may be employed for controlling the operation of computing device <NUM>. Software system <NUM> and its components, including their connections, relationships, and functions, is meant to be exemplary only, and not meant to limit implementations of the example embodiment(s). Other software systems suitable for implementing the example embodiment(s) may have different components, including components with different connections, relationships, and functions.

Software system <NUM> is provided for directing the operation of computing device <NUM>. Software system <NUM>, which may be stored in system memory (RAM) <NUM> and on fixed storage (e.g., hard disk or flash memory) <NUM>, includes a kernel or operating system (OS) <NUM>.

The OS <NUM> manages low-level aspects of computer operation, including managing execution of processes, memory allocation, file input and output (I/O), and device I/O. One or more application programs, represented as 502A, 502B, 502C. 502N, may be "loaded" (e.g., transferred from fixed storage <NUM> into memory <NUM>) for execution by the system <NUM>. The applications or other software intended for use on device <NUM> may also be stored as a set of downloadable computer-executable instructions, for example, for downloading and installation from an Internet location (e.g., a Web server, an app store, or other online service).

Software system <NUM> includes a graphical user interface (GUI) <NUM>, for receiving user commands and data in a graphical (e.g., "point-and-click" or "touch gesture") fashion. These inputs, in turn, may be acted upon by the system <NUM> in accordance with instructions from operating system <NUM> and/or application(s) <NUM>. The GUI <NUM> also serves to display the results of operation from the OS <NUM> and application(s) <NUM>, whereupon the user may supply additional inputs or terminate the session (e.g., log off).

OS <NUM> can execute directly on the bare hardware <NUM> (e.g., processor(s) <NUM>) of device <NUM>. Alternatively, a hypervisor or virtual machine monitor (VMM) <NUM> may be interposed between the bare hardware <NUM> and the OS <NUM>. In this configuration, VMM <NUM> acts as a software "cushion" or virtualization layer between the OS <NUM> and the bare hardware <NUM> of the device <NUM>.

VMM <NUM> instantiates and runs one or more virtual machine instances ("guest machines"). Each guest machine comprises a "guest" operating system, such as OS <NUM>, and one or more applications, such as application(s) <NUM>, designed to execute on the guest operating system. The VMM <NUM> presents the guest operating systems with a virtual operating platform and manages the execution of the guest operating systems.

In some instances, the VMM <NUM> may allow a guest operating system to run as if it is running on the bare hardware <NUM> of device <NUM> directly. In these instances, the same version of the guest operating system configured to execute on the bare hardware <NUM> directly may also execute on VMM <NUM> without modification or reconfiguration. In other words, VMM <NUM> may provide full hardware and CPU virtualization to a guest operating system in some instances.

In other instances, a guest operating system may be specially designed or configured to execute on VMM <NUM> for efficiency. In these instances, the guest operating system is "aware" that it executes on a virtual machine monitor. In other words, VMM <NUM> may provide para- virtualization to a guest operating system in some instances.

The above-described basic computer hardware and software is presented for purpose of illustrating the basic underlying computer components that may be employed for implementing the example embodiment(s). The example embodiment(s), however, are not necessarily limited to any particular computing environment or computing device configuration. Instead, the example embodiment(s) may be implemented in any type of system architecture or processing environment that one skilled in the art, in light of this disclosure, would understand as capable of supporting the features and functions of the example embodiment(s) presented herein.

Claim 1:
A data processing method comprising:
storing (<NUM>), in a columnar data store, one or more datasets;
storing (<NUM>), in an index data repository comprising one or more indexes of the one or more datasets stored in the columnar data store, at least one dataset, and one or more views of the one or more datasets stored in the columnar data store;
receiving (<NUM>) a query to perform an analysis on a particular dataset;
determining (<NUM>) whether a view of the particular dataset is stored in the index data repository based on index metadata identifying the datasets stored in the index data repository and access controls for the at least one dataset;
wherein the index metadata is stored in the index data repository;
determining (<NUM>), when a view of the particular dataset is stored in the index data repository, whether the particular dataset is subject to access controls based on the index metadata;
adding, when the particular dataset is subject to particular access controls, a filter to the query based on the particular access controls,
wherein the filter identifies resources in the columnar store accessible under the particular access controls; and
executing the query using the columnar data store.