Benchmark framework for cost-model calibration

In some aspects, there is provided a method including receiving an execution plan file, the execution plan file utilizing at least one operator of interest and further utilizing other actions separate from the at least one operator of interest. The method further includes forming an execution plan object by modifying the execution plan file by isolating the at least one operator of interest from the other actions. The method further includes performing a series of tests executing an extended execution plan object. The series of tests can include receiving the input data identified by the one or more pointers in the extended execution plan object, executing the extended execution plan object using the received input data, measuring, based on the execution of the extended execution plan object, at least one cost metric representative of execution of the at least one operator of interest, and outputting the measured cost metric.

TECHNICAL FIELD

The subject matter described herein relates to database management, and more particularly, cost-based optimization for query execution planning.

BACKGROUND

Database management systems have become an integral part of many computer systems. For example, some systems handle hundreds if not thousands of transactions per second. On the other hand, some systems perform very complex multidimensional analysis on data. In both cases, the underlying database may need to handle responses to queries very quickly in order to satisfy systems requirements with respect to transaction time.

Query execution engines may use an optimizer to find the optimal execution plans for database queries. This may involve the ordering of operators and the selection of the best-fitting implementation for each of these operators. To come up with such an optimal execution plan, a thorough investigation of each operator's performance characteristics is needed. The gathered information can then be used as the basis for a linear regression analysis to derive a mathematical model predicting the operator's performance.

Optimization typically relies on a collection of precise performance information for each operator. One approach for collecting this performance information would be to run a certain set of queries involving the operators of interest on a certain set of input data via an SQL interface of the database system and measure the execution time of these queries. This approach has a number of drawbacks, which render it basically useless for execution plan optimization.

SUMMARY

In some aspects, there is provided a system including at least one data processor; and at least one memory storing instructions which, when executed by the at least one data processor, result in operations including receiving an execution plan file, the execution plan file utilizing at least one operator of interest and further utilizing other actions separate from the at least one operator of interest. The operations further include forming, based on the execution plan file, an execution plan object by modifying the execution plan file by isolating the at least one operator of interest from the other actions; performing a series of tests executing an extended execution plan object, the extended execution plan object formed by modifying one or more dummy operators in the execution plan object to include pointers identifying input data. The series of tests include receiving the input data identified by the one or more pointers in the extended execution plan object, executing the extended execution plan object using the received input data, and measuring, based on the execution of the extended execution plan object, at least one cost metric representative of execution of the at least one operator of interest; and outputting the measured cost metric.

In some variations, the operations can further include one or more features disclosed herein including the following. Forming the extended execution plan object can include deserializing the execution plan file. Modifying the execution plan file can include replacing one or more of the other actions with one or more of at least one pointer identifying input data and one or more dummy operators. Replacing the one or more other actions can include replacing a table scan with the at least one pointer identifying the input data. Replacing the one or more other actions can include replacing a projection operator with a dummy projection operator that receives data produced by the operator of interest. The cost metric can be execution time of the extended execution plan object. The operations can further include generating the input data using a random number generator. The operations can further include determining, based on the at least one output cost metric, a cost-model for the operator of interest based on the measured cost metric. The system can further include a database execution engine; a benchmark framework communicatively coupled to the database execution engine; and one or more databases, where the benchmark framework receives the execution plan file, and forms the extended execution plan object, the benchmark framework performs the series of tests and outputs the measured cost metric, the input data is received from the one or more databases, and the benchmark framework determines the cost-model for the operator of interest based on the measured cost metric. The system can further include a cost-based optimizer communicatively coupled to the database execution engine and configured to perform operations including receiving a query request, compiling the received query request into an initial execution plan, using the compiled execution plan, identifying database parameters associated with the received query request, identifying, based on the identified database parameters and using one or more cost-models associated with one or more operators of interest associated with the received query request, an optimum version of the one or more operators of interest to execute the query, and executing the received query request using the identified optimum version.

In some aspects, there is provided a computer program product including a non-transitory machine-readable medium storing instructions that, when executed by at least one programmable processor, cause the at least one programmable processor to perform operations including receiving an execution plan file, the execution plan file utilizing at least one operator of interest and further utilizing other actions separate from the at least one operator of interest, forming, based on the execution plan file, an execution plan object by modifying the execution plan file by isolating the at least one operator of interest from the other actions, performing a series of tests executing an extended execution plan object, the extended execution plan object formed by modifying one or more dummy operators in the execution plan object to include pointers identifying input data. The series of tests include receiving the input data identified by the one or more pointers in the extended execution plan object, executing the extended execution plan object using the received input data, measuring, based on the execution of the extended execution plan object, at least one cost metric representative of execution of the at least one operator of interest, and outputting the measured cost metric.

In some variations, the operations can further include one or more features disclosed herein including the following. Forming the extended execution plan object can include deserializing the execution plan file. Modifying the execution plan file can include replacing one or more of the other actions with one or more of at least one pointer identifying input data and one or more dummy operators. Replacing the one or more other actions can include replacing a table scan with the at least one pointer identifying the input data. Replacing the one or more other actions can include replacing a projection operator with a dummy projection operator that receives data produced by the operator of interest.

In some aspects, there is provided a method including receiving an execution plan file, the execution plan file utilizing at least one operator of interest and further utilizing other actions separate from the at least one operator of interest, forming, based on the execution plan file, an execution plan object by modifying the execution plan file by isolating the at least one operator of interest from the other actions, performing a series of tests executing an extended execution plan object, the extended execution plan object formed by modifying one or more dummy operators in the execution plan object to include pointers identifying input data. The series of tests can include receiving the input data identified by the one or more pointers in the extended execution plan object, executing the extended execution plan object using the received input data, measuring, based on the execution of the extended execution plan object, at least one cost metric representative of execution of the at least one operator of interest, and outputting the measured cost metric.

The cost metric can be execution time of the extended execution plan object. The method can further include generating the input data using a random number generator. The method can further include determining, based on the at least one output cost metric, a cost-model for the operator of interest based on the measured cost metric. Replacing the one or more other actions can include replacing a projection operator with a dummy projection operator that receives data produced by the operator of interest.

DETAILED DESCRIPTION

In some example embodiments, a cost-based optimizer is used to find an optimal execution plan for performing database queries. The cost-based optimizer utilizes a cost model, which allows for predicting the “cost” of applying a certain operator implementation at a certain position in the execution plan. In various example embodiments, cost can be a performance metric, such as execution time, memory consumption, etc. The cost-based optimizer selects the “least costly” plan according to the cost model and passes it to an execution framework to be executed.

Running a complete query may involve a lot of overhead which is included in most cost-based measurements of the actual database query. The overhead cannot be separated from the cost information of an operator implementation of interest that is being optimized. For example, if one is only interested in the execution time of a join operator for two tables of certain input sizes, running a query with a join operator will inevitably also involve the execution of two table scans, which will consume an unknown portion of the execution time of the query. For some example embodiments of the cost-model used by the cost-based optimizer described herein, it is desirable to have a cost measurement only for the join operator itself, since the cost-based optimizer is only interested in the cost of the one or more individual operators being optimized.

Controlling the execution plan resulting from running a SQL query is difficult. For example, if one is interested in the performance of a certain implementation of a join operator, it may not be possible to control the database system not to use any other join implementation for the execution of the respective SQL query. In typical database systems, the decision of what specific join implementation to use is intentionally hidden from the user interface and not necessarily meant to be influenced by the user.

To get meaningful cost measurement numbers, a rather large set of input parameter combinations is needed. For example, a user might be interested in the runtime measurements of a scan operator for table sizes ranging from 1 to 1,000,000,000 rows in relatively small increments, say 10,000 rows. Collecting this information in the database system itself would involve the creation of a large number of tables with the desired table sizes and the insertion of data into these tables. For example, 100,000 tables would be needed to provide table sizes ranging from 1 to 1,000,000,000 rows in increments of 10,000 rows.

To overcome these problems, a benchmark framework allowing for measuring the execution times of select operators by a database execution engine for an arbitrary set of determinant parameters in an easy-to-use manner has been developed. The benchmark framework makes use of one or more benchmark libraries of operators, which provide all the tools necessary for automatic generation of input parameter values and reliable measurement of a cost metric. In some example embodiments, the input to the benchmark framework is a JSON-file encoding the query execution plan containing the operator of interest to be optimized. The JSON-file is deserialized into an execution plan object designed to be executed by the database execution engine. This provides the user with full control over what exactly is executed in the execution plan and for which operator or operators cost is/are measured.

The input data used for the optimization testing is, in some example embodiments, provided by providing the execution plan object with “dummy” or “placeholder” operators (e.g., dummy scan operators) that simply pass the input data to the operator of interest without any significant overhead. This allows for isolating the cost of executing the operator of interest from other superfluous operations that surround the operator of interest in a regular execution.

Using the benchmark framework described herein, it is possible to collect precise cost data on the performance of a certain select operator or operators subject to a wide range of influence factors (e.g., a number of distinct values in a number of rows or columns, or a selectivity of an operator predicate) which can then be input into the cost-model to be used by the cost-based optimizer. In some example embodiments, the cost data is determined using standard statistical analysis software of a number of cost measurements. Conducting these cost-based performance tests does not require a running instance of the database system. The entire process can be automated to develop and/or update the cost-model describing the performance of the select operator whenever the characteristics of an operator change during ongoing development.

In some example embodiments, the cost-based benchmark framework provides the following enhanced features:The system is easy to use and does not require a running instance of a database system. In some example embodiments, the system works similarly to a unit test, which only requires the compilation of the code that is executed (in this case code executed by the database execution engine), but not that executed by the whole database system.The process of running the benchmark generated tests can be automated within a build structure of the database execution engine and can done on a recurring basis to re-calibrate the cost-model during ongoing development.The benchmark framework provides full control over which operators are executed and on specified input data. The input data can be generated for each operator execution and the input characteristics (e.g., distribution, value range, etc.) are controlled by the user.One or more selected operators can be analyzed in isolation. The results are not blurred by the effects of other operators that are typically contained in a regular query execution plan, such as table scans at the leaves of an operator tree and/or a projection at the root of the operator tree.The results can be provided in a format that may be processed by standard statistics software to derive cost-formulas for the respective operators (e.g., using CSV or JSON).

In some example embodiments, there may be provided an execution engine that may decouple a higher-level, application layer from a lower database layer (e.g., the persistence or storage layer where data including database tables may be stored and/or queried using instructions, such as commands and/or the like). The execution engine may be implemented separately from the database layer and/or the application layer. Furthermore, the execution engine may be configured to receive a query, generate a query plan (including for example query algebra), optimize the query plan, and/or generate executable code, which can be executed at runtime. The executable code may include pre-compiled code (which can be selected for certain operations in the query plan) and/or code that is generated just-in-time specifically for execution of the query plan.

The execution engine may be configured to perform some operations itself, while the execution engine may send some operations (e.g., relatively basic commands, such as reads, writes, scans, and/or the like) to the database layer. Furthermore, the execution engine may receive corresponding responses from the database layer where data is stored/persisted and certain commands, such as reads, writes, scans, and/or the like, can be performed. The execution engine may perform more complex execution operations, such as rule-based operations including relatively more complex operations such as joins, projections, and/or the like, while accessing the database's storage/persistence layer when needed to read, write, update, and/or perform other operations.

The execution engine may be configured to support a wide range of database types to reduce, if not eliminate, the need for specialized execution engines for each type of database. For example, rather than having an execution engine for each type of database (e.g., an execution engine for an OLAP database, another execution engine for an OLTP database, an execution engine for a row-store database, an execution engine for a column-store database, and/or the like), the execution engine disclosed herein can perform query execution for a variety of database types and send queries to the different types of database layers (and/or their storage/persistence layer) and handle the corresponding responses.

FIG. 1shows a block diagram of an example embodiment of a system100, in accordance with some example implementations.

The system100may include one or more client user equipment102A-N, such as a computer, a smart phone, a tablet, an Internet of Things (IoT) device, and/or other computer or processor-based devices. The client user equipment102may include a user interface, such as a browser or other application to enable access to one or more applications, database layer(s), and/or databases, to generate queries to one or more databases190A-N, and/or to receive responses to those queries.

In the example ofFIG. 1, the databases190A-N represent the database layer of a database management system where data may be persisted and/or stored in a structured way, and where the data can be queried or operated on using operations including SQL commands or other types of commands/instructions to provide reads, writes, and/or perform other operations. To illustrate by way of an example, client user equipment102A-N may send a query plan describing an execution plan including one or more operators to a database execution engine150. The database execution engine150may send the query plan to the database layer190A-B, which may represent a persistence and/or storage layer where database tables may be stored and/or queried. The query plan may be sent via a connection, such as a wired and/or wireless connection (e.g., the Internet, cellular links, WiFi links, and/or the like).

The database execution engine150may include a cost-based optimizer110, such as a SQL optimizer and/or another type of optimizer, to receive at least one query from a user equipment102and generate a query plan (which may be optimized) for execution by the execution engine112. The cost based optimizer110may receive a request, such as a query request, and then form or compile the received query request into an initial execution plan. The cost-based optimizer110may then identify database parameters associated with an operator of interest in the received query request. The database parameters may include one or more of a number of rows, a number of columns, a number of distinct values for each column, and a selectivity of an operator predicate, for which cost-models for the operator of interest have been previously generated. Using the identified database parameters and one or more cost-models, the cost-based optimizer may identify an optimum version of the operator of interest to execute the query request. The execution engine150may then execute the received query request using the identified optimum version. Further details of the cost-based optimization of the operators of interest are described below.

For example, SELECT Columns from Table A and Table B, and perform an INNER JOIN on Tables A and B may represent a query received by the database execution engine150including the cost-based optimizer110. There may be several ways of implementing execution of this query. As such, the query plan may offer hints or propose an optimum query plan with respect to the execution time of the overall query. To optimize a query, the cost-based optimizer110may obtain one or more costs for the different ways of executing of the query plan. The costs may be obtained via the execution interface112A from a cost function114, which responds to the cost-based optimizer110with the cost(s) for a given query plan (or portion thereof), and these costs may be in terms of execution time at the database layer190A-N, for example.

The cost-based optimizer110may form an optimum query plan, which may represent a query algebra, as noted above. To compile a query plan, the cost-based optimizer110may provide the query plan to the query plan compiler116to enable compilation of some, if not all, of the query plan. The query plan compiler116may compile the optimized query algebra into operations, such as program code and/or any other type of command, operation, object, or instruction. This code may include pre-compiled code (which can be pre-compiled and stored, and then selected for certain operations in the query plan) and/or just-in-time code generated specifically for execution of the query plan. For example, plan compiler116may select pre-compiled code for a given operation as part of the optimization of the query plan, while for another operation in the query plan the plan compiler116may allow a compiler to generate the code. The pre-compiled and generated code represent code for executing the query plan, and this code may be provided to the plan generator118, which interfaces with the query execution engine115.

In some implementations, the cost-based optimizer110may optimize the query plan by compiling and generating code. Moreover, the cost-based optimizer110may optimize the query plan to enable pipelining during execution.

The query execution engine115may receive, from the plan generator118, compiled code to enable execution of the optimized query plan, although the query execution engine may also receive code or other commands directly from a higher-level application or other device, such as user equipment102A-N.

The query execution engine115may then forward, via an execution interface112B, the code to a plan execution engine120. The plan execution engine120may then prepare the plan for execution, and this query plan may include pre-compiled code125and/or generated code127. When the code for the query plan is ready for execution during runtime, the query execution engine115may step through the code performing some of the operations within the database execution engine150and sending some of the operations (or commands in support of an operation, such as a read, write, and/or the like) to an execution engine application programming interface (API)199for execution at one or more of databases layers190A-N.

In some example embodiments, the user equipment102A-102N is connected to or can operate directly a benchmark framework160. The benchmark framework160may be connected to a benchmark library165via a connection, such as a wired and/or wireless connection (e.g., the Internet, cellular links, WiFi links, and/or the like). In some embodiments, the benchmark library165may be local to the user equipment102A-102N. The benchmark framework160allows for measuring one or more cost metrics (e.g., the execution time, memory allocation, etc.) of select operators for an arbitrary set of determinant parameters.

The benchmark framework160makes use of one or more benchmark libraries165of operators, which provide all the tools necessary for automatic generation of input parameter values and reliable measurement of the cost metric. In some example embodiments, the input to the benchmark framework160is a JSON-file encoding the query execution plan containing the operator of interest to be analyzed to form the cost-model. The JSON-file is deserialized into an execution plan object designed to be executed by the database execution engine150. This provides a user of one of the client user equipment102with full control over what exactly is executed in the execution plan and for which operator or operators the cost metric is/are measured in developing the cost-model.

FIG. 2shows an example embodiment of a high-level flow diagram of a benchmark framework loop process200for determining a cost-model of one or more operators of interest contained in an execution plan.

The process200starts with the benchmark framework160receiving, as an input (e.g., from a user of client user equipment102A-102N), an execution plan file (e.g., a JSON file)202that includes an encoding of a query plan directed at one or more of the databases190A to190N. The execution plan file202contains information needed by the execution engine150to generate an execution plan object206, including the one or more operators of interest to be optimized by the cost-based optimizer110. The execution plan file202also contains expressions (e.g., predicates, field expressions, etc.) utilized by the query execution engine115to execute the query plan represented by the execution plan file202.

Upon receiving the execution plan file202, the benchmark framework may form an execution plan object206by processing (e.g., deserializing)205the execution plan file202. In some example embodiments, the processing205of the execution plan file202into the execution plan object206includes converting a JSON execution plan file202into an object in memory. The processing205may include, in some example embodiments, the construction of an object in memory that contains all the information that is also contained in the execution plan file202. In general, no information is added or modified during the processing205. However, various placeholder strings may be replaced by other strings during the processing205. The execution plan file202is a representation of an in-memory object which can be persistently stored on disk in contrast to a volatile in-memory representation. The benchmark framework160modifies the execution plan object206by isolating the operators of interest from other superfluous actions (e.g., other operators, scans, projections, etc.) that consume overhead separate from the operators of interest. Before the execution plan object206is executed, the execution plan object206is further processed by adding one or more data sources as leaf nodes of an operator tree structure representing the execution plan object206. The benchmark framework160revises the execution plan object206by representing table scans included in the execution plan file202with operators (referred to as DummyData) resulting in an extended execution plan object214. In some example embodiments, pointers to the input data are held by the DummyData operators. The previously generated input data resides in memory and can be directly streamed to the one or more operators of interest when under test in a benchmark framework loop process210.

In some example embodiments, the input data used to fill tables being operated on by the operator(s) of interest is generated on the fly by a data generator212of the benchmark framework150for each execution in the benchmark framework loop process210according to parameters specified by a user (e.g., of one of the client user equipment102A to102N). In this way, the input data does not have to be stored in a file on disk. Typical examples for the user specified parameters may include a number of rows, a number of columns, a number of distinct values for each column, a selectivity of an operator predicate, etc. Selectivity is a probability that any row will satisfy a predicate. For example, a selectivity of 0.01 (1%) for a predicate operating on a table with 1,000,000 rows means that the predicate will return an estimated 10,000 rows and discard an estimated 990,000 rows. The specified parameters later become independent variables of the cost-function114employed by the cost-based optimizer110. In contrast to an actual table scan, which typically makes use of some more complex data structures and comprises a separate materialization step, the streaming of input data using the pointers of the DummyData causes no significant overhead, thus not affecting the cost-function114.

The input data used to fill the tables utilized by the operators of interest may be generated by the data generator212using a random number generator, in some example embodiments. This provides a fast and simple way to fill tables of various sizes according to the input parameters specified by the user.

Before executing the extended execution plan object214, the benchmark framework160further revises the execution plan object206by replacing one or more projection operators with one or more “dummy” projection operators added at a root of the operator tree structure. When being executed, the dummy projection operators receive the data produced by the operator of interest under test and basically ignore the received data. In an actual execution of the execution plan object206, a projection operator buffers a query result and potentially applies a set of projection functions to it, which causes some overhead that are not to be included in the cost-metric measurements.

Having revised the execution plan object206, as described above, to form the extended execution plan object214, the extended execution plan object214is then passed to an execution engine216to be executed in the benchmark framework loop process210. In some example embodiments, the execution engine216is the query execution engine115.

The benchmark framework160runs a series of tests, represented by the benchmark framework loop process210, using the tables filled with the random input data generated by the data generator212using a random number generator as described above. While running the series of tests, the benchmark framework160measures at least one cost metric (e.g., execution time, memory allocation, etc.) representative of execution of a candidate version of at least one operator of interest.

The benchmark framework160outputs the at least one measured cost metric for individual tests to a cost result file220. In some example embodiments where execution time is the cost-metric, the time taken for the execution of individual tests is measured and communicated to and stored in the cost result file220. In some example embodiments, the benchmark framework loop process210starting with generating the input data with the data generator212to executing the extended execution plan object214at the execution engine210, measuring the cost metric and communicating and storing the measured cost metric in the cost result file220is conducted by the benchmark library165which provides a mechanism for running a series of benchmarks in benchmark framework loop210with varying input parameters. In this way, a plurality of cost-based experiments can be executed using the same extended execution plan object214repeatedly with a large number of different input sizes and data distributions while executing the benchmark framework loop process210just once.

Further processing of the outputted cost result file220can be provided using standard statistical software. An example cost result file220is described below. The cost-based optimizer110may identify, based on the at least one output cost metric, an optimum one of the plurality of candidate versions of the at least one operator of interest. The optimum candidate version of the operator of interest may be the candidate version that results in the lowest average measured execution time or the lowest memory allocation, for example.

FIG. 3depicts an example embodiment of an execution plan file300(e.g., the execution plan file202ofFIG. 2) that can be received from one of the user equipment102A-102N and formed into the extended execution plan object file214executed during tests using the benchmark framework loop process210ofFIG. 2. The execution plan file300encodes a query execution plan containing a hash-join operator310comprising lines 01-25.

The hash-join operator310, in this example, is the operator of interest for which the benchmark framework loop process210will collect information needed to derive a cost-model. Lines 3 to 9 define children operators315A and315B of the hash-join operator310. The children operators315A and315B are of the type DummyData as described above, which is a name for a data source operator. In the execution plan object file300, the DummyData operators act as placeholders for the input data that are later created by the data generator212and inserted into the execution plan object file300before being executed in the benchmark framework loop process210.

Lines 10-21 define a specification of a predicate of the hash-join' operator310, including a collect expression320(the build side of the hash-join operator310) and a lookup expression325(the probe side of the hash-join operator310). Strings330A,330B and330C with prefix “PH” (lines 14, 20 and 22) are placeholder strings that are later replaced by a different string during the processing205. The strings are replaced during processing205. They can be replaced by a string that has a certain meaning to the database execution engine150. For example, “PH_JOINTYPE” may be replaced by one of the following specific strings: “Inner”, “LeftOuter”, “RightOuter”, “FullOuter”, “LeftSemi”, “RightSemi”, “LeftAntiSemi” or “RightAntiSemi.” Similarly, there are a set of valid values for the data type of an operators result (“PH_RESULTTYPE” in line 14 ofFIG. 3). So, the string placeholder mechanism, allows just one generic file for all supported join types and users do not have to create and store 8 different plan files, which only differ in the join type to cover all supported join types. This allows for further parameterization of the plan. In the example execution plan object file300, the data type of the join result and the join type (e.g., inner, left outer, etc.) is not fixed in the execution plan object file300. In this way, only one relatively generic execution plan object file300needs to be written and stored for all benchmark experiments involving a hash-join.

Table 1 shows an example of a CSV-file version of the cost result file220produced by the benchmark for a plan containing an inner hash join on integer columns. Each HashJoinINT entry contains the input parameter values including: a number of distinct values “1Distinct” in each row of a first (left) table, a number of rows “lRows” in the first table, a number of distinct values “rDistinct” in a second (right) table, a number of rows “rRows” in the second table, and a selectivity “sel.” The input parameter values are used in the respective execution of each iteration resulting in output values for the cpu_time and real_time elapsed during the execution (e.g., in nanoseconds) for each iteration.

The file format of Table 1 is exemplary and can be provided by the benchmark library165, and can be used by the benchmark framework160when running the benchmark framework loop process210for a plurality of iterations. The iterations continue until the variance of the measurements of cpu_time reaches a certain lower bound, upon which the framework loop process210outputs the statistics (e.g., average and standard deviation) of all the iterations to the cost result file220. This may guarantee meaningful numbers being output, because the first iteration typically acts as a “warmup” run for the following iterations and usually is much slower.

FIG. 4Adepicts an example process400for developing a cost-model for analyzing performance of an operator of interest, in accordance with some example embodiments.

At410, an execution plan file may be received from a user. For example, the benchmark framework160may receive an execution plan file from user equipment102A. The execution plan file utilizes at least one operator of interest and further utilizes other actions separate from the at least one operator of interest. The benchmark framework160may form, based on the execution plan file, an execution plan object at420. For example, the benchmark framework160may modify the execution plan file by isolating the at least one operator of interest from the other actions.

At420, the benchmark framework160may form the execution plan object by processing (e.g., deserializing) the execution plan file as described above. The benchmark framework160may replace one or more of the other actions with one or more operators including, for example, at least one DummyData operator including a pointer identifying input data and one or more other dummy operators (e.g., projections). The benchmark framework160may replace a table scan with at least one pointer to the input data. The benchmark framework may replace a projection operator with a dummy projection operator that receives data produced by the operator of interest. As discussed above, during this process, the benchmark framework160may replace placeholder strings in the execution plan file with various values of actual strings.

At430, the benchmark framework160may perform a series of tests executing an extended execution plan object. The extended execution plan object may be formed by replacing the DummyData operators with the corresponding input data and adding one or more dummy projection operators to the in-memory object. The series of tests, in some example embodiments may include receiving the input data identified by the one or more pointers, executing the extended execution plan object using the received input data with the at least one operator of interest. The benchmark framework160may generate the input data using a random number generator as described above. The benchmark framework160may further measure, based on the execution of the extended execution plan object, at least one cost metric representative of execution of the at least one operator of interest. The benchmark framework160may output the measured cost metric at440to be received by the cost-based optimizer110. The cost metric may be one or more of execution time and memory allocation of the extended execution plan object.

At450, the benchmark framework160determines a cost-model for the tested operator of interest based on the at least one output measured cost metric. The cost-model may be generated using statistical analysis processes to generate curves to fit the variations in input parameters (e.g., a number of rows, a number of columns, a number of distinct values for each column, a selectivity of an operator predicate, etc.) for which the series of tests performed at430covered.

The process400may be repeated a number of times for different versions of the operator of interest in order to generate several cost-models related to the operator of interest.

FIG. 4Bdepicts an example of a process455for determining an optimum format to execute a query request, in accordance with some example embodiments. The process455may be executed by the cost-based optimizer110.

At460, the cost-based optimizer110receives a query request. The query request may be received from one of the user equipment120A-120N. The query request will include one or more operators of interest for which one or more cost-models were generated using the process400ofFIG. 4A.

At465, the cost-based optimizer110compiles the received query request to form an initial execution plan. At470, the cost-based optimizer110identifies, using the compiled initial execution plan, database parameters associated with the received query request. The database parameters may include one or more of a number of rows, a number of columns, a number of distinct values for each column, a selectivity of an operator predicate, for which the cost-models of the operator of interest were generated.

At475, the cost-based optimizer110identifies, based on the identified database parameters and using one or more of the cost-models associated with the one or more operators of interest associated with the received query, an optimum version of the one or more operators of interest to execute the received query. The identified optimum version may be the version that, according to the cost-model, resulted in the lowest estimated execution time based on the identified database parameters associated with the received query request.

At480, the received query request is executed, e.g., by the database execution engine, using the identified optimum version of the one or more operators of interest in the received query request.

FIG. 5depicts a block diagram illustrating a computing system500consistent with implementations of the current subject matter. Referring toFIGS. 1 and 5, the computing system500can be used to implement the benchmark framework160and the database execution engine150and/or any components therein.

As shown inFIG. 5, the computing system500can include a processor510, a memory520, a storage device530, and input/output device540. The processor510, the memory520, the storage device530, and the input/output device540can be interconnected via a system bus550. The processor510is capable of processing instructions for execution within the computing system500. Such executed instructions can implement one or more components of, for example, the data ingestion engine120. In some example embodiments, the processor510can be a single-threaded processor. Alternately, the processor510can be a multi-threaded processor. The processor510is capable of processing instructions stored in the memory520and/or on the storage device530to display graphical information for a user interface provided via the input/output device540.

The memory520is a computer readable medium such as volatile or non-volatile that stores information within the computing system500. The memory520can store data structures representing configuration object database management systems, for example. The storage device530is capable of providing persistent storage for the computing system500. The storage device530can be a floppy disk device, a hard disk device, an optical disk device, a solid-state device, a tape device, and/or any other suitable persistent storage means. The input/output device540provides input/output operations for the computing system500. In some example embodiments, the input/output device540includes a keyboard and/or pointing device. In various implementations, the input/output device540includes a display unit for displaying graphical user interfaces.

The illustrated methods are exemplary only. Although the methods are illustrated as having a specific operational flow, two or more operations may be combined into a single operation, a single operation may be performed in two or more separate operations, one or more of the illustrated operations may not be present in various implementations, and/or additional operations which are not illustrated may be part of the methods.