Techniques for dynamically benchmarking cloud data store systems

In various embodiments, a benchmarking engine automatically tests a data store to assess functionality and/or performance of the data store. The benchmarking engine generates data store operations based on dynamically adjustable configuration data. As the benchmarking engine generates the data store operations, the data store operations execute on the data store. In a complementary fashion, as the data store operations execute on the data store, the benchmarking engine generates statistics based on the results of the executed data store operations. Advantageously, because the benchmarking engine adjusts the number and/or type of data store operations that the benchmarking engine generates based on any changes to the configuration data, the workload that executes on the data store may be fine-tuned as the benchmarking engine executes.

BACKGROUND OF THE INVENTION

Field of the Invention

Embodiments of the present invention relate generally to computer science and, more specifically, to techniques for dynamically benchmarking cloud data store systems.

Description of the Related Art

Many software applications rely on external services known as “cloud data stores,” which are systems that execute on cloud computing platforms and are designed to store and manage collections of client data. Examples of cloud data stores include Netflix Dynomite, Apache Cassandra, and Amazon Elastic File System, to name a few. Oftentimes, the overall functionality and performance of applications that rely on cloud data stores correlate to the functionality and performance of the cloud data stores themselves. Thus, to evaluate the applications, the cloud data stores also need to be evaluated. However, executing a complex application or system of applications across a wide range of operating conditions to evaluate the functionality and performance of a cloud data store or to compare multiple cloud data stores for a variety of use cases is prohibitively time consuming.

To reduce the time required to evaluate the functionality and performance of a cloud data store, an engineer may implement a benchmarking engine instead of an application or system of applications to test the cloud data store. In operation, the benchmarking engine typically generates different workloads that can be executed on the cloud data store. Workload operations that execute on the cloud data store are referred to herein as “data store operations.” During testing, as various data store operations execute on the cloud data store, the benchmarking engine monitors the performance of the cloud data store. In general, the workloads are designed to emulate loads on the cloud data store for one or more use cases. For example, to emulate the load on a cloud data store while a video streaming service responds to clients during evening hours, the benchmarking engine could generate and execute a workload that is characterized by a high number of read operations per second. In another example, to emulate the load on a cloud data store during a denial of service attack, the benchmarking engine could generate and execute a workload that includes a dramatic spike in a number of read operations per second.

One limitation of a typical benchmarking engine is that the workloads cannot be adjusted while the benchmarking engine executes. Thus, the workloads cannot be dynamically updated based on the performance of the cloud data store during any given testing scenario. For example, an engineer could determine that a current workload exceeds the throughput of the cloud data store and, therefore, want to reduce the number of data store operations per second. However, with a conventional benchmarking engine, the engineer would not be able to adjust the number of data store operations per second without terminating the benchmarking engine, configuring the benchmarking engine to generate a new workload, and re-executing the benchmarking engine. Re-configuring and re-executing the benchmarking engine to modify the workload can dramatically increase the time required to evaluate the cloud data store.

Another limitation of a typical benchmarking engine is that the benchmarking engine usually executes for a predetermined amount of time and then terminates. Thus, evaluating the performance of a cloud data store with respect to an application that rarely terminates is quite difficult. For example, conventional benchmarking engine testing likely would not be able to detect or indicate the presence of a memory leak that incrementally reduces the amount of available application memory over long periods of time. Such memory leaks could be quite problematic because they could cause a long-running application to terminate unexpectedly. Among other things, the architecture of a conventional benchmarking engine typically constrains the length of time that the benchmarking engine can successfully execute without overflow errors, etc.

As the foregoing illustrates, what is needed in the art are more effective techniques for benchmarking cloud data stores.

SUMMARY OF THE INVENTION

One embodiment of the present invention sets forth a computer-implemented method for testing a data store. The method includes processing one or more workload generation operations to generate first data store operations based on first configuration data; executing at least one of the data store operations included in the first data store operations on a data store to obtain first statistics that are associated with a performance of the data store; while continuing to process the one or more workload generation operations, receiving second configuration data, modifying the one or more workload generation operations to generate second data store operations based on the second configuration data, executing at least one of the data store operations included in the second data store operations on the data store to obtain second statistics that are associated with the performance of the data store; and displaying or transmitting for further processing at least one of the first statistics and the second statistics.

One advantage of the disclosed techniques is that, unlike conventional testing techniques, the workload generation operations may be modified without terminating the workload generating operations. Consequently, a user may adjust the workload based on statistics that are generated as the data store executes the workload. Fine-turning the workload as the data store executes the workload can dramatically reduce the time required to evaluate the performance of the data store.

DETAILED DESCRIPTION

In the following description, numerous specific details are set forth to provide a more thorough understanding of the present invention. However, it will be apparent to one of skilled in the art that the present invention may be practiced without one or more of these specific details.

System Overview

FIG. 1is a conceptual illustration of a benchmarking system100configured to implement one or more aspects of the present invention. As shown, the benchmarking system100includes, without limitation, a cloud data store110, a benchmarking cluster120, and an analysis cluster190. In alternate embodiments, the benchmarking system100may include any number of cloud data stores110, any number of benchmarking clusters120, and any number of analysis clusters190. For explanatory purposes, multiple instances of like objects are denoted with reference numbers identifying the object and parenthetical numbers identifying the instance where needed.

The cloud data store110executes on a cloud computing platform and is designed to store and manage collections of client data. The cloud data store110is also commonly referred to as a “data store cluster.” Examples of cloud data stores110include Netflix Dynomite, Apache Cassandra, and Amazon Elastic File System, to name a few. As shown, the cloud data store110includes, without limitation, any number of data store nodes112. The data store nodes112within the cloud data store110are typically interconnected computers or virtual machines, where each computer or virtual machine supplies data store services via a client-server architecture. In alternate embodiments, each of the data store nodes130may be any instruction execution system, apparatus, or device capable of executing software applications. The data store nodes130may be organized in any technically feasible fashion and across any number of geographical locations.

Oftentimes, the overall functionality and performance of applications that rely on cloud data stores correlate to the functionality and performance of the cloud data stores themselves. Thus, to evaluate applications that rely on the cloud data stores, the cloud data stores also need to be evaluated. However, executing a complex application or system of applications across a wide range of operating conditions to evaluate the functionality and performance of a cloud data store or to compare multiple cloud data stores for a variety of use cases is prohibitively time consuming.

To reduce the time required to evaluate the functionality and performance of a cloud data store, an engineer may implement a conventional benchmarking engine instead of an application or system of applications to test the cloud data store. In operation, the conventional benchmarking engine typically generates different workloads that can be executed on the cloud data store. Workload operations that execute on the cloud data store are referred to herein as “data store operations.” During testing, as various data store operations execute on the cloud data store, the conventional benchmarking engine monitors the performance of the cloud data store. In general, the workloads are designed to emulate loads on the cloud data store for one or more use cases.

One limitation of a typical conventional benchmarking engine is that the workloads cannot be adjusted while the conventional benchmarking engine executes. Thus, the workloads cannot be dynamically updated based on the performance of the cloud data store during any given testing scenario. For example, an engineer could want to optimize a number of data store operations per second based on the observed throughput of the cloud data store. However, with a conventional benchmarking engine, the engineer would not be able to adjust the number of data store operations per second without terminating the conventional benchmarking engine, configuring the conventional benchmarking engine to generate a new workload, and re-executing the conventional benchmarking engine. Re-configuring and re-executing the conventional benchmarking engine to modify the workload can dramatically increase the time required to evaluate the cloud data store

Another limitation of a typical conventional benchmarking engine is that the conventional benchmarking engine usually executes for a predetermined amount of time and then terminates. Thus, evaluating the performance of a cloud data store with respect to an application that rarely terminates is quite difficult. For example, conventional benchmarking engine testing likely would not be able to detect or indicate the presence of a memory leak that incrementally reduces the amount of available application memory over long periods of time. Such memory leaks could be quite problematic because they could cause a long-running application to terminate unexpectedly.

Efficiently and Flexibly Testing a Cloud Data Store

To enable engineers to more efficiently and flexibly test the cloud data store110, the benchmarking system100includes the benchmarking cluster120. The benchmarking cluster120supports a variety of integration standards, enables dynamic re-configuration of workloads, and executes workloads for indeterminate amounts of time. As shown, the benchmarking cluster120includes, without limitation, any number of benchmarking nodes130. The benchmarking nodes130within the benchmarking cluster120are typically computers or virtual machines, where each computer or virtual machine independently executes an instance of a benchmarking subsystem140. The benchmarking nodes130may be organized in any technically feasible fashion and across any number of geographical locations.

As shown for the benchmarking node130(1), each of the benchmarking nodes130includes, without limitation, a processor132and a memory136. In alternate embodiments, each of the benchmarking nodes130may be configured with any number (including zero) of processors132and memories136, and the configuration of the benchmarking nodes130may vary. In other embodiments, each of the benchmarking nodes130may be any instruction execution system, apparatus, or device capable of executing software applications. In operation, the processor132(1) is the master processor of the benchmarking node130(1), controlling and coordinating operations of other components included in the benchmarking node130(1).

The memory136(1) stores content, such as software applications and data, for use by the processor132(1) of the benchmarking node130(1). As shown, the memory136(1) includes, without limitation, the instance of the benchmarking subsystem140that executes on the processor132(1). In a complementary fashion, for each of the other benchmarking nodes130(2)-130(N), the memory136(x) includes a different instance of the benchmarking subsystem140that executes on the processor132(x).

As shown, the benchmarking subsystem140includes, without limitation, a benchmarking interface150, a driver interface170that interfaces with a data store driver180, and a benchmarking engine160. The benchmarking interface150may be any type of interface that enables configuration of the benchmarking subsystem140via any number and type of configuration data. In some embodiments, the benchmarking interface150comprises a graphical user interface (GUI). In other embodiments, the benchmarking interface150comprises an application programming interface (API). For instance, in some embodiments, the benchmarking interface150may comprise a Representational State Transfer (REST) API. The REST API may support any number and type of data interchange formats, such as JavaScript Object Notation (JSON), HyperText Markup Language (HTTP), and Extensible Markup Language (XML), to name a few.

The configuration data includes, without limitation, a data store selection, a driver configuration, workload properties, a workload type, and benchmarking commands. The data store selection specifies the cloud data store110that is to be tested. The driver configuration selects the data store driver180through which the benchmarking subsystem140interfaces with the cloud data store110. As depicted with a dotted box, the data store driver180implements the driver interface170to enable the benchmarking subsystem140to interact with the cloud data store110. More specifically, in some embodiments, the data store driver180implements the driver interface170to enable the benchmarking subsystem140to:initialize the cloud data store110,shutdown the cloud data store110,perform a single read operation on the cloud data store110,perform a single write operation on the cloud data store110,get connection information from the cloud data store110, andrun a workflow for a functional test on the cloud data store110.

The workload properties, the workload type, and the benchmarking commands configure the benchmarking engine160. When configured to test the cloud data store110, the benchmarking engine160generates a workload based on the workload properties and the workload type and causes the workload to execute on the cloud data store110. For each of the benchmarking nodes130, the workload properties and the workload type may be specified separately via the benchmarking interface150. Accordingly, the various instances of the benchmarking engine160included in the different instances of the benchmarking subsystems140may be configured to generate different workloads. As a general matter, the workload properties may include any number and type of data, and the benchmarking engine160may generate the workload based on the workload properties in any technically feasible fashion.

For instance, in some embodiments, the workload properties include, without limitation:numKeys that specifies the sample space for randomly generated keys,numValues that specifies the sample space for generated values,dataSize that specifies the size of each value,numWriters that specifies the number of threads per benchmarking node130that execute write operations on the cloud data store110,numReaders that specifies the number of threads per benchmarking node130that execute read operations on the cloud data store110,writeEnabled that enables or disables write operations on the cloud data store110,readEnabled that enables or disables read operations on the cloud data store110,writeRateLimit that specifies the number of write operations per second on the cloud data store110readRateLimit that specifies the number of read operations per second on the cloud data store110userVariableDataSize that enables or disables the ability of the payload to be randomly generated

In general, the workload type specify a different pluggable workload traffic pattern. For instance, in some embodiments, the workload type specifies random traffic pattern, a sliding window traffic pattern, or sliding window flip traffic pattern. The sliding window traffic pattern specifies a workload that concurrently exercises data that is repetitive inside a window, thereby providing a combination of temporally local data and spatially local data. For example, the window could be designed to exercise both a caching layer provided by the cloud data store110and the Input/Output Operations Per Second (IOPS) of a disk managed by the cloud data store110.

The workload generation commands may configure any number of the instances of the benchmarking engine160to start, pause, and/or finish generating and executing the workload in any technically feasible fashion. For example, if a “run writes” command and a single benchmarking node130is selected, then the instance of the benchmarking engine160associated with the selected benchmarking node130executes write operations on the cloud data store110. By contrast, if “run writes” and “run reads” commands and multiple benchmarking nodes130are selected, then multiple instances of the benchmarking engines160independently and substantially in parallel execute write and read operations on the cloud data store110.

As the cloud data store110executes data store operations, the cloud data store110transmits results to the benchmarking engine160. For example, the result of a read operation could be client data and the result of a write operation could be an acknowledgment. The benchmarking engine160generates statistics (not shown inFIG. 1) based on the results and transmits the statistics to the benchmarking interface150for display purposes. The statistics may include any number and type of data that provide insight into the performance of the cloud data store110, and the benchmarking engine160may generate the statistics in any technically feasible fashion.

The benchmarking engine160also transmits the statistics to the analysis cluster190. In general, the benchmarking subsystem140provides plugin functionality that enables the benchmarking engine160to interface with any number of compatible analysis clusters190. The analysis cluster190may be any number and type of software applications (e.g., external time series database, monitoring system, etc.) that provides analysis and/or monitoring services. For example, the analysis cluster190could include, without limitation, a Netflix Servo interface that exposes publishing metrics in Java, and a Netflix Altas backend that manages dimensional time series data. After the analysis cluster190receives the statistics, the analysis cluster190generates any number of metrics based on the statistics.

In some alternate embodiments, the benchmarking engine160may transmit the statistics to the analysis cluster190, but may not transmit the statistics to the benchmarking interface150. In other alternate embodiments, the benchmarking engine160may transmit the statistics to the benchmarking interface150, but may not transmit the statistics to the analysis cluster190. In such embodiments, the benchmarking system100may not include the analysis cluster190. In various embodiments, the cloud data store110may transmit data store statistics to the analysis cluster190instead of or in addition to the statistics that the benchmarking engine160transmits to the analysis cluster190.

Notably, while the benchmarking engine160generates and executes the workload based on “current” workload properties, the benchmarking engine160may be dynamically reconfigured via the benchmarking interface150. More specifically, the benchmarking engine160may receive “new” workload properties via the benchmarking interface150. In response, the benchmarking engine160generates and executes the workload based on the new workload properties instead of the current workload properties without ceasing to generate and execute the workload. Accordingly, the new workload properties become the current workload properties.

In some embodiments, all of the workload properties may be dynamically configured. In other embodiments, one or more of the workload properties may be dynamically configured, while the remaining workload properties are statically configured prior to connecting to the cloud data store110and/or generating and executing the workload. For example, in some embodiments, the writeRateLimit and the readRateLimit may be dynamically configured, while the remaining workload properties are statically configured prior to configuring the driver connection. In alternate embodiments, the workload type may be dynamically configured.

In operation, the benchmarking engine160continues to generate and execute the workload based on the current workload properties until the benchmarking engine160receives a “pause” or an “end” workload generation command via that benchmarking interface150. Unless an end workload command is received, the benchmarking node130continues to generate and execute the workload for an infinite amount of time, thereby efficiently emulating the operating conditions of long-running applications.

In alternate embodiments, the benchmarking interface150may be configured to generate an “end” workload generation command in any technically feasible fashion. For example, the benchmarking interface150could implement a servlet context listener that detects when an application that is associated with the benchmarking interface150is terminated. When the servlet context listener detects that the application is terminated, then the benchmarking interface150could generate an end workload generation command. In some embodiments, the workload generation commands may include any number and type of additional commands that customize and/or optimize the benchmarking of the cloud data store110.

For instance, in some embodiments, the workload generation commands include a “backfill” command. If the benchmarking subsystem140receives the backfill command, then the benchmarking subsystem140executes one or more write commands on the cloud data store110prior to executing the workload. The one or more write commands store initial data in the cloud data store110. The stored initial data, also commonly referred to as “hot” data, reduces the time required to test the cloud data store110.

Note that the techniques described herein are illustrative rather than restrictive, and may be altered without departing from the broader spirit and scope of the invention. In particular, the functionality provided by the benchmarking subsystem140, the benchmarking engine160, the benchmarking interface150, the driver interface170, the data store driver180, the analysis cluster190, and the cloud data store110may be implemented in any number of software applications in any combination. Further, in various embodiments, any number of the techniques disclosed herein may be implemented while other techniques may be omitted in any technically feasible fashion.

Many modifications and variations on the functionality provided by the benchmarking subsystem140, the benchmarking engine160, the benchmarking interface150, the driver interface170, the data store driver180, the analysis cluster190, and the cloud data store110will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. Alternate embodiments include any benchmarking application that enables dynamic reconfiguration of a workload executing on a database and/or generates and executes a workload on the database until explicitly terminated.

FIG. 2is a more detailed illustration of the operations performed by the benchmarking subsystem140ofFIG. 1when testing the cloud data store110, according to various embodiments of the present invention. For explanatory purposes only,FIG. 2depicts a sequence of events involved in testing the cloud data store110with circles that are labeled1through13. In alternate embodiments, the number and sequence of events involved in testing the cloud data store110may vary.

First, as depicted with the circle labeled1, the benchmarking subsystem140receives a data store selection210via the benchmarking interface150. As shown, the data store selection210specifies that the cloud data store110is to be the target of testing operations. The benchmarking subsystem140may enable the specification of the data store selection210in any technically feasible fashion. For example, in various embodiments, the benchmarking subsystem140could identify a number of available cloud data stores110via a discovery process and configure the benchmarking interface150to display the available cloud data stores110in a selection window.

Subsequently, as depicted with the circle labeled2, the benchmarking subsystem140receives a driver configuration220that configures the benchmarking subsystem140to interface with the cloud data store110via the data store driver180(1). In operation, upon receiving the driver configuration220that specifies the data store driver180(1), the benchmarking subsystem140connects to the cloud data store110via the data store driver180(1).

As shown, the data store driver180(1) is included in a driver list215that includes, without limitation, any number of the data store drivers180and a dynamic plugin282. As a general matter, each of the data store drivers180and the dynamic plugin282implement the driver interface170. The data store drivers180are typically written in a programming language and, consequently, are configured statically. Examples of the data store drivers180include DataStax Java Driver (Cassandra Query Language), Cassandra Astyanax (Thrift), ElasticSearch API, and Dyno (Jedis support). By contrast, the dynamic plugin282is dynamically configured via a script that is written in a scripting language, such as Groovy. In alternate embodiments, the driver list215may include any number and type of software applications that implement the driver interface170.

Subsequently, as depicted with the circle numbered3, the benchmarking engine160receives workload properties230via the benchmarking interface150. The benchmarking engine160may receive any number and type of workload properties230in any technically feasible fashion. Similarly, as depicted with the circle numbered5, the benchmarking engine160receives a workload type235. As depicted with the circle numbered6, the benchmarking engine160then receives a start workload command240that configures the benchmarking engine160to generate and execute the workload based on the workload properties230and the workload type235.

As part of generating and executing the workload, the benchmarking engine160configures a thread pool260that includes any number of threads262that execute on the cloud data store110. More specifically, the benchmarking engine160performs operations that configure the thread pool260based on the workload properties230. The benchmarking engine160may configure the thread pool260based on any number and type of the workload properties230. For instance, in some embodiments, the benchmarking engine160configures the thread pool260to include a number of the threads262that execute read operations on the cloud data store110based on the benchmarking parameter230“numReaders.” The benchmarking engine160may configure the thread pool260and manage the threads262in any technically feasible fashion as known in the art.

After the benchmarking engine160configures the thread pool260, the benchmarking engine generates data store operations250based on the workload properties230and the workload type235. The benchmarking engine160may generate the data store operations250based on any number and type of the workload properties230and the workload type235. For example, the benchmarking engine160could generate read operations at a rate that is specified by the benchmarking parameter230“readRateLimit” and based on a sliding window traffic pattern that is specified by the workload type235.

As depicted with the circle labeled7, as the benchmarking engine160generates each of the data store operations250, the benchmarking engine160assigns the data store operation250to one of the threads262included in the thread pool260. The thread262then executes the data store operation250on the cloud data store110via the data store driver180(1). In alternate embodiments, the benchmarking engine160may cause the data store operations250to execute on the cloud data store110in any technically feasible fashion. The processes of generating the data store operations250and causing the data store operations250to execute on the cloud data store110is also referred to herein as “generating and executing the workload.”

As the cloud data store110executes the data store operations250, the cloud data store110transmits the results to the benchmarking engine160. The benchmarking engine160receives the results of the data store operations250and generates statistics280. The statistics280may include any amount and type of data that measures the functionality and/or performance of the cloud data store110. As depicted with the circles labeled8, the benchmarking engine160transmits the statistics280to the analysis cluster190and the benchmarking interface150. The benchmarking interface150then displays the statistics280.

As a general matter, the benchmarking engine160is configured to execute multiple operations during the benchmarking process substantially in parallel. For example, the benchmarking engine160typically generates the statistics280associated with the data store operations250that have finished executing on the cloud data store110data store while generating new data store operations250.

Dynamically Re-Configuring a Workload

In particular, as depicted with the circle labeled9, as the benchmarking engine160generates and executes the workload based on current workload properties230and the workload type235, the benchmarking engine160receives new workload properties230. As depicted with the circle labeled10, the benchmarking engine160modifies the workload based on the new workload properties230. More specifically, if the new workload properties230are associated with the thread pool260, then the benchmarking engine160re-configures the thread pool260. For example, the benchmarking engine160could increase or decrease the number of threads262that are included in the thread pool260. Further, the benchmarking engine160generates subsequent data store operations250based on the new workload properties230instead of the current workload properties230. Accordingly, the new workload properties230become the current workload properties230.

As depicted with the circle labeled10, the benchmarking engine160continues to assign the data store operations250to the threads262that execute the data store operations250on the cloud data store110. In a complementary fashion, as depicted with the circle labeled12, the benchmarking engine160continues to generate and transmit the statistics280to the analysis cluster190and the benchmarking interface150. Such a process causes the benchmarking interface150to dynamically display the statistics280. The benchmarking subsystem140continues to perform testing operations in this fashion until the benchmarking subsystem140receives an end workload command290(depicted with the circle labeled13) via the benchmarking interface150.

FIG. 3illustrates an example configuration of the benchmarking interface150ofFIG. 2, according to various embodiments of the present invention. As shown, the benchmarking interface150includes, without limitation, the data store selection pane310, the driver configuration pane320, the workload properties subpane332, the workload generation pane340, and the statistics display pane380. In alternate embodiments, the benchmarking interface150may include any number and type of interface widgets (e.g., panes, sliders, buttons, menus, etc.) that enable an engineer to configure and execute the benchmarking subsystem140to test the cloud data store110. In yet other alternate embodiments, the benchmarking interface150may be replaced with an API.

As shown, the data store selection pane310identifies the data store selection210of the cloud data store110“localhost.” The driver configuration pane320identifies that the benchmarking subsystem140is connected to the cloud data store110via the data store driver180(1) “InMemoryTest.” The workload properties subpane330displays the values for the workload properties230. As shown, the workload properties230include “initial settings” and “runtime settings.” The initial settings can be modified before the benchmarking engine160connects to the cloud data store110via the data store driver180(1). By contrast, the runtime settings can be modified at any time.

The workload generation pane340identifies that the benchmarking engine160included in the benchmarking node130“localhost:8080” is generating and executing a workload on the data store110. The workload generation pane340further identifies that the workload is of the workload type235“random.” In a complementary fashion, the statistics display pane380displays a selection of the statistics280that the benchmark engine160included in the benchmarking node130generates based on the results received from the data store210.

FIG. 4is a flow diagram of method steps for testing a cloud data store, according to various embodiments of the present invention. Although the method steps are described with reference to the systems ofFIGS. 1-3, persons skilled in the art will understand that any system configured to implement the method steps, in any order, falls within the scope of the present invention. For explanatory purposes only, the context ofFIG. 4is that a single instance of the benchmarking subsystem140included in a single benchmarking node130is executing the method steps. As a general matter, any number of instances of the benchmarking subsystem140included in any number of the benchmarking nodes130may execute any number of the method steps independently and substantially in parallel

As shown, a method400begins at step404, where the benchmarking subsystem140receives the data store selection210via the benchmarking interface150. The data store selection210specifies the cloud data store110. At step406, the benchmarking subsystem140receives the driver configuration220via the benchmarking interface. The driver configuration220requests that the benchmarking engine160interface with the cloud data store110via the data store driver180(1). At step408, the benchmarking engine160connects to the cloud data store110via the data store driver180(1).

At step410, the benchmarking engine160receives the workload properties230via the benchmarking interface150, and the benchmarking engine160sets “current” workload properties230equal to the workload properties230. At step412, the benchmarking engine160receives the workload type235via the benchmarking interface150. At step413, the benchmarking engine160receives the start workload command240via the benchmarking interface150.

At step414, the benchmarking engine160configures the thread pool260based on the current workload properties230. At step416, the benchmarking engine160generates the data store operations250based on the current workload properties230and the workload type235. As the benchmarking engine160generates each of the data store operations250, the benchmarking engine160assigns the data store operation250to one of the threads262included in the thread pool260for execution on the cloud data store110via the data store driver180(1).

At step416, the benchmarking engine160receives results of the executed data store operations250from the cloud data store110, generates the statistics280based on the results, and transmit the statistics280to the analysis cluster190and the benchmarking interface150. The benchmarking interface150displays the statistics280via the statistics display pane380. At step422, the benchmarking engine160determines whether the benchmarking engine160has received new workload properties230via the benchmarking interface150.

If, at step422, the benchmarking engine160determines that the benchmarking engine160has received new workload properties230, then the method400proceeds to step424. At step424, the benchmarking engine sets the current workload properties230equal to the new workload properties230, and the method400returns to step414where the benchmarking engine160adjusts the workload based on the current workload properties230. The benchmarking engine160continues to cycle through steps414-424, dynamically adjusting the workload based on new workload properties230until the benchmarking engine160does not receive any new workload properties230.

If, however, at step422, the benchmarking engine160determines that the benchmarking engine160has not received any new workload properties230, then the method400proceeds directly to step426. At step426, the benchmarking engine160determines whether the benchmarking engine160has received the end workload command290. If, at step426, the benchmarking engine160determines that the benchmarking engine160has not received the end workload command290, then the method400returns to step416, where the benchmarking engine160continues to generate the data store operations250. The benchmarking engine160continues to cycle through steps416-426, generating and executing the workload until the benchmarking engine160receives the end workload command290. If, however, at step426, the benchmarking engine160determines that the benchmarking engine160has received the end workload command, then the method400terminates.

In sum, the disclosed techniques may be used to test a cloud data store. A benchmarking subsystem includes, without limitation, a benchmarking interface, a driver interface, and a benchmarking engine. The benchmarking interface enables selection of a cloud data store and a data store driver, dynamic specification of workload properties, and generation and execution of a workload on the cloud data store. The driver interface enables the benchmarking engine to interface with the cloud data store via a compatible data store driver.

To test the cloud data store, the benchmarking engine generates data store operations based on the workload properties and a workload type. The benchmarking engine attaches the data store operations to threads that execute on the cloud data store via the data store driver. As the cloud data store executes the data store operations, the benchmarking engine generates statistics based on the results of the executed data store operations. The benchmarking engine transmits the statistics to an analysis cluster that performs any number of analysis operations. The benchmarking engine also transmits the metrics to the benchmarking interface for display purposes.

Notably, the workload properties may be dynamically updated via the benchmarking interface while the benchmarking engine generates and executes data store operations without terminating the benchmarking engine. Upon receiving new workload properties, the benchmarking engine generates data store operations based on the new workload properties instead of the previously specified workload properties. As a general matter, the benchmarking engine continues to generate data store operations based on the current workload properties until the benchmarking engine receives an end command via the benchmarking interface.

Advantageously, the benchmarking subsystem may be configured to automatically test data stores for use cases that are not efficiently supported by conventional benchmarking engines. Unlike conventional benchmarking engines, because the workload properties may be updated without terminating the benchmarking engine, a user may adjust the workload based on the statistics generated by the benchmarking engine. Fine-turning the workload as the benchmarking engine executes can dramatically reduce the time required to evaluate the performance of the cloud data store. Further, because the benchmarking engine executes until receiving an end command, the benchmarking engine may be configured to provide statistics for long-running use cases that are not supported by conventional benchmarking engines. Finally, because the benchmarking subsystem implements a variety of flexible interfaces, the benchmarking engine may be integrated with a wide range of cloud data stores, data store drivers, analysis clusters, cloud services, and the like.

1. In some embodiments, a method comprises processing one or more workload generation operations to generate a first plurality of data store operations based on first configuration data; executing at least one of the data store operations included in the first plurality of data store operations on a data store to obtain first statistics that are associated with a performance of the data store; while continuing to process the one or more workload generation operations, receiving second configuration data, modifying the one or more workload generation operations to generate a second plurality of data store operations based on the second configuration data, executing at least one of the data store operations included in the second plurality of data store operations on the data store to obtain second statistics that are associated with the performance of the data store; and displaying or transmitting for further processing at least one of the first statistics and the second statistics.

2. The method of clause 1, further comprising receiving an end command and, in response, ceasing to process the workload generation operations.

3. The method of clauses 1 or 2, further comprising, prior to processing the one or more workload generation operations, receiving a first command that specifies the data store; receiving a second command that specifies a driver; and establishing a connection to the data store through the driver.

4. The method of any of clauses 1-3, wherein the driver comprises a driver application that is written in a programming language or a dynamic plugin that is associated with a script.

5. The method of any of clauses 1-4, wherein executing at least one of the data store operations included in the first plurality of data store operations comprises assigning the at least one of the data store operations to at least one thread included in a thread pool to generate at least one configured thread; and causing the data store to execute the at least one configured thread.

6. The method of any of clauses 1-5, further comprising, prior to modifying the one or more workload generation operations, modifying a number of threads included in the thread pool based on the second configuration data.

7. The method of any of clauses 1-6, further comprising, while continuing to process the one or more workload generation operations, receiving at least one subsequent configuration data; and for each subsequent configuration data included in the at least one subsequent configuration data, modifying the one or more workload generation operations to generate a subsequent plurality of data store operations based on the subsequent configuration data, executing at least one of the data store operations included in the subsequent plurality of data store operations on the data store to obtain subsequent statistics that are associated with the performance of the data store, and displaying or transmitting for further processing the subsequent statistics.

8. The method of any of clauses 1-7, wherein the first configuration data includes at least one of a rate of read operations, a rate of write operations, a number of threads, a size of data, and a traffic pattern.

9. The method of any of clauses 1-8, wherein the second configuration data includes at least one of an updated rate of read operations, an updated rate of write operations, and an updated number of threads.

10. In some embodiments, a computer-implemented computer-readable storage medium includes instructions that, when executed by a processor, cause the processor to perform the steps of establishing a connection to a data store through a driver; generating a first workload based on first configuration data; causing the first workload to execute on the data store to obtain first statistics that are associated with a performance of the data store and the first configuration data; while remaining connected to the data store, generating a second workload based on second configuration data, causing the second workload to execute on the data store to obtain second statistics that are associated with the performance of the data store and the second configuration data; and displaying or transmitting for further processing at least one of the first statistics and the second statistics.

11. The computer-implemented method of claim10, wherein generating the first workload comprises processing one or more workload generation operations to generate a first plurality of data store operations based on the first configuration data; and generating the second workload comprises modifying the one or more workload generation operations to generate a second plurality of data store operations based on the second configuration data.

12. The computer-readable storage medium of clauses 10 or 11, further comprising receiving an end command and, in response, ceasing to process the one or more workload generation operations.

13. The computer-readable storage medium of any of clauses 10-12, wherein the first workload comprises a plurality of data store operations, and causing the first workload to execute on the data store comprises assigning at least one of the data store operations included in the plurality of data store operations to at least one thread included in a thread pool to generate at least one configured thread; and causing the data store to execute the at least one configured thread.

14. The computer-readable storage medium of any of clauses 10-13, further comprising, prior to generating the second workload, modifying a number of threads included in the thread pool based on the second configuration data.

15. The computer-readable storage medium of any of clauses 10-14, wherein the first configuration data includes at least one of a rate of read operations, a rate of write operations, a number of threads, a size of data, and a traffic pattern.

16. The computer-readable storage medium of any of clauses 10-15, wherein the traffic pattern comprises a sliding window of data that is characterized by at least one of temporally proximate data and spatially proximate data.

17. The computer-readable storage medium of any of clauses 10-16, wherein transmitting at least one of the first statistics and the second statistics comprises transmitting at least one of the first statistics and the second statistics to an analysis application for further processing.

18. In some embodiments, a system comprises a memory storing instructions associated with a benchmarking engine; and a processor that is coupled to the memory and, when executing the instructions, is configured to process one or more workload generation operations to generate a first plurality of data store operations based on first configuration data; assign at least one of the data store operations included in the first plurality of data store operations to at least a first thread included in a thread pool to generate at least a first configured thread; cause the data store to execute the at least a first configured thread to obtain first statistics that are associated with a performance of the data store; while continuing to process the one or more workload generation operations, receive second configuration data, modify at least one of the workload generation operations and the thread pool based on the second configuration data, assign at least one of the data store operations included in the second plurality of data store operations to at least a second thread included in the thread pool to generate at least a second configured thread; and cause the data store to execute the at least a second configured thread to obtain second statistics that are associated with the performance of the data store; and display or transmit for further processing at least one of the first statistics and the second statistics.

19. The system of clause 18, wherein the processor is further configured to receive an end command and, in response, cease to process the workload generation operations.

20. The system of clause 18 or 19, wherein the first configuration data includes at least one of a rate of read operations, a rate of write operations, a number of threads, a size of data, and a traffic pattern.

21. The system of any of clauses 18-20, wherein the processor is further configured to, prior to processing the one or more workload generation operations, generate one or more write operations that store initial data in the data store.