Metrics-based analysis for testing a service

Techniques are described for determining test cases to test a service, such as a service to manage a purchase contract in an e-commerce environment. Log data may be generated during execution of the service to process requests, the log data indicating a plurality of code segments that executed during the execution of the service. The log data may be analyzed to generate metrics data comprising a plurality of records corresponding to the processed requests. A record may include any number of counters that each indicates whether, or how many times, a particular code segment executed during processing of a request. The metrics data may be analyzed to identify a subset of records comprising representative combinations of the counters in the metrics data. The requests corresponding to one or more of the subset of records may be replayed to test the service.

BACKGROUND

A business or other organization may deploy software to provide various services related to online shopping, electronic commerce, digital media delivery, gaming, communications, web search, social networking, and so forth. Such services may be provided through one or more software systems executing on the computing devices, as in a distributed software deployment on the computing devices. In some cases, distributed software may include front-end software that provides an interface with end-users, and back-end software that processes data received from the front-end software and generates data to be presented through the front-end software. An organization may employ personnel and computing resources in testing the software. Such testing may seek to identify broken functionality or sub-optimal performance of the software. Testing may also seek to identify aspects of the software that may lead to a potentially negative user experience, higher operational costs, inefficiency in operations, or other problems. An organization may at least partly automate its software testing processes to enable such processes to be performed more efficiently and with less operator involvement.

DETAILED DESCRIPTION

This disclosure describes systems, devices, methods, and computer-readable media to generate test cases for testing a service based on an analysis of metrics data collected during execution of the service. Log data may be generated during execution of a production version of a service to process request(s) to the service. In some cases, the log data may be emitted by instructions present in the executable code of the service. Such instructions may be generated during compilation of source code lines inserted into the source code of the service by developers. Instructions to output log data may also be added to the executable code of the service by a compiler at compile time. In some cases, instructions to output log data may be injected into the executable code during the execution of the service. In some cases, log data may be generated by a mediator service that mediates calls between the service and other (e.g., remote) network services. The log data may be stored on storage device(s) and analyzed to generate metrics data. In some embodiments, the metrics data may include a plurality of records that each corresponds to a request processed by the service. A record may include a plurality of counters each corresponding to a code segment of the service. The value of a counter may indicate whether, or how many times, a code segment was executed during execution of the service. Accordingly, a record of the metrics data may provide an indication of those code segments of the service that were executed during processing of a particular request. The metrics data may be analyzed to identify a subset of one or more records that includes representative combination(s) of the counters among the records in the metrics data. The request corresponding to one or more of the identified record(s) in the subset may then be replayed to test a test version (e.g., a subsequent build) of the service in a test environment.

Through the identification of the various representative combinations of counters, and the generation of test cases based on the representative combinations of counters, embodiments provide a set of test cases that may cover a substantially complete set of the code paths that were traversed during the execution of the service. Traditional methods of identifying test cases may include a random selection of previously processed requests, and may therefore fail to generate test cases corresponding to infrequently traversed code paths (e.g., corner cases). Accordingly, embodiments may provide a set of test cases that test the service more completely than traditional methods and identify bugs that may otherwise go unnoticed. Embodiments also provide for the automatic identification of new test cases following the addition of new code or new features to the service. For example, a build of the service may include a new software module or portion of a software module, such as a class, method, code branch, subroutine, method call, and so forth. While the service is processing request(s), log data may be emitted that includes indications that the newly added code is being executed. Subsequent analysis of the log data may generate test cases that test the newly added code. Accordingly, embodiments provide for a flexible testing system that is able to adapt to new code that has been added into the service under test.

FIG. 1illustrates an example environment in which one or more embodiments may operate. As shown inFIG. 1, the environment may include one or more contract processing devices102. The contract processing device(s)102may execute one or more versions of a contract service104. One or more client processes106may communicate one or more requests108to one or more versions of the contract service104executing on the contract processing device(s)102. In some embodiments, the contract processing device(s)102may include one or more backend server devices that operate within a system for providing an e-commerce service to end-users. In such cases, the contract service104may include any number of software modules that execute to receive, generate, send, or otherwise process data related to a contract within the e-commerce service. For example, the contract service104may process a purchase contract or other type of contract arising from an end-user's selection of one or more items to be purchased via the e-commerce service. In some embodiments, the contract service104may be a backend service that is not directly accessible by end-users (e.g., customers). Alternatively, the contract service may provide a user interface (UI) or other features configured to interact directly with an end-user.

The client process(es)106may include front end processes that provide an interface, such as a web interface or other UI accessible by end-users. The action(s) of an end-user in the UI provided by the client process(es)106may generate the request(s)108that are processed by the contract service104. For example, and end-user may select one or more items for purchase through an online store, e-commerce web site, e-commerce app, or other UI provided by the client process(es)106. The client process(es)106may then generate the request(s)108to indicate that a purchase contract is to be created to manage the end-user's purchase of the item(s) through the UI. The contract service104may generate a contract object that includes data for the contract. In some cases, at least some of the data included in the contract object may be communicated to the client process(es)106and presented to the end-user whose action(s) caused the request(s)108to be sent. In some embodiments, the contract service104may be a data aggregation service that calls multiple network services (e.g., remote services) to generate or process data to be included in the contract object. Such a data aggregation service is described further with reference toFIGS. 2 and 3.

In some embodiments, the client process(es)106may execute on server computing device(s) to generate a UI that is communicated to, and presented on, and end-user device such as a tablet computer, smartphone, laptop computer, desktop computer, home entertainment system, and so forth. For example, the client process(es)106may include one or more web servers or application servers that generate and serve content for the UI. In some embodiments, the client process(es)106may execute on the end-user device(s), such as device(s) directly operated by customer(s) of an e-commerce service. For example, the client process(es)106may include one or more software components of an application, or app, executing on an end-user device. Such an application may be a native application that is configured to execute within the particular hardware and software configuration of the end-user device. The application may also be configured to execute on a variety of hardware or software configurations. In some cases, the client process(es)106may be configured to execute partly on an end-user device and partly on one or more server device(s) in communication with the end-user device.

In some embodiments, the contract processing device(s)102may be configured to execute one or both of a contract service104A or a contract service104B that are, respectively, a production version and a test version of the contract service104. In some cases, the contract service104A and the contract service104B may be different versions, such as different revisions or different builds of the contract service104. The contract service104A and the contract service104B may at least partly differ with regard to the executable code, features, or other aspects of the different versions. A production version of the contract service104A may execute to process request(s)108generated based on actual end-user(s) (e.g., customer(s)) accessing the client process(es)106. A test version of the contract service104B may process request(s) that are generated through a testing process, such as test request(s) sent by a test script, test application, or other test module, or generated through manual testing. The test version of the contract service104B may be a subsequent build or later version of the contract service104relative to the production version of the contract service104A. In some cases, the contract service104B may execute in a controlled test environment, on separate device(s) than those that execute the contract service104A.

In some embodiments, the contract service104A may generate log data110during execution of the contract service104A. The log data110may be arranged in any format, and may indicate one or more code segments that executed during the operations of the contract service104A to process the request(s)108. For example, log data110may indicate that a particular line or block of code (e.g., source code or executable code) was accessed. The log data110may indicate that a particular function, method, or subroutine was called, entered, or exited during execution of the contract service104A. In some cases, the log data110may indicate that a particular code branch was accessed during execution of the contract service104A. For example, the log data110may indicate that an IF branch or an ELSE branch was accessed during execution of a section of code that includes an IF/ELSE block of code. As another example, the log data110may indicate that a TRY branch or a CATCH branch was accessed during execution of a section of code that includes a TRY/CATCH block of code. Log data110may also indicate that other type of conditional code segments were accessed.

Embodiments support the use of log data110generated from a variety of sources. In some cases, the log data110may be generated by instructions included in the executable code of the contract service104A. Such instructions may be based on source code line(s) inserted into the source code of the contract service104A by a developer or other individual(s). For example, a developer may write a line of code println (“I am in classX.methodY”) into the source code of the contract service104A, e.g., within the implementation of a particular method in a particular class. The line of code may compile to generate one or more executable instructions within the executable code of the contract service104A. During execution of the contract service104A, the executable instruction(s) may output (e.g., to standard output, standard error, a file, a console, and so forth) “I am in classX.methodY” when the code for a particular method is executed. Such output may be captured as a portion of the log data110in log files written when the contract service104A executes. In some embodiments, the instruction(s) to write log data110may be inserted into the executable code of the contract service104A, by a compiler or other build process, during the compilation of the source code for the contract service104A. In such cases, the instruction(s) may be injected or otherwise added by a compiler even when there is no source code corresponding to the instruction(s). A build process may specify a compiler option such that the instruction(s) are added to the executable code that is generated during the build process.

In some embodiments, the instruction(s) may be injected into the executable code of the contract service104A while the contract service104A is executing. For example, in cases where the contract service104A executes within a virtual machine, runtime engine, or other execution container, the execution container may emit log data110to indicate accessed code segment(s) of the contract service104A. In some embodiments, instruction(s) to output log data110may be automatically added into branches of conditional statements, calls to methods, class constructors, or elsewhere in the generated code, at compile time or at run time.

In some embodiments, the log data110may be generated by external processes that have access to information regarding the execution of the contract service104A. For example, calls from the contract service104A to other service(s) may be monitored, mediated, or otherwise managed through a mediator service, and the mediator service may generate log data110describing the various calls made by the contract service104A. Such embodiments are described further with reference toFIG. 3.

The log data110may include any amount or type of information to indicate accessed code segment(s) of the contract service104A. For example, the log data110may identify a code segment by one or more of the following: package name, class name, method name, subroutine name, code branch traversed, line of code, block of code, line number, or other identifications. The log data110may include the values of one or more variables or parameters when the code segment is reached, such as parameter(s) of a method call, or value(s) returned from a method call. The log data110may also include a timestamp (e.g., date, time, or date and time) indicating when the code segment was accessed. In some cases, the log data110may indicate a build number, version number, build date, revision number, or other indication of the particular version of the contract service104A. The log data110may also include other information. The log data110may be generated by the contract service104A itself, or by other processes that have access to information regarding the execution of the contract service104A. In some embodiments, the log data110for a request108may include a request identifier (ID) that identifies (e.g., uniquely identifies) the request108among a plurality of requests108.

In some embodiments, request data112may be generated during execution of the contract service104A. The request data112may include at least a portion of the data included with the request(s)108, and may enable the request(s)108to be substantially replayed during testing of the contract service104B. In cases where the request(s)108are communicated using a particular protocol such as a version of HyperText Transfer Protocol (HTTP), the request data112may include the request(s)108formatted according to the protocol. The request data112for a request108may include one or more of the following: an indication of the communication protocol used to communicate the request108; a network address or location, such as a Uniform Resource Locator (URL), Uniform Resource Name (URN), or other Uniform Resource Indicator (URI), or an Internet Protocol (IP) address; a path to a requested resource; one or more parameters; one or more headers; one or more cookies; or other information. The request data112for a request108may also include the request ID identifying the particular request108from which the request data112was captured. The request data112may be generated by the contract service104A itself, or by another process that has access to information regarding the request(s)108received by the contract service104A. For example, the request data112may be captured and stored by a communications interface that conveys the request(s)108from the client process(es)106to the contract service104A. In some cases, the request data112for a particular request108may include a timestamp indicating when the request108was received by the contract service104A.

One or both of the log data110or the request data110may be stored on one or more storage devices114. The storage device(s)114may store data in any format, using any type of storage technology. For example, the storage device(s)114may store data in a relational or non-relational storage format. One or both of the log data110or the request data112may be stored in a compressed format. One or both of the log data110or the request data112may be stored in an encrypted format.

In some embodiments, separate datastores or separate sets of data storage device(s)114A and114B may be employed to respectively store the log data110and the request data112. For example, the storage device(s)114A may be tasked with storing the log data110generated during execution of the contract service104A. The storage device(s)114A may be associated with the contract process device(s)102. In some cases, the storage device(s)114A may be component(s) of the contract processing device(s)102. The storage device(s)114B may be separate from the contract processing device(s)102. Using separate sets of storage device(s)114A and114B to respectively store the log data110and the request data112may contribute to the scalability of one or both of the log data110or the request112, enabling storage of larger amounts of data.

As shown inFIG. 1, the environment may include one or more analysis devices116. The analysis device(s)116may include any type of computing device(s). The analysis device(s)116may execute a log data analyzer118. The log data analyzer118may access and analyze the log data110to generate metrics data120that indicates a number of executions of one or more code segments during the processing of one or more requests108. As described above, and as illustrated inFIG. 1, the metrics data120may include any number of records122. Each record122may include a request ID124identifying a particular request108processed by the contract service104A. Each record122may also include any number of counters126that each indicates whether, or how many times, a particular code segment was executed during the processing of the request108identified by the request ID124. In some embodiments, a counter126for a code segment may have a Boolean value (e.g., true or false, 0 or 1, etc.) that indicates whether the code segment was executed zero times or at least one time. In some embodiments, a counter126may have a value (e.g., numeric value) indicating a number of times the code segment was executed, such that the value may be zero or higher. In some embodiments, the metrics data120may be arranged, and stored, in a relational database table such that the records122correspond to rows in the table, the request ID124may be a primary key of the table, and the counters126correspond to columns in the table. In some embodiments, the metrics data120may be stored in the storage device(s)114or elsewhere.

The metrics data120may be accessed by a test case generator128executing on the analysis device(s)116. The test case generator128may identify a subset of the records122of the metrics data120, the subset including records122that exhibit representative combinations of the values of the counters126among all the records122of the metrics data120. For example, the metrics data120may include five different records122(e.g., for five different requests108) that each exhibits counter values “3”, “1”, and “0” for three counters126. The test case generator128may generate a subset of records122that includes a representative one of the five records122that exhibits the particular counter values. In this way, the test case generator128may collapse the metrics data120into a representative subset of the metrics data120that includes one record122for each of the combinations of counter values present in the full metrics data120. Accordingly, the representative subset of the metrics data120may include a plurality of records122that are unique among the records122in the representative subset. For each of the records122in the subset, the test case generator128may generate a test request130, also described herein as a test case. The test case generator128may access the request data112for the request ID124of a record122in the representative subset, and incorporate the request data112into the test request130. The test request130may then be employed, as a test case, to test a test version of the contract service104B. Accordingly, embodiments may identify each combination of counters126present in the metrics data120, and for each combination replay the corresponding request data112as a test request130to test the contract service104B.

In some embodiments, a test client132may execute to replay the test request(s)130on the contract service104B under test. The test client132may execute, e.g., in a test environment, to substantially mimic the behavior of the client process(es)106sending request(s)108. AlthoughFIG. 1depicts the test client132as a software module executing on the analysis device(s)116, embodiments are not so limited. In some embodiments, the test client132may run on a separate set of one or more computing device(s), such as dedicated test server(s).

AlthoughFIG. 1depicts the contract processing device(s)102, analysis device(s)116, and storage device(s)114as separate devices in the environment, embodiments are not so limited. The operations performed by the contract processing device(s)102, analysis device(s)116, and storage device(s)114may be distributed across any number of computing devices, or on a same device or cluster of devices.

FIG. 2is a block diagram illustrating an example of a data aggregation service202that calls multiple network services206to populate a shared data object212, according to embodiments. In some embodiments, the contract service104(e.g., one or both of the contract service104A or the contract service104B) may comprise a data aggregation service202executing on the contract processing device(s)102. On receiving a request108from the client process(es)106, the data aggregation service202may generate a shared data object212to store data associated with the request108. For example, in cases where the request108indicates a purchase of one or more items through an e-commerce service, the shared data object212may be a contract object including data describing one or more aspects of the contract governing the purchase. The contract object may store any amount of data that describes any aspect of the contract including, but not limited to, the purchase price, taxes, discounts, gift information, item(s) to be purchased, description of item(s), quantity of item(s) to be purchased, availability of item(s), shipping date or time estimate, shipping method, requested delivery period, seller of the item(s), manufacturer of the item(s), buyer's name, recipient's name, buyer's location, recipient's location, shipping address, payment information, and so forth.

On receiving the request108, the data aggregation service202may create an initial shared data object212. The initial shared data object212may include information that the data aggregation service202is able to determine based on the request108itself, but the initial shared data object212may otherwise be a shell object in which subsequently generated data is to be written. To populate the shared data object212, the data aggregation service202may perform any number of service calls204to one or more network services206. Each service call204may include input data208, and may call a network service206that is configured to process the input data208and generate service-generated data210. For example, the data aggregation service202may perform a service call204A to that includes input data208A describing the purchase price of item(s) in a purchase, and location information regarding the buyer. The service call204A may be to a network service206A that is configured to calculate an amount of tax to be paid with the purchase, based on the purchase price and the location information, or other input data208A. The network service206A may generate and return service-generated data210A that includes the calculated amount of tax. The data aggregation service202may then incorporate the received service-generated data210A into the shared data object212. Alternatively, one or more network services206may have (e.g., direct) access to the shared data object212, and may be configured to add the service-generated data210into the shared data object212without sending the service-generated data210to the data aggregation service202.

The network service(s)206may be configured to perform operations including, but not limited to, one or more of the following: calculate taxes based on purchase price, location, or other input data; determine availability of item(s) to be purchased; determine an estimated delivery time, date, or time and date for delivering purchased item(s); validate or process a gift certificate, credit card, bank card, store credit, or other method of purchase; process gift information, such as a gift certificate amount, gift message, gift wrapping options, and so forth; determine address information, contact information, or other stored data associated with a buyer or recipient of a purchase; determine information describing the seller, manufacturer, distributor, or other entity associated with the item(s); or other operation(s).

In some embodiments, the network service(s)206may be remote with respect to the data aggregation service202, executing on different computing device(s) than the data aggregation service202. In such cases, the service call(s)204and the service generated data210may be communicated over one or more networks such as wide area networks (WANs) or local area networks (LANs). In some cases, one or more of the service call(s)204or the service-generated data210may be encrypted for communication. In some embodiments, one or more of the network services206may be a local service with respect to the data aggregation service202, such that the network service(s)206execute on a same computing device or cluster of devices as the data aggregation service202.

In some embodiments, the data aggregation service202may generate change tracking data214. The change tracking data214may describe the service-generated data210that has been added to the shared data object212. The change tracking data214may also describe one or more portions of service-generated data to be added to the shared data object212. Accordingly, the change tracking data214may be described as a task list comprising: completed tasks, for which the service-generated data210has been added to the shared data object212; and incomplete tasks, for which the service-generated data210has not yet been generated or added to the shared data object212. The change tracking data214may indicate when particular instances of service-generated data210were added to the shared data object212, and which network service(s)206generated the service-generated data210. The change tracking data214may also indicate when the service call(s)204were made to the network service(s)206to request determination of service-generated data210to be added to the shared data object212.

In some embodiments, the data aggregation service202may generate log data216describing the service call(s)204made to the network service(s)206to populate the shared data object212for a particular request ID. The log data216may also describe the input data208sent to the network service(s)206, or the service-generated data210provided by the network service(s)206. The log data216may be stored, accessed, and processed as part of the log data110to generate test request(s)130.

FIG. 3is a block diagram illustrating a data aggregation service202that calls multiple network services206via a service call mediator302to populate a shared data object212, according to embodiments. Elements inFIG. 3may be configured similarly to like-numbered elements ofFIG. 2, or may perform similar operations to those performed by like-numbered elements ofFIG. 2.FIG. 3illustrates embodiments in which a service call mediator302is employed to manage, convey, or otherwise mediate the service calls204made by the data aggregation service202to the network service(s)206. In such embodiments, the data aggregation service202may send service call(s)204to the service call mediator302, which may then forward the service call(s)204to the appropriate network service(s)206. The service call mediator302may receive the service-generated data210provided by the network service(s)206in response to the service call(s)204, and forward the service generated data210to the data aggregation service202. In the example ofFIG. 3, the log data216describing the various service call(s)204may be generated by the service call mediator302and stored in the storage device(s)114or elsewhere. In some embodiments, the service call mediator302may also generate request data304. The request data304may include information describing the service call(s)204made from the data aggregation service202to the network service(s)206, as mediated by the service call mediator302. In some cases, the service call mediator302may also capture and emit request data304describing the request(s)108made from the client process(es)106to the contract service104A. The request data304may be employed in subsequent test case generation as described herein.

The service call mediator302may execute on the same computing device(s) as the data aggregation service202, or on different computing device(s). Embodiments may employ the service call mediator302to mediate all the service call(s)204made by the data aggregation service202to the network service(s)206. Alternatively, embodiments may employ the service call mediator302to mediate one or more of the service call(s)204made by the data aggregation service202, while other service call(s)204are sent by the data aggregation service202to the network service(s)206without using the service call mediator302.

The data aggregation service202shown inFIGS. 2 and 3may be configured to generate, maintain, update, and provide a shared data object212. In some embodiments, the data aggregation service202might be utilized in conjunction with software components and hardware devices that provide e-commerce functionality. For example, and without limitation, the technologies disclosed herein may be utilized with an online shopping or other e-commerce module (not shown) that provides a web site or other type of site for online shopping. The online shopping module may be configured to maintain and utilize an item catalog data store (e.g., an “item catalog”) that stores records for items available for purchase through the web site. The web site may also provide functionality for browsing and searching for web pages corresponding to items in the item catalog, for purchasing the items, and for performing other functions. The technologies disclosed herein might also be utilized with other types of e-commerce systems. In this regard, it should be appreciated that the configurations disclosed herein are not limited to use by e-commerce systems and may be utilized to optimize the process of data aggregation in other technical environments.

In cases where the data aggregation service202is utilized in conjunction with an online shopping module, such as that described above, the shared data object212may be utilized to store data associated with a customer purchase such as purchase contract data. For example, and without limitation, the service-generated data210, and other information in the shared data object212, may describe items to be purchased by the customer, payment information, customer information such as a billing or delivery address, or other types of information relating to a purchase contract.

The data aggregation service202may obtain the data to be stored in the shared data object212from any number of the network services206. For example, and without limitation, the data aggregation service202might make one or more service calls204that comprise one or more input values of input data208to the network service(s)206. In response to the service call(s)204, the network service(s)206may perform various types of processing to generate instances of the service-generated data210for inclusion in the shared data object212. Various other services may then access at least a portion of the data stored in the shared data object212. In some cases, the client process(es)106may access data of the shared data object212and present such data through a UI, such as through an online shopping website. For example, the client process(es)106may access data regarding delivery or shipping estimated, calculated tax to be paid, item(s) descriptions, or other data after the data has been stored in the shared data object212, and present the data to an end-user in an e-commerce web site or other UI.

When utilized in conjunction with an e-commerce system, the network service(s)206may be configured to provide various types of information about a customer purchase for storage in the shared data object212and use by the client process(es)106or other services. For example, and without limitation, one network service206A may be utilized to provide service-generated data210A that includes a customer's shipping address, while another network service206B may be configured to provide service-generated data210B that includes information about the item(s) to be purchased. Other services206may compute the sales tax for the purchase, the shipping cost, or the total purchase price for the purchase, and return the computed information as the service-generated data210to the data aggregation service202. Other service(s)206may provide other types of data. The service-generated data210returned by the network service(s)206may be stored in the shared data object212. The data aggregation service202may also provide functionality enabling the client process(es)106or other services to retrieve or modify the data stored in the shared data object212.

In some cases, the types or instances of data stored in the shared data object212may have various interdependencies, such that the modification of one portion or instance of service-generated data210may cause the updating of other portion(s) or instance(s) of service-generated data210. For example, if data is added to the shared data object212describing an additional item to be purchased by a customer, additional network service call(s)204may be made to network service(s)206to update the data in the shared data object212describing the total order price, the shipping cost, delivery estimates, whether free shipping is available, or other types of information. Other service call(s)204may be made based upon other changes or additions to the data in the shared data object212.

To update the data in the shared data object212, the data aggregation service202may make one or more service calls204to the appropriate network service(s)206. In some cases, a short period of time may have elapsed since a previous service call204such that the input data208to a network service206may not have changed since the previous call, and such that the service-generated data210returned by the network service206is not likely to have changed. For example, if the items in a customer purchase from an e-commerce merchant have not changed, then there may be a high probability that there is no change in the shipping cost for the item(s), unless other data has since changed that may impact the shipping cost, such as the shipping address. In this example, a service call204may be made to a network service206unnecessarily, thereby increasing latency or reducing availability of the network service206or the data aggregation service202. This may be particularly true when there is a large amount of data in the shared data object212or when there are a large number of network services206to be called. Accordingly, in some embodiments at least a portion of the service-generated data210may be accessed from a cache instead of repeating a service call204to a network service206, in cases where the service-generated data210is unlikely to have changed since a previous call. In some embodiments, the data may be cached with a time-to-live (TTL) indicating how long since the data has been cached and how fresh the cached data is. Cached data with a high TTL may be not used, and instead a service call204may be made to request updated service-generated data210from the network service206. Older cached data may be purged from the cache, and replaced with updated service-generated data210.

Another mechanism that the data aggregation service202may utilize to avoid unnecessary network service calls204to network services206involves the use of the change tracking data214. In some embodiments, the data aggregation service202may be configured to track changes to data in the shared data object212. For example, and without limitation, the data aggregation service202might maintain the change tracking data214that describes the changes to the various instances of service-generated data210in the shared data object212. When the data aggregation service202determines that a second or subsequent call to a network service206may be necessary (e.g. as a result of the modification of data in the shared data object212), the data aggregation service202may employ the change tracking data214to determine whether the data in the shared data object212that is used as input data208to the network service206has changed since the previous service call204to the network service206. If the change tracking data214indicates that the input data208has not changed, the data aggregation service202may not make an additional service call204to the network service206. The caching mechanism and the change tracking mechanism described above may be utilized in any combination in some embodiments.

The data aggregation service202may also utilize a filtering mechanism to perform caching on a more granular level than at the level of a single instance of service-generated data210, and to make communication with the network service(s)206more efficient. The network service(s)206may be configured to provide filtering data to the data aggregation service202. The filtering data may describe the specific data from the shared data object212that each network service206utilizes. For example, and without limitation, a network service206A might provide filtering data indicating that the network service206A should be subsequently called if (e.g., only if) particular field(s) of data are changed in the shared data object212. Similarly, a network service206B might provide filtering data indicating that the network service206B is to be subsequently called if other field(s) are changed. Based on the filtering data, the data aggregation service202may call network service(s)206a subsequent time if the data specified by the network service(s)206in the filtering data has changed (and the associated TTL has not elapsed). Additionally, the data aggregation service202may utilize the filtering data to minimize the amount of data sent to a network service206in a service call204. For example, the field(s) specified in the filtering data may be provided to the network service(s)206as input data208.

In some cases, the data aggregation service202may be configured to be expandable to support additional network service(s)206providing additional service-generated data210. In some embodiments, a new network service206may register itself with the data aggregation service202to indicate one or more of: the particular service-generated data210that may be provided by the network service206; or the particular input data208that may be sent to request the service generated data210. Following registration, the data aggregation service202may then begin sending service call(s)204to the newly registered network service206to request service-generated data210to populate the shared data object212. Accordingly, the data aggregation service202may be described as having a pluggable architecture to facilitate bringing new network service(s)206online, and the network service(s)206may be described as plug-in(s) to the data aggregation service202.

FIG. 4is a block diagram illustrating an example of analyzing log data110to generate metrics data120, according to embodiments. In the example shown, the log data analyzer118accesses log data110A and log data110B that describe code segments executed during the processing of two different requests108having different request IDs124. The log data analyzer118may analyze log data110associated with any number of requests108. The log data analyzer118generates the metrics data120that includes records122A and122B respectively corresponding to the two requests108. The log data analyzer118may examine the log data110for each request108. For each instance where the log data110indicates that a code segment has been executed during processing of a request108, the log data analyzer118may update the record122for the request108by incrementing a counter126associated with the code segment. Accordingly, each record122may comprise a summary of the code segments executed during the processing of the corresponding request108.

AlthoughFIG. 4depicts the metrics data120as being arranged in a relational table, embodiments support the use of other data formats or data structure to store the metrics data120, the log data110, or any number of data sets that are intermediate between the log data110and the metrics data120. In some embodiments, the log data110may not explicitly identify a particular class or a particular method in which the code segment may be found, as in the example ofFIG. 4. In some cases, the log data110may include descriptive information regarding the operations being performed in the code segment during its execution. In some embodiments, the particular location of the code segment in the executable code may be inferred based on this descriptive information. In some cases, the log data110may indicate a particular code branch that is being followed, such as whether the execution is in an IF or ELSE branch, whether the execution is in a TRY or CATCH branch, and so forth. The particular example of the log data110shown inFIG. 4is not limiting of embodiments.

In some embodiments, the log data110may be initially processed to generate intermediate data402in a key-value format. As shown in the example ofFIG. 4, a record of the intermediate data402may include a key that is a combination of the request ID (e.g., “8695307”) and a code segment ID (e.g., “X.A 394”). The record of the intermediate data402may include a value that is the value of the counter126corresponding to the request ID and the code segment. In the example ofFIG. 4, the record122A for request ID “8695307” in the metrics data120may correspond to seven key-value records in the intermediate data402. Each of the seven records may correspond to a different code segment, and the value of each of the seven records may be the value of the counter126indicating a number of executions of the code segment, or whether the code segment executed at least once. In embodiments that employ the intermediate data402, or other intermediate data sets, the records of the intermediate data402corresponding to a particular request ID may be joined or otherwise combined to generate a record of the metrics data120. The processing of the log data110to generate one or both of the intermediate data402or the metrics data120may be performed by the log data analyzer118or other software module(s). Use of the intermediate data402in a key-value format may facilitate the addition of additional code segments to be tracked in the intermediate data402and the metrics data120.

FIG. 5is a block diagram illustrating an example of generating test cases such as the test request(s)130based on an analysis of the metrics data120, according to embodiments. The test case generator128may access the metrics data120generated as described above with reference toFIG. 4. The test case generator128may determine a representative subset of the metrics data120that includes those records122that exhibit a representative combination502of the values of the counters126. In the example ofFIG. 5, the metrics data120includes four records122that have counter values of “1”, “0”, “1”, “0”, “2”, “0”, and “1”. The test case generator128may select a representative one of the four records122to include in the representative subset of records122to be used in test case generation. For each of the records122in the representative subset, the test case generator may retrieve the request data112corresponding to the request ID124in the record122. The test case generator128may generate a test request130that includes the request data112for one or more of the records122in the subset of representative combinations502. Accordingly, as shown in the example ofFIG. 5, the test requests130may include a test request130A that comprises the request data112A previously recorded for request ID “8695307”, and a test request130B that comprises the request data112B previously recorded for request ID “8695493”. The test requests130may then be sent to the contract service104B, e.g., replayed via the contract service104B, to test the contract service104B. In some embodiments, the intermediate data402may be analyzed to generate the test request(s)130, e.g., without generating or using the metrics data120. One or more of the log data110, the intermediate data402, or the metrics data120may be stored on the storage device(s)114. In some cases, one or more of the log data110, the intermediate data402, or the metrics data120may be stored in a data warehouse or other storage that employs a non-relational data storage format.

The examples ofFIGS. 4 and 5are not limiting of embodiments. One or both of the log data110or the metrics data120may include different information, or may be arranged in a different format, than shown inFIGS. 4 and 5. Although the examples ofFIGS. 4 and 5depict the metrics data120in table form, e.g., arranged as in a relational database table, embodiments support other formats and arrangements for the metrics data. The metrics data120may include any number of counters126corresponding to different code segments. In some cases, the metrics data120may include a large number such as thousands of counters126.FIGS. 4 and 5depict a smaller number of counters126solely for clarity of illustration. In some embodiments, as described above, the metrics data120may include counters126that hold a Boolean value to indicate whether or not a particular code segment has executed at least once or not executed during processing of a request108.

FIG. 6is a flow diagram illustrating one or more embodiments of a process for generating test cases based on analyzing the log data110and the metrics data120. Operations of the process may be performed by one or more of the log data analyzer118, the test case generator128, other software module(s) executing on the analysis device(s)116, or other software module(s) executing on other device(s).

At602, the log data110is accessed. As described above, the log data110may be generated or collected during execution of the contract service104A to process one or more requests108, and the log data110may describe one or more code segments of the contract service104A that were executed or otherwise accessed during the processing of the requests108.

At604, the log data110is analyzed to generate the metrics data120. As described above, the metrics data120may include a record122corresponding to each processed request108, and a record122may include a plurality of counters126each indicating whether, or a number of times, a particular code segment was executed during processing of the request108. At606, in some embodiments the metrics data120may be stored in the storage device(s)116or elsewhere. The metrics data120may be stored in one or more tables of a relational database or in some other storage format.

At608, the metrics data120may be analyzed to identify a subset of one or more records122that each includes a representative combination of the counters126present in the metrics data120, as described above.

At610, for one or more of the subset of records(s)122identified at608, the request data112may be retrieved. As described above, the request data112may have been previously captured and stored during the processing of request(s)108.

At612, the test request(s)130may be generated to include the request data112corresponding to the subset of records122. At614, in some embodiments the test request(s)130may be stored in the storage device(s)116or elsewhere.

At616, the test request(s)130may be employed to test the contract service104B. Testing may include sending the test request(s)130, with the previously captured request data112, to the contract service104B to effectively replay the previous request(s)108on a later build of the contract service104. In some cases, testing may be performed as part of a build process, such as in build verification testing, regression testing, or otherwise. Testing may also include other types of testing, such as performance testing.

In some cases, the test request(s)130may be replayed in the contract service104B to test the contract service104B in isolation from other service(s) called by the contract service104B. For example, the contract service104B may be tested, using the test request(s)130, while the various communications with network service(s)206are replaced with mocked communications, dummy data, or spoofed data that substantially mimics responses (e.g., the service-generated data210) that may otherwise be received from network service(s)206in a production environment.

FIG. 7is a flow diagram illustrating one or more embodiments of a process for generating test cases based on analyzing the metrics data120, including filtering based on various criteria. Operations of the process may be performed by one or more of the log data analyzer118, the test case generator128, other software module(s) executing on the analysis device(s)116, or other software module(s) executing on other device(s).

At702, the metrics data120may be analyzed to identify a subset of one or more records122that each includes a representative combination of the counters126present in the metrics data120, as described above.

In some embodiments, each of the subset of records122may be employed to generate a test request130to replay its corresponding request data112in the contract service104B. However, in some cases the subset of records122exhibiting representative combinations of counters126may include too many records122to be employed as test requests130, e.g., given constraints on time or resources available for testing the contract service104B. Accordingly, in some embodiments the subset of records122may be initially filtered based on one or more criteria, and one or more of the subset of records122may be employed to generate test request(s)130.

At704, in some embodiments the subset of records122may be filtered based on a number of occurrences of the combination of counters126in the metrics data120. For example, the subset of records122may be filtered to determine those records122exhibiting a particular combination of counters126that occurred at least a minimum number of times in the metrics data120.

At706, in some embodiments the subset of records122may be filtered based on particular value(s) of one or more counters126. For example, the subset of records122may be filtered to determine those records122for which the value of a particular counter126is at least a minimum value.

At708, in some embodiments, the subset of records122may be filtered based on particular client process(es)106that generated the requests108. In some cases, the request(s)108generated by different client processes106may cause the traversal of different code paths, or different sets of code segments, in the executable code of the contract service104. Accordingly, embodiments may filter based on client process(es)106to ensure that such different code paths are exercised during testing of the contract service104B. Filtering may be performed to include, in the subset of records122, at least one record corresponding to a request108sent by individual ones of the client processes106. Filtering may include selecting the records122corresponding to a client process106A instead of other records122corresponding to a different client process106B, and employing the selected records122to generate test requests130. Filtering based on client process106may ensure that the code segments are adequately tested for those client process(es)106that may generate fewer requests108than other, more active client process(es)106.

In some embodiments, the contract service104may expose or otherwise provide a plurality of interfaces, such as application programming interfaces (APIs), through which the client process(es)106may submit request(s)108. At710, in some embodiments the subset of records122may be filtered based on the particular interface(s) that were employed to communicate the request(s)108to the contract service104A. Filtering may be performed to include, in the subset of records122, at least one record corresponding to a request108sent to individual ones of the plurality of interfaces. Embodiments may filter based on interface(s) to ensure that different code paths associated with the different interface(s) are exercised during testing of the contract service104B.

At712, for one or more of the records(s)122identified at702and filtered according to one or more of704,706,708, or710, the request data112may be retrieved. As described above, the request data112may have been previously captured and stored during the processing of request(s)108.

At714, the request data112for the filtered record(s)122may be employed to generate test request(s)130for testing the contract service104B, as described above. Embodiments may employ any of the filtering methods described herein, alone or in any combination. Embodiments may also employ other methods for filtering the records122to determine test cases.

FIG. 8is a block diagram illustrating a computer system800configured to implement at least a portion of the queue-based communication described herein according to various embodiments. The computer system800may include any number of computing devices, and may execute or otherwise include any number of software modules such as applications, processes, libraries, interfaces, APIs, and so forth. For example, computer system800may be configured to implement one or more of the contract processing device(s)102, the analysis device(s)116, the storage device(s)114, end-user client device(s), and so forth. The computer system800may include any type of computing device including but not limited to: a personal computer system, a desktop computer, a rack-mounted computing device, a laptop or notebook computer, a tablet computer, an electronic book (e-book) reader, a wearable computer, an implanted computer, a mainframe computer system, a distributed computing device (e.g., cloud server), a handheld computer, a workstation, a network computer, a consumer device, an automotive computer, a home entertainment device, a smart appliance, a storage device, a telephone, a remote control, a game controller, a gaming system, a mobile telephone, a smartphone, or any other type of computing device.

The computing system800may include one or more physical computing devices. The computing system800may also include one or more of a virtual computing environment, a hypervisor, an emulation, or a virtual machine executing on one or more physical computing devices. In some embodiments, the computing system800may comprise a cluster, cloud, farm, or other grouping of multiple devices that coordinate operations to provide load balancing, failover support, parallel processing capabilities, shared storage resources, or other aspects.

Computer system800may include one or more processors810coupled to a system memory820via one or more input/output (I/O) interfaces830. One or more of the processor(s)810may include multiple cores, which may be configured to execute single-threaded or multi-threaded code. In various embodiments, computer system800may be a uniprocessor system including one processor810, or a multiprocessor system including multiple processors810(e.g., two, four, eight, or any other number). The processor(s)810may include any processor(s) capable of executing instructions. For example, in various embodiments, the processor(s)810may include general-purpose or embedded processor(s) implementing any of a variety of instruction set architectures (ISAs), such as the x86, PowerPC, SPARC, or MIPS ISAs, or any other ISA. In multiprocessor systems, each of processors810may implement the same ISA. Alternatively, different ones of the processors810may implement different ISAs.

The computer system800may include one or more system memories820, described herein as system memory820, configured to store one or more of code825or data826such that the code825and the data826are accessible by the processor(s)810. The system memory820may comprise one or more computer-readable storage media that include one or more of the following: an electronic storage medium, a magnetic storage medium, an optical storage medium, a quantum storage medium, a mechanical computer storage medium, a solid state storage medium, and so forth. The system memory820may be implemented using any memory technology, including but not limited to one or more of the following: read-only memory (ROM), random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), Rambus DRAM (RDRAM), extended data out (EDO) RAM, synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), electrically erasable programmable ROM (EEPROM), flash memory, non-volatile memory, volatile memory, or any other type of memory. The system memory820may be incorporated into the computer system800. In some cases, at least a portion of the system memory820may be external to the computer system800and accessible via the network interface(s)840or the I/O device(s)870.

The system memory820may include the code825. The code825may include executable instructions that are executable by processor(s)810to implement the embodiments described herein. The code825may implement one or more executable software modules, such as applications, programs, processes, libraries, interfaces, APIs, scripts, routines, and so forth. The code825may be in a compressed or uncompressed form. The code825may be in an encrypted or unencrypted form. The code825may include any number of instructions that may be employed to program a computing device, or other electronic device, to perform the operations described herein. The code825may be arranged in any format and according to any language. In some embodiments, the code825may include machine-executable binary instructions that are configured to execute on the processor(s)810, the instructions generated by compiling, linking, or otherwise processing source code written in any programming language. In some embodiments, the code825may include intermediate language instructions (e.g., bytecodes) that execute within a runtime application such as a Java™ Virtual Machine (JVM), C#™ runtime, or any other runtime, interpreter, virtual machine, or execution engine running on the computer system800. In some embodiments, the code825may include instructions written in a scripting language or interpreted language, such as JavaScript™, ActiveScript™, VBScript™, Perl™, and so forth. In such cases, the code825may execute within a runtime, interpreter, virtual machine, scripting engine, or other process that executes on the computer system800.

The code825may include instructions to implement one or more of the contract service(s)104, the client process(es)106, the log data analyzer118, the test case generator128, the data aggregation service202, the network service(s)206, or the service call mediator302. The code825may also include instructions to implement at least one operating system (OS) that executes on the computer system800. The at least one OS may include one or more of the following: any version of the UNIX™ OS; any version of the Linux™ OS; any version of iOS™ or OSX™ from Apple Corp. of Cupertino, Calif., USA; any version of Windows™ or Windows Mobile™ from Microsoft Corp. of Redmond, Wash., USA; any version of Android™ from Google Corp. of Mountain View, Calif., USA and its derivatives from various sources; any version of Palm OS™ from Palm Computing, Inc. of Sunnyvale, Calif., USA and its derivatives from various sources; any version of BlackBerry OS™ from Research In Motion Ltd. of Waterloo, Ontario, Canada; any version of VxWorks™ from Wind River Systems of Alameda, Calif., USA; or other operating systems.

The system memory820may include data826employed during operations of the computer system800. The data826may include one or more of the following: the request(s)108, the log data110, the request data112, the metrics data120, the test request(s)130, the shared data object212, the change tracking data214, the log data216, the service-generated data210, the input data208, or the service call(s)204. The data826may be stored in any format. In some embodiments, at least a portion of the data826may be stored externally to the computer system800, on one or more other devices or storage media that may communicate with the computer system800via the network interface(s)840, the I/O interface(s)830, or the I/O device(s)870. The system memory820may include persistent storage such as one or more hard drives, tape drives, solid state memory, other mass storage devices, or any other persistent storage device. The system memory820may also include active memory, physical memory, or virtual memory that is employed by processes executing on the computer system800. The system memory820may include cache memory.

The system memory820may comprise one or more non-transitory storage media that store information such as one or both of the code825or the data826. Non-transitory storage media may store information in any arrangement, and using any storage technology, such that the stored information is computer-readable, i.e., readable by a machine such as a computing device. Non-transitory storage media may include any media that is configured to store information such that the stored information persists for at least a minimum duration relative to the operations described herein. Non-transitory storage media may include any media that is transportable as a physical object. Embodiments may include software comprising one or both of the code825or the data826stored on the system memory820that comprises one or more non-transitory storage media. Such software may be used to program the computer system800, or other electronic devices, to perform one or more operations according to various embodiments.

Embodiments may also include software that is transmitted in a transitory form as an electromagnetic transmission, optical transmission, acoustical transmission, or any other type of signal or communication. Such software may be communicated using any communication protocol over the Internet or any other communications network, using a wired or wireless communication path. In such cases, the software may be received using the network interface(s)840and employed to program the computer system800, or other electronic devices, to perform one or more operations according to various embodiments.

The I/O interface(s)830may be configured to coordinate I/O traffic between the processor(s)810, the system memory820, and any peripheral devices accessible to the computer system800through the network interface(s)840or other peripheral interface(s). In some embodiments, the I/O interface(s)830may perform protocol, timing or other data transformations to convert data from one component (e.g., the system memory820) into a format suitable for use by another component (e.g., the processor(s)810). In some embodiments, the I/O interface(s)830may include support for devices attached through various types of peripheral buses that support any bus standard such as any variant of the Peripheral Component Interconnect (PCI) bus standard or the Universal Serial Bus (USB) standard. In some embodiments, the function of I/O interface(s)830may be split into two or more separate components, such as a north bridge and a south bridge. Moreover, in some embodiments at least some of the functionality of I/O interface(s)830, such as an interface to the system memory820, may be incorporated directly into the processor(s)810.

The computer system800may include one or more network interfaces840coupled to the I/O interface(s)830. The one or more network interfaces840may be employed by the various components or software of the computer system800to communicate with other systems and/or components over one or more communications networks850. The network interface(s)840may include one or more network interface controllers (NICs), transceiver devices, or other types of network communications devices configured to send and receive communications over the network(s)850.

The computer system800may employ the network interface(s)840to communicate and exchange data with one or more other devices860over the network(s)850. The network interface(s)840may support one or more wireless networking protocols such as any version of the Institute of Electrical and Electronics Engineers (IEEE) 802.11 standard or other wireless networking standard. The network interface(s)840may also support communication via any wired data networks, such as Ethernet networks. The network interface(s)840may also support communication via any telecommunications or telephony network such as analog voice networks or digital fiber communications networks, via storage area networks such as Fibre Channel storage area networks (SANs), or via any other suitable type of network and/or protocol.

In some embodiments, the network interface(s)840may be configured to enable communication between the computer system800and one or more I/O devices870, or between the computer system800and external (e.g., remote) storage device(s). The I/O device(s)870may include one or more data input devices such as a keyboard, a keypad, a mouse, a pen, a game controller, a touch input device, an audio input device (e.g., a microphone), a gestural input device, a haptic input device, an image or video capture device (e.g., a camera), or other data input devices. In some cases, the I/O device(s)870may include one or more data output devices such as a display, an audio output device (e.g., a speaker), a printer, a haptic output device, and so forth.

The I/O device(s)870may be components of all the devices included in the computer system800or may be components of different devices that comprise the computer system800. The I/O device(s)870may be physically incorporated with the computer system800. In some embodiments, one or more of the I/O device(s)870may be externally placed relative to the computer system800and may communicate with the computer system800using a wired or wireless connection, such as over the network interface(s)840. In various embodiments, the computer system800may include more, fewer, or different components than those illustrated inFIG. 8.

The network(s)850may include public networks such as the Internet, private networks such as an institutional or personal intranet, or some combination of private and public networks. The network(s)850may include any type of wired or wireless network, including but not limited to local area networks (LANs), wide area networks (WANs), wireless WANs (WWANs), wireless LANs (WLANs), mobile communications networks (e.g., 3G, 4G, etc.), and so forth. The various computing systems, devices, and processes described herein may employ the network(s)850for communication. Such communications may be encrypted or otherwise secured. For example, such communications may employ one or more public or private cryptographic keys, ciphers, digital certificates, or other credentials supported by a security protocol, such as any version of the Secure Sockets Layer (SSL) or the Transport Layer Security (TLS) protocol. The network(s)850may be employed for communications between any of the processes or devices depicted inFIGS. 1-3.

The various methods, processes, and routines illustrated in the figures and described herein represent example embodiments. The methods may be implemented as software, as hardware, as manual operations, or as any combination thereof. The order of operations performed by any method, process, or routine described herein may be changed, and one or more operations may be added, reordered, combined, omitted, or modified. The operations may be performed serially or in parallel. In cases where the methods, process, and routines described herein are implemented as computer programs, any number of instances of these programs may be executed on any number of separate computer systems or on the same computer system. Although certain operations may be described herein as performed by particular devices, software programs, processes, or entities, embodiments are not limited to these examples. A variety of alternative embodiments will be understood by those having ordinary skill in the art.