Patent ID: 9483393
Filing Date: 2016-11-01
CPC Classification: G06F

Claim Text:
1. A computer-implemented method for performing an experiment on a software application, the method comprising performing computer-implemented operations for: receiving an experiment configuration from a computing device associated with a developer of the software application, the experiment configuration comprising a test factor for the software application, a set of test values for the test factor, and an objective metric to be optimized in the experiment, the test factor representing a feature of the software application to be tested in the experiment; determining an experiment strategy based at least in part upon the experiment configuration, the experiment strategy comprising a plurality of experience configurations of the software application, a number of rounds of the experiment, and two or more of the test values to be tested in individual rounds of the experiment, wherein the plurality of experience configurations comprise different application settings; identifying a plurality of user computing devices associated with the software application and allocating the plurality of experience configurations to the plurality of user computing devices according to the experiment strategy, the plurality of experience configurations being used to configure corresponding instances of the software application executing on the plurality of user computing devices and individual ones of the plurality of experience configurations specifying one of the set of test values to be used as a value of the test factor in the corresponding instance of the software application; causing a first round of the experiment to be performed on one or more of the plurality of user computing devices to test a first subset of test values; analyzing experiment data related to the first round of the experiment to identify a sub-optimized test value from among the first subset of the test values based on the objective metric; including the sub-optimized test value into a second subset of the test values to be tested in a second round of the experiment; causing the remaining experiment to be performed and collecting additional experiment data related to the experiment; and analyzing the additional experiment data to identify an optimized test value for the test factor from among the set of test values based on the objective metric.