Patent ID: 8776021

Claim:
A method for performing an experiment on a software application, the method comprising: under control of an application experiment system comprising computing hardware including one or more processors: providing to a developer computing system an interface configured to receive an experiment configuration for a software application, wherein the experiment configuration includes: one or more treatment configurations, wherein individual ones of the one or more treatment configurations include: one or more test factors, wherein individual ones of the one or more test factors indicates factors to be changed in an instance of the software application during execution of the software application; one or more test values corresponding to the one or more test factors, wherein individual ones of the one or more test values are applied to the corresponding test factor in an instance of the software application during execution of the software application; a treatment distribution parameter, wherein the treatment distribution parameter includes information indicating how the treatment configuration is to be distributed to application computing systems executing instances of the software application; and wherein one of the treatment configurations is a control treatment configuration and at least one of the treatment configurations is a test treatment configuration; receiving from the developer computing system via the interface the experiment configuration; receiving from the developer computing system via the interface a request to run the experiment; in response to a request to allocate a treatment for a running experiment received from an application computing system that is executing an instance of the software application: retrieving an experiment configuration for the running experiment; allocating, based at least in part on the experiment configuration and treatment distribution parameters for the running experiment, a treatment configuration to the application computing system; and providing, based at least on the allocated treatment configuration, one or more test values to the application computing system; logging the treatment data, wherein the treatment data includes information related to the treatment configuration allocated to the application computing system; and providing to the developer computing system experiment results data, wherein the experiment results data includes statistical data generated based at least partly on the treatment data.