Experimentation in internet-connected applications and devices

A content variation experiment system for performing variation testing of web pages is disclosed. A content provider receives requests for a web page undergoing an experiment. The content provider determines a variation from a plurality of variations of the web page to provide to the user. The content provider makes the determination without sending a network request to an experiment definition system used to define the experiment thereby reducing network latency.

BACKGROUND

The embodiments described herein pertain to internet-connected applications and devices.

A content variation experiment tests changes to content against one or more different versions of the content. One example of a content variation experiment is a web page variation experiment where an experiment is conducted that tests different variations of web pages. During variation testing of a web page, changes to a web page that produce positive results are determined. The positive results associated with a particular change to a web page validates that the change to the web page should be used in a production version of the web page, for example.

In client-side web content variation experiments, a third-party content provider works with a variation testing system to establish an experiment for one or more web pages of the third-party content provider. When a user requests a web page undergoing an experiment from the third-party content provider, the third-party content provider must communicate with the variation testing system to determine which variation of the web page to provide to the user. Since the third-party content provider must communicate with the variation testing system to determine which version of a web page to provide to the user, unnecessary latency may be added to the web page load time. Furthermore, in conventional variation experiments, the third-party content provider cannot easily make complex changes to the experiment as there is only a limited set of functionality that can be changed client-side on the web page. Client-side web variation testing systems cannot be used to test deeper logic in the third-party's backend such as algorithms or other product features.

BRIEF SUMMARY

The embodiments herein describe a variation experiment environment for performing variation experiments of any content item. Examples of content items include any internet-connected application, device, or web page. The variation experiment environment includes an experiment definition system and a third-party content provider. The third-party content provider works in conjunction with the experiment definition system to define a variation experiment in the code for the content item provided by the third party content provider. The variation experiment for the content item includes multiple variations of the content item where each variation includes a different change to the content item.

The definition of the variation experiment is stored at the third-party content provider. By storing the definition at the third party content provider, the third-party content provider beneficially minimizes application latency since the third-party content provider does not need to send a network request to the experiment definition system to determine which variation of the content item to provide to a client device requesting the content item. Rather, the third-party content provider itself can determine which variation of the content item to provide to the client device based on the definition of the variation experiment stored at the third-party content provider.

Furthermore, by storing the definition of the experiment at the third-party content provider, the third-party content provider can experiment with complex changes to the functionality of the content item undergoing experimentation. The third-party content provider can directly make revisions to the definition of the experiment that is stored at the third party content provider given that the experiment definition system is incapable of handling the complex changes directly at the experiment definition system.

DETAILED DESCRIPTION

FIG. 1is a block diagram of a variant experiment system environment100according to one embodiment. Generally, the variant experiment system environment100uses third-party content provider side experiment testing. The definition of a variation experiment is stored at a third-party content provider107. By storing the definition at the third party content provider107, the third-party content provider107beneficially minimizes application latency at client devices102requesting a content item undergoing experimentation since the third-party content provider itself can determine which variation of the content item to provide to the client devices102. Thus, the third-party content provider107does not need to send a network request to an experiment definition system105to determine which variation of the content item to provide to a client device102requesting the content item. Furthermore, by storing the definition of the experiment at the third-party content provider107, the third-party content provider107can experiment with complex changes to content item functionality of the content item undergoing experimentation.

Environment100includes client devices102(e.g., client device102A and client device102B). Client device102A may represent one or more devices of a first user and client device102B may represent one or more devices of a second user. Client devices102are connected to an experiment definition system105and a third-party content provider107via a network109. Although the environment100shown inFIG. 1only includes two client devices102, the environment100can include any number of client devices (e.g., thousands of client devices102).

FIG. 1and the other figures use like reference numerals to identify like elements. A letter after a reference numeral, such as “102A,” indicates that the text refers specifically to the element having that particular reference numeral. A reference numeral in the text without a following letter, such as “102,” refers to any or all of the elements in the figures bearing that reference numeral.

Network109enables communication among the entities connected to it. In one embodiment, network109is the Internet and uses standard communications technologies and/or protocols. Thus, network109can include links using technologies such as Ethernet, 802.11 (WiFi), worldwide interoperability for microwave access (WiMAX), 3G, Long Term Evolution (LTE), digital subscriber line (DSL), asynchronous transfer mode (ATM), InfiniBand, etc. Similarly, the networking protocols used on network109can include multiprotocol label switching (MPLS), the transmission control protocol/Internet protocol (TCP/IP), the User Datagram Protocol (UDP), the hypertext transport protocol (HTTP), the simple mail transfer protocol (SMTP), the file transfer protocol (FTP), etc. The data exchanged over the network109can be represented using technologies and/or formats including the hypertext markup language (HTML), the extensible markup language (XML), etc. In addition, all or some of links can be encrypted using conventional encryption technologies such as the secure sockets layer (SSL), transport layer security (TLS), virtual private networks (VPNs), Internet Protocol security (IPsec), etc. In one embodiment, network109may include but is not limited to any combination of a local area network (LAN), a metropolitan area network (MAN), and a wide area network (WAN). In another embodiment, the entities use custom and/or dedicated data communications technologies instead of, or in addition to, the ones described above.

Client devices102communicate with the third-party content provider107via the network109to receive content items from the third-party content provider107. In one embodiment, each client device102is a computer system capable of communicating with the experiment definition system105and the third-party content provider107. Examples of client devices102include a personal computer, a mobile phone, a tablet, a personal digital assistant (PDA), a laptop, a mobile application, a television, an Internet-of-Things (IoT) application. As shown inFIG. 1, each client device102includes a web browser103(e.g., web browser103A and web browser103B) in one embodiment. Web browser103is a computer program stored on a client device102that allows the user of client102to access web pages on the World Wide Web such as web pages provided by the third-party content provider107. Suitable web browsers include, but are not limited to, GOOGLE CHROME, MICROSOFT INTERNET EXPLORER, MICROSOFT EDGE, MOZILLA FIREFOX, and APPLE SAFARI.

As mentioned above, the environment100includes a third-party content provider107. Hereinafter, the third-party content provider107is referred to as a “content provider107” for ease of description. In one embodiment, the content provider107is an entity such as a company. However, the content provider107can be an individual or a group of individuals. Generally, the content provider107provides content items to client devices102. While only one content provider107is shown, it is understood that any number of content providers are supported and can be in the environment100at any time.

In one embodiment, the content items provided by the content provider107include web pages, mobile applications, video, audio, a combination of video and audio, still images (e.g., JPEG), text documents, advertisements, or any type of content item that requires backend logic to implement functionality of the content item. The embodiments described herein can be applied to any type of content item. However, the present disclosure is described with respect to a web page provided by the content provider107that is undergoing a variation experiment, but is not limited to web page experimentation.

FIG. 2illustrates a user interface200of a web page provided by content provider107where the web page is undergoing a variation experiment. The user interface200includes a search result of shoes that is provided to a client device102in response to a user search request for “basketball shoes.” For purposes of convenience and the description of one embodiment, the content items provided by the content provider107will be referred to as web pages, but no limitation on the type of content items are intended by this terminology.

Referring back toFIG. 1, generally the content provider107operates in conjunction with the experiment definition system105to perform variation experiments on web pages provided by the content provided107. The experiment definition system105defines variation experiments for the content provider107where the variation experiment for a web page tests different changes to the web page. By testing the different variations of web page, the content provider107determines whether the changes implemented in the variations produce a desired result. Thus, the variation experiment validates any changes made to the web page.

In one embodiment, the specific content included in a web page provided by the content provider107is determined using an algorithm implemented at the content provider107. That is, the content provider107may utilize an algorithm to determine what content to include in a web page that is requested by a client device102. One example of a variation experiment tests how changes to the underlying algorithms operating on the web page affect the goals of the experiment. For example, the content provider107may test different ranking criteria used to rank the search results of shoes shown inFIG. 2to determine which ranking criteria led to the desired result of an increased selection of the “shop now” buttons201. Alternatively, changes to a web page may be aesthetic changes such as changes to the layout of different elements displayed on the web page.

In one embodiment, the experiment definition system105includes a definition module111. As is known in the art, the term “module” refers to computer program logic utilized to provide the specified functionality. Thus, a module can be implemented in hardware, firmware, and/or software. In one embodiment, program modules are stored on a non-transitory storage device (i.e., a computer program product), loaded into a memory, and executed by one or more computer processors. Additionally, those of skill in the art will recognize that other embodiments of the experiment definition system105and content provider107shown inFIG. 1can have different and/or other modules than the ones described here, and that the functionalities can be distributed among the modules in a different manner.

The definition module111establishes a definition for a variation experiment of a web page based on attributes provided by the content provider107. In one embodiment, the definition of a variation experiment includes at least the following attributes provided by the content provider107:Experiment key: an identifier for the variation experiment;Variation key: an identifier for a variation of the web page being tested;Event key: a conversion metric being optimized; andTraffic allocation: percentage of user traffic that are assigned to the different variations of the variation experiment.

The definition module111receives the definition for a variation experiment of a web page that the content provider107wishes to test via user interfaces provided by the experiment definition system105.FIGS. 3-7describe different user interfaces provided by the definition module111for defining a variation experiment of a web page as described below.

FIG. 3illustrates a user interface300provided by the definition module111to create a project. User interface300includes various project types from which a content provider107can select from for the content provider's variation experiment. In one embodiment, the project types are classified into projects that include conventional client-side variation experiments or server-side variation experiments.

Examples of conventional client-side variation experiments where experiments are run at the client device102include web projects301where the variation experiment is optimized for a website using JavaScript. A server-side variation experiment in contrast is a full stack project303. In one embodiment, a full stack project is a custom project that includes customized code that offers developer software development kits (SDKs) allowing a content provider107to conduct a variation experiment at the content provider107while tracking events at the experiment definition system105. Generally, full stack projects303are platform-agnostic in contrast to the other types of projects.

If the full stack project303is used for the variation experiment, the content provider107selects the primary language that will be used for the experiment code blocks for the web page. For example, the content provider107may select from various programming languages that will be used to implement the web page including Java, JavaScript, Node, Python, and Ruby. The variation experiment for the web page may be implemented in other programming languages than those shown inFIG. 3such as PHP, CXX, and ROOT.

In the description herein, the example variation experiment tests different ranking criteria of the results of the web page200shown inFIG. 2. Once the content provider107selects the programming language that will be used to implement the web page, the definition module111receives the type of project selected by the content provider107. In the example herein, the project being created is a full stack project307that implements a content provider-side experiment.

Responsive to receiving a submission of the selected project type (e.g., a full stack project), the definition module111provides a user interface400for creating the variation experiment for the project as shown inFIG. 4A. The user interface400includes a UI element401that sends a request to the definition module111to create a new experiment responsive to the content provider107selecting the UI element401.

In response to receiving the request to create the new experiment, the definition module111provides a screen403for the content provider107to edit the experiment as shown inFIG. 4B. For example, the content provider107defines the experiment key for the variation experiment as shown inFIG. 4B. The experiment key is an identifier for the variation experiment that the content provider107will include in the source code of the web page undergoing the experiment. In the example shown inFIG. 4B, the experiment key is “SHOE_SORT_EXPERIMENT.” The content provider107may optionally include a description of the variation experiment such as the description “Test different sorting of shoes” as shown inFIG. 4B.

The screen403for editing the experiment also allows the content provider107to define one or more variation keys for the variation experiment as shown inFIG. 4B. A variation key is an identifier for a variation of the web page being tested. A variation experiment includes at least one variation and a corresponding variation key for the variation. InFIG. 4B, the content provider107defined the first variation key as “default.” The screen403also includes an area for optionally providing a description of the variation. In this example, the content provider107provided the description that the default variation key tests sorting shoes by category.

The screen403also includes a UI element405for adding an additional variation key for another variation of the web page. Responsive to the selection of the UI element405, the definition module111updates the screen403with text fields where the content provider107provides the variation key and optional description for the additional variations for testing. For example,FIG. 4Billustrates that the variation experiment includes a second variation key for the variation experiment for web page200in addition to the default variation key. The second variation includes a variation key “SORT_BY_PRICE” for ranking the shoes on web page200by price. The content provider107may also remove a variation key from the variation experiment by selecting UI element407.

In one embodiment, the content provider107also defines the traffic allocation for each variation key for the variation experiment. The traffic allocation for each variation key defines the percentage of users that are subject to the variation of the variation experiment using UI elements409(e.g., text fields). Thus, the content provider107may customize the specific distribution of the variations of the web page to the users requesting the web page. For example, the content provider107indicated that 50% of the users receive the “default” variation of the web page and 50% of the users receive the “SORT_BY_PRICE” variation of the web page as shown inFIG. 4B.

After the content provider107has completed editing the experiment as shown inFIG. 4B, the definition module111provides a screen500for defining the events of the experiment as shown inFIG. 5A. An event for the variation experiment indicates at least one metric to track in the variation experiment. In one embodiment, a variation experiment includes at least one event. The content provider107may either create a new event via UI element501.

FIG. 5Billustrates screen503for defining a new event for the variation experiment. The screen503requires the content provider107to provide an event key and an optional description of the event key. As mentioned above, the event key indicates the conversion metric being optimized in the variation experiment. In the example described herein, the event key is “CLICK_SHOP_NOW.” That is, the goal for the experiment is to determine which variation resulted in the increased selection of the “shop now” buttons201shown inFIG. 2. In one embodiment, the definition module111also provides a tracking function (e.g., track( )) used for tracking conversion events that occur on the web page being tested given a user ID. The content provider107includes the tracking function in the source code of the web page after the activate function has been called for the experiment in the source code. For example,FIG. 5Bshows the code block507that includes the tracking function which tracks the number of conversions on the web page that are related to the event key. In the example shown inFIG. 5B, the tracking function tracks the number of selections of the “shop now” button201included in the different variations of the web page200included in the variation experiment. In one embodiment, the definition module111provides the code block507in the programming language selected by the content provider107for the variation experiment.

Responsive to selecting the UI element505, the event key and optional event description is created and screen507shown inFIG. 5Cis displayed that lists the events for the variation experiment. InFIG. 5C, the content provider107may also add an additional event by searching for previously defined events using the search UI509. Thus, the content provider107may apply an event previously defined by the content provider107to the variation experiment.

If no additional events are required, the definition module111provides screen600shown inFIG. 6for defining the traffic allocation for the variation experiment. The traffic allocation describes the percentage of users (e.g., traffic) that are included in the variation experiment. The content provider107may define the traffic allocation by using the text field601. In other embodiments, a slider or other mechanism may be used to specify the traffic allocation for the variation experiment. In the example shown inFIG. 6, 5% of all users requesting the web page200are included in the variation experiment.

Additionally, screen600also illustrates the percentage of users that are subject to each of the different variations of the variation experiment as defined inFIG. 4B. The content provider107may revise the specific distribution of the variations of the web page to the users requesting the web page if needed in screen600using UI elements409. Thus, in the example described herein, 5% of all users that request web page200are included in the variation experiment and of those users included in the variation experiment, 50% of the users receive the “default” variation of the web page and 50% of the users receive the “SORT_BY_PRICE” variation of the web page as shown inFIG. 6.

In one embodiment the definition module111creates a code block701for the content provider107to include in the source code of the web page as shown inFIG. 7. The code block701includes various functions needed to run the variation experiment and track events on the web page undergoing experimentation. For example, the code block701shown inFIG. 7includes an activate function (e.g., activate( )) that is used to allocate a user to a variation of the variation experiment given the user's user identification (ID). In one embodiment, the activate function may also allocate a user to a given variation based on any specified targeting criteria such as demographic information.

As shown inFIG. 7, the content provider107is advantageously provided with code block701created by the definition module111for the variation experiment. The content provider107merely needs to include the code block701in the source code for the web page being tested and add in the variation code for each of the different variations being tested. In this example, the content provider107is required to provide the code that describes what occurs if the user is assigned to the default variation or the variation that sorts by price. In one embodiment, the definition module111provides the code block in the programming language selected by the content provider107inFIG. 3. For example, if the content provider107is implementing the web page in Java, the code block701is provided in Java.

In one embodiment, the definition module111receives optional targeting attributes from the content provider107based on attributes of the users requesting the web page. Attributes can include for example demographic information about the users, interest information about users, geographic information about users, and any type of information related to users. The attributes allow the experiment to targeting a particular audience. For example, the content provider107can determine whether to include or exclude a user from an experiment based on the user's attributes in comparison with the targeting attributes received from the content provider. If a user's attributes match the targeting attributes for the experiment, the content provider107may include the user in the experiment for example.

After reviewing the code block701, the content provider107creates the experiment by selecting UI element703. In one embodiment, the definition module111includes the attributes of the variation experiment in an experiment definition file. The experiment definition file represents all of the “instructions” needed for the content provider107to activate the variation experiment in the source code of the web page undergoing experimentation and track events on the web page. In one embodiment, the experiment definition file is a JavaScript Objection Notation (JSON) representation of the attributes of the variation experiment.

In one embodiment, definition module111sends the experiment definition file for the variation experiment to the associated content provider107. As will be discussed below, the content provider107stores the experiment definition file locally and therefore does not need to communicate with the experiment definition system105when conducting the variation experiment. The definition module111may optionally store the experiment definition file associated with the experiment and the content provider in the enterprise data file database113shown inFIG. 1. The enterprise data file database113stores the experiment definition files of different variation experiments established using the experiment definition system105.

If the variation experiment does not require any further events and the content provider107has established the code for the web page including the code blocks created by the definition module111, the content provider107starts the experiment by selecting UI element703shown inFIG. 7. Once the variation experiment is started, the experiment is live and users requesting the web page are automatically routed to one of the experiments and the events of the experiment are tracked. For example,FIG. 8illustrates that the status of the experiment is “running” indicating that the experiment is live. The experiment can also be paused by the content provider107at any time.

Referring back toFIG. 1, the experiment definition system105also includes a statistics module115. The statistics module115calculates statistics regarding the events of the variation experiment. The statistics module115receives from the content provider107events that occur on the web page being tested as described in the experiment definition file for the experiment and the statistics module115calculates statistics related to the received events. Events can be any type of interaction with the web page such as merely providing the web page to unique viewers or any interactions with elements related to the events of the experiment that are being tracked on the web page such as the “shop now” button inFIG. 2.

In one embodiment, the statistics module115receives events from the content provider107in real time as the events are received on the web page if network connectivity to the content provider107is available. Alternatively, the statistics module115periodically receives events in a batch of events if the experiment definition system105has network connectivity to the content provider107. If the experiment definition system105lacks network connectivity to the content provider107, the experiment definition system105receives the events at a later time when network connectivity is available.

FIG. 9illustrates a UI900of the performance summary of the variation experiment on web page200that is provided to the content provider107via the experiment definition system105. The performance summary provides a summary of the statistics of the variation experiment calculated by the statistics module115. For example, the performance summary indicates the number of unique visitors that received the web page200, the number of visitors directed to each of the different variations of the web page200, and the percentage of conversions (e.g., selection of the “shop now” button) that occurred on each variation of the web page.

Referring back toFIG. 1, in one embodiment the content provider107includes a web server117, an experiment module119, a reporting module121, and a data file database123. The data file123database117stores experiment definition files of the content provider's variation experiments defined on the experiment definition system105. The content provider107receives an experiment definition file for a variation experiment from the experiment definition system105after the content provider107defines the variation experiment on the experiment definition system105as described above.

In one embodiment, the web server117links the content provider107to one or more client devices102and the experiment definition system10105via the network109. The web server117serves content items such as web pages, as well as other web related content, such as Java, Flash, XML, and so forth. The web server117may include a mail server or other messaging functionality for receiving and routing messages between the experiment definition system105and one or more client devices102.

The experiment module119conducts a variation experiment on a web page hosted by the content provider107according to the associated experiment definition file stored in the data file database123. The experiment module119receives requests for a web page that is undergoing an experiment from client devices102. The experiment module119determines whether to include a user requesting the web page in the variation experiment based on the traffic allocation specified in the experiment definition file for the experiment. In the example variation experiment discussed herein, the traffic allocation indicates to include all users requesting the web page in the variation experiment. Thus, the experiment module119includes any user requesting the web page in the experiment.

The experiment module119also determines which variation to provide to the client device102of the user. In one embodiment, the experiment module119includes a SDK received from the experiment definition system105that is installed at the content provider107. The SDK includes the code for the activation function and tracking function.

As discussed earlier, the source code for the web page being tested includes a call to the activate function which allocates a user to a variation based on a given user ID. In one embodiment, the experiment module119randomly assigns the user to one of the variations of the web page using the SDK while satisfying the specified allocation of the number of users that should be directed to each variation as defined in the experiment definition file. For example, the experiment definition file for the shoe demo indicates that 33.33% of all users should be directed to the default variation, 33.33% of all users should be directed to the “SORT_BY_PRICE” variation, and 33.33% of all users should be directed to the “SORT_BY_REVIEWS” variation of the web page. The experiment module119may randomly assign the user to one the variations while satisfying the specified allocation of users to each variation. The experiment module119then provides the determined version of the web page to the client device119and tracks events related to the web page according to the experiment definition file.

Since the experiment definition file for the variation experiment is stored at the content provider107, the content provider107does not need to make a network request to the experiment definition system105to determine which variation of the web page to provide to the client device102. Given that the content provider107does not need to communicate with the experiment, the content provider107beneficially minimizes application latency, which could translate to web page load time in the web page example described herein. Furthermore, the content provider107can decide which variations to provide to client devices102as soon as the requests for web pages are received by the content provider107thereby reducing latency. For example, 10 ms is saved each time a user requests a web page undergoing an experiment since the content provider107does not need to send a network request to the experiment definition system105to determine which variation of the web page to provide to the user.

In one embodiment, the experiment module119tracks events on the web page provided by the content provider107. The events describe various actions related to the web page such as each request for the web page from a unique visitor. The events may also describe user interactions with the web page. Specifically, the experiment module119tracks events describing user interactions with objects on the web page that are related to the goals of the experiment. For example, the experiment module119tracks the number of selections of the “shop now” UI element inFIG. 2.

Furthermore, the experiment module119may revise the attributes of a variation experiment in the experiment definition file stored in the data file database123. Since the experiment definition file for a variation experiment is stored at the content provider107, the content provider107is given the flexibility to easily change any of the attributes of the variation experiment without having to communicate with the experiment definition system105. The content provider119merely makes the revision to the source code of the web page and any corresponding revisions to the experiment definition file for the variation experiment. For example, if the content provider109wants to change the variation experiment to sort the results based on shoe color rather than sort by reviews of shoes, the content provider109updates the source code for the web page with the code for the revised sorting algorithm and revises the variation key and traffic allocation according to the change. Thus, the content provider107is allowed to test complex changes that occur at the content provider107which would be impossible in conventional client-side variation testing.

In one embodiment, the reporting module121reports events related to variation experiments to the experiment definition system105. The reporting module121may send the events to the experiment definition system105in real time as the events are received on the web page or periodically in a batch of events if the content provider107has network connectivity to the experiment definition system105. If the content provider107lacks network connectivity to the experiment definition system105, the reporting module121sends the events to the experiment definition system105at a later time when network connectivity is available.

Although the description herein is applied to a variation experiment applied to a web page, the embodiments of the present disclosure are applicable to any type of content items where it is desirable to test back end logic or algorithms. For example, a content item may be a mobile application that ranks search results of automobiles for sale. The content provider107may have different variations of the mobile application that each operate using a different algorithm to rank automobiles based on different ranking criteria. The embodiments herein may be used to test which of the different variations of the mobile application produced desired results.

In another example, the content item may be a digital media player provider by the content provider107. The digital media player may stream content to users such as videos, television, movies, etc. The content provider107may provide different variations of the digital media player that each implement a different algorithm used to determine which content to provide to users for example. The embodiments herein may be used to test which of the different variations of the digital media player produced the desired results.

Thus, a variation experiment can be performed on any type of content item using the embodiments herein where the definition of the variation experiment is stored by the content provider107that provides the content item. By storing the definition at the content provider107, the content provider107beneficially minimizes application latency since the content provider107does not need to send a network request to the experiment definition system to determine which variation of the content item to provide to a client device requesting the content item.

Providing Variations of Web Pages

FIG. 10is an interaction diagram of a process for providing a variation of a web page to a client device102A according to one embodiment. The interaction diagram illustrates the steps performed by client device102, content provider107, and experiment definition system105according to one embodiment.

In one embodiment, the content provider107transmits1001a request to create a content provider-side variation experiment to the experiment definition system105. The experiment definition system105creates1003the content provider-side variation experiment as described above. For example, the experiment definition system105creates an experiment definition file for the variation experiment that includes the attributes of the experiment provided by the content provider107. The experiment definition system sends1005the experiment definition file for the content provider-side variation experiment to the content provider107. The content provider107stores1007the experiment definition file for the content provider-side variation experiment.

The content provider107receives1009a request for a web page undergoing the experiment from the client device102. The content provider107determines1011the version of the web page to present to the client device102based on the experiment definition file. Since the experiment definition file is stored at the content provider107, the content provider107does not need to make a network request to the experiment definition system105to determine which variation to provide to the client device102. Rather, the content provider107itself can determine which variation of the web page to provide to the client device102.

The content provider107provides1013the determined variation of the web page to the client device102. The client device102interacts1015with the web page and the interactions are sent to the content provider107. For example, the client device102selects the “shop now” UI element on web page200. The content provider107receives the interaction with the shop now UI element and transmits1017the interactions with the web page to the experiment definition system105. The experiment definition system105calculates1019statistics based on the interactions received from the content provider107. The content provider107can view the calculated statistics via a user interface provided by the experiment definition system105.

Additional Considerations

The foregoing described embodiments have been presented for the purpose of illustration; they are not intended to be exhaustive or to limiting to the precise forms disclosed. Persons skilled in the relevant art can appreciate that many modifications and variations are possible in light of the above disclosure.

Some portions of this description describe the embodiments in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are commonly used by those skilled in the data processing arts to convey the substance of their work effectively to others skilled in the art. These operations, while described functionally, computationally, or logically, are understood to be implemented by computer programs or equivalent electrical circuits, microcode, or the like. Furthermore, described modules may be embodied in software, firmware, hardware, or any combinations thereof.

Embodiments of the invention may also relate to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, and/or it may include one or more general-purpose computing devices selectively activated or reconfigured by one or more stored computer programs. A computer program may be stored in a non-transitory, tangible computer readable storage medium, or any type of media suitable for storing electronic instructions, which may be coupled to a computer system bus. Furthermore, any computing systems referred to in the specification may include a single processor or may be architectures employing multiple processor designs for increased computing capability.