Assigning variants of content to users while maintaining a stable experimental population

A variation testing system environment for performing variation testing of web pages and applications is disclosed. The variation testing system applies a weighted consistent hash function to user attributes of users to assign the users to a variant of a web page that is undergoing experimentation. The usage of the weighted consistent hash function allows for a stable experimental population.

FIELD OF TECHNOLOGY

The embodiments described herein pertain in general to conducting experiments on content and more particularly to determining which variant of content that is undergoing an experiment to display to users in a manner that maintains a stable experiment population.

BACKGROUND

Content variation testing is an experiment that tests changes to content against other variants of the content. One example of content variation testing is an experiment that tests changes to a web page against one or more different changes in other variants of the web page. During the experiment, changes to a web page that produce positive results are determined. The positive results associated with a particular change to a web page that is included in a variant validates which variant of the web page to use in a production version of the web page, for example. However, conventional variation testing is unable to maintain a consistent distribution of users across different variants when changes to an experiment occur such as changing of variant weights, adding variants, or removing variants from the experiment.

BRIEF SUMMARY

The embodiments herein describe a variation testing system environment for performing variation testing of content such as a web page. Users issues requests for a web page from a content provider where the web page is undergoing experimentation. The experiment for the web page may include multiple variants of the web page that are provided to the different users that request the web page. Each variant of the web page may have a different version of a particular feature of the web page. In one embodiment, the content provider uses a hashing function such as a weighted consistent hash function to assign the users to one of the variants of the experiment. User interactions with the different variants of the web page are monitored according to one or more performance metrics to determine which variant is “winning” the experiment.

Any subsequent requests by the users to view the web page again results in the content provider either maintaining the previously assigned variant for each user or reassigning the user to a different variant. The content provider may provide the previously assigned variant to a user if that user was previously provided the variant that is “winning” the experiment. However, the content provider may reassign the user to the “winning” variant if the user was previously assigned to a “losing” variant in the experiment. In one embodiment, the weighted consistent hash function used by the content provider inherently handles the re-distribution of users responsive to subsequent requests for the content from the users. Using the weighted consistent hash function to assign users to variants allows for a stable population of users in the experiment even if changes to the experiment occur such as changing of variant weights or removing variants from the experiment.

DETAILED DESCRIPTION

FIG. 1is a block diagram of a variant testing system environment100according to one embodiment. Environment100includes client devices102(e.g., client device102A and client device102B). In one embodiment, client device102A is a device of a first user and client device102B is a device of a second user. Client device102A may represent one or more devices of the first user and client device102B may represent one or more devices of the second user. Client devices102are connected to a content provider system101via a network109. Although the environment100shown inFIG. 1only includes two client devices102, the environment100can include any number of client devices (e.g., thousands of client devices102).

FIG. 1and the other figures use like reference numerals to identify like elements. A letter after a reference numeral, such as “102A,” indicates that the text refers specifically to the element having that particular reference numeral. A reference numeral in the text without a following letter, such as “102,” refers to any or all of the elements in the figures bearing that reference numeral.

Network109enables communication among the entities connected to it. In one embodiment, network109is the Internet and uses standard communications technologies and/or protocols. Thus, network109can include links using technologies such as Ethernet, 802.11 (WiFi), worldwide interoperability for microwave access (WiMAX), 3G, 4G, 5G, Long Term Evolution (LTE), digital subscriber line (DSL), asynchronous transfer mode (ATM), InfiniBand, etc. Similarly, the networking protocols used on network109can include multiprotocol label switching (MPLS), the transmission control protocol/Internet protocol (TCP/IP), the User Datagram Protocol (UDP), the hypertext transport protocol (HTTP), the simple mail transfer protocol (SMTP), the file transfer protocol (FTP), etc. The data exchanged over the network109can be represented using technologies and/or formats including the hypertext markup language (HTML), the extensible markup language (XML), etc. In addition, all or some of links can be encrypted using conventional encryption technologies such as the secure sockets layer (SSL), transport layer security (TLS), virtual private networks (VPNs), Internet Protocol security (IPsec), etc. In one embodiment, network109may include but is not limited to any combination of a local area network (LAN), a metropolitan area network (MAN), and a wide area network (WAN). In another embodiment, the entities use custom and/or dedicated data communications technologies instead of, or in addition to, the ones described above.

Client devices102communicate with the content provider101via the network109to receive content items such as web pages from the content provider101. In one embodiment, each client device102is a computer system capable of communicating with the content provider101. Examples of client devices102include a personal computer, a mobile phone, a tablet, a personal digital assistant (PDA), or a laptop. As shown inFIG. 1, each client device102includes a web browser103(e.g., web browser103A and web browser103B). Web browser103is a computer program stored on a client device102that allows the user of client102to access web pages on the World Wide Web such as web pages provided by the content provider101. Suitable web browsers include, but are not limited to, GOOGLE CHROME, MICROSOFT INTERNET EXPLORER, MICROSOFT EDGE, MOZILLA FIREFOX, and APPLE SAFARI.

In one embodiment, the content provider101may be an individual, a group of individuals, or an entity such as a company. The content provider101provides content items to client devices102. While only one content provider101is shown, it is understood that any number of content providers are supported and can be in the environment100at any time.

In one embodiment, the content items provided by the content provider101include web pages. However, the content provider101may provide other types of content items such as video, audio, a combination of video and audio, still images (e.g., JPEG), text documents, advertisements, and/or any other types of content. For purposes of convenience and the description of one embodiment, the content items provided by the content provider101will be referred to as a web pages, but no limitation on the type of content items are intended by this terminology.

In one embodiment, the content provider101performs experimentation on web pages provided by the content provider101. The content provider101may display different variants of a web page to client devices102according to an experiment (e.g., a variation test) being performed on the web page by the content provider101. An experiment for a web page tests changes to one or more features of a web page included in multiple variants of the web page (e.g., the current version of the web page and different variants) to determine whether the changes to the one or more features produce a desired result in a performance metric under evaluation. An example of a performance metric under evaluation is increased conversions (e.g., purchases) of a product advertised on a web page resulting from a change to a feature of a web page. While conversions are one type of performance metric that may be monitored during an experiment, the experiment may be targeted towards any type of performance metric.

Thus, experiments on features of web pages performed by the content provider101validates a new design of a web page or changes on features on the web page before the new design or changes are put into production by the content provider101such as during a gradual roll out. Experiments performed by the content provider101also allow for rollout of the features, a holdback group of users in order to evaluate long-term effects of the features, and multi-armed bandit situations.

In one embodiment, a rollout is when the proportion of users that are exposed to variants is expanded while keeping the population in each variant of the web page as stable as possible. For example, an experiment may have an initial distribution of users to Variants A, B, and C of 10%, 10%, and 80%, respectively. Over time, the distribution of users to Variants A, B, and C may be changed to 30%, 30%, and 40%, respectively.

A holdback group may be implemented when a winner of the experiment is declared, the feature is launched, and a small proportion of the control is kept to understand long-term impacts of the feature corresponding to the winning variant. Consider the example where Variant A is the control and Variants B and C are variants that are being tested. The experiment may have an allocation of 40%, 30%, and 30% to Variants A to C, respectively. If B is declared the winner of the experiment, a new distribution of users to Variants A to C is established such as 5%, 95%, and 0%, respectively. A small percentage of users that are still assigned to the control (e.g., Variant A) is a holdback group and is kept for a predetermined period of time (e.g., 2 months) to understand the longer term impact of declaring Variant B as the winner of the experiment.

Multi-armed bandits and other adaptive sampling methods may vary the proportion of variants assigned or even add or subtract variants. By modifying the proportion of the variants, the value of the feature (e.g. maximize the CTR) may be maximized or the leanings (minimize the time to find the best variant) in the experiments may be maximized.

For a given web page, the content provider101may store one or more variants of the web page that are used in an experiment for the web page. In one embodiment, a variation test of a web page involves an experiment that tests an “A” variant known as the control, a “B” variant known as a first variant, and a “C” variant known a second variant. For ease of discussion, the embodiments discussed herein describe a web page having only three variants: the control and two variants. However, in other embodiments, a web page can have any number of variants.

FIG. 2Ais an example of the “A” variant (e.g., the control variant) of a web page200A that represents the current implementation of the web page provided by content provider101. The control variant of the web page200A includes a plurality of features including a picture of a product201being sold on the web page200A, a description of the product203, a price205of the product, a quantity207of the product for purchase, a purchase mechanism209A, and a plurality of related products211A to211D. Any of the features included in the web page200A may be changed in one or more variants during an experiment. In the example described herein, the experiment performed on the web page200tests different purchase mechanisms209to see which is more effective in increasing conversions of the product on the web page200. In the “A” variant shown inFIG. 2A, the purchase mechanism209A includes the label “+Add to cart”.

FIG. 2Bis an example of a “B” variant of a web page200B. Web page200B includes a change (i.e., a modification) to the feature being evaluated. As mentioned above, the purchase mechanism209is under evaluation while all other features of the web page remain consistent across the different versions of the web page200. For example, the picture of the product201being sold on the web page200A, the description of the product203, the price205of the product, the quantity207of the product for purchase, and the plurality of related products211A to211D are the same across web page200A to web page200C shown inFIGS. 2A to 2C. However, the purchase mechanism209is different across the different versions of the web page200. InFIG. 2B, the purchase mechanism209B includes the label “+Buy It!” rather than the label “+Add to cart” inFIG. 2A.

FIG. 2Cis an example of a “C” variant of a web page200C. Web page200C includes yet another change (i.e., a modification) to the purchase mechanism209while the remaining features of the web page200C are the same as in web pages200A and200B. In web page200C shown inFIG. 2C, the purchase mechanism209B includes the label “+Purchase”. In example shown inFIGS. 2A to 2C, the experiment determines which of the labels for the purchase mechanism209result in a higher rate of purchase conversions of the product advertised on the web page200. The variant with highest rate of purchase conversions may be declared the “winner” of the experiment by the content provider101.

Referring back toFIG. 1, in one embodiment the content provider101includes a web server105, an identification (ID) module107, a variation selection module111, a result module117, an ID database113, and an experiment database115. As is known in the art, the term “module” refers to computer program logic utilized to provide the specified functionality. Thus, a module can be implemented in hardware, firmware, and/or software. In one embodiment, program modules are stored on a non-transitory storage device (i.e., a computer program product), loaded into a memory, and executed by one or more computer processors. Additionally, those of skill in the art will recognize that other embodiments of the content provider101shown inFIG. 1can have different and/or other modules than the ones described here, and that the functionalities can be distributed among the modules in a different manner.

In one embodiment, the experiment database115stores multiple (i.e., a plurality) experiment records that describe experiments conducted by the content provider101. Each experiment record is associated with a web page, an experiment identifier unique to the experiment, and identifiers of different variants of a web page included in the experiment. The experiment identifier may be a numerical identifier or an alphanumeric identifier that describes the name of the feature being tested. An experiment record may also include a description of the change(s) made to the feature(s) of the web page in each variant.

In one embodiment, each experiment stored in the experiment database115comprises an associated rule set. A rule set for an experiment includes one or more rules that determine if a feature should be enabled for a user of the client device102that requested the web page and if so, which variant should be presented to the user. In one embodiment, each rule defines a criteria about the user that must be valid in order to satisfy the rule so that the user is assigned the variant associated with the rule. In one embodiment, all the rules of a rule set must be satisfied for a user to be targeted.

In one embodiment, a rule included in a rule set may define any type of criteria that must be satisfied in order for a user to be assigned a particular variant of a web page. The criteria for a rule may be based on a user attribute such as the user's user identifier (ID), location (e.g., zip code), type of device platform associated with the user's client device102(e.g., iOS or ANDROID). For example, only users in California that are using iOS devices may be included in an experiment of a web page.

In one embodiment, a rule may be a static rule that indicates a large set of explicitly set values which determine which variant a user is assigned to. For example, a static rule may specify that a first threshold of users (e.g., first 1,000 users) that are exposed to the web page receive a first variant of a web page, the second threshold of users (e.g., second 1,000 users) that are exposed to the web page receive a second variant of the web page, and the third threshold of users (e.g., third 1,000 users) that are exposed to the web page receive the third variant of the web page.

In one embodiment, a rule for an experiment may define an exposure rate of users to the experiment. The exposure rate indicates a percentage of users that are exposed to the experiment. In one embodiment, all users may be subject to an experiment (e.g., 100% exposure) or only a percentage of all the users are subject to the experiment. For example, if the exposure rate is 20%, 20% of all users that request a web page that is under experimentation will be assigned to one of the variants of the experiment.

A rule for the experiment may also define weights for each variant included in the experiment. The weight assigned to each variant impacts the distribution of users to each variant. For example, a rule may specify a distribution split between the different variants. The distribution split may be different for the different variants of an experiment. For example, a rule may specify a distribution split of 20% for variant “A”, 30% for variation “B”, and 50% for variant “C” in the examples shown inFIGS. 2A to 2C. In other embodiments, the distribution split may the same for the different variants. For example, the rule may specify a distribution split of 33.3% for variant “A”, 33.3% for variation “B”, and 33.4% for variant “C”.

In one embodiment, the web server105links the content provider101to one or more client devices102via the network109. The web server105serves content items such as web pages, as well as other web related content. The web server111may include a mail server or other messaging functionality for receiving and routing messages between the variation determination system105and one or more client devices102.

The ID module107generates user identifiers for users of client devices102. In one embodiment, the user identifier is used by the user of a client device102to log into a web page or application provided by the content provider107. In one embodiment, a user identifier is a string of numbers that is unique to a particular client device102. Thus, client device102A may have a different user identifier than the user identifier for client device102B.

In one embodiment, the ID module107generates a user identifier by automatically incrementing the value of the last user identifier generated by the ID module113using an auto increment function. Alternatively, the ID module107generates a unique random number and assigns the unique random number as a user identifier for a client device102. In another embodiment, the ID module107may receive a user identifier from a client device102. The user identifier received from the client device102may be generated by the client device102or the user identifier may be input by the user into the client device102. The ID module107stores user identifiers in the ID database113that maintains a record of all assigned user identifiers.

In one embodiment, responsive to the content provider107receiving a request for a web page from a client device, the ID module107communicates with the client device102to determine whether the client device102already has a user identifier. For example, the ID module107may query the client device102to transmit the user identifier via a login form displayed on the web page. In another example, the ID module107queries the client device102to determine whether the client device102includes a stored cookie that comprises a user identifier for the client device102. If the client device102lacks a cookie with a user identifier for the client device102, the ID module107may query the client device102to transmit the user identifier via a login form displayed on the web page. Alternatively, the ID module107generates a user identifier for the client device102and communicates the user identifier to the client device102for storage in a cookie at the client device102if the client device102lacks a cookie with a user identifier. The client device102will continue to use the user identifier generated by the ID module107for subsequent requests for the web page.

The variation selection module111selects (e.g., assigns) a variant of a web page to provide to a user of a client device102. In one embodiment, the variation selection module11selects the variant from a plurality of different variants of the web page undergoing an experiment. The variation selection module111may receive client device requests for web pages from the web server105. For each received request, the variation selection module111determines whether the user associated with the request is subject to the experiment and if so, selects a variant for assignment to the user. For example, the variation selection module111determines whether to provide the variant “A”, variant “B”, or variant “C described above.

In one embodiment, the request received by the variation selection module11includes user attributes such as at least a user identifier of the client device102that requested the web page. In addition, the request may also include other user attributes such as the location of the client device (e.g., the zip code) and/or the platform of the client device102(e.g., iOS or ANDROID). The variation selection module111determines which variant of a web page to provide to a client device102based on one or more of the user attributes.

In one embodiment, the variation selection module111selects a variant to provide to a user's client device using a hash function. The variation selection module111applies the hash function to the user identifier to generate a hashed user identifier. The hashed user identifier may be a numeric string. For example, if the user identifier is “367” the application of the function to the user identifier results in the hashed user identifier “12367.” In another embodiment, the variation selection module111applies the hash function to a combination of the user identifier and any other features such as the experiment identifier and/or other user attributes. The combination of the user identifier and the experiment identifier may be a concatenation of the user identifier and the experiment identifier, for example. In one embodiment, the function applied by the variation selection module111to a user identifier is a weighted consistent hash function such as the Weighted Rendezvous Hash function. However, other weighted consistent hash functions may be used.

In one embodiment, the variation selection module111normalizes the hashed user identifier to create an assignment identifier that is the basis for assigning the user to a variation of a web page. Normalizing the hashed user identifier restricts the assignment identifier to a fixed interval such as [0, 1]. To normalize the hashed user identifier, the variation selection module111divides the hashed user identifier by the maximum possible value of the hashed user identifier. Thus, in one embodiment the assignment identifier for the client device102is the quotient of the hashed user identifier and the maximum possible value for the hashed user identifier. In this way, the range of values for the output of the hash function is mapped to the range [0, 1].

In one embodiment, the variation selection module111determines which variant of a web page to provide to a client device102based the assignment identifier for the client device102. In one embodiment, the variation selection module111compares the user attributes and the assignment identifier to a rule set for the experiment of the web page to determine whether the user is subject to the experiment, and if so, selects a variant to present to the user's client device102based on the comparison.

For example, the variation selection module111applies the weighted consistent hash function to the user attribute (e.g., the user identifier and experiment identifier) to generate the assignment identifier. Next the variation selection module111applies the assignment identifier and any user attributes to the rule set for the experiment, as applicable.

Consider the example where the experiment includes a rule specifying that the experiment is applied to 20% of all users. Thus, only 20% of all possible assignable assignment identifiers will be eligible for the experiment (e.g., 1.0 (all users)*0.20 (exposure percentage)=0.2 (assignable assignment identifiers)).

Furthermore, the experiment may also include a variation split of 20% for variant “A”, 30% for variant “B”, and 50% for variant “C”. The variation selection module111determines bounds of assignment identifiers which are used to assign a user to one of the variants. For example, the variation selection module111may determine the following bounds for each variant in the experiment from the rule set for the experiment:

B: 0.04<X<=0.10 (e.g., middle bound between A and C); and

In the mapping above, X is the assignment identifier of the user. The rule set for the experiment may define a mapping between a set of different ranges of assignment identifiers and a corresponding set of variants of the web page. This mapping may be implemented by, for example, a table of different ranges of assignment identifiers and corresponding variations of the web page wherein, the variation selection module111selects from the table of assignment identifiers a variant of the web page that corresponds to the assignment identifier for the client device102.

For example, the rule set above may describe a first range of assignment identifiers that is associated with variant A of the web page, a second range of assignment identifiers that is associated with variant B of the web page, and a third range of assignment identifiers that is associated with variant C of the web page. The variation selection module111determines whether the assignment identifier is included in the first range of assignment identifiers, the second range of assignment identifiers, or the third range of assignment identifiers. The variation selection module111then selects the variant of the web page associated with the range of assignment identifiers that includes the assignment identifier for the user of the client device102.

Once the variation selection module111determines which variant of the web page should be provided to the client device102, the variation selection module111transmits a notification to the web server105. The notification instructs the web server105to provide a particular variant of the web page to the client device102that requested the web page. In one embodiment, the variation selection module111may also store in the experiment database which variant was assigned to each user to track the history of assignments.

In one embodiment, the variation selection module111receives a subsequent request for a web page from a client device102. The variation selection module111has already previously assigned a variant of the web page to the client device102. The variation selection module11determines whether to maintain the previously assigned variant or re-assign another variant of the web page to the client device102.

To determine whether to maintain the previously assigned variant ore re-assign the variation, the variation selection module111may determine a performance metric of each of the different variants responsive to the request. The performance metric may be calculated by the results module117and provided to the variation selection module111as will be further described below. The variation selection module111selects either the previously assigned variant of the web page or another variant of the web page to provide to the client device102based on the performance metrics of the variants of the web page. For example, if the previously assigned variant is the variant with the “best” performance metric then the assignment is maintained. However, if another one of the variants of the web page has the “best” performance metric, the variation selection module111re-assigns the variant with the “best” performance metric to the client device102.

In one embodiment, the result module117tracks data describing user interactions of web pages by users of client devices102. Interactions include conversions that occur on each of the different variants. The result module117determines from the data the results of the experiment of the web page based on the interactions. The interactions are indicative of a performance metric for each variant. For example, the result module117determines based on the interaction data that a higher percentage of conversions (e.g., a performance metric) occurred in variant C compared to variants A and B. Thus, based on the results of the experiment, the content provider101may promote variant C of the web page as the control version of the web page thereby replacing the original control version of the web page.

Generally, an experiment is ended when it has been conducted for a long enough period of time such that a large enough sample size is acquired to achieve a predetermined level of statistical power. As mentioned above, a weighted consistent hash function may be used to assign users to variants. The weighted consistent hash function allows the use of standard statistical methods such as sequential statistical methods to decide when to end an experiment.

In one embodiment, the result module117may determine the results of the experiment based on various criteria. For example, the result module117may determine the results after a threshold amount of time has passed since the experiment was started (e.g., 3 months). In another example, the result module117may determine the results after a threshold number of users were exposed to the experiment (e.g., 1000 users). In yet another example, the result module117may determine the results when one or more metrics of interest show a statistically significant improvement and counter metrics show no degradation. The result module117may also end an experiment at any time due to operational or business concerns. However, any types of criteria may be used to end an experiment.

As mentioned above, the variation selection module111applies a weighted consistent hash function such as the Weighted Rendezvous Hash function to a user identifier to generate the assignment identifier. Using the weighted consistent hash function beneficially maintains a stable experimental population. Even when the variant weights change or a variant is removed from an experiment, the users that are assigned to the variant that is “winning” the experiment variant (e.g., has more conversions) remain assigned to the variant and any subsequent requests from users that were previously assigned to non-winning variants are subsequently re-assigned to the “winning” variant due to the usage of the weighted consistent hash function as is illustrated inFIGS. 3A to 3C.

As described with respect toFIGS. 2A to 2C, an experiment on a web page that is testing different versions of the purchase mechanism may include three different web page variations: variant A, variant B, and variant C. The variation distribution defined by the rule set for the experiment may specify that 20% of users receive variant A, 30% of users receive variant B, and 50% of users receive variant C.FIG. 3Aillustrates the above distribution where a total of ten users have been assigned to the experiment. As shown inFIG. 3A, two of the ten users (e.g., users U1and U2) are initially assigned to variant A, three of the ten users (e.g., users U3to U5) are initially assigned to variant B, and five of the ten users (e.g., users U6to U10) are initially assigned to variant C.

FIG. 3Billustrates the re-assignment of a user to a “winning” variant. A user U1may have been initially assigned to variant A and may subsequently request to receive the web page at a later instance in time. The variation selection module111receives the request from the user U1and may re-assign the user to the “winning” variant which is variant C in this example as a result of the application of the weighted consistent hash function to the assignment identifier of user U1. Variant C may be determined to be the “winning” variant as it has the highest conversion rate for example.

FIG. 3Cillustrates maintaining users that are already assigned to a “winning” variant. A user U6may have been initially assigned to variant C and may subsequently request to receive the web page at a later instance in time. The variation selection module111receives the request from the user U6and may maintain the assignment of user U6to variant C since variant C is the “winning” variant. Thus, the usage of the weighted consistent hash function in the assignment of user's to variants of an experiment allows for implementing certain experiment designs that require changing variant weights such as gradual roll outs, holdback groups for evaluating long-term effects, and multi-armed bandits.

FIG. 4is a block diagram of a variant testing system environment400according to an alternative embodiment. As shown inFIG. 4, environment400includes client devices402in communication with a content provider401via network411. The content provider401includes a web server403, an ID module405, an ID database407, and an experiment database409which perform similar functionality as described with respect toFIG. 1. Thus, the description of similar functionality inFIG. 4is omitted.

However, in contrast to the embodiment shown inFIG. 1, the content provider401lacks functionality for selecting a variant of a web page for display on a client device. Rather, the client devices402include the variation determination functionality of the content provider shown inFIG. 1. Thus, the client devices402themselves determine which variant of a web page to provide on the client device402without consulting the content provider401.

In one embodiment, a client device402includes a variation selection module413, a web browser415, and an experiment database417. The experiment database417may be an exact copy of the experiment database409stored in the content provider401. The content provider401may include the experiment database409merely to maintain a record of the experiments being conducted by the client devices. The components of the client devices402shown inFIG. 4perform similar functionality as their respective components described above with respect toFIG. 1and are omitted for ease of description.

For example, a client device402A may receive a request from a user of the client device402A for a web page from a content provider400. The variation selection module413A of client device402A applies the weighted consistent hash function such as the Weighted Rendezvous Hash function to user attributes of the client device402(e.g., the user identifier of the client device102or a concatenation of the user identifier and experiment identifier (e.g., retrieved from the experiment database417A) of the requested web page to generate an assignment identifier. The variation selection module413A may select which variation of a web page should be displayed on the client device402A according to the assignment identifier and the rule set for the experiment that is stored in the experiment database417A. The client device402A requests the selected variant of the web page from the content provider401and receives the selection variant of the web page from the content provider401.

Providing Variations of Web Pages

FIG. 5is an interaction diagram of a process for providing a variant of a web page to a client device102A according to the embodiment shown inFIG. 1. The interaction diagram illustrates the steps performed by client device102A and content provider101according to one embodiment.

Client device102A transmits a first request501of a web page to the content provider101. In one embodiment, the web page requested by th client device102is undergoing an experiment where there are multiple variants of the web page available where each variant has a different feature of the web page that is best tested. For example, each variant of the web page may have a different purchase mechanism used to purchase a product as described with respect toFIGS. 2A to 2C.

The content provider101determines503an assignment identifier for the client device102A. In one embodiment, the assignment identifier is used by the content provider101to select one of the multiple variants of the web page to provide to the client device102A. The assignment identifier for the client device102A may be determined from the user identifier for example. In other embodiments, the assignment identifier may be determined from any other user attribute such as location, device type of the client device102A used by the user, etc.

The content provider101assigns505a variant from the multiple variants of the web page to the client device102based on the assignment identifier. The content provider101transmits507the assigned variant of the web page to the client device102A based on the assignment and the client device102A displays509the variant of the web page provided by the content provider107.

Subsequently, the client device102A may transmit519a second request for the web page to the content provider101. For example, the user may desire to view the web page again at some later time. The content provider101either maintains521the previously assigned variant or reassigns the user to a different variant of the web page. As mentioned above, the consistent hash algorithm employed by the content provider101to assign users to variants of web pages maintains the initial assignment to a web page if the previously assigned variant of the web page is the “winning” variant in the experiment or will re-assign the user to the “winning” variant if the user was previously assigned to a “non-winning” variant of the web page. By maintaining users that are already assigned to the “winning” variant, the population of the experiment is kept stable regardless of changes made to the experiment such as a modification of weighting applied to the variants of the experiment or removal of a variant from the experiment. The content provider101then transmits the variant523of the web page to the client device102A to fulfill the second request for the web page and the client device102A displays the variant of the web page525.

FIG. 6illustrates one embodiment of a method used by the content provider101to assign505a variant of the web page to the client device102A based on the user identifier of the client device102A. In one embodiment, the content provider101applies601a weighted consistent hash function to at least the user identifier to generate an assignment identifier for the user. An example of the weighted consistent hash function is the Weighted Rendezvous Hash. The content provider101then determines603a variant of the web page that is associated with the assignment identifier. For example, the content provider101compares the assignment identifier for the user to a rule set for the experiment to select one of the variants from the experiment to assign to the user.

FIG. 7is an interaction diagram of a process for providing a variation of a web page to a client device402A according to the embodiment shown inFIG. 4. The interaction diagram illustrates the steps performed by client device402A and content provider401according to one embodiment.

Client device402A receives a request701for a web page from a user of the client device402A. The client device402A determines703an assignment identifier for the user of the client device402A. For example, the client device402A determines the assignment identifier based on the user's user identifier used for logging into the content provider401. The client device402A may apply a weighted consistent hash function to the user identifier to generate the assignment identifier.

The client device402A assigns705a variant of the requested web page to the user. The variant is assigned by comparing the assignment identifier with the rule set of the experiment to determine which variant from the multiple variants to assign to the user. The client device402A transmits707a request for the determined variant of the web page to the content provider401and the content provider401transmits709the requested variant of the web page811to the client device402A. The client device402A displays711the variant of the web page provided by the content provider401.

Subsequently, the client device402A may receive711a second request for the web page from the user. For example, the user may desire see the web page again at some later time. The client device402A either maintains715the previously assigned variant or reassigns the user to a different variant of the web page. As mentioned above, the consistent hash algorithm employed by the client device402A to assign users to variants of web pages maintains the initial assignment to a web page if the previously assigned variant of the web page is the “winning” variant in the experiment or will re-assign the user to the “winning” variant if the user was previously assigned to a “non-winning” variant of the web page. By maintaining the user in the previously assigned “winning” variant, the population of the experiment is kept stable regardless of changes made to the experiment such as a modification of weighting applied to the variants of the experiment or removal of a variant from the experiment. The client device402A transmits717a request for the determined variant of the web page to the content provider401and the content provider401transmits719the requested variant of the web page to the client device402A. The client device402A displays721the variant of the web page provided by the content provider401.

Hardware Components

FIG. 8is a diagram illustrating a computer system800upon which embodiments described herein may be implemented within the content provider system101,401, and client devices102,402. For example, in the context ofFIG. 1, the content provider system101and client devices102may each be implemented using a computer system such as described byFIG. 8. The content provider system101and/or client devices102may also be implemented using a combination of multiple computer systems as described byFIG. 8.

In one implementation, content provider system101and/or client devices102each include processing resources801, main memory803, read only memory (ROM)805, storage device807, and a communication interface809. The content provider system101and/or client devices102include at least one processor801for processing information and a main memory803, such as a random access memory (RAM) or other dynamic storage device, for storing information and instructions to be executed by the processor801. In one embodiment, multiple processors are employed by the content provider system101and/or client devices102to perform the techniques described above in order to improve efficiency of the content provider system101and/or client devices102and reduce computation time when assigning variants. Main memory803also may be used for storing temporary variables or other intermediate information during execution of instructions to be executed by processor801. Content provider system101and/or client devices102may each also include ROM805or other static storage device for storing static information and instructions for processor801. The storage device807, such as a magnetic disk or optical disk or solid state memory device, is provided for storing information and instructions.

The communication interface809can enable each of the content provider system101and/or client devices102to communicate with each other through use of a communication link (wireless or wireline). Each of the content provider system101and/or client devices102can optionally include a display device811, such as a cathode ray tube (CRT), an LCD monitor, an LED monitor, OLED monitor, a TFT display or a television set, for example, for displaying graphics and information to a user. An input mechanism813, such as a keyboard that includes alphanumeric keys and other keys, can optionally be coupled to the computer system800for communicating information and command selections to processor801. Other non-limiting, illustrative examples of input mechanisms813include a mouse, a trackball, touch-sensitive screen, or cursor direction keys for communicating direction information and command selections to processor801and for controlling cursor movement on display device811.

Examples described herein are related to the use of the content provider system101and/or client devices102for implementing the techniques described herein. According to one embodiment, those techniques are performed by each of the content provider system101and/or client devices102in response to processor801executing one or more sequences of one or more instructions contained in main memory803. Such instructions may be read into main memory803from another machine-readable medium, such as storage device807. Execution of the sequences of instructions contained in main memory803causes processor801to perform the process steps described herein. In alternative implementations, hard-wired circuitry may be used in place of or in combination with software instructions to implement examples described herein. Thus, the examples described are not limited to any specific combination of hardware circuitry and software. Furthermore, it has also proven convenient at times, to refer to arrangements of operations as modules, without loss of generality. The described operations and their associated modules may be embodied in software, firmware, hardware, or any combinations thereof.

Reference in the specification to “one embodiment” or to “an embodiment” means that a particular feature, structure, or characteristic is included in at least one embodiment of the disclosure. The appearances of the phrase “in one embodiment” or “a preferred embodiment” in various places in the specification are not necessarily referring to the same embodiment.

Certain aspects disclosed herein include process steps and instructions described herein in the form of a method. It should be noted that the process steps and instructions described herein can be embodied in software, firmware or hardware, and when embodied in software, can be downloaded to reside on and be operated from different platforms used by a variety of operating systems.

While the disclosure has been particularly shown and described with reference to a preferred embodiment and several alternate embodiments, it will be understood by persons skilled in the relevant art that various changes in form and details can be made therein without departing from the spirit and scope of the invention.