Server side experimentation for client side software

A computer-implemented method for comparing multiple client application variations, including, providing a copy of a client application to each of a plurality of client devices, wherein each copy of the client application comprises multiple application variations, receiving demographic information from one or more of the plurality of client devices and providing a list of available experiments to each of the plurality of client devices based on the demographic information, wherein each of the available experiments corresponds with one or more application variations. In certain implementations, the method further includes steps for receiving usage information for each copy of the client application associated with each of the plurality of client devices and comparing two or more application variations of the client application based on the usage information. Systems and computer-readable media are also provided.

BACKGROUND

The disclosed technology provides a method for comparing multiple features across a common application version. Specifically, the instant disclosure provides methods by which multiple variations of a client application can be tested to compare different features, configurations and parameters among the variations.

In conventional software releases, client application binaries are updated and/or replaced in order to add a new feature or to change current configurations or parameters. As such, comparisons between different features can only be made between different releases made at different points in time, making it difficult to register the direct impact any particular changes have had on client application use or user behavior.

SUMMARY

The disclosed subject matter relates to a computer-implemented method for comparing multiple client application variations, comprising, providing a copy of a client application to each of a plurality of client devices, wherein each copy of the client application comprises multiple application variations, receiving demographic information from one or more of the plurality of client devices and providing a list of available experiments to each of the plurality of client devices based on the demographic information, wherein each of the available experiments corresponds with one or more application variations. In certain aspects, the method further comprises steps for receiving usage information for each copy of the client application associated with each of the plurality of client devices and comparing two or more application variations of the client application based on the usage information.

The disclosed subject matter also relates to a system for comparing multiple client application variations, the system comprising, one or more processors and a computer-readable medium comprising instructions stored therein, which when executed by the processors, cause the processors to perform operations comprising providing a copy of a client application to each of a plurality of client devices, wherein each copy of the client application comprises multiple application variations, receiving demographic information from one or more of the plurality of client devices and providing a list of available experiments to each of the plurality of client devices based on the demographic information, wherein each of the available experiments corresponds with one or more application variations. In certain aspects the processors are further configured to perform operations for receiving usage information for each copy of the client application associated with each of the plurality of client devices, comparing two or more application variations of the client application based on the usage information and determining a preferred application variation based on the comparison of the two or more application variations.

In other implementations, the disclosed subject matter relates to a computer-readable storage medium comprising instructions stored therein, which when executed by a processor, cause the processor to perform operations comprising, providing a copy of a client application to each of a plurality of client devices, wherein each copy of the client application comprises multiple application variations, receiving demographic information from one or more of the plurality of client devices, wherein the demographic information comprises information regarding a user's use of the client application associated with the respective client device and providing a list of available experiments to each of the plurality of client devices based on the demographic information, wherein each of the available experiments corresponds with one or more application variations. In certain aspects, the method further comprises steps for receiving usage information for each copy of the client application associated with each of the plurality of client devices and comparing two or more application variations of the client application based on the usage information.

In yet other implementations, the disclosed subject matter also relates to a method for comparing multiple client application variations, comprising receiving, from a remote server, a copy of a client application on a client device, providing demographic information to the remote server, wherein the demographic information comprises one or more of language information or geographic information associated with the client application and receiving a list of available experiments based on the demographic information, wherein each of the available experiments corresponds with one or more application variations. In certain aspects the method further comprises steps for selecting, from among the available experiments, an experiment to be performed with respect to the client application and providing usage information for the client application, based on a user's engagement with the client application.

DETAILED DESCRIPTION

Variance in user use and behavior with respect to a client application can be due to a multitude of factors, both related and unrelated to the features and configuration provided by the client application. For example, use of an application released at a time corresponding with a signification political or cultural event may be relatively low compared when with that of a release performed at another time. Because of the multitude of factors that can potentially affect how (and how much) users will engage with a client application, it can be difficult to determine what features or configurations were actually preferred by users.

Aspects of the subject technology address the above issue by providing a method for facilitating experimentation between software configurations and parameters within a common application version or release. Specifically, aspects of the subject technology provide for a client application that includes multiple application variations (e.g., “variations”) associated with different configurations and features. By implementing (e.g., flagging) different variations in the client applications of separate users, different features/configurations can be provided to separate users of the same application version. For example, a first arrangement of elements in a browser application may correspond with a first variation, and a second arrangement may correspond with a second variation. Subsequently, experiments can be performed to compare the first and second variations (in the same browser version), for example, by analyzing and comparing data collected for users using the first variation with those using the second variation.

Although the possible experiments that may be performed for a particular application version are only limited by the number of variations, in certain aspects only a predetermined sub-set of all possible experiments will be available. A list of available experiments may be stored on a server accessible by the client application. In certain implementations, the client application will provide demographic information to the server to help further limit the set of available experiments. For example, the availability of a particular experiment to a particular client application may be based on the language version (or geographic region, etc.) of the client application. After the list of available experiments is determined, one or more experiments may be selected by the client application, causing the client application to be run with the features/configuration corresponding with the variation(s) associated with the selected experiment(s).

By comparing data collected from multiple client applications running different variations, inferences can be made as to the efficacy and/or desirably of certain configurations and features. Thus, the subject technology provides a means for facilitating experimentation between application features and configurations within a common application version.

FIG. 1illustrates a block diagram of an example system100that can be used to implement some aspects of the subject technology. Specifically, system100includes remote server102, network104and four user devices (106,108,110and112).

As illustrated in the example ofFIG. 1, remote server102is communicatively coupled to each of user devices (106,108,110and112), via network104. It is understood that any number of devices may be coupled via network104, including one or more additional servers and/or a greater (or fewer) number of user devices.

Server102can be used to distribute copies of a client application to different users associated with user devices106-112. For example, Version A of a client application can be distributed to each of user devices106-112, via network104. In certain implementations, each copy of client application Version A will include multiple variations. As used herein, “variation” can refer to a particular application feature or configuration, or to a combination of features/configurations. As such, different variations of the client application can correspond with different features, settings and/or configurations. By way of example, different variations can control differences in the look, behavior and/or functionality of the client application.

In certain aspects, each version of a client application can include multiple variations, each of which will affect the configuration or functionality of the client application. By way of example, a particular variation may correspond with a visual arrangement or placement of objects in a client application.

In one or more implementations, different variations for a client application will be selected. As illustrated in system100ofFIG. 1, each of client devices106-112that have received Version A of the client application will run a different variation of the client application. For example, client device106runs Variation 1 of Version A, client device108runs Variation 2 of Version A, client device110runs Variation 3 of Version A and client device112runs Variation 4 of Version A. As such, each of the users associated with client devices106-112will experience slightly different features and/or configurations of client application Version A.

The different variations of the client application running on each of client devices106-112can provide the associated users with a different user experience. By way of example, each variation may provide the users with a different arrangement of visual features in the corresponding client application, for example, a browser application. Subsequently, information (e.g., “usage information”) regarding the user's interaction with the client application can be collected and provided, for example, to a remote server. Usage information can include any type of information about the user, the client application (and associated variation), and/or the associated client device. In certain aspects, usage information will include data indicating the current variation and version of the client application, as well as information regarding the user's use of the client application. For example, usage information could include data related to features used by the user of the client application (e.g., types of features and frequency), and/or crash report information, etc. In some aspects, usage information may also include information regarding how a user uses a particular feature of the client application (e.g., if use of the feature involves an interface prompting the user to make a choice, the user's choice selection may be included as usage information). Additionally, usage information may include timing data related to a user's use of client application features, for example, a time duration between actions, such as, opening a new tab within a browser window.

In some implementations, usage information collected from multiple users (using different client devices running different variations of the client application), can be compared in order to perform experiments regarding different variations of the client application. By comparing different variations of the client application, various configurations, features and parameters can be compared across a single version of the client application.

FIG. 2illustrates an example of steps of a method200for implementing certain aspects of the subject technology. The method200begins with step202in which a copy of a client application is provided to each of a plurality of client devices, wherein each copy of the client application includes multiple application variations. In certain aspects, application variations may be implemented as different code-paths within the same application (e.g., the client application) such that each variation can correspond with a specific set of conditionals. As discussed above, each variation included in each of the client application copies can correspond with one or more features, configurations or parameters that can affect the look and/or behavior of the associated client application.

In step204, demographic information is received from one or more of the plurality of client devices. Demographic information can include any information about the user and/or the client device for which the client application copy is provided (e.g., provided for download by the client device). By way of example, the demographic information can include information regarding the client device, such as operating system version information and/or information regarding hardware components of the client device, or hardware components connected to the client device. Demographic information may also include information that relates to a client application that is installed or executed on the client device. For example, demographic information may include the date on which the client application was installed and/or what version was installed, etc. In certain aspects, demographic information may also include language information for the user and/or information identifying the geographic region in which the client device resides. In some examples, the demographic information may further include information about user preferences related to the use of the client application.

In step206, a list of available experiments is provided to each of the plurality of client devices based on the demographic information, wherein each of the available experiments corresponds with one or more application variations. As used herein “experiments” correspond with one or more selections or combinations of client software variations that can be analyzed and compared, for example, to make relative determinations as to the utility and/or desirability of certain features and/or configurations of the client application.

Because different experiments may be relevant to different users (or groups of users), the list of available experiments provided to various client devices may be different. For example, different client devices (associated with different users) will have different demographic information; as such, the list of available experiments provided to each client device may be different or unique.

Based on the list of experiments that are available to a particular user device and associated client application, a particular variation (or combination of variations) can be flagged for use by the client application. Thus, each client application copy may implement an assortment of different client application variations. The selection of a particular variation (or combination of variations) can be performed by the client application or by a remote server. For example, a client application can be configured to choose and implement a particular variation (or combination of variations) based on a list of available experiments provided by a remote server. Because the list of available experiments provided to any particular client device is based on the demographic information for that client device, client application and/or associated user, the list of available experiments can be limited to those which are relevant (or most relevant) to a particular client application and/or user. That is, the variations (or variations) that are implemented in any of the client application copies, may implement changes that are most relevant to the associated user and/or client device.

In step208, usage information is received for each copy of the client application associated with each of the plurality of client devices. Usage information can include any information pertaining to the user's use of the client application. By way of example, if the client application is a web browser, the usage information may include data indicating how many searches were performed by the user and/or how many times the user used the web browser's “back” function. Such types of information may be relevant in assessing the user's experience with respect to the client application.

Subsequently, in step210, two or more application variations of the client application will be compared based on the usage information. Comparisons of various application variations will correspond with one or more of the available experiments, as discussed above with respect to step206. In some implementations, collections of experiments (e.g., “studies”) may be performed with respect to multiple features and/or configurations within a common version of the client application, for example, by comparing different experiments that have been performed across multiple client application copies.

FIG. 3illustrates an example network system300that can be used to implement some aspects of the subject technology. Specifically, network system300includes user devices302,304and306, network308and remote server310. As illustrated, the user devices (302-306) and remote server310are communicatively connected via network308. It is understood that in addition to the user devices (302-306) and remote server310, any number of processor-based devices could be communicatively connected to network308. Furthermore, as will be discussed in greater detail below, network308could comprise multiple networks, such as a network of networks, e.g., the Internet.

In some examples, one or more of the process steps of the subject technology can be carried out by one or more of the user devices302-306, and remote server310, over network308. By way of example, remote server310could be configured to provide a copy of a client application to each of a plurality of client devices (e.g., user devices302-306), wherein each copy of the client application comprises multiple application variations. Additionally, remote server310can be configured to receive demographic information from one or more of the plurality of user devices302-306and to provide a list of available experiments to each of the plurality of the user devices302-306based on the demographic information, wherein each of the available experiments corresponds with one or more application variations. Remote server310can also be configured to receive (e.g., via network308) usage information for each copy of the client application associated with each of the user devices302-306and to compare two or more application variations of the client application based on the usage information. In certain implementations, remote server310may be further configured to determine a preferred application variation based on the comparison of the two or more application variations.

FIG. 4illustrates an example of an electronic system400with which some aspects of the subject technology can be implemented. In some examples, electronic system400can be a single computing device such as a server, e.g., remote server310, discussed above. Furthermore, in some implementations, electronic system400can be operated alone or together with one or more other electronic systems e.g., as part of a cluster or a network of computers.

As illustrated, electronic system400comprises storage402, system memory404, output device interface406, system bus408, ROM410, processor(s)412, input device interface414and a network interface416. In some aspects, system bus408collectively represents all system, peripheral, and chipset buses that communicatively connect the numerous internal devices of the electronic system400. For instance, system bus408communicatively connects the processor(s)412with the ROM410, the system memory404, the output device interface406and the permanent storage device402.

In some implementations, the various memory units, the processor(s)412retrieve instructions to execute (and data to process) in order to execute the steps of the subject technology. Processors(s)412can be a single processor or a multi-core processor in different implementations. Additionally, the processor(s) can comprise one or more graphics processing units (GPUs) and/or one or more decoders, depending on implementation.

ROM410stores static data and instructions that are needed by processor(s)412and other modules of electronic system400. Similarly, processor(s)412can comprise one or more memory locations such as a CPU cache or processor in memory (PIM), etc. Storage device402, is a read-and-write memory device. In some aspects, this device can be a non-volatile memory unit that stores instructions and data even when electronic system400is without power. Some implementations of the subject disclosure can use a mass-storage device (such as solid state, magnetic or optical storage devices) e.g., a permanent storage device402.

Other implementations can use one or more a removable storage devices (e.g., magnetic or solid state drives) such as permanent storage device402. Although the system memory can be either volatile or non-volatile, in some examples the system memory404is a volatile read-and-write memory, such as a random access memory. System memory404can store some of the instructions and data that the processor needs at runtime.

In some implementations, the processes of the subject disclosure are stored in system memory404, permanent storage device402, ROM410and/or one or more memory locations embedded with the processor(s)412. From these various memory units, processor(s)412retrieve instructions to execute and data to process in order to execute the processes of some implementations of the instant disclosure.

Bus408also connects to the input device interface414and output device interface406. The input device interface414enables a user to communicate information and select commands to the electronic system400. Input devices used with the input device interface414may include for example, alphanumeric keyboards and pointing devices (also called “cursor control devices”) and/or wireless devices such as wireless keyboards, wireless pointing devices, etc.

Finally, as shown inFIG. 4, bus408also communicatively couples the electronic system400to a network (not shown) through a network interface416. It is be understood that the network interface416can be either wired, optical or wireless and may comprise one or more antennas and transceivers. In this manner, the electronic system400can be a part of a network of computers, such as a local area network (“LAN”), a wide area network (“WAN”), or a network of networks, such as the Internet (e.g., the network310, as discussed above).

In practice the methods of the subject technology can be carried out by the electronic system400. In some aspects, instructions for performing one or more of the method steps of the present disclosure will be stored on one or more memory devices such as the storage402and/or the system memory404.

All structural and functional equivalents to the elements of the various aspects described throughout this disclosure that are known or later come to be known to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be encompassed by the claims.