Method and system for detecting data bucket inconsistencies for A/B experimentation

The present teaching generally relates to identifying data bucket overlap with online experiments. In a non-limiting embodiment, first data representing a first set of identifiers associated with a first data bucket of a first online experiment may be obtained. Second data representing a second set of identifiers associated with a second data bucket of the first online experiment may be obtained. Based on the first data and the second data, a first number of identifiers that are associated with the first data bucket and the second data bucket may be determined. In response to determining that the first number exceeds a threshold, a data flag indicating that results associated with the first online experiment are inconsistent may be generated.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is related to U.S. patent application Ser. No. 15/677,925, entitled “Method and System for Detecting Gaps in Data Buckets for A/B Experimentation,” filed on Aug. 15, 2017, and U.S. patent application Ser. No. 15/677,724, entitled “Method and System for Providing Pre-Approved A/A Data Buckets,” filed on Aug. 15, 2017, the disclosures of each being incorporated herein by reference in their entireties.

BACKGROUND

1. Technical Field

The present teaching generally relates to online experimentation. More specifically, the present teachings relate to providing pre-approved A/A data buckets. Further, the present teachings relate to detecting and monitoring a gap in data buckets for online experimentation. Still further, the present teachings relate to detecting and monitoring inconsistencies in data buckets for online experimentation.

2. Technical Background

In the age of the Internet, online experimentation, and in particular controlled online experimentation, is a commonly used and effective tool for product development. One such type of controlled online experiment is A/B testing. A/B testing in a more classical sense corresponds to having two (or more) groups, where one group—the control group—is given a controlled experience, while another group (or groups) are given a test experience. For example, in drug tests, a control group may receive a placebo while a test group may receive a test drug. In online experimentation, one set of users may receive one user experience at their user device, while another set of users may receive a different user experience. This allows a service provider to gauge an effectiveness of the user experience based on various user metrics computed for each of the various users of each set of users. For example, a service provider may test an effectiveness of a new website format (experimental user experience) against an original website format (control user experience). Users accessing the website via their user devices may randomly be provided with one of the new website format or the original website format, and user interaction metrics (e.g., clicks, scrolls, advertisement impressions, click-through-rate, etc.) may be computed for each website format for each user to determine how “effective” the new website format is as compared to the original website experience.

There are some requirements to these types of online experiments in order for the results to be accurate. For example, the two or more groups (e.g., control group and experimental group(s)) should include users that are randomly selected without any predisposition. Additionally, a user placed into one group should remain in that group throughout the duration of the experiment. Further still, the size (e.g., number of users) in each group should be substantially equal. Failure to adhere to these conditions may result in compromised results that do not accurately reflect the outcome that is obtained.

One way to prevent potential errors in the experimentation process is to perform A/A validation. A/A validation, for example, serves to try and validate the control and experimentation groups, or in the case of online experimentation, the control data bucket and the experimentation data bucket(s). This includes determining whether or not there are any pre-existing differences in the control and/or experimentation buckets, as well as determining whether or not there are any systematic errors in the experimentation system that would lead to unexpected results.

Typically, A/A validation takes time. For example, in order for data to be obtained and analyzed, four to five days' worth of data may be needed. Additionally, it may be expected for some data buckets of online experiments to fail the A/A validation process. Therefore, it is common for experimenters to open more data buckets for the A/A validation process then may actually be needed. This may cause the experimenter to have to decide which data bucket, if any, to use for the experiment. These issues, amongst various others, may cause a delay in the start of an online experiment, which in the ever evolving and fast paced online world, is undesirable.

When online experiments are performed, as mentioned above, users may be placed into one of a control group or an experimentation group. In certain scenarios, a single experiment may include two or more experimentation groups, depending on the conditions and aspects sought to be tested by the experimenter. Each of the control group and the experimentation group (or groups) is designed such that they have a same size. For example, a control data bucket may be designed such that it is to include 10,000 randomly selected users, while an experimental data bucket may also be designed such that it is to include 10,000 randomly selected, but different, users. If, during or after experimentation it is determined that one or more of these data buckets included substantially less than the designed number of users, the results of the experiment may be compromised. For example, if the data buckets were designed to include 10,000 users, but after the experiment it is determined that the data bucket online includes 7,000 users, this may lead to inaccurate results. Inaccurate results may corrupt the data, and undermine the findings of that experiment.

Further still, when placing users in one of the control data buckets or one of the one or more experimental data buckets, it is believed that each user will only be placed in one of these data buckets. Intuitively, if a user is to be placed in both the control data bucket and the experimental data bucket, this may lead to compromised results. Similarly, if a user is placed in two different experimental data buckets, inconsistent results across these two data buckets will arise.

Each of these issues, as described above, are further exacerbated by the platform of online experimentation as the number of users is extremely large, and the time scale of randomly selecting and placing a user into a data bucket, and rolling out various user experiences is very small.

Therefore, providing techniques to reduce the amount of time needed to validate and accuracy of data buckets for use create online experiments is needed. Further, providing techniques to identify a gap between an expected data bucket size and an actual data bucket size is needed. Still further, providing techniques to identify inconsistencies between data buckets is needed.

SUMMARY

The teachings disclosed herein generally relate to methods, systems, and programming for providing data buckets for online experiments. The teachings disclosed herein further generally relate to methods, systems, and programming for detecting data bucket discrepancies associated with online experiments. The teachings disclosed herein still further generally relate to methods, systems, and programming for identifying data bucket overlap with online experiments.

In one example, a method for providing data buckets for online experiments may be implemented on at least one machine including at least one processor, memory, and communications circuitry is described. User activity data representing user activity for a first plurality of user identifiers may be obtained. A first set of values representing a first user engagement parameter for each user identifier of the first plurality may be generated based on the user activity data, and a second set of values representing a second user engagement parameter for each user identifier of the first plurality may also be generated based on the user activity data. A first ranking for the first set may be determined, and a second ranking for the second set may also be determined. A first exclusion range including a first number of values to be removed from the first set and the second set may be determined. A homogenous value set may be generated by removing the first number of values from the first set and the second set, where each value from the homogenous value set corresponds to a user identifier that is available to be placed in a data bucket for an online experiment.

In another example, a system for providing data buckets for online experimentation is described. The system includes a metric computation system configured to obtain user activity data representing user activity for a first plurality of user identifiers, generate a first set of values representing a first user engagement parameter for each user identifier of the first plurality based on the user activity, and generate a second set of values representing a second user engagement parameter for each user identifier of the first plurality based on the user activity. The system also includes a hash value ranker configured to determine a first ranking for the first set, and determine a second ranking for the second set. The system further includes a hash value exclusion system configured to determine a first exclusion range including a first number of values to be removed from the first set and the second set. The system still further includes a hash value selector system configured to generate a homogenous value set by removing the first number of values from the first set and the second set, wherein each value from the homogenous value set corresponds to a user identifier that is available to be placed in a data bucket for an online experiment.

Other concepts relate to software for implementing the present teaching on providing data buckets for online experiments. A software product, in accord with this concept, includes at least one machine-readable non-transitory medium and information carried by the medium. The information carried by the medium may be executable program code data, parameters in association with the executable program code, and/or information related to a user, a request, content, or information related to a social group, etc.

In one example, a non-transitory computer readable medium having instructions recorded thereon for providing data buckets for online experiments is described. The instructions, when read by a computer, causes the computer to obtain user activity data representing user activity for a first plurality of user identifiers, generate a first set of values representing a first user engagement parameter for each user identifier of the first plurality based on the user activity, and generate a second set of values representing a second user engagement parameter for each user identifier of the first plurality based on the user activity. The instructions further, when read by the computer, causes the computer to determine a first ranking for the first set, and determine a second ranking for the second set. The instructions further, when read by the computer, causes the computer to determine a first exclusion range including a first number of values to be removed from the first set and the second set, and to generate a homogenous value set by removing the first number of values from the first set and the second set, where each value from the homogenous value set corresponds to a user identifier that is available to be placed in a data bucket for an online experiment.

In yet another example, a method for detecting data bucket discrepancies associated with online experiments implemented on at least one machine including at least one processor, memory, and communications circuitry is described. A monitoring layer may be generated within an online experimentation platform. The online experimentation platform may include at least a first layer, where a first online experiment is associated with the first layer, the monitoring layer includes a monitoring layer data bucket, and the first layer includes at least a first data bucket. First data representing user activity associated with a first plurality of identifiers may be obtained, the user activity being associated with the first layer. Second data including representing at least one user engagement parameter may be generated based on the first data. A first discrepancy between the first data and the second data may be determined, where the first discrepancy indicates a first amount of identifiers that include a first metadata tag associated with the first layer and lack a second metadata tag associated with the monitoring layer.

In still yet another example, a system for detecting data bucket discrepancies associated with online experiments is described. The system includes an experimentation system configured to generate a monitoring layer within an online experimentation platform. The experimentation platform includes at least a first layer, and where a first online experiment is associated with the first layer, the monitoring layer includes a monitoring layer data bucket, and the first layer comprises at least a first data bucket. The system also includes a discrepancy detection system configured to obtain first data representing user activity associated with a first plurality of identifiers, the user activity being associated with the first layer. The discrepancy detection system is further configured to determine a first discrepancy between the first data and the second data, where the first discrepancy indicates a first amount of identifiers that include a first metadata tag associated with the first layer and lack a second metadata tag associated with the monitoring layer.

Other concepts relate to software for implementing the present teaching on detecting data bucket discrepancies associated with online experiments. A software product, in accord with this concept, includes at least one machine-readable non-transitory medium and information carried by the medium. The information carried by the medium may be executable program code data, parameters in association with the executable program code, and/or information related to a user, a request, content, or information related to a social group, etc.

In one example, a non-transitory computer readable medium having instructions recorded thereon for detecting data bucket discrepancies associated with online experiments. The instructions, when read by a computer, cause the computer to generate a monitoring layer within an online experimentation platform. The experimentation platform may include at least a first layer, and where a first online experiment is associated with the first layer, the monitoring layer includes a monitoring layer data bucket, and the first layer includes at least a first data bucket. The instructions, when read by the computer, also causes the computer to obtain first data representing user activity associated with a first plurality of identifiers, the user activity being associated with the first layer. The instructions, when read by the computer, further cause the computer to generate second data representing at least one user engagement parameter based on the first data. The instructions, when read by the computer, still further cause the computer to determine a first discrepancy between the first data and the second data, where the first discrepancy indicates a first amount of identifiers that include a first metadata tag associated with the first layer and lack a second metadata tag associated with the monitoring layer.

In still yet another example, a method for identifying data bucket overlap with online experiments implemented on at least one machine including at least one processor, memory, and communications circuitry is described. First data representing a first set of identifiers associated with a first data bucket of a first online experiment may be obtained. Second data representing a second set of identifiers associated with a second data bucket of the first online experiment may be obtained. Based on the first data and the second data, a first number of identifiers that are associated with the first data bucket and the second data bucket may be determined. In response to determining that the first number exceeds a threshold, a data flag may be generated that indicates that results associated with the first online experiment are inconsistent.

In still further yet another example, a system for identifying data bucket overlap with online experiments is described. The system includes a user identifier extraction system configured to obtain first data representing a first set of identifiers associated with a first data bucket of a first online experiment, and obtain second data representing a second set of identifiers associated with a second data bucket of the first online experiment. The system further includes a user identification comparison system configured to determine, based on the first data and the second data, a first number of identifiers that are associated with the first data bucket and the second data bucket. The system still further includes a data bucket abnormality system configured to generate, in response to determining that the first number exceeds a threshold, a data flag indicating that the results associated with the first online experiment are inconsistent.

Other concepts relate to software for implementing the present teaching on identifying data bucket overlap with online experiments. A software product, in accord with this concept, includes at least one machine-readable non-transitory medium and information carried by the medium. The information carried by the medium may be executable program code data, parameters in association with the executable program code, and/or information related to a user, a request, content, or information related to a social group, etc.

In one example, a non-transitory computer readable medium having information recorded thereon for identifying data bucket overlap with online experiments. The information, when read by a computer, causes the computer to obtain first data representing a first set of identifiers associated with a first data bucket of a first online experiment, and obtain second data representing a second set of identifiers associated with a second data bucket of the first online experiment. The information, when read by the computer, further causes the computer determine, based on the first data and the second data, a first number of identifiers that are associated with the first data bucket and the second data bucket. The information, when read by the computer, still further causes the computer to generate, in response to determining that the first number exceeds a threshold, a data flag indicating that results associated with the first online experiment are inconsistent.

DETAILED DESCRIPTION

The present disclosure generally relates to systems, methods, medium, and other implementations for providing data buckets for online experiments. The present disclosure generally relates to systems, methods, medium, and other implementations for detecting data bucket discrepancies associated with online experiments. The present disclosure generally relates to systems, methods, medium, and other implementations for identifying data bucket overlap with online experiments.

FIG.1is an illustrative diagram of an exemplary system for providing valid data buckets for an online experiment, in accordance with various embodiments of the present teachings. In the non-limiting example embodiment, an exemplary system100is described. System100includes, in one embodiment, a user device110. A user102may interact with user device110, for example, via user interface112. User device102may correspond to any suitable type of electronic device including, but are not limited to, desktop computers, mobile computers (e.g., laptops, ultrabooks), mobile phones, smart phones, tablets, televisions, set top boxes, smart televisions, personal display devices, personal digital assistants (“PDAs”), gaming consoles and/or devices, smart household devices (e.g., refrigerators, microwaves, etc.), smart vehicles (e.g., cars, trucks, motorcycles, etc.), smart transportation devices (e.g., boats, ships, trains, airplanes, etc.), wearable devices (e.g., watches, pins/broaches, headphones, etc.), and/or smart accessories (e.g., light bulbs, light switches, electrical switches, etc.). Although only one user device110is shown within system100, persons of ordinary skill in the art will recognize that any suitable number of user devices may be included within system100. Furthermore, various additional details related to user device110are described in greater detail below.

In one embodiment, user102may access content and/or perform one or more actions using user device110. In some embodiments, user102may access content, such as a website, hosted by a provider by inputting a uniform resource location (“URL”) into user device associated with the site. For example, user102may input a sports webpage's URL into user interface112and, in turn, user device110may access the website by communicating with the sports webpage's server across one or more networks104.

Network(s)104may be a single network or a combination of different networks. For example, a network may be a local area network (“LAN”), a wide area network (“WAN”), a public network, a private network, a proprietary network, a Public Telephone Switched Network (PSTN), the Internet, a wireless network, a cellular network, a virtual network, or any combination thereof. A network may also include various network access points, (e.g., wired or wireless access points such as base stations or Internet exchange points) through which a data source may connect to the network(s)104in order to transmit information via network(s)106, and a network node may connect to the network(s)104in order to receive information. In one embodiment, the network(s)104may correspond to an online advertising network or an ad network, which connects one or more devices, systems, servers, and/or databases/data stores, with one or more other systems, devices, servers, etc.

As illustrated, system100may include web services120, which may allow a user to interact with a web site hosted thereby. For example, web services120may host a platform with which user102may set up an online experiment. As another example, web services120may provide content to user device110via network(s)104. Generally speaking, web services120may correspond to a suitable service provider/functionality capable of being interacted with by user102via user device110across network(s)104.

In some embodiments, system100may include one or more databases, which may also be referred to as data stores herein. For instance, system100may include an experimentation metadata database105, an analytics database115, a historical database125, and a dimension database135. Although databases105,115,125, and135are each separate in the illustrative embodiment, persons of ordinary skill in the art will recognize that two or more of databases105,115,125, and135may be combined. Additionally, data associated with any of databases105,115,125, and135may be distributed over one or more computing devices (e.g., server farms, database centers, and the like). In some embodiments, information stored by one or more of databases105,115,125, and135may be accessed by user device110, web services120, and/or a data pipeline130, as described in greater detail below, via network(s)104.

Data pipeline130, in one example embodiment, may be configured to validate data buckets for use in online experimentation. As described in greater detail herein, data pipeline130may include one or more components, and may be capable of communicating with one or more of databases105,115,125, and135, as well as web services120and user device110via network(s)104. In one example, data pipeline130may perform offline hashing and hash value validation for online experiments (e.g., A/A validation). For instance, data pipeline130may be configured to perform offline hashing using historical data for a certain temporal range (e.g., one day, two days, one week, one month, etc.), rank hash values based on one or more metrics, and validate the hash values. Validated hash values may then be provided to web services120, in one embodiment, for use in creation and execution of an online experiment.

FIG.2is an illustrative diagram of an exemplary multi-layer experimental platform, in accordance with various embodiments of the present teachings. Experimental platform200, in the illustrative embodiment, corresponds to a multi-layer platform including two or more layers. For example, multi-layer experimental platform200may include a first layer202a(e.g., Layer 1), and a second layer202b(e.g., Layer N). In the illustrative embodiment, platform200may include N layers. Each layer may have one or more experiments existing thereon. For example, first layer202a(e.g., Layer 1) may include a first experiment204a(e.g., Experiment 1) and a second experiment204b(e.g., Experiment 2). Second layer202b(e.g., Layer N) may include a single experiment204c(e.g., Experiment M). Furthermore, each experiment may include one or more data buckets, with which a user will be assigned to. As seen in the illustrative embodiment, first experiment204a(e.g., Experiment 1) may include a first data bucket206a(e.g., Bucket 1) and a second bucket206b(e.g., Bucket 2). For example, first data bucket206a(e.g., Bucket 1) may correspond to a control bucket, whereas second bucket206b(e.g., Bucket 2) may correspond to an experimental bucket. Further still, as seen in the illustrative embodiment, second experiment204b(e.g., Experiment 2) may include a first data bucket206c(e.g., Bucket 3), a second data bucket206d(e.g., Bucket 4), and a third data bucket206e(e.g., Bucket 5). For example, data bucket206cmay correspond to a control bucket, and each of data buckets206dand206emay correspond to an experimental bucket, where each of data buckets206dand206emay provide a different experimental variant to be tested.

In some embodiments, an experiment may use an identifier associated with each user as the experimental unit, where the experimental unit corresponds to the item to be randomized into a data bucket. As an illustrative example, an identifier that may be used is a browser cookie associated with each user device110that accesses web services120and/or data pipeline130, however persons of ordinary skill in the art will recognize that additional types of identifiers may be employed, and the use of browser cookies is merely exemplary. The various user identifiers (e.g., browser cookies), may be grouped into data buckets, where user devices110of a particular data bucket are provided with a same user experience, as described below with reference toFIGS.3A and3B.

Web services and the data pipeline, in some embodiments, correspond to components of the experimentation platform. The data pipeline may be configured to process data produced by the experiments and store that information in the data store. This enables the other components of the experiment system to access information. Web services may correspond to a hosted service which a layer between the UI component data component. The web server serves a web page that receives user requests and/or processes a request to enable a component to run. Experiments may be created via a web page user interface, which may be served by web services. Experiments that may be set up through a webpage user interface may be backed by a web service and saved in a database offline. The data buckets (experiment metadata) may then be provided to the user while browsing apps and/or webpages by another web service at run time. Thus, when an experiment is “running,” this may correspond to the experiment being set up and data buckets being served to users.

FIGS.3A and3Bare illustrative diagrams of an exemplary control user experience and an exemplary experimental user experience, respectively, in accordance with various embodiments of the present teachings. User experience300ofFIG.3A, in one embodiment, may correspond to a control user experience. In this particular scenario, for example, a website may be configured to display a first button302. However, an experiment designer may want to determine how receptive users would be to a change to the website where, instead of displaying first button302, a different button, button352, is displayed.

In order to determine the effectiveness of the new design (e.g., including button352as opposed to button302), an experiment designer may create an experiment (e.g., experiment204a) within a layer (e.g., layer202a), which includes two data buckets (e.g., data buckets206aand206b). Users who access the web site may be randomly placed into one of the two data buckets. Depending on which data bucket they are assigned to, one of user experience300, including button302, or user experience350ofFIG.3B, including button352, may be provided to a corresponding user. If positive feedback is obtained regarding user experience350, the experiment designer may modify his/her website such that instead of providing all users with user experience300, all users are provided with user experience350. Conversely, if the feedback received is poor regarding user experience350, then user experience300may be retained.

Returning toFIG.2, as described above, each user associated with a user identifier that is placed in one of an experiment's data buckets is provided with a same user experience. For instance, all of the users whose user identifiers are placed in a control data bucket may be provided with user experience300, whereas all of the users whose user identifiers are placed in an experimental data bucket may be provided with user experience350.

In some embodiments, each user identifier may be hashed (e.g., assigned and/or attributed) with an integer value within a pre-determined integer value range. For example, browser cookies for various user devices110may be hashed to an integer in the range of 0 to 999 (e.g., [0, 999]). Each data bucket of an experiment is assigned a range of these hash values, where a length of the range is associated with a desired size of that data bucket. In one embodiment, data buckets are sized in increments of 0.1% of total traffic volume for a particular website, however this is merely exemplary. As an illustrative example, a front webpage for a site may have, on average, daily traffic of 30 million users accessing the site. In this particular example, each hash value would correspond to 30,000 users. So, if the experiment designer wants to create a data bucket having a size of 5% of daily traffic, then 50 hash values would be allocated for that bucket.

Each experiment, as mentioned above, includes one or more data buckets. In one embodiment, each experiment may include two or more data buckets. For instance, each experiment may include one control data bucket and one or more test, or experimental, data buckets. All of the test data buckets may be compared with the control data bucket to test a particular feature associated with that test data bucket. For example, user experience300ofFIG.3Amay be associated with a control data bucket, and user experience350ofFIG.3Bmay be associated with a test data bucket. User activity data associated with the user identifiers of the control data bucket may be analyzed and compared with the user activity data associated with the user identifiers of the test data bucket. If, for example, one or more additional user experiences are to be tested, then another test data bucket is created, and user activity associated with user identifiers that are provided with the additional user experience of that test data bucket may also be analyzed and compared with the user activity data associated with the user identifiers of the control data bucket.

In one embodiment, each experiment, and therefore the data buckets associated therewith, exists on a single layer of platform200. Each layer covers an entire range of integer values (e.g., [0, 999]), and thus data buckets and experiments on a same layer are mutually exclusive as every user identifier is deterministically hashed to a same single hash value. Thus, a total bucket size over all experiments may not exceed 100% of the total traffic volume. Each layer has a unique seed that it is assigned, which is different and random, and may be used by a hash function to hash each user identifier to a particular integer. Therefore, each layer of platform200includes all available traffic, however traffic may be split across layers, and each layer is orthogonal to one another. Each user identifier may only be placed into, at most, one data bucket/experiment on each layer, but may also be placed into multiple buckets/experiments on different layers.

FIG.4is an illustrative diagram of an exemplary data pipeline system, in accordance with various embodiments of the present teachings. Data pipeline system130, in the non-limiting embodiment, includes an offline hash generation system410, a metric computation system420, a hash value ranker430, a hash value exclusion system440, and a hash value selector system450. Each of offline hash generation system410, metric computation system420, hash value ranker430, hash value exclusion system440, and hash value selector system450may include one or more processor(s)402, memory/storage404, and communications circuitry406, amongst other components.

Processor(s)402may include any suitable processing circuitry capable of controlling operations and functionality of one or more components/modules of data pipeline system130, as well as facilitating communications between various components within data pipeline system130and/or with one or more other systems/components of system100. In some embodiments, processor(s)402may include a central processing unit (“CPU”), a graphic processing unit (“GPU”), one or more microprocessors, a digital signal processor, or any other type of processor, or any combination thereof. In some embodiments, the functionality of processor(s)402may be performed by one or more hardware logic components including, but not limited to, field-programmable gate arrays (“FPGA”), application specific integrated circuits (“ASICs”), application-specific standard products (“ASSPs”), system-on-chip systems (“SOCs”), and/or complex programmable logic devices (“CPLDs”). Furthermore, each of processor(s)402may include its own local memory, which may store program systems, program data, and/or one or more operating systems. However, processor(s)402may run an operating system (“OS”) for one or more components of data pipeline system130(e.g., offline hash generation system410, metric computation system420, hash value ranker430, hash value exclusion system440, and hash value selector system450), and/or one or more firmware applications, media applications, and/or applications resident thereon. In some embodiments, processor(s)402may run a local client script for reading and rendering content received from one or more websites. For example, processor(s)402may run a local JavaScript client for rendering HTML or XHTML content received from a particular URL.

Storage/memory404may include one or more types of storage mediums such as any volatile or non-volatile memory, or any removable or non-removable memory implemented in any suitable manner to store data for one or more of offline hash generation system410, metric computation system420, hash value ranker430, hash value exclusion system440, and/or hash value selector system450. For example, information may be stored using computer-readable instructions, data structures, and/or program systems. Various types of storage/memory may include, but are not limited to, hard drives, solid state drives, flash memory, permanent memory (e.g., ROM), electronically erasable programmable read-only memory (“EEPROM”), CD-ROM, digital versatile disk (“DVD”) or other optical storage medium, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, RAID storage systems, or any other storage type, or any combination thereof. Furthermore, storage/memory404may be implemented as computer-readable storage media (“CRSM”), which may be any available physical media accessible by processor(s)402to execute one or more instructions stored within storage/memory404. In some embodiments, one or more applications (e.g., gaming, music, video, calendars, lists, etc.) may be run by processor(s)402, and may be stored in memory404.

Communications circuitry406may include any circuitry allowing or enabling one or more components of data pipeline system130to communicate with one another, and/or with one or more additional devices, servers, and/or systems. For example, communications circuitry406may facilitate communications between two or more of offline hash generation system410, metric computation system420, hash value ranker430, hash value exclusion system440, and/or hash value selector system450, or between one or more components of data pipeline system130, or between one or more components of system100. In some embodiments, communications between one or more components of system100may communicate with user devices110and/or data pipeline system130and/or web services120across network(s)104via communications circuitry406. For example, network(s)104may be accessed using Transfer Control Protocol and Internet Protocol (“TCP/IP”) (e.g., any of the protocols used in each of the TCP/IP layers), Hypertext Transfer Protocol (“HTTP”), WebRTC, SIP, and/or wireless application protocol (“WAP”). Various additional communication protocols may be used to facilitate communications between various components of data pipeline system130and/or to/from data pipeline system130, including, but not limited to, Wi-Fi (e.g., 802.11 protocol), Bluetooth, radio frequency systems (e.g., 900 MHz, 1.4 GHz, and 5.6 GHz communication systems), cellular networks (e.g., GSM, AMPS, GPRS, CDMA, EV-DO, EDGE, 3GSM, DECT, IS 136/TDMA, iDen, LTE or any other suitable cellular network protocol), infrared, BitTorrent, FTP, RTP, RTSP, SSH, and/or VOIP.

Communications circuitry406may use any communications protocol, such as any of the previously mentioned exemplary communications protocols. In some embodiments, one or more components of data pipeline system130(e.g., offline hash generation system410) may include one or more antennas to facilitate wireless communications with a network using various wireless technologies (e.g., Wi-Fi, Bluetooth, radiofrequency, etc.). In yet another embodiment, one or more components of user activity detection system may include one or more universal serial bus (“USB”) ports, one or more Ethernet or broadband ports, and/or any other type of hardwire access port so that communications circuitry406facilitates communications with one or more communications networks.

Offline hash generation system410, in an example embodiment, may be configured to performing hashing and metric computation using historical data stored by historical database125. Offline hash generation system410may be capable of accessing data stored by historical database125via network(s)104, however alternatively and/or additionally, offline hash generation system410may access data from historical database125directly without network(s)104. In some embodiments, offline hash generation system410may be configured to obtain historical data, such as, and without limitation, user activity data associated with one or more webpages for one or more user identifiers, for various temporal intervals. For example, offline hash generation system410may be configured to obtain user activity data associated with user activities from a previous N days.

User activity data may be stored by historical database125with temporal metadata indicating a time/date with which the activity occurred. The user activity data may be obtained at predefined temporal intervals (e.g., every minute, every hour, every day, etc.), and/or upon a request being received by data pipeline system130. For example, in response to receiving a valid hash value request, user activity may be accessed from historical database125. Data pipeline system130, in one embodiment, may include a timer470, which offline hash generation system410may be operatively in communication with, to determine when to obtain user activity data from historical database125. For instance, timer470may count various temporal intervals, and at each temporal interval, timer470may notify offline hash generation system410to obtain user activity data from historical database125representing user activity occurring within a particular amount of time prior to an expiration of that temporal interval. For example, every day, offline hash generation system410may access user activity data associated with a past seven days of user activity from historical database125.

Metric computation system420, in one embodiment, may be configured to compute values for one or more metrics based on the user activity data obtained from offline hash generation system410. Metric computation system420may be in communication with dimension database135. Dimensions database135, in one embodiment, may store dimension information indicating metric computation parameters, techniques, and processes, such that metric computation system may perform the one or more metric computations. For example, metric computation system420may access dimension information from dimension database135across network(s)135. As another example, metric computation system420may access dimension information from dimension database135directly.

As described in greater detail below with reference toFIGS.6A and6B, metric computation system420may be configured to compute one or more metrics for each user identifier from historical user activity data. Various types of metrics that may be computed on a user identifier basis may include, but are not limited to, a days visited metric, a page view (PV) metric, and a session-based metric. In some embodiments, a number of distinct identifiers may also be computed. The various types of user identifiers that may be used as a basis for the metric computation include, but are not limited to, browser cookie (e.g., b-cookie), IP address, device identifier, MAC address, telephone number, and the like. As an illustrative example, metric computation system420may obtain, for each browser cookie, activity data for user engagement parameters such as days visited, page view, sessions, as well as a distinct number of browser cookies from the user activity data. In one embodiment, however, dimension database135may store user engagement parameters, and thus metric computations for such user engagement parameters, which may be obtained by metric computation system420.

Hash value ranker system430, in one embodiment, may be configured to rank each hash value based on the one or more metrics that have been computed by metric computation system420. For instance, hash value ranker system430may rank the user engagement parameters such as days visited, page view, sessions, and distinct identifiers, separately, to produce a ranked list of these hash values for each parameter. In some embodiments, for each hash value, a mean hash value for each user engagement parameter may be computed, and the mean hash value may be used for the ranking. The net result of hash value ranker system430may be one or more lists that indicate, for each hash value, a rank of that hash value for each user engagement parameter that has been computed. A more detailed explanation of hash value ranker system430may be seen below with reference toFIG.7A.

Hash value exclusion system440, in one embodiment, may determine an exclusion range for hash values from the ranked hash values obtained by hash value ranker system430. In some embodiments, the hash value exclusion range may be based on one or more exclusion parameters460. After obtaining the ranked hash value list, hash value exclusion system440may be configured to identify the exclusion range to be employed for a particular user activity range. In some embodiments, this determination may be based on temporal information obtained from timer470. For instance, the exclusion range may vary depending on the historical range of data and/or the frequency of which that data is received. Additionally, in some embodiments, hash value exclusion system440may be configured to remove any hash values associated with one or more of the user engagement parameters falling within the exclusion range identified from exclusion range parameters460. After the excluded hash values are removed from the hash value list, the remaining hash values, which may correspond to a homogenous hash value set, may be stored in analytics database115. A more detailed explanation of hash value exclusion system440may be seen below with reference toFIG.8A.

Hash value selector system450, in one embodiment, may be configured to facilitate creation of an online experiment using validated hash values, such as those stored by analytics database115. In some embodiments, in response to receiving a request to create an experiment, an experiment may be created within a layer of multi-layer experimentation platform200. The request, for example, may be received by web services120from user interface112of user device110. Hash value selector system450may be configured to determine a data bucket size for the experiment, as requested by the user, and fill each of the one or more data buckets with hash values from the homogenous hash value set stored by analytics database115. Upon the data buckets being filled, hash value selector system450may provide a notification to web services120, which relies the notification to user interface112, indicating that the experiment is ready and may proceed. A more detailed explanation of hash value selector system450may be seen below with reference toFIG.9A.

FIG.5is an illustrative flowchart of an exemplary processor for generating a homogenous value set of hash values, in accordance with various embodiments of the present teachings. Process550, in a non-limiting embodiment, may begin at step502. At step502, a request may be received. For example, user102may submit a request via user interface112of user device110to web services120via network(s)104. The request may be for an experiment to be created, to update a homogenous value set of hash values, or may be any other suitable type of request. In some embodiments, data pipeline system130may receive the request from web services120and/or user device110via network(s)104.

At step504, one or more metrics to be computed are determined. The one or more metrics may, in some embodiments, correspond to user engagement parameters. For example, days visited, page views, sessions, as well as a total number of distinct user identifiers, may all correspond to metrics to be determined.

At step506, historical user activity data may be obtained. For instance, offline hash value generation system410may obtain historical user activity data from historical database125. In some embodiments, the historical user activity data may correspond to user activity that occurred within a predetermined temporal range. For example, the obtained historical user activity data may represent user activity from a past seven days, however persons of ordinary skill in the art will recognize that the historical user activity data may represent user activity from any suitable previous amount of time, and the use of seven days is merely exemplary. In some embodiments, offline hash generation system410may determine what data to obtain from historical database125based on a current time, indicated from time470.

At step508, the one or more metrics may be computed based on the historical user activity data that was obtained. In this particular example, user engagement parameters of days visited, page views, and sessions may be computed as metrics based on each user identifier included within the historical user activity data. Furthermore, the number of distinct user identifiers included within the historical user activity data may also be determined.

At step510, hash values for each metric may be ranked. After the metrics are computed for each user identifier (e.g., daily visits metric, page view metric, sessions based metric), in some embodiments, each user identifier may be hashed to an integer within a particular integer range. For example, there may be 1,000 integers in the integer range [0, 999] (e.g., 0 to 999). Each user identifier may then be assigned to one of those integers, and then the identifiers may be grouped for the particular integer values assigned thereto. As described in greater detail below, the hashing may be performed by a hash function that randomly assigns the user identifiers to a particular integer based on a unique seed. In some embodiments, a mean value for each of the user engagement parameters may be computed for each hash value (e.g., [0, 999]). In one embodiment, a data bucket size—or in other words a count of distinct identifiers—may also be computed for each hash value. The hash values may then be ranked for each metric.

At step512, one or more exclusion ranges may be determined. In some embodiments, the exclusion range(s) may be determined based on one or more exclusion parameters460. As an illustrative example, a top fifty and a bottom fifty hash values may correspond to one exemplary exclusion range. At step514, hash values within the exclusion range(s) for each metric may be excluded from the list of hash values. For example, the hash values in the top 50 and the bottom 50 of the ranked list for the page view user engagement parameter may be removed. A similar process may also occur for the any additional user engagement parameters that are employed. As a result, a substantially homogenous hash value set may be obtained. At step516, the remaining hash values (e.g., the homogenous hash value set) may be stored within analytics database115.

At step520, a determination may be made as to whether or not a temporal range has ended. For example, a determination may be made as to whether or not a new set of historical user activity data is needed. If not, then process550may proceed to step522, where process550may end. However, in some embodiments, process550may continually loop at step520until it is determined that the temporal range has ended. If, at step520, it is determined that the temporal range has ended, then process550may proceed to step518, where a new temporal data range for historical user activity data is determined. For example, the historical user activity data may correspond to user activity occurring during a past seven days, and the historical user activity data may be updated every six hours. If a temporal difference between a time when the request is received, and a time when a determination as to whether the temporal range has ended is less than then a threshold amount of time (e.g., one minute, one hour, six hours, one day, etc.), then no new historical user activity data may be obtained. However, if the temporal difference is greater than the threshold amount of time, then process550may return to step504where new historical user activity data associated may be obtained. As an illustrative example, the threshold amount of time may be one day. If the previous historical data corresponded to June 1 to June 7, then after the threshold amount of time has elapsed, the new historical data may correspond to June 2 to June 8.

FIG.6Ais an illustrative diagram of an exemplary metric computation system, in accordance with various embodiments of the present teachings. Metric computation system420, in the non-limiting, illustrative embodiment, may include a data bucket size configuration system610, a metric selection system620, and a metric calculation system630. Each of data bucket size configuration system610, metric selection system620, and metric calculation system630may include one or more processors602, memory/storage604, and communications circuitry606, amongst other components. In some embodiments, processor(s)602, memory/storage604, and communications circuitry606may be substantially similar to processor(s)402, memory/storage404, and communications circuitry406ofFIG.4, and the previous descriptions may apply.

Data bucket size configuration system610may, in one embodiment, be configured to receive historical user activity data from offline hash generation system410and/or historical database125. Furthermore, data bucket size configuration system610may, in one embodiment, be configured to determine a size of a data bucket to be created for an online experiment based on a request for creating an experiment that was received by data pipeline130. For example, if the request indicates that the data buckets to be created are 5% of the daily traffic, then data bucket size configuration system610may determine a total amount of daily traffic, and may determine a number of hash values to use to provide data buckets meeting the specifications of the request. Still further, in some embodiments, data bucket size configuration system610may be configured to determine a historical data range for user activity data to be obtained, or that has been obtained. For example, a determination may be made that seven days of user activity data may be needed, and data bucket size configuration system610may request that data from historical database125(and/or offline hash generation system410).

Metric selection system620may be configured to determine and/or select, in one embodiment, the one or more metrics to be used for user engagement analysis and hash value selection. For instance, metric selection system620may select one or more metrics605to use for calculating user engagement associated with the historical user activity data. Metrics605may correspond to user engagement parameters that are computed. For example, the various types of user engagement parameters may include, but are not limited to, daily visits, page views, sessions, and the like. A metric for a particular user engagement determines a value for that user engagement parameter that may be selected by metric selection system620to be computed. Additionally, a metric associated with an amount of distinct user identifiers within the user activity data may also be determined. In some embodiments, certain metrics may be selected by metric selection system620as opposed to others based on a particular experiment, or based on a request from a user. Generally speaking, metrics for determining user engagement may be selected based on data integrity. For example, it may be determined that days visited, page view, and sessions user engagement parameters produce historically accurate metrics quantifying usage of a particular website and/or product in terms of visitation frequency and level of interaction. Persons of ordinary skill in the art will recognize that additional user engagement parameters may be employed for metrics605and/or substituted for one or more of the previously mentioned parameters, and the aforementioned is merely exemplary.

Metric calculation system430, in one embodiment, may be configured to calculate the one or more metrics that have been selected based on the historical user activity data. In some embodiments, one or more models615may be employed by metric calculation system630to calculate/compute the one or more metrics. Model(s)615, for instance, may describe formulations to be employed to compute each metric605that has been selected by metric selection system620. For example, model(s)615may include a days visited model used for a days visited metric, a page view model for a page view metric, and a sessions model for a sessions based metric, however additional models may be used as well. In some embodiments, metric calculation system430may calculate metric values for each user identifier, where metric values correspond to the metrics for each user engagement parameter that have been computed analyzed. Upon completion, the metrics values for each user engagement parameter may be stored by metric value(s) database625. In some embodiments, metric value(s) database625may be in communication with analytics database115such that, upon computation of the various metric values, the metric values may be stored additionally, or alternatively, by analytics database115. In some embodiments, metric value(s) database625may serve as a temporary storage for the various metric values computed, however this is merely exemplary.

FIG.6Bis an illustrative flowchart of an exemplary processor for generating and storing one or more metric values, in accordance with various embodiments of the present teachings. Process650, in a non-limiting embodiment, may begin at step652. At step652, a historical data range may be determined. For instance, an amount of time with which user activity data is to be obtained may be determined. As an illustrative example, user activity data representing user activity occurring over a past seven days may be desired. Therefore the historical data range, in this particular scenario, would correspond to seven days.

At step654, one or metrics to be computed are selected. For example, metric selection system620may select one or more metrics605to be used for determining/quantifying user engagement. In one example embodiment, days visited metrics, page view metrics, and sessions based metrics may be selected. Additionally, a distinct number of user identifiers (e.g., browser cookies, MAC addresses, device identifiers, etc.) may be determined.

At step656, the historical user activity data may be obtained. In some embodiments, the historical user activity data may be obtained from historical database125, which may store user activities occurring on a website for each user identifier whose corresponding user device110accessed the site. The historical user activity data may represent various user activities that occurred. For example, user activities, which may also be referred to as interactions, with content being rendered, such as images, text, video, hyperlinks, and the like, may include, but is not limited to clicks, dwelling, hovering on/over, tapping, swiping, scrolling, flicking, etc., the content. In the context of at least one embodiment, user activities related to the content may be observed and used to explore, calculate, determined, and/or enhance the effectiveness of the content and the user experience.

At step658, data bucket information may be obtained. For instance, the data bucket information may indicate a desired data bucket size. The data bucket size, for instance, may reflect the distinct number of user identifiers associated with the historical user activity data.

At step660, one or more metric values, corresponding to the one or more metrics selected to be computed, may be generated using the historical data. For example, a metric value or values for the days visited metric may be generated using the historical user activity data. At step662, the metric value(s) may be stored within metric value(s) database625. In some embodiments, the metric value(s) may be additionally and/or alternatively stored within analytics database115.

FIG.7Ais an illustrative diagram of an exemplary hash value ranker system, in accordance with various embodiments of the present teachings. Hash value ranker system430, in the illustrative, non-limiting embodiment, may include a hash value assignment/grouping system710, a metric calculation system720, a hash value size determination system730, and a hash ranking system740. Each of hash value assignment/grouping system710, metric calculation system720, hash value size determination system730, and hash ranking system740may include one or more processors702, memory/storage704, and communications circuitry706. Processor(s)702, memory/storage704, and communications circuitry706, in one embodiment, may be substantially similar to processor(s)402, memory/storage404, and communications circuitry406ofFIG.4, and the previous description may apply.

Hash value assignment/grouping system710, in one embodiment, may be configured to assign each identifier from the user activity data to an integer value (e.g., a hash value). For example, a range of integer values may be chosen, and each user identifier may be assigned to one of those integers. In some embodiments, the assignment of an identifier to a particular integer value, also referred to as a hash value, may be performed by a hash function. The hash function may be a randomizing function that randomly assigns each user identifier to one integer within the integer range using a unique seed. In one embodiment, each layer of a multi-layer experimentation platform may be associated with a unique seed for the hash function.

Hash value assignment/grouping system710may further be configured to generate grouping of user identifiers based on an integer value. For example, user identifiers assigned to a same integer may be grouped together. This may yield integer values including a number of user identifiers which have each been assigned thereto. Typically, the hash function employed distributes user identifiers uniformly across each hash value (e.g., 0, 1, 999), and the means of each metric representing a particular user engagement parameter are normally distributed by that hash value.

Metric calculation system720, in one embodiment, may be configured to determine an average of each metric associated with a corresponding user engagement parameter for each hash value. For example, for each hash value including a grouping of user identifiers, an average (e.g., a mean) metric value for that user engagement parameter may be determined. In some embodiments, metrics705may be employed to calculate the metric values by metric calculation system720. Metrics705may be substantially similar to metrics605ofFIG.6and the previous description may apply. In some embodiments, metrics705may also include information for determining the means for each metric to be calculated by metric calculation system720.

Hash value size determination system730, in one embodiment, may be configured to determine a quantity of distinct user identifiers within each set of hash values of each user engagement parameter. For example, data bucket determination system730may determine a number of unique user identifiers within each grouping of hash values. The quantity of user identifiers for each grouping of hash values should be substantially similar across each hash value, however in some embodiments there may be slight variations. Generally speaking, a good hash function will provide a relatively homogenous distribution of user identifiers across each hash value.

Hash ranking system740, in one embodiment, may be configured to determine a ranking of each set of values representing each of the user engagement parameters with which metrics were computed. For example, based on the mean hash values for each user engagement parameter, as well as the number of unique identifiers, a rank of the hash values for each user engagement parameter may be produced. The rank may list each user engagement parameter's top ranked hash value to its least ranked hash value.

FIG.7Bis an illustrative flowchart of an exemplary processor for generating metric ranking data for one or more metrics, in accordance with various embodiments of the present teachings. Process750may, in a non-limiting embodiment, begin at step752. At step752, user identifier data representing user identifiers may be obtained. In some embodiments, the user identifier data may be included within user activity data. For example, the user activity data may represent user activity for each of a plurality of user identifiers. The user identifier data may include, in one embodiment, each user identifier that accessed the webpage/website with which the online experimentation is to be associated with. For example, the user identifier data may represent a plurality of browser cookies, device identifiers, MAC address, and the like, that interact with particular content (e.g., a webpage).

At step754, each user identifier of the user identifier data may be assigned to an integer in a hash range. For instance, the hash range may correspond to a set of hash values (e.g., integers). In one example, the hash range may include 1,000 integers ranging from 0 to 999 (e.g., [0, 999]). In one embodiment, hash value assignment/grouping system may be configured to apply a hash function to each user identifier to generate a plurality of values, where each user identifier is assigned to one value of the plurality of values. In some embodiments, assigning each user identifier to an integer may include attributing a metadata tag associated with a particular hash value to that identifier. For example, a metadata tag may be attributed to a first user identifier indicating that the first user identifier is associated with a first hash value.

At step756, the user identifiers may be grouped together by a corresponding integer. Continuing the example above, hash value assignment/group system710may be configured to determine which user identifiers have been attributed with a same metadata tag, and may group those user identifiers together. Typically, the number of user identifiers within each grouping is fairly consistent, however this may depend on the hash function used to assign user identifiers to integer values (e.g., hash values).

At step758, one or more metric values for each integer may be generated for each metric. In some embodiments, a metric value for each hash value may be generated for each user engagement parameter's associated metric. For example, if the user engagement parameters are associated with days visited, page views, and sessions, then the metrics would correspond to a days visited metric, a page views metric, and a sessions based metric. Thus, a metric value, or values, may be generated, for each of these metrics, for each of the hash values. As an illustrative example, a metric value associated with a page views metric for each hash value may be generated. In some embodiments, a mean value for each metric value may be determined. For example, a mean value for each user engagement metric may be computed for each hash value.

At step760, a number of distinct user identifiers per integer may be determined. For example, the number of distinct user identifiers grouped into each integer value may be determined by hash value size determination system730. In some embodiments, the number of distinct user identifiers grouped into each integer value (e.g., hash value) is consistent across the various hash values. However, persons of ordinary skill in the art will recognize that variations across the number of distinct user identifiers grouped into each integer value may also occur.

At step762, metric ranking data may be generated by ranking each integer value for each of the metrics. For example, the various hash values may be ranked based on their computed metric value. In some embodiments, the ranking may be based on the mean metric value. Furthermore, in some embodiments, the ranking may be based on the various metric values computed as well as the number of distinct user identifiers within each of the integer values. The metric ranking data, therefore, may indicate which hash values have a greatest ranking for each selected metric, and which hash values have a lowest ranking for each selected metric. In one example embodiment, if there are 1,000 hash values, then rankings may rank each of those 1,000 hash values from the greatest metric value for a particular metric to the lowest metric value for that particular metric. More detail regarding the metric ranking may be seen from Table I, below.

FIG.8Ais an illustrative diagram of an exemplary hash value exclusion system, in accordance with various embodiments of the present teachings. Hash value exclusion system440, in one non-limiting embodiment, may include an exclusion range identification system800and an exclude range removal system810. Each of exclusion range identification system800and exclude range remove system810may include one or more processors802, memory/storage804, and communications circuitry806, amongst other components. Processor(s)802, memory/storage804, and communications circuitry806may, in some embodiments, be substantially similar to processor(s)402, memory/storage404, and communications circuitry406ofFIG.4, and the previous description may apply.

Exclusion range identification system800may be configured, in some embodiments, to determine which hash value from each ranking of hash values fall within an exclusion range. The exclusion range may be selected from exclusion parameters460, and may be based on a user preference, and experimental preference, or a default setting. As an illustrative example, the exclusion range may specify that hash values falling within the top 50 and bottom 50 are to be removed from consideration. However, persons of ordinary skill in the art will recognize that these values are merely exemplary. In some embodiments, the exclusion range may be symmetric (e.g., top X and bottom X hash values), whereas asymmetry between the top and bottom hash values may also be considered. Generally speaking, as the means of each of the user engagement metrics typically are distributed in a Gaussian distribution, the Central Limit Theorem would predict that an upper tail and a lower tail of the distribution would provide hash values that would traditionally fail A/A validation. Thus, restriction to a central part of the Normal distribution may occur.

Looking at Table 1, for hash value 1, may be identified as a removal candidate as the session's metric for hash value 1 has a ranking of 33 (e.g., if the exclusion range corresponds to the top 50 and bottom 50 hash values). Similarly, hash value 457 may also be removed as the distinct identifier count metric has a ranking of 971. Further still, hash value 998 may also be a candidate for removal as the page view metric ranking for hash value 998 is 19.

Excluded range removal system810, in some embodiments, may be configured to remove the hash values that have been determined to fall within the exclusion range. Thus, excluded range removal system810may be configured to generate a substantially homogenous value set by removing each of the number of values of the exclusion range from each ranked set, leaving the remaining hash value set (e.g., the homogenous hash value set) corresponding to user identifiers that are available to be placed in an experiment's data bucket. In some embodiments, excluded range removal system810may be configured to apply a metadata tag to each user identifier that falls within the exclusion range(s) such that the identifiers associated with those hash values are prevented from being selected for an experiment's data bucket. Alternatively, excluded range removal system810may be configured to apply a metadata tag to each user identifier that does not fall within the exclusion range(s) such that those identifiers may be used for selection into an experiment's data bucket. The remaining hash values may be stored by analytics database115for future use with an experiment.

As an illustrative example, looking at Table 1, a first metadata tag may be applied to user identifiers associated with hash value 0, hash value 2, and hash value 999, where the first metadata tag indicates that the corresponding user identifiers are available for use in an experiment. Continuing this example, a second metadata tag may be applied to user identifiers associated with hash value 1, hash value 457, and hash value 998, where the second metadata tag indicates that the corresponding user identifier is unavailable for us in the experiment.

FIG.8Bis an illustrative flowchart of an exemplary processor for generating and storing ranked metric data, in accordance with various embodiments of the present teachings. Process850, in a non-limiting embodiment, may begin at step852. At step852, ranking metric data may be obtained. For example, metric data representing the ranked list of user engagement parameters may be obtained from hash value ranker system430. At step854, one or more exclusion rules may be determined. For instance, exclusion rules460may be selected, indicating that hash values associated with a top X metric values are to be excluded, and a bottom Y metric values are to be excluded. In some embodiments, X and Y may be integers of equal value.

At step856, exclusion range rules may be applied to the ranking metric data. For instance, if the exclusion range indicates that has values associated with the top X metric values are to be excluded (e.g., removed from consideration), then those hash values may be excluded. In some embodiments, hash values that are determined to be excluded may have a metadata tag applied thereto to indicate that those hash values are unavailable for placement in an experiment's data bucket. At step858, ranked metric data may be generated. The ranked metric data may represent a homogenous value set including hash values that are available for use in an experiment (e.g., to be placed in an experiment's data bucket). At step860, the ranked metric data may be stored. For instance, the ranked metric data representing the homogenous value set may be stored in analytics database115. In some embodiments, the ranked metric data may include the excluded hash values, albeit those hash values may be flagged (e.g., metadata tag applied thereto) to indicate that the user identifiers associated with those hash values are unavailable and/or should not be used.

FIG.9Ais an illustrative diagram of an exemplary hash value selector system, in accordance with various embodiments of the present teachings. Hash value selector system450, in the illustrative embodiment, includes a user interface receiver system900, an experiment layer generation system910, a data bucket size system920, a data bucket filler system830, and a user interface notification system940. Each of user interface receiver system900, experiment layer generation system910, data bucket size system920, data bucket filler system830, and user interface notification system940includes one or more processors902, memory/storage904, and communications circuitry906, amongst other components. Processor(s)902, memory/storage904, and communications circuitry906, in one embodiment, may be substantially similar to processor(s)402, memory/storage404, and communications circuitry406ofFIG.4, and the previous description may apply.

User interface receiver system900may, in one embodiment, be configured to receive a request to populate two or more data buckets for an online experiment. For instance, an experiment designer (e.g., user102) may create, or seek to create, an online experiment (e.g., as described in greater detail above with reference toFIGS.2,3A, and3B). The experiment designer may create two or more data buckets (e.g., a control data bucket and a test data bucket or buckets), and may desire for these data buckets to be populated by user identifiers with which the experiment will be conducted with. The request may be transmitted by user device110to web services120via network(s)104. In turn, web services120may provide the request to data pipeline130, and/or receive the request independently.

Experimental layer generation system910may be configured to generate an experiment within a layer of multi-layer experimentation platform200, in some embodiments. For example, in response to receiving the request, a layer may be selected, and the experiment may be generated within that layer. In some embodiments, the experiment may encompass the entire layer, whereas in other embodiments, the experiment may only encompass a portion of that layer.

Data bucket size system920may be configured, in one embodiment, to determine a size of the data buckets to be used for the experiment being created. For example, the request from user102may indicate that a 5% data bucket is desired. Therefore, as data buckets may be sized in increments of 0.1% of total traffic volume for a particular website, if the site has, on average, daily traffic of 30 million users accessing the site, then each hash value would correspond to 30,000 users. So, if the experiment designer wants to create a data bucket having a size of 5% of daily traffic, then 50 hash values would be allocated for that bucket. In response to determining the size of the data buckets, data bucket size system920may provide data bucket size information to analytics database115for filling of the data buckets.

Data bucket filler system930may, in one embodiment, be configured to randomly select a plurality of user identifiers to be placed in one of the control data bucket or a test data bucket. Data bucket filler system930may receive user identifiers from a homogenous set of values stored by analytics database115. As mentioned previously, the homogenous hash value set may previously be validated from A/A validation, and thus an experiment designer need not wait to perform A/A validation tests prior to conducting the experiment. Data bucket filler system930may further be configured to assign a metadata tag associated with a corresponding data bucket that each user identifier has been assigned to, to each user identifier. This metadata tag may persist throughout the experiment such that that user identifier is continually provided with the user experiences attributed to that data bucket with which it is assigned. In some embodiments, a random assignment function may be used to randomly assign each user identifier to one of the data buckets of the online experiment. Persons of ordinary skill in the art will recognize that any suitable random assignment function may be employed.

User interface notification system940may be configured, in one embodiment, to generate and send a notification to user device110indicating that the data buckets have been filled, and that the experiment may begin. For instance, visualization data representing a graphical user interface may be generated and sent to user device110for rendering by user interface112. In some embodiments, the graphical user interface may indicate that the experiment is ready for use, as well as present analytics to be monitored for the experiment.

FIG.9Bis an illustrative flowchart of an exemplary processor for populating data buckets with user identifiers from pre-approved bucket data, in accordance with various embodiments of the present teachings. Process950, in one embodiment, may begin at step952. At step952, a request to start an experiment may be received. The request may indicate a data bucket size for the experiment, a name of the experiment, a number of data buckets needed for the experiment, and the like. At step954, an experiment may be generated within a layer. For example, an experiment may be generated within a layer of multi-layer experimental platform200. At step956, a data bucket size for the experiment may be determined. In some embodiments, the data bucket size may be determined from the previously received request. However, alternatively, the data bucket size information may be obtained after the experiment has been generated.

At step958, pre-approved bucket data may be identified. For example, the metric ranking data previously stored in analytics database115may be identified. The pre-approved bucket data may correspond to the metric ranking data representing a homogenous hash value set, which may include user identifiers that have been pre-approved for A/A validation. At step960, at least a first data bucket and a second data bucket may be generated for the experiment. For example, at least a control data bucket and one test data bucket may be generated. The control data bucket may provide a control user experience to each user with whose user identifier is assigned thereto, whereas a test user experience may be provided to each user whose user identifier is assigned to a particular test data bucket. While it is possible to use a single data bucket within an experiment, the results may not provide an accurate estimation of the effects of the experiment as there may be no basis for comparison. Therefore, typically, experiments will employ one control data bucket and one or more test data buckets.

At step962, the first data bucket and the second data bucket may be populated with user identifiers obtained from the pre-approved bucket data. The user identifiers may be randomly assigned to one of the first data bucket and the second data bucket. When a user identifier is assigned to a particular data bucket, a metadata tag is attributed to that data bucket such that the user experience associated with that data bucket is maintained for that particular user identifier. In some embodiments, the user identifier and metadata data pairing may be stored by analytics database115for continually monitoring of the user activity data of that particular user.

At step964, the experiment may be conducted. For instance, the control user experience for the control data bucket may be provided to the corresponding users whose user identifiers are associated with the control data bucket. Further, the test user experience for the test data bucket may be provided to the corresponding users whose user identifiers are associated with the test data bucket.

FIG.10is an illustrative diagram of a system for detecting data bucket discrepancies associated with online experiments, in accordance with various embodiments of the present teachings. System1000, in the illustrative non-limiting embodiment, may include user device110operated by user102, which may communicate with analytics system1010via network(s)104. In some embodiments, content may be provided to user102on user device110. For example, a user experience associated with an online experiment hosted by an experimentation system1030may be provided.

In some embodiments, user102may interact with the content associated with the user experience being provided for a particular online experiment. The interactions with the content may be detected by user device110and transmitted via network(s)104to analytics system1010. In some embodiments, each interaction that occurs with the provided content may be logged within a data logging system1060. For instance, event logs1004may store records of each user interaction that occurs. Event logs1004may be created by data highway1006which may correspond to a centralized event and log collection service. User engagement database1005, in one embodiment, may store events and user activity data associated with discrepancy monitoring data buckets. In some embodiments, user activity data associated with discrepancy monitoring data buckets may be aggregated hourly, daily, weekly, etc., and which point the aggregated user activated data may also be stored by user engagement database1005.

Analytics system1010may also include an interface1020. Interface1020may be configured to receive user requests, and provide responses to those user requests. For example, if user experience350ofFIG.3Bis provided to user device110, and user102clicks on button352, the indication that the click occurred may be received by user interface1020. In response, user interface1020may be configured to receive, from experimentation system1030, a predetermined response to the clicking of button352. The response may in turn be communicated back to user device110for consumption by user102.

Analytics system1010may further include an experimentation platform1030, which may be in communication with interface1020, as mentioned above. Experimentation system1030may be configured to host one or more online experiments to be provided to one or more users. In some embodiments, analytics system1050may facilitate operation of an experimentation platform1050. For example, experimentation platform1050may correspond to a multi-layer experimental platform, such as multi-layer experimental platform200ofFIG.2. Experimentation system1030may further be in communication with user engagement database1005such that interactions associated with discrepancy monitoring may be stored thereby.

Analytics system1040may further include, in one embodiment, a discrepancy detection system1040. Discrepancy detection system1040may be configured, in one embodiment, to determine a discrepancy between an expected amount of user identifiers and an actual amount of user identifiers. This discrepancy, or gap, may reflect a difference between an expected sample size of an experiment and the actual sample size of the experiment. Large discrepancies may lead to inaccurate experimentation results and poor user experience. Therefore, reducing/minimizing the discrepancy is of upmost importance. Further still, identifying experiments where discrepancy exists, and in particular where large discrepancy exists, is crucial.

FIG.11is an illustrative diagram of a system for hashing a user device identifier into one or more layers of a multi-layer experimental platform, in accordance with various embodiments of the present teachings.FIG.11illustrates one embodiment of a multi-layered experimentation platform, such as that described byFIG.2. Each user102may be associated with a corresponding user device110, and each user device may be associated with a particular user identifier1102. Various types of user identifiers1102may include, but are not limited to, browser cookies, device identifiers, MAC identifiers, IP addresses, telephone numbers, and the like. Each user identifier1102may be provided to a hash function system1110, which may be configured to hash the user identifier to a particular hash value1104. Hash function system1110may, for example include a hash function that randomly assigns each user identifier to a hash value1104.

In some embodiments, each layer of the multi-layer experimentation platform1050may include a unique seed for the hash function. In some embodiments, hash values1104may encompass integer values within the range of 0 to 999 (e.g., [0, 999]), however persons of ordinary skill in the art will recognize that this is merely exemplary. Each layer of multi-layer experimentation platform1050may be orthogonal to the other layers, such that the same hash function of hash function system1110may be used for each layer with a corresponding unique seed associated with that layer. Thus, a user identifier may be hashed into a same segment in a given layer, however that user may be hashed into a different segment in a different layer.

FIG.12Ais an illustrative diagram of an exemplary experimentation system, in accordance with various embodiments of the present teachings. In the non-limiting embodiment, experimentation system1030one or more locations1220a,1220b, and1220c. When a request comes in to experimentation platform1030, it is routed to a nearest location by a load-balancer1210. Load-balancer1210, in one embodiment, may be configured to distribute incoming requests to one or more locations1220a-cbased on that locations proximity to a user device that sent the request, bandwidth restrictions of network(s)104, bandwidth settings of the various servers and processors of each of locations1220a-c, and the like.

Each of locations1220a-cincludes, in one embodiment, one or more edge servers and one or more application servers. For instance, location1220amay include edge server(s)1222aand application server(s)1224a, location1220bmay include edge server(s)1222band application server(s)1224b, and location1220cmay include edge server(s)122cand application server(s)1224c. Each of edge servers1222a-c, which may collectively referred to as edge server(s)1222, may include one or more proxy servers configured to route requests received by that edge server to a corresponding location's application server(s). In some embodiments, the one or more proxy servers may include support for data traffic splitting as well as infrastructure for meta-data distribution. Edge server(s)1222may run sampling functions to randomize the user identifiers, for example. Application servers1224a-c, which collectively may be referred to as application server(s)1224, may process received requests routed thereto and may be configured to generate and send back a response with an appropriate user experience to an end user's requesting user device110.

FIG.12Bis an illustrative diagram of the exemplary experimentation system ofFIG.12Acapable of providing various user experiences, in accordance with various embodiments of the present teachings. As mentioned above, experimentation system1030may include one or more locations1220, each of which may include one or more edge server(s)1222and one or more application server(s)1224.

In some embodiments, edge server(s)1222may include a user identifier (“ID”) receiver system1212and a data bucket assigner system1214. Each of user ID receiver system1212and data bucker assigner system1214may include one or more processor(s)1202, memory/storage1204, and communications circuitry1206. Each of processor(s)1202, memory/storage1204, and communications circuitry1206are, in some embodiments, substantially similar to processor(s)402, memory/storage404, and communications circuitry406ofFIG.4, and the previous description may apply.

User ID receiver system1212may, in some embodiments, be configured to receive incoming user requests received by experimentation platform1030and determine a user identifier associated with the user device with which the request is associated. For instance, requests received from user device110may be distributed to one of locations1220a-c. Upon receipt at one of locations1220a-c, a corresponding user ID receiver system1212of that location may determine a user ID associated with the user device that sent that the request. In some embodiments, the user identifier, or other characteristics used to identify the particular user that sent the request, may be determined using any suitable user identifier determination technique such as, and without limitation, user agent string analysis.

Data bucket assigner1214may be configured to determine a data bucket with which a user identifier that has been identified by user ID receiver system1212is to be assigned to. For example, data bucket assigner system1214may include a randomizer function that is configured to randomly assign a user identifier to a particular data bucket such that that user identifier's associated user device will receive a particular user experience. In one example embodiment, data bucket assigner system1214may input the user identifier into the randomizer function that will randomly pick a data bucket to assign that user identifier too. For example, the randomizer function may randomly assign the user identifier to one of a control data bucket or a test data bucket associated with a particular online experiment. In some embodiments, data bucket assigner system1214may determine that a user identifier associated with an incoming request is already associated with a particular data bucket. For example, the user identifier may have a metadata tag attributed thereto that indicates a data bucket that the user identifier has previously been assigned to.

Application server(s)1224may, in some embodiments, include one or more user experience modules1216. User experience module(s)1216may be configured to receive the user identifier and corresponding data bucket assignment from data bucket assigner system1214, and may retrieve an appropriate user experience to be rendered for that user identifier. For example, user experience module1216may determine that a user identifier that is received is attributed with a control data bucket, and therefore is to be provided with a control user experience1230. As another example, user experience module1216may determine that a user identifier that is received is attributed with a test data bucket, and therefore is to be provided with a test user experience1240. In response, application server(s)1224may be configured to send data representing the appropriate user experience (e.g., one of control user experience1230and test user experience1240) to the user device (e.g., user device110) associated with the user identifier that the request is associated with.

FIG.13is an illustrative flowchart of an exemplary process for providing user experience(s) to data bucket(s) and computing user metric(s), in accordance with various embodiments of the present teachings. Process1300may, in a non-limiting embodiment, begin at step1302. At step1302, a user identifier may be retrieved. For instance, upon a request being received, a user identifier associated with the user device that provided the request may be determined. The request, in some embodiments, may correspond to a request for a user experience to be provided, or a general request for content. For example, inputting of a URL into a web browser by user102on user device110may correspond to one type of request. Alternatively, a request may correspond to an interaction with content already rendered by user device110, such as, and without limitation, a click on a hyperlink, a click on an advertisement, a scroll, a tap, a swipe, or any other type of interaction, or any combination thereof. In any of the aforementioned instances, the user identifier associated with the received request may be determined by user ID receiver system1212. In some embodiments, however, the user identifier may previously be determined, and therefore at step1302, the user identifier may be retrieved from local storage (e.g., memory/storage1204).

At step1304, the user identifier may be provided to a hash function system. The hash function system may be part of data bucket assigner system1214. The hash function, which may also be referred as a randomizer function, may be configured to randomly assigned the user identifier to a particular data bucket, which may be associated with a particular user experience (e.g., control user experience1230, test user experience1240). At step1306, a seed value for the hash function for each online experiment may be determined. As mentioned previously, each experiment may exist on a layer in the multi-layer experimental platform. Each layer may include a unique seed value for the hash function, which is used for randomly assigning user identifiers to data buckets associated with that layer. At step1308, a hash value for the user identifier for each layer may be determined. For example, as seen inFIG.11, a same user identifier may be assigned to a different hash value in different layers. Persons of ordinary skill in the art will recognize that each user identifier need not be assigned to each layer of the multi-layer experimental platform. The generated hash value may also indicate the data bucket with which the user identifier is to be assigned.

At step1310, the user identifier is assigned to a data bucket. For example, the user identifier may be assigned to a data bucket based on the hash value. At step1312, a user experience associated with the data bucket that the user identifier has been assigned to may be provided to a corresponding user device. For example, if the user identifier has been assigned to a control data bucket, then the corresponding user device associated with that user identifier may be provided with a control user experience (e.g., control user experience1230). As another example, if the user identifier has been assigned to a test data bucket, then the corresponding user device associated with that user identifier may be provided with the test user experience (e.g., test user experience1240). At step314, one or more use engagement metrics may be computed. For example, a days visited metric, a page view metric, and/or a sessions-based metric may be computed.

FIG.14Ais an illustrative diagram of an exemplary discrepancy detection system, in accordance with various embodiments of the present teachings. Discrepancy detection system1040, in a non-limiting embodiment, may include an Extract Transform and Load (“ETL”) system1410, a visualization system1420, and an analytics data store1415. ETL system1410and visualization system1420, in one embodiment, may include one or more processors1402, memory/storage1404, and communications circuitry1406. In some embodiments, processor(s)1402, memory/storage1404, and communications circuitry1406may be substantially similar to processor(s)402, memory/storage404, and communications circuitry406ofFIG.4, and the previous description may apply.

ETL system1410, in some embodiments, may be configured to receive user engagement data from user engagement database1005. ETL system1410may aggregate the user engagement data such that hourly and daily user engagement data is generated, which ETL system1410may in turn provide to analytics data store1415for storage. For example, ETL system1410may call on user engagement database1005periodically (e.g., every few seconds, every minute, every few minutes, every hour, etc.) to obtain user engagement data representing user activities with experimentation platform1030that occurred. Upon receipt, ETL system1410may aggregate temporally common data, which may be based on a particular temporal period with which the aggregation is to occur, and may generate the aggregated user engagement data and send the aggregated user engagement data to analytics data store1415for storage. In some embodiments, user engagement database1005may perform and/or obtain some or all of the aggregated user engagement data, and thus when user engagement database1005is called on by ETL system1410, the previously aggregated user engagement data may be provided to ETL system1410and then transmitted to analytics data store1415.

Analytics data store1415, in the illustrative embodiment, stores the aggregated user engagement data for use in real time data analytics and reporting. Analytics data store1415may, in some embodiments, be configured to perform/facilitate aggregation of user engagement data, however persons of ordinary skill in the art will recognize that this is merely exemplary. In one non-limiting embodiment, analytics data store1415may be a column-oriented distributed data store, however any suitable data structure configuration may be employed for storing the data received from ETL system1410.

Visualization system1420may, in some embodiments, be configured to process aggregated user engagement data received from analytics data store1415, and may generate visualization data representing one or more graphics (e.g., images, text, video, etc.) to be displayed within a user interface1430for a user. Visualization system1420may be configured to employ various filters for visualizing and segmenting data for review by an experiment designer (e.g., user102). For example, a visualization of a discrepancy between stamped user identifiers (e.g., user identifiers that have been properly flagged for a particular experiment) and non-stamped user identifiers (e.g., user identifiers that have been mistakenly or mysteriously assigned to a data bucket of an experiment).

FIG.14Bis an illustrative flowchart of an exemplary processor for providing visualization data for various metrics, in accordance with various embodiments of the present teachings. Process1450, in one embodiment, may begin at step1452. At step1452, user engagement data may be retrieved. For example, user engagement data representing user activities/user engagement information with content associated with an online experiment rendered by experimentation platform1030may be retrieved from user engagement database1005.

At step1454, aggregate user engagement data may be generated for one or more temporal durations. In some embodiments, ETL system1410may generate the aggregate user engagement data. For example, the user engagement data may be aggregated for hourly user engagement data and/or daily user engagement data, or any other suitable temporal duration of user engagement data. At step1456, the aggregate user engagement data that was generated by ETL system1410may be provided to analytics data store1415, and at step1458, the aggregate user engagement data may be stored by analytics data store1415. At step1460, a data visualization request may be received by visualization system1420. The data visualization request may be received from user device110, from the experiment designer, or may be received automatically upon each instance of user engagement data being aggregated by ETL system1410. At step1462, the aggregate user engagement data may be obtained from analytics data store1462. In some embodiments, the aggregate user engagement data may be obtained for a particular amount of time, which may be specified by the data visualization request. For example, the data visualization request may indicate the discrepancy data for a past seven days is to be presented, and therefore visualization system1420may obtain seven days' worth of aggregate user engagement data from analytics data store.

At step1464, one or more metrics reflective of one or more user engagement parameters to be visualized may be computed. For example, a discrepancy between a number of user identifiers assigned to a data bucket for an online experiment as compared to a number of user identifiers expected to be assigned to the data bucket may be determined. At step1466, visualization data representing the computed metric(s) may be generated. For example, visualization data representing the discrepancy between the expected size of a data bucket (e.g., number of user identifiers assigned to that data bucket) and the actual size of the data bucket (e.g., number of user identifiers receiving a metadata tag for that data bucket) may be generated. At step1468, the visualization data may be provided to the user interface such that the computed metrics may be visualized by one or more users and/or experiment designers. For example, graphs1800and1900ofFIGS.18and19, respectively, may correspond to various types of visualization data that may be rendered by a user interface to detail discrepancy information associated with user engagement data.

FIG.15Ais an illustrative diagram of an exemplary ETL system, in accordance with various embodiments of the present teachings. In the non-limiting embodiment, ETL system1410may include an experiment monitoring layer module1510, a quality bucket stamping module1520, and a data processing module1530. Each of experiment monitoring layer module1510, quality bucket stamping module1520, and data processing module1530may include one or more processors1502, memory/storage1504, and communications circuitry1506. Processor(s)1502, memory/storage1504, and communications circuitry1506may, in some embodiments, be substantially similar to processor(s)402, memory/storage404, and communications circuitry406ofFIG.4, and the previous description may apply.

Experiment monitoring layer module1510may, in one embodiment, be configured to receive a setup request to set up an online experiment, or to set up discrepancy monitoring for an online experiment. In response to the request, experiment monitoring layer module1510may create a monitoring layer within the experimental platform (e.g., experimental platform200,1050). The monitoring layer may be a special layer which will be used for tracking quality issues associated with one or more experiments operating within the multi-layer experimentation platform1050. The monitoring layer may, for instance, include a single data bucket, which may be referred to as a “quality bucket,” and this data bucket uses 100% of the monitoring layer.

Quality bucket stamping module1520may be configured to assign a metadata tag to each user identifier placed into the monitoring layer. This process, which may also be referred to as “stamping,” causes each user identifier to be attributed to the monitoring layer. Therefore, as platform1050is a multi-layer platform, the user identifiers not assigned into the monitoring layer may be subject to one or more underlying issues that may lead to discrepancies between actual data bucket size and expected data bucket size.

Data processing module1530may, in some embodiments, be configured to provide process the user engagement data that has been stamped by quality bucket stamping module1520. For instance, data processing module1530may aggregate the user engagement data for various temporal durations. As an illustrative example, aggregated user engagement data may be generated every hour, every few hours, every day, and the like, and may be provided to analytics data store1415for storage. In some embodiments, one or more thresholds1515may also be obtained by data processing module1530and stored by analytics data store such that visualization system1430may generate appropriate visualization data. Threshold(s)1515may, for example, indicate whether or not a discrepancy is large enough to invalidate results of an online experiment.

FIG.15Bis an illustrative flowchart of an exemplary processor for generating discrepancy data, in accordance with various embodiments of the present teachings. Process1550may begin, in one embodiment, at step1552. At step1552, a monitoring layer may be generated for a multi-layer experimentation platform. For example, a monitoring layer may be generated within multi-layer experimentation platform1050. The monitoring layer may include a single data bucket, which may occupy 100% of the monitoring layer.

At step1554, user engagement data may be generated by stamping each user identifier with an additional metadata tag. For instance, each user identifier may, in addition to be distributed to one or more of the layers of multi-layer experimentation platform1050, be distributed to the monitoring layer. Therefore, these user identifiers may further receive a metadata tag associated with the monitoring layer or, more particularly, attributed to a data bucket of the monitoring layer.

A user agent string may include a user identifier (e.g., a string of characters unique to a user device, browser, etc.), followed by one or more additional indicators. In some embodiments, some of the additional indicators may indicate whether that user identifier is associated with a particular layer of multi-layer experimentation platform. As an illustrative example, multi-layer experimentation platform1050may include a monitoring layer and a first layer. The first layer may include a single online experiment including two data buckets: a control data bucket associated with a control user experience, and a test data bucket associated with a test user experience. The user agent string may, therefore, include a certain amount of characters representative of a user identifier (e.g., abcd1234), followed by two additional characters. The first additional character may correspond to a logical 1 (e.g., “1”), which may indicate that the user identifier has been assigned to the monitoring layer's data bucket. The second additional character may correspond to either a logical 1 or a logical 0 (e.g., “0”), indicating that the user identifier has been assigned to either the control data bucket of the first layer or the test data bucket of the second layer, respectively. Therefore, when the user engagement data is stamped, each user identifier is attributed with an additional character in their user agent string (e.g., logical 1) reflective of the fact that the user identifier has been attributed to the monitoring layer.

At step1556, the user engagement data may be aggregated for various temporal durations. For example, the user engagement data may be aggregated such that hourly data, daily data, weekly data, and the like may be obtained. At step1558, data analytics may be performed to the user engagement data. For example, user engagement metrics may be computed describing one or more user engagement parameters based on the user engagement data. In some embodiments, the data analytics that are performed may further be performed to determine a number of user identifiers included within the user engagement data that include the monitoring layer's metadata tag. For example, the user engagement data for a first data bucket of an experiment may indicate that 70% of the user identifiers include a metadata tag associated with the monitoring layer. This may indicate, for example, that 30% of the user identifiers do not include the monitoring layer metadata tag, indicating that there is an abnormality occurring within the system architecture that causes those user identifiers to not be attributed with the additional metadata tag. Therefore, the results of the first experiment may be erroneous, or not completely accurate, as the difference between the number of user identifiers expected to be within the data bucket as opposed to the number of user identifiers actually within the data bucket is greater than expected.

At step1560, the data analytics may be provided to a user interface. For instance, discrepancy detection system1040may provide the data analytics to visualization system1420. At step1562, unstamped data analytics may be obtained. In some embodiments, unstamped data analytics may be determined by data processing module1530. For example, as described above, unstamped data analytics may indicate a number of user identifiers within each data bucket that lack the additional metadata tag associated with the monitoring layer. At step1564, discrepancy data may be generated that indicates a difference between events associated with stamped user identifiers and events associated with unstamped user identifiers. The discrepancy data, in some embodiments, may take the user engagement data obtained and may determine the difference between the number of user identifiers including the metadata tag associated with the monitoring layer and the number of user identifiers expected to be in the data bucket. In some embodiments, data processing module1530may generate the discrepancy data, and may also obtain one or more thresholds1515. The thresholds may be used to determine whether the discrepancy is large enough that results of the experiment being performed re unable to be used. In some embodiments, the discrepancy data may further be provided to visualization system1420for rendering on a user interface.

FIG.16is an illustrative graph of exemplary discrepancy data indicating a discrepancy for a data bucket of an online experiment, in accordance with various embodiments of the present teachings. Graph1600illustrates, in the non-limiting embodiment, discrepancies between stamped used identifiers and unstamped user identifiers over time for a particular experiment. For example, unstamped user identifiers1602and stamped user identifiers1604may exist in a same data bucket of an experiment. This implies that some user identifiers (e.g., unstamped user identifiers1602) may be receiving one user experience that is not being measured by experimentation platform1030. Furthermore, if the total number of user identifiers is the desired amount for a data bucket, then without determining that some of those user identifiers are unstamped, an experiment designer may erroneously read in findings to results of their online experiment. Additionally, user experience may suffer as the particular user experience provided to unstamped user identifiers1602may be unknown to the experimentation platform, as the data bucket assignment process may yield null results.

FIG.17is an illustrative graph of balanced data buckets where a portion of the user identifiers are not stamped, in accordance with various embodiments of the present teachings. In the non-limiting embodiment, graph1700describes a situation where data bucket size may be balanced across various data buckets of an online experiment. However, in the illustrative embodiment, roughly half of the user activity events receive a quality layer data bucket stamp. Therefore, even though A/A validation and A/B validation for the online experiment may indicate that the size of the data buckets is evenly distributed for each data bucket, at least some of the identifiers of those data buckets have not been stamped with the quality layer's metadata tag.

FIG.18is an illustrative diagram of various visualization data for rendering on a user interface detailing data bucket discrepancy for an online experiment, in accordance with various embodiments of the present teachings. User interface1800may, in some embodiments, display discrepancies in data bucket size over time. User interface1800may include various types of data analytics, as described above, to indicate an amount of unstamped user identifiers within a data bucket. For example, for a particular website, one or more webpages may be tracked, and a determination of a number of user identifiers that have been stamped and a number of user identifiers that have not been stamped may be displayed for each webpage of the website as they change over time. Furthermore, totals for a daily traffic volume, stamped user identifier volume, and changes in these parameters may also be visualized by user interface1800.

FIG.19is an illustrative diagram of an exemplary user interface detailing data bucket discrepancy over time, in accordance with various embodiments of the present teachings. In the non-limiting embodiment, graph1900describes one example graphic that may be rendered by user interface1800. For instance, a ratio of the number of stamped user identifiers as compared to a number of unstamped user identifiers over a particular amount of time may be presented via graph1900. As seen from graph1900, a number of unstamped user identifiers may have accounted for a large portion of the user activity early on, however over time, with implementation of a monitoring layer within the experimentation platform, the ratio may decrease. Therefore, fewer and fewer unstamped user identifiers may remain in a data bucket for an experiment, thereby increasing the reliability of the experiment's results.

An additional benefit to implementing the monitoring layer to detect data bucket discrepancies is the ability to quickly and effectively identify experiments that are experiencing data bucket discrepancies. For example, at a time1902, discrepancy detection system1040may detect that a ratio of unstamped user identifiers to stamped user identifiers for a particular data bucket may be in excess of a threshold (e.g., threshold1515). In response, an experiment designer may dive into the details associated with the setup of the experiment within online experimentation platform1030, and may rectify any issues that may exist. As seen from graph1900, after time1902, when an underlying issue was discovered, the number of unstamped user identifiers infiltrating the experiment's data bucket decreases.

FIG.20is an illustrative diagram of an exemplary system for identifying data bucket overlap with online experiments, in accordance with various embodiments of the present teachings. In the non-limiting example embodiment, a system2000may include one or more user devices2010a,2010b, and2010c. User devices2010a,2010b, and2010cmay be substantially similar to user device110ofFIG.1, and the previous description may apply. Furthermore, additional details associated with user devices2010a,2010b, and2010c, which collectively may be referred to as user device(s)2010, are described below. Although only three user devices2010a-care shown within system2000, persons of ordinary skill in the art will recognize that any number of user devices may be employed, and the use of three user devices is merely exemplary.

User device2010may, in some embodiments, include one or more processors2002, memory/storage2004, communications circuitry2006, a display or display functionality2008, one or more input components2012, and one or more output components2014. In some embodiments, processor(s)2002, memory/storage2004, and communications circuitry2006may be substantially similar to processor(s)402, memory/storage404, and communications circuitry406ofFIG.4, and the previous description may apply. In some embodiments, memory/storage2004may be further configured to store configuration data2016, which as described in greater detail below, may store user experience information indicating a user experience to be provided by user device2010when user device2010accesses experimentation provider system2020.

Display2008may correspond to any suitable type of visual output component capable of presenting images and/or videos to user102via user device2010. Various types of displays include, but are not limited to, liquid crystal displays (“LCD”), monochrome displays, color graphics adapter (“CGA”) displays, enhanced graphics adapter (“EGA”) displays, variable graphics array (“VGA”) display, or any other type of display, or any combination thereof. Display2008may, in some embodiments, include a touch screen, such as a touch screen including capacitive sensing panels capable of recognizing touch inputs thereon. For instance, a touch screen may correspond to a projected capacitive touch (“PCT”), screen include one or more row traces and/or driving line traces, as well as one or more column traces and/or sensing lines. As described in greater detail herein, display2008may be configured to render a user experience2040thereon. User experience2040may correspond to one type of user experience associated with an online experiment with which user device2010is participating in via user device2010accessing experimentation provider system2020. For example, user experience2040may correspond to a control user experience or a test user experience, and the type of user experience2040rendered by display2008may depend on configuration data2016. Furthermore, a type of user experience2040to be rendered by display2008may be determined by experimentation provider system2020, based on a data bucket assignment process, which is described in greater detail above.

Input components2012may correspond to any suitable input mechanism. For example, continuing the aforementioned discussion, if display2008includes touch screen capabilities, display screen2008may also function as an input component. Additionally, or alternatively, input components2012may include, but are not limited to, a mouse, a stylus, a keyboard, voice processing components, a microphone, facial/object recognition functionality, and/or any other suitable input component, or any combination thereof.

Output component(s)2014may correspond to any suitable output mechanism for user device2010. For example, continuing the aforementioned discussion, display2008may correspond to one type of output component (e.g., a visual output component). In some embodiments, output component(s)2014may correspond to one or more audio output components and/or one or more haptic components. For example, user device2010may include one or more speakers and/or transducers for outputting sound, and/or one or more haptic devices for output haptic feedback (e.g., vibrating components). In some embodiments, output component(s)2014may further include one or more interfaces that allow user device2010to communicate with one or more additional components, devices, and/or systems. For instance, output component(s)2014may include a USB interface, and HDMI interface, and/or any other interface component as persons of ordinary skill in the art will recognize.

System2000may further include an experimentation provider system2020, which may be in communication with user device(s)2010via network(s)104. In some embodiments, experimentation provider system2020may be configured to provide an online experiment that is to be experienced and interacted with by user devices2010. For instance, experimentation provider system2020may be substantially similar to experimentation system1030ofFIG.10, and the previous description may apply.

System2000may further include a data inconsistency detection system2030. Data inconsistency detection system2030may be configured, as described in greater detail below, to determine whether any user identifiers have been assigned to two or more data buckets of experimentation provider system2020. As mentioned previously, user identifiers may be assigned to data buckets for online experiments, and user devices whose user identifiers are assigned to same data bucket may receive a same user experience when accessing the online experiment. However, in the mobile realm, for example, slight differences occur between that of the non-mobile realm.

As mentioned previously, when a user device (e.g., user device110) access an experimentation system (e.g., experimentation system1030), a user identifier associated with that user device may be assigned to a data bucket for an experiment. This process may occur randomly, and if performed accurately, attributed each identifier to one data bucket in one layer of the multi-layer experimentation platform (if a multi-layer experimentation platform is employed).

In the mobile setting, however, there are slight variations to the data bucket assignment process. For instance, for mobile devices (e.g., user devices2010a-c), instead of assigned a user identifier to a data bucket, and then the experimentation system storing the user identifier and assigned data bucket within their system, the mobile device may store an indication of the data bucket assigned thereto locally. For example, user device2010may store configuration data2016within memory/storage2004that indicates a user experience to be provided to user device2010. When user device2010first accesses experimentation provider system2020, in one embodiment, the data bucket assignment process may occur, and information indicating the corresponding user experience to be provided to user device2010may be stored within a configuration file locally on user device2010. The next instance that user device2010accesses experimentation provider system2020, the configuration file may transmit the corresponding user experience information to experimentation provider system2020, which in turn may provide the appropriate user experience2040to user device2010.

FIG.21is an illustrative graph indicating apparent data bucket size consistency over time, in accordance with various embodiments of the present teachings. In the illustrative, non-limiting embodiment, graph2100includes five different trend lines describing five different data buckets of an online experiment. As seen by graph2100, each of the five data buckets appears to have a substantially same number of user identifiers. For example, each of the five data buckets appears to have a sample size of approximately 1.7 million. Additionally, as seen by graph2100, the sample size of each of the five data buckets holds relatively steady compared with one another over time. This may appear as showing good distribution of identifiers to data buckets during experimentation, however there is no way of knowing whether or not one or more identifiers of one data bucket also is assigned to another data bucket or other data buckets. The assumption that there is no overlap of data bucket assignment for identifiers may prove to be detrimental to an experimentations validity and overall user experience. For instance, if one user device receives two or more user experiences associated with a same online experiment, then the results associated with that experiment may not be accurate. Furthermore, this can provide a poor user experience as the user may receive one user experience when accessing a webpage at one instance, but may receive a different user experience when accessing the same webpage at another instance.

FIG.22is an illustrative flowchart of an exemplary process for sending user experience data to a user device based on a configuration file associated with the user device, in accordance with various embodiments of the present teachings. Process2200may, in a non-limiting embodiment, begin at step2202. At step2202, a notification may be received by experimentation provider system2020that an application has launched on a user device. For instance, user102may open an application on user device2010. An application, as described herein, may correspond to a web application that allows a user to view a website in a mobile setting. For example, the application, which may be stored within memory/storage2004, may include data that, when read by processor(s)2002, causes a web view of the website to be rendered by display2008. As an illustrative example, the application may include JavaScript code that describes the layout and format to be used to render the content associated with the website. The differences between a web view version of a website and a browser view of the website may be primarily in appearance, as the content may be substantially the same, however persons of ordinary skill in the art will recognize that this is merely exemplary. For example, a social media site may have substantially the same content but a different appearance when accessed via one's mobile device as compared to a web browser.

In response to opening the application, which user102may accomplish by tapping an icon on display2008, if display2008includes touch screen functionality, pressing a button, or performing any other suitable action. Upon determining that user102has invoked the application, the application may generate a notification to be sent by user device2010to experimentation provider system2020(or to the corresponding website) indicating that user device2010is requesting that content be rendered thereby. In some embodiments, the notification may include configuration data2016, which may indicate, amongst other aspects, a user identifier associated with user device2010(e.g., a device identifier, MAC address, IP address), a location of user device2010, a timestamp with which the request was generated, and/or any other suitable information, and/or any combination thereof.

At step2204, a user experience stored by a configuration file may be determined. In response to receiving the notification, experimentation provider system2020may determine that configuration data2016includes a configuration file, which may also be referred to as configuration information and/or user experience information, indicating a user experience to be provided to user device2010. For example, configuration data2016may include information indicating that user device2010has been assigned to a control data bucket. Therefore, in the illustrative example, experimentation provider system2020may determine that a control user experience is to be provided to user device2010.

At step2206, user experience data may be obtained. The user experience data may include visual data, audio data, text data, and/or any other type of data, reflective of the corresponding user experience to be provided to user device2010. For instance, the user experience data may include a particular web view rendering of a website's content with which display2008may be configured to display. In some embodiments, the user experience data may include information capable of being read by user device2010(e.g., JavaScript information) such that user device2010may render the appropriate content without having to access the content via a web browser functionality of user device2010. At step2208, the user experience data may be sent to the user device that the notification was received from. In some embodiments, some aspects of the user experience, and thus portions of the user experience data, may be stored locally by user device2010(e.g., within memory/storage2004), however persons of ordinary skill in the art will recognize that this is merely exemplary.

FIG.23is an illustrative diagram of an exemplary data inconsistency detection system, in accordance with various embodiments of the present teachings. Data inconsistency detection system2030, as mentioned above, may be in communication with experimentation provider system2020directly and/or via network(s)104. User activity data associated with an online experiment being provided by experimentation provider system2020and provided to user device(s)2010may additionally be received by data inconsistency detection system2030. In some embodiments, data inconstancy detection system2030and experimentation provider system2020may receive the user activity data in parallel, however this is merely exemplary.

Data discrepancy detection system, in a non-limiting embodiment, may include a data bucket identifier2310, a user identifier extraction system2320, a user identification comparison system2330, and a data bucket abnormality system2340. Each of data bucket identifier2310, user identifier extraction system2320, user identification comparison system2330, and data bucket abnormality system2340may include one or more processors2302, memory/storage2304, and communications circuitry2306. Processor(s)2303, memory/storage2304, and communications circuitry2306may, in some embodiments, be substantially similar to processor(s)402, memory/storage404, and communications circuitry406ofFIG.4, and the previous description may apply.

In some embodiments, data inconsistency detection system2030may include multi-layer experimentation platform2312, which may correspond to substantially similar experimentation layers provided by experimentation provider system2020. For instance, data inconsistency detection system2030may reproduce the experimentation platform used by experimentation provider system2020for accurately monitoring inconstancies in data buckets. In some embodiments, however, platform2312may be optional, and thus may be omitted from data consistency detection system2030. In this particular scenario, information regarding the activities and actions associated with each layer of the experimentation platform may be received from experimentation provider system2020. In some embodiments, where multi-layer experimentation platform2312resides on data inconstancy detection system2030, platform2312may be configured substantially similar to platform1050ofFIG.10and/or platform200ofFIG.2, and the previous descriptions may apply.

As described in greater detail above, multi-experimentation platform2312may include one or more layers, such as a first layer2314aand a second layer2314b. In the illustrative embodiment, first layer2314aand second layer2314bmay be orthogonal to one another, such that a user identifier may be capable of being placed in one data bucket in one layer and another data bucket in another layer. First layer2314amay include one or more data buckets2316a, and second layer2314bmay include one or more data buckets2316b. In one embodiment, data buckets2316aand2316bmay each correspond to two or more data buckets. For example, each of layers2314aand2314bmay include at least a control data bucket and one test data bucket.

User activity data corresponding to activities performed to content associated with each of the layers of platform2312may be provided to data bucket identifier system2310. Upon receipt, data bucket identifier system2310may be configured, in some embodiments, to determine a data bucket that a corresponding user identifier is associated. For example, the user activity data may include configuration data2016, which indicates a data bucket that a corresponding user device has been assigned to. Upon accessing platform2312, a user experience is provided to user device2010based on an assigned data bucket that configuration data2016identifies. Data bucket identifier2310, as described in greater detail below, may be configured to determine the user experience based on which data bucket the corresponding user activity data is obtained from.

User identifier extraction system2320may, in some embodiments, be configured to determine a user identifier associated with the user activity data. For instance, in response to determine which data bucket a particular user activity or activities relate(s) to, user identifier extraction system2320may determine the user identifier associated with user device2320. User identifier extraction system2320may then provide the user identifier to user ID database2315. User ID database2315may, for instance, store pairing information indicating a user identifier and a corresponding data bucket or buckets that the user identifier is assigned to. In some embodiments, user ID database2315may further store a corresponding layer that each of the data buckets for that user identifier is associated with.

User identification comparison system2330may, in some embodiments, be configured to compare the user identifier associated with a particular user activity event with additional user identifiers stored by user ID database2315. As an illustrative example, a first user identifier, ID 1, may be indicated as being associated with data bucket 1, DB 1, of layer 1, L1. Upon extraction by user identifier extraction system2320, user identification comparison system2330may compare ID1 with each of the user identifiers that are also associated with L1. For instance, if layer 1 includes two data buckets, DB1 and DB 2, ID 1 may be compared with each of the user identifiers stored within user ID database2315as being associated with DB 2. If ID 1 does not match any of the user identifiers associated with DB 2, then this may indicate that there is no overlap between DB 1 and DB 2, at least with relation to ID 1. However, if ID 1 does match a user identifier associated with DB 2, then this may indicate that there is overlap between DB 1 and DB 2, as configuration data2016associated with ID 1 may be attributed to both DB 1 and DB 2, meaning user device2010may be capable of receiving two different user experiences. A similar process may also be performed across each of the various layers included within platform2312to determine overlap existing for each user identifier between two or more data buckets of each layer.

Data bucket abnormality system2340may be configured to receive the comparison results from system2330to determine a severity of the inconsistency. For example, a number of overlapping user identifiers between two or more data buckets of a same layer may be received by system2340. System2340may then be configured to compare that number to experimental overlap threshold(s)2325to determine the extent of the overlap. In some embodiments, threshold(s)2325may be set by an experiment designer, or they may be defaulted to certain parameters. For example, overlaps of 10% or more of total user traffic may signify inconsistencies large enough to invalidate experimental results, however this is merely exemplary.

FIG.24is an illustrative flowchart of an exemplary processor for detecting data bucket inconsistency, in accordance with various embodiments of the present teachings. In the non-limiting embodiment, process2400may being at step2402. At step2402, identification metadata indicating a user identifier associated with a user device2010may be received. For instance, the identification metadata may be received by platform2312and/or experimentation provider system2020. At step2404, the user identifier may be assigned to a data bucket in an experiment. For instance, upon first interacting with the online experiment, the user identifier may be assigned to a data bucket associated with the experiment. For example, the user identifier may be assigned to a control data bucket, associated with a control user experience, or a test data bucket, associated with a test user experience.

At step2406, a user experience associated with the assigned data bucket may be provided to the user device. For example, if the user identifier was assigned to the control data bucket, then the control user experience may be provided to that user device2010. In some embodiments, user device2010may store information indicating which user experience they have been attributed for the online experiment within configuration data2016. At step2408, experiment information may be determined. In some embodiments, the experiment information may indicate, amongst other aspects, a number of data buckets associated with the experiment, a number of additional experiments occurring within the platform, a number of user identifiers associated with the experiment, and the like.

At step2410, data bucket identification data from each data bucket in the experiment may be obtained. The data bucket identification data may include a listing of each user identifier stored within user ID database2315for a particular experiment. At step2412, a user identifier for a user activity event associated with the user activity data that is received may be identified. At step2414, the user identifier and the data bucket identification data may be provided to comparison system2330. Comparison system2330, at step2416, may be configured to determine whether the user identifier overlaps with any user identifiers associated another data bucket. For instance, the user identifier associated with one user activity event may be compared with the user identifiers associated with each other data bucket of an experiment.

If, at step2416, comparison system2330determines that no overlap occurs, then process2400may proceed to step2424, where the user experience(s) may continue being rendered. For example, an online experiment may include a first data bucket and a second data bucket, DB 1 and DB 2, respectively. A first user identifier ID 1 assigned to DB 1 may be compared with each user identifier associated with DB 2. If ID 1 does not match any of the user identifiers associated with DB 2, then the user experience being provided to ID 1 may continue to be provided thereto.

However, if at step2416, comparison system2330determines that the user identifier does overlap with a user identifier in another data bucket, then process2400may proceed to step2418. At step2418, an amount of overlapping user identifiers may be determined. Continuing the example above, if ID 1 substantially matches another user identifier, ID 2, of DB 1, then the amount of overlapping user may correspond to one. However, if there are more user identifiers of DB 1 that also are associated with DB 2, then the amount of these identifiers may be determined.

At step2420, a determination may be made as to whether or not the amount is greater than a threshold. For instance, the amount may be compared with threshold2315by system2340to determine an extent of the overlap. If, at step2420, it is determined that the amount is less than or equal to the threshold, then process2400may proceed to step2424. However, if at step2420it is determined the amount of overlapping user identifiers is greater than the threshold, then process2400may proceed to step2422. At step2422, an inconsistency flag may be generated. For instance, data bucket abnormality system2340may generate data bucket validity data including an inconsistency flag. The inconsistency flag may indicate to an experiment designer that there is significant overlap of users between data buckets, and therefore the results of the experiment may be inaccurate.

FIG.25Ais an illustrative diagram of an exemplary data bucket identifier system, in accordance with various embodiments of the present teachings. Data bucket identifier system2310, in the non-limiting embodiment, may include an experiment identifier system2510, a user experience information extractor2520, and a data bucket counter2530. Each of experiment identifier system2510, user experience information extractor2520, and data bucket counter2530may include one or more processors2502, memory/storage2504, and communications circuitry2506. In some embodiments, processor(s)2502, memory/storage2504, and communications circuitry2506may be substantially similar to processor(s)402, memory/storage404, and communications circuitry406ofFIG.4, and the previous descriptions may apply.

In the non-limiting embodiment, an analysis request received by data bucket identifier system2310may be provided to experiment identifier system2510. Experiment identifier system2510may, in some embodiments, be configured to determine an experiment with which the analysis request is associated with. For example, experiment platform2312may include multiple experiments corresponding to different layers of platform2312. Therefore, upon receipt of the request, experiment identifier2510may determine which experiment that the current request is associated. Additionally, in some embodiments, experiment identifier system2510may be configured to determine a corresponding layer of platform2312that the experiment occupies.

User experience information extractor system2520may, in some embodiments, be configured to determine user experience information associated with the identified experiment. For example, user experience information extractor system2520may determine a number of different data buckets associated with the experiment, and therefore a number of different user experiences capable of being provided for that experiment. Furthermore, user experience information extractor system2520may be configured to determine which user experience is a control user experience and which user experience, or experiences, is/are test user experiences.

Data bucket counter system2530may, in some embodiments, be configured to determine data bucket information associated with each data bucket of the experiment. For example, data bucket counter system2530may determine a total user identifier count (e.g., a sample size) associated with the each data bucket of the experiment, and thus a number of user devices2010that will, or that should, receive a particular user experience. Data bucket information, in one embodiment, may thus be output by data bucket counter system2530, and therefore output by data bucket identifier system2310, indicating a number of distinct data buckets associated with an experiment, and an approximate sample size of the data buckets.

FIG.25Bof an illustrative flowchart of an exemplary process for determining data bucket information associated with each user experience of an online experiment, in accordance with various embodiments of the present teachings. Process2550may, in a non-limiting embodiment, begin at step2552. At step2552, a data analysis request may be received by data bucket identifier system2310. In one embodiment, the request may be received by experiment identifier system2510. At step2554, an experiment to be analyzed may be identified by experiment identifier system2510. For instance, experiment identifier system2510may determine an experiment associated with the request (e.g., an experiment associated with user device2010with which the request was received).

At step2556, user experience information associated with the experiment may be obtained. For instance, user experience information extractor system2520may determine a number of data buckets associated with the identified experiment, and may determine the number and types of user experiences that are associated with each of the data buckets. At step2558, data bucket information associated with each user experience associated with the experiment may be determined. For example, data bucket counter system2530may determine a total user identifier count (e.g., a sample size) associated with the each data bucket of the experiment, and thus a number of user devices2010that will, or that should, receive a particular user experience.

FIG.26Ais an illustrative diagram of an exemplary user identifier extraction system, in accordance with various embodiments of the present teachings. User identifier extraction system2320may, in some embodiments, include a data bucket separator system2610, a user identifier collector system2620, and a user/experiment information system2630. Each of data bucket separator system2610, user identifier collector system2620, and user/experiment information system2630may include one or more processors2602, memory/storage2604, and communications circuitry2606. Processor(s)2602, memory/storage2604, and communications circuitry2606may, in some embodiments, be substantially similar to processor(s)402, memory/storage404, and communications circuitry406ofFIG.4, and the previous description may apply.

Data bucket separator system2610may, in some embodiments, be configured to receive data bucket user information. For instance, data bucket separator system2610may obtain data including user identifiers associated with an experiment (or more than one experiment), and corresponding data bucket metadata tags associated with those user identifiers. Each user identifier of the experiment should be associated with one data bucket, however in some embodiments, as described herein, additional data buckets of the same experiment may be assigned to a user identifier. Data bucket separator system2610may further be configured to separate out the user identifiers into groups by data bucket. For example, if there are two data buckets, DB 1 and DB 2, then data bucket separator system2610may group user identifiers together that include a metadata tag associated with DB 1, and may also group user identifiers together that include a metadata tag associated with DB 2.

User identifier collector2620may, in some embodiments, be configured to collect user identifiers for each data bucket together. For instance, first data bucket user identifier data may be generated by user identifier collector system2620, which may include the various user identifiers associated with the first data bucket. Similarly, second data bucket user identifier data may be generated including the various user identifiers associated with the second data bucket. If more than two data buckets exists for a particular experiment, then data including the user identifiers for that bucket may also be included. Additionally, in some embodiments, user identifier data for data buckets of other experiments may also be collected by user identifier collector2620such that data for each data bucket of each experiment may be obtained.

User/experiment information system2630may, in some embodiments, be configured to determine the experiment associated with each collection of data generated by user identifier collector system2620. For instance, system2630may determine a name associated with each experiment, a sample size designed for the data buckets of the experiments, and/or any other additional information, and/or any combination thereof. The user identification data and the experiment data may then be provided to user ID database2315for storage. User ID database2315may, therefore, be capable of storing data indicating, for each experiment existing in the experimentation platform1050, a number of data buckets associated with each experiment, an expected data bucket size of each experiment, and the user identifiers that have been assigned to those data buckets.

FIG.26Bis an illustrative flowchart of an exemplary processor for storing data indicating user identifiers and data buckets therefore, in accordance with various embodiments of the present teachings. Process2650, in one example embodiment, may begin at step2652. At step2652, data bucket user information may be received. For example, data bucket user information may be received by data bucket separator system2610of user identifier extraction system2320. At step2654, user identifiers may be separated by data bucket based on the data bucket user information. For instance, data bucket separator system2610may be configured to separate the user identifiers based on the data bucket with which they are attributed to. At step2656, the user identifiers for each data bucket of the experiment may be collected. For example, user identifier collector2620may collect, or otherwise group, user identifiers attributed with a common metadata tag associated with a same data bucket. In this way, all of the user identifiers having a metadata tag associated with a first data bucket may be grouped together, and all of the user identifiers having a metadata tag associated with a second data bucket may be grouped together. At step2658, data indicating the user identifiers and data buckets associated therewith may be stored by user ID database2315. In some embodiments, experiment information system2630may further provide information associated with the experiment(s) corresponding to those data buckets such that the information may also be stored by user ID database. For example, user ID database2315may store an expected sample size of a data bucket, and/or other information associated with the data buckets and experiments (e.g., whether a data bucket is a control data bucket or a test data bucket, a user experience associated with a data bucket, etc.).

FIG.27Ais an illustrative diagram of an exemplary user identification comparison system, in accordance with various embodiments of the present teachings. In the non-limiting embodiment, user identification system2330may include a user identifier selector system2710, a different data bucket user identifier collector system2720, and a comparison module2730. Each of user identifier selector system2710, different data bucket user identifier collector system2720, and comparison module2730may include one or more processors2702, memory/storage2704, and communications circuitry2706. In some embodiments, processor(s)2702, memory/storage2704, and communications circuitry2706may be substantially similar to processor(s)402, memory/storage404, and communications circuitry406, and the previous description may apply.

User identifier selector system2710, in one embodiment, may be configured to obtain user identification and experiment data from user ID database2315. In some embodiments, user identifier selector2710may be configured to select a user identifier from the received data, and may determine a data bucket associated with the selected user identifier. For example, user identification and experiment data may include, amongst other features, user identifier data representing a plurality of user identifiers associated with at least one experiment. Selector system2710may be configured to isolate one (or possibly more) user identifier from the data, with which to use for comparison purposes. In some embodiments, user identification comparison system2330may include multiple instances of selector system2710(and/or systems2720and2730) such that a comparison process may be performed for multiple user identifiers in parallel.

Different data bucket user identifier collector system2720may, in some embodiments, be configured to determine a data bucket associated with the selected user identifier, and may obtain user identifiers associated with each of the one or more other data buckets. For example, if the selected user identifier, ID 1, is associated with a first data bucket DB 1 of a first experiment, then system2720may collect all of the user identifiers associated with a second data bucket DB 2 of the first experiment. Similarly, system2720may be configured to collect user identifier data representing user identifiers of data buckets associated with other experiments also occurring within the experimentation platform.

Comparison module2730may, in some embodiments, be configured to receive the selected user identifier associated with a first data bucket and the plurality of user identifiers associated with at least a second data bucket used for comparison. Comparison module2730may then be configured to determine whether or not the selected user identifier also exists in the collection of user identifiers associated with the second data bucket. If so, then this may indicate that the selected user identifier has been attributed to more than one data bucket within a same experiment. Comparison module2730, and similarly systems2710and2720, may be configured to perform a similar procedure for each user identifier in each data bucket such that a total number of overlapping user identifiers may be determined. Comparison module2730, and thus system2330, may therefore be configured to generate overlapping user identifier data that indicates a number of overlapping user identifiers determined to exist for a given data bucket. Comparison module2730may further be configured to output data bucket size information indicating an expected/measured size of the data bucket.

FIG.27Bis an illustrative flowchart of an exemplary process for generating data indicating overlap, in accordance with various embodiments of the present teachings. Process2750, in a non-limiting embodiment, may begin at step2752. At step2752, user identifier data for an experiment may be received. For instance, user identifier data may be received by user identifier selector system2710from user ID database2315. At step2754, a first user identifier from a first data bucket may be isolated. The first user identifier may be selected at random, in some embodiments, by selector system2710.

At step2756, the first user identifier may be compared with user identifiers in additional data buckets. In some embodiments, system2720may obtain user identifier data representing user identifiers associated with a different data bucket than the selected user identifier's corresponding data bucket. In one embodiment, the different data bucket and the selected user identifier's data bucket are both associated with a same online experiment. At step2758, a determination may be made as to whether a match has been identified between the selected user identifier of one data bucket and any of the user identifiers associated with the different data bucket.

If, at step2758, it is determined that a match, or matches, have been found, then process2750may proceed to step2760. At step2760, count data indicating overlap between data buckets may be generated. In some embodiments, comparison module2730may generate count data indicating a number of matches found at step2758. For example, if the first user identifier is determined to match another user identifier of the different data bucket, then the count data may indicate that one overlap has been found between the two data buckets. If, however, there are more than two data buckets, then step2756and2758may similarly be performed for each of the other data buckets to determine if overlap between the first user identifier and the user identifiers of the other data buckets exist. Additional count data reflecting the overlap for that other data bucket(s) may also be generated.

After step2760, process2750may proceed to step2762. Similarly, if no match was identified at step2758, process2750may proceed to step2762. At step2762, a determination may be made as to whether the first user identifier isolated at step2754is a last identifier in the first data bucket. If not, then process2750may proceed to step2764. At step2764, a second user identifier may be isolated from the first data bucket, and process2750may proceed to step2756may repeat using the second user identifier. This process may loop until it is determined at step2762that the compared user identifier is the last identifier in the data bucket, or in other words, there are no more user identifiers to be compared. However, in some embodiments, multiple instances of systems2710,2720, and2730may be implemented within system2330, and therefore some of the comparisons of user identifiers with other identifiers may occur in parallel with one another.

If, at step2762, it is determined that the identifier user for comparison is in fact the last identifier associated with the data bucket (e.g., there are no more user identifiers left to perform comparisons with), then process2750may proceed to step2766. At step2766, the count data may be aggregated. For instance, each instance of an overlap being found may increment a counter associated with comparison module2730, and the aggregate count may be determined at step2766. At step2768, output data indicating a total overlap may be generated. Additionally, in some embodiments, output data indicating a total data bucket size may also be generated. For example, if 100 user identifiers of a first data bucket are also determined to exist in the second data bucket, then the output data may indicate that the overlap is 100. If the total number of user identifiers in both data buckets is 1,000 identifiers, then the overlap percentage/ratio may be referred to as 10% overlap, however persons of ordinary skill in the art will recognize that this is merely exemplary.

FIG.28Ais an illustrative diagram of an exemplary data bucket abnormality system, in accordance with various embodiments of the present teachings. Data bucket abnormality system2340may, in the illustrative non-limiting embodiment, include an overlap percentage system2810, a threshold excess determination system2820, and a data bucket inconsistency notification system2830. Each of overlap percentage system2810, threshold excess determination system2820, and data bucket inconsistency notification system2830may include one or more processors2802, memory/storage2804, and communications circuitry2806. Processor(s)2802, memory/storage2804, and communications circuitry2806may, in some embodiments, be substantially similar to processor(s)402, memory/storage404, and communications circuitry406ofFIG.4, and the previous description may apply.

Overlap percentage system2810, in some embodiments, may be configured to receive overlapping user identifier data from system2330, as well as data bucket size information, and may determine a percentage of overlap for a particular data bucket. For example, as mentioned previously, if 100 user identifiers of a first data bucket are determined to be associated with a second data bucket, and the total number of identifiers in the first data bucket is approximately 1,000 identifiers, then the percentage of overlap determined by system2810would be approximately 10%. Persons of ordinary skill in the art will recognize that a number of identifiers in each data bucket of a same experiment is substantially consistent across each data bucket, and therefore the percentage of overlap may be considered with respect to any of the data buckets under consideration.

Threshold excess determination system2820, in some embodiments, may be configured to determine whether or not the overlap percentage is in excess of an experimental overlap threshold2325. For example, if the threshold for an experiment is set at 10%, and the overlap is 20%, then threshold excess determination system2820may determine that the overlap exceeds the threshold. However, if the overlap is 10% and the threshold is 20%, then system2820may determine that the overlap does not exceed the threshold. Persons of ordinary skill in the art will recognize that different experiments may employ different thresholds2325, which may also be based on a sample size associated with an experiment, and the aforementioned are purely exemplary.

Data bucket inconsistency notification system2830may be configured, in some embodiments, to determine whether threshold excess determination system2820indicates that the overlap exceeds the threshold, or does not exceed threshold, and may generate output data reflective of the results. For example, if data bucket inconsistency notification system2830determines that the overlap exceeds the threshold for the experiment, then system2830may generate an inconsistent data bucket notification, which may be provided to an experiment designer to indicate that an inconsistency in the distribution of user identifiers within an experiment is present.

FIG.28Bis an illustrative flowchart of an exemplary processor for determining whether to generate an inconsistent data bucket flag, in accordance with various embodiments of the present teachings. Process2850, in a non-limiting embodiment, may begin at step2852. At step2852, overlapping user identifier data may be received. For instance, overlapping user identifier data may be received from user identification comparison system2330. At step2854, data bucket size information may be received. The data bucket size information may indicate a sample size associated with the data bucket that the overlap was detected for.

At step2856, a proportion associated with the overlap may be determined. For example, a proportion of the overlap as compared to the total data bucket size may be determined. At step2858, thresholds associated with the online experiment that the data bucket is associated with may be obtained. For instance, thresholds2325for the online experiment associated with the overlapping user identifier data's data bucket may be obtained. At step2860, a determination may be made as to whether the proportion is greater than the threshold. If, at step2860, it is determined that the proportion is not greater than threshold, then process2850may proceed to step2862. At step2862, normal user experience may continue. For example, each user identifier of the online experiment may continue to receive their appropriate user experience based on the data bucket assigned thereto. However, if at step2860it is determined that the proportion is greater than the threshold, the process2850may proceed to step2864. At step2864, an inconsistent data bucket flag may be generated. For example, data bucket inconstancy notification system2830may generate an inconsistent data bucket notification indicating that an overlap exists for a particular data bucket of an experiment, and that the overlap exceeds a threshold. This may indicate that the results of the experiment may be incorrect/inaccurate, thereby invalidating the experiment. Furthermore, this allows an experiment designer to research into the possible causes of the error to eliminate them, thereby potentially salvaging the experiment.

FIGS.29A and29Bare illustrative graphs indicating data bucket inconstancies over time and data bucket inconstancies within a threshold limit, in accordance with various embodiments of the present teachings. Graph2900ofFIG.29Acorresponds to a graph indicating a change in bucket consistency over time. As compared to graph2100ofFIG.21, even though the data buckets may appear to be consistent in terms of sample size over time, graph2900ofFIG.29Aindicates that there may still exists inconsistencies across the data buckets over time. This inconsistency, as mentioned above, not only affects the experiment's validity, but also may affect a user's experience with the content provider associated with the experiment.FIG.29B, however, includes graph2950, which illustrates an exemplary embodiment where inconsistencies have been minimized. For instance, over time, the inconstancies are fairly minimal (e.g., less than 1%). The minimal inconsistencies reflected herein may result in changes to the experiment's setup such as, and without limitation, metadata changes and/or data bucket opt-in configurations.

FIG.30is an illustrative diagram of exemplary computing system architecture, in accordance with various embodiments of the present teaching. Such a specialized system incorporating the present teaching has a functional block diagram illustration of a hardware platform which includes user interface elements. Computer3000may be a general purpose computer or a special purpose computer. Both can be used to implement a specialized system for the present teaching. Computer3000may be used to implement any component of the user activity detection system, as described herein. For example, data pipeline130, discrepancy detection system1040, and data inconsistency detection system2030may each be implemented on a computer such as computer3000via its hardware, software program, firmware, or a combination thereof. However, other components/systems of the aforementioned figures may additionally or alternatively be implemented on a computer such as computer3000via its hardware, software program, firmware, or combination thereof, and the aforementioned is merely exemplary. Although only one such computer is shown, for convenience, the computer functions relating to the various embodiments described herein may be implemented in a distributed fashion on a number of similar platforms, to distribute the processing load.

Computer3000, for example, may include COM ports3050connected to and from a network connected thereto to facilitate data communications. Computer3000also includes a central processing unit (CPU)3020, in the form of one or more processors, for executing program instructions. For example, CPU3020may include one or more processors such as those described by processor(s)402. The exemplary computer platform may also include an internal communication bus3010, program storage and data storage of different forms (e.g., disk3070, read only memory (ROM)3030, or random access memory (RAM)3040), for various data files to be processed and/or communicated by computer3000, as well as possibly program instructions to be executed by CPU3020. For instance, one or more of memory/storage404may be included by ROM3030and/or RAM3040, as described in greater detail above. Computer3000may also include an I/O component3060supporting input/output flows between the computer and other components therein such as user interface elements3080. Computer3000may also receive programming and data via network communications, such as via communications circuitry similar to communications circuitry406described in greater detail above.

All or portions of the software may at times be communicated through a network such as the Internet or various other telecommunication networks. Such communications, for example, may enable loading of the software from one computer or processor into another, for example, from a management server or host computer of abnormal user activity processing operator or other abnormal user activity detection system into the hardware platform(s) of a computing environment or other system implementing a computing environment or similar functionalities in connection with abnormal user activity detection. Thus, another type of media that may bear the software elements includes optical, electrical and electromagnetic waves, such as used across physical interfaces between local devices, through wired and optical landline networks and over various air-links. The physical elements that carry such waves, such as wired or wireless links, optical links or the like, also may be considered as media bearing the software. As used herein, unless restricted to tangible “storage” media, terms such as computer or machine “readable medium” refer to any medium that participates in providing instructions to a processor for execution.

Those skilled in the art will recognize that the present teachings are amenable to a variety of modifications and/or enhancements. For example, although the implementation of various components described above may be embodied in a hardware device, it may also be implemented as a software only solution—e.g., an installation on an existing server. In addition, the abnormal user activity detection system, as disclosed herein, may be implemented as a firmware, firmware/software combination, firmware/hardware combination, or a hardware/firmware/software combination.

While the foregoing has described what are considered to constitute the present teachings and/or other examples, it is understood that various modifications may be made thereto and that the subject matter disclosed herein may be implemented in various forms and examples, and that the teachings may be applied in numerous applications, only some of which have been described herein. It is intended by the following claims to claim any and all applications, modifications and variations that fall within the true scope of the present teachings.