Computationally and network bandwidth-efficient technique to determine network-accessible content changes based on computed models

Technologies are disclosed for determining network-accessible content changes based on computed models and providing a long term forecast of user interaction at a network accessible site based upon a short term experiment at the site. A forecast model for a period of time is generated based upon historical data of user interactions at the site. An experiment is run for a short term at the site based upon a potential change at the site. Based upon data obtained during the experiment, scores are generated for a control group (no change) and a treatment group (potential change) and compared. If there are statistically significant differences between the control group and the treatment group scores, the long term forecast may be used to forecast what the long term impact of the experiment would be based upon the short term experiment.

BACKGROUND

Network accessible sites offer users a variety of experiences. For example, users may access such sites over a network such as, for example, websites accessed over the Internet, in order to view media, play games, purchase goods and/or services, access information, provide information, etc. Operators of such sites may wish to know the effect various changes to the site may have on user behavior. For example, operators of network accessible sites may wish to know what effect changing the font size of print displayed on the site has on user behavior.

In order to predict what effect changes at a network accessible site may have on user behavior, operators may run experiments. For example, an experiment might be to change the font size of print at a site. In order to determine the effect such a change may have, the experiment may be run as what is referred to as an A/B experiment where two groups are randomly established. A first group may be a control group, where the font size remains unchanged. A second group may be a treatment group, where the font size is changed, for example, doubled in size. The operator of the site running the experiment may wish to know if the change in font size affects how long users remain at the website. Thus, the operator runs the experiment and at the end of the experiment compares the results from the two groups. However, generally in order to obtain enough data for a period of time for there to be statistical significance and any difference, the experiment may need to run for three months, six months or even a year. This can create a burden for the operator of the site who may wish to know the effect much sooner in order to implement the change in a timely manner.

The disclosure made herein is presented with respect to these and other considerations.

DETAILED DESCRIPTION

The following detailed description is directed to technologies for providing a model build service within a service provider network for generating long term forecast models of user interaction with at network accessible sites based upon short term experiments. An implementation of the technologies described herein can improve the speed with which changes may be made to network accessible sites.

Traditional experimental systems (A/B experimental systems) use change in metrics during the course of the experiment related to a potential change at a network accessible site to evaluate if the experiment was successful. The impact of the experiment is also evaluated and recommendations may be made for launching the potential change full time. However, an experiment might not have an immediate impact but rather result in a long term change in customer behavior.

For example, assume the long term effect of interest is the impact of the experiment for 365 days. In such a case, based on the existing solutions, an experiment will need to run for 365 days. However, this is a significant amount of time to run an experiment (which requires a lot of additional resources, e.g., processing time, network bandwidth, storage, services to correlate and analyze the data, etc.) and slows down the pace of innovation as it is necessary to wait a year before knowing the long term impact.

By integrating a long term forecast model, it is no longer necessary to run the experiment for such a long period of time. The forecast model can use short run experiment data (e.g., 2 weeks) to forecast what the 365 day impact of the experiment would be along with a measure of accuracy (e.g., X %). A recommendation on how much longer an experiment should run to improve accuracy (e.g., 2X %) may be made. A recommendation to launch or not to launch the potential change may be made if the current accuracy of the forecast (e.g., X %) is acceptable. Thus, by generating forecast models, the experiments do not need to run as long, thereby saving processor cycles, computing resources, power, etc. Technical benefits other than those specifically identified herein might also be realized through an implementation of the disclosed technologies.

In configurations, a forecast model may be generated that forecasts user behavior for users that visit a network accessible site. The forecast model may predict an expected contribution, e.g., monetary contribution, for the users of the site based upon a past history of users accessing the site. For example, an operator of a site provided by a website service of a service provider network may wish to know the effect on user behavior when the font size of print displayed at the site is doubled. Thus, an experiment may be developed for checking the effect that doubling the font size has on user behavior.

Prior to running the experiment, a forecast model may be built for forecasting future user behavior. The forecast model may then be scored with respect to expected user behavior e.g., the expected contribution or behavior of users that visit the site based upon the original font size. The forecast model takes into account various characteristics or features of previous user behavior over a period of time. For example, the period of time may be one year. Thus, the forecast model may forecast how many times users will interact with a user interface (UI) at the site (as an example of expected user behavior) for an upcoming period of time, e.g., the next year, based upon various user behavior characteristics and metrics data gathered over the past year.

Once the forecast model has been generated, users accessing the site are randomly split into two groups, e.g. 50-50, utilizing user identifiers. A first group may be a control group where the font remains unchanged. The second group may be a treatment group where the font is doubled in size. Before the experiment starts, the forecast model may be scored on these two groups of users using their behavior characteristics upon the beginning of the experiment. The scores may be a predicted number of expected interactions with the UI at the site for each user. Afterwards, the experiment may then be run.

As users trigger into the experiment, e.g., users access the site and are assigned to one of the groups, the site is displayed to the users based upon which group they have been assigned to. Users assigned to the control group will see the site displayed in the original, unchanged font. Users assigned to the treatment group will see the site displayed using a font twice the size of the original font. Actions of the users are monitored and logged by a log service, which provides the data to an analytics service that is running the experiment.

After a predetermined amount of time, for example, one month, the analytics related to user behavior during the experiment may be provided, and the forecast model may be scored twice, once for the control group and once for the treatment group, using the users' behavior characteristics as of the end of the predetermined amount of time. For each user, the original score computed upon the start of the experiment may be subtracted from the recent score computed upon the end of the experiment to provide a pre-post score change. The pre-post score change may reflect how the forecast outlook might change from using the pre-experiment characteristics for scoring to using the post-experiment characteristics for scoring.

An average pre-post score change for the two groups may be compared in order to determine if there is a statistically significant difference between the two groups with respect to the pre-post score change. Since users were randomly assigned to the control group and the treatment group prior to the experiment, the pre-post score change is not expected to differ between the two groups if the experiment does not affect users behavior. However, if the change in font size has an effect on user behavior, then the pre-post score change for the treatment group may be significantly different than that for the control group.

In configurations, results of the pre-post score change for the control group are subtracted from results of the pre-post score change for the treatment group in order to determine a difference. Any difference may then be analyzed in order to determine if the difference is statistically significant or if the difference is simply random, i.e. noise.

If the difference is deemed to be statistically significant, then it may be determined that the changing of the font size is either desirable, undesirable or neutral depending upon the results. If it is desirable, then the change may be instituted and the forecast model may be utilized to predict the expected contribution or behavior of users based upon the change in font size for a much longer period of time, e.g. 6 months or a year.

If the change is undesirable then the change in font size may be disregarded and not instituted. In configurations, if the differences are deemed to be not statistically significant, the experiment may run for a longer period of time to see if the proposed change does eventually cause a change in user behavior, i.e. increase or decrease the number of clicks on the user interface, that is statistically significant. Likewise, in configurations, the experiment may be designed to run for one month but after two weeks it may already be determined that the proposed change does have a statistically significant effect on user behavior and thus, the experiment may end early.

In configurations, the characteristics or features that are used as inputs for the forecast model may include various features or characteristics such as, for example, purchase related features, browsing related features, event or site engagement type features, etc. In accordance with various configurations, the model may utilize two stages where a first stage predicts a user's propensity for a certain action, e.g. a user's propensity for clicking on the user interface. The second stage may predict that, given that a user will click on the user interface, the number of times the user will click on the user interface. The forecast model result is based upon the product of multiplying results from the first stage by results from the second stage. Additional details regarding the various components and processes described above for creating a contact center within a service provider network will be presented below with regard toFIGS. 1-10.

Those skilled in the art will also appreciate that aspects of the subject matter described herein can be practiced on or in conjunction with other computer system configurations beyond those described herein, including multiprocessor systems, microprocessor-based or programmable electronic devices, minicomputers, mainframe computers, handheld computers, personal digital assistants, e-readers, cellular telephone devices, special-purposed hardware devices, network appliances, and the like. The configurations described herein can also be practiced in distributed computing environments, where tasks can be performed by remote computing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in and executed from both local and remote memory storage devices.

In the following detailed description, references are made to the accompanying drawings that form a part hereof, and that show, by way of illustration, specific configurations or examples. The drawings herein are not drawn to scale. Like numerals represent like elements throughout the several figures (which might be referred to herein as a “FIG.” or “FIGS.”).

FIG. 1schematically illustrates a service provider network100that hosts various services including a website service102that provides websites that may be accessed by various users or customers104over a network, e.g., the Internet. A log service106may also be provided by the service provider network100that may log various actions of customers104that access a website108provided by the website service102. For example, the log service106may log and store data related to the identity of customers108that access the website108, the time and day that the customers104access the website106, purchases, subscription purchases, return purchases, cancelled subscriptions, media streaming activity, etc. In configurations, users generally consent to monitoring of their activities for such logging and storing data. The log service106may provide the data to an analytics service110, which may analyze the data for various purposes and generate metrics128. While the present disclosure is being discussed with respect to websites108, as previously noted, other sites that are accessible by a network may be used with the techniques and architecture described herein.

In configurations, the service provider network100also includes an experiment service112. The experiment service112may provide various experiments for execution at the website108of the website service102in order to determine the effect of possible changes to the website108on customers' behavior. For example, the experiment service112may implement an experiment114to determine the effect of font size on customers' purchase behavior. The experiment114may be designed to determine the effect of doubling the size of the font at the website108.

The experiment service112may provide the data related to the definition and implementation of the experiment114to the analytics service110. The analytics service110may then interact with the website service102to double the font size at the website108for customers104that are assigned to a treatment group116. For the experiment114, the font size will remain unchanged for a control group118of customers104. As customers104access the website108, they are randomly assigned to either the control group118or the treatment group116. An example of a split between the control group118and the treatment group116is a 50-50 split. Other splits may be utilized if desired.

Thus, as a customer104accesses, via a computing device, the website108at the website service102, a customer identifier (ID)120is provided to the analytics service110, which then assigns the customer104to either the control group118or the treatment group116utilizing the customer ID120. Customers104may be identified, for example, after they have provided credentials. In other configurations, customers104may be identified based upon various other metrics, such as, for example, cookies received from a customer's web browser, device IDs, actions performed using a client device, etc. Based upon the group assignment130, the analytics service110controls the font size displayed to the customer104at the website108.

In configurations, a model build service122is provided by the service provider network100. As will be discussed further herein, the model build service122builds a forecast model124for user or customer behavior based upon the data provided by the log service106. For example, based upon data provided by the log service106, the model build service122may build the forecast model124that predicts, or forecasts, how much customers104will spend at the website108in the next year. The forecast model124may predict a per customer average spend and/or a total, aggregated spend amount for all customers104on any type of cadence (which may be regular or irregular), such as, for example, daily, hourly, etc.

The forecast model124may be provided to the analytics service110for running the experiment114. Upon the start of the experiment, the list of the treatment group customers118and the list of the control group customers116may be provided to a model scoring service126to score the control group118and the treatment group116based upon the forecast model124using customer behavior characteristics recorded by log service106. The scores may reflect any number of predicted future interactions with the network accessible content, such as, for example, predicted purchase behavior at the site for each user in the two groups116,118. Once the experiment114is complete, the list of the treatment group customers118and the list of the control group customers116may be provided to the model scoring service126. The model scoring service126may then score the control group118and the treatment group116based upon the forecast model124and metrics128related to customer behavior during the experiment114provided by the analytics service110and/or customer behavior characteristics from the log service106as of the end of the experiment.

For each customer in the control group118and treatment group116, the model scoring service126can provide the score as of the start of the experiment and the score as of the end of the experiment to the analytics service110to compute a pre-post score change by taking the difference between the two scores for every customer. The analytics service110can then compare an average pre-post score changes for the two groups (control group118and treatment group116). In configurations, this may be done by subtracting the average pre-post score change of the control group from the average pre-post score change of the treatment group.

If there is a difference in the average pre-post score change between the control group118and the treatment group116, then the analytics service110may determine if the difference is statistically significant. If the difference is not statistically significant, then the font size may not be changed at the website108.

In configurations, the prediction of the forecast model124for the upcoming year may include a confidence or accuracy factor, i.e. a plus or minus amount, a percentage, etc., for the expected purchase amount, since the experiment ran for a shorter amount of time than the time period for the prediction, e.g., a few weeks versus one year. In configurations, it may be determined that the experiment108should run for a longer period of time to see if any differences do become statistically significant, or if the confidence or accuracy factor is not acceptable. In configurations, the model scoring service126may determine how much longer to run the experiment114based upon the models124.

If the differences between the treatment group116and the control group118are statistically significant and/or the confidence or accuracy factor is acceptable, then the change in font size may be implemented at the website108. The forecast model124, applied to the randomized treatment and control groups, may be used as the prediction for incremental purchase amount per customer over the next year based upon the change in font size. Thus, instead of running the experiment for a long period of time, e.g., a year to determine an initial effect of a change in font size, the expected effect of the font size change on customer behavior at the website108may be determined in a much shorter period of time, e.g., a few weeks. Thus, by generating a forecast model based upon short term experiments, the experiments114do not need to run as long, thereby saving processor cycles, power, network bandwidth, storage, etc.

In configurations, the experiment114may be designed to run for one month, but after two weeks the analytics service110may determine that the proposed change does have a statistically significant effect on customer behavior and thus, the experiment may end early. This may be accomplished by periodically providing customer data for the treatment group116and the control group118to the model scoring service126for periodically scoring and comparing the two groups with respect to the pre-post score change by the analytics service110based upon the forecast model124. In configurations, the confidence factor may also be used to determine that the experiment has run long enough, e.g., the confidence factor is acceptable.

Customers104may be assigned to categories for the experiments and the model build. For example, active customers may be customers104who made at least one purchase within the last 12 months. Dormant customers may be customers104who have made at least one purchase between 12 and 24 months, but none in the previous 12 months. “Other” types of customers may include customers who have either never made any purchases or who made their most recent purchase more than 24 months ago.

In configurations, the forecast model124built by the model build service122may be based upon active customers. In configurations, if a dormant customer or an “other” customer, e.g. a new customer, accesses the website108during an experiment, then that customer may be assigned a mean value of their history. For example, a mean value may be calculated for a dormant customer while a value of zero may be assigned to a new customer.

While only one model124and one experiment114is illustrated, there may be multiple models built and there may be multiple experiments run at a time. Thus, there may be multiple control groups and multiple treatment groups.

FIG. 2schematically illustrates the operation of some of the services illustrated inFIG. 1in more detail. As may be seen, customers104access the website108at the website service102. Customer actions and data202are logged to the log service106. Customer actions and data202may include, for example, the identity of customers108that access the website108, the time and day that the customers104access the website106, purchases, subscription purchases, return purchases, cancelled subscriptions, media streaming activity, time spent at the website108, interactions with UIs, etc. The log service106may provide data related to the customer actions and data202to the analytics service110.

The experiment service112includes an experiment generator204that may be utilized to generate experiments for the website108at the website service102in order to determine the effects potential changes at the website108may have on customer behavior. The experiment generator204may generate data related to the definition and implementation of the experiment114related to doubling the font size at the website108and provides the experiment114to the analytics service110.

The analytics service110includes a website control function206and a group assign function208. As customers104access the website108, the customers104are assigned to either the control group118by the group assign function208, where the font size at the website108will remain unchanged by the website control function206via a website effect212, or to the treatment group116, where the font size at the website108will be changed by the website control function206via a website effect212. The identity120of the customers104accessing the website108are provided to the analytics service110for random assignment to the two groups,116,118.

Once the customers104are assigned to a particular group, the analytics service110uses the website control function206to control the function of the font displayed at the website108for each customer depending upon which group the customer was assigned to. For customers104in the control group118, the current font size is utilized. For customers104in the treatment group, the proposed new font size, e.g., double, is utilized.

Based upon customer actions and data202provided by the log service, a metrics generator210at the analytics service110may provide metrics128to the experiment service112for the control group118and the treatment group116. Such metrics may include, for example, time spent by customers104at the website108, purchases made by customers104at the website108, subscriptions purchased by customers104at the website108, content consumed, content viewed, content viewed and dismissed, etc. Thus, the experiment service112can see the difference in such metrics between the control group118and the treatment group116.

FIG. 3illustrates an example time period for calculating metrics128for the experiment114by the analytics service110. For example, during the experiment, from TNto T0, historical data based upon various metrics may be utilized to predict future customer behavior from T0to T1for an expected behavior model generated by the analytics service110. TNmay be for a period of 1 year, 2 years or more, including all historical data available. TNmay be for a less than a year if desired.

InFIG. 4, an example of determining a difference in the expected behavior of customers104based upon the experiment114is illustrated. As can be seen, when a customer104first enters the website108during experiment114, i.e. triggers into the experiment, the customer's expected behavior is calculated based upon historical data in the data from the log service106. At the end of the experiment114, the expected behavior for the customer is calculated again.

The customer behavior, e.g., actions and data202, during the experiment114is included in the new prediction as part of the customer's historical behavior. The expected contribution prediction on the first day is subtracted from the expected contribution prediction from the experiment end date, which in this example is one year. This reflects the change in expected future customer behavior from this customer over the course of the experiment. The difference in expected customer behavior is collected for all customers and the average expected customer behavior difference across various experiments may be compared in order to determine if there is a significant change in the expected customer behavior difference caused by any of the experiments.

In configurations, in a comparison between groups, e.g. the control group and treatment group, a significant change in the expected customer behavior difference metric leads to the conclusion that the treatment caused a significantly different change in the future customer behavior than the control experience. A negative significant result indicates that the expected customer behavior difference was significantly lower in the treatment while a positive significant result suggests that the treatment caused an increase in the expected customer behavior difference.

Referring toFIG. 5, as previously noted, in configurations, a model build service122is provided that builds a forecast model that forecasts a change in customer behavior based upon various characteristics or input features502of customer behavior. For example, the model build service122may build a forecast model124that forecasts how much customers104will spend in the next 365 days when accessing the website108.

The model build service502receives input data related to input features502that relate to various aspects of customer behavior. An example of a category of input features502may include event or engagement characteristics. Such event or engagement characteristics may include, for example, a subscription to a purchase service, a subscription to a particular consumer group, creation of a web wishlist at the website, creation of a wedding registry, creation of a baby registry, etc.

Such events may be evaluated prior to the beginning of the experiment and utilized in the forecast model build as a factor in expected future purchases of customers. Likewise, if such events are cancelled during the experiment, then these features may have a negative effect on predicted future purchases by customers. An additional example category of input features502may include purchase history characteristics such as, recent purchase history, long term purchase history, amounts of purchases, etc. A third example category of input features502may include website browsing characteristics, e.g. frequency of visits to the website108, most recent visit to the website108, etc.

An outlier and missing feature function504may eliminate outliers of various features as inputs. For example, the highest and lowest values for various input features may be dropped in order to not skew the input data. For example, a single customer104may have purchased only a large, expensive TV in the last year. This can skew the expected future purchases from this customer104. The outlier and missing feature function504may also impute a value for missing input features for use in building the forecast model124.

A feature engineering function506may also be included in the model build service122. For example, the feature engineering function506may take an input feature502of what a customer104spends in a year at the website108and another input feature502of what the customer104earns in a year and engineer another input feature502that is a ratio between what the customer104spends at the website108in a year and what the customer104earns in a year. The ratio may now be used as an input feature502with the other two input features502if desired, or one or both of the other input features502may be eliminated by a feature reduction function508as will be described further herein.

The feature reduction function508(also referred to as a multi-co-linearity function) of the model build service122may evaluate various input features502and eliminate input features502that may overlap. For example, if one input feature502that is selected for use in building the model is a 30-day purchase history and another feature selected for input into the model is a 60-day purchase history, then the 30-day purchase history may be dropped as an input feature502for the model building since it overlaps with the 60-day purchase history.

A model fit function510of the model build service122may, for a first stage of the model, determine a probability of a customer104making a purchase at the website108in the next year. For example, after the feature reduction function508, the first stage model may be fit using a logistic regression model. The units of observation may be customers, with input features measured as of a specific pivot date. The target variable takes value 1 if a customer made at least one purchase in the n days after the pivot date (where n is the downstream horizon) and 0 otherwise. The input features may be all features selected during the feature reduction function508. To further guard against multi-colinearity in predictor variables and to reduce the prediction variance, the model may use L2 regularization. At the second stage of the model fitting, the model fit function510may determine that given that a customer104will make a purchase, how much will that customer104spend. For example, the second stage model may be fit using a linear regression model. The units of observation may be customers who made at least one purchase in the n days after the pivot date. The target variable may be the customer's total order product sales (OPS) or contribution profit (CP) during the n-day downstream horizon. The input features may be all features selected during the feature reduction feature508. To further encourage model sparsity, to guard against multi-colinearity in the predictor variables, and to reduce the prediction variance, from the model uses L1 and L2 regularization. The model124may then be generated for a prediction of future spend of customers104by multiplying the first stage by the second stage.

A model validation function512may also be provided that may be utilized to validate the model124based upon future gathering of data. For example, if the model predicts a 100 dollar per customer purchase amount over the next year, but the actual average purchase per customer only total 80 dollars per year, then features502may be added or eliminated from the model124in the future in order to improve future model builds and predictions of customer behavior.

Additionally, as another example, a model124may be built for 2016 based upon input features502from 2015. The model may then be compared with data from 2014 for accuracy to see if any input features502should be added or eliminated. It should be noted that this example of the model build service122is only an example and that other types of model builds for model build service122may be utilized as desired.

FIG. 6illustrates another example of one type of metric128that may be generated by the analytics service110for use in model building by the model build service122and use of customer characteristics and actions as input features502. The metrics128may include various customer actions202from 1 to X.

As previously noted, the log service106monitors the actions of customers104and maintains a record of customer actions202. During an experiment, the metrics generated and provided to the experiment service112and/or the model build service122may include a determination that Action 1 was performed by W number of customers104within the control group118and Y number of customers104within the treatment group116.

For Action 2, zero customers104within the control group118performed Action 2, while Z number of customers104within the treatment group116performed Action 2. Such information may be utilized in order to determine which customer actions202caused significant differences between the control group118and the treatment group116. For example, if Z is a large number, e.g., 8,000,000, then it may be an indication that Action 2 caused the difference in purchase amounts between the treatment group116and the control group118during the experiment114.

It is to be appreciated that the examples given above are merely illustrative and that other techniques can be used in other configurations. Additional details regarding the arrangements shown inFIGS. 1-6will be provided below with respect toFIG. 7.

FIG. 7includes a flow diagram showing a routine700that illustrates aspects of utilizing long term forecast models generated by a model build service, e.g. the model build service122of the service provider network100ofFIGS. 1 and 5, to predict long term customer behavior based upon short term experiments.

It is to be appreciated that the logical operations described herein with respect toFIG. 7(and the other FIGS.) can be implemented (1) as a sequence of computer implemented acts or program modules running on a computing system and/or (2) as interconnected machine logic circuits or circuit modules within the computing system. The implementation of the various components described herein is a matter of choice dependent on the performance and other requirements of the computing system. Accordingly, the logical operations described herein are referred to variously as operations, structural devices, acts, or modules.

These operations, structural devices, acts, and modules can be implemented in software, in firmware, in special purpose digital logic, and any combination thereof. It should also be appreciated that more or fewer operations can be performed than shown in the FIGS. and described herein. These operations can also be performed in parallel, or in a different order than those described herein.

The routine700begins at operation702, where a forecast model, e.g., forecast model124, is built by the model build service122based upon data related to historical user behavior, e.g., behavior of customers104, at a network accessible site, e.g., website108provided by website service102of the service provider network100. The forecast model may be used to forecast expected user behavior at the network accessible site for a period of time.

At operation704, the model is pre-scored with respect to a control group and a treatment group to generate a prediction of future user behavior with respect to at least one user characteristic. At operation706, an experiment, e.g., experiment114, is generated by the experiment service112. The experiment is with respect to a potential change at the network accessible site and is set to run for an amount of time that is less than the period of time.

At operation708, the control group and the treatment group are scored based up the forecast model and metrics and data gathered during the experiment. The scoring is with respect to expected user behavior at the network accessible site for the period of time. At operation710, differences between the pre-scores and the post-scores for the control group and the treatment group are determined.

From operation710, the routine700continues to operation712, where it is determined whether a difference between the control group and the treatment group is statistically significant and/or if the accuracy or confidence factor is acceptable. If so, then the routine700proceeds to operation714, where it is determined whether the difference is acceptable. If so, then the routine700proceeds to operation716where the potential change is implemented at the network accessible site. If the difference is not acceptable, then at operation718the network accessible site is left as is. From operations716and718, the routine700proceeds to operation720, where it ends. Note that operations716and718also proceed back to operation702. This is to provide data and metrics related to user behavior back to the model build step for continued model building and/or evaluation.

If, at operation712, it is determined that the difference is not statistically significant and/or the accuracy or confidence factor is not acceptable, then the routine700proceeds to operation722where it is determined whether the experiment should run longer. If the experiment is to run longer, then the routine700proceeds back to operation704where the experiment continues to run in the manner described above. If the experiment is not to run longer, then the routine proceeds to operation718, where the network accessible site is left as is. The routine700then ends at operation720.

FIG. 8is a system and network diagram that shows one illustrative operating environment for the configurations disclosed herein that includes a service provider network100. As discussed above, the service provider network100can execute network services that provide computing resources on a permanent or an as-needed basis. Among other types of functionality, the computing resources provided by the service provider network100can be utilized to implement the various network services described herein. As also discussed above, the computing resources provided by the service provider network100can include various types of computing resources, such as data processing resources like VM instances, stateless event-driven compute services, data storage resources, networking resources, data communication resources, network services, and the like.

As also discussed above, the computing resources provided by the service provider network100are enabled in one implementation by one or more data centers804A-804D (which might be referred herein singularly as “a data center804” or in the plural as “the data centers804”). The data centers804are facilities utilized to house and operate computer systems and associated components. The data centers804typically include redundant and backup power, communications, cooling, and security systems. The data centers804can also be located in geographically disparate locations. One illustrative configuration for a data center804that can be utilized to implement the technologies disclosed herein will be described below with regard toFIG. 6.

The customers and other users of the service provider network100can access the computing resources provided by the service provider network100over a network802, which can be a wide area communication network (“WAN”), such as the Internet, an intranet or an Internet service provider (“ISP”) network or a combination of such networks. For example, and without limitation, a computing device800operated by a customer or other user of the service provider network100can be utilized to access the service provider network100by way of the network802. It should be appreciated that a local-area network (“LAN”), the Internet, or any other networking topology known in the art that connects the data centers804to remote customers and other users can be utilized. It should also be appreciated that combinations of such networks can also be utilized.

FIG. 9is a computing system diagram that illustrates one configuration for a data center804that implements aspects of the technologies disclosed herein. The example data center804shown inFIG. 9includes several server computers902A-902F (which might be referred to herein singularly as “a server computer902” or in the plural as “the server computers902”) for providing the computing resources904A-904E.

The server computers902can be standard tower, rack-mount, or blade server computers configured appropriately for providing the computing resources904described herein (illustrated inFIG. 9as the computing resources904A-904E). As mentioned above, the computing resources904provided by the service provider network100can be data processing resources such as VM instances or hardware computing systems, data storage resources, database resources, networking resources, and others. Some of the servers902can also be configured to execute network services906A-906E, respectively, capable of instantiating, providing and/or managing the computing resources904, some of which are described in detail below with regard toFIG. 7.

The data center804shown inFIG. 9also includes a server computer902F that can execute some or all of the software components described above. For example, and without limitation, the server computer902F can be configured to execute the onboarding service130, which was described in detail above. The server computer902F can also be configured to execute other components and/or to store data for providing some or all of the functionality described herein. In this regard, it should be appreciated that the can execute on many other physical or virtual servers in the data centers804in various configurations.

In the example data center804shown inFIG. 9, an appropriate LAN908is also utilized to interconnect the server computers902A-902F. The LAN908is also connected to the network802illustrated inFIG. 8. It should be appreciated that the configuration of the network topology described herein has been greatly simplified and that many more computing systems, software components, networks, and networking devices can be utilized to interconnect the various computing systems disclosed herein and to provide the functionality described above.

Appropriate load balancing devices or other types of network infrastructure components can also be utilized for balancing a load between each of the data centers804A-804D, between each of the server computers902A-902F in each data center804, and, potentially, between computing resources904in each of the data centers804. It should be appreciated that the configuration of the data center404described with reference toFIG. 9is merely illustrative and that other implementations can be utilized.

FIG. 10shows an example computer architecture for a computer1000capable of executing program components for implementing the functionality described above. The computer architecture shown inFIG. 10illustrates a conventional server computer, workstation, desktop computer, laptop, tablet, network appliance, e-reader, smartphone, or other computing device, and can be utilized to execute any of the software components presented herein.

The computer1000includes a baseboard1002, or “motherboard,” which is a printed circuit board to which a multitude of components or devices can be connected by way of a system bus or other electrical communication paths. In one illustrative configuration, one or more central processing units (“CPUs”)1004operate in conjunction with a chipset1006. The CPUs1004can be standard programmable processors that perform arithmetic and logical operations necessary for the operation of the computer1000.

The chipset1006provides an interface between the CPUs1004and the remainder of the components and devices on the baseboard1002. The chipset1006can provide an interface to a RAM1008, used as the main memory in the computer1000. The chipset1006can further provide an interface to a computer-readable storage medium such as a read-only memory (“ROM”)1010or non-volatile RAM (“NVRAM”) for storing basic routines that help to startup the computer1000and to transfer information between the various components and devices. The ROM1010or NVRAM can also store other software components necessary for the operation of the computer1000in accordance with the configurations described herein.

The computer1000can operate in a networked environment using logical connections to remote computing devices and computer systems through a network, such as the network1002shown inFIG. 10. The chipset1006can include functionality for providing network connectivity through a NIC1012, such as a gigabit Ethernet adapter. The NIC1012is capable of connecting the computer1000to other computing devices over the network1008. It should be appreciated that multiple NICs1012can be present in the computer1000, connecting the computer to other types of networks and remote computer systems.

The computer1000can be connected to a mass storage device1018that provides non-volatile storage for the computer. The mass storage device1018can store an operating system1020, programs1022, and data, which have been described in greater detail herein. The mass storage device1018can be connected to the computer1000through a storage controller1014connected to the chipset1006. The mass storage device1018can consist of one or more physical storage units. The storage controller1014can interface with the physical storage units through a serial attached SCSI (“SAS”) interface, a serial advanced technology attachment (“SATA”) interface, a fiber channel (“FC”) interface, or other type of interface for physically connecting and transferring data between computers and physical storage units.

In addition to the mass storage device1018described above, the computer1000can have access to other computer-readable storage media to store and retrieve information, such as program modules, data structures, or other data. It should be appreciated by those skilled in the art that computer-readable storage media is any available media that provides for the non-transitory storage of data and that can be accessed by the computer1000.

As mentioned briefly above, the mass storage device1018can store an operating system1020utilized to control the operation of the computer1000. According to one configuration, the operating system comprises the LINUX operating system. According to another configuration, the operating system comprises the WINDOWS® SERVER operating system from MICROSOFT Corporation. According to further configurations, the operating system can comprise the UNIX operating system or one of its variants. It should be appreciated that other operating systems can also be utilized. The mass storage device1018can store other system or application programs and data utilized by the computer1000.

In one configuration, the mass storage device1018or other computer-readable storage media is encoded with computer-executable instructions which, when loaded into the computer1000, transform the computer from a general-purpose computing system into a special-purpose computer capable of implementing the configurations described herein. These computer-executable instructions transform the computer1000by specifying how the CPUs1004transition between states, as described above. According to one configuration, the computer1000has access to computer-readable storage media storing computer-executable instructions which, when executed by the computer1000, perform the various processes described above with regard toFIGS. 1-7. The computer1000can also include computer-readable storage media for performing any of the other computer-implemented operations described herein.

Based on the foregoing, it should be appreciated that technologies for generating long term forecast models of customer behavior at a network accessible site based upon short term experiments at the network accessible site operating within a service provider network have been presented herein. Moreover, although the subject matter presented herein has been described in language specific to computer structural features, methodological acts, and computer readable media, it is to be understood that the invention defined in the appended claims is not necessarily limited to the specific features, acts, or media described herein. Rather, the specific features, acts, and mediums are disclosed as example forms of implementing the claims.

The subject matter described above is provided by way of illustration only and should not be construed as limiting. Furthermore, the claimed subject matter is not limited to implementations that solve any or all disadvantages noted in any part of this disclosure. Various modifications and changes can be made to the subject matter described herein without following the example configurations and applications illustrated and described, and without departing from the true spirit and scope of the present invention, which is set forth in the following claims.