A / B testing - wikipedia A / B testing Example of A / B testing on a website. By randomly serving visitors two versions of a website that differ only in the design of a single button element, the relative efficacy of the two designs can be measured In marketing and web analytics, A / B testing ( bucket tests or split - run testing ) is a controlled experiment with two variants, A and B. It is a form of statistical hypothesis testing or `` two - sample hypothesis testing '' as used in the field of statistics. In online settings, such as web design ( especially user experience design ), the goal of A / B testing is to identify changes to web pages that increase or maximize an outcome of interest ( e.g., click - through rate for a banner advertisement ). Formally the current web page is associated with the null hypothesis. A / B testing is a way to compare two versions of a single variable typically by testing a subject 's response to variable A against variable B, and determining which of the two variables is more effective. As the name implies, two versions ( A and B) are compared, which are identical except for one variation that might affect a user 's behavior. Version A might be the currently used version ( control ), while version B is modified in some respect ( treatment ). For instance, on an e-commerce website the purchase funnel is typically a good candidate for A / B testing, as even marginal improvements in drop - off rates can represent a significant gain in sales. Significant improvements can sometimes be seen through testing elements like copy text, layouts, images and colors, but not always. Multivariate testing or multinomial testing is similar to A / B testing, but may test more than two versions at the same time or use more controls. Simple A / B tests are not valid for observational, quasi-experimental or other non-experimental situations, as is common with survey data, offline data, and other, more complex phenomena. A / B testing has been marketed by some as a change in philosophy and business strategy in certain niches, though the approach is identical to a between - subjects design, which is commonly used in a variety of research traditions. A / B testing as a philosophy of web development brings the field into line with a broader movement toward evidence - based practice. The benefits of A / B testing are considered to be that it can be performed continuously on almost anything, especially since most marketing automation software now, typically, comes with the ability to run A / B tests on an on - going basis. This allows for updating websites and other tools, using current resources, to keep up with changing trends. Contents 1 Common test statistics 2 History 3 An emailing campaign example 4 Segmentation and targeting 5 Acceptance 6 See also 7 References 8 Further reading Common test statistics `` Two - sample hypothesis tests '' are appropriate for comparing the two samples where the samples are divided by the two control cases in the experiment. Z - tests are appropriate for comparing means under stringent conditions regarding normality and a known standard deviation. Student 's t - tests are appropriate for comparing means under relaxed conditions when less is assumed. Welch 's t test assumes the least and is therefore the most commonly used test in a two - sample hypothesis test where the mean of a metric is to be optimized. While the mean of the variable to be optimized is the most common choice of estimator, others are regularly used. For a comparison of two binomial distributions such as a click - through rate one would use Fisher 's exact test. Assumed Distribution Example Case Standard Test Alternative Test Gaussian Average Revenue Per Paying User Welch 's t - test ( Unpaired t - test ) Student 's t - test Binomial Click Through Rate Fisher 's exact test Barnard 's test Poisson Transactions Per Paying User E-test C - test Multinomial Number of each product Purchased Chi - squared test Unknown -- Mann -- Whitney U test Gibbs sampling History Like most fields, setting a date for the advent of a new method is difficult because of the continuous evolution of a topic. Where the difference could be defined is when the switch was made from using any assumed information from the populations to a test performed on the samples alone. This work was done in 1908 by William Sealy Gosset when he altered the Z - test to create Student 's t - test. Google engineers ran their first A / B test in the year 2000 in an attempt to determine what the optimum number of results to display on its search engine results page would be. The first test was unsuccessful due to glitches that resulted from slow loading times. Later A / B testing research would be more advanced, but the foundation and underlying principles generally remain the same, and in 2011, 11 years after Google 's first test, Google ran over 7,000 different A / B tests. An emailing campaign example A company with a customer database of 2,000 people decides to create an email campaign with a discount code in order to generate sales through its website. It creates two versions of the email with different call to action ( the part of the copy which encourages customers to do something -- in the case of a sales campaign, make a purchase ) and identifying promotional code. To 1,000 people it sends the email with the call to action stating, `` Offer ends this Saturday ! Use code A1 '', and to another 1,000 people it sends the email with the call to action stating, `` Offer ends soon ! Use code B1 ''. All other elements of the emails ' copy and layout are identical. The company then monitors which campaign has the higher success rate by analyzing the use of the promotional codes. The email using the code A1 has a 5 % response rate ( 50 of the 1,000 people emailed used the code to buy a product ), and the email using the code B1 has a 3 % response rate ( 30 of the recipients used the code to buy a product ). The company therefore determines that in this instance, the first Call To Action is more effective and will use it in future sales. A more nuanced approach would involve applying statistical testing to determine if the differences in response rates between A1 and B1 were statistically significant ( that is, highly likely that the differences are real, repeatable, and not due to random chance ). In the example above, the purpose of the test is to determine which is the more effective way to encourage customers to make a purchase. If, however, the aim of the test had been to see which email would generate the higher click - rate -- that is, the number of people who actually click onto the website after receiving the email -- then the results might have been different. For example, even though more of the customers receiving the code B1 accessed the website, because the Call To Action did n't state the end - date of the promotion many of them may feel no urgency to make an immediate purchase. Consequently, if the purpose of the test had been simply to see which email would bring more traffic to the website, then the email containing code B1 might well have been more successful. An A / B test should have a defined outcome that is measurable such as number of sales made, click - rate conversion, or number of people signing up / registering. Segmentation and targeting A / B tests most commonly apply the same variant ( e.g., user interface element ) with equal probability to all users. However, in some circumstances, responses to variants may be heterogeneous. That is, while a variant A might have a higher response rate overall, variant B may have an even higher response rate within a specific segment of the customer base. For example, the breakdown of the response rates by gender could have been : Gender Overall Men Women Total sends 2,000 1,000 1,000 Total responses 80 35 45 Variant A 50 / 1,000 ( 5 % ) 10 / 500 ( 2 % ) 40 / 500 ( 8 % ) Variant B 30 / 1,000 ( 3 % ) 25 / 500 ( 5 % ) 5 / 500 ( 1 % ) In this case, we can see that while variant A had a higher response rate overall, variant B actually had a higher response rate with men. As a result, the company might select a segmented strategy as a result of the A / B test, sending variant B to men and variant A to women in the future. In this example, a segmented strategy would yield an increase in expected response rates from 5 % = 40 + 10 500 + 500 ( \textstyle 5 % = ( \frac ( 40 + 10 ) ( 500 + 500 ) ) ) to 6.5 % = 40 + 25 500 + 500 ( \textstyle 6.5 % = ( \frac ( 40 + 25 ) ( 500 + 500 ) ) ) -- constituting a 30 % increase. It is important to note that if segmented results are expected from the A / B test, the test should be properly designed at the outset to be evenly distributed across key customer attributes, such as gender. That is, the test should both ( a ) contain a representative sample of men vs. women, and ( b ) assign men and women randomly to each `` variant '' ( variant A vs. variant B). Failure to do so could lead to experiment bias and inaccurate conclusions to be drawn from the test. This segmentation and targeting approach can be further generalized to include multiple customer attributes rather than a single customer attribute -- for example, customers ' age AND gender -- to identify more nuanced patterns that may exist in the test results. Acceptance Many companies use the `` designed experiment '' approach to making marketing decisions, with the expectation that relevant sample results can improve positive conversion results. It is an increasingly common practice as the tools and expertise grows in this area. There are many A / B testing case studies which show that the practice of testing is increasingly becoming popular with small and medium - sized businesses as well. See also Adaptive control Choice modelling Multi-armed bandit Multivariate testing Test statistic References Jump up ^ Kohavi, Ron ; Longbotham, Roger ( 2015 ). `` Online Controlled Experiments and A / B Tests ''. In Sammut, Claude ; Webb, Geoff. Encyclopedia of Machine Learning and Data Mining ( PDF ). Springer. pp. to appear. ^ Jump up to : `` The ABCs of A / B Testing - Pardot ''. Pardot. Retrieved 2016 - 02 - 21. Jump up ^ `` Split Testing Guide for Online Stores ''. webics.com.au. August 27, 2012. Retrieved 2012 - 08 - 28. Jump up ^ Christian, Brian ( 2000 - 02 - 27 ). `` The A / B Test : Inside the Technology That 's Changing the Rules of Business Wired Business ''. Wired.com. Retrieved 2014 - 03 - 18. Jump up ^ Christian, Brian. `` Test Everything : Notes on the A / B Revolution Wired Enterprise ''. Wired.com. Retrieved 2014 - 03 - 18. Jump up ^ Cory Doctorow ( 2012 - 04 - 26 ). `` A / B testing : the secret engine of creation and refinement for the 21st century ''. Boing Boing. Retrieved 2014 - 03 - 18. Jump up ^ Krishnamoorthy, K. ; Thomson, Jessica ( 2004 ). `` A more powerful test for comparing two Poisson means ''. Journal of Statistical Planning and Inference. 119 : 23. doi : 10.1016 / S0378 - 3758 ( 02 ) 00408 - 1. Jump up ^ `` Brief history and background for the one sample t - test ''. Jump up ^ Box, Joan Fisher ( 1987 ). `` Guinness, Gosset, Fisher, and Small Samples ''. Statistical Science. 2 ( 1 ) : 45 -- 52. doi : 10.1214 / ss / 1177013437. Jump up ^ Amazon.com. `` The Math Behind A / B Testing ''. Developer.amazon.com. Archived from the original on 2015 - 09 - 21. Retrieved 2015 - 04 - 12. Jump up ^ Kohavi, Ron ; Longbotham, Roger ; Sommerfield, Dan ; Henne, Randal M. ( 2009 ). `` Controlled experiments on the web : survey and practical guide '' ( PDF ). Data Mining and Knowledge Discovery. Berlin : Springer. 18 ( 1 ) : 140 -- 181. ISSN 1384 - 5810. doi : 10.1007 / s10618 - 008 - 0114 - 1. Jump up ^ `` Advanced A / B Testing Tactics That You Should Know Testing & Usability ''. Online-behavior.com. Retrieved 2014 - 03 - 18. Jump up ^ `` Eight Ways You 've Misconfigured Your A / B Test ''. Dr. Jason Davis. 2013 - 09 - 12. Retrieved 2014 - 03 - 18. Jump up ^ `` The Complete Guide To Conversion Rate Optimization ''. Omniconvert. Retrieved 2017 - 01 - 05. Jump up ^ `` A / B Split Testing Multivariate Testing Case Studies ''. Visual Website Optimizer. Retrieved 2015 - 09 - 08. `` A / B Testing Case Studies ''. Optimizely. Retrieved 2015 - 11 - 24. `` Apptimize Mobile A / B Testing Case Studies ''. Apptimize. Further reading `` A Marketer 's Guide to A / B Testing ''. CleverTap Blog. CleverTap. Retrieved from `` https://en.wikipedia.org/w/index.php?title=A/B_testing&oldid=800608234 '' Categories : Market research Experiments Software testing Talk Article Talk More Contents About Wikipedia Azərbaycanca Čeština Dansk Deutsch Español فارسی Français 한국어 עברית Nederlands 日本 語 Polski Português Русский Українська Edit links This page was last edited on 14 September 2017, at 16 : 02. Text is available under the Creative Commons Attribution - ShareAlike License ; additional terms may apply. By using this site, you agree to the Terms of Use and Privacy Policy. Wikipedia ® is a registered trademark of the Wikimedia Foundation, Inc., a non-profit organization. About Wikipedia

Question: what variable do you test in an experiment
Answer: variable A against variable B