document: Overlapping experiment infrastructure : more , better , faster experimentation At Google , experimentation is practically a mantra ; we evaluate almost every change that potentially affects what our users experience . Such changes include not only obvious user-visible changes such as modifications to a user interface , but also more subtle changes such as different machine learning algorithms that might affect ranking or content selection . Our insatiable appetite for experimentation has led us to tackle the problems of how to run more experiments , how to run experiments that produce better decisions , and how to run them faster . In this paper , we describe Google 's overlapping experiment infrastructure that is a key component to solving these problems . In addition , because an experiment infrastructure alone is insufficient , we also discuss the associated tools and educational processes required to use it effectively . We conclude by describing trends that show the success of this overall experimental environment . While the paper specifically describes the experiment system and experimental processes we have in place at Google , we believe they can be generalized and applied by any entity interested in using experimentation to improve search engines and other web applications .
doc_bio_tags: O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O