Patent ID: 11960387
Assignee: MICROSOFT TECHNOLOGY LICENSING, LLC
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 9:
10. A method comprising:
detecting a sample ratio mismatch (SRM) in two online controlled experiments (OCEs) for a product;
determining that the two OCEs use a same assignment mechanism and data pipeline;
analyzing a first segment and a second segment included in the two OCEs for the product to determine segment SRMs, the first segment representing a first set of participants sharing a first attribute value and the second segment representing a second set of participants sharing a second attribute value, with the first attribute value being different from the second attribute value, and wherein each of the first segment and the second segment comprises a combination including both participants assigned to an experiment variant of the product and participants assigned to a control variant of the product;
determining, based on a determination that the first segment corresponding to the first attribute value and the second segment corresponding to the second attribute value both include a segment SRM, that a root cause of the SRM is widespread within recent OCEs for the product; and
before the two OCEs each compute a subsequent scorecard, stopping the two OCEs from each running the subsequent scorecard based on the root cause of the SRM being widespread within the recent OCEs for the product, wherein the subsequent scorecard is more expensive to compute than a prior scorecard.