Patent ID: 11960387
Assignee: MICROSOFT TECHNOLOGY LICENSING, LLC
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 0:
1. A method for determining a time period of a root cause of a sample ratio mismatch (SRM) in an online controlled experiment (OCE), wherein the OCE comprises a treatment experience and a control experience, the OCE is configured to compute scorecards according to a pre-defined schedule, the method comprising:
receiving a first scorecard from the OCE, wherein the first scorecard was computed by the OCE, the OCE includes an expected proportion of users to be assigned to the treatment experience, the first scorecard is for a first period of time, and the first scorecard includes a first proportion of users in the treatment experience;
determining that the first scorecard does not have the SRM based on a threshold and a first p-value, wherein the first p-value is based on the first proportion of users in the treatment experience and the expected proportion of users to be assigned to the treatment experience;
receiving a second scorecard from the OCE, wherein the second scorecard is for a second period of time that includes the first period of time and is longer than the first period of time, the second scorecard was computed by the OCE, and the second scorecard includes a second proportion of users in the treatment experience, and wherein the second scorecard is more expensive to compute than the first scorecard;
determining that the second scorecard has the SRM based on the threshold and a second p-value, wherein the second p-value is based on the second proportion of users in the treatment experience and the expected proportion of users to be assigned to the treatment experience;
determining that the root cause of the SRM in the second scorecard existed in the first scorecard based on the first proportion of users in the treatment experience in the first scorecard and the second proportion of users in the treatment experience in the second scorecard; and
in response to determining that the root cause of the SRM in the second scorecard existed in the first scorecard and prior to completing the OCE, stopping the OCE from computing additional scorecards of the OCE subsequent to the second scorecard, wherein the additional scorecards are more expensive to compute than each of the first scorecard and the second scorecard.