Patent ID: 11960387
Assignee: MICROSOFT TECHNOLOGY LICENSING, LLC
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 6:
7. A method for determining and presenting an indication of a sample ratio mismatch (SRM) for a scorecard of an online controlled experiment (OCE), the method comprising:
receiving a first scorecard from the OCE, wherein the OCE includes a treatment experience, a control experience, and an expected treatment-to-control (T/C) ratio, and wherein the first scorecard is for a first period of time;
determining an observed first T/C ratio for the first scorecard;
determining a first confidence interval using the observed first T/C ratio based on the expected T/C ratio and a threshold, the first confidence interval determined using a difference between the observed first T/C ratio and the expected T/C ratio;
displaying, on a graphical user interface of a display screen, a marker representing the observed first T/C ratio;
displaying, on the graphical user interface of the display screen, a line representing the first confidence interval;
displaying, on the graphical user interface of the display screen, a line representing the expected T/C ratio, wherein whether the line representing the first confidence interval intersects with the line representing the expected T/C ratio indicates whether the first scorecard has an SRM;
determining, based on the line representing the first confidence interval not intersecting the line representing the expected T/C ratio, that the SRM exists in the first scorecard; and
before completing a second scorecard of the OCE, the second scorecard more expensive to compute than the first scorecard, stopping the OCE based on whether the first scorecard has an SRM.