Patent ID: 11960387

Abstract:
A sample ratio mismatch (SRM) analyzer receives data from an online controlled experiment (OCE) and provides information to help determine a root cause of an SRM. The SRM analyzer may identify one or more segments in the data that include an SRM and may determine whether a triggered scorecard of the OCE includes an SRM. The data may include one or more scorecards. The SRM analyzer may determine whether each scorecard has an SRM. The SRM analyzer may test a difference in proportion of users assigned to treatment between a last scorecard without an SRM and a first scorecard with an SRM. If the difference in proportion is statistically meaningful, the SRM analyzer may determine that the SRM arose after the last scorecard. If the difference in proportions is not statistically meaningful, the SRM analyzer may determine that the SRM existed from a beginning of the OCE.