YEAST: Yet Another Sequential Test
Authors: Alexey Kurennoy
Majed Dodin
Tural Gurbanov
Ana Peleteiro Ramallo
Venue: NeurIPS 2025 poster ()
Keywords: A/B testing
experiment monitoring
sequential test
early stopping
Primary area: evaluation
TL;DR: We propose a new sequential test for online experiments that permits early stopping without increasing false detection risk and outperforms existing methods in statistical power, as shown in semi-synthetic simulations.

Abstract: The online evaluation of machine learning models is typically conducted through A/B experiments. Sequential statistical tests are valuable tools for analysing these experiments, as they enable researchers to stop data collection early without increasing the risk of false discoveries. However, existing sequential tests either limit the number of interim analyses or suffer from low statistical power. In this paper, we introduce a novel sequential test designed for the continuous monitoring of A/B experiments. We validate our method using semi-synthetic simulations and demonstrate that it outperforms current state-of-the-art sequential testing approaches. Our method is derived using a new technique that inverts a bound on the probability of threshold crossing, based on a classical maximal inequality.

YEAST: Yet Another Sequential Test
Alexey Kurennoy
Majed Dodin Delivery Hero
Meta akurennoymmeta.cmm com
majed wodinn@delivvrryerreer
Tural Gurbanov SoundCloud tural. www..urrraaorossunnoooo.ddddd com
Ana Peleteiro Ramallo Preply ana. pw..peeetiiropprppyyyco com
Abstract
The online evaluation of machine learning models is typically conducted through A/B experiments. Sequential statistical tests are valuable tools for analysing these experiments, as they enable researchers to stop data collection early without increasing the risk of false discoveries. However, existing sequential tests either limit the number of interim analyses or suffer from low statistical power. In this paper. introduce novel sequential test designed for the continuous monitoring of A/B experiments. We validate our method using semi-synthetic simulations and demonstrate that outperforms current state-of-the-art sequential testing approaches. Our method is derived using new technique that "inverts" a bound on the probability of threshold crossing, based on a classical maximal inequality.
Introduction
Online A/B experiments have become standard approach for evaluating the impact of machine learning models on the systems and products where they are deployed. Although this approach offers significant advantages, it also poses the risk of exposing users to harmful changes that could degrade their experience. Unlike traditional techniques such as the Student's test, sequential testing supports repeated significance checks without increasing the false positive rate. This makes it well-suited for real-time monitoring of experiments, allowing early detection of harmful effects, which helps protect user experience and reduce financial losses.
Sequential testing methods can broadly divided into discrete and continuous. Discrete sequential tests permit only fixed number of interim checks while continuous methods impose no such limit and allow checking for significance at the experimenter's 's will even after each new observation. In this paper, focus continuous sequential tests. We find that the absence of restriction the number of interim checks useful in practice, as it avoids situation where the experimenter and the stakcholders observe degradation in the metric of interest but have to wait until the next checkpoint before assessing the statistical significance and making decisions.
In this paper, we propose novel continuous sequential test that demonstrates better control of Type-I error and greater power than current state-of-the-art methods for continuous experiment monitoring, The theoretical technique use to derive and (asymptotically) justify this novel test is, to the best of our knowledge, also new.
We validate the proposed sequential test using semi-synthetic simulation experiment based on public real-world data set. We share the associated code for greater reproducibility (16).
To summarise, our main contributions are as follows.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

We propose novel (asymptotic) sequential test. This test supports unlimited interim significance checks.
We present new theoretical technique that consists of inverting the bounds obtained from the (generalised) Levy inequalities [2T] Paragraph 29.1, A].
We conduct an empirical study and demonstrate that the proposed sequential test has better Type-I error control and higher power than current stateoof-the-ae methods for continuous experiment monitoring.
We share the associated code 16 for greater reproducibility and to promote further research on this topic.
The remainder of this paper is structured follows. Section 2 reviews related work. Section introduces the proposed method, explains its derivation, and then presents formal theory. In Section 4 assess the correctness of the proposed method using semi-synthetic experiment (based on a public real-world data set). Section disccsses the limitations of the proposed approach and Section concludes.
2 Related Work
Our work is related to a large literature on sequential decision-making based on sequences of hypothesis tests, confidence sequences, sequences of "always-valid" p-values. While early work in this field was primarily concerned with clinical trials and industrial quality control [6 4 2). more recent work is typically motivated by the sequential nature of online experiments, where users arrive sequentially and generate outcomes that are observed quickly relative to the duration of the experiment [17 [4] [1]. Here, the goal of sequential testing is typically either reduce the opportunity cost of longer experiments, as in the context of best-arm identification in multi-armed bandit problems
13
39 12. or to manage risk by faster detection of harmful treatments 20
2.1 Martingale Methods and E-Processes
Building on the seminal work of Wald (36 and Doob 77. most modern sequential procedures are based on maximal inequalities for appropriately constructed martingales that provide unme-uniform statistical guarantees. While Wald's original sequential probability ratio test (SPRT) was developed from scratch, its construction essentially of this "martingale-type"" SPRT sequentially evaluates the likelihood ratio test statistic at each sample size for the null hypothesis H, e Ou against the simple alternative H1 0 01. The test continues until the statistic crosses one of two predefined bounds, at which point the null hypothesis is either rejected or not rejected accordingly. SPRT requires specifying a single-point alternative, which limits use for general-purpor To address this limitation, likelihood ratio methods using the method of mixtures (mSPRT) have been developed. These approaches replace the single-point alternative with a composite hypothesis and the standard likelihood ratio with a mixture, which remains martingale under the null (27).
experimental monitoring.
In addition mSPRT, the class of martingale methods encompasses more sophisticated approaches constructed using other martingales and falling under the framework of generalized always-valid inference (GAVI)
19
Another recent branch martingale harde methods built upon the principle of testing by betting (see 25 and references therein).
The martingale-based approach was generalised through the concept of e-processes
stochas- tic processes characterised by property that allows for straightforward construction of sequential tests. The roots of this perspective can traced back to (35), who introduced the use of test mar- tingales within game-theoretic framework foundation for statistical testing. This remains an active area research, with growing literature developing new constructions and applications of e-processes
31 32
See (26 for a recent overview of the topic and further references.
2.2 Group Sequential Methods
conceptually different approach followed by methods based on group sequential tests (GST)
44
23
33
8
GST methods rely (asymptotic) characterizations of the joint distribution of

---

vector of test statistics across increments of data, which permits the computation of sequential boundaries that uniformly control the false discovery rate multivariate (numerical) integration. Farlier GST methods (such as [24 23 required fixed number and timing of interim analyses. In contrast, the later generalisation g permits flexible (in theory up to continuous) monitoring. This flexibility is achieved using alpha-spending function, which controls the allocation of Type I error across analyses, based on a prespecified maximum sample size.
2.3 Simple Sequential A/B Testing
Our approach has similarities to simple sequential A/B testing [22]. However, the test from [22 only allows monitoring count-type metrics (for example, the number of clicks or purchases), whilst our proposed method can be used to monitor metrics of any type, including real-valued financial metrics such as revenue. One can view our work as generalisation of [22], yet we emphasise that this generalisation is not at all trivial as the theory presented in [22] fundamentally supports only count-type metrics.
3 YEAST: YEt Another Sequential Test
In this section, we present novel, alternative sequential testing method proposed in this paper.
3.1 Problem Setup and Notation
Suppose we are conducting an A/B test and have a certain metric interest. We want to constantly track this metric and have tool check if the difference between the groups is at any point large enough to conclude that the treatment has an impact on the metric. The tool must have good control of the false detection rate despite the repeated checks. other words, must allow for peeking into the interim results of the experiment without inflating the false detection rate.
To develop tool satisfying the above requirement, consider the difference S in the total value of the metric of interest accumulated by the control (W = 0) and the treatment (W = 1) groups sum of increments over stream of events. For
Typically, this difference can be represented example, if the metric of interest is revenue then the tracked difference S can be decomposed into the sum of order values if it comes from the control group and with a negative sign if it comes from the treatment arm. Mathematically, the increments can be written as
checkout events where the order value is taken with a positive sign
(1) where Y; is the value outcome associated with the i-th event (such the order value in the example above) and W; is the event assignment indicator (it equals 0 if the event was generated by a subject from the control group and 1 if its subject belongs to the treatment group). With this notation at hand, the difference in the metric interest after observing n events can written as the runming sum
=1 2W,) Y,
Sn
n n L, X
(2)
illustrated in the example data set displayed in Table
1
Table 1: An Example of Experiment Data
timestamp
group
Y
X
S
2023-08-01
12:00:00
control
175.0
175.0
175.0
2
2023-08-01
12:00:02
treatment
35.5
-35.5
139.5
3
2023-08-01
12:00:05
treatment
20.0
-20.0
119.5
4
2023-08-01 12:00:10
control
100.0
100.0
219.5
So far we have used revenue monitoring as an example but one can think of examples from other domains such video streaming (metric: hours streamed; the events are user sessions; the event value
3

---

is the total duration of videos streamed within the session). online advertisement (metric: number of clicks; the events are clicks; the event value is 1), medical drug trial (metric: number of cases of side-effects; the events are incoming patients; the event value is the indicator of side-effect presence after taking the drug), online subscription services (metrie: number of new users who convert into a paid subscription; the events are newly acquired users; the event value is the indicator of subscribing), banking (metric: total transaction fee; the events are transactions; the event value fee amount), etc.
3.2 Assumptions and the Null Hypothesis
We will now state and discuss our assumptions and the null hypothesis.
Assumption 1 (Random Subject Assignment) The subjects are assigned to the control or treatment uniformly at random and with equal probability.
We work with the following null hypothesis.
Ho: the treatment does not affect either the event outcome distribution or the frequency of events, so that
(a) the event outcomes and the assignment indicators are independent from each other, i.e., {Yi, Y2, ...} TT (W1, ...).
(b) and the frequency of events in each group is the same, implying (under Assumption that
Pr(W, 1) 0.5 11 1.
(4)
In the setting of this paper, it is important to distinguish between subjects (such as users) and events generated by subjects (such as orders). Throughout the paper, we work with variables associated with events and not subjects. For example. Y,, 2 1, are event outcomes and W,, 2 1, are event assignment indicators. One implication of this is that even though the subjects are assigned one of the two groups independently at random, the variables associated with individual events can be dependent (as some of them can be generated by the same user). Therefore, in our theory (Theorems and[), we allow the event outcomes Y, i 2 1, to be dependent and, likewise for the event assignment indicators W,, i 2 Part (a) the null hypothesis stated above only assumes that the two sets of variables ([Yi, Y2, . and (W1, W2, ...) are independent from each other, but the variables within each the two sets are allowed be dependent.
The null hypothesis stated above exhibits the difference between our setup and the classical testing for a zero average treatment effect. Specifically,
our testing procedure acts on the level of events generated by experiment subjects and not the experiment subjects themselves (for example, orders and not customers);
we assume that under the null hypothesis, the treatment does not affect the "event generation process" in any way.
In the revenue monitoring example, the latter implies that both the frequency of orders and the order value distribution stay the same in both groups under the null. We discuss the limitations of the considered setup in Section 5
The approach described so far allows us to derive simple but effective sequential testing procedure we present next.
3.3 Proposed Sequential Testing Method
The proposed monitoring method consists of tracking the difference in the metric of interest between the experiment groups and (continuously) comparing with a constant (alerting) boundary. Whenever the boundary is crossed, can conclude (with a predefined significance level a) that the treatment has an impact on the metric.
The procedure requires setting the maximum number of events to be observed during the monitoring period that we denote by N. In other words, the boundary is computed assuming that the monitoring will not continue after N events have been collected. See Section 3.3.2 for how to set N in practice.

---

The proposed method has only one other parameter that needs to be estimated beforehand the (scaled) variance of the difference in the metric of interest the end of monitoring, VN v/r(Sx)/N. The estimator Vy needs to be consistent under the null hypothesis. Given the estimate Vi (computed by such estimator), the alerting boundary is set
b" =1 NVN.
(5)
where 21-a/2 is the quantile of the standard normal distribution of level 1 ax/2. The proposed simple sequential test proceeds by tracking S, and comparing it against b*
The necessity to fix or estimate N and to provide an estimate of VN beforehand can be perceived as disadvantage the proposed approach. However, we argue that in many practical scenarios, those parameters can be reliably set based on pre-experiment data (in fact, the underlying data distribution has to have some level of stability or regularity in time for the A/B testing to be meaningful in the first place).
3.3.1 Informal Derivation of the Proposed Method
The derivation of the alerting boundary (5) follows two steps. First, we use (generalised) Levy's inequality" to bound the false detection rate by the probability that the tracked sum exceeds the threshold at the end of monitoring, i.c.,
N max S. >>) b) s 2r (S > b}. n=1
FDR = Pr
(6)
In the second step we use the Central Limit Theorem to approximate the latter probability,
(S > bj Rz
Vvar(Sw)
VNVV
where denotes the CDF of the standard normal distribution. Then by setting the threshold b to 21 21-a/2 VNVN (cf. (5), we ensure that the false detection rate is under control, i.c., FDR S This is formalised in Theorem bellow.
The proposed testing procedure is defined in Algorithm 1 and will be referred to as YEAST (from YEt Another Sequential Test). The estimation of Vx is discussed in Section 3.3.2 below.
Algorithm 1 YEAST
Require: N. VN
b" + 21-0/2 V NVv for 1, ...... N do
if S, b' then flag significance end if end for
For two-sided test, we track |.S,| and compare it against bi-sided 21-0/4 VNVN.
Mind that differently from the non-sequential testing, the significance level needs be divided by for the one-sided test and by 4 for the two-sided test. This is due to the factor of two in the right-hand side of bound (6)
Note that while baring some resemblance in the formulas, YEAST is fundamentally different from GST (8) methods. The latter a statistic normalized by the observed information (i.e., the variance of the cumulative data up to time n) and adjust the rejection threshold at each look. In contrast, our method normalizes the partial cumulative sum S, by the variance of the full dataset var (Sx) and compares it to a fixed threshold. As a result, the boundary of YEAST is constant (does not change from one look another) while this is not the case for GST. The computation of the YEAST boundary is computationally simple and does not require numerical integration.
We leverage generalised version of the inequality from 21 Paragraph 29.1. A] that allows the observations be dependent.
5

---

3.3.2 Setting the Input Parameters
As was described in the previous subsection, the computation of the alerting boundary of the proposed sequential testing procedure requires two inputs: the number of events N to be observed during the monitoring and an estimate Vy of the (scaled) variance of the tracked metric difference at the end of the monitoring. In this subsection, we discuss how to provide those inputs in practice.
To set N, we estimate the expected number of events to be collected over the experiment time frame. In the simplest form, this can amount to setting N to the number of events observed in pre-experiment period of the same duration as the experiment we want to monitor.
For variance estimation, we suggest using pre-experiment data as well. Obtaining an accurate estimate this way requires the pre-experiment period to be representative of the experiment time. However, in cases where this cannot be assumed, the validity of fixed-duration randomised experiments is questionable in principle (as the effects measured during such experiments may not generalise beyond their time frames). So in the context of A/B testing, this is a natural assumption. Note that estimating variance from pre-experiment data is standard procedure performed before an A/B test because is used in computing the MDE (minimum detectable effect) and identifying the necessary sample size.
As noted in Section 3.2. the event outcomes Y,, 2 1, can be dependent because some of the events are generated by the same user. The same holds for the event assignment indicators W,, i 2 1. Consequently, the increments X,, i 2 1, of the monitored trajectory can be serially correlated, in which case the variance var(SN) does not equal the sum of the variances var(X;) The serial correlation of the increments has clustered nature in this case because variables corresponding to different users are independent due to random assignment. Therefore, we propose to use an estimator that is robust clustered serial correlation ① to estimate VN. Implementations of such estimators readily available, for example as part of the sandwich package in R [4 [4) (see function vcovCL). The cluster variable in our case is the user identifier. As we show in our experiments with real-woria data in Section 4 the use of a robust (clustered) variance estimator can be utterly important for
effective control of the false detection rate.
Although the need to set a finite horizon N can be viewed as a disadvantage of YEAST, we have not found it problematic in practice. When rumning A/B experiments it is typical to plan for certain duration, and the number of observations collected during the experiment time frame is usually predictable. In addition, the decision boundary of YEAST depends on the square root of N, which makes the method more robust to inaccuracies in the estimation of N. That said, we acknowledge that there can be situations where it is difficult to reliably estimate N and, in that case, it may be better to resort to other methods (e.g., (II) that do not require setting a finite horizon. In the empirical evaluations in Section 4 we never assume that N is known and estimate it from pre-experiment data (as part of the method). Hence, the respective empirical results incorporate the uncertainty in the estimation of N.
3.3.3 Formal Statements
In this subsection, we will justify the correctness of the proposed testing procedure. This includes a statement demonstrating that the proposed method controls the false detection rate (Theorem [ and a statement showing that the power of the method is asymptotically one (Theorem
In the formal statements below, (Ω, F. Pr) is probability space and IR stands for the set of real numbers. As before, & denotes the CDF of the standard normal distribution. The proofs of the statements can be found in Appendix A.
The first theorem states that when the treatment has no actual impact, detections occur sufficiently rarely.
Theorem (False Detection Rate Control), Let Y;: Ω + IR and Wi: S2 + {0, 1 N. be random variables and let S,, > 1, be the running sum defined by 11-2 Suppose that Assumption and the null hypothesis (3) 4 hold and that a version of the Central Limit Theorem
valid for the sequence of random variables X,, for all N Z and set VN == var(Sx))
defined by (1. Assume that var(Sx) > 0
Then for any 7 > O and any & > O there exists No,e such that for all N 2 Ny.e we have
N S, TV NVN 22 1 (7)) +
Pr max n=1

---

In particular. for any significance level a E (0. 1) and any > Q the false detection rate Pr mmax Sr =1-a/2VNVV is bounded by a + e, provided that N is sufficiently large.
Note that we deliberately do not spell out specific conditions, ensuring that the central limit properties hold for the sequence of increments {Xt}.. Multiple sets of such conditions exist. A good overview of them can be found in [38 Chapter V].
The above theorem not only justifies the proposed method, but also provides insight into how the quality of the false detection rate control depends on the error in setting the input parameters of the method (NV and/or Vvv) Specifically, if we overestimate or underestimate the product NVv by a factor of r the effective threshold used during monitoring will equal 21 = /V/NN and, as
follows from I the false detection rate will be (approximately) bound by 2 (1 ① (21-a/2VT)
instead of a. For example, if NVv is overestimated by 20% the bound becomes equal approximately 3% instead of the nominal significance level of 5%.
Our second theorem claims that, when treatment has an effect, its detection can be made as certain as desired if we monitor for sufficiently long.
Theorem (Power). Let Y;: S2: IR and W;: + f0. 13. N. be random variables, and let Sa, n. 2 1, be the running sum defined by [T) 1-- Suppose thut Assumption 1 holds and
a version of the Central Limit Theorem valid for the sequence of random variables X1, X2, defined by (T) Assume that war(Sx) > O for all N 2 lnnd that the set of variables VN
var(SN) //, N 2 1, is stochastically bounded. Finally, let E|X,] = μ > 0 (implying that the null hypothesis
is violated). Then Pr maxx x= Sn 21 21-a/22
1 N - 00
4
Validation Using Real-World Data: Semi-Synthetic Experiment
The best possible real-world validation would be using data from very large number of A/B tests (some with statistically significant results and some with neutral outcomes under high power). Having those data, one could apply the compared sequential tests retrospectively and measure their false detection rates and sensitivity (power). As we do not have access to a sufficient number of past A/B tests, we chose a different approach and conducted semi-synthetic study, in which we took a public real-world data set ("Online Retail" dataset [5]), randomly assigned users in that data set to control and test, computed the cumulative difference in the metric of interest (revenue) between control and treatment for each assignment replication, and measured the associated detection rates. (For power assessment, we also artificially decreased the order values considering multiple effect sizes, see Section 4.2 below.)
The code implementing our experiments is openly available in the git repository [16
Here are the methods we compared.
YEAST The proposed sequential method using the significance boundary
(5) as presented in Sec-
tion 3
mSPT The mixture sequential probability ratio test 27 20
We set the tuning parameter of the method to 11, 25, and 100 and denote the corresponding versions as mSRTphill, mSRTphi25, and mSRTphi50.
we set the numerator parameter p of the method to 250, 500, and 750 and denote the corresponding instances of the method by GAVI250, GAVIS00, and GAVI750, respectively. We also included GAVI with the tuning parameter p set to 10,000 (the default setting used by Eppo).
GAVI The generalization of the always valid inference, as proposed in 11 As in 28
LanDeMetsOBB The GST method g with the O' fieem-n alpha-spending function 23 For computational reasons, we constructed the boundary using 100 interim checkpoints and then extended it in a piecewise-constan manner to support continuous monitoring.
Seq The sequential predictive test from 25 with the online Newton step (ONS) strategy for selecting betting fractions. We used QDA (quadratic discriminant analysis) as classification model for this method.
7

---

4.1 Type-I Error (False Detection Rate)
The public dataset [5] consists of all transactions that occurred between 2010-12-01 and 2011-12-09 on a UK-based online store. We used the data up until 2011-11-30 to have 12 complete months and dropped orders with missing customer identifiers. The resulting dataset had 21,269 orders. We split it into two halves: the first 6 months were used for estimating parameters (NV and Vx) and the latter 6 months for validation. Using the validation period, we randomly assigned customers to control and treatment 100,000 time?" computed the cumulative difference the revenue between control and treatment for each assignment replication, applied the sequential tests to monitor the cumulative difference, and measured the frequency of detections over the replications. Since no actual treatment was applied to the replicated treatment groups, the null hypothesis was true, and the computed rates were false detection rates.
It is typical of online transaction data to contain outliers (e.g., large wholesale customers that can easily skew the monitored metric difference in favour of one of the groups). Therefore, before replicating the assignments, we removed outliers via revenue capping at the 99,9th percentile of the total revenue generated by customer. The capping was applied progressively as follows: given customer was only monitored while his/her revenue stayed within the cap and all subsequent events from that customer were discarded.
The measured detection rates are reported in Table Table together with the associated confidence intervals (computed with Bonferroni correction). Detection rates within one percentage point of the nominal significance level (set to 5%) are shown in bold.
As mentioned in Section 3.1 employed sequential tests with two different variance estimates. One of the two assumed that the increments X; of the monitored difference- -nnrrencc----uumm are independent. The other increments coming from the same user in our case 3 The results corresponding to the use of the two estimators reported in columns 'non-robust" and 'robust' of Table 2 respectively
estimator robust cluster serial correlation, that is to the correlation between
Two observations from our experiments are in order. Firstly, the proposed method (YEAST) with the robust variance estimator was the only method to demonstrate accurate false detection rate control (i.e., with the detection rate staying within one percentage point of the nominal significance level). Secondly, when the non-robust estimator used, the detection rate considerably inflated. This highlights the importance of accounting for the correlation in the events generated by the same user when estimating the variance.
Table 2: False Detection Rate
YEAST
1.044:0.000
0.167 i 0.004
MSPRT100
0.028 + 0.002
0.159 + 0.004
3
mSPRT1I
0.007
0.127 + 0.003
4
mSPRT25
0.002
0.141 # 0.003
5
GAVI250
0.002
0.148 # 0.003
GAVI500
0.027 0.002
0.156 0.004
GAV1750
0.029 10990000 0.002
0,160 + 0.004
8
GAVIIOK
0.028 + 0.002
0.160 + 0.004
9
LanDeMetsOBF
0.026 + 0.006
0.133 0.011
10
SeqC2ST-QDA
0.102 0.003
"We generated the assignments in parallel using 13 processes. The random seed of the i-th process was set to i. Each process but the last made 7692 replications while the last one made 7696 so the total number of replications was 100,000. The simulations take about 15 min to 18 GB RAM.
laptop with an Apple M3 Pro CPU and
'We used the vcovCL function of the sandwich R package to compute the robust variance estimate. To reduce the sparsity in the time dimension, summed the data (revenue increments) by user-hour before computing the estimator. See the linked repository [16] for details.
"SeqC2ST-QQ QDA does not take an estimate of the increment variance as input hence there is only one result for this method.
8

---

4.2 Type-II Error (Power) Assessment
To assess power, we conducted the same simulations as in Section 4.1 but additionally decreased the values of order values belonging to the treatment group in each replication. The relative decrease was equal 5, 10, and 20%.
To evaluate the sensitivity of the tests, we included an additional baseline: the standard non-sequential Student's I-test, applied once the end of each experiment replication using the full dataset. This test serves as a benchmark because it is widely used in practice and represents the level of power achievable when no early stopping employed (and thus no adjustments are needed control the false detection rate).
The reported results for the robust variance estimate. The detection rates be found in Table As can be seen, the proposed method (YEAST) is the only method that was not underpowered relative to the standard (non-sequential) Student's t-test.
Table 3: Semi-synthetic Experiment: Power (Online Retail)
method
relative decrease in the order value
0.05
0.1
0.2
0.5
Non-seq. Hest
0.115 0.004
0.253 0.005
0.686
0.005
1.000
0.000
1
YEAST
0.116 + 0.004
0.250 = 0.005
0.678 + 0.005
1.000 = 0.000
2
MSPRTI00
0.006 = 0.001
0.018 # 0.002
0.139 + 0.004
0.998 + 0.001
3
mSPRT011
0.002 = 0.001
0.007 # 0.001
0.072 # 0.003
0.993 # 0.001
4
mSPRT025
0.003 = 0.001
0.010 0.001
0.093 0.003
0.996 + 0.001
GAVI250
0.003
0.012 0.001
0.107 = 0.003
0.997 = 0.001
6
GAVISOD
0.005 0.000
0.017
0.131 # 0.004
0.998 + 0.001
7
GAVI750
0.006 + 0.001
0.00
0.146 # 0.004
0.999 + 0.000
8
GAVIIOK
0.012
0.008
0.232
1.000
LanDeMetsOBE
0.076 = 0.003
1.000
10
ScqC2ST-QDA
0.091 = 0.003
0.090 = 0.003
0.094 + 0.003
0.107 # 0.003
5 Limitations
We saw in Section that despite its simplicity, the proposed monitoring procedure demonstrated effectiveness in controlling type-I error and a higher power relative to existing SOTA approaches. Yet there are scenarios where the underlying null hypothesis of no effect on the data generation process is too strong, potentially leading to some "undesirable sensitivity". In other words, sometimes the treatment can alter the data generation process but have a neutral average effect on the metric of interest. We can think of (at least) two such scenarios.
Situations where the treatment brings multiple changes to the data generation process and those changes bantee-balance each other leading to a neutral average effect on the metric. For example, the treatment can make users place fewer orders but increase the average order value at the same time so the two effects neutralise each other and the revenue per user stays intact.
Situations where the treatment changes the variance of the event outcome, leaving the average unaffected. For example, the treatment can increase the variability in the order values without changing their average.
If in a given practical application, having sensitivity one of the scenarios above is highly undesirable, it is safer to adhere to alternative sequential testing methods (e.g., 11 T0).
Note that none of the problematic two scenarios can materialise when the event outcome is constant (i.e., the metric of interest simply counts events). In addition, the first scenario cannot hold when the treatment cannot affect the number of events, for example, when the events correspond to incoming patients in a medical study.
From a theoretical perspective, the amount of oversensitivity of the proposed method in the above two situations would depend on how much the median of the "residual sum" (or the sum of the remaining increments until the end of monitoring) deviates from zero Intuitively, in the early steps of monitoring, this should not be problem because the number of remaining increments is large and their sum's 's distribution would be approximately symmetric due to the Central Limit Theorem.

---

Yet, closer to the end of monitoring, this deviation can be considerable. Mitigation strategies can include stopping the monitoring early or setting the threshold if the monitoring is be run for longer than in reality. The effectiveness of those mitigation strategies and the tradeoff with the power would depend on how quickly the distribution increment sums converges to a normal distribution.
We see studying the behaviour of our proposed method in situations mentioned in this section and seeking mitigation strategies large topic for future research.
worth emphasising that in some situations, it can be desirable to capture all relevant potential defects (beyond the effect on the mean) (20). For example, increased outcome variance indicate that the effect of the treatment is heterogeneous and some important subgroups of the population can be negatively affected (even though the overall mean stays the same). In such scenarios, the extra sensitivity mentioned in this section not really a limitation but an advantage instead.
Conclusion
This paper proposes novel sequential test for continuous experiment monitoring. The proposed test supports both discrete and real-valued metrics. Its effectiveness is demonstrated in an empirical study semi-synthetic experiment based on real-world data). We hope that the proposed method will enable a wider use of sequential testing for online evaluation.
Acknowledgments and Disclosure of Funding
The authors are grateful to Evan Miller for insightful comments and his excellent blog [22], which sparked the idea for this research.
This research was carried out while all the authors worked at Zalando. The paper does not represent, imply, or establish any form of partnership, joint venture, or collaboration whether formal or informal - between the organizations with which the authors are affiliated at present.
References
[1] Jonah B. Gelbach A. Colin Cameron and Douglas Miller. Robust inference with multiway clustering. Journal of Business & Economic Statistics, 29(2):238-249, 2011.
[2] Peter Armitage et al. Sequential tests in prophylactic and therapeutic trials. Quarterly Journal of Medicine, 23(91): 1954.
[3] Aurélien F. Bibaut, Nathan Kallus, and Michael Lindon. Near-optimal non-pan con-aramettric sequential tests and confidence sequences with possibly dependent observations. 2022.
[4] I Bross. Sequential medical plans. Biometrics, 8(3):188-25) 205, 1952.
[5] Daqing Chen. Online Retail. http://doi.org// http:////g..2442///BWW333
UCI Machine Learning Repository, 2015. DOI:
[6] Harold French Dodge and Harry G Romig. A method of sampling inspection. The Bell System Technical Journal, 8(4))613-63 1929.
[7] J.L. Doob. Stochastic Processes. Probability and Statisties Series. Wiley, 1953.
[8] K. K. Gordon Lan and David L. Demets. Discrete sequential boundaries for clinical trials. Biometrika, 70(3):659-66) -03,3, 12 1983.
[9] Peter Grünwald, Rianne de Heide, and Wouter M. Koolen. Safe testing. Journal of the Royal Statistical Society. Series B: Statistical Methodology, 86(5):1091-1125 2024.
[10] Dae Woong Ham, lavor Bojinov, Michael Lindon, and Martin Tingley. Design- confidence sequences: general approach to risk mitigation online experimentation, 2023.
[11] Steven R. Howard, Aaditya Ramdas, Jon McAuliffe, and Jasjeet Sekhon. Time-uniform, nonparametrie, nonasymptotic confidence sequences. The Annals of Statistics, 49(2), apr 2021.
10

---

[12] Kevin Jamieson and Lalit Jain. A bandit approach to multiple testing with false discovery control. Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS' 18, page 3664-3674, Red Hook, NY, USA, 2018. 8. Curran Associates Inc.
[13] Kevin Jamieson, Matthew Malloy, Robert Nowak, and Sébastien Bubeck. lil' ucb An optimal exploration algorithm for multi-armed bandits, 2013.
[14] Ramesh Johari, Pete Koomen, Leonid Pekelis, and David Walsh. Always valid inference: Continuous monitoring of a/b tests. Operations Research, 706):1806-1822) 2022.
[15] Emilie Kaufmann, Olivier Cappé, and Aurélien Garivier. On the complexity of best arm identification multi-armed bandit models. Journal of Machine Learning Reseurch, 17:1-42, 2016.
[16] Alexey Kurennoy. Yeast: Yet another sequential test. Git Repository, 2024. This paper's code companion. httpps://gt thub. com/ akur akurennoyyyeattt
[17] Nicholas Larsen, Jonathan Stallrich, Srijan Sengupta, Alex Deng, Ron Kohavi, and Nathaniel Stevens. Statistical challenges in online controlled experiments: A review of a/b testing methodology, 2023.
[18] Michael Lindon, Dae Woong Ham, Martin P. Tingley, and lavor Bojinov. Anytime-valie valid linear models and regression adjusted causal inference in randomized experiments. 2022.
[19] Michael Lindon and Alan Malek. Anytime-valic valid inference for multinomial count data. In
S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 2817-2831. Curran Associates, Inc., 2022.
[20] Michael Lindon, Chris Sanden, and Vaché Shirikian. Rapid regression detection in software deployments through sequential testing. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '22, page 3336-3346, New York, NY, USA, 2022. Association for Computing Machinery.
[21] M. Loeve. Probability Theory. 2nd Edition. D. Van Nostrand Company Inc., 1960.
[22] Evan Miller. Simple sequential A/B testing. Blog Post, 2015. https: http::://ww..vaatt wwanmi.wwwwa..m..mmmmm.mmm...... 1ler
atz//aquantiial----ttttiit
htmml
[23] Peter C. O'Brien and Thomas R. Fleming. A multiple testing procedure for clinical trials. Biometrics, 35(3):549-556, 1979.
[24] Stuart Pocock. Group sequential methods in the design and analysis of clinical trials. Biometrika, 6412):91-1199, 1977.
[25] Aleksandr Podkopaev and Aaditya Ramdas. Sequential predictive two-sample and independence testing. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 53275-53307 Curran Associates, Inc., 2023.
(26) Aaditya Ramdas and Ruodu Wang. Hypothesis Testing with E-values. Statistics & Data Science Monographs Now Publishers. Now Publishers, 2025. Monograph; unified treatment of e-values, including e-processes; 16 chapters, 426 pp-
[27] Herbert Robbins. Statistical Methods Related the Law of the Iterated Logarithm. The Annals of Mathematical Statistics, 41(5):1397 1409, 1970.
[28] Märten Schultzberg and Sebastian Ankargren. Choosing sequential testing framework comparisons and discussions. Blog Post, 2023.
[29] Märten Schultzberg and Sebastian Ankargren. Simulation files for Schultzberg & Ankargren's blogpost. GitHub repository, 2023. choosing- sequential testingg frattiin-framworrkk Brramewor--comppaiisons-oooooe and and-discussions
11

---

[30] Glenn Shafer. Testing by betting: A strategy for statistical and scientific communication. Journal of the Royal Statistical Society. Series A: Statistics in Society, 184(2):00-431, 2021.
[31] Shubhanshu Shekhar and Aaditya Ramdas. -heoethe formulations of sequential non- parametric one- and two-sample tests. arXiv preprint a/X0v221 2021. Later version published in IEEE Transactions on Information Theory (2023).
[32] Shubhanshu Shekhar and Aaditya Ramdas. Nonparametrie two-sample testing by betting. IEEE Transactions on Information Theory, 70(2), 2023.
[33] Eric V. Slud and L. J. Wei. Two-sample repeated significance tests based on the modified wilcoxon statistic. Journal of the American Statistical Association, 77:862-868, 1982.
[34] A.W. van der Vaart. Asymptotic Statistics. Asymptotic Statistics. Cambridge University Press, 2000.
[35] Vladimir Vovk. A logic of probability with application to the foundations of statistics. Annals of Statistics, 21(4):247-276, 1993.
[36] A. Wald. Sequential Tests of Statistical Hypotheses. The Annals of Mathematical Statisties, 16(2):117 186, 1945.
[37] G. Wassmer and W. Brannath. Group Sequential and Confirmatory Adaptive Designs in Clinical Trials. Springer Series in Pharmaceutical Statistics. Springer International Publishing, 2016.
[38] H. White. Asymptotic Theory for Econometricians. Economic Theory, Econometries, and Mathematical Economics. Elsevier Science, 1984.
[39] Fanny Yang, Aaditya Ramdas, Kevin Jamieson, and Martin Wainwright. A framework for multi-a(rme)/b(andiit testing with online fdr control. Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS' 17, page 5959-5968, Red Hook, NY, USA, 2017. Curran Associates Inc.
[40] Achim Zeileis. Econometric computing with HC and HAC covariance matrix estimators. Journal of Statistical Software, 11(10):1-17, 2004.
[41] Achim Zeileis, Susanne Köll, and Nathaniel Graham. Various versatile variances: An object- oriented implementation of clustered covariances in R. Journal of Statistical Software, 95(1):1- 36, 2020.
NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's s contributions and scope?
Question:
Answer: [Yes]
Justification: the abstract and introduction state which class the proposed sequential test belongs to (continuous monitoring, i.e., unlimited interim checks), how it differs from SOTA methods (better Type-I error control and higher power as established empirically); the main contributions are explicitly enumerated; the assumptions and limitations of the method are too technical/nuanced to be stated the introduction and the reader is referred to the dedicated sections, coming after an appropriate problem setting.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: the paper has a dedicated section discussing limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
12

---

Answer: [Yes]
Justification: Each of the two theoretical results presented the paper (Theorems and has full list of assumptions in its statement and is supplied with complete formal proof (moved to the appendix due to space limitations). A sketch of the proof of the main theorem (Theorem D is included in the main contents of the paper (Section 3.3.T.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: The paper has a clear statement of the proposed algorithm and detailed description of the experimental setup, including a reference public dataset [5] and the random seeds for the (semi-synthetic) simulations (see Section 4). In addition, we provide link to openly available code implementing the experiment [16].
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The dataset used in the paper is public [5] and the code used for the experiments is made openly available too via an anonymous repository [16]
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Data preprocessing, splits, and other experimental setting details described in Section
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: All experimental results are reported together with confidence intervals (cor- rected for multiple hypothesis testing using the Bonferroni procedure).
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: This is specified in footnote in Section
9. Code of ethics
Question: Does the research conducted in the paper conform. in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The authors not aware of any potential harmful consequences of the research and belive is fully conformant to the NeurlPS Code Ethics.
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
13

---

Answer: [Yes]
Justification: There is no direct societal impact of the work performed beyond making A/B testing more efficient and less disruptive experience, which is mentioned in the introduction (end the first paragraph).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper does not release datasets or models.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The public dataset used in the experiments is licensed under CC BY 4.0 which allows its use for any purpose provided that an appropriate credit is given. We make explicit references the dataset when describing the experiments. We also include the associated introductory paper for the dataset in the bibliography and cite it in the manuscript.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The code released together with the paper has a README file and comments.
Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects. 15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing or research with human subjects.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: LLMs were used solely for text editing purposes and do not impact the methodology, scientific rigorousness, or originality of the research in any way.
14

---

Appendix
A Proofs and Auxiliary Statements
Lemma 1. Let Y;: + IR and W - {0, -,, N, be random variables, and let Sn.
1. be the running sum defined by T Suppose that Assumption 1 and the null hypothesis hold.
Then for any N 2 and any k = 1, N the distribution of SN Sg conditional on Si...., Sk symmetric around
(for N = 1 k N the statement is trivial). Let T = (-1, +1 +1] -k be the set of all possible combinations of -1 and +1 of length N Furthermore, for any E T, let 7 E 'T be the combination obtained by negating every element of Next, define
Proof. Fix any N > 2 and any k N
ie., Tj- = TT for all j = 1, ....,
A, = {1 - 2W; = Ti-k Vi = k N}. TET.
Note that
A, NA 0 r, r' T and UtET A, USET Ω Also, Assumption together with the null hypothesis imply that
(8)
Pr(A, V1, WI...., Yk, Wx) = Pr(A; Yi, WI,..., YR, Wk) a.s. VrET. Finally, fix an arbitrary Borel set Q C IR and define
(9)
== {Sx - S₁ E Qh, B= (Sp SN E EQ-
Denoting the conditional probability Pr(- Y1. WI...., Yz. Wa) by Prk, we will now derive the
key relationship for the proof of the lemma. Specifically, and (9) imply that for any t E T
have
N = Prk ((- Σ
T- T
Pra Pra(BnA)
i=k+1
Prk
Σ
T k ViE
Pr*(At)
1=++1
PTk
Σ
Pra(Ar)
i=++1
=
Prk
Σ T-
Prx(At)
1=t+1
5
Prk Σ PI-KYi E Q E nAp
iik+1
(rr(BnA a.s.
(10)
From 10 and (8) it follows that
Pr
Σ (Bn
10
Pry (BaBAAS Pra(B) a.s.,
TET
TET
which by the tower property implies that
Pr(B S1, ... St) P Si,..., Sτ)
(11)
Since the fixed Borel set Q was arbitrary, [T] proves that conditionally on S....., Sk. the distribu- tion of SN S₁. and S₁ Sy is the same or, equivalently, the distribution of SN Sg is symmetric about zero.
In the second and next to the last transitions, we leverage the fact that if two collections of random variables are independent, they remain conditionally independent after conditioning on any subset of variables from either both collections.
15

---

Lemma 2. Suppose that the assumptions of Lemmna 7 hold.
Then for any N and any b O, we have
Pr
max n=1
Sn 2 2b] < 2Pr[SN 2 b}.
(12)
and
Pr
max
S 2 b) < P 2 b}.
(13)
Proof. From Lemma | it follows that for any = 1, N, the distribution of SN -S. conditional
on S1, ..... Sg is symmetric about 0 and therefore the conditional median y(SN -S, Sx St, .... St) zero. Then 12 and (13) easily follow from the generalised Levy's inequalities 21 Paragraph 29.1.
A].
Proof of Theorem 7 This proof has two steps. First, we bound the chance of "crossing the threshold" any moment during the monitoring by twice the probability of crossing Secondly, we approximate the latter probability with the help of the Central Limit Theorem (noting that the tracked difference the end of monitoring is the sum of large number of increments). This allows us to construct the threshold appropriately, ensuring that the method's 's false detection rate is small enough.
the last moment.
Fix an arbitrary y > 0 and & > 0.
By the continuity of Φ there exists a sufficiently small > O such that
2 ( 5) 4
(14)
Furthermore, the null hypothesis implies that
E[X,] [(1 2W.) YY]
B Pr L))
for all 2 1. Therefore, E[5x] = 0 for all 2 and by the Central Limit Theorem we have that
Pr SN/ V (Sx) < 5 0} ( 6) S 4
(15)
for all sufficiently large N. relations
14
give
N Pr max Sn > NVN n=1
Pr
max S, > 7 / n=1
Pr
max Sn 2 vvrr(SS))
2Pr
SN 2 1V vvrr(s))
2Pr [sx/ war var(SN) 2 27}
Pr { (x// var
var(SN) (SN) > 7- -6
<
(1 Pr SSN (SNN var (SN) S -7-66) 67)
2 (1)) (7) (7 8))
+ (0 5) - Pr SN //l//V/ari var(SN) S 6))
H 15
2 (1 (1-((1))
2
2 2 ((+)) +
for all large enough N. particular, for any (0, 1) and any > we have that
Pr max n=
Sn 21 NVN < (1 (21-a/2)) ===a+t
for all sufficiently large N. This completes the proof.
16

---

Proof of Theorem The idea of the proof is to note that the probability of detecting the effect at any moment during the monitoring is at least high as the probability of detecting at the very end. Then we leverage the Central Limit Theorem to show that (when the treatment effect exists) the latter probability approaches as the monitoring duration increases.
Let En stand for the centered and standardised sum (Sx - NNu)/va(Sxx). By the Central Limit Theorem EN S (0, 1) and therefore EN is stochastically bounded (see, for example, [3 Theo- rem 2.4]).
Now, fix an arbitrary e > 0. From the stochastical boundedness of EN and VN, it follows that there exist M1 and M2 such that for all sufficiently large N we have
Pr [1EN = MI 11-18-8 2
(16)
and
Pr S M2] 2 2
(17)
From 16
17
it follows that
Pr EN + HVNN/VV 2 M1 + uNN / M2 21-6
for all large enough N. But then for any sufficiently large N, we have that
Pr
PrfEN EN + WVN N/NVVVVV a-a/2 s
(18)
(-M M/NNN///
because
as N co. Then derive that
Pr
N max Sn > 31 21/2 NV 2 n=1
Pr Pr SN > 21-0/2 /
Pr + HVN/N////VV 21-0/2
--c
for all sufficiently large N. Since E was arbitrary, this completes the proof.
B Experiments using synthetic data
Our simulation setup follows a recent study (28. To the best of our knowledge, this is the only existing comparative study of sequential tests. We extended implementation (2 with the method proposed in Section 3 of the paper (YEAST).
As in 29 we generate observations from the control, Y -=.i= 1, .... N. and observations from treatment, Y,, with N set to 500. Both Y and Y, are generated as IID random variables. The effect size (on the mean) took values 0.0, 0.1, 0.2, 0.3, and 0.4. The X, variables (increments in the metric difference between the two groups) were computed as X; Y Y,, I..., N.
We conducted two simulation experiments. The first experiment repeats the simulation study from 28 but adds YEAST to the comparison. In the second set of simulations, we explored the behaviour of the compared methods when the increments are non-normal.
In experiments, the target significance level was set at 5% and the number of replications was set to 100,000.
The simulation code for these experiments can be found the associated git repository [16].
B.1 Simulation Results
In this experiment, observations Y and Y were drawn from normal distributions with parameters (1, 1) and (1 E, 1), respectively. The effect size & took values 0.0, 0.1, 0.2, 0.3, and 0.4. The random generator seed was set to 8163 (as in 28 29
In the following present the list of methods we compared.
17

---

YEAST The proposed sequential method.
YEASTuy(K) (with K 80, 90, 110, 120) are the instances of the proposed method with the product NV misestimated by a factor 0.8, 0.9, 1.1 and 1.2, respectively (ie the method is applied with NVN set at 10% and 20% below and above the true value). Since the alerting boundary of YEAST depends on the product of N and Vy we can study the effect of inaccuracies in their estimation together.
mSPRT
The mixture sequential probability ratio test 20]. method to 11, 25, and 100 and denote the corresponding versions as mSRTphil1, mSRTphi25, and mSRTphi50.
We set the tuning parameter of the
GAVI The generalization of the always valid inference, as proposed in [1]] As in 28 we set the numerator parameter P of the method to 250, 500, and 750 and denote the corresponding instances of the method by GAVI250, GAVI500, and GAVI750, respectively.
LanDeMetsOBB The GST method with the O' O'Brien-Flemine alpha-spending function 223 For computational reasons, we constructed the boundary using 100 interim checkpoints and then extended it in a piecewise-constant manner support continuous monitoring.
CegC2ST-QDA The sequential predictive test from 23] with the online Newton step (ONS) strategy for selecting betting fractions. We used QDA (quadratic discriminant analysis) classification model for this method.
Bonferroni A naive approach using Bonferroni corrections.
All of the compared methods employed in the continuous monitoring mode meaning that the check for significance was performed after each observation. In Appendix [ we report additional evaluation results for the case where YEAST was employed in the "discrete mode" (i.e., with fixed number of interim significance checks).
For each experimental setting, conducted 100,000 replications. Each replication result in detection or no-detection. A detection occurs when the respective test flags significance (at any point of the monitoring process). We compute the share of replications where detection occurs. When the treatment does not have an effect this share is the so-called (empirical) false detection rate (or, synonymously, false positive rate, type-I error, test "size"). In settings where the treatment has an effect, this share is the (empirical) power. Tableeppreeentt the measured false detection rate and power, along with their corresponding 95% confidence intervals adjusted for multiple comparisons.
The methods that keep the false detection rate below the nominal level 5% and have the highest power are shown in bold.
One can see from the table that YEAST demonstrated the highest power among the continuous monitoring approaches that did not inflate the false detection rate. It means that while it kept the false positive rate under control when there was no treatment effect, YEAST had the highest sensitivity among the compared approaches when the treatment had an effect on the metric of interest.
Table 4: Simulation Experiment:False Detection Rate and Power
effect size
0.0
0.1
0.2
0.3
method
YEAST
0.047 : 0.002
0.448 # 0.005
0.902 # 0.003
0.995 # 0.001
YEASTa110
0.038 = 0.002
0.408 = 0.005
0.883 0.003
0.994 # 0.001
3
YEASTH120
0.030 +.00000002
0,372
0.004 0.004
0.864 0.003
0,992 : 0.001
4
YEASTH80
0.534
0.933 0.002
0.997 + 0.000
5
YEASTH90
0.059 + 0.002
0.490 + 0.005
0.918 0.003
0.996 + 0.001
mSPRT10M 100
0.016 0.001
0.235 + 0.004
0.751 # 0.004
0.976 + 0.001
mSPRTO11
0.032 4 0.002
4.004
0.739 4 0.004
0.972 + 0.002
8
mSPRT025
0.028 + 0.002
4.000
0.758 4 0.004
0.976 4 0.001
9
GAVI250
0.025 + 0.001
0.260
4.004
0.977 + 0.001
10
GAVI500
0.019 0.001
0.245
+.58
0.977 # 0.001
GAVI750
0
0.207
0.744
0.975 # 0.001
12
Bonferroni
0.008 + 0.001
0.067 1
0.440 + 0.005
0.877 # 0.003
13
LanDeMetsOBF
+.002
0.466
0.005
0.927 + 0.002
0.999 + 0.000
14
SeqC2ST QDA
0.022 + 0.001
0.037 + 0.002
0.092 # 0.003
0.220 # 0.004
18

---

Figure 1: Power Curves (under non-normal increment distributions)
student
gamma
1.00
YEAST
YEASTn110
YEASTn120
MSPRT100
detection rate
MSPRTO11
MSPRTO25
0.50
GAVI250
GAVI500
GAVI750
Bonferroni
LanDeMetsOBF
0.05
SeqC2ST_ODA
0.00
0.0
0.1
0.2
0.3
0.4
0.0
0.1
0.2
0.3
0.4
true relative effect
true relative effect
In Section D we additionally report sample (or, equivalently, time) savings that each of the methods produced on average (due to the early effect detection an interim check).
B.2 Non-Normal Data
The derivation of YEAST involves the application of the Central Limit Theorem to sums of X; (increments in the metric difference between the two groups). For fixed sample size, the quality of the normal approximation depends on the distribution of X, In this section, we explore the performance of YEAST in situations where the distribution of Y and Y is not normal. Specifically, we used two alternative distributions: Student's is which has heavier tales than the normal distribution and Gamma which asymmetric. Student's distribution had degrees of freedom and was shifted by v3 for the control and by v3(1 (11+14 {) for the treatment. The shifting was done to maintain the same coefficient of variation as in Section B.I.. The Gamma distribution had its shape parameter set to 1.0. The scale parameter equaled 2 for the control and 2(1 - <) for the treatment. The effect size & took the same values as in the first simulation experiment: random seed was set to 2023 for the simulations with the Student's distribution and to 2024 for the simulations with the Gamma distribution. The two seeds were different avoid dependence across the two sets of simulations.
0.0, 0.1, 0.2, 0.3, 0.4. The
The results are depicted in Figure The first data point each line corresponds to the case of no treatment effect and therefore represets the false detection rate. The remaining points represent the power for different treatment effect sizes. Similarly to the experiment with normal data, YEAST had a considerably higher power curve (both for the Gamma and Student's distribution cases) than other methods that did not inflate the false detection rate.
C Discrete Monitoring
In this section, we report additional evaluation results for the case where YEAST was employed in the discrete mode (i.e., with only fixed number of interim checks). The evaluation was performed against the same replications as in Section B.I We again follow the protocol from (28) and perform 14, 28, 42, and 56 significance checks (spaced equally across the timeline). Since in these evaluations, we operate in a discrete setting, we were able to include the discrete baselines from [28] in the comparison. Namely, compare against the following benchmark.
The group sequential test with alpha spending as proposed in 8]. The test performs a signifi- cance check after the arrival of each batch of observations. To schedule prespecified number of checks therefore needs an estimate during the experiment time frame. In the evaluations, we consider three different scenarios: when the sample size estimated precisely (right number checks), when the sample size is underesti- mated (leading to more checks than planned), and when the sample size is overstimated (leading
GST
the total number of observations that would be collected
19

---

to making fewer checks than planned). The respective entries in the evaluation table are referred as GST, GSToversampled. and GSTundersanpled, respectively. The actual sample size was 500 and the respective sample size estimates for the three scenarios were 500, 250, and 750. In the case of oversampling (i.e., the sample size underestimated), we apply the correction the bounds proposed in [37 pp. 78-79]. We consider quadratic and cubic alpha-spending, the latter having the "phi3" prefix in the name.
We report the share of replications where the test detects an effect (i.e., significance is detected in ("False Positive Rate") reports this share for the case
least one of the interim checks). Table ("False where no actual effect was present. All the compared methods except oversampled versions of GST keep the false detection rate below the nominal level of 5%. Table ( ("Power") contains the share of replications with a detection for the case where the treatment effect was set to 0.2 standard deviations. Methods with inflated false detection rate were excluded from the power comparisons. The GST method with cubic alpha-spending demonstrated the highest power, closely followed by YEAST and GST with quadratic alpha-spending. We find remarkable that our proposed method, despite supporting continuous monitoring, performed par with the GST method in the discrete monitoring setting.
Table 5: False Positive Rate
type
14
28
42
56
YEAST
0.04
0.04
0.04
0.04
2
GST
0.05
0.05
0.05
0.05
GSTphi3
0.05
0.05
0.05
0.05
GSToversampled
0.07
0.07
0.07
0.08
GSToversampledhhi
0.09
0.10
0.10
0.07
GSTundersampled
0.03
0.02
0.03
0.03
7
GSTundersampledphii
0.01
0.01
0.01
0.01
Table 6: Power (under a treatment effect of 0.2 standard deviations)
type
14
28
42
56
stream
YEAST
0.90
0.91
0.91
0.91
0.92
2
GST
0.90
0.90
0.90
0.89
3 GSTphi3
0.93
0.92
0.93
0.93
GSTundersampled
0.83
0.82
0.82
0.82
D Sample/Time Savings
The main benefit of sequential testing is the ability to stop the experiment early (once significance is flagged upon one the interim checks). This allows saving time and, the treatment harmful, reduce the negative impact. Thus, the amount of savings that is generated by sequential test on average is another important metric and we report the savings observed in the experiment from Section B. in Table The savings are measured as follows: if method identified the effect after the arrival 10% of the total number of observations, the associated sample (or, equivalently, time) savings would be 90%.
20

---

Table 7: Simulation Experiment: Sample/Time Savings, %
effect size
0.1
0.2
0.3
0.4
method
YEAST
13
39
58
69
mSPRTphi100
9
35
62
75
3
mSPRTphill
12
41
68
82
mSPRTphi25
12
41
68
81
5
GAVI250
12
40
67
80
GAVI500
10
37
63
77
7
GAVI750
8
34
60
74
Bonferroni
2
19
45
67
9
LanDeMetsOBIF
14
41
60
70
10
SeqC2ST-QDA
3
14
21


Reviews:
[{"review_id": "I93CCULAZw", "reviewer": "Reviewer_eKju", "summary": "A new sequential test procedure is proposed and analyzed. It is experimentally shown to outperform current/recent methods in both false detection rate (FDR) control as well as power. In terms of theoretical analysis, the authors essentially leverage Levy's inequality and the Central Limit Theorem to establish tunable FDR control and that the probability of correctly making a true detection tends to 1 asymptotically.", "questions": "**Question 1:**  Regarding Section 4.2's experiments to assess Power, I thought it was just a little curious that to synthetically generate a difference between control and treatment, a multiplier effect was chosen, rather than, say, an additive effect. I think an additive separation between control and treatment signals could be quite a reasonable phenomenon in a variety of contexts. While the results in Table 3 indicate that YEAST can outperform other methods by orders of magnitude under their choice of a multiplier effect, I just can't help but wonder how things behave under the additive effect. And considering the authors have well-documented their code, I wonder if it would be possible to ask for them to reconfigure their experiment to show how YEAST performs (relative to other methods) under the additive effect. On a related note, I am wondering if the multiplier effect setting benefits YEAST in some way that is related to its use of the scaled variance $V_N$ = var($S_N$)/N (or its estimate).", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n- The idea is short and clear/simple\n- YEAST has the capacity to significantly outperform current/recent methods in some settings. Meanwhile, it is not expensive in terms of parameters\n- Sequential Testing is a topic that is current \n\n**Negatives:** \n- YEAST requires a finite horizon N, whereas another method like GAVI makes no assumptions on the stopping rule. This requirement by YEAST makes it so that it's less straightforward to claim that YEAST is superior to GAVI. \n- The semi-synthetic simulations presented haven't fully convinced me that YEAST is necessarily a superior sequential test method (see questions) in terms of FDR control and Power, but I remain open, and look forward to hearing from the authors during the rebuttal period.", "was_revised": true, "final_justification": "To me, there aren't really any glaring downsides/weaknesses. The authors' efforts to provide follow-up discussions and experiments were satisfactory. \n\nTheir Table 3 results indicating much higher power over other methods is intriguing, and they were able to show a similar table under additive effects too. \n\nI personally think this paper gets a 4 at the very least, which was my original score. After the rebuttal, in which everything in my review was addressed, I am willing to give a slight bump to 5.", "rebuttal": ""}, {"review_id": "7gVT5SiyGy", "reviewer": "Reviewer_aBPe", "summary": "The authors propose a new sequential hypothesis testing method called YEAST. It generalizes a previous method for count-type metrics to any type of metric. Compared with alternative methods, YEAST offers more power by better controlling the Type-I error rate. This claim is supported by semi-synthetic experiments based on real-world data. The paper also provided theoretical guarantees for the error control and power of YEAST.", "questions": "1. How is YEAST different from alpha spending with the O’Brien–Fleming function?\n2. Does YEAST stop significantly earlier than N (both under H_0 and H_1)?", "limitations": "The authors have a substantial section discussing the limitations of their method (more detailed than a typical submission).", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 1, "strengths_and_weaknesses": "The paper is written very clearly and the exposition is very easy to follow. In particular, I really appreciated the informal introduction of the method in (6) and (7) before the formal proof in Theorem 1.\n\nThe method requires estimating V_N (the normalized variation of the metric being monitored). This is a limiting factor of the method but the authors have a nice discussion regarding the impact of the estimation error on the Type-I error based on their Theorem 1.\n\nMy main concern is the novelty of the proposed method. Is it not the case that checking $S\\_n>b^*=z\\_{1-\\alpha/2}\\sqrt{N\\hat{V}\\_N}$ is essentially an alpha spending strategy? More precisely, the false discovery rate until the $n$-th sample can be bounded as\n\n$$P(\\max \\_{i=1}^n S\\_i > b^* ) \\leq 2 P(S\\_n > b^*) \\leq 2(1-\\phi(z\\_{1-\\alpha/2}/\\sqrt{n/N}))$$\n\nwhich is exactly the O’Brien–Fleming alpha spending function. It is not clear to me what is the contribution of the paper to this strategy (I may have missed an important nuance). Because of this concern, I have rated the paper a borderline reject. However, if the authors highlight the novel parts of their approach during the rebuttal, I am open to increasing my score as I like the simplicity of the paper otherwise.\n\nRelated to my previous point, the baselines in the experiment section do not seem to include any alpha spending strategy.\n\nExperiments show that YEAST has comparable error control and power to a non-sequential test. However, the authors do not consider stopping time as a metric. Does YEAST use almost all of the N samples to be able to achieve this comparable performance? If the answer is yes, its benefit over a non-sequential test would be very limited.", "was_revised": true, "final_justification": "Authors addressed both my concerns adequately in their response.", "rebuttal": ""}, {"review_id": "jsggvIIlwu", "reviewer": "Reviewer_aybv", "summary": "This paper addresses sequential statistical tests for randomized online experiments. In these experiments, we focus on a specific metric of interest, comparing a control group and a treatment group. Our objective is to continuously track this metric and determine if, at any point, the observed difference between the groups is sufficiently large to conclude that the treatment has had an impact on the metric.\n\nFor this setting, the authors propose a novel testing procedure. This method involves tracking the running sum $S_n$​, which represents the difference in the total value of the metric of interest between the treatment and control groups. If this difference crosses a predefined constant alerting boundary, it is concluded that the treatment has an impact on the metric. The paper demonstrates that this method effectively controls the false detection rate and that, when a treatment effect exists, it can be detected with high certainty given sufficiently long monitoring. The effectiveness of this proposed method is further validated through semi-synthetic simulation experiments using real-world data.", "questions": "N/A", "limitations": "yes", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "On the positive side, the presentation of the paper is clear and well-structured, making the proposed method and its derivation accessible. Additionally, the empirical results presented in Section 4 show that YEAST achieves accurate false detection rate control and high power.\n\n\nOn the negative side, while Theorems 1 and 2 provide theoretical guarantees for a sufficiently large monitoring period N, the paper does not determine what constitutes a sufficiently large N in real-world scenarios. This lack of clear guidance regarding the practical choice of N limits the theoretical depth of the proposed method, as its performance guarantees are asymptotic. It would be valuable to explore how the choice of N impacts the test's performance.\n\n\nFurthermore, the results of the following paper would be applicable to the setting:\n> Oufkir, A., Fawzi, O., Flammarion, N. and Garivier, A., 2021. Sequential algorithms for testing closeness of distributions. Advances in Neural Information Processing Systems, 34, pp.11655-11664.\n\nMinor comment:\n- l.101: \"Section ??\"", "was_revised": true, "final_justification": "The authors have addressed many of my concerns, but I am still not sure whether the results are significant.", "rebuttal": ""}, {"review_id": "0zxInGfmpm", "reviewer": "Reviewer_n3tA", "summary": "This paper introduces a sequential test for online A/B testing. The method allows unlimited interim checks without inflating type-I error by tracking the cumulative difference in metrics between control and treatment groups against a constant boundary derived by holdout data. The key innovation is deriving this boundary by \"inverting\" bounds from generalized Levy inequalities. The authors validate their approach using semi-synthetic experiments on real-world \"Online Retail\" dataset, demonstrating superior Type-I error control and statistical power compared to existing methods like mSPRT and GAVI.", "questions": "Please improve Sections 1-2 as mentioned in the weaknesses.\n\nQ1: Robustness to model misspecification: How sensitive is YEAST to violations of its assumptions, particularly when the treatment affects event frequency or outcome variance? Can the authors provide theoretical or empirical analysis of performance degradation in such scenarios?\n\nQ2: Comparison with martingale based and group sequential tests: Why were GST and martingale based methods excluded from the empirical comparison? Those methods are widely used and provide an important benchmark, specifically methods that build on coin-betting.\n\nQ3: Ablation study: What is the impact of N on the test? It’s crucial to assess the impact of the sample size on the test.", "limitations": "The authors adequately acknowledge key limitations in Section 5, particularly the restrictive null hypothesis and potential oversensitivity to variance changes. However, the discussion could be expanded to include more detailed analysis of when the assumptions are likely to be violated in practice and mitigation strategies beyond those briefly mentioned.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\nS1. Theoretical foundation: The paper presents a rigorous approach by inverting bounds from generalized Levy inequalities (with several weaknesses, as will be detailed below). The theoretical results (Theorems 1 and 2) provide formal guarantees for asymptotic false detection rate control and asymptotic power analysis.\n\nS2. Practical simplicity with a concrete guideline for parameter tuning: YEAST requires estimating only two parameters ($N$ and $V_N$) that can be reliably obtained from pre-experiment data using standard statistical techniques.\n\nS3. Important practical insight about variance estimation: The paper explains the effect of the error in variance estimation.\n\nS4. Superior performance compared to tested benchmarks: While all methods tested control the type-I error rate, YEAST is the only method (out of the ones compared) that obtained power comparable to non-sequential t-tests.\n\n**Weaknesses**\n\nW1 (main issue). Incomplete comparison with group sequential tests and martingale based tests: While GST and martingale methods are mentioned in related work, they are not included in the empirical comparison, making it difficult to assess YEAST's performance against this important class of methods. There is a massive recent progress in using e-processes for online testing, which is somewhat ignored. For example, it is not clear at all why this approach is better than sequential classification-based testing the optimize the coin-betting strategy using ONS?\n\nPodkopaev, Aleksandr, and Aaditya Ramdas. \"Sequential predictive two-sample and independence testing.\" Advances in neural information processing systems (2023).\n\nW2. Asymptotic validity: In contrast to other existing tests, such as GAVI, the proposed test relies on asymptotic assumptions to grant its type-I error control.\n\nW3. Limited scope of null hypothesis:  Assuming that the frequency of events in each group is the same is quite restrictive. Why not relaxing this assumption?\n\nW4. Requirement for pre-specification of N and $V_N$: The method requires fixing the maximum number of events N beforehand, which limits its flexibility compared to truly adaptive methods. While the authors argue this is practical, it may not suit all experimental contexts. Regarding the estimation of $V_N$ - although affecting the validity of the test, I think that using robust estimation is a good way to deal with events that are correlated by nature (such as customers’ actions as presented in the paper).\n\nW5. Limited experimental validation: The evaluation relies on a single dataset and semi-synthetic experiments. Validation on diverse domains would strengthen the claims significantly. Also, an ablation study for different N values can be beneficial.\n\nW6. Clarity and writing quality: The writing quality can be improved, specifically:\n- It takes some time to understand what is the exact problem the paper deals with (A/B testing). It will be good to give concrete examples early on.\n\n- It might be understood from the introduction that the paper presents a non-asymptotic valid sequential test, which is not the case.\n\n- The related work section is not comprehensive: In Section 2.1 you detailed about martingale methods without explaining the essence of testing martingales. In Section 2.3 you mentioned that YEAST can be viewed as a generalisation of Miller (2015) test, even without briefly presenting its method. More recent developments in e-process are missing.\n\n- Citations don’t fit the text (either use numerical citations, or adjust the text to fit the named citations). Some references are broken, e.g., in line 101. Please go over the references and improve the citation format.", "was_revised": true, "final_justification": "This is a borderline paper. I decided to update my final rating to 4 (weak accept). The authors did address my major concern about the comparison to existing work, which I appreciate, although it remains difficult to know exactly how the comparison was implemented.\n\nI am still concerned about the assumption that the frequency of events in each group is the same (Eq. 4): it is quite a strong assumption. While the authors explained their reasoning for making this assumption and vaguely suggested possible relaxations, clear guidelines on how to address this issue in practice would be important.", "rebuttal": ""}]

Author rebuttal:

Meta-review: This paper introduces a sequential test for online A/B testing. For the theoretical analysis, the authors use Levy's inequality and the Central Limit Theorem to establish tunable FDR control and that the probability of correctly making a true detection tends to 1 asymptotically.

The initial reviews had somewhat diverging opinions and ratings. All reviewers appreciated the discussion with the authors which allowed to clarify a number of points. One reviewer still maintained their low rating (2) due to "uncertainty about significance of the results".

Overall with 4 positive reviewers (2 strongly so) and that the discussion appeared to be quite constructive and fruitful, I recommend to accept the submission.
Decision: Accept (poster)