Title: YEAST: Yet Another Sequential Test
Venue: NeurIPS 2025
Accepted: yes
Keywords: A/B testing

experiment monitoring

sequential test

early stopping
Abstract: The online evaluation of machine learning models is typically conducted through A/B experiments. Sequential statistical tests are valuable tools for analysing these experiments, as they enable researchers to stop data collection early without increasing the risk of false discoveries. However, existing sequential tests either limit the number of interim analyses or suffer from low statistical power. In this paper, we introduce a novel sequential test designed for the continuous monitoring of A/B experiments. We validate our method using semi-synthetic simulations and demonstrate that it outperforms current state-of-the-art sequential testing approaches. Our method is derived using a new technique that inverts a bound on the probability of threshold crossing, based on a classical maximal inequality.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a sequential test for online A/B testing. The method allows unlimited interim checks without inflating type-I error by tracking the cumulative difference in metrics between control and treatment groups against a constant boundary derived by holdout data. The key innovation is deriving this boundary by \"inverting\" bounds from generalized Levy inequalities. The authors validate their approach using semi-synthetic experiments on real-world \"Online Retail\" dataset, demonstrating superior Type-I error control and statistical power compared to existing methods like mSPRT and GAVI.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\nS1. Theoretical foundation: The paper presents a rigorous approach by inverting bounds from generalized Levy inequalities (with several weaknesses, as will be detailed below). The theoretical results (Theorems 1 and 2) provide formal guarantees for asymptotic false detection rate control and asymptotic power analysis.\n\nS2. Practical simplicity with a concrete guideline for parameter tuning: YEAST requires estimating only two parameters ($N$ and $V_N$) that can be reliably obtained from pre-experiment data using standard statistical techniques.\n\nS3. Important practical insight about variance estimation: The paper explains the effect of the error in variance estimation.\n\nS4. Superior performance compared to tested benchmarks: While all methods tested control the type-I error rate, YEAST is the only method (out of the ones compared) that obtained power comparable to non-sequential t-tests.\n\n**Weaknesses**\n\nW1 (main issue). Incomplete comparison with group sequential tests and martingale based tests: While GST and martingale methods are mentioned in related work, they are not included in the empirical comparison, making it difficult to assess YEAST's performance against this important class of methods. There is a massive recent progress in using e-processes for online testing, which is somewhat ignored. For example, it is not clear at all why this approach is better than sequential classification-based testing the optimize the coin-betting strategy using ONS?\n\nPodkopaev, Aleksandr, and Aaditya Ramdas. \"Sequential predictive two-sample and independence testing.\" Advances in neural information processing systems (2023).\n\nW2. Asymptotic validity: In contrast to other existing tests, such as GAVI, the proposed test relies on asymptotic assumptions to grant its type-I error control.\n\nW3. Limited scope of null hypothesis:  Assuming that the frequency of events in each group is the same is quite restrictive. Why not relaxing this assumption?\n\nW4. Requirement for pre-specification of N and $V_N$: The method requires fixing the maximum number of events N beforehand, which limits its flexibility compared to truly adaptive methods. While the authors argue this is practical, it may not suit all experimental contexts. Regarding the estimation of $V_N$ - although affecting the validity of the test, I think that using robust estimation is a good way to deal with events that are correlated by nature (such as customers’ actions as presented in the paper).\n\nW5. Limited experimental validation: The evaluation relies on a single dataset and semi-synthetic experiments. Validation on diverse domains would strengthen the claims significantly. Also, an ablation study for different N values can be beneficial.\n\nW6. Clarity and writing quality: The writing quality can be improved, specifically:\n- It takes some time to understand what is the exact problem the paper deals with (A/B testing). It will be good to give concrete examples early on.\n\n- It might be understood from the introduction that the paper presents a non-asymptotic valid sequential test, which is not the case.\n\n- The related work section is not comprehensive: In Section 2.1 you detailed about martingale methods without explaining the essence of testing martingales. In Section 2.3 you mentioned that YEAST can be viewed as a generalisation of Miller (2015) test, even without briefly presenting its method. More recent developments in e-process are missing.\n\n- Citations don’t fit the text (either use numerical citations, or adjust the text to fit the named citations). Some references are broken, e.g., in line 101. Please go over the references and improve the citation format.\n\nQUESTIONS:\nPlease improve Sections 1-2 as mentioned in the weaknesses.\n\nQ1: Robustness to model misspecification: How sensitive is YEAST to violations of its assumptions, particularly when the treatment affects event frequency or outcome variance? Can the authors provide theoretical or empirical analysis of performance degradation in such scenarios?\n\nQ2: Comparison with martingale based and group sequential tests: Why were GST and martingale based methods excluded from the empirical comparison? Those methods are widely used and provide an important benchmark, specifically methods that build on coin-betting.\n\nQ3: Ablation study: What is the impact of N on the test? It’s crucial to assess the impact of the sample size on the test.\n\nLIMITATIONS:\nThe authors adequately acknowledge key limitations in Section 5, particularly the restrictive null hypothesis and potential oversensitivity to variance changes. However, the discussion could be expanded to include more detailed analysis of when the assumptions are likely to be violated in practice and mitigation strategies beyond those briefly mentioned.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses sequential statistical tests for randomized online experiments. In these experiments, we focus on a specific metric of interest, comparing a control group and a treatment group. Our objective is to continuously track this metric and determine if, at any point, the observed difference between the groups is sufficiently large to conclude that the treatment has had an impact on the metric.\n\nFor this setting, the authors propose a novel testing procedure. This method involves tracking the running sum $S_n$​, which represents the difference in the total value of the metric of interest between the treatment and control groups. If this difference crosses a predefined constant alerting boundary, it is concluded that the treatment has an impact on the metric. The paper demonstrates that this method effectively controls the false detection rate and that, when a treatment effect exists, it can be detected with high certainty given sufficiently long monitoring. The effectiveness of this proposed method is further validated through semi-synthetic simulation experiments using real-world data.\n\nSTRENGTHS AND WEAKNESSES:\nOn the positive side, the presentation of the paper is clear and well-structured, making the proposed method and its derivation accessible. Additionally, the empirical results presented in Section 4 show that YEAST achieves accurate false detection rate control and high power.\n\n\nOn the negative side, while Theorems 1 and 2 provide theoretical guarantees for a sufficiently large monitoring period N, the paper does not determine what constitutes a sufficiently large N in real-world scenarios. This lack of clear guidance regarding the practical choice of N limits the theoretical depth of the proposed method, as its performance guarantees are asymptotic. It would be valuable to explore how the choice of N impacts the test's performance.\n\n\nFurthermore, the results of the following paper would be applicable to the setting:\n> Oufkir, A., Fawzi, O., Flammarion, N. and Garivier, A., 2021. Sequential algorithms for testing closeness of distributions. Advances in Neural Information Processing Systems, 34, pp.11655-11664.\n\nMinor comment:\n- l.101: \"Section ??\"\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a new sequential hypothesis testing method called YEAST. It generalizes a previous method for count-type metrics to any type of metric. Compared with alternative methods, YEAST offers more power by better controlling the Type-I error rate. This claim is supported by semi-synthetic experiments based on real-world data. The paper also provided theoretical guarantees for the error control and power of YEAST.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is written very clearly and the exposition is very easy to follow. In particular, I really appreciated the informal introduction of the method in (6) and (7) before the formal proof in Theorem 1.\n\nThe method requires estimating V_N (the normalized variation of the metric being monitored). This is a limiting factor of the method but the authors have a nice discussion regarding the impact of the estimation error on the Type-I error based on their Theorem 1.\n\nMy main concern is the novelty of the proposed method. Is it not the case that checking $S\\_n>b^*=z\\_{1-\\alpha/2}\\sqrt{N\\hat{V}\\_N}$ is essentially an alpha spending strategy? More precisely, the false discovery rate until the $n$-th sample can be bounded as\n\n$$P(\\max \\_{i=1}^n S\\_i > b^* ) \\leq 2 P(S\\_n > b^*) \\leq 2(1-\\phi(z\\_{1-\\alpha/2}/\\sqrt{n/N}))$$\n\nwhich is exactly the O’Brien–Fleming alpha spending function. It is not clear to me what is the contribution of the paper to this strategy (I may have missed an important nuance). Because of this concern, I have rated the paper a borderline reject. However, if the authors highlight the novel parts of their approach during the rebuttal, I am open to increasing my score as I like the simplicity of the paper otherwise.\n\nRelated to my previous point, the baselines in the experiment section do not seem to include any alpha spending strategy.\n\nExperiments show that YEAST has comparable error control and power to a non-sequential test. However, the authors do not consider stopping time as a metric. Does YEAST use almost all of the N samples to be able to achieve this comparable performance? If the answer is yes, its benefit over a non-sequential test would be very limited.\n\nQUESTIONS:\n1. How is YEAST different from alpha spending with the O’Brien–Fleming function?\n2. Does YEAST stop significantly earlier than N (both under H_0 and H_1)?\n\nLIMITATIONS:\nThe authors have a substantial section discussing the limitations of their method (more detailed than a typical submission).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 1, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nA new sequential test procedure is proposed and analyzed. It is experimentally shown to outperform current/recent methods in both false detection rate (FDR) control as well as power. In terms of theoretical analysis, the authors essentially leverage Levy's inequality and the Central Limit Theorem to establish tunable FDR control and that the probability of correctly making a true detection tends to 1 asymptotically.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n- The idea is short and clear/simple\n- YEAST has the capacity to significantly outperform current/recent methods in some settings. Meanwhile, it is not expensive in terms of parameters\n- Sequential Testing is a topic that is current \n\n**Negatives:** \n- YEAST requires a finite horizon N, whereas another method like GAVI makes no assumptions on the stopping rule. This requirement by YEAST makes it so that it's less straightforward to claim that YEAST is superior to GAVI. \n- The semi-synthetic simulations presented haven't fully convinced me that YEAST is necessarily a superior sequential test method (see questions) in terms of FDR control and Power, but I remain open, and look forward to hearing from the authors during the rebuttal period.\n\nQUESTIONS:\n**Question 1:**  Regarding Section 4.2's experiments to assess Power, I thought it was just a little curious that to synthetically generate a difference between control and treatment, a multiplier effect was chosen, rather than, say, an additive effect. I think an additive separation between control and treatment signals could be quite a reasonable phenomenon in a variety of contexts. While the results in Table 3 indicate that YEAST can outperform other methods by orders of magnitude under their choice of a multiplier effect, I just can't help but wonder how things behave under the additive effect. And considering the authors have well-documented their code, I wonder if it would be possible to ask for them to reconfigure their experiment to show how YEAST performs (relative to other methods) under the additive effect. On a related note, I am wondering if the multiplier effect setting benefits YEAST in some way that is related to its use of the scaled variance $V_N$ = var($S_N$)/N (or its estimate).\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-30"}