Anytime-Valid Inference For Multinomial Count Data
Authors: Michael Lindon
Alan Malek
Venue: NeurIPS 2022 Accept (2022)
Keywords: Anytime Valid
Sequential Testing
Experimentation
Bayesian Methods
Martingales
A/B Testing
Confidence Sequences
e-processes

Abstract: Many experiments compare count outcomes among treatment groups. Examples include the number of successful signups in conversion rate experiments or the number of errors produced by software versions in canary tests. Observations typically arrive in a sequence and practitioners wish to continuously monitor their experiments, sequentially testing hypotheses while maintaining Type I error probabilities under optional stopping and continuation. These goals are frequently complicated in practice by non-stationary time dynamics. We provide practical solutions through sequential tests of multinomial hypotheses, hypotheses about many inhomogeneous Bernoulli processes and hypotheses about many time-inhomogeneous Poisson counting processes. For estimation, we further provide confidence sequences for multinomial probability vectors, all contrasts among probabilities of inhomogeneous Bernoulli processes and all contrasts among intensities of time-inhomogeneous Poisson counting processes. Together, these provide an ``anytime-valid'' inference framework for a wide variety of experiments dealing with count outcomes, which we illustrate with several industry applications.

Anytime-Valid Inference For Multinomial Count Data
Michael Lindon
Alan Malek alan. www.mmllllgmal..mmmmmmmmmmmmmmmm com
Netflix michael.. s. s.l.ndon@gmall... com
Abstract
Many experiments compare count outcomes among treatment groups. Examples include the number of successful signups in conversion rate experiments the number of errors produced by software versions in canary tests. Observations typically arrive in sequence and practitioners wish to continuously monitor their experiments, sequentially testing hypotheses while maintaining Type er- ror probabilities under optional stopping and continuation. These goals are fre- quently complicated in practice by non-stationary time dynamics. We provide practical solutions through sequential tests of multinomial hypotheses, hypotheses about many inhomogeneous Bernoulli processes and hypotheses about many time- inhomogeneous Poisson counting processes. For estimation, we further provide confidence sequences for multinomial probability vectors, all contrasts among probabilities of inhomogeneous Bernoulli processes and all contrasts among inten- sities of time-inoomogeneous Poisson counting processes. Together, these provide "anytime-val vaaidd inference framework for wide variety of experiments dealing with count outcomes, which we illustrate with several industry applications.
Introduction
Many experiments compare count outcomes among treatment groups (arms). For example, in online conversion experiments we seek the treatment group that achieves the greatest number of conversions (sigmups, purchases etc..... software experiments, we seek the software version that produces the fewest number of errors [Lindon et al. 2022a Kuo and Yang. 1996). These experiments frequently complicated in practice by time variability over the course of the experiment caused by uncontrollable external factors. The Bernoulli success probability of conversion is often time-varying due to uncontrollable product changes and market conditions. The instantancous rate of software errors is time-varying due time-varying usage patterns of the software application among users. Fortunately, the time-variability is often common to all arms of the experiment. Therefore, the count outcomes among arms should be in some sense comparable under the null hypothesis that all arms behave equally.
In testing these hypotheses, it is desirable if a statistical test can both (i) detect large treatment effects early, and (ii) detect small treatment effects eventually. If there is a large negative effect then is important end the experiment company. However, is also important to detect small negative effects, as these can accumulate over time. Classical fixed-n tests cannot satisfy both of these objectives, as there only one opportunity to perform the test. Performing the test earlier achieves objective (i), but doesn't 't achieve objective (ii) may be underpowered. Performing the test later achieves objective (ii), but doesn't 't achieve objective (i) and treatments may affect experimental units negatively for lengthy amounts of time. For this reason we provide sequential tests that allow experiments be continuously monitored. We have included discussion of fixed-n testing vs sequential testing and corresponding literature review in the appendix.
as possible to reduce risk the experimental units and/or
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Sequential testing in experiments with Bernoulli outcomes is complicated in practice because failure outcomes are typically not directly observed. In many fixed-n approaches, Berouuli fails are often inferred from the absence of successes within an arbitrary window of time. This is not possible for real-time sequential testing when the raw data are streams of Bernoulli successes with associated timestamps. We are able to circumvent this issue by basing the sequential test solely on the counts of successes accumulated for each arm so far. This has the advantage of also removing the time-varying components in the success probabilities.
We begin by providing an saytime--vaidd inference approach for multinomial families based on a mixture martingale, which is widely applicable to experiments dealing with count outcomes. We build upon this test to provide sequential tests of iimeinnnomogeneuus Bernoulli processes and time-inhomen immooooooooess Poisson counting processes. The former is widely applicable to industry A/B experiments with Bernoulli outcomes, which we illustrate with a payments funnel experiment. The latter is widely applicable to industry A/B experiments which observe point processes in time, which we illustrate with an A/B test used to monitor the safe roll-out of new software application version. Our results are general for arbitrary number of arms.
A Sequential Multinomial Test
2.1 Theory
Our development of sequential tests for different kinds of count data begins with a sequential test for multinomial observations. The construction follows the following sequence of steps common in the literature [Shafer et al. [201] 1] Waudby-Smith and Ramdas] 2020 Howard et al., 2021]. Step 1: Define relevant test statistic. Step Show that the test statistic is test martingale
nonnegative
supermartingale under the null hypothesis. Step 3: Use martingale inequalities to bound the frequentist Type I error probability uniformly over time below a desired level u. Step 4: Invert the sequential test based on the test martingale to obtain confidence sequence with time-uniform coverage guarantee of at least u. Extensions to other kinds of count data, including Bernoulli, Binomial and Poisson counting processes, then obtained by recognizing relationships that exist to the multinomial distribution.
Our test martingale is obtained through a mixture martingale construction, integrating the multinomial likelihood with respect to a Dirichlet mixture distribution [Kaufmann and Koolen 202T]. The likelihood mixture is also the statistic used in mixture sequential probability ratio tests (mSPRT) [Wald] 1945). Bayesians will immediately recognize this statistic as a Bayes factor [Jeffreys] 1935 Kass and Raftery [995), interpreting the Dirichlet mixture distribution as prior distribution, which could be used for computing posterior probabilities. However, not wishing to confuse the reader, we emphasize that we are not developing a Bayesian procedure. We aim to develop procedure for "anytime- valid" inference, providing u-level sequential tests and 1 - u confidence sequences. We will borrow some language from Bayesian analysis, but the frequentist Type error and coverage guarantees do not depend on the particular choice of Dirichlet distribution. To avoid confusion, we encourage the reader view the Bayes factor simply as test statistic.
The test statistic can also be viewed as an e-process (Grünaald et al. 2021 Pérez-Ortiz et al. 2022 Hendriksen et al. 2021]. A review of these constructions and their relationships is discussed in Ramdas et al. [2022]. The purpose of this paper is not to study the theoretical properties of these constructions [alld and Wolfowitz/ 1948). J. but rather to develop anytime-vall valid tests for several challenging applications in modern online experimentation dealing with count outcomes.
For step 1, consider a sequence I1, Z2, T..... of independent Multinomial(1, 0) random variables with 0 E A4, the - 1 simplex. We use bold typeset to denote vectors. Under the null hypothesis, Mo. it is assumed that
3τ, 22, ... Mo iid Multinomial(1, 0o).
To construct a model over alternatives, M1, we consider the marginal distribution obtained from the following joint model
3; 22,
00 M,
iid. ~ Multinomial(1,00 0),
0 M ~ Dirichlet(ag).

---

The following expressions reduce to simple forms when a uniform distribution over the simplex is used, achieved by setting dg, = 1 for i l..... d, which may be a good default value. The Type I error and coverage guarantees are unaffected by this choice, but the expected stopping time of the sequential test is influenced. The optimal choice of this parameter depends upon the value of under the alternative, which is unknown. Stopping times under large departures from the null are reduced by using the uniform distribution. Whereas stopping times under very small departures from the null are reduced by concentrating the Dirichlet distribution about Oo. which can be achieved with the choice 00.; = k0o.; for concentration parameter E IR., In general, the average stopping time reduced by matching the Dirichlet distribution to the distribution of effects upon which these sequential tests will be performed.
ST) E Rd In addition, let |o| = E; D denote the element- wise sum of a vector at = II. ,/ to denote element-wise exponentiation of two vectors v and w, and Beta to denote the multivariate Beta function Beta(v) = (II, ((:))/TT) v.). The resulting Bayes factor comparing models M1 to Mu is given by
Let S Σラ= Fy and S, = ........
B BF1o(21:)) = Beta( + S.) 1
Beta(
which appears early Good [967]] (derivation in Appendix A.44. In Bayesian analysis, the Bayes factor multiplied by the prior odds gives the posterior odds of M1 over Mo. In this work, we take the prior odds to be unity so that the terms Bayes factor and posterior odds can be used interchangeably.
In sequential applications, it often makes sense to compute Equation 3 recursively. Let (n,) denote the posterior odds at n, then
O₂ (o)) Beta xn) 1
On
On-1
(ela((nn1) 0%
where Cx,, Cn-1 + Dn and Oo(Oo) p(M1)/p((MM)
1. When 0o,i are integer valued, Equation (4) can be expressed in terms of the posterior mean [01m|]] for each 0, as
E 20mm
(n))
II 1=1
Ono)
(5)
0o,5
Details are provided in Appendix A.5 Equation 5) yields an intuitive betting interpretation (Shafer 2021). It represents our bet against the null hypothesis O, using 0 learned from the data. In this case, our estimate of f based on the data is the posterior mean. If a uniform prior is used, our initial bet on each outcome žı,; is simply 1/d, and our bet on each outcome In is (1 - 3/-1)/(n= + d). The dependence of O,, (0)) on the observed data i: is implicit in this notation, yet the null value 0g being tested is made explicit to aid the discussion of confidence sequences in Theorem 2.4
Step 2 in our construction is to demonstrate that O,, (0)) is nonnegative supermartingale under the null hypothesis Mo.
Theorem 2.1. Let F2....... be a sequence of independent Multinomial(1, 0) random variables with the filtration Fa 0 Ø2, .2....... ,,,,,,. -1n and consider the sequence of posterior odds O, (Op)
defined in Equation with (O,) 1. Then
EM, (0,) (m0)
The proof is found in Appendix A.6 Theorem 2.1 states that (.,) is a nonnegative martingale under the null hypothesis with respect to the canonical filtration.
Step 3 use the posterior odds to construct test martingale.
Theorem 2.2. Let 11, ...... be sequence of independent Multinomial(1, 0) random variables and consider the sequence of posterior odds On (0o) defined in Equation with Oo(Oo) = 1. Then
Po-a, (3n N (n)) 1/0) S
for all u E [0, 1].
3

---

The proof is provided in Appendix A.7 The time- uine-uniform bound presented in Theorem 2.2 controls the deviations of stochastic process for all simultaneously and is essential for proving the correctness of sequential tests and verifying the optional stopping and optional continuation properties. provides a valid stopping rule: reject the null at time r = (n E N (n(()) 2 1/u)- Simply stated, a practitioner who rejects the null hypothesis as soon as the posterior odds become larger than 1/u faces a frequentist Type error with probability at most u. Shafer et al. [201I). Johari et al.
[2021] bring this idea back to more familiar territory by constructing sequential p-value by tracking the running supremum of the posterior odds and taking its inverse, or equivalently
po 1 and
Pr. = min (Pn-1, 1))))
It follows from this definition and equation 7 that
Po=o. (En E e::: N: 5 uu) u.
(8)
which is an easily digestible generalization of a fixed-n p-value to sequential settings. Instead of holding only at some pre-specified N, this guarantee holds for all E N. This construction shown in Figure continuous monitoring relative to a x² test is shown in Appendix A.9 A.9
simulation empirically demonstrating the control of false positives under
Before completing Step 4, it is useful show that this sequential test is not trivial. For this test to have utiliiy it must possess the ability to control not only Type errors, as in theorem 2.2 but also Type errors. This provided by the following theorem.
Theorem 2.3. Let 11, F2... be a sequence of independent Multinomial(1 1. 0) random variables and consider the sequence of posterior odds (,,( defined in Equation with Oo(O,) = 1. If # 0o then
log (0,) + (KI(0))00))
O.S.,
where DKL (0)()) is the Kallback-Leible divergence of the true multinomial distribution with true parameter from the multinomial distribution under the null hypothesis with null parameter Og.
The proof is given in Appendix A.8 Theorem 2.3 states that if the null hypothesis is not true, with
* 0g. then the Bayes factor will diverge to infinity and exceed the 1/u threshold in Theorem 2.2 (a.s.), or equivalently that the sequential p-value converges zero and falls below the threshold (a.s.). In other words, this test is guaranteed to reject the null almost surely if the null is incorrect, which is considered be asymptotically power I by Robbins (1970]. This result follows simply from the posterior consistency of Bayes factors. We now state step 4 of the construction.
Theorem 2.4.
Let I D2.... be a sequence of independent Multinomial(1, 9) random variables
and consider the sequence of posterior odds for testing the null hypothesis = Os. (nn),, defined in Equation with Oo(O 1. Let Cn(u) {0 A (,) 1/u} denote the set of parameter vectors that would not be rejected by the test at the u level, then
Po (0 E Cu(()) for all n E EN 2 1 1-u
(10)
for all u E [0, 1].
A simple corollary of theorem 2.4 is that Po (0 E N Cn))) 2 1 This result provides a confidence statement for sequentially estimating the true parameter vector O as the experiment progresses. The confidence set Cn (u) for 0 is convex subset of Ad, with convexity following from the concavity of the multinomial log- log-likelihood. Confidence intervals on the individual elements can be obtained by projecting C, (u) onto the coordinate axes in the following manner.
Corollary 2.5. For Cn (u) as in Theorem 2.4 let
(w) sup 0 Cn("),, fu,(u) = inf (u)),,
then
Po
Vi: 0, E n (w),
2 21-w u.
(II)
==1

---

(u) can be computed by solving the following convex optimization program
max
c+ log S Σ S log 0;
s.t.
(12)
Ed= 00=11 =1
where log (ela + Sw) - log (eoa The constraints in the optimization program simply define (u). Similarly, Jn, (u) is obtained by minimizing 0g over this set.
Lastly, the confidence set in Theorem 2.4 is equal to the Bayesian support set (aggemakerss et al.) 2022].
Corollary 2.6. The Bayesian support set, defined as 10 E A p Mi) 2 up(0)Mi)) has time-uniform, frequentist coverage of at least
The proof is a simple application of the Savage-Dickey density ratio (x) = p (M0MM)/p(((e Ti:n: M1) (Dickey 1971].
2.2 Application: Sample Ratio Mismatch Tests
Comparing the counts of experimental units assigned to each arm multi-arm experiment can often detect bias and errors in the experiment. Most online controlled experiments follow simply randomized designs, whereby a new unit is randomly assigned to one of arms according to a vector of pre-specified probabilities O. The assignment outcome for new unit is independent of other units (individualistic), unit-level covariates, potential outcomes (unconfounded) and can therefore be summarized an independent Multinomial(1. 0) random variable. Under these assumptions, the assignment mechanism can be considered ignorable when performing inference on causal estimands such the average treatment effect [Imbens and Rubin 2015].
Although simple in theory, the systems that perform assignments quickly grow in complexity as the number of concurrent experiments increases [Tang et al. 2010). This increased complexity increases the risk of introducing bugs that cause departures from the intended assignment mechanism, breaking the assumption of ignorability and rendering causal estimates invalid. /hao al. [2016) provide an account of an incorrect hashing algorithm introducing bias into the assignment mechanism.
After assignment and measurement, data passes through processing and cleansing pipelines before analysis. If incorrect cleansing logic applied, there risk that specific observations may be selectively removed, introducing "missing not at random" missing data mechanism, rendering causal estimates invalid [Rubin] 1976). Fabijan et al. [2099] describes an experiment which units from the treatment arm were unintentionally removed, with the probability of their removal depending on their observed outcomes.
An arm with a surprisingly low or high number of units is usually symptomatic of an implementation error in the experiment and is colloquially referred to as sample ratio mismatch (SRM) (Fabijan et al. 2019] These errors can be caught by comparing the counts of experimental units in each arm against the intended assignment probabilities. is now considered good practice to validate the experiment setup by performing x test, comparing the observed counts against the expected counts under the intended assignment mechanism (Chen al., 20188). However, the x" test an example of fixed-n test, providing statistical guarantees when performed once. Due to this limitation, it is typically performed after data collection and before analysis. To reveal a bug that renders an expensive experiment invalid only after the experiment is finished would be less than ideal. Idcally, SRMs are detected as early as possible so that the implementation error can be corrected before more units enter the experiment. This necessitates sequential testing. We provide sequential multinomial test for testing point null based on the counts of each outcome.
Figure D shows the application of the sequential multinomial test to representative experiment exhibiting an SRM. this example the intended assignment probabilities are On = (0.1, 0.4, 0.5). These propensity scores are assumed to be known by the analyst and will subsequently be used to perform inference about average treatment effects. Due to a bug, either in randomization or data processing, some units are preferentially assigned certain arms based their covariates, or data is
5

---

(a) Joint Confidence Set n 144
(b) Simultaneous Marginal Confidence Sequences
AAY
(c) Sequential p-value and Bayes Factor
Figure (a) The confidence set (0.05)) n 144 as defined Theorem 2.4 The true = (0.1, 0.3, 0.6) is marked with a black cross, and the null hypothesis O, = (0.1, 0.4, 0.5) is denoted with red cross. (b) Simultancous 0.95 confidence sequences that cover the individual elements of 0 (0.1. 0.3. 0.6) obtained from Corollary 2.5 and computed via the optimization program in Equation (12). (c) (Left axis) Sequential p-value (blue) defined Equation (8). Critical
value u = 0.05 (dashed-black). O = (0.1, 0.3, 0.6), 0n = (0.1, 0.4, 0.5). Pr. < 0.05 for all n 2 144.
(Right axis) The posterior odds defined in Equation (3) (red), with the running supremum (green).
being lost random, resulting observations arriving with probabilities 0 (0.1, 0.6). The sequential p-value is less than 0.05 for all n 2 144. This is the smallest for which 6, & (0.05)..
3
Inhomogeneous Bernoulli Processes
Wald 1947] described an experiment to test the null hypothesis that two guns are equally precise. Two guns fired at target simultancously for many attempts, and their success is recorded. Each attempt is modelled as a Bernoulli trial and, under the null hypothesis, the probability of each gun hitting the target on the same attempt equal. However, the success probability varies across attempts due to gusty wind conditions. As the guns are shot simultaneously, the wind condition affects each gun equally. the null is rejected, choosing the gun that obtained the greatest number of successes seems reasonable.
3.1 Theory
Suppose new experimental unit randomly assigned to one experiment treatment groups (arms) at time 1, according to assignment probabiliities E A and Bernoulli outcome is observed. The Bernoulli probability for arm at time / is parameterized by (t) e*(t), t so that the time-varying effect is multiplicative and common all arms. The improvement of arm j over arm at any time is then Pj (t)/p:(t) (t) = cxp(6; - &,), and the difference on the log-scale is simply 8, 8, Suppose Bernoulli failures are ignored and arms are compared only through their counts Bermoulli successes. The (conditional) probability that the next Bernoulli success comes from arm i is
(13)
Σ 2
which independent of the vare-narying effect. The arm from which the next Bernoulli success arrives is, therefore, Multinomial(1, 0) random variable, and the counts of Bernoulli successes for

---

each arm are independent of the time-varying nuisance parameter μtt). Framing the problem this way allows the sequential multinomial test to perform inference on 6. Simple hypotheses about ò can therefore be translated into testing simple hypotheses about O. Equality among success probabilities can be tested by simply testing the null multinomial hypothesis 0g p. The individual components 6, are not identifiabl,, adding a constant to each element results in the same 0, yet contrasts of the form >; a.d; for E; ag = 0 are identifiable.
Let opp IRd + A denote a generalization of the softmax function to include p, with op(6), equal to the right hand side of Equation
(3)..
Theorem 3.1. Let (w) = o (C (u)), then
P E (nr a N
(14)
The proof is direct consequence of Theorem 2.4 The following corollary yields simultaneous confidence sequences for all contrasts.
Corollary 3.2. Let (u) = o = (Cn(u)) and A {a E Rd E, a = =0} denote the set of all
d-dimensional contrasts. For all a E A define
n,a (v. = supt Σ
a;d; 8 kn(4)} and
(u) = inf Σ
a 5 (u))
Then
00
Pe
Va A" Ea i
The upper bound +t (u) is the solution to the convex optimization
Σa.o a;::
max
(15)
s.t.
Σ
ST
+ log log Σ Epee3s
Σn
where c = log Beta(ar + S,)) - log Beta (x0) + logu.
u. Convexity follows from the log-sum- exponential function. The lower bound 17, (u) is the solution to the corresponding minimization problem. This visualized Figure
A hypothesis can be rejected at the u level as soon as the set that it defines fails to intersect with the confidence set (u). Note that (,, C (wz) for u1 > U2. To obtain sequential p-value, we seek the largest such that the null is not rejected. That is, we seek the smallest set K,,u (u) such that there is a non-empty intersection with the subset of IR defined by the hypothesis. This too can be achieved by convex optimization program. One can simply maximize over the feasible defined by the intersection of the K, (u) and the set defined by the hypothesis. Suppase one wishes to test the hypothesis &n 2 ó1 and dg 2 82. The sequential p-value at time the inverse of the solution to the following convex program
min a
s.t. c S llo(gg) +
5m Σ
6; + log - -log Σ Σpses,
(16)
8
8n 251
8n 252
where e = log Beta + S.)) log Beta(o))) A simulation study presenting sequential p-values and confidence sequences on is given in Appendix A.11
7

---

330
440
310
(a)
(b)
Figure 2: (a) Assuming P1 and Po are stationary and estimated using multinomial confidence se-
quences from from Corollary 2.5, (b) Assuming P1 (t) and po(t) are dynamic but with P1 (t)/po(( (t) exp(o1 - 6n) and estimating this quotient using the confidence sequences from equation Corollary 3.2 (Left axis) Confidence sequences visualized with shaded regions and MLE estimates are visualized with solid lines. (Right axis) sequential p-value.
3.2 Application: Conversion Rate Optimization
Many modern experiments are designed to increase conversion rate and drive additional revenue. The term conversion used to describe Bernoulli trial such as user signup or purchase. The typical setup is that there are d different experiences (such as different versions of a signup page), and each experience i assumed to have some conversion probability pi- When new user visits the experience, they either convert or they do not, and successful or unsuccessful conversion event logged. One might think that it would be easy estimate p; sequentially using the multinomial confidence sequences of section and the counts of successful and unsuccessful conversions of arm i. However, one soon runs into problems, as illustrated below.
The following case study comes from signup funnel experiment at Netflix. It was hypothesized that adding additional methods of payments might increase the overall number of conversions. Alternatively, it could also be plausible that additional payment methods may simply cannibalize conversions from other payment methods, not increasing the overall number of conversions at all. An A/B test was created to test these hypotheses. The control group received the standard set of payment methods, while the treatment group were offered the standard set plus additional payment methods. To begin, let's 's first incorrectly assume that the success probabilities are constant. Figure 2 shows the application the multinomial confidence sequences from Corollary [.5 to separately estimate the conversion probabilities for arms 0 and 1. Note that the running intersection of anytime- valid confidence intervals becomes the empty set, and the MLE exits
confidence sequence. An empty intersection would be event (with probability less than a) if the assumption of constant probabilities in Section 2 were true. Instead, it indicates that the conversion probabilities not constant but time-varying invalidating commonly made assumption in conversion rate experimentation.
Many online conversion experiments share similarities with Wald's gun-shooting example. The success probability of conversion is often time-varying due to external factors such as the day of the week, recent product launches, new promotions. These external factors likely affect all arms of the experiment equally, so reasonable to expect the time variation be common to all. Moreover, it is often the case in these experiments that an unsuccessful signup (a 0 or "fail" Bernoulli outcome) is not directly observed. When a successful signup occurs, an event is typically logged, but if no signup occurs, then unsuccessful signup must be inferred from the absence of successful signup (typically through an ad-hoc definition such as "an experimental unit failed to convert within T' units of time since allocation"). This definition of conversion arbitrary choice of time window, and can present challenges to a sequential analysis if one must wait T units of time before the first Bernoulli outcome is realized.
make the analysis sensitive the
Instead, we propose using the assumptions of section 3 which only requires the counts of successful signups. The assumption of constant conversion probabilities is relaxed to constant multiplicative difference berween them, which is typically more realistic practice. The confidence sequence on the multiplicative constant is shown in 2 (b). A winning arm can be declared in approximately one
8

---

week instead of nine, toward which all subsequent visitors can be directed, dramatically increasing signups relative the fixed-n experiment.
4 Inhomogeneous Poisson Counting Processes
4.1 Theory
counting process is a stochastic process {N(1) (11):.1 :120) satisfying N(O) 0, N(1) E No and (s)) < (1) for s < I. The inhomogeneous Poisson counting process is defined by an intensity function A: RR - R+ that locally integrable, fB A(1)dl S op for all bounded Borel measurable sets E R. defining a measure A(B) f X(t)dt [Kingman 1992 For any collection of disjoint Borel measurable sets B1, B2.... the inhomogeneous Poisson counting process has the property that (B,) are independent Poisson A B,)) random variables. The inhomogeneous Poisson counting process can be defined in terms of inhomogeneous Poisson point process on the real line by simply counting the number of points in a set. For our applications, these points correspond to times of events. At any time / the probability density of the time-difference s to the next point is given by g(s) = X(L + 8) exp(- fg X(( s)ds). The independent increments property implies a memoryless property of the process: the counts in the next time interval waiting time until the next point independent of the process history. This is necessary property to establish the following theorem.
Theorem 4.1. Consider d inhomogeneous Poisson point processes indexed by i E (1,....d) d] where process has intensity functions (²(t) = p.e Xt).. Let each point produced by process be marked with the corresponding process index At any time t, such as immediately after the previous point, the probability that the next point has mark is given by
P:e
(17)
Σj=1 Pze0:
This gives the probability that the next point in time is from process i. The proof is given in the Appendix A.10 Theorem [4.] states that the sequence of marks can be considered a sequence of Multinomial(T. 9) random variables, allowing the sequential multinomial test to perform inference on For example, a sequential test of equality among d ume-inhhoooeeneus Poisson point processes (tx(t) (t) for all pairs i and j) can be obtained from the sequential multinomial test of the hypothesis 0, = (1/d...., 1/ /d). Once again, the total counts in each arm is a statistic that is independent of the time-narying nuisance parameter A(t). A simulation study presenting sequential p-vulues and confidence sequences on ô is given in Appendix AA.2
4.2 Application: Software Canary Testing
When continuously deploving new software to users, engineers often adopt the practice of canary testing [Lindon et al. 2022al Schermann et al. 2018 A canary test is a controlled experiment in which users randomly assigned to the current software newer release candidate. The experi- menter's goal is to study the performance of the release candidate in a production environment before releasing it globally, essentially acting as a quality control gate before full deployment. If the release candidate performs significantly worse, it is blocked, and developers must resolve the offending issues. This strategy helps prevent bugs from reaching all users. However, performance regressions are still experienced by the users in the experiment. Reducing the risk for these users motivates the experiment's stoppage soon a performance regression is detected. Optional stopping requires performance to be measured in real-time and requires sequential testing methodology.
Performance regressions to errors, failures, or any occurrence of interest sent to a central logging service by each instance of the software. The data naturally forms marked I-dimensional point process in time, recording the timestamp and type of event. The instantaneous rate expected to be time-varying due to varying traffic and usage patterns.
measured in terms of the rates of events. These events could correspond
The following example is taken from a canary test Netflix. This acts as quality control gate in the software delivery process when releasing new versions of the client application. A regression that would have affected approximately 60% of all devices was detected with this methodology less than one second. Successful play starts (SPS) are carefully monitored in this experiment between

---

(a)
Figure 3: (a) Ruglplot shows the timestamps of SPS events being received, while solid lines show the counting processes for arms 0 and 1. (b) (Left axis) Shaded region shows the confidence sequence for the quotient of inhomogeneous Poisson process intensity functions At obtained from Corollary 3.2 Blue solid line shows the MLE (t)/Aoot) = eh - (Right axis) Black solid line shows sequential p-value.
the cxisting software and the release candidate. An SPS event is sent by the streaming application to the central logging system whenever requested title successfully begins playback. If significantly fewer SPS events being received from the treatment group running the release candidate, then indicates there an issue with the new software preventing some streams from starting. Figure B shows that the confidence sequence X₁ (t)/Ao(? falls below 1.0 in less than second, indicating that the instantaneous rate of SPS events for arm I is less than the instantaneous rate of SPS events for O. In this case, the canary test aborted and the offending bug was identified, preventing serious regression from being released to all users.
5 Conclusion
The contributions of this paper provide an "anytime-valid" approach to inference in several important applications dealing with count data. The anytime-valid guarantees permit experiments to be contin- uously monitored and enables optional stopping. which can greatly reduce both the time required for experiments to complete and the potential harm to units in the experiment. We first introduced sequential test for Multinomial hypotheses using mixture martingale construction, which has already proven to be an effective solution for rapidly detecting sample ratio mismatches in online controlled experiments. We then used this result develop a sequential test of equality and contrasts in inhomogeneous Bernoulli processes, which has practically demonstrated dramatic speedups in decision-makiin making in conversion experiments. Contrary to many widely used models, our approach does not assume that the conversion probabilities are constant, which is often violated in the real world. Moreover, only successful conversions, such signups, are observed practice, This is different from Bernoulli outcome models in which it is assumed that both successes and failures are observed. Our proposed sequential test enjoys the added convenience only requiring successful Bernoulli outcomes to be observed. Lastly, we used the sequential Multinomial test to develop sequential test for equality and contrasts in time-inhomogeneous Poisson counting processes. These play an important role in the monitoring of systems, such as data pipelines and software usage.
The confidence sequences provided in this paper allow inference be made any time and data- dependent stopping rules for ending an experiment. This in of itself can dramatically speed up the time reach conclusions with valid statistical guarantees. An obvious extension to this work is to combine these confidence sequences with a strategy for adapting the assignment probabilities with the goal of assigning fewer units suboptimal arms. The confidence sequences presented here can be used to construct an adaptive algorithm that identifies the best arm with tunably high probability through the least upper confidence bound (LUCB) algorithm, as has been done successfully in Howard and Ramdas 2022
References
Michael Lindon, Chris Sanden, and Vaché Shirikian. Rapid regression detection in software de- ployments through sequential testing. In Proceedings of the 28th ACM SIGKDD Conference
10

---

Knowledge Discovery and Data Mining, KDD '22, page 3336-3346, New York, NY, USA, 2022a.
Association for Computing Machinery. ISBN 978 1450393858 doi: 011144//3534688.5399999 org/10.
URL htttps://doi
Lynn Kuo and Tae Young Yang. Bayesian computation for nonhomogeneous poisson processes in software reliability. Jourmal of the American Statisticall Association, 91(434):763-773, 1996. ISSN
01621459. URL /tLt://:://wrr jstor arg/stablee/221677
Glenn Shafer, Alexander Shen, Nikolai Vereshchagin, and Vladimir Vovk. Test martingales, bayes factors and -values. Starist. Sci., 26(1):84-100 101, 02 2011. doi: 10.1214/0-STS3377 URL
https::/doi. org/ 10rg/10.10 1214/10-STS344
lan Waudby-Sminn and Aaditya Ramdas. Confidence sequences for sampling without replace- ment. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Ad- vances in Neural Information Processing Systems, volume 33, pages 20204-20214. Cur- Associates, Inc., 2020. URL https: httttpp:/pproceediggs. @96c7/e@ff33001ecc 051.:9000.6675564444444455550-00 wtw.eeeeennnnn...uii.s...... hecuripp.cc/ppprr2220//111..
pdf
Steven R. Howard, Aaditya Ramdas, Jon McAuliffe, and Jasjeet Sekhon. Time-uniform, nonparamet- ric, nonasymptotic confidence sequences. The Amnals of Statistics, 49(2):1055 1080, 2021. doi:
10.1214/0-AOSS199) URL. /tattps://doi.rrre 1214:100..1111111111 20-14990
Emilie Kaufmann and Wouter M. Koolen. Mixture martingales revisited with applications to sequential tests and confidence intervals, Journal of Machine Learning Research, 22(246):1
2021. URL http://::/ http://wwww..rrrrpppps////////// 798. html
A. Wald. Sequential tests of statistical hypotheses. Ann. Math. Statist,, 16(2):117-166, 06 1945. doi: 10.1214/aom//11777111 URL https:////// 10. 1214 aww////m///////////// /
Harold Jeffreys. Some tests of significance, treated by the theory of probability. Mathe-
matical Proceedings of the Cambridge Philosophical Sociery, 31(2):203-225 1935. doi:
10.1017/5030500400003333
Robert E. Kass and Adrian E. Raftery. Bayes factors. Journal of the American Statistical Association, 90(430):773-755, 1995. ISSN 01621459. URL htttp://www/ jstto/www..trrrroooo.o/.oo.oo.rs org/ htt///ttbbeee2291/////
Peter Grünwald, Rianne de Heide, and Wouter Koolen. Safe testing, 2021.
Muriel Felipe Pérez-Ortiz, Tyron Lardy, Rianne de Heide, and Peter Grünwald. E-statistics, group invariance and anytime valid testing, 2022. httttttts///aiivi wttp://///aaa......./m 07610
Allard Hendriksen, Rianne de Heide, and Peter Grünwald. Optional Stopping with Bayes Factors: A Categorization and Extension Folklore Results, with an Application to Invariant Situations. Bayesian Analysis, 16(3):961 - 989, 2021. doi: 1..1214/20-BA1244 https:///ddo org/ 10. 20044/20-B11230
Aaditya Ramdas, Peter Grünwald, Vladimir Vovk, and Glenn Shafer. Game-theoretic statisties and safe anytime-valid inference, 2022. htttst:////: www.aaiwwi.rr..ooeeeeeaeaae absp///s/221..... 01948
A. Wald and J. Wolfowitz. Optimum character of the sequential probability ratio test. The Annals of Mathematical Statistics, 19(3):326-33)) 1948. ISSN 00034851, URL htttp://www. jstor org/
Stable/2235633
J.
J. Good. A bayesian significance test for multinomial distributions. Journal of the Royal Statistical Society. Series B (Methodological), 29(3):399 431, 1967. ISSN 00359246. URL http: : www
Glenn Shafer. Testing by betting: A strategy for statistical and scientific communication. Journal of the Royal Statistical Society: Series A (Statistics in Society), 184(2)4007-331 2021. doi: https:
1oooioorg11.1111 I/rssa. 12647. URL http::ttp//ssstt www.eesooonnnneiibbrarra wiley..i................o www.doo/mas.aa//eoooooooooooooat
1111/rssa. 12647
Ramesh Johari, Pete Koomen, Leonid Pekelis, and David Walsh. Always valid inference: Continuous monitoring of a/b tests. Operations Research, 2021. doi: 10.1 00.1287/0re.2221.22333 URL https
Afoiiorrg/00 1 /ppe. 0122.2135

---

Herbert Robbins. Statistical methods related to the law of the iterated logarithm. The Annals of Mathematical Statistics, 41(5):137711499, 1970. ISSN 00034851. URL htLphtt::0/ h/www/ jstor
Drg/stable/222300 2239848
Eric-Jan Wagenmakers, Quentin Gronau, Fabian Dablander, and Alexander Etz. The support inter-
val. Erkennimis, 87(2):589 601, Apr 2022. ISSN 1572-8420. doi: 10.1007//110670--199-000000
URL Sattps://don org 107.:000000000000--00-00
James M. Dickey. The Weighted Likelihood Ratio, Linear Hypotheses on Normal Location Parameters.
The Annals of Mathematical Statistics, 42(1):204 223, 1971. doi: 10. (0.1214//mm///7776655777
URL https: //doi. org/10. /
aoms/ 11776935077
Guido W. Imbens and Donald B. Rubin. Causal Inference for Statistics, Social, and Biomedical
Sciences: An Introduction. Cambridge University Press, 2015. doi: 10.1017/CB09781 330057751.
3902577
Diane Tang, Ashish Agarwal, Deirdre O'Brien, and Mike Meyer. Overlapping experiment infras- tructure: More, better, faster experimentation. In Proceedings 16th Conference on Knowledge Discovery and Data Mining, pages 17-26, Washington, DC, 2010.
Zhenyu Zhao, Miao Chen, Don Matheson, and Maria Stone. Online experimentation diagnosis and troubleshooting beyond aa validation. 20/6 IEEE International Conference on Data Science and Advanced Analytics (DSAA), pages 498-507, 2016. doi: 10. 10.1109//SAAA2216..61
Donald B. Rubin. Inference and missing data. Biometrika, 63 12 1976. ISSN 0006-344 -3443444. doi: 10. 10.1093/biomet/33..5881 URL http:::/dti.oooooooooooo http:///o....r.......01/////e/// 10w.:/1110100///00///ee.e... 3.51
Aleksander Fabijan, Jayant Gupchup, Somit Gupta, Jeff Omhover, Wen Qin, Lukas Vermeer, and Pavel Dmitriev. Diagnosing sample ratio mismatch in online controlled experiments: A taxonomy and rules of thumb for practitioners. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD '19, page 2156-2164, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781 9781450362016. 3330722. URL ttttps:///ol: htt////o.or///////////oo 10. 1145/3292500 3330722
doi: 10. 10.1145/3992000
Nanyu Chen, Min Liu, and Ya Xu. Automatic detection and diagnosis biased online experiments, 2018.
A. Wald. Sequential analysis. J. Wiley & sons, Incorporated, 1947. URL https::/books.goont
J.F.C. Kingman. Poisson Processes. Oxford Studies in Probability. Clarendon Press, 1992. ISBN 9780191591242. https::/: http:///oooos.oolleee google wwtgool...oooooo/iiiiooooooooo.e
Gerald Schermann, Jürgen Cito, Philipp Leitner, Uwe Zdun, and Harald C. Gall. We're doing
live: A multi- empirical study on continuous experimentation. Information and Software
Technology, 99:41-57, 2018 ISSN 0950-5849, doi: https://do..org//00001//// 1016: ittp:///.iinssff2118.0222222
URL https:tpppo www. sci sciencedirect.. www wwt.eiic....ccccccccc../ete/o//a
Steven R. Howard and Aaditya Ramdas. Sequential estimation of quantiles with applications to A/B testing and best-arm identification. Bernoulli, 1704):1704- - 1728, 2022. doi: 10.31500211BBBJ3338
URL https:///oi...... org.10........ 300.0.110/2111 BEJ1388
P. Armitage. Interim Analyses in Clinical Trials. Multiple Comparisons. Selection and Applications in Biometry. CRC Press, 1993. ISBN 9780824788957.
Jerzy Neyman, Egon Sharpe Pearson, and Karl Pearson. Ix. on the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society of London, Series A, Containing Papers of a Mathematical or Physical Character, 2316694-700)2289 337, 1933. doi: 10.1098//sta.11 1933.0009. URI httpphttt:://ee hutpppps//oogglo.cc.eyoaalaaaaap
1098/
rsta. 1933. 0009
Herbert Robbins. Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society, 58(5):527 -335, 1952. doi: bams/ 183551777. URL https: http::////oooooooo org/
12

---

G. Casella and R.L. Berger. Statistical Inference. Duxbury advanced series in statistics and decision sciences. Thomson Learning, 2002. ISBN 9780534243128. / com/
books?id=Oxx vAAAAMAAJ
F. J. Anscombe. Fixed-sample-size analysis of sequential observations. Biometrics, 1011)0 1954. ISSN 0006341X, 15410420. URL httt:://www..ssort wtt.att...rr//taaaa//0116665555
Ramesh Johari, Pete Koomen, Leonid Pekelis, and David Walsh. Peeking at a/b tests: Why it matters, and what to do about it. In Proceedings of the 23rd ACM SIGKDD International Conference Knowledge Discovery and Data Mining, KDD '17, page 1517-1525, New York, NY, USA, 2017. Association for Computing Machinery. ISBN 9781450348874. doi: 10.11457399798333099992
URL htttp:://dol.l org/10. 10. 1145
P. Armitage, C. K. McPherson, and B. C. Rowe. Repeated significance tests on accumulating data. Journal of the Royal Statistical Society. Series A (General), 132(2):235225 1969. ISSN 00359238.
URL http:/ //wp: jstor org wtgg/stabbee23437811
Joseph B. Kadane, Mark Schervish, and Teddy Seidenfeld. Reasoning to a foregone conclusion.
Journal of the American Statistical Association, 91(435):1228-1235, 1996. ISSN 01621459, URL http:/ /ttp:://www..o www. jstor org/
Jean Ville. Étude critique de la notion de collectif. 1939. URL.ttpp///uddh..oo///b////
James O. Berger, Lawrence D. Brown, and Robert L. Wolpert. A unified conditional frequentist and bayesian test for fixed and sequential simple hypothesis testing. Ann. Statist, 22( 178--1887 12 1994. doi: 10. 121121444 10111///////1111// waeaa/os//1163330 176325757. UtL. 1tt/:/1..1111//////1763333333
J. O. Berger, B. Boukai, and Y. Wang. Unified frequentist and bayesian testing of precise Statist. Sci., 12(3):133-166) 160, 09 1997. doi: //s////////33/044 URL http:t:t:////...rr//// 1214/ss/103003790
hyptthee hypothesis.
James Berger, Benzion Boukai, and Yinping Wang. Simultaneous bayesian frequentitt sequential testing of nested hypotheses. Biometrika, 86(1):79 92, 1999. ISSN 00063444. URL http:
//ww..jstor org/stabbe//673553
Jerome Cornfield. A bayesian test of some classical hypotheses, with applications to sequential clinical trials. Journal of the American Statistical Association, 61(315):577-594, 1966. ISSN 01621459. URL/http: h jstor. org/ wwrg/stablee/222777
D. A. Darling and Herbert Robbins. Confidence sequences for mean, variance, and median. Pro- ceedings of the National Academy of Sciences, 58(1):66-68 1967. ISSN 0027-8424. doi:
(0.10773/pna.58.1.66 URL httthhttt//www/t wwtp::/wwwwwwppw.waaaaaaaaaaaaaa pnas org/ wttpg//cooeeett/e./
Ian Waudby- Smith, David Arbour, Ritwik Sinha, Edward H. Kennedy, and Aaditya Ramdas. Doubly robust confidence sequences for sequential causal inference, 2021.
Michael Lindon, Dae Woong Ham, Martin Tingley, and lavor Bojinov. Anytime-valid f-tests for faster sequential experimentation through covariate adjustment, 2022b. URL https://axxxi org/abs/ 2atbba//221.... 08589
Dae Woong Ham, lavor Bojinov, Michael Lindon, and Martin Tingley. Design-based confidence sequences for anytime-valid causal inference, 2022. URL hatp::://axxivoor//bbb/ 08639
C. Jennison and B.W. Turnbull. Group Sequential Methods with Applications Clinical Trials. Chapman & Hal//CRC Interdisciplinary Statistics. CRC Press, 1999. ISBN 978 9781584888581. URL
https://books. www.lkss..ooge...ooooo www.//oooooodaaaeeaooeo bookss?dd
Ivair R Silva and Martin Kulldorff. Continuous versus group sequential analysis for post-market drug and vaccine safety surveillance. Biometrics, 71(3):851-853 2015.
Steven R. Howard, Aaditya Ramdas, Jon McAuliffe, and Jasjeet Sekhon. Time-uniform Chernoff bounds via nonnegative supermartingalls. Probability Surveys, 17(none):257 -317, 2020. doi: 10.1214/18-PSS3 BRLhhttps: ://o::///oi. org 11tp://8...11111PP23222
13

---

P. A. W. Lewis and G. S. Shedler. Simulation of nonhomogeneous poisson processes with degree- two exponential polynomial rate function. Operations Research, 27(5):1026 1040, 1979. doi: 10. 10.1287/opre.27.55 27.5.110......2 URIhttps: :/tt::://dooooo http:::://oi..o//////// org/10. 1287/opre 27.5.1026
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] Our work motivated by three important sequential applications in online experimentation. Testing proper randomization via sample ratio mismatch testing. Testing hypotheses among time-varying probabilities conversion rate optimization and testing hypotheses among time-varying poisson process intensities software canary experiments. We have provided viable sequential testing solutions to all of three of these applications via sequential multinomial tests, sequential tests of inhomogeneous Bernoulli processes, and sequential tests of mem-innooooeneouus Poisson counting processes.
(b) Did you describe the limitations of your work? [Yes] We have explained in the introductory section that we do not study the theoretical properties such as optimality our proposed tests. Our are specific instances of generic construction known as the method of mixture martingales and mixture sequential probability ratio tests. The optimality properties of these tests are well established and we have linked the relevant literature. Our work focuses on providing specific solutions to some important applications using these constructions.
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] All proofs include the assumptions, and we have even presented examples of what can go wrong when these assumptions do not hold see the application of multinomial confidence sequences when the probabilities are time varying in section
(b) Did you include complete proofs of all theoretical results? [Yes] All proofs are in the appendix
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material as URL)? [Yes] We have several simulations in the appendix, which all include clear instructions on how they were created
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [N/A]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [N/A]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] We include two real-world case studies of how this methodology is employed with practical value at leading internet streaming company. A draft of this paper has been reviewed internally by said company and cleared for submission
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
14

---

5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "kt_kNkMM9M8", "reviewer": "Reviewer_PFqX", "summary": "This paper proposes new sequential hypothesis tests and corresponding confidence sets and intervals for multinomial data. These tests provide \"always-valid\" guarantees, i.e. controlled Type I error probability under optional stopping/continuation.\nFirst, the authors consider the direct application of detecting departures from a specified multinomial parameter, that is sample ratio mismatch tests.\nSecond, they consider the problem of comparing the success rate of k arms corresponding to inhomogeneous Bernoulli processes. The parameters of the Bernoullis can vary with time (but in the same way for all arms). \nThis problem is mapped to a multinomial one by considering the probability of each arm producing the next success conditioned on the event that there is a success.\nThis point of view has several advantages:\n1. It allows using the sequential constructions for tests, confidence sets and intervals proposed in the first part.\n2. It only needs to count success observations.\n3. The time varying effect gets canceled out.\n\nFinally, the problem of comparing the intensity of inhomogeneous Poisson counting processes is also mapped to a multinomial problem by showing that the probability of the next observation belonging to the i-th process is a multinomial random variable.\n", "questions": "The test of [1, Section 6.2] (based on [2])  is also based on a sequential test for multinomial count data, with the difference that the multinomial count data is conditioned on a continuous variable, making the alternative hypothesis composite and allowing to build a nonparametric two-sample test. I think it is worth mentioning it since it has a similar spirit and is based on the same tools/properties (likelihood ratio, multinomial model, sequential Bayesian updating, nonnegative supermartingales, optional stop/continuation).\n\n*[1] Lhéritier, A., & Cazals, F. (2019). Low-complexity nonparametric Bayesian online prediction with universal guarantees. Advances in Neural Information Processing Systems, 32.*\n\n*[2] Lhéritier, A., & Cazals, F. (2018). A sequential non-parametric multivariate two-sample test. IEEE Transactions on Information Theory, 64(5), 3361-3370.*\n\nMinor comments:\n\nLine 49: $\\mathbf{x}_1,\\mathbf{x}_2,\\mathbf{x}_3,…$ are vectors so I guess they are dummy coded variables, maybe it is worth mentioning it.\n\nLine 176: maybe you could say \"so that the time-varying effect $\\mu(t)$\" to explicitly introduce $\\mu(t)$\n\nEq just after line 196: $\\mathbf{a}$ should be in bold font\n\nReferences are not sorted alphabetically or numerically, so it is difficult to find things in there.\n", "limitations": "Yes", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I enjoyed reading this paper that addresses difficult and interesting statistical problems.\nAlthough the paper is very well written, I found it a bit unfortunate that part of the introductory material had to go to the Appendix, but I understand it was necessary for space constraints.\nAs explained in the paper, the construction of the sequential test for multinomial point hypotheses follows standard steps that are well known in the community and the theoretical results of this part are straightforward consequences of known results.\n\nFrom my point of view, the most interesting and novel contributions are those of Section 3 and 4, i.e. how to turn the problem of comparing time-inhomogeneous Bernoulli and Poisson counting processes into a multinomial problem.\nMoreover, these problems have important practical applications that are very relevant for the NeurIPS community.\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "hXux12cnliT", "reviewer": "Reviewer_6qpV", "summary": "This paper proposes a sequential multinomial test where Type I error is controlled through the martingale properties and the test is consistent. Then, the paper show how to apply the proposed test to test the data generated from the Poisson counting process and the Bernoulli process. \n\n== After rebuttal ==\\\nI want to thank the authors for their efforts to clarify my misunderstanding. I have more appreciation of this work now and meanwhile still have some concerns remaining. \n\nThe pros of this work after the clarification: controlling the Type I of this proposed multinomial test naturally comes from the martingale properties of a variant of the sequential probability ratio test (SPRT), and also the consistency of the proposed test is also based on the nice consistent property of the SPRT. The Dirichlet prior with $\\alpha_{0,i}=k\\theta_{0,i}$ is a reasonable set-up to simulate an alternative where there is only a small departure from the null. \n\nSome concerns after the clarification: the authors point out that the comparison with the $\\chi^2$ test is shown in A9 Figure 4, but from what I read, isn't this a Type I error rate comparison? Does this work have the power (Type II) comparison between the proposed test and the $\\chi^2$ test? Besides, one of the biggest benefits of a sequential test is the early (optional) stopping, but the drawback is that, with the same sample size allowed, its power will be attenuated compared with a fixed-size hypothesis testing. I tried to find through the appendix to see the experimental evidence that the stopping time increases with the increasing difficulties of the alternative hypothesis, but I didn't see that in the appendix. Lastly, a sequential test usually wouldn't enjoy too much benefit when the considered alternative hypothesis only has a minimum departure from the null, and this is the setting (small departure from the null) considered in Section 2.1. I wonder does the proposed test still can reject the alternative fast in this setting. If so, what's the intuition of that? \n\nI would love to kindly ask the authors to remind me if I miss something here. Thank you. \n\n=== correction ===\nIn the end of the comments above, I was asking can the test reject the null fast under the alternative. \n\n===final score ===\nThe authors did a great job to largely relieve my concerns and help me understand their work more. Thus I am willing to increase my score to 6.", "questions": "Thm 2.4 is confusing. From my understanding, $\\mathcal{O}$($\\theta$)_n is a random variable depending on $\\theta$, and data points. generated according to some distributions. It seems the authors want to define a set of $\\theta$ (under the null) that would not be rejected by the test, but isn't that every $\\theta$ under the null has a chance to be rejected, given the data points are randomly generated? ", "limitations": "See above.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The design of the test is technically solid. The Type I error is controlled and the test is a consistent test.\nWeakness: (1) The assumption on the parameters (line 52 to line 56) about the alternative may not be accurate, but the consistency of the test heavily depends on such an assumption. (2) The motivation for developing the test instead of using some two-sample tests such as a T-test is missing. At least the author should show the experimental comparisons with some classical A/B tests.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "BgP3FXP5D76", "reviewer": "Reviewer_FAoW", "summary": "The paper introduces a sequential test for multinomial hypotheses using a martingale construction. It then uses this result to develop a sequential test of equality and contrasts in inhomogeneous Bernoulli processes and time-inhomogeneous Poisson counting processes.", "questions": "Why do you assume  \"departures from the null to be small \"? Do the results hold even when this assumption is not true?\n\nWhy do you focus on Byes factors and then derive results  about frequentist Type I and II errors? Then why did you not consider a likelihood ratio test instead of the Bayes factor ?.", "limitations": "The assumption \"departures from the null to be small\" is very strong.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The strengths are (1) the authors use martingale inequalities to construct a test martingale that\ncontrols the frequentist Type I probability below a desired level u. (2) they obtain a confidence with a coverage guarantee of at least\n1-u.\n\nThe main assumption of the test is that \"departures from the null to be small and encode this information into the Dirichlet prior\". Null-hypotheses are never fully true and, therefore, the above assumption decreases the robustness of the test. \nI am not sure the result is really novel and in any case it seems to be an application of the results in Test Martingales, Bayes Factors\nand p-Values by Shafer et al.. I believe a statistical conference/journal would be a more appropriate venue for this paper, where the contributions of this paper can be discussed with more details with respect to the state of the art. \nIt is also not clear to me why the authors focus on Bayes factors and then derive results about frequentist Type I and II errors. Then why did they not consider a likelihood ratio test instead of a Bayes factor ? From a Neurips perspective, the authors are considering a simple problem (count data) and other approaches could be used. For instance, one could simply compute the posterior of theta and then use alpha-% credible intervals for drawing conclusions about theta_0. This approach would work quite well in practice considering the goal is to estimate theta from counts.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "4deKyodG1W0", "reviewer": "Reviewer_QXQV", "summary": "This paper introduces two online sequential tests of equality and contrasts, for Bernoulli and Poisson process, based on multinomial hypothesis. In simulated studies, the new tests show the speed up in decision making and reduce the opportunity cost in data collection process.", "questions": "- \\cite is all wrong in the draft.\n- mSPRT-type is not defined\n- Figure legends are hard to see clearly\n", "limitations": "adequately, in fact the discussion is thorough. ", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n- sequential testing is an important problem and authors demonstrated an effect approach\n- some theoretical results are provided on the tests\n- multiple applications are tested\n\nWeakness:\n- the results are not surprising. For example, theorem 4.1  and Lemma A.3 seems known/straightforward. The resulting test, which is just a ratio test, is not so difficult to come up with.\n- no comparison with baselines are conducted in the experiments. It didn't demonstrate the efficiency gain of the proposed approach against potential baselines. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept