Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper introduces two online sequential tests of equality and contrasts, for Bernoulli and Poisson process, based on multinomial hypothesis. In simulated studies, the new tests show the speed up in decision making and reduce the opportunity cost in data collection process.
strengths_and_weaknesses: Strength:
- sequential testing is an important problem and authors demonstrated an effect approach
- some theoretical results are provided on the tests
- multiple applications are tested

Weakness:
- the results are not surprising. For example, theorem 4.1  and Lemma A.3 seems known/straightforward. The resulting test, which is just a ratio test, is not so difficult to come up with.
- no comparison with baselines are conducted in the experiments. It didn't demonstrate the efficiency gain of the proposed approach against potential baselines. 
questions: - \cite is all wrong in the draft.
- mSPRT-type is not defined
- Figure legends are hard to see clearly

limitations: adequately, in fact the discussion is thorough.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a sequential multinomial test where Type I error is controlled through the martingale properties and the test is consistent. Then, the paper show how to apply the proposed test to test the data generated from the Poisson counting process and the Bernoulli process. 

== After rebuttal ==\
I want to thank the authors for their efforts to clarify my misunderstanding. I have more appreciation of this work now and meanwhile still have some concerns remaining. 

The pros of this work after the clarification: controlling the Type I of this proposed multinomial test naturally comes from the martingale properties of a variant of the sequential probability ratio test (SPRT), and also the consistency of the proposed test is also based on the nice consistent property of the SPRT. The Dirichlet prior with $\alpha_{0,i}=k\theta_{0,i}$ is a reasonable set-up to simulate an alternative where there is only a small departure from the null. 

Some concerns after the clarification: the authors point out that the comparison with the $\chi^2$ test is shown in A9 Figure 4, but from what I read, isn't this a Type I error rate comparison? Does this work have the power (Type II) comparison between the proposed test and the $\chi^2$ test? Besides, one of the biggest benefits of a sequential test is the early (optional) stopping, but the drawback is that, with the same sample size allowed, its power will be attenuated compared with a fixed-size hypothesis testing. I tried to find through the appendix to see the experimental evidence that the stopping time increases with the increasing difficulties of the alternative hypothesis, but I didn't see that in the appendix. Lastly, a sequential test usually wouldn't enjoy too much benefit when the considered alternative hypothesis only has a minimum departure from the null, and this is the setting (small departure from the null) considered in Section 2.1. I wonder does the proposed test still can reject the alternative fast in this setting. If so, what's the intuition of that? 

I would love to kindly ask the authors to remind me if I miss something here. Thank you. 

=== correction ===
In the end of the comments above, I was asking can the test reject the null fast under the alternative. 

===final score ===
The authors did a great job to largely relieve my concerns and help me understand their work more. Thus I am willing to increase my score to 6.
strengths_and_weaknesses: Strengths: The design of the test is technically solid. The Type I error is controlled and the test is a consistent test.
Weakness: (1) The assumption on the parameters (line 52 to line 56) about the alternative may not be accurate, but the consistency of the test heavily depends on such an assumption. (2) The motivation for developing the test instead of using some two-sample tests such as a T-test is missing. At least the author should show the experimental comparisons with some classical A/B tests.
questions: Thm 2.4 is confusing. From my understanding, $\mathcal{O}$($\theta$)_n is a random variable depending on $\theta$, and data points. generated according to some distributions. It seems the authors want to define a set of $\theta$ (under the null) that would not be rejected by the test, but isn't that every $\theta$ under the null has a chance to be rejected, given the data points are randomly generated? 
limitations: See above.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes new sequential hypothesis tests and corresponding confidence sets and intervals for multinomial data. These tests provide "always-valid" guarantees, i.e. controlled Type I error probability under optional stopping/continuation.
First, the authors consider the direct application of detecting departures from a specified multinomial parameter, that is sample ratio mismatch tests.
Second, they consider the problem of comparing the success rate of k arms corresponding to inhomogeneous Bernoulli processes. The parameters of the Bernoullis can vary with time (but in the same way for all arms). 
This problem is mapped to a multinomial one by considering the probability of each arm producing the next success conditioned on the event that there is a success.
This point of view has several advantages:
1. It allows using the sequential constructions for tests, confidence sets and intervals proposed in the first part.
2. It only needs to count success observations.
3. The time varying effect gets canceled out.

Finally, the problem of comparing the intensity of inhomogeneous Poisson counting processes is also mapped to a multinomial problem by showing that the probability of the next observation belonging to the i-th process is a multinomial random variable.

strengths_and_weaknesses: I enjoyed reading this paper that addresses difficult and interesting statistical problems.
Although the paper is very well written, I found it a bit unfortunate that part of the introductory material had to go to the Appendix, but I understand it was necessary for space constraints.
As explained in the paper, the construction of the sequential test for multinomial point hypotheses follows standard steps that are well known in the community and the theoretical results of this part are straightforward consequences of known results.

From my point of view, the most interesting and novel contributions are those of Section 3 and 4, i.e. how to turn the problem of comparing time-inhomogeneous Bernoulli and Poisson counting processes into a multinomial problem.
Moreover, these problems have important practical applications that are very relevant for the NeurIPS community.



questions: The test of [1, Section 6.2] (based on [2])  is also based on a sequential test for multinomial count data, with the difference that the multinomial count data is conditioned on a continuous variable, making the alternative hypothesis composite and allowing to build a nonparametric two-sample test. I think it is worth mentioning it since it has a similar spirit and is based on the same tools/properties (likelihood ratio, multinomial model, sequential Bayesian updating, nonnegative supermartingales, optional stop/continuation).

*[1] Lhéritier, A., & Cazals, F. (2019). Low-complexity nonparametric Bayesian online prediction with universal guarantees. Advances in Neural Information Processing Systems, 32.*

*[2] Lhéritier, A., & Cazals, F. (2018). A sequential non-parametric multivariate two-sample test. IEEE Transactions on Information Theory, 64(5), 3361-3370.*

Minor comments:

Line 49: $\mathbf{x}_1,\mathbf{x}_2,\mathbf{x}_3,…$ are vectors so I guess they are dummy coded variables, maybe it is worth mentioning it.

Line 176: maybe you could say "so that the time-varying effect $\mu(t)$" to explicitly introduce $\mu(t)$

Eq just after line 196: $\mathbf{a}$ should be in bold font

References are not sorted alphabetically or numerically, so it is difficult to find things in there.

limitations: Yes
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper introduces a sequential test for multinomial hypotheses using a martingale construction. It then uses this result to develop a sequential test of equality and contrasts in inhomogeneous Bernoulli processes and time-inhomogeneous Poisson counting processes.
strengths_and_weaknesses: The strengths are (1) the authors use martingale inequalities to construct a test martingale that
controls the frequentist Type I probability below a desired level u. (2) they obtain a confidence with a coverage guarantee of at least
1-u.

The main assumption of the test is that "departures from the null to be small and encode this information into the Dirichlet prior". Null-hypotheses are never fully true and, therefore, the above assumption decreases the robustness of the test. 
I am not sure the result is really novel and in any case it seems to be an application of the results in Test Martingales, Bayes Factors
and p-Values by Shafer et al.. I believe a statistical conference/journal would be a more appropriate venue for this paper, where the contributions of this paper can be discussed with more details with respect to the state of the art. 
It is also not clear to me why the authors focus on Bayes factors and then derive results about frequentist Type I and II errors. Then why did they not consider a likelihood ratio test instead of a Bayes factor ? From a Neurips perspective, the authors are considering a simple problem (count data) and other approaches could be used. For instance, one could simply compute the posterior of theta and then use alpha-% credible intervals for drawing conclusions about theta_0. This approach would work quite well in practice considering the goal is to estimate theta from counts.
questions: Why do you assume  "departures from the null to be small "? Do the results hold even when this assumption is not true?

Why do you focus on Byes factors and then derive results  about frequentist Type I and II errors? Then why did you not consider a likelihood ratio test instead of the Bayes factor ?.
limitations: The assumption "departures from the null to be small" is very strong.
review_title: null

Meta-review: metareview: While there is no unanimity, the majority is positive and sees the potential value of the approach for machine learning. It is good to mention that the authors did a good job in replying to comments and criticisms, which has clarified a few misunderstandings. Personally I would recommend the authors not to use the confusing terminology between Bayesian and frequentist approaches that are employed in the paper, because it has generated more harm than benefit (at least among those in the discussion here). The theoretical results seem sound and useful and the contribution is good (even if the targeted problem might be seen as too specific by some).
confidence: Less certain
Decision: Accept