SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[1] Shira Mitchell, Eric Potash, Solon Barocas, Alexander D'Amour, and Kristian Lum. Algorithmic fairness: Choices, assumptions, and definitions. Annual Review of Statistics and Its Application, 8:141-163, 2021. 1
[2] Peter Henderson, Ben Chugg, Brandon Anderson, and Daniel E Ho. Beyond ads: Sequential decision-making algorithms in law and public policy. In Proceedings of the 2022 Symposium on Computer Science and Law, pages 87-100, 2022. 1
[3] Deborah Hellman. Measuring algorithmic fairness. Virginia Law Review, 106(4):811-866, 2020. 1
[4] Jon Kleinberg, Jens Ludwig, Sendhil Mullainathan, and Ashesh Rambachan. Algorithmic fairness. In AEA papers and proceedings, volume 108, pages 22-27, 2018.
[5] Allison Woodruff, Sarah E Fox, Steven Rousso-Schindler, and Jeffrey Warshaw. A qualitative exploration of perceptions of algorithmic fairness. In Proceedings of the 2018 CHI conference on human factors in computing systems, pages 1-14, 2018. 1
[6] Inioluwa Deborah Raji and Joy Buolamwini. Actionable auditing: Investigating the impact of publicly naming biased performance results of commercial AI products. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pages 429-435, 2019. 1
[7] Jack Bandy. Problematic machine behavior: A systematic literature review of algorithm audits. Proceedings of the ACM on human-computer interaction, 5:1-34, 2021.
[8] Danaë Metaixa, Joon Sung Park, Ronald E. Robertson, Karrie Karahalios, Christo Wilson, Jeff Hancock, and Christian Sandvig. Auditing algorithms: Understanding algorithmic systems from the outside in. Foundations and Trends® in Human-Computer Interaction, 14(4):272-344, 2021. 1
[9] Briana Vecchione, Karen Levy, and Solon Barocas. Algorithmic auditing and social justice: Lessons from the history of audit studies. In Equity and Access in Algorithms, Mechanisms, and Optimization, pages 1-9. 2021. 1
[10] Inioluwa Deborah Raji, Andrew Smart, Rebecca N. White, Margaret Mitchell, Timnit Gebru, Ben Hutchinson, Jamila Smith-Loud, Daniel Theron, and Parker Barnes. Closing the AI accountability gap: Defining an end-to-end framework for internal algorithmic auditing. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pages 33-44, 2020. 1
[11] John R Zech, Marcus A Badgeley, Manway Liu, Anthony B Costa, Joseph J Titano, and Eric Karl Oermann. Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross-sectional study. PLoS medicine, 15(11):e1002683, 2018. 1
[12] Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In Proceedings of the European conference on computer vision (ECCV), pages 456-473, 2018. 1
[13] Xiaoxuan Liu, Ben Glocker, Melissa M McCradden, Marzyeh Ghassemi, Alastair K Denniston, and Lauren Oakden-Rayner. The medical algorithmic audit. The Lancet Digital Health, 2022. 1
[14] Bahar Taskesen, Jose Blanchet, Daniel Kuhn, and Viet Anh Nguyen. A statistical test for probabilistic fairness. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 648-665, 2021. 1, 2
[53] Ilmun Kim, Sivaraman Balakrishnan, and Larry Wasserman. Minimax optimality of permutation tests. The Annals of Statistics, 50(1):225-251, 2022. 9
[54] Muhammad Aurangzeb Ahmad, Arpit Patel, Carly Eckert, Vikas Kumar, and Ankur Teredesai. Fairness in machine learning for healthcare. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 3529-3530, 2020. 10
[55] Douglas B White and Derek C Angus. A proposed lottery system to allocate scarce COVID-19 medications: promoting fairness and generating knowledge. Jama, 324(4):329–330, 2020. 10
[56] Ben Chugg, Lisa Lu, Derek Ouyang, Benjamin Anderson, Raymond Ha, Alexis D'Agostino, Anandi Sujeer, Sarah L Rudman, Analilia Garcia, and Daniel E Ho. Evaluation of allocation schemes of COVID-19 testing resources in a community-based door-to-door testing program. In JAMA health forum, volume 2, pages e212260–e212260. American Medical Association, 2021. 10
[57] Inioluwa Deborah Raji, Peggy Xu, Colleen Honigsberg, and Daniel Ho. Outsider oversight: Designing a third party audit ecosystem for AI governance. In Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pages 557-571, 2022. 10
[58] Nick Harvey. A second course in randomized algorithms. 2022. 17
A Testing in batches
Of course, it is the rare application in which we can expect to receive an audit from each group at each timestep. We can modify the strategy in Section 3 to handle different arrival times by simply waiting to bet until we have new audits from each group. If multiple audits from the other group have accumulated during that period, we bet with their average. More formally, define
$$
N _ {t} ^ {b} = T _ {b} [ t ] \cap \{n: n \geq \max  (T _ {1 - b} [ t ]) \}, \text {a n d} G _ {t} ^ {b} = \frac {1}{\left| N _ {t} ^ {b} \right|} \sum_ {j \in N _ {t} ^ {b}} \widehat {Y} _ {j} ^ {b}, \tag {15}
$$
where we recall that  $T_{b}[t] = T_{b} \cap [t]$  is the set of times at which we receive audits from group  $b$  up until time  $t$ . In words,  $N_{t}^{b}$  is simply all timesteps for which we receive an audit from group  $b$  without receiving an audit from the other group. From here, we can define a payoff function as
$$
S _ {t} = \left\{ \begin{array}{l l} 1 + \lambda_ {t} \left(G _ {t} ^ {0} - G _ {t} ^ {1}\right), & \text {i f N} _ {t} ^ {0} \neq \emptyset \text {a n d} N _ {t} ^ {1} \neq \emptyset , \\ 1, & \text {o t h e r w i s e .} \end{array} \right. \tag {16}
$$
If  $S_{t} = 1$  then  $\mathcal{K}_t = \mathcal{K}_{t-1}$ , so this can be interpreted as abstaining from betting at time  $t$ . This occurs when we have no new audits from one group  $(N_{t}^{b} = \emptyset$  for some  $b)$ . Note that if  $N_{t}^{0}$  and  $N_{t}^{1}$  are both non-empty, then one of them contains only  $t$ . Thus, in this case,  $G_{t}^{0} - G_{t}^{1} \in \{\widehat{Y}_{t}^{0} - G_{t}^{1}, G_{t}^{0} - \widehat{Y}_{t}^{1}\}$ . As in Section 3 we choose  $\lambda_{t}$  using ONS, but we use  $g_{t} = G_{t}^{0} - G_{t}^{1}$  in its definition. The expected stopping time of the test defined by (16) follows as a corollary from Proposition 1 after correcting for the number of timesteps during which non-trivial bets were placed. More specifically, the stopping-time in Proposition 1 is with reference to the number of times we actually bet, i.e., with respect to the times  $\{t : N_{t}^{0} \neq \emptyset, N_{t}^{1} \neq \emptyset\}$ . The same remark holds for other propositions throughout this paper and, as such, we will state these propositions under the assumption that  $T_{0} = T_{1} = \mathbb{N}$ .
Finally, let us emphasize that the above discussion is still assuming an online data gathering procedure. If we are in a nonsequential setting (i.e., all the data has already been gathered), then we may simulate a sequential setting by simply revealing the outcomes one at a time. Thus, all of our procedures hold for fixed-time hypothesis testing as well.
B Auditing multiple groups
Here we consider the case when there are more than two groups. Suppose we have  $J + 1$  groups  $\{0, 1, \ldots, J\}$ . In accordance with Definition 1, the null and alternative become
$$
H _ {0}: \mathbb {E} _ {\rho} [ \varphi (X) | \xi_ {i} ] = \mathbb {E} _ {\rho} [ \varphi (X) | \xi_ {j} ], \quad \forall i, j \in \{0, \dots , J \}, \tag {17}
$$
$$
H _ {1}: \exists i, j \in \{0, \dots , J \} \text {s u c h t h a t} \mathbb {E} _ {\rho} [ \varphi (X) | \xi_ {i} ] \neq \mathbb {E} _ {\rho} [ \varphi (X) | \xi_ {j} ]. \tag {18}
$$
As before, let  $\mu_{i} = \mathbb{E}_{\rho}[\varphi (X)|\xi_{i}]$ ,  $i\in \{0,\dots ,J\}$ . One could derive a sequential test by applying Algorithm 1 to each pair of means  $\mu_{i}$  and  $\mu_{j}$ . Game-theoretically, this can be interpreted as splitting your initial wealth among multiple games and playing each simultaneously. If you grow rich enough in any one game, you reject the null. Of course, one needs to adjust the significance level to account for the number of games being played, thus reducing the (nonasymptotic) power of the test.
Of course, it is not necessary to test each mean against all others. We need only test whether  $\mu_{b} = \mu_{b + 1}$  for all  $b\in \{0,\dots ,J\}$ . That is, we can play  $J$  games instead of  $\Omega (J^2)$  games. In order to ensure this constitutes a level-  $\alpha$  test, we reject when the wealth process of any game is at least  $J / \alpha$ . For any single game, this occurs with probability  $\alpha /J$  under the null. Therefore, the union bound then ensures that the type-I error of this procedure is bounded by  $\alpha$ . Moreover, the asymptotic power remains one since, if  $\mu_i\neq \mu_j$  for some  $i,j$  then  $\mu_b\neq \mu_{b + 1}$  for some  $b$ . The guarantees we've provided on Algorithm 1 ensure that the wealth process for this particular game will eventually grow larger than  $J / \alpha$  and our test will reject. We summarize this discussion with the following proposition, which is the equivalent of Proposition 1 for auditing multiple groups.
Proposition 5. Let  $\alpha \in (0,1)$ . Consider running Algorithm 1 on groups  $b, b + 1$ , for  $b \in \{0,1,\ldots,J - 1\}$  in parallel with input parameter  $\alpha / J$ . This constitutes a level- $\alpha$  sequential test for (17) with asymptotic power one against (18). If we receive an audit from each group at each timestep, then the expected stopping time  $\tau$  of this procedure obeys
$$
\mathbb {E} [ \tau ] \lesssim \min  _ {b \in \{0, \dots , J - 1 \}} \frac {1}{| \mu_ {b} - \mu_ {b + 1} | ^ {2}} \log \left(\frac {J}{| \mu_ {b} - \mu_ {b + 1} | ^ {2} \alpha}\right). \tag {19}
$$
The expected stopping time follows from Proposition 1 after correcting for the significance level and the difference between the means. We take the minimum over all  $b$  because the procedure rejects as soon as any of the wealth processes grow too large. Equivalent versions of Propositions 2 and 3 for multiple groups can be obtained similarly.
Let us end by remarking that similar ideas may be applied to test other notions of group fairness which posit the equivalence of multiple means. This is the case of equalized odds, for instance. As above we simply imagine playing multiple games simultaneously, each testing for the equivalence of one pair of means.
C Omitted proofs
We break the proof into three components.
Level- $\alpha$  sequential test. Combining the discussion at the beginning of Section 3 with Ville's inequality demonstrates why our procedure constitutes a level- $\alpha$  sequential test. However, let us prove it formally here for completeness. Let  $P \in H_0$  and note that  $\mathbb{E}_P[\widehat{Y}_t^0 - \widehat{Y}_t^{1}] = \mathbb{E}_P[\varphi(X_t^0) - \varphi(X_t^1)] = \mu_0 - \mu_1 = 0$ . Therefore, using the fact that  $\lambda_t$  is predictable (i.e.,  $\mathcal{F}_{t-1}$ -measurable)
$$
\mathbb {E} _ {P} \left[ \mathcal {K} _ {t} \mid \mathcal {F} _ {t - 1} \right] = \mathbb {E} _ {P} \left[ \prod_ {j = 1} ^ {t} \left(1 + \lambda_ {j} \left(\widehat {Y} _ {j} ^ {0} - \widehat {Y} _ {j} ^ {1}\right)\right) \middle | \mathcal {F} _ {t - 1} \right] = \mathcal {K} _ {t - 1} \left(1 + \lambda_ {t} \mathbb {E} _ {P} \left[ \widehat {Y} _ {t} ^ {0} - \widehat {Y} _ {t} ^ {1} \right]\right) = \mathcal {K} _ {t - 1},
$$
so  $(\mathcal{K}_t)_{t\geq 1}$  is a  $P$ -martingale, with initial value 1. Moreover, it is nonnegative since  $|\lambda_t|\leq 1 / 2$  for all  $t$  by definition of ONS. Thus, Ville's inequality implies  $P(\exists t\geq 1:\mathcal{K}_t\geq 1 / \alpha)\leq \alpha$ , meaning that rejecting at  $1 / \alpha$  yields a level- $\alpha$  sequential test. Finally, as discussed in the main paper, the last lines of Algorithm 1 are justified by the randomized Ville's inequality of Ramdas and Manole [43], which states that, for all stopping times  $n$ ,
$$
P (\exists t \leq n: \mathcal {K} _ {t} \geq 1 / \alpha \text {o r} \mathcal {K} _ {n} > U / \alpha) \leq \alpha ,
$$
where  $U\sim \mathrm{Unif}(0,1)$  is independent of everything else.
Asymptotic power. Next, let us demonstrate that Algorithm 1 has asymptotic power one. That is, for  $P \in H_1$ ,  $P(\tau < \infty) = 1$ . It suffices to show that  $P(\tau = \infty) = 0$ . To see this, define
$$
g _ {t} := \widehat {Y} _ {t} ^ {0} - \widehat {Y} _ {t} ^ {1}, \quad S _ {t} := \sum_ {i = 1} ^ {t} g _ {i}, \quad V _ {t} := \sum_ {i = 1} ^ {t} g _ {i} ^ {2}.
$$
We have the following guarantee on the wealth process, which can be translated from results concerning ONS from Cutkosky and Orabona [44, Theorem 1]:
$$
\mathcal {K} _ {t} \geq \frac {1}{V _ {t}} \exp \left\{\frac {S _ {t} ^ {2}}{4 \left(V _ {t} + | S _ {t} |\right)} \right\} \geq \frac {1}{t} \exp \left\{\frac {S _ {t} ^ {2}}{8 t} \right\}, \quad \forall t \geq 1. \tag {20}
$$
Since  $\{\tau = \infty\} \subset \{\tau \geq t\}$  for all  $t \geq 1$ , we have  $P(\tau = \infty) \leq \lim \inf_{t \to \infty} P(\tau > t) \leq \lim \inf_{t \to \infty} P(\mathcal{K}_t < 1 / \alpha)$ , where the final inequality is by definition of the algorithm. Using the second inequality of (20),
$$
P (\mathcal {K} _ {t} <   1 / \alpha) \leq P \bigg (\exp \left\{\frac {S _ {t} ^ {2}}{8 t} \right\} <   t / \alpha \bigg) \leq P \bigg (- \sqrt {\frac {8 \log (t / \alpha)}{t}} <   \frac {S _ {t}}{t} <   \sqrt {\frac {8 \log (t / \alpha)}{t}} \bigg).
$$
By the SLLN,  $S_{t} / t$  converges to  $\mu_0 - \mu_1 \neq 0$  almost surely. On the other hand,  $8\log (t / \alpha) / t \to 0$ . Thus, if we let  $A_{t}$  be the event that  $\exp (S_t^2 /8t) < t / \alpha$ , we see that  $\mathbf{1}(A_t) \to 0$  almost surely. Hence, by the dominated convergence theorem,
$$
P (\tau = \infty) \leq \lim _ {t \to \infty} \inf P (A _ {t}) = \lim _ {t \to \infty} \inf \int {\bf 1} (A _ {t}) d P = \int \lim _ {t \to \infty} \inf {\bf 1} (A _ {t}) d P = 0.
$$
This completes the argument.
Expected stopping time. Last, let us show the desired bound on the expected stopping time. Fix  $P \in H_1$ . Let  $\tau$  be the stopping time of the test. Since it is nonnegative, we have
$$
\mathbb {E} [ \tau ] = \sum_ {t = 1} ^ {\infty} P (\tau > t) = \sum_ {t = 1} ^ {\infty} P (\log \mathcal {K} _ {t} <   \log (1 / \alpha)) = \sum_ {t = 1} ^ {\infty} P (E _ {t}),
$$
for  $E_{t} = \{\log \mathcal{K}_{t} < \log (1 / \alpha)\}$ . Note that the second equality is by definition of the algorithm. Employing the first inequality of (20) yields
$$
\begin{array}{l} E _ {t} \subset \left\{S _ {t} ^ {2} <   4 \left(V _ {t} + \left| S _ {t} \right|\right) \left(\log (1 / \alpha) - \log (1 / V _ {t}) \right. \right\} \\ \subset \left\{S _ {t} ^ {2} <   4 \Big (V _ {t} + \sum_ {i \leq t} | g _ {i} | \Big) (\log (1 / \alpha) - \log (1 / V _ {t}) \right\}. \\ \end{array}
$$
To analyze the probability of this event, we first develop upper bounds on  $W_{t} \coloneqq \sum_{i \leq t} |g_{i}|$  and  $V_{t}$ . We begin with  $W_{t}$ . Since  $W_{t}$  is the sum of independent random variables in [0, 1], we apply the multiplicative Chernoff bound (e.g., [58]) to obtain
$$
P (W _ {t} > (1 + \delta) \mathbb {E} [ W _ {t} ]) \leq \exp (- \delta^ {2} \mathbb {E} [ W _ {t} ] / 3).
$$
Setting the right hand side equal to  $1 / t^2$  and solving for  $\delta$  gives  $\delta = \sqrt{6\log t / \mathbb{E}W_t}$ . Thus, with probability  $1 - 1 / t^2$ , we have
$$
W _ {t} \leq \mathbb {E} W _ {t} + \sqrt {6 \mathbb {E} \left[ W _ {t} \right] \log t} = t + \sqrt {6 t \log t} \leq 2 t \quad \forall t \geq 1 7, \tag {21}
$$
where we've used that  $\mathbb{E}[W_t] = \sum_{i\leq t}\mathbb{E}[|g_i|]\leq t$  since  $|g_{i}|\leq 1$ . Following a nearly identical process for  $V_{t}$ , we have that with probability  $1 - 1 / t^{2}$
$$
V _ {t} \leq \mathbb {E} \left[ V _ {t} \right] + \sqrt {6 \mathbb {E} \left[ V _ {t} \right] \log t} \leq t + \sqrt {6 t \log t} \leq 2 t, \quad \forall t \geq 1 7, \tag {22}
$$
where again we use that  $|g_i|^2 \leq |g_i| \leq 1$ . Let  $H_t = \{V_t \leq 2t\} \cap \{W_t \leq 2t\}$ . Then,
$$
E _ {t} \cap H _ {t} \subset \{S _ {t} ^ {2} <   1 6 t (\log (1 / \alpha) + \log (2 t) \} \subset \{| S _ {t} | <   \underbrace {4 \sqrt {t \log (2 t / \alpha)}} _ {:= D} \}.
$$
We now argue that  $|S_t|$  is unlikely to be so small. Indeed, since  $S_t$  is the sum of independent random variables in  $[-1, 1]$ , applying a Chernoff bound for the third time gives  $P(|S_t - \mathbb{E}S_t| \geq u) \leq 2\exp(-u^2 / t)$ . So, with probability  $1 - 1 / t^2$ , by the reverse triangle inequality,
$$
\left| \left| S _ {t} \right| - \left| \mathbb {E} S _ {t} \right| \right| \leq \left| S _ {t} - \mathbb {E} S _ {t} \right| \leq \sqrt {t \log 2 t ^ {2}},
$$
implying that,
$$
\left| S _ {t} \right| \geq \left| \mathbb {E} S _ {t} \right| - \sqrt {t \log 2 t ^ {2}} \geq t \Delta - \sqrt {2 t \log 2 t}.
$$
This final quantity is at least  $D$  for all  $t \geq \frac{81}{\Delta^2} \log \left( \frac{162}{\Delta^2 \alpha} \right)$ . Now, combining what we've done thus far, by the law of total probability,
$$
P (E _ {t}) = P (E _ {t} \cap H _ {t}) + P (E _ {t} | H _ {t} ^ {c}) P (H _ {t} ^ {c}) \leq P (| S _ {t} | <   D) + P (H _ {t} ^ {c}) \leq 3 / t ^ {2},
$$
and so, for  $t$  large enough such that (21), (22), and  $S_{t} > D$  all hold, that is
$$
T = \frac {8 1}{\Delta^ {2}} \log \left(\frac {1 6 2}{\Delta^ {2} \alpha}\right),
$$
$$
\mathbb {E} [ \tau ] = \sum_ {t \geq 1} P (E _ {t}) \leq T + \sum_ {t \geq T} \frac {3}{t ^ {2}} \leq T + \frac {\pi^ {2}}{2}.
$$
The proof is similar to that of Proposition 1, so we highlight only the differences.
The wealth process remains a martingale due to the IPW weights (5). Indeed, since  $\lambda_{t}$  and  $L_{t}$  are  $\mathcal{F}_{t - 1}$  measurable, under the null we have
$$
\begin{array}{l} \mathbb {E} \left[ \mathcal {K} _ {t} \mid \mathcal {F} _ {t - 1} \right] = \mathbb {E} \left[ \prod_ {j = 1} ^ {t} \left(1 + \lambda_ {j} L _ {j} \left(\widehat {Y} _ {j} ^ {0} \omega_ {j} ^ {0} - \widehat {Y} _ {j} ^ {1} \omega_ {j} ^ {1}\right)\right) \middle | \mathcal {F} _ {t - 1} \right] \\ = \mathcal {K} _ {t - 1} (1 + \lambda_ {t} L _ {t} \mathbb {E} [ \widehat {Y} _ {t} ^ {0} \omega_ {t} ^ {0} - \widehat {Y} _ {t} ^ {1} \omega_ {t} ^ {1} ]) = \mathcal {K} _ {t - 1} (1 + \lambda_ {t} L _ {t} (\mu_ {0} - \mu_ {1})) = \mathcal {K} _ {t - 1}. \\ \end{array}
$$
Moreover, as described in the text, multiplication by  $L_{t}$  ensures that  $\mathcal{K}_t$  is nonnegative, since
$$
\begin{array}{l} | L _ {t} | | \widehat {Y} _ {t} ^ {0} \omega_ {t} ^ {0} (X _ {t} ^ {0}) - \widehat {Y} _ {t} ^ {1} \omega_ {t} ^ {1} (X _ {t} ^ {1}) | \leq L _ {t} | \widehat {Y} _ {t} ^ {0} \omega_ {t} ^ {0} (X _ {t} ^ {0}) | + L _ {t} | \widehat {Y} _ {t} ^ {1} \omega_ {t} ^ {1} (X _ {t} ^ {1}) | \\ \leq L _ {t} \omega_ {t} ^ {0} \left(X _ {t} ^ {0}\right) + L _ {t} \omega_ {t} ^ {1} \left(X _ {t} ^ {1}\right) \leq 1, \\ \end{array}
$$
$$
L _ {t} \leq \frac {1}{2 \omega_ {t} ^ {b} \left(X _ {t} ^ {b}\right)},
$$
for each  $b$  by definition. Therefore,  $(\mathcal{K}_t)_{t\geq 1}$  is a nonnegative martingale and, as before, Ville's inequality implies that rejecting at  $1 / \alpha$  gives a level-  $\alpha$  sequential test.
The asymptotic power follows by replacing  $g_{t}$  in Appendix C.1 with
$$
h _ {t} = L _ {t} \left(\widehat {Y} _ {t} ^ {0} \omega_ {t} ^ {0} - \widehat {Y} _ {t} ^ {1} \omega_ {t} ^ {1}\right).
$$
Under the alternative,  $h_t$  has non-zero expected value, so identical arguments apply.
Regarding, the expected stopping time, we again argue about  $h_t$  instead of  $g_t$ . Since  $|h_t| \leq 1$  (see above), the bounds on  $V_t$  and  $W_t$  remain as they are in the proof of Proposition 1. The bound on  $|\mathbb{E}[S_t]|$  is where the proof departs that in Appendix C.1. In this case we have
$$
\mathbb {E} [ S _ {t} | \mathcal {F} _ {t - 1} ] = S _ {t - 1} + L _ {t} \mathbb {E} [ \widehat {Y} _ {t} ^ {0} \omega_ {t} ^ {0} - \widehat {Y} _ {t} ^ {1} \omega_ {t} ^ {1} | \mathcal {F} _ {t - 1} ] = S _ {t - 1} + L _ {t} (\mu_ {0} - \mu_ {1}).
$$
$$
\mathbb {E} [ S _ {t} ] = \mathbb {E} [ \mathbb {E} [ S _ {t} | \mathcal {F} _ {t - 1} ] ] = \mathbb {E} [ S _ {t - 1} + L _ {t} (\mu_ {0} - \mu_ {1}) ] = \mathbb {E} [ S _ {t - 1} ] + (\mu_ {0} - \mu_ {1}) \mathbb {E} [ L _ {t} ].
$$
Induction thus yields
$$
| \mathbb {E} [ S _ {t} ] | = \left| (\mu_ {0} - \mu_ {1}) \sum_ {i \leq t} \mathbb {E} [ L _ {i} ] \right| = \Delta \left| \sum_ {i \leq t} \mathbb {E} [ L _ {i} ] \right| \geq \Delta t L _ {\inf}.
$$
From here, we may replace  $\Delta$  in the proof in Appendix C.1 with  $\Delta L_{\mathrm{inf}}$  and the arithmetic remains the same. This yields the desired result.
C.3 Proof of Proposition 3
Again, the proof mirrors that of Proposition 1 so we highlight only the differences.
First let us ensure that Algorithm 1 yields a level- $\alpha$  sequential test. As before, it suffices to demonstrate that the wealth process is a nonnegative martingale. The time-varying means do not change this fact from before:
$$
\mathbb {E} [ \mathcal {K} _ {t} | \mathcal {F} _ {t - 1} ] = \mathbb {E} \bigg [ \prod_ {j = 1} ^ {t} (1 + \lambda_ {j} (\widehat {Y} _ {j} ^ {0} - \widehat {Y} _ {j} ^ {1})) \bigg | \mathcal {F} _ {t - 1} \bigg ] = \mathcal {K} _ {t - 1} (1 + \lambda_ {t} \mathbb {E} [ \widehat {Y} _ {t} ^ {0} - \widehat {Y} _ {t} ^ {1} | \mathcal {F} _ {t - 1} ]) = \mathcal {K} _ {t - 1},
$$
since, under the null,  $\mathbb{E}[Y_t^0 |\mathcal{F}_{t - 1}] = \mathbb{E}[\varphi (X)|\xi_0,\mathcal{F}_{t - 1}] = \mu_0 = \mu_1 = \mathbb{E}[\varphi (X)|\xi_1,\mathcal{F}_{t - 1}] =$ $\mathbb{E}[Y_t^1 |\mathcal{F}_{t - 1}]$ . Nonnegativity once again follows from the ONS strategy.
Asymptotic power follows an identical argument as in Appendix C.1, so we focus on expected stopping time. The event  $E_{t}$  remains the same as in Appendix C.1. We again apply a Chernoff bound
to  $W_{t}$  (the values remain independent, even though they are not necessarily identically distributed), and obtain
$$
W _ {t} \leq \mathbb {E} W _ {t} + \sqrt {6 \mathbb {E} [ W _ {t} ] \log t} \leq 2 t,
$$
for  $t \geq 17$  with probability  $1 - 1/t^2$ , since again,  $|g_i| \leq 1$  for each  $i$ . Similarly,  $\mathbb{E}V_t \leq 2t$  with probability  $1 - 1/t^2$  for  $t \geq 17$ . Let the shift begin at time  $n$ , and set  $\Delta = \inf_{t \geq n} |\mu_0(t) - \mu_1(t)|$ . Then  $|\mathbb{E}S_t| \geq (t - n)\Delta$ . As above, we want to find  $t$  such that
$$
\left| S _ {t} \right| \geq \left| \mathbb {E} S _ {t} \right| - \sqrt {t \log 2 t ^ {2}} \geq (t - n) \Delta - \sqrt {t \log 2 t ^ {2}} \geq D.
$$
Rearranging and simplifying this final inequality, we see that it suffices for  $t$  to satisfy
$$
t - n \geq \frac {6}{\Delta} \sqrt {t \log (2 t / \alpha)}. \tag {23}
$$
We claim this holds for all
$$
t \geq n + \max  \left\{n, \frac {1 0 8}{\Delta^ {2}} \log \left(\frac {4 \cdot 1 0 8}{\Delta^ {2} \alpha}\right) \right\}.
$$
To see this, suppose first that  $n \geq \beta$  where
$$
\beta = \frac {1 0 8}{\Delta^ {2}} \log \left(\frac {4 \cdot 1 0 8}{\Delta^ {2} \alpha}\right).
$$
Then, at  $t = 2n$ , the right hand side of (23) is
$$
\frac {6}{\Delta} \sqrt {2 n \log (2 n / \alpha)} \leq n,
$$
where the final inequality holds for all  $n \geq \beta$ , which was assumed. Now suppose that  $n < \beta$ , so that (23) should hold for  $t \geq n + \beta$ . Since the left hand side of (23) grows faster than the right hand side, it suffices to show that it holds at  $t = n + \beta$ . To this end, write