SlowGuess/NeurIPS_2023_OCR huggingface dataset:
selected from the population. Let  $\rho^b$  be the density of  $X|\xi_b$ . Following standard propensity weighting techniques [46, 47], introduce the weighted estimates
$$
\omega_ {t} ^ {b} (x) := \frac {\rho^ {b} (x)}{\pi_ {t} ^ {b} (x)}. \tag {5}
$$
We may write simply  $\omega_{t}^{b}$  when  $x$  is understood from context. The policy  $\pi_t^b$  (and hence the weight  $\omega_{t}^{b}$ ) need not be deterministic, only  $\mathcal{F}_{t - 1}$  measurable (so it can be considered a deterministic quantity at time  $t$ ). This is motivated by various practical applications of sequential decision-making, in which the policy used to collect data often changes over time in response to either learning or policy [48, 49]. In many applications,  $\pi_t^b$  is a function of a select few covariates only—for instance, income, education level, or career. In some settings it is reasonable to assume knowledge of  $\rho^b$  (by means of a census, for instance). However, this is not always the case. Section 3.3 will therefore discuss strategies which do not require precise knowledge of the density. We will assume that for all  $x,t$  and  $b$ ,  $\omega_{t}^{b}(x) < \infty$ . In this case, (5) enables an unbiased estimate of  $\mu_{b} = \mathbb{E}_{\rho}[\varphi (X)|\xi_{b}]$  when  $X_{t}$  is sampled according to  $\pi_t^b$ :
$$
\mathbb {E} _ {X \sim \pi_ {t} ^ {b}} [ \hat {Y} _ {t} ^ {b} \omega_ {t} ^ {b} (X) | \mathcal {F} _ {t - 1}, \xi_ {b} ] = \int_ {\mathcal {X}} \varphi (x) \omega_ {t} ^ {b} (x) \pi_ {t} ^ {b} (x) d x = \int_ {\mathcal {X}} \varphi (x) \rho^ {b} (x) d x = \mu_ {b}. \tag {6}
$$
Our payoff function is similar to the previous section, but reweights the samples by  $\omega_{t}^{b} (= \omega_{t}^{b}(X_{t}^{b}))$ , and then adds a corrective factor to ensure that the payoff is nonnegative:
$$
S _ {t} \left(\widehat {Y} _ {t} ^ {0}, \widehat {Y} _ {t} ^ {1}\right) = 1 + \lambda_ {t} L _ {t} \left(\widehat {Y} _ {t} ^ {0} \omega_ {t} ^ {0} - \widehat {Y} _ {t} ^ {1} \omega_ {t} ^ {1}\right), \text {w h e r e} L _ {t} := \min  _ {b \in \{0, 1 \}} \underset {x \in \mathcal {X}} {\operatorname {e s s}} \inf  _ {\frac {1}{2 \omega_ {t} ^ {b}} (x)}, \tag {7}
$$
and  $\lambda_{t}$  is once again selected via ONS. Multiplication by  $L_{t}$  is required to ensure that the values  $L_{t}(\widehat{Y}_{t}^{0}\omega_{t}^{0} - \widehat{Y}_{t}^{1}\omega_{t}^{1})$  lie in  $[-1,1]$  in order to be compatible with ONS. Here "ess inf" is the essential infimum, which is the infimum over events with nonzero measure. For most practical applications one can simply replace this with a minimum. For such a strategy, we obtain the following guarantee.
Proposition 2. Algorithm 1 with input  $\alpha \in (0,1)$  and betting strategy (7) is a level- $\alpha$  sequential test with asymptotic power one. Moreover, suppose that  $0 < L_{\mathrm{inf}} \coloneqq \inf_{t \geq 1} L_t$  and let  $\kappa = \Delta L_{\mathrm{inf}}$  and  $\Delta = |\mu_0 - \mu_1|$ . Then, treating log-log factors as constant, under the alternative the expected stopping time  $\tau$  obeys
$$
\mathbb {E} [ \tau ] \lesssim \frac {1}{\kappa^ {2}} \log \left(\frac {1}{\kappa^ {2} \alpha}\right). \tag {8}
$$
Thus we see that we pay the price of allowing randomized data collection policies by a factor of roughly  $1 / L_{\mathrm{inf}}$ . Note that the payoff function (7) does not require knowledge of  $L_{\mathrm{inf}}$ . Indeed, this strategy remains valid even if  $L_{\mathrm{inf}} = 0$  as long as  $L_{t}$  is nonzero for each  $t$ . It is only the analysis that requires  $L_{\mathrm{inf}}$  to be finite and known. The proof of Proposition 2 is provided in Appendix C.2.
3.3 Time-varying policies for unknown densities
A reasonable objection to the discussion in the previous section is that the densities  $\rho^b$  may not always be known. In this case we cannot compute the propensity weights in (5). Here we provide an alternative payoff function which uses an estimate of the density,  $\widehat{\rho}^b$ . We assume we know upper and lower bounds on the multiplicative error of our estimate:  $\delta^{\max} \geq \max_b \sup_x \{\widehat{\rho}^b(x) / \rho^b(x)\}$ , and  $\delta^{\min} \leq \min_b \inf_x \{\widehat{\rho}^b(x) / \rho^b(x)\}$ . We assume that  $\delta^{\min} > 0$ .
Let  $\widehat{\omega}_t^b$  be the propensity weights at time  $t$  using the estimated density, i.e.,  $\widehat{\omega}_t^b (x) = \widehat{\rho}^b (x) / \pi_t^b (x)$ . Notice that  $\mathbb{E}_{X\sim \pi_t^b}[\widehat{\omega}_t^b (X)\varphi (X)|\mathcal{F}_{t - 1}] = \int_{\mathcal{X}}\varphi (x)\widehat{\rho}^b (x)\mathrm{d}x = \int_{\mathcal{X}}\varphi (x)(\widehat{\rho}^b (x) / \rho^b (x))\rho^b (x)\mathrm{d}x\leq \delta^{\max}\mu_b$ . Similarly,  $\mathbb{E}_{X\sim \pi_t^b}[\widehat{\omega}^b (X)\varphi (X)|\mathcal{F}_{t - 1},\xi_b]\geq \delta^{\min}\mu_b$ . Recall that  $\widehat{Y}_t^b = \varphi (X_t^b)$ . Consider the payoff function
$$
S _ {t} \left(\widehat {Y} _ {t} ^ {0}, \widehat {Y} _ {t} ^ {1}\right) = 1 + \lambda_ {t} B _ {t} \left(\frac {\widehat {\omega} _ {t} ^ {0} \widehat {Y} _ {t} ^ {0}}{\delta^ {\operatorname* {m a x}}} - \frac {\widehat {\omega} _ {t} ^ {1} \widehat {Y} _ {t} ^ {1}}{\delta^ {\operatorname* {m i n}}}\right), \tag {9}
$$
where  $B_{t}$  is  $\mathcal{F}_{t - 1}$  measurable. Then
$$
\mathbb {E} [ S _ {t} | \mathcal {F} _ {t - 1} ] = 1 + \lambda_ {t} B _ {t} \bigg (\frac {\mathbb {E} _ {\pi_ {t} ^ {0}} [ \widehat {\omega} _ {t} ^ {0} \widehat {Y} _ {t} ^ {0} | \mathcal {F} _ {t - 1} ]}{\delta^ {\mathrm {m a x}}} - \frac {\mathbb {E} _ {\pi_ {t} ^ {1}} [ \widehat {\omega} _ {t} ^ {1} \widehat {Y} _ {t} ^ {1} | \mathcal {F} _ {t - 1} ]}{\delta^ {\mathrm {m i n}}} \bigg) \leq 1 + \lambda_ {t} B _ {t} (\mu_ {0} - \mu_ {1}).
$$
Figure 2: Two illustrations of our sequential test adapting to distribution shift. In both settings, the observations at time  $t$  are Bernoulli with bias determined by the respective mean at that time. Shaded areas in the bottom plots represent the standard deviation across 100 trials. Left: For the first 100 time steps, we have  $\mu_0(t) = \mu_1(t) = 0.3$ . At time 100,  $\mu_1(t)$  begins to smoothly slope upward. Right: Here we assume that both means are non-stationary and non-smooth. Both are sinusoidal with Gaussian noise but  $\mu_1(t)$  drifts slowly upwards over time.
Under the null,  $\mu_0 - \mu_1 = 0$ , implying that  $\mathbb{E}[S_t|\mathcal{F}_{t - 1}]\leq 1$ . We select  $B_{t}$  so that  $S_{t}$  is nonnegative (thus ensuring that the capital process is a nonnegative supermartingale) and is compatible with ONS. The following condition on  $B_{t}$  suffices:  $B_{t}\leq \delta^{\mathrm{min}}\min_{b}\inf_{x}(2\widehat{\omega}_{t}^{b}(x))^{-1}$ . Note that if we know the true densities then we can take  $\delta^{\mathrm{max}} = \delta^{\mathrm{min}} = 1$  and we recover (7).
3.4 Handling distribution shift
Until this point we've assumed that the means  $\mu_{b}$  remain constant over time. A deployed model, however, is susceptible to distribution shift due to changes in the underlying population, changes in the model (retraining, active learning, etc), or both. Here we demonstrate that our framework handles both kinds of drift. In fact, we need not modify our strategy (we still deploy Algorithm 1), but we must reformulate our null and alternative hypotheses and our analysis will change. Before introducing the mathematical formalities, we draw the reader's attention to Figure 2 which illustrates two cases of distribution shift and the response of our sequential test to each. The left panel is a case of "smooth" drift in which case  $\mu_0(t) = \mu_1(t)$  are equal for some number of timesteps after which  $\mu_{1}(t)$  begins to drift upward. The right panel exemplifies a situation in which both means change each timestep (the weekly prevalence of disease in a population, for instance), but  $\mu_{1}(t)$  has a marked drift upward over time. In both cases, the wealth process is sensitive to the drift and grows over time.
To handle changes in the underlying population we let  $\rho_{t}$  denote the population distribution over  $\mathcal{X}$  at time  $t$ . Likewise, to handle changes to the model, we let  $\varphi_t:\mathcal{X}\to [0,1]$  be the model at time  $t$ . In order to define our hypotheses, let  $\Delta_t\coloneqq |\mu_0(t) - \mu_1(t)|$  where  $\mu_b(t) = \mathbb{E}_{X\sim \rho_t}[\varphi_t(X)|\xi_b,\mathcal{F}_{t - 1}], b\in \{0,1\}$ . Then, we write the null and alternative hypothesis as
$$
H _ {0}: \Delta_ {t} = 0 \forall t \geq 1, \quad H _ {1}: \exists T \in \mathbb {N} \text {s . t .} \Delta_ {t} > 0 \forall t \geq T. \tag {10}
$$
Of course,  $H_0$  and  $H_1$  do not cover the space of possibilities. In particular, neither contains the event that  $\mu_1(t) \neq \mu_0(t)$  for some finite interval  $t \in [a,b]$  and are otherwise equal. However, we believe (10) strikes a desirable balance between analytical tractability and practical utility; defining  $H_1$  such that we will reject the null for each finite window  $[a,b]$  places too extreme a burden on the growth of the wealth process. Moreover, as illustrated by Figure 2, if  $\Delta_t > 0$  for a large enough window, then our sequential tests will reject the null.
If we are working with randomized data collection policies per Section 3.2, then one should change the definition of the weights in Equation (5) to incorporate the time-varying distribution  $\rho_{t}$ . That is,  $\omega_{t}^{b}(x) = \rho_{t}^{b}(x) / \pi_{t}^{b}(x)$ . Otherwise, all procedures detailed in the previous sections remain the same, and we can provide the following guarantee on their performance.
Proposition 3. Algorithm 1 with input  $\alpha \in (0,1)$  and betting strategies (2) and (7) is a level- $\alpha$  sequential test for problem (10). It has power one under the alternative if  $\Delta_{\mathrm{inf}} := \inf_{t\geq n}\Delta_t > 0$  where  $n$  is some time at which drift begins. Moreover, under the alternative the expected stopping
Permutation test (M1)
Permutation test (M2)
Figure 3: Comparisons of false positives rates (FPRs) and stopping times on credit loan data and US census data. The left two columns plot  $\tau$  under  $H_{1}$  versus the FPR as  $\alpha$  is varied from 0.1 to 0.01. The FPR is grossly inflated when under method M1, as illustrated the first and third columns. Betting is a Pareto improvement over the permutation tests.
Credit Default Dataset
$\times$  Betting
time  $\tau$  obeys
$$
\mathbb {E} [ \tau ] \lesssim n + \frac {1}{\Delta_ {\inf } ^ {2}} \log \left(\frac {1}{\Delta_ {\inf } ^ {2} \alpha}\right). \tag {11}
$$
Furthermore, if the data are gathered by randomized policies  $\pi_t^b$  and  $L_{\mathrm{inf}} > 0$  for  $L_{\mathrm{inf}}$  as in Proposition 2, then the expected stopping time follows by replacing  $\Delta_{\mathrm{inf}}$  in (14) with  $\Delta_{\mathrm{inf}}L_{\mathrm{inf}}$ .
Let us make two remarks about this result. First, the condition  $\Delta_{\mathrm{inf}} > 0$  ensures that, after some point, the means  $\mu_0(t)$  and  $\mu_1(t)$  remain separated. If they diverge only to later reconverge, there is no guarantee that the test will reject (though, in practice, it will if they are separated for sufficiently long). Second, the reader may have guessed that the expected stopping time under drift beginning at time  $n$  follows from Propositions 1 and 2 after simply adding  $n$ . However, it is a priori feasible that the reliance of ONS on past predictions would result in slower convergence under distribution shift. Proposition 3 verifies that this is not the case, and that the rate is the same up to constants.
3.5 Composite Nulls
In practice, it may be unreasonable to require that the means between two groups are precisely zero. Instead, we may only be interested in detecting differences greater than  $\epsilon$  for some  $0 < \epsilon \ll 1$ . In this case we may write the null and alternative as
$$
H _ {0}: \left| \mu_ {0} - \mu_ {1} \right| \leq \epsilon \text {v s} H _ {1}: \left| \mu_ {0} - \mu_ {1} \right| > \epsilon . \tag {12}
$$
To formulate our test, we introduce two pairs of auxiliary hypotheses:  $H_0': \mu_0 - \mu_1 \leq \epsilon$  vs  $H_1': \mu_0 - \mu_1 > \epsilon$  and  $H_0'': \mu_1 - \mu_0 \leq \epsilon$  vs  $H_1'': \mu_1 - \mu_0 > \epsilon$ . Observe that if either  $H_0'$  or  $H_0''$  is false, then the alternative  $H_1$  is true. Our approach will therefore entail testing both  $H_0'$  vs  $H_1'$  and  $H_0''$  vs  $H_1''$ . Let  $\phi_t^Q$  denote the sequential test for the former, and  $\phi_t^R$  for the latter. The test given by  $\phi_t = \max\{\phi_t^Q, \phi_t^R\}$  (i.e., rejecting  $H_0$  if either  $H_0'$  or  $H_0''$  is rejected) is then a test for (12). Game-theoretically, this can be interpreted as splitting our initial capital in half and playing two games simultaneously. To test  $H_0'$  and  $H_0''$  consider the two payoff functions
$$
Q _ {t} \left(\widehat {Y} _ {t} ^ {0}, \widehat {Y} _ {t} ^ {1}\right) = 1 + \lambda_ {t} \left(\widehat {Y} _ {t} ^ {0} - \widehat {Y} _ {t} ^ {1} - \epsilon\right), \text {a n d} R _ {t} \left(\widehat {Y} _ {t} ^ {0}, \widehat {Y} _ {t} ^ {1}\right) = 1 + \lambda_ {t} \left(\widehat {Y} _ {t} ^ {1} - \widehat {Y} _ {t} ^ {0} - \epsilon\right). \tag {13}
$$
If  $\lambda_t \in [\frac{-1}{1 - \epsilon}, \frac{1}{1 + \epsilon}]$  then both  $Q_t$  and  $R_t$  are nonnegative. We thus select  $\lambda_t$  via ONS as usual. The wealth processes  $(\mathcal{K}_t^Q)$  and  $(\mathcal{K}_t^R)$  defined by  $Q_t$  and  $R_t$  (i.e.,  $\mathcal{K}_t^Q = \prod_{i \leq t} Q_i (\widehat{Y}_i^0, \widehat{Y}_i^1)$  and  $\mathcal{K}_t^R = \prod_{i \leq t} R_i (\widehat{Y}_i^0, \widehat{Y}_i^1)$ ) are then nonnegative supermartingales under  $H_0'$  and  $H_0''$ , respectively. We reject  $H_0$  if either  $\mathcal{K}_t^R \geq 2/\alpha$  or  $\mathcal{K}_t^Q \geq 2/\alpha$  which results in an overall level- $\alpha$  sequential test.
Proposition 4. Algorithm 1 with input  $\alpha /2\in (0,1)$  with betting strategy  $Q_{t}$  (resp.,  $R_{t}$ ) is a level-  $\alpha /2$  sequential test for  $H_0^{\prime}$  vs  $H_1^{\prime}$  (resp.,  $H_0''$  vs  $H_1''$ ). Rejecting  $H_0$  if either  $H_0^{\prime}$  or  $H_0''$  is rejected results in a level-  $\alpha$  sequential test with asymptotic power one for problem (12). Let  $\Delta = |\mu_0 - \mu_1|$ . Then, under the alternative the expected stopping time  $\tau$  obeys
$$
\mathbb {E} [ \tau ] \lesssim \frac {1}{(\Delta - \epsilon) ^ {2}} \log \left(\frac {1}{\alpha (\Delta - \epsilon) ^ {2}}\right). \tag {14}
$$
Distribution Shift, Census Data
Figure 4: Response of the tests to distribution shift on the census data. We use a fair model for 400 timesteps, after which we switch to an unfair model with  $\Delta \approx 1.0$ . The leftmost plot uses permutation tests under M1, resulting in inflated type-I error. Values are plotted as  $\alpha$  ranges from 0.01 to 0.1.
We note that under the alternative,  $\epsilon < \Delta$ , so  $0 < \Delta - \epsilon < \Delta$ . Proposition 4 states that the expected stopping time depends on the interplay of  $\Delta$  and  $\epsilon$ , and will increase as they approach each other. This is intuitive. For a fixed  $\Delta$  one would expect that the problem becomes harder as  $\epsilon \uparrow \Delta$ .
Here we provide several sets of experiments to demonstrate the benefits of our sequential test compared to fixed-time tests. First we need to consider how fixed-time tests might be applied in practice to deal with sequential settings. We consider two such methods.
M1. For some prespecified  $k \in \mathbb{N}$ , wait until we have collected  $k$  audits, then perform the test. If the test does not reject, collect another  $k$  audits and repeat. We emphasize that if one does not adjust the significance level over time, then this is not a valid level-  $\alpha$  test. However, it may be used unwittingly in practice, so we study it as one baseline.
M2. We batch and test in the same way as above, but we apply a Bonferroni-like correction in order to ensure it is level  $\alpha$ . More precisely, for the  $j$ th-batch,  $j \geq 1$ , we set the significance level to  $\alpha / 2^j$ . The union bound then ensures that the FPR is at most  $\alpha$  over all batches.
We run experiments on three real world datasets: a credit default dataset [50], US census data [51], and health insurance data [52]. All can be made to violate equality of opportunity if trained with naive models; we provide the details in Appendix D. Suffice it to say here that the absolute difference in means,  $\Delta$ , is  $\approx 0.03$  for the credit default dataset,  $\approx 1.1$  for the census data, and  $\approx 0.06$  for the insurance data. We employ various distinct models to test our methods, from forest-based models to logistic regression. We use a permutation test as our baseline fixed-time test because of its ubiquity in practice, exact type-I error control under exchangeability, and minimax optimality in various scenarios [53].
Figure 3 compares the results of Algorithm 1 (betting strategy (2)) with the permutation test baselines M1 and M2. The left two columns plot the empirical false positive rate (FPR) against the stopping time under the alternative. Values concentrated in the lower left corner are therefore preferable. We plot values for each test as  $\alpha$  is varied from 0.01 to 0.1. The betting test typically Pareto-dominates the (stopping time, FPR) values achievable by the baselines. In a small number of cases, M1 (baseline without Bonferroni correction) achieves a faster stopping time; however, this is only possible with a very high FPR ( $>0.5$ ). This inflated FPR is verified by the final column, which shows that M1 does not respect the desired type-I error rate. This encourages the use of the Bonferroni-style correction of M2. However, doing so results in an overly conservative test which is slower to reject than betting, as seen in the center column of Figure 3. Betting, meanwhile, respects the desired FPR over all  $\alpha$ , and almost always has the fastest stopping time at any given FPR. We note that while it may appear that the permutation tests improve as  $k$  gets larger, this trend does not hold for large  $k$ . Indeed,  $k$  is
2We refrain from comparing our betting-style sequential tests to other sequential tests; we refer the reader to [28] for such comparisons. Instead, our goal in this section is to persuade the fairness community that sequential tests are superior tools to fixed-time tests for auditing deployed systems.
Various Data Collection Policies, Insurance Dataset
Figure 5: Illustration of our betting method when using various data collection strategies,  $\pi_1$ ,  $\pi_2$ , and  $\pi_3$ , which were based on each individual's region (NE, NW, SE, SW). Operationally,  $\pi_i$  samples an individual by first sampling a region with the given probability, and then sampling an individual uniformly at random from that region. We compare results against our method with data sampled uniformly from the population (red crosses), and to permutation tests (M2), also with uniformly sampled data. Even with randomized policies, we continue to outperform permutation tests.
also the minimum rejection time for the permutation test. Increasing  $k$  to 1,500 on the credit default dataset, for instance, would result in vertical line at  $\tau = 1$ , 500.
The robustness of betting to distribution shift is illustrated by Figure 4. Here we use a fair model for the first 400 timesteps, after which we switch to a random forest classifier which is unfair ( $\Delta \approx 1.1$ ). The rejection time for all tests suffers accordingly, but betting remains a Pareto improvement over the permutation tests using M2. As before, using M1 results in an enormous FPR. Interestingly, we see that M2 also results in FPR higher than 0.1 (the maximum value of  $\alpha$ ) a nontrivial percentage of the time. Indeed, under distribution shift, the permutation test using M2 is no longer level- $\alpha$  since the data are not exchangeable. Betting, on the other hand, maintains its coverage guarantee.
Finally, Figure 5 illustrates the performance of various algorithms when audits are not conducted on predictions received uniformly at random from the population (discussed in Section 3.2). Instead, predictions were received based on an individual's region. We test three distinct strategies,  $\pi_1$ ,  $\pi_2$  and  $\pi_3$ , each of which deviates from the population distribution to varying degrees (LHS of Figure 5). As expected, the rejection time of our strategy suffers compared to when the predictions are received uniformly from the population (denoted by the red crosses). The desired level- $\alpha$  guarantee is still met, however, and betting continues to outperform permutation tests in all cases.
5 Summary
We have argued that practitioners in the field of algorithmic fairness should consider adopting sequential hypothesis tests in lieu of fixed-time tests. The former enjoy two desirable properties absent in the latter: (i) the ability to continually monitor incoming data, and (ii) the ability to reject at data-dependent stopping times. Both (i) and (ii) are useful for high-stakes and/or time-sensitive applications in which fixing the budget beforehand and waiting for more data to arrive may be unsatisfactory, such as auditing the fairness of healthcare models [54] or resource allocation strategies during a public health crisis [55, 56]. We provided a sequential test with asymptotic power one for group fairness (Algorithm 1) inspired by the paradigm of game-theoretic probability and statistics [25, 21, 28]. It is fast, easy to implement, and can handle time-varying data collection policies, distribution shift, and composite nulls, all of which are common in practice. We also provided bounds on the expected stopping time of the test. We hope that the simplicity of our methods combined with their strong theoretical guarantees prove useful for practitioners in the field.
Limitations and societal impact. Auditing is a complex socio-technical problem which involves a range of questions: choosing appropriate audit targets, gaining access to data, ensuring the credibility and independence of external auditors, creating accountability based on audit results, and more [57]. Indeed, poorly designed audits may certify that a system is "fair" while masking harder to verify issues such as data provenance. Our work addresses only one portion of the overall auditing task: providing statistically sound methods for the sequential testing of a specific criteria. This is complementary to, and does not substitute for, careful overall design of an auditing framework.
Acknowledgements. We thank Jing Yu Koh and Shubhanshu Shekhar for helpful conversations. We also thank the anonymous referees for helpful feedback which improved the paper. BC and AR acknowledge support from NSF grants IIS-2229881 and DMS-2310718. BC was supported in part by the NSERC PGS D program, grant no. 567944. BW and SCG were supported in part by the AI2050 program at Schmidt Futures, grant no. G-22-64474.