Title: Auditing Fairness by Betting
Venue: NeurIPS 2023
Accepted: yes
Keywords: fairness

auditing

sequential analysis

martingales

testing by betting
Abstract: We provide practical, efficient, and nonparametric methods for auditing the fairness of deployed classification and regression models. Whereas previous work relies on a fixed-sample size, our methods are sequential and allow for the continuous monitoring of incoming data, making them highly amenable to tracking the fairness of real-world systems. We also allow the data to be collected by a  probabilistic policy as opposed to sampled uniformly from the population. This enables auditing to be conducted on data gathered for another purpose. Moreover, this policy may change over time and different policies may be used on different subpopulations. Finally, our methods can handle distribution shift resulting from either changes to the model or changes in the underlying population. Our approach is based on recent progress in anytime-valid inference and game-theoretic statistics---the ``testing by betting'' framework in particular. These connections ensure that our methods are interpretable, fast, and easy to implement. We demonstrate the efficacy of our approach on three benchmark fairness datasets.

Auditing Fairness by Betting

Abstract

We provide practical, efficient, and nonparametric methods for auditing the fairness of deployed classification and regression models. Whereas previous work relies on a fixed-sample size, our methods are sequential and allow for the continuous monitoring of incoming data, making them highly amenable to tracking the fairness of real-world systems. We also allow the data to be collected by a probabilistic policy as opposed to sampled uniformly from the population. This enables auditing to be conducted on data gathered for another purpose. Moreover, this policy may change over time and different policies may be used on different subpopulations. Finally, our methods can handle distribution shift resulting from either changes to the model or changes in the underlying population. Our approach is based on recent progress in anytime-valid inference and game-theoretic statistics-the 'testing by betting' framework in particular. These connections ensure that our methods are interpretable, fast, and easy to implement. We demonstrate the efficacy of our approach on three benchmark fairness datasets.

1 Introduction

As algorithmic decision-making continues to increase in prevalence across both the private and public sectors [1, 2], there has been an increasing push to scrutinize the fairness of these systems. This has lead to an explosion of interest in so-called 'algorithmic fairness', and a significant body of work has focused on both defining fairness and training models in fair ways (e.g., [3-5]). However, preventing and redressing harms in real systems also requires the ability to audit models in order to assess their impact; such algorithmic audits are an increasing area of focus for researchers and practitioners [6-9]. Auditing may begin during model development [10], but as model behavior often changes over time throughout real-world deployment in response to distribution shift or model updates [11, 12], it is often necessary to repeatedly audit the performance of algorithmic systems over time [8, 13]. Detecting whether deployed models continue to meet various fairness criteria is of paramount importance to deciding whether an automated decision-making system continues to act reliably and whether intervention is necessary.

In this work, we consider the perspective of an auditor or auditing agency tasked with determining if a model deployed 'in the wild' is fair or not. Data concerning the system's decisions are gathered over time (perhaps with the purpose of testing fairness but perhaps for another purpose) and our goal is to determine if there is sufficient evidence to conclude that the system is unfair. If a system is in fact unfair, we want to determine so as early as possible, both in order to avert harms to users and because auditing may require expensive investment to collect or label samples [8, 13]. Following the recent work of Taskesen et al. [14] and Si et al. [15], a natural statistical framework for thinking about this problem is hypothesis testing. Informally, consider the null and alternative hypotheses, H 0 and H 1 , defined respectively as

H 0 : the model is fair , H 1 : the model is unfair .

Unfortunately, traditional hypothesis testing requires stringent assumptions on the data; a fixed number of iid data points, for example. Such assumptions are unrealistic in our setting. We should, for instance, be able to continually test a system as we receive more information, i.e., perform sequential hypothesis testing. Moreover, we would like to be able stop collecting additional samples at arbitrary data-dependent stopping times if we have sufficient evidence against the null. This is not allowed in traditional statistical frameworks, where it is known as 'peeking' or 'p-hacking.'

To overcome these challenges, we take advantage of recent progress in safe, anytime-valid inference (SAVI) to construct sequential procedures for determining whether an existing decision-making system is fair. SAVI is part of sequential analysis , a branch of statistics concerned-as the name suggests-with analyzing data sequentially while maintaining statistical validity. This subfield traces its origins back to Wald, Lai, Robbins, and several others beginning in the 1940s [16-20]. More recently, the power of sequential analysis to enable inference under continuous monitoring of data and data-dependent stopping rules (hence safe and anytime-valid ) has led to an explosion of work in the area [21-23]. A further exciting development has been the connection between such methods and so-called 'game-theoretic probability' [24, 25] which is consistent with modern measure theoretic probability 1 but provides an alternative foundation based on repeated games. Importantly for our purposes, this renders many of the tools of SAVI interpretable as well as statistically powerful. We refer the interested reader to the recent survey by Ramdas et al. [27] for further detail on the relationship between sequential analysis, SA VI, and game-theoretic statistics.

Contributions. We develop tools to sequentially audit both classifiers and regressors. In particular:

We formulate the problem of auditing the fairness of classification and regression models in terms of sequential hypothesis testing. The focus on sequential testing is distinct from other work, and highlights various desiderata that are important in practice-specifically, (i) being able to continuously monitor the data and (ii) a focus on rejecting the null as early as possible (i.e., reducing the number of costly samples needed to detect unfairness).

Next, we design nonparametric sequential hypothesis tests which hold under various definitions of group fairness. We treat auditing as sequential two-sample testing and adapt the recent work of Shekhar and Ramdas [28] on two-sample testing by betting to the fairness setting. We also provide novel bounds on the expected stopping time of our tests under the alternative, and demonstrate how to handle distribution drift (due to either model changes or changes in the underlying population), time-varying data collection policies, and composite nulls which more accurately reflect practical demands.

Finally, we demonstrate the real world applicability of our methods on three datasets: credit default data, US census data, and insurance data. We show that our method is robust to distribution shift resulting from model retraining and to various randomized data-collection policies whose densities deviate significantly from the underlying population. All code is publicly available at https://github.com/bchugg/auditing-fairness .

In order to provide a preview of our methodology, we suggest the following thought experiment. Imagine a fictitious better who is skeptical that a machine learning system is fair. She sets up an iterated game wherein she bets on the results of the audit as it is conducted. Her bets are structured such that, if the system is unfair, her expected payoff will be large. Conversely, if the system is fair, her expected payoff is small. Thus, if her wealth increases over time, it is evidence that the system is unfair. Our sequential test thus rejects the null if her wealth crosses some predetermined threshold. Of course, the trick is to design her bets in such a way the above conditions are satisfied, and that under the alternative, her wealth grows as quickly as possible. Mathematically, the above is made rigorous via nonnegative (super)martingales and results concerning their behavior over time [29, 30].

Additional Related Work. This work sits at the intersection of several fields. On the fairness side, the closest work to ours is that of Taskesen et al. [14] and Si et al. [15], both of which study fairness through the lens of (fixed-time) hypothesis testing, based on a batch of n iid observations. Given iid data Z t ∼ ρ , they formulate the null as H 0 : ρ ∈ G , where G is the set of all 'fair' distributions and derive a test statistic based on projecting the empirical distribution onto G (minimizing a Wasserstein distance). They work with classifiers only. On the more technical side, we view our work as

a continuation and application of the testing by betting framework [21] and more specifically of nonparametric two sample testing by betting [28]. We will deploy similar 'betting strategies' (to be defined later) as [28], but provide novel analyses which are particular to our setting, and consider several extensions. Relatedly, but slightly further removed from our setting, Duan et al. [31] also employ game-theoretic ideas to construct interactive rank tests. The idea of a auditing a system to verify its veracity extends well beyond the domain of algorithmic fairness. Statistical procedures have been developed, for instance, to audit election results [32] and recently several SA VI inspired methods have been developed for several such scenarios [33, 34]. More broadly and outside the scope of this paper, betting techniques have been deployed with great success in optimization, online learning, and statistics [35-38, 26].

2 Preliminaries

Consider a feature space X and model φ : X → [0 , 1] . Depending on the application, φ ( x ) might be a risk score for an individual with covariates x , a classification decision, or the probability that some action is taken. Each x ∈ X is associated with some 'sensitive attribute' A = A x ∈ { 0 , 1 , . . . , J } (indicating, e.g., the result of a private health test, whether they went to college, their income strata, etc). We will often refer to A as group membership. The classifier may or may not observe A . For the sake of exposition, we will assume that A ∈ { 0 , 1 } , i.e, there are two groups. However, our methods extend straightforwardly to more than two groups. The details are provided in Appendix B. Before formally stating the problem, let us discuss the notion of fairness we will employ in this work.

Fairness. We focus on the concept of 'group' fairness. Roughly speaking, this involves ensuring that groups of individuals sharing various attributes are treated similarly. There have been multiple notions of group fairness proposed in the literature. We introduce the following generalized notion of group fairness, which can be instantiated to recapture various others.

Definition 1. Let { ξ j ( A,X,Y ) } j ∈{ 0 ,...,J } denote a family of conditions on the attributes A , covariates X , and outcomes Y . We say a predictive model φ : X → [0 , 1] is fair with respect to { ξ j } and a distribution ρ over X if, for all i, j ∈ [ J ] , E X ∼ ρ [ φ ( X ) | ξ i ( A,X,Y )] = E X ∼ ρ [ φ ( X ) | ξ j ( A,X,Y )] .

̸

As was mentioned above, unless otherwise stated we will assume that there are only two conditions ξ 0 , ξ 1 . For our purposes, the important feature of Definition 1 is that it posits the equality of means. Indeed, letting µ b = E X ∼ ρ [ φ ( X ) | ξ b ] for b ∈ { 0 , 1 } we can write our null as H 0 : µ 0 = µ 1 and alternative as H 1 : µ 0 = µ 1 . Different choices of conditions ξ j lead to various fairness notions in the literature. For instance, if φ is a classification model, then:

Taking ξ 0 = { A = 0 , Y = 1 } and ξ 1 = { A = 1 , Y = 1 } results in equality of opportunity [39]. Predictive equality [40] is similar, corresponding to switching Y = 1 with Y = 0 .

Taking ξ 0 = { A = 0 } , ξ 1 = { A = 1 } results in statistical parity [41].

Taking ξ j = { A = j, ℓ ( X ) } for some projection mapping ℓ : X → F ⊂ X onto 'legimitate factors' results in conditional statistical parity [42, 41].

Problem Formulation. We consider an auditor who is receiving two streams of predictions Z 0 = ( φ ( X 0 t )) t ∈ T 0 and Z 1 = ( φ ( X 1 t )) t ∈ T 1 , where X b t obeys condition ξ b (i.e., is drawn from some distribution over X| ξ b ). For brevity, we will let ̂ Y b t = φ ( X b t ) , b ∈ { 0 , 1 } . The index sets T 0 , T 1 ⊂ N ∪ {∞} denote the times at which the predictions are received. We let the index differ between groups as it may not be feasible to receive a prediction from each group each timestep. We refer to this entire process as an audit of the model φ .

For b ∈ { 0 , 1 } , let T b [ t ] = T b ∩ [ t ] be the set of times at which we receive predictions from group b up until time t . The auditor is tasked with constructing a sequential hypothesis test ϕ ≡ ( ϕ t ) t ≥ 1 where ϕ t = ϕ t (( ∪ t ∈ T 0 [ t ] Z 0 t ) ∪ ( ∪ t ∈ T 1 [ t ] Z 1 t )) ∈ { 0 , 1 } is a function of all predictions received until time t . We interpret ϕ t = 1 as 'reject H 0 ', and ϕ t = 0 as 'fail to reject H 0 .' Once we reject the null, we stop gathering data. That is, our stopping time is τ = arg inf t { ϕ t = 1 } . We say that ϕ is a level -α sequential test if

In words, the test should have small false positive rate (type I error) simultaneously across all time steps . (Note that a fixed-time levelα test would simply drop the quantifier ∃ t ≥ 1 and concern

itself with some fixed time t = n .) We also wish to design tests with high power. Formally, we say that ϕ has asymptotic power 1 -β if sup P ∈ H 1 P ( ∀ t ≥ 1 : ϕ t = 0) ≤ β , or equivalently sup P ∈ H 1 P ( τ = ∞ ) ≤ β . That is, in all worlds in which the alternative is true, we fail to reject with probability at most β (type II error). Typically the type-II error β ≡ β n decreases with the sample size n . In this work we will develop asymptotic power one tests, meaning that β n → 0 as n →∞ .

Martingales and filtrations. Our techniques rely on the machinery of nonnegative (super)martingales and results concerning their behavior over time. We introduce some of the technicalities here. A (forward) filtration F ≡ ( F t ) t ≥ 0 is an increasing sequence of σ -fields F t ⊂ F t +1 . Throughout this paper, we will consider the 'canonical filtration' F t = σ ( Z 1 , . . . , Z t ) which can heuristically (but very usefully) be thought of as all the information known at time t . We say that a stochastic process S = ( S t ) t ≥ 1 is adapted to ( F t ) if S t is F t measurable for all t ≥ 1 , and predictable if S t is F t -1 measurable for all t ≥ 1 . A P -martingale is an adapted stochastic process M = ( M t ) t ≥ 1 such that E P [ M t +1 |F t ] = M t for all t ≥ 1 . If the equality is replaced with ≤ , then M is a P -supermartingale . A particularly useful result in sequential analysis is Ville's inequality [29], which states that if M is a nonnegative P -supermartingale, then for all α > 0 , P ( ∃ t ≥ 0 : M t ≥ 1 /α ) ≤ α E P [ M 0 ] . We will also employ the following randomized improvement to Ville's inequality [43, Corollary 4.1.1]: For M as above and any F -stopping time τ , P ( ∃ t ≤ τ : M t ≥ 1 /α or M τ ≥ U/α ) ≤ α , where U is a uniform random variable on [0 , 1] which is independent of M and τ .

3 Methods

We begin by introducing the methodology in the setting where the predictions are received uniformly at random from the population. We will then progressively generalize the setting: Sections 3.2 and 3.3 will enable time-varying data collection policies, Section 3.4 will allow the means µ 0 and µ 1 to change with time, and Section 3.5 will consider composite nulls of the form | µ 0 -µ 1 | ≤ ϵ .

To ease the presentation, let us make the assumption that we receive an audit from both groups each timestep, so that T 0 = T 1 = N . This is without loss of generality; one can simply wait until multiple audits from each group are available. However, Appendix A provides a more detailed discussion on how to modify our framework and theorems if this condition is unmet.

3.1 Testing by betting

Returning to the intuition for a moment, recall the fictitious bettor discussed in the introduction. She attempts to prove that the system is unfair by betting on the results of the audits before they are revealed. If the system is unfair, then the average difference between ̂ Y 0 t and ̂ Y 1 t (i.e., the model's outputs across different groups) will be non-zero. The skeptic's bets are therefore structured such that if µ 0 = E ̂ Y 0 t = E ̂ Y 1 t = µ 1 , then her expected wealth will increase over time. More formally, at time t , the skeptic designs a payoff function S t : R × R → R ≥ 0 which is F t -1 measurable and E P [ S t ( ̂ Y 0 t , ̂ Y 1 t ) |F t -1 ] ≤ 1 if P ∈ H 0 (i.e, µ 0 = µ 1 ). Next, the skeptic receives the model predictions ̂ Y 0 t = φ ( X 0 t ) and ̂ Y 1 t = φ ( X 1 t ) . We assume the skeptic starts with wealth of K 0 = 1 . At each time t , she reinvests all her wealth K t -1 on the outcome and her payoff is K t = S t ( ̂ Y 0 t , ̂ Y 1 t ) · K t -1 = ∏ t i =1 S t ( ̂ Y 0 t , ̂ Y 1 t ) . We call ( K t ) t ≥ 0 the skeptic's wealth process . The wealth process is a supermartingale starting at 1 under the null, due to the constraint on the payoff function. Thus, by Ville's inequality, the probability that K t ever exceeds 1 /α is at most α when the model is fair. The skeptic's goal, as it were, is to design payoff functions such that the wealth process grows quickly under the alternative, and thus rejection occurs sooner rather than later.

̸

Inspired by a common idea in game-theoretic statistics (cf. [21, 26-28]), let us consider the following payoff function:

where λ t is predictable and lies in [ -1 , 1] to ensure that S t ( ̂ Y 0 t , ̂ Y 1 t ) ≥ 0 . Note that for P ∈ H 0 , E P [ K t |F t -1 ] = K t -1 E P [1+ λ t ( ̂ Y 0 t -̂ Y 1 t ) |F t -1 ] = K t -1 , so ( K t ) t ≥ 1 is a nonnegative P -martingale. Rejecting when K t > 1 /α thus gives to a valid levelα sequential test as described above. We will select λ t using Online Newton Step (ONS) [44, 45], which ensures exponential growth of the wealth process under the alternative. Figure 1 illustrates the behavior of the wealth process under various hypotheses when using ONS to choose λ t . As the difference between the means ∆ = | µ 0 -µ 1 |

Algorithm 1

Growthofwealthprocess

150

100

Wealth

50

0

0

50

100

150

200

250

300

Time

=0.5

△ =0.1

= ==

1/α

=0.3

△=0

increases, the wealth grows more quickly which leads to faster rejection of the null. To define ONS, let g t = ̂ Y 0 t -̂ Y 1 t and initialize λ 1 = 0 . For all t ≥ 1 , recursively define

The machinery just introduced is sufficient to define a levelα sequential test. However, we add a final ingredient to increase the power of this procedure. Motivated by the randomized Ville's inequality mentioned in Section 2, we introduce one final step: If we stop the procedure at some stopping time τ but have not yet rejected the null (because, for instance, our budget ran out), we can check if K τ ≥ U/α , where U is uniform on [0,1] and drawn independently from everything observed so far . We emphasize that the final step can only be performed once. Thus, if it is possible that more data will be collected in the future, we advise waiting. We summarize the process in Algorithm 1.

The following proposition gives a bound on the expected stopping time of this sequential test under the alternative. We note that Shekhar and Ramdas [28] also provide a bound on the expected stopping time of a multivariate betting-style sequential test. However, due to the generality of their setting, their result is defined in terms of quantities which are difficult to analyze. We thus provide a more direct analysis specific to the difference of univariate means. The proof may be found in Appendix C.1.

Proposition 1. Algorithm 1 with input α ∈ (0 , 1) and betting strategy (2) is a levelα sequential test with asymptotic power one. Moreover, letting ∆ = | µ 0 -µ 1 | , under the alternative the expected stopping time τ obeys

It is possible to demonstrate a lower bound of E [ τ ] ≳ σ 2 log(1 /α ) / ∆ 2 where σ 2 = E [( ̂ Y 0 -̂ Y 1 ) 2 ] [28, Prop. 2]. Since σ 2 ≳ 1 in the worst case, our result is optimal up to a factor of log(1 / ∆ 2 ) .

3.2 Time-varying data collection policies

Here we extend the setting to allow for time-dependent data collection policies. This is motivated by the fact that organizations are often collecting data for various purposes and must therefore evaluate fairness on data which is not necessarily representative of the population at large. We also allow the data collection policies to differ by group. Mathematically, for b ∈ { 0 , 1 } let π b t ( · ) = π t ( ·| ξ b ) denote the randomized policies which determine the probability with which a set of covariates are

selected from the population. Let ρ b be the density of X | ξ b . Following standard propensity weighting techniques [46, 47], introduce the weighted estimates

We may write simply ω b t when x is understood from context. The policy π b t (and hence the weight ω b t ) need not be deterministic, only F t -1 measurable (so it can be considered a deterministic quantity at time t ). This is motivated by various practical applications of sequential decision-making, in which the policy used to collect data often changes over time in response to either learning or policy [48, 49]. In many applications, π b t is a function of a select few covariates only-for instance, income, education level, or career. In some settings it is reasonable to assume knowledge of ρ b (by means of a census, for instance). However, this is not always the case. Section 3.3 will therefore discuss strategies which do not require precise knowledge of the density. We will assume that for all x, t and b , ω b t ( x ) < ∞ . In this case, (5) enables an unbiased estimate of µ b = E ρ [ φ ( X ) | ξ b ] when X t is sampled according to π b t :

Our payoff function is similar to the previous section, but reweights the samples by ω b t (= ω b t ( X b t )) , and then adds a corrective factor to ensure that the payoff is nonnegative:

and λ t is once again selected via ONS. Multiplication by L t is required to ensure that the values L t ( ̂ Y 0 t ω 0 t -̂ Y 1 t ω 1 t ) lie in [ -1 , 1] in order to be compatible with ONS. Here 'ess inf' is the essential infimum, which is the infimum over events with nonzero measure. For most practical applications one can simply replace this with a minimum. For such a strategy, we obtain the following guarantee.

Proposition 2. Algorithm 1 with input α ∈ (0 , 1) and betting strategy (7) is a levelα sequential test with asymptotic power one. Moreover, suppose that 0 < L inf := inf t ≥ 1 L t and let κ = ∆ L inf and ∆ = | µ 0 -µ 1 | . Then, treating log-log factors as constant, under the alternative the expected stopping time τ obeys

Thus we see that we pay the price of allowing randomized data collection policies by a factor of roughly 1 /L inf . Note that the payoff function (7) does not require knowledge of L inf . Indeed, this strategy remains valid even if L inf = 0 as long as L t is nonzero for each t . It is only the analysis that requires L inf to be finite and known. The proof of Proposition 2 is provided in Appendix C.2.

3.3 Time-varying policies for unknown densities

A reasonable objection to the discussion in the previous section is that the densities ρ b may not always be known. In this case we cannot compute the propensity weights in (5). Here we provide an alternative payoff function which uses an estimate of the density, ̂ ρ b . We assume we know upper and lower bounds on the multiplicative error of our estimate: δ max ≥ max b sup x { ̂ ρ b ( x ) /ρ b ( x ) } , and δ min ≤ min b inf x { ̂ ρ b ( x ) /ρ b ( x ) } . We assume that δ min > 0 .

Let ̂ ω b t be the propensity weights at time t using the estimated density, i.e., ̂ ω b t ( x ) = ̂ ρ b ( x ) /π b t ( x ) . Notice that E X ∼ π b t [ ̂ ω b t ( X ) φ ( X ) |F t -1 ] = ∫ X φ ( x ) ̂ ρ b ( x ) d x = ∫ X φ ( x )( ̂ ρ b ( x ) /ρ b ( x )) ρ b ( x ) d x ≤ δ max µ b . Similarly, E X ∼ π b t [ ̂ ω b ( X ) φ ( X ) |F t -1 , ξ b ] ≥ δ min µ b . Recall that ̂ Y b t = φ ( X b t ) . Consider the payoff function

where B t is F t -1 measurable. Then

1.0

0.75

μo(t)

E[X|b]

0.50

μ1(t)

0.5

0.25

Kt

Wealth

100

100

1/α

0

0

0

100

200

300

400

0

100

200

300

400

Time

Time

Under the null, µ 0 -µ 1 = 0 , implying that E [ S t |F t -1 ] ≤ 1 . We select B t so that S t is nonnegative (thus ensuring that the capital process is a nonnegative supermartingale) and is compatible with ONS. The following condition on B t suffices: B t ≤ δ min min b inf x (2 ̂ ω b t ( x )) -1 . Note that if we know the true densities then we can take δ max = δ min = 1 and we recover (7).

3.4 Handling distribution shift

Until this point we've assumed that the means µ b remain constant over time. A deployed model, however, is susceptible to distribution shift due to changes in the underlying population, changes in the model (retraining, active learning, etc), or both. Here we demonstrate that our framework handles both kinds of drift. In fact, we need not modify our strategy (we still deploy Algorithm 1), but we must reformulate our null and alternative hypotheses and our analysis will change. Before introducing the mathematical formalities, we draw the reader's attention to Figure 2 which illustrates two cases of distribution shift and the response of our sequential test to each. The left panel is a case of 'smooth' drift in which case µ 0 ( t ) = µ 1 ( t ) are equal for some number of timesteps after which µ 1 ( t ) begins to drift upward. The right panel exemplifies a situation in which both means change each timestep (the weekly prevalence of disease in a population, for instance), but µ 1 ( t ) has a marked drift upward over time. In both cases, the wealth process is sensitive to the drift and grows over time.

To handle changes in the underlying population we let ρ t denote the population distribution over X at time t . Likewise, to handle changes to the model, we let φ t : X → [0 , 1] be the model at time t . In order to define our hypotheses, let ∆ t := | µ 0 ( t ) -µ 1 ( t ) | where µ b ( t ) = E X ∼ ρ t [ φ t ( X ) | ξ b , F t -1 ] , b ∈ { 0 , 1 } . Then, we write the null and alternative hypothesis as

̸

Of course, H 0 and H 1 do not cover the space of possibilities. In particular, neither contains the event that µ 1 ( t ) = µ 0 ( t ) for some finite interval t ∈ [ a, b ] and are otherwise equal. However, we believe (10) strikes a desirable balance between analytical tractability and practical utility; defining H 1 such that we will reject the null for each finite window [ a, b ] places too extreme a burden on the growth of the wealth process. Moreover, as illustrated by Figure 2, if ∆ t > 0 for a large enough window, then our sequential tests will reject the null.

If we are working with randomized data collection policies per Section 3.2, then one should change the definition of the weights in Equation (5) to incorporate the time-varying distribution ρ t . That is, ω b t ( x ) = ρ b t ( x ) /π b t ( x ) . Otherwise, all procedures detailed in the previous sections remain the same, and we can provide the following guarantee on their performance.

Proposition 3. Algorithm 1 with input α ∈ (0 , 1) and betting strategies (2) and (7) is a levelα sequential test for problem (10) . It has power one under the alternative if ∆ inf := inf t ≥ n ∆ t > 0 where n is some time at which drift begins. Moreover, under the alternative the expected stopping

Credit Default Dataset

No correction (M1)

With correction (M2)

Type-l error (M1)

0.6

0.6

k=250

Desired FPR

(H

0.06

k=500

(under

0.4

0.4

k=1000

0.04

R

0.2

0.02

0.2

0.0

0.00

0.0

500

1000

1500

1000

2000

0.00

0.05

0.10

Rejection time T (under H1)

Rejection time T (under H1)

Permutation test (M1)

Permutation test (M2)

Betting

time τ obeys

Furthermore, if the data are gathered by randomized policies π b t and L inf > 0 for L inf as in Proposition 2, then the expected stopping time follows by replacing ∆ inf in (14) with ∆ inf L inf .

Let us make two remarks about this result. First, the condition ∆ inf > 0 ensures that, after some point, the means µ 0 ( t ) and µ 1 ( t ) remain separated. If they diverge only to later reconverge, there is no guarantee that the test will reject (though, in practice, it will if they are separated for sufficiently long). Second, the reader may have guessed that the expected stopping time under drift beginning at time n follows from Propositions 1 and 2 after simply adding n . However, it is a priori feasible that the reliance of ONS on past predictions would result in slower convergence under distribution shift. Proposition 3 verifies that this is not the case, and that the rate is the same up to constants.

3.5 Composite Nulls

In practice, it may be unreasonable to require that the means between two groups are precisely zero. Instead, we may only be interested in detecting differences greater than ϵ for some 0 < ϵ ≪ 1 . In this case we may write the null and alternative as

To formulate our test, we introduce two pairs of auxiliary hypotheses: H ′ 0 : µ 0 -µ 1 ≤ ϵ vs H ′ 1 : µ 0 -µ 1 > ϵ and H ′′ 0 : µ 1 -µ 0 ≤ ϵ vs H ′′ 1 : µ 1 -µ 0 > ϵ . Observe that if either H ′ 0 or H ′′ 0 is false, then the alternative H 1 is true. Our approach will therefore entail testing both H ′ 0 vs H ′ 1 and H ′′ 0 vs H ′′ 1 . Let ϕ Q t denote the sequential test for the former, and ϕ R t for the latter. The test given by ϕ t = max { ϕ Q t , ϕ R t } (i.e., rejecting H 0 if either H ′ 0 or H ′′ 0 is rejected) is then a test for (12). Game-theoretically, this can be interpreted as splitting our initial capital in half and playing two games simultaneously. To test H ′ 0 and H ′′ 0 consider the two payoff functions

If λ t ∈ [ -1 1 -ϵ , 1 1+ ϵ ] then both Q t and R t are nonnegative. We thus select λ t via ONS as usual. The wealth processes ( K Q t ) and ( K R t ) defined by Q t and R t (i.e., K Q t = ∏ i ≤ t Q i ( ̂ Y 0 i , ̂ Y 1 i ) and K R t = ∏ i ≤ t R i ( ̂ Y 0 i , ̂ Y 1 i ) ) are then nonnegative supermartingales under H ′ 0 and H ′′ 0 , respectively. We reject H 0 if either K R t ≥ 2 /α or K Q t ≥ 2 /α which results in an overall levelα sequential test.

Proposition 4. Algorithm 1 with input α/ 2 ∈ (0 , 1) with betting strategy Q t (resp., R t ) is a levelα/ 2 sequential test for H ′ 0 vs H ′ 1 (resp., H ′′ 0 vs H ′′ 1 ). Rejecting H 0 if either H ′ 0 or H ′′ 0 is rejected results in a levelα sequential test with asymptotic power one for problem (12) . Let ∆ = | µ 0 -µ 1 | . Then, under the alternative the expected stopping time τ obeys

Distribution Shift, Census Data

No correction

(M1)

With correction (M2)

k=250

k=500

0.10

(under

0.5

k=1000

×

Betting

Permutation test (M1)

0.05

FPR

Permutation test (M2)

0.0

0.00

200

400

600

800

1000

2000

Rejection time T (under H1)

Rejection time T (under H1)

We note that under the alternative, ϵ < ∆ , so 0 < ∆ -ϵ < ∆ . Proposition 4 states that the expected stopping time depends on the interplay of ∆ and ϵ , and will increase as they approach each other. This is intuitive. For a fixed ∆ one would expect that the problem becomes harder as ϵ ↑ ∆ .

4 Experiments

Here we provide several sets of experiments to demonstrate the benefits of our sequential test compared to fixed-time tests. 2 First we need to consider how fixed-time tests might be applied in practice to deal with sequential settings. We consider two such methods.

M1. For some prespecified k ∈ N , wait until we have collected k audits, then perform the test. If the test does not reject, collect another k audits and repeat. We emphasize that if one does not adjust the significance level over time, then this is not a valid levelα test . However, it may be used unwittingly in practice, so we study it as one baseline.

M2. We batch and test in the same way as above, but we apply a Bonferroni-like correction in order to ensure it is level α . More precisely, for the j th-batch, j ≥ 1 , we set the significance level to α/ 2 j . The union bound then ensures that the FPR is at most α over all batches.

We run experiments on three real world datasets: a credit default dataset [50], US census data [51], and health insurance data [52]. All can be made to violate equality of opportunity if trained with naive models; we provide the details in Appendix D. Suffice it to say here that the absolute difference in means, ∆ , is ≈ 0.03 for the credit default dataset, ≈ 1.1 for the census data, and ≈ 0 . 06 for the insurance data. We employ various distinct models to test our methods, from forest-based models to logistic regression. We use a permutation test as our baseline fixed-time test because of its ubiquity in practice, exact type-I error control under exchangeability, and minimax optimality in various scenarios [53].

Figure 3 compares the results of Algorithm 1 (betting strategy (2)) with the permutation test baselines M1 and M2. The left two columns plot the empirical false positive rate (FPR) against the stopping time under the alternative. Values concentrated in the lower left corner are therefore preferable. We plot values for each test as α is varied from 0.01 to 0.1. The betting test typically Pareto-dominates the (stopping time, FPR) values achievable by the baselines. In a small number of cases, M1 (baseline without Bonferroni correction) achieves a faster stopping time; however, this is only possible with a very high FPR (> 0.5). This inflated FPR is verified by the final column, which shows that M1 does not respect the desired type-I error rate. This encourages the use of the Bonferroni-style correction of M2. However, doing so results in an overly conservative test which is slower to reject than betting, as seen in the center column of Figure 3. Betting, meanwhile, respects the desired FPR over all α , and almost always has the fastest stopping time at any given FPR. We note that while it may appear that the permutation tests improve as k gets larger, this trend does not hold for large k . Indeed, k is

Various Data Collection Policies, Insurance Dataset

Population

0.10

×

p

probabilit

0.6

T1

T2

T3

0.4

T3

under

k =100

0.05

k=200

Relative

k=500

R

0.2

0.00

0.0

NE

NW

SE

SW

0

2500

5000

7500

10000

Region

also the minimum rejection time for the permutation test. Increasing k to 1,500 on the credit default dataset, for instance, would result in vertical line at τ = 1 , 500 .

The robustness of betting to distribution shift is illustrated by Figure 4. Here we use a fair model for the first 400 timesteps, after which we switch to a random forest classifier which is unfair ( ∆ ≈ 1 . 1 ). The rejection time for all tests suffers accordingly, but betting remains a Pareto improvement over the permutation tests using M2. As before, using M1 results in an enormous FPR. Interestingly, we see that M2 also results in FPR higher than 0.1 (the maximum value of α ) a nontrivial percentage of the time. Indeed, under distribution shift, the the permutation test using M2 is no longer levelα since the data are not exchangeable. Betting, on the other hand, maintains its coverage guarantee.

Finally, Figure 5 illustrates the performance of various algorithms when audits are not conducted on predictions received uniformly at random from the population (discussed in Section 3.2). Instead, predictions were received based on an individual's region. We test three distinct strategies, π 1 , π 2 and π 3 , each of which deviates from the population distribution to varying degrees (LHS of Figure 5). As expected, the rejection time of our strategy suffers compared to when the predictions are received uniformly from the population (denoted by the red crosses). The desired levelα guarantee is still met, however, and betting continues to outperform permutation tests in all cases.

5 Summary

We have argued that practitioners in the field of algorithmic fairness should consider adopting sequential hypothesis tests in lieu of fixed-time tests. The former enjoy two desirable properties absent in the latter: (i) the ability to continually monitor incoming data, and (ii) the ability to reject at data-dependent stopping times. Both (i) and (ii) are useful for high-stakes and/or timesensitive applications in which fixing the budget beforehand and waiting for more data to arrive may be unsatisfactory, such as auditing the fairness of healthcare models [54] or resource allocation strategies during a public health crisis [55, 56]. We provided a sequential test with asymptotic power one for group fairness (Algorithm 1) inspired by the paradigm of game-theoretic probability and statistics [25, 21, 28]. It is fast, easy to implement, and can handle time-varying data collection policies, distribution shift, and composite nulls, all of which are common in practice. We also provided bounds on the expected stopping time of the test. We hope that the simplicity of our methods combined with their strong theoretical guarantees prove useful for practitioners in the field.

Limitations and societal impact. Auditing is a complex socio-technical problem which involves a range of questions: choosing appropriate audit targets, gaining access to data, ensuring the credibility and independence of external auditors, creating accountability based on audit results, and more [57]. Indeed, poorly designed audits may certify that a system is 'fair" while masking harder to verify issues such as data provenance. Our work addresses only one portion of the overall auditing task: providing statistically sound methods for the sequential testing of a specific criteria. This is complementary to, and does not substitute for, careful overall design of an auditing framework.

Acknowledgements. We thank Jing Yu Koh and Shubhanshu Shekhar for helpful conversations. We also thank the anonymous referees for helpful feedback which improved the paper. BC and AR acknowledge support from NSF grants IIS-2229881 and DMS-2310718. BC was supported in part by the NSERC PGS D program, grant no. 567944. BW and SCG were supported in part by the AI2050 program at Schmidt Futures, grant no. G-22-64474.

References

Shira Mitchell, Eric Potash, Solon Barocas, Alexander D'Amour, and Kristian Lum. Algorithmic fairness: Choices, assumptions, and definitions. Annual Review of Statistics and Its Application , 8:141-163, 2021. 1

Peter Henderson, Ben Chugg, Brandon Anderson, and Daniel E Ho. Beyond ads: Sequential decision-making algorithms in law and public policy. In Proceedings of the 2022 Symposium on Computer Science and Law , pages 87-100, 2022. 1

Deborah Hellman. Measuring algorithmic fairness. Virginia Law Review , 106(4):811-866, 2020. 1

Jon Kleinberg, Jens Ludwig, Sendhil Mullainathan, and Ashesh Rambachan. Algorithmic fairness. In AEA papers and proceedings , volume 108, pages 22-27, 2018.

Allison Woodruff, Sarah E Fox, Steven Rousso-Schindler, and Jeffrey Warshaw. A qualitative exploration of perceptions of algorithmic fairness. In Proceedings of the 2018 CHI conference on human factors in computing systems , pages 1-14, 2018. 1

Inioluwa Deborah Raji and Joy Buolamwini. Actionable auditing: Investigating the impact of publicly naming biased performance results of commercial AI products. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society , pages 429-435, 2019. 1

Jack Bandy. Problematic machine behavior: A systematic literature review of algorithm audits. Proceedings of the ACM on human-computer interaction , 5:1-34, 2021.

Danaë Metaxa, Joon Sung Park, Ronald E. Robertson, Karrie Karahalios, Christo Wilson, Jeff Hancock, and Christian Sandvig. Auditing algorithms: Understanding algorithmic systems from the outside in. Foundations and Trends® in Human-Computer Interaction , 14(4):272-344, 2021. 1

Briana Vecchione, Karen Levy, and Solon Barocas. Algorithmic auditing and social justice: Lessons from the history of audit studies. In Equity and Access in Algorithms, Mechanisms, and Optimization , pages 1-9. 2021. 1

Inioluwa Deborah Raji, Andrew Smart, Rebecca N. White, Margaret Mitchell, Timnit Gebru, Ben Hutchinson, Jamila Smith-Loud, Daniel Theron, and Parker Barnes. Closing the AI accountability gap: Defining an end-to-end framework for internal algorithmic auditing. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency , pages 33-44, 2020. 1

John R Zech, Marcus A Badgeley, Manway Liu, Anthony B Costa, Joseph J Titano, and Eric Karl Oermann. Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross-sectional study. PLoS medicine , 15(11):e1002683, 2018. 1

Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In Proceedings of the European conference on computer vision (ECCV) , pages 456-473, 2018. 1

Xiaoxuan Liu, Ben Glocker, Melissa M McCradden, Marzyeh Ghassemi, Alastair K Denniston, and Lauren Oakden-Rayner. The medical algorithmic audit. The Lancet Digital Health , 2022. 1

Bahar Taskesen, Jose Blanchet, Daniel Kuhn, and Viet Anh Nguyen. A statistical test for probabilistic fairness. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency , pages 648-665, 2021. 1, 2

Nian Si, Karthyek Murthy, Jose Blanchet, and Viet Anh Nguyen. Testing group fairness via optimal transport projections. In International Conference on Machine Learning , pages 9649-9659. PMLR, 2021. 1, 2

Abraham Wald. Sequential analysis . John Wiley and Sons, 1947. 2

Abraham Wald and Jacob Wolfowitz. Optimum character of the sequential probability ratio test. The Annals of Mathematical Statistics , pages 326-339, 1948.

Herbert Robbins. Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society , 58(5):527-535, 1952.

Donald A Darling and Herbert Robbins. Confidence sequences for mean, variance, and median. Proceedings of the National Academy of Sciences , 58(1):66-68, 1967.

Tze Leung Lai. On confidence sequences. The Annals of Statistics , pages 265-280, 1976. 2

Glenn Shafer. Testing by betting: A strategy for statistical and scientific communication. Journal of the Royal Statistical Society: Series A (Statistics in Society), with discussion , 184(2): 407-431, 2021. 2, 3, 4, 10

Aaditya Ramdas, Johannes Ruf, Martin Larsson, and Wouter Koolen. Admissible anytime-valid sequential inference must rely on nonnegative martingales. arXiv preprint arXiv:2009.03167 , 2020.

Peter Grünwald, Rianne de Heide, and Wouter M Koolen. Safe testing. Journal of the Royal Statistical Society, Series B (Methodological), forthcoming with discussion , 2023. 2

Glenn Shafer and Vladimir Vovk. Probability and finance: It's only a game! , volume 491. John Wiley & Sons, 2005. 2

Glenn Shafer and Vladimir Vovk. Game-theoretic foundations for probability and finance , volume 455. John Wiley & Sons, 2019. 2, 10

Ian Waudby-Smith and Aaditya Ramdas. Estimating means of bounded random variables by betting. Journal of the Royal Statistical Society, Series B (Methodological), forthcoming with discussion , 2023. 2, 3, 4

Aaditya Ramdas, Peter Grünwald, Vladimir Vovk, and Glenn Shafer. Game-theoretic statistics and safe anytime-valid inference. Statistical Science , 2023. 2

Shubhanshu Shekhar and Aaditya Ramdas. Nonparametric two-sample testing by betting. IEEE Transactions on Information Theory , 2023. 2, 3, 4, 5, 9, 10

Jean Ville. Étude critique de la notion de collectif. Bull. Amer. Math. Soc , 45(11):824, 1939. 2, 4

Johannes Ruf, Martin Larsson, Wouter M Koolen, and Aaditya Ramdas. A composite generalization of Ville's martingale theorem. Electronic Journal of Probability (forthcoming) , 2023. 2

Boyan Duan, Aaditya Ramdas, and Larry Wasserman. Interactive rank testing by betting. In Conference on Causal Learning and Reasoning , pages 201-235. PMLR, 2022. 3

Philip B Stark. Conservative statistical post-election audits. The Annals of applied statistics , 2 (2):550-581, 2008. 3

Ian Waudby-Smith, Philip B Stark, and Aaditya Ramdas. RiLACS: Risk limiting audits via confidence sequences. In Electronic Voting: 6th International Joint Conference, E-Vote-ID 2021, Proceedings 6 , pages 124-139. Springer, 2021. 3

Shubhanshu Shekhar, Ziyu Xu, Zach Lipton, Pierre J. Liang, and Aaditya Ramdas. Risklimiting financial audits via weighted sampling without replacement. Uncertainty in Artificial Intelligence (UAI) , 2023. 3

Francesco Orabona and Dávid Pál. Coin betting and parameter-free online learning. Advances in Neural Information Processing Systems , 29, 2016. 3

Kwang-Sung Jun and Francesco Orabona. Parameter-free online convex optimization with sub-exponential noise. In Conference on Learning Theory , pages 1802-1823. PMLR, 2019.

Keyi Chen, John Langford, and Francesco Orabona. Better parameter-free stochastic optimization with ODE updates for coin-betting. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 6239-6247, 2022.

Ian Waudby-Smith, Lili Wu, Aaditya Ramdas, Nikos Karampatziakis, and Paul Mineiro. Anytime-valid off-policy inference for contextual bandits. ACM/IMS Journal of Data Science , 2023. 3

Moritz Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. Advances in neural information processing systems , 29, 2016. 3, 21

Sam Corbett-Davies, Emma Pierson, Avi Feller, Sharad Goel, and Aziz Huq. Algorithmic decision making and the cost of fairness. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data mining , pages 797-806, 2017. 3

Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. In Proceedings of the 3rd innovations in theoretical computer science conference , pages 214-226, 2012. 3

Faisal Kamiran, Indr˙ e Žliobait˙ e, and Toon Calders. Quantifying explainable discrimination and removing illegal discrimination in automated decision making. Knowledge and information systems , 35:613-644, 2013. 3

Aaditya Ramdas and Tudor Manole. Randomized and exchangeable improvements of Markov's, Chebyshev's and Chernoff's inequalities. arXiv preprint arXiv:2304.02611 , 2023. 4, 16

Ashok Cutkosky and Francesco Orabona. Black-box reductions for parameter-free online learning in Banach spaces. In Conference On Learning Theory , pages 1493-1529. PMLR, 2018. 4, 16

Elad Hazan, Amit Agarwal, and Satyen Kale. Logarithmic regret algorithms for online convex optimization. Machine Learning , 69(2-3):169-192, 2007. 4

Daniel G Horvitz and Donovan J Thompson. A generalization of sampling without replacement from a finite universe. Journal of the American statistical Association , 47(260):663-685, 1952. 6

RD Narain. On sampling without replacement with varying probabilities. Journal of the Indian Society of Agricultural Statistics , 3(2):169-175, 1951. 6

Peter Henderson, Ben Chugg, Brandon Anderson, Kristen Altenburger, Alex Turk, John Guyton, Jacob Goldin, and Daniel E Ho. Integrating reward maximization and population estimation: Sequential decision-making for Internal Revenue Service audit selection. AAAI Conference on Artificial Intelligence , 2023. 6

Ben Chugg, Peter Henderson, Jacob Goldin, and Daniel E Ho. Entropy regularization for population estimation. AAAI Conference on Artificial Intelligence , 2023. 6

I-Cheng Yeh and Che-hui Lien. The comparisons of data mining techniques for the predictive accuracy of probability of default of credit card clients. Expert systems with applications , 36(2): 2473-2480, 2009. 9

Frances Ding, Moritz Hardt, John Miller, and Ludwig Schmidt. Retiring adult: New datasets for fair machine learning. Advances in Neural Information Processing Systems , 34, 2021. 9, 21

Brett Lantz. Machine learning with R: expert techniques for predictive modeling . Packt Publishing Limited, 2019. 9, 21

Ilmun Kim, Sivaraman Balakrishnan, and Larry Wasserman. Minimax optimality of permutation tests. The Annals of Statistics , 50(1):225-251, 2022. 9

Muhammad Aurangzeb Ahmad, Arpit Patel, Carly Eckert, Vikas Kumar, and Ankur Teredesai. Fairness in machine learning for healthcare. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 3529-3530, 2020. 10

Douglas B White and Derek C Angus. A proposed lottery system to allocate scarce COVID-19 medications: promoting fairness and generating knowledge. Jama , 324(4):329-330, 2020. 10

Ben Chugg, Lisa Lu, Derek Ouyang, Benjamin Anderson, Raymond Ha, Alexis D'Agostino, Anandi Sujeer, Sarah L Rudman, Analilia Garcia, and Daniel E Ho. Evaluation of allocation schemes of COVID-19 testing resources in a community-based door-to-door testing program. In JAMA health forum , volume 2, pages e212260-e212260. American Medical Association, 2021. 10

Inioluwa Deborah Raji, Peggy Xu, Colleen Honigsberg, and Daniel Ho. Outsider oversight: Designing a third party audit ecosystem for AI governance. In Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society , pages 557-571, 2022. 10

Nick Harvey. A second course in randomized algorithms. 2022. 17

A Testing in batches

Of course, it is the rare application in which we can expect to receive an audit from each group at each timestep. We can modify the strategy in Section 3 to handle different arrival times by simply waiting to bet until we have new audits from each group. If multiple audits from the other group have accumulated during that period, we bet with their average. More formally, define

where we recall that T b [ t ] = T b ∩ [ t ] is the set of times at which we receive audits from group b up until time t . In words, N b t is simply all timesteps for which we receive an audit from group b without receiving an audit from the other group. From here, we can define a payoff function as

̸

̸

̸

If S t = 1 then K t = K t -1 , so this can be interpreted as abstaining from betting at time t . This occurs when we have no new audits from one group ( N b t = ∅ for some b ). Note that if N 0 t and N 1 t are both non-empty, then one of them contains only t . Thus, in this case, G 0 t -G 1 t ∈ { ̂ Y 0 t -G 1 t , G 0 t -̂ Y 1 t } . As in Section 3 we choose λ t using ONS, but we use g t = G 0 t -G 1 t in its definition. The expected stopping time of the test defined by (16) follows as a corollary from Proposition 1 after correcting for the number of timesteps during which non-trivial bets were placed. More specifically, the stoppingtime in Proposition 1 is with reference to the number of times we actually bet, i.e., with respect to the times { t : N 0 t = ∅ , N 1 t = ∅} . The same remark holds for other propositions throughout this paper and, as such, we will state these propositions under the assumption that T 0 = T 1 = N .

̸

Finally, let us emphasize that the above discussion is still assuming an online data gathering procedure. If we are in a nonsequential setting (i.e., all the data has already been gathered), then we may simulate a sequential setting by simply revealing the outcomes one at a time. Thus, all of our procedures hold for fixed-time hypothesis testing as well.

B Auditing multiple groups

Here we consider the case when there are more than two groups. Suppose we have J +1 groups { 0 , 1 , . . . , J } . In accordance with Definition 1, the null and alternative become

̸

As before, let µ i = E ρ [ φ ( X ) | ξ i ] , i ∈ { 0 , . . . , J } . One could derive a sequential test by applying Algorithm 1 to each pair of means µ i and µ j . Game-theoretically, this can be interpreted as splitting your initial wealth among multiple games and playing each simultaneously. If you grow rich enough in any one game, you reject the null. Of course, one needs to adjust the significance level to account for the number of games being played, thus reducing the (nonasymptotic) power of the test.

̸

Of course, it is not necessary to test each mean against all others. We need only test whether µ b = µ b +1 for all b ∈ { 0 , . . . , J } . That is, we can play J games instead of Ω( J 2 ) games. In order to ensure this constitutes a levelα test, we reject when the wealth process of any game is at least J/α . For any single game, this occurs with probability α/J under the null. Therefore, the union bound then ensures that the type-I error of this procedure is bounded by α . Moreover, the asymptotic power remains one since, if µ i = µ j for some i, j then µ b = µ b +1 for some b . The guarantees we've provided on Algorithm 1 ensure that the wealth process for this particular game will eventually grow larger than J/α and our test will reject. We summarize this discussion with the following proposition, which is the equivalent of Proposition 1 for auditing multiple groups.

̸

Proposition 5. Let α ∈ (0 , 1) . Consider running Algorithm 1 on groups b, b + 1 , for b ∈ { 0 , 1 , . . . , J -1 } in parallel with input parameter α/J . This constitutes a levelα sequential test for (17) with aymptotic power one against (18) . If we receive an audit from each group at each timestep, then the expected stopping time τ of this procedure obeys

The expected stopping time follows from Proposition 1 after correcting for the significance level and the difference between the means. We take the minimum over all b because the procedure rejects as soon as any of the wealth processes grow too large. Equivalent versions of Propositions 2 and 3 for multiple groups can be obtained similarly.

Let us end by remarking that similar ideas may be applied to test other notions of group fairness which posit the equivalence of multiple means. This is the case of equalized odds, for instance. As above we simply imagine playing multiple games simultaneously, each testing for the equivalence of one pair of means.

C Omitted proofs

C.1 Proof of Proposition 1

We break the proof into three components.

Levelα sequential test. Combining the discussion at the beginning of Section 3 with Ville's inequality demonstrates why our procedure constitutes a levelα sequential test. However, let us prove it formally here for completeness. Let P ∈ H 0 and note that E P [ ̂ Y 0 t -̂ Y 1 t ] = E P [ φ ( X 0 t ) -φ ( X 1 t )] = µ 0 -µ 1 = 0 . Therefore, using the fact that λ t is predictable (i.e., F t -1 -measurable)

so ( K t ) t ≥ 1 is a P -martingale, with initial value 1. Moreover, it is nonegative since | λ t | ≤ 1 / 2 for all t by definition of ONS. Thus, Ville's inequality implies P ( ∃ t ≥ 1 : K t ≥ 1 /α ) ≤ α , meaning that rejecting at 1 /α yields a levelα sequential test. Finally, as discussed in the main paper, the last lines of Algorithm 1 are justified by the randomized Ville's inequality of Ramdas and Manole [43], which states that, for all stopping times n ,

where U ∼ Unif (0 , 1) is independent of everything else.

Asymptotic power. Next, let us demonstrate that Algorithm 1 has asymptotic power one. That is, for P ∈ H 1 , P ( τ < ∞ ) = 1 . It suffices to show that P ( τ = ∞ ) = 0 . To see this, define

We have the following guarantee on the wealth process, which can be translated from results concerning ONS from Cutkosky and Orabona [44, Theorem 1]:

Since { τ = ∞} ⊂ { τ ≥ t } for all t ≥ 1 , we have P ( τ = ∞ ) ≤ lim inf t →∞ P ( τ > t ) ≤ lim inf t →∞ P ( K t < 1 /α ) , where the final inequality is by definition of the algorithm. Using the second inequality of (20),

̸

By the SLLN, S t /t converges to µ 0 -µ 1 = 0 almost surely. On the other hand, 8 log( t/α ) /t → 0 . Thus, if we let A t be the event that exp( S 2 t / 8 t ) < t/α , we see that 1 ( A t ) → 0 almost surely. Hence, by the dominated convergence theorem,

This completes the argument.

Expected stopping time. Last, let us show the desired bound on the expected stopping time. Fix P ∈ H 1 . Let τ be the stopping time of the test. Since it is nonnegative, we have

for E t = { log K t < log(1 /α ) } . Note that the second equality is by definition of the algorithm. Employing the first inequality of (20) yields

To analyze the probability of this event, we first develop upper bounds on W t := ∑ i ≤ t | g i | and V t . We begin with W t . Since W t is the sum of independent random variables in [0 , 1] , we apply the multiplicative Chernoff bound (e.g., [58]) to obtain

Setting the right hand side equal to 1 /t 2 and solving for δ gives δ = √ 6 log t/ E W t . Thus, with probability 1 -1 /t 2 , we have

where we've used that E [ W t ] = ∑ i ≤ t E [ | g i | ] ≤ t since | g i | ≤ 1 . Following a nearly identical process for V t , we have that with probability 1 -1 /t 2 ,

where again we use that | g i | 2 ≤ | g i | ≤ 1 . Let H t = { V t ≤ 2 t } ∩ { W t ≤ 2 t } . Then,

We now argue that | S t | is unlikely to be so small. Indeed, since S t is the sum of independent random variables in [ -1 , 1] , applying a Chernoff bound for the third time gives P ( | S t -E S t | ≥ u ) ≤ 2 exp( -u 2 /t ) . So, with probability 1 -1 /t 2 , by the reverse triangle inequality,

This final quantity is at least D for all t ≥ 81 ∆ 2 log( 162 ∆ 2 α ) . Now, combining what we've done thus far, by the law of total probability,

and so, for t large enough such that (21), (22), and S t > D all hold, that is

implying that, we have

This completes the proof.

C.2 Proof of Proposition 2

The proof is similar to that of Proposition 1, so we highlight only the differences.

The wealth process remains a martingale due to the IPW weights (5). Indeed, since λ t and L t are F t -1 measurable, under the null we have

Moreover, as described in the text, multiplication by L t ensures that K t is nonnegative, since

since

for each b by definition. Therefore, ( K t ) t ≥ 1 is a nonnegative martingale and, as before, Ville's inequality implies that rejecting at 1 /α gives a levelα sequential test.

The asymptotic power follows by replacing g t in Appendix C.1 with

Under the alternative, h t has non-zero expected value, so identical arguments apply.

Regarding, the expected stopping time, we again argue about h t instead of g t . Since | h t | ≤ 1 (see above), the bounds on V t and W t remain as they are in the proof of Proposition 1. The bound on | E [ S t ] | is where the proof departs that in Appendix C.1. In this case we have

Therefore,

Induction thus yields

From here, we may replace ∆ in the proof in Appendix C.1 with ∆ L inf and the arithmetic remains the same. This yields the desired result.

C.3 Proof of Proposition 3

Again, the proof mirrors that of Proposition 1 so we highlight only the differences.

First let us ensure that Algorithm 1 yields a levelα sequential test. As before, it suffices to demonstrate that the wealth process is a nonnegative martingale. The time-varying means do not change this fact from before:

since, under the null, E [ Y 0 t |F t -1 ] = E [ φ ( X ) | ξ 0 , F t -1 ] = µ 0 = µ 1 = E [ φ ( X ) | ξ 1 , F t -1 ] = E [ Y 1 t |F t -1 ] . Nonnegativity once again follows from the ONS strategy.

Asymptotic power follows an identical argument as in Appendix C.1, so we focus on expected stopping time. The event E t remains the same as in Appendix C.1. We again apply a Chernoff bound

to W t (the values remain independent, even though they are not necessarily identically distributed), and obtain

for t ≥ 17 with probability 1 -1 /t 2 , since again, | g i | ≤ 1 for each i . Similarly, E V t ≤ 2 t with probability 1 -1 /t 2 for t ≥ 17 . Let the shift begin at time n , and set ∆ = inf t ≥ n | µ 0 ( t ) -µ 1 ( t ) | . Then | E S t | ≥ ( t -n )∆ . As above, we want to find t such that

Rearranging and simplifying this final inequality, we see that it suffices for t to satisfy

To see this, suppose first that n ≥ β where

Then, at t = 2 n , the right hand side of (23) is

where the final inequality holds for all n ≥ β , which was assumed. Now suppose that n < β , so that (23) should hold for t ≥ n + β . Since the left hand side of (23) grows faster than the right hand side, it suffices to show that it holds at t = n + β . To this end, write

where the final inequality uses the (loose) bound log log( x ) ≤ log 2 ( x ) .

C.4 Proof of Proposition 4

First let us note that ( K Q t ) are indeed nonnegative supermartingales. Under the null H ′ 0 , we have E H ′ 0 [ ̂ Y 0 t -̂ Y 1 t ] ≤ ϵ , so

Moreover, for λ t ∈ [ -1 1 -ϵ , 1 1+ ϵ ] , Q t ≥ 0 . The argument is similar for R t . Therefore, by Ville's inequality,

so the union bound gives P ( ∃ t ≥ 1 : K Q t ≥ 2 /α or K R t ≥ 2 /α ) ≤ α . This implies that the proposed test, which involves rejecting if max {K Q t , K R t } ≥ 2 /α is a levelα sequential test.

We claim this holds for all

Now, note that ONS restricts λ t to be in [ -1 / 2 , 1 / 2] , which is a subset of [ -1 1 -ϵ , 1 1+ ϵ ] . We can therefore use a similar ONS analysis as above, without needing to modify the guarantees to account for the range of λ . Recall from Section C.1 the definitions g t , S t , and V t . Because Q t and R t have an extra ϵ term, we need to modify these terms for this new setting. Define

The guarantee given by (20) translates, in this setting, to

where we've simply replaced S t and V t by S ′ t and V ′ t . The same guarantee holds for K Q t (there g i is replaced with -g i but it makes no difference).

Results for asymptotic power (for both K Q t and K R t ) follow from similar arguments as in Section C.1 but replacing S t and V t with S ′ t and V ′ t . Therefore, let us turn our attention to expected stopping time. We will consider the expected stopping time of the test based on K Q t . Since identical arguments hold for K R t , one need only multiply the result by 2 to get the overall expected stopping time.

Following what was done in previous proofs, write E [ τ ] = ∑ ∞ t =1 P ( E t ) , where E t = { log K Q t < log(1 /α ) } . We have

Here we've used that V ′ t ≤ V t + tϵ 2 and | S ′ t | ≤ | S t | + tϵ . As in Section C.1, we have W t = ∑ i ≤ t | g i | ≤ 2 t and V t ≤ 2 t with probability 1 -t 2 for all t ≥ 17 . Let H t be the event that both V t ≤ 2 t and W t ≤ 2 t . Note that by the reverse triangle inequality, | S ′ t | ≥ | S t | -tϵ . Therefore,

where we've used that ϵ < 1 . With probability 1 -1 /t 2 , | S t | ≥ t ∆ -√ 2 t log(2 t ) . The minimum t = T such that t ∆ -√ 2 t log(2 t ) ≥ tϵ + √ 18 t (log(1 /α ) + log(3 t )) is on the order of

Therefore,

Note that this is under the alternative, so ∆ > ϵ and 0 < ∆ -ϵ < ∆ .

D Simulation Details

Code to recreate all plots and run the simulations is available at https://github.com/bchugg/ auditing-fairness . Here we provide more extensive details on each figure.

Figure 1. Given ∆ , we generate the two means µ 0 and µ 1 as µ 0 = 0 . 5+∆ / 2 and µ 1 = 0 . 5 -∆ / 2 . We take φ ( X ) | ξ b to be distributed as Ber ( µ b ) . (Thus, this simulates a scenario for which we witness the classifcation decisions, not e.g., a risk score.) We set α = 0 . 01 , so we reject when the wealth process is at least 100 . We receive a pair of observations each timestep. Each experiment was run 100 times to generate the plotted standard deviation around the mean of each wealth process.

Figure 2. As above, we take the distribution of model observations φ t ( X ) | ξ b to be Ber ( µ b ( t )) . For the left hand side of Figure 2 we take µ 0 ( t ) = µ 1 ( t ) = 0 . 3 for t = 1 , . . . , 99 . At t = 100 , we add a logistic curve to µ 1 . In particular, we let

We keep µ 0 ( t ) at 0.3 for all t . For the right hand side of Figure 2, we let both µ 1 and µ 0 be noisy sine functions with different wavelengths. We let

for all t , where ϵ 0 t ∼ N (0 , 0 . 01) . Meanwhile,

where, again, ϵ 1 t ∼ N (0 , 0 . 01) . The mean µ 1 ( t ) thus has a constant upward drift over time. As before, we assume we receive a pair of observations at each timestep and we take α = 0 . 01 . We generate the means once, but run the sequential test 100 times in order to plot the standard deviation around the mean of the wealth process.

Figures 3 and 4. For a given sequential test and a given value of α , we run (i) the test under the null hypothesis (i.e., with a fair model; we describe how we generated a fair model below), and (ii) the test under the alternative. Repeating 300 times and taking the average gives the FPR and average rejection time for this value of α . This procedure is how the leftmost two columns of Figure 3 were constructed. The final column then simply plots the FPR versus the value of α .

We used a vanilla random forest for both the credit default dataset and the US census data. For the credit default dataset, the model does not satisfy equality of opportunity [39] when Y indicates whether an individual has defaulted on their loan, and A indicates whether or not they have any university-level education. One can imagine loans being given or withheld on the basis of whether they are predicted to be returned; we might wish that this prediction does not hinge on educational attainment. For the census data, the model does not satisfy equality of opportunity when A indicates whether an individual has an optical issues, and Y indicates whether they are covered by public insurance. Admittedly, this example is somewhat less normative than the other. It is unclear whether we should expect perfect equality of opportunity in this context. However, we emphasize that our experiments are for illustrative purposes only. They are not meant as comments on the actual fairness or unfairness of these datasets. We interface with the census data by means the folktables package [51].

For the credit default dataset and random forest classifier, we have ∆ = | µ 0 -µ 1 | = 0 . 034 . We have ∆ = 0 . 09 for the census data. To construct the fair model (in order to test the null), we add ∆ to the model predictions of the group with the lower mean. Thus, the distributions of predictions are different but the means are identical.

Figure 4 follows similar experimental logic, but we begin with a fair model (i.e., group predictions with the same mean, generated as above), and then switch to the unfair random forest classifier at time t = 400 .

Figure 5. We use US health insurance data [52], which is synthetic data generated based on US census information. We train a logistic regression model to predict the risk of non-repayment based on 11 covariates which include age, BMI, gender, region, and whether the individual is a smoker. We find that the predicted risks are different among males (0.235) and females (0.174), so we use A as gender and use equality of opportunity as our notion of group fairness.

We construct three data collection policies, π 1 , π 2 , and π 3 , which are probability distributions over four regions: NE, SE, NW, SW. Data-collection works by first sampling a region with the probability prescribed by π i , and then sampling an individual in that region uniformly at random. We defined the three strategies as follows:

We also ran our-betting based method with data sampled uniformly for the population, which served as the comparison point. Of course, as is expected from Proposition 1 and 2, data sampled uniformly from the population yields better results. We also compared to permutation tests run on data sampled uniformly from the population. These were implemented as described above and we do not belabor the details here.

Peer reviews:
comments: SUMMARY:
The paper proposes a safe anytime-valid inference (SAVI) testing procedure to audit the potential unfairness of a classifier or regression model. The paper posits group-conditioned fairness as a test of equality of means under some finite set of conditions, this definition includes several common fairness notions such as equality of odds and equality of opportunity. In this setting, the authors propose a technique allows for sequential testing of equality of means that can accommodate repeated testing, finite or infinite testing time horizons, batched testing, and importance weighted testing. The authors also provide expected stopping times for their testing procedure.

STRENGTHS:
The paper is well written and structured. All the necessary information to understand the paper's contributions is provided within the main text. Several practical considerations (time horizon, batched testing, importance weighting) are clearly outlined and addressed. The method itself is extremely simple to implement.

WEAKNESSES:
Overall the paper is very solid. The main weakness I see is that there is no immediate way of extending these ideas to settings where equality of means may be too strict of a definition, and instead one would want to test that the absolute difference of means falls below some epsilon value.

QUESTIONS:
See weaknesses above.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces practical and efficient methods for auditing the fairness of deployed classification and regression models. The methods are sequential, allowing for continuous monitoring of incoming data, and can handle distribution shifts resulting from changes in the model or the underlying population. The approach is based on recent advancements in anytime-valid inference and game-theoretic statistics. The methods are demonstrated to be effective through experiments on benchmark fairness datasets.\n\nSTRENGTHS:\nOriginality:\n\nThe concept of developing a system that can continuously monitor and track the fairness of deployed real-world systems in a sequential evaluation context is a novel contribution. The idea of extending fairness evaluation beyond the development phase is an innovative approach.\n\nQuality:\n\nThe paper demonstrates a high level of quality, with well-founded theoretical reasoning and rigorous analysis. The authors present a solid theoretical foundation that supports their proposed methods.\n\nClarity:\n\nThe paper is written in a clear and understandable manner. The research problem is effectively formulated, and the theoretical foundations are explained comprehensively. The clarity of the paper contributes to its accessibility and readability.\n\nSignificance:\n\nEnsuring fairness in systems goes beyond the initial testing phase and requires continuous monitoring of deployed models. The significance of the paper lies in its contribution to the field by providing a way to quantify and evaluate the fairness of deployed models. This capability is becoming increasingly important as the need for ongoing monitoring and accountability in real-world systems grows. The paper's focus on continuous fairness monitoring and its impact on the research community make it a valuable contribution to the field.\n\nWEAKNESSES:\nIt would greatly enhance the paper's findings to include more than two datasets for result verification. Additionally, since only one model (Random Forest) was trained on the dataset, it would be intriguing to explore the effects of different fairness methods, such as in-processing, post-processing, and preprocessing approaches, by testing multiple models. By incorporating these suggestions, the paper's results would become more robust and comprehensive, providing a more thorough understanding of the proposed methods and their performance across various datasets and models.\n\nQUESTIONS:\nI would like to discuss a thought experiment regarding a fair model that exhibits some degree of non-determinism. In this scenario, the model generally produces fair outputs for a given instance, but with a certain probability (e.g. 10%) the same instance would have an unfair output. My intention in raising this thought is to gain insights into the robustness and reliability of your framework. I am curious to understand whether such non-deterministic behavior would impact or challenge the effectiveness of your proposed framework.\n\nLIMITATIONS:\nI acknowledge that cost considerations and reaching a sufficient amount of evidence to make a decision are factors that may lead to the discontinuation of data collection. However, it is crucial to recognize that the fairness of a deployed system can evolve over time, necessitating continuous evaluation to ensure ongoing fairness. In addition, the emergence of regulatory frameworks like the AI Act in the EU emphasizes the importance of continuous evaluation, as it may become a legal requirement. These factors highlight the need for continuous monitoring and evaluation to address fairness concerns and comply with legal obligations. So even if the null is rejected at some other time in the future this could not be seen any more. How does the framework could handle this? Is this a limitation?", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes an approach based on recent results of super-martingales to detect when a certain predictor is demographically unfair. Studying different formulations of group fairness (such as equal opportunity, demographic parity, etc), the authors frame the problem as that of sequential hypothesis test, where a null corresponding to equalized metrics across groups. The problem is then to reject the null, if the alternative is true, as soon as possible while controlling for type-1 error. Because this is set up as a sequential hypothesis test, this implies strong conditions on the probability of false positive (i.e. for any future time). Relying on the idea of “betting”, the authors then design a pay-off function that allows the “payoff” to be a super-martingale under the null (and remain controlled), whereas the payoff grows exponentially under the alternative. While the authors derive their methodology a a “basic”/standard, they further comment on how the same approach can be minimally modified to accommodate for data collected at different rates from different demographic groups, with different data-collection policies (not iid), and under “distribution shifts”. They finally showcase their method in a couple of real world datasets, comparing with relatively simple alternative.\n\nSTRENGTHS:\n- The studied problem is quite important and timely, given the increasing use of predictive algorithms in society.\n- The proposed solution is relatively simple and elegant, provably correct, and seems effective in practice.\n- This paper is beautifully written, and it was a pleasure to read.\n- The experiment results are very nice and easy to understand\n\nWEAKNESSES:\n- The experiment results do not seem to address the time-varying data collection policies setting.\n\nQUESTIONS:\n1. A potentially trivial question is how to employ the same ideas to not only equal opportunity but also to equalized odds (which requires not only equal TPR but also FPRs to be the same). Is it true that in this case one can simply generalize the null hypothesis as one where both rates being the same, and the results would naturally follow? A footnote on this might be useful.\n\n2. While their approach seems to work quite nicely in practice, I wonder about the tightness of Proposition 1 (and its corollaries that follow). What are the typical constants associated with this bound? Moreover, do the authors think it would be possible to obtain a lower bound on $\\mathbb E[\\tau]$ as a function of $\\Delta$?\n\n3. As far as I understand, the experiments section did not showcase their results on being able to incorporate different data-collection policies (Sec 3.2). Could the authors comment on why this is the case? Even if the data they used involved static policies, could they simulate different policies a posteriori to showcase their results?\n\nLIMITATIONS:\nThe results all hinge on the auditing process being conducted properly. The paper realizes this and appropriately mentions it in the limitations section.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper clarifies the issue of fairness auditing, wherein an auditor verifies the fairness of a system through sequential interaction with it. The goal is to develop a methodology that facilitates anytime-valid tests of system fairness. Essentially, this involves devising a sequence of test statistics for the null hypothesis (i.e., the system is fair), enabling the auditor to conduct statistical tests using these statistics at any given stopping time. To achieve this, the authors design a payoff for the invalidity of observed information against the null hypothesis. A sufficiently high value of this payoff can safely lead to the rejection of the null hypothesis. Both theoretical and experimental analyses of the proposed method substantiate the efficacy of this sequential fairness testing approach.\n\nSTRENGTHS:\nThis is a well-written and fascinating paper. Recently, the problem of fairness auditing has garnered attention, driven by societal demands for fair systems incorporating machine learning techniques. Sequential testing of fairness has become an essential tool for such audits. This paper delivers a robust technical solution for sequential fairness testing, offering impressive theoretical and empirical analyses of its performance.\n\nThe principal technical innovation is the reduction of the fairness auditing problem into a two-sample testing problem. This transformation facilitates the application of the established and theoretically sound methodology of anytime-valid two-sample testing.\n\nThe empirical experiments conducted in this paper effectively illustrate the potential for achieving higher power with fewer interactions, all while controlling the Type-I error in comparison to the permutation test. This evidence underscores the practicality of the proposed method.\n\nWEAKNESSES:\nA notable weakness is the limited innovation in methodology. The construction of the testing algorithm is merely an application of the anytime-valid two-sample testing developed by Shekhar and Ramdas. Additionally, even though Shekhar and Ramdas did not account for adjustments to time-varying data collection and distributional shifts, such an extension might be relatively straightforward.\n\nIn the context of time-varying data collection, the authors make the assumption that the density of the covariate distribution is readily accessible. This assumption, however, isn't practically feasible. Therefore, it would be advantageous to address the limitations of this assumption more thoroughly.\n\nQUESTIONS:\nThe existence of group-wise streams may not be feasible in practice. In a typical scenario, an auditor might receive a sequence of i.i.d. data drawn from a composite distribution that combines the group-wise distributions. Therefore, it would be more beneficial to illustrate the application in a situation where a single, mixed stream is present.\n\nLIMITATIONS:\nThere are no specific limitations or potential negative social impacts that need to be addressed in relation to this paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}