Fair Sortition Made Transparent

1 Introduction . In a citizens ’ assembly , a panel of randomly chosen citizens is convened to deliberate and ultimately make recommendations on a policy issue . The defining aspect of citizens ’ assemblies is the randomness of the process , sortition , by which participants are chosen . In practice , the sortition process works as follows : first , volunteers are solicited via thousands of letters or phone calls , which target individuals chosen uniformly at random . Those who respond affirmatively form the pool of volunteers , from which a final panel will be chosen . Finally , a selection algorithm is used to randomly select some pre-specified number k of pool members for the panel . To ensure adequate representation of demographic groups , the chosen panel is often constrained to satisfy some upper and lower quotas on feature categories such as age , gender , and ethnicity . We call a quota-satisfying panel of size k a feasible panel . As this process illustrates , citizens ’ assemblies offer a way to involve the public in informed decision-making . This potential for civic participation has recently spurred a global resurgence in the popularity of citizens assemblies ; they have been commissioned by governments and led to policy changes at the national level [ 19 , 23 , 12 ] . Prompted by the growing impact of citizens ’ assemblies , there has been a recent flurry of computer scientific research on sortition , and in particular , on the fairness of the procedure by which participants are chosen [ 2 , 13 , 12 ] . The most practicable result to date is a family of selection algorithms proposed by Flanigan et al . [ 12 ] , which are distinguished from their predecessors by their use of randomness toward the goal of fairness : while previously-used algorithms selected pool members in 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . a random but ad-hoc fashion , these new algorithms are maximally fair , ensuring that pool members have as equal probability as possible of being chosen for the panel , subject to the quotas.1 To encompass the many interpretations of “ as equal as possible , ” these algorithms permit the optimization of any fairness objective with certain convexity properties . There is now a publicly available implementation of the techniques of Flanigan et al . [ 12 ] , called Panelot , which optimizes the egalitarian notion that no pool member has too little selection probability via the Leximin objective from fair division [ 21 , 14 ] . This algorithm has already been deployed by several groups of panel organizers , and has been used to select dozens of panels worldwide . Fairness gains in the panel selection process can lend legitimacy to citizens ’ assemblies and potentially increase their adoption , but only insofar as the public trusts that these gains are truly realized . Currently , the potential for public trust in the panel selection process is limited by multiple factors . First , the latest panel selection algorithms select the final panel via behind-the-scenes computation . When panels are selected in this manner , observers can not even verify that any given pool member has any chance of being chosen for the panel . A second and more fundamental hurdle is that randomness and probability , which are central to the sortition process , have been shown in many contexts to be difficult for people to understand and reason about [ 24 , 20 , 28 ] . Aiming to address these shortcomings , we propose and pursue the following notion of transparency in panel selection : Transparency : Observers should be able to , without reasoning in-depth about probability , ( 1 ) understand the probabilities with which each individual will be chosen for the panel in theory , and ( 2 ) verify that individuals are actually selected with these probabilities in practice . In this paper , we aim to achieve transparency and fairness simultaneously : this means advancing the defined goal of transparency , while preserving the fairness gains obtained by maximally fair selection algorithms . Although this task is reminiscent of existing AI research on trade-offs between fairness or transparency with other desirable objectives [ 4 , 11 , 3 , 27 ] , to our knowledge , this is the first investigation of the trade-off between fairness and transparency . Setting aside for a moment the goal of fairness , we consider a method of random decision-making that is already common in the public sphere : the uniform lottery . To satisfy quotas , a uniform lottery for sortition must randomize not over individuals , but over entire feasible panels . In fact , this approach has been suggested by practitioners , and was even used in 2020 to select a citizens ’ assembly in Michigan . The following example , which closely mirrors that real-world pilot,2 illustrates that panel selection via uniform lottery is naturally consistent with the transparency notion we pursue . Suppose we construct 1000 feasible panels from a pool ( possibly with duplicates ) , numbered 000- 999 , and publish an ( anonymized ) list of which pool members are on each panel . We then inform spectators that we will choose each panel with equal probability . This satisfies criterion ( 1 ) : spectators can easily understand that all panels will be chosen with the same probability of 1/1000 , and can easily determine each individual ’ s selection probability by counting the number of panels containing the individual . To satisfy criterion ( 2 ) , we enact the lottery by drawing each of the three digits of the final panel number individually from lottery machines . Lottery spectators can confirm that each ball is drawn with equal probability ; this provides confirmation that panels are indeed being chosen with uniform probabilities , thus confirming the enactment of the proposed individual selection probabilities . In addition to its conventionality as a source of randomness , decision-making via drawing lottery balls invites an exciting spectacle , which can promote engagement with citizens ’ assemblies . This simple method neatly satisfies our transparency criteria , but it has one obvious downside : a uniform lottery over an arbitrary set of feasible panels does not guarantee any measure of equal probabilities to individuals . In fact , it is not even clear that the fairest possible uniform lottery over m panels , where m is a number conducive to selection by physical lottery ( e.g . m =1000 ) , would not be significantly less fair than maximally fair algorithms , which sample the fairest possible unconstrained distribution over panels . For example , if m is too small , there may be no uniform lottery which gives all individuals non-zero selection probability , even if each individual appears 1Quotas can preclude giving individuals exactly equal probabilities : if the panel must be 1/2 men , 1/2 women but the pool is split 3/4 men , 1/4 women , then some women must be chosen more often than some men . 2Of By For ’ s pilot of live panel selection via lottery can be viewed at https : //vimeo.com/458304880 # t=17m59s from 17:59 to 21:23 . For a more detailed description , see Figure 3 and surrounding text in [ 12 ] . on some feasible panel ( and so can attain a non-zero selection probability under an unconstrained distribution ) . Fortunately , empirical evidence suggests that there is hope : in the 2020 pilot mentioned above , a uniform lottery over m =1000 panels was found that nearly matched the fairness of the maximally fair distribution generated by Panelot . Motivated by this anecdotal evidence , we aim to understand whether such a fair uniform lottery is guaranteed to exist in general , and if it does , how to find it . We summarize this goal in the following research questions : Does there exist a uniform lottery overm panels that nearly preserves the fairness of the maximally fair unconstrained distribution over panels ? And , Algorithmically , how do we compute such a uniform lottery ? Results and Contributions . After describing the model in Section 2 , in Section 3 we prove that it is possible to round an ( essentially ) arbitrary distribution over panels to a uniform lottery while preserving all individuals ’ selection probabilities up to only a small bounded deviation . These results use tools from discrepancy theory and randomized rounding . Intuitively , this bounded change in selection probabilities implies bounded losses in fairness ; we formalize this intuition in Section 4 , showing that there exists in general a uniform lottery that is nearly maximally fair , with respect to multiple choices of fairness objective . Although we would ideally like to give such bounds for the Leximin fairness objective , due to its use practice , we can not succinctly represent bounds for this objective because it is not scalar valued . We therefore give bounds for Maximin , a closely related egalitarian objective which only considers the minimum selection probability given to any pool member [ 7 ] . We discuss in Section 4 why bounds on loss in Maximin fairness are , in the most meaningful sense , also bounds on loss in Leximin fairness . We additionally give upper bounds on the loss in Nash Welfare [ 21 ] , a similarly well-established fairness objective that has also been implemented in panel selection tools [ 18 ] . Finally , in Section 5 , we consider the algorithmic question in practice : given a maximally fair distribution over panels , can we actually find nearly maximally fair uniform lotteries that match our theoretical guarantees ? To answer this question , we implement two standard rounding algorithms , along with near-optimal ( but more computationally intensive ) integer programming methods , for finding uniform lotteries . We then evaluate the performance of these algorithms in 11 real-world panel selection instances . We find that in all instances , we can compute uniform lotteries that nearly exactly preserve not only fairness with respect to both objectives , but entire sets of Leximin-optimal marginals , meaning that from the perspective of individuals , there is essentially no difference between using a uniform lottery versus the optimal unconstrained distribution sampled by the latest algorithms . We discuss these results , their implications , and how they can be deployed directly into the existing panel selection pipeline in Section 6 . 2 Model . Panel Selection Problem . First , we formally define the task of panel selection for citizens ’ assemblies . Let N = [ n ] be the pool of volunteers for the panel—individuals from the population who have indicated their willingness to participate in response to an invitation . Let F = { ft } t denote a fixed set of features of interest . Each feature ft : N → Ωt maps each pool member to their value of that feature , where Ωt is the set of ft ’ s possible values . For example , for feature ft = “ gender ” , we might have Ωt = { “ male ” , “ female ” , “ non-binary ” } . We define individual i ’ s feature vector F ( i ) = ( ft ( i ) ) t ∈ ∏ t Ωt to be the vector encoding their values for all features in F . As is done in practice and in previous research [ 13 , 12 ] , we impose that the chosen panel P must be a subset of the pool of size k , and must be representative of the broader population with respect to the features in F . This representativeness is imposed via quotas : for each feature f and corresponding value v ∈ Ω , we may have lower and upper quotas lf , v and uf , v . These quotas require that the panel contain between lf , v and uf , v individuals i such that f ( i ) = v. In terms of these parameters , we define an instance of the panel selection problem as : given ( N , k , F , l , u ) —a pool , panel size , set of features , and sets of lower and upper quotas—randomly select a feasible panel , where a feasible panel is any set of individuals P from the collection K : K : = { P ∈ ( Nk ) : lf , v ≤ | { i ∈ P : f ( i ) = v } | ≤ uf , v for all f , v } . Maximally Fair Selection Algorithms . A selection algorithm is a procedure that solves instances of the panel selection problem . A selection algorithm ’ s level of fairness on a given instance is determined by its panel distribution p , the ( possibly implicit ) distribution over K from which it draws the final panel . Because we care about fairness to individual pool members , we evaluate the fairness of p in terms of the fairness of selection probabilities , or marginals , that p implies for all pool members.3 We denote the vector of marginals implied by p as π , and we will sometimes specify a panel distribution as p , π to explicitly denote this pair . We say that π is realizable if it is implied by some distribution p over the feasible panels K. Maximally fair selection algorithms are those which solve the panel selection problem by sampling a specifically chosen p : one which implies marginals π that allocate probability as fairly as possible across pool members . The fairness of p , π is measured by a fairness objective F , which maps an allocation—in this case , of selection probability to pool members—to a real number measuring the allocation ’ s fairness . Fixing an instance , a fairness objective F , and a panel distribution p , we express the fairness of p as F ( p ) . Existing maximally fair selection algorithms can maximize a wide range of fairness objectives , including those considered in this paper . Leximin , Maximin , and Nash Welfare . Of the three fairness objectives we consider in this paper , Maximin and Nash Welfare ( NW ) have succinct formulae . For p , π they are defined as follows , where πi is the marginal of individual i : Maximin ( p ) : = min i∈N πi , NW ( p ) : = ( ∏ i πi ) 1/n . Intuitively , NW maximizes the geometric mean , prioritizing the marginal πi of each individual i in proportion to π−1i . Maximin maximizes the marginal probability of the individual least likely to be selected . Finally , Leximin is a refinement of Maximin , and is defined by the following algorithm : first , optimize Maximin ; then , fixing the minimum marginal as a lower bound on any marginal , maximize the second-lowest marginal ; and so on . Our task : quantize a maximally fair panel distribution with minimal fairness loss . We define a 1/m-quantized panel distribution as a distribution over all feasible panels K in which all probabilities are integer multiples of 1/m . We use p̄ to denote a panel distribution with this property . Formally , while an ( unconstrained ) panel distribution p lies in D : = { p ∈ R|K|+ : ‖p‖1 = 1 } , a 1/m-quantized panel distribution in p̄ lies in D : = { p̄ ∈ ( Z+/m ) |K| : ‖p̄‖1 = 1 } . Note that a 1/m-quantized distribution p̄ immediately translates to a physical uniform lottery of over m panels ( with duplicates ) : if p̄ assigns probability ` /m to panel P , then the corresponding physical uniform lottery would contain ` duplicates of P . Thus , if we can compute a 1/m-quantized panel distribution p̄ with fairness F ( p̄ ) , then we have designed a physical uniform lottery over m panels with that same level of fairness . Our goal follows directly from this observation : we want to show that given an instance and desired lottery size m , we can compute a 1/m-quantized distribution p̄ that is nearly as fair , with respect to a fairness notion F , as the maximally fair panel distribution in this instance p∗ ∈ arg maxp∈D F ( p ) . We define the fairness loss in this quantization process to be the difference F ( p∗ ) − F ( p̄ ) . We are aided in this task by the existence of practical algorithms for computing p∗ Flanigan et al . [ 12 ] , which allows us to use p∗ as an input to the quantization procedure we hope to design . For intuition , we illustrate this quantization task in Figure 1 , where π∗ , π̄ are the marginals implied by p∗ , p̄ , respectively . Since the fairness of p∗ , p̄ are computed in terms of π∗ , π̄ , it is intuitive that a quantization process that results in small marginal discrepancy , defined as the maximum change in any marginal ‖π− π̄‖∞ , should also have small fairness loss . This idea motivates the upcoming section , in which we give quantization procedures with provably bounded marginal discrepancy , forming the foundation for our later bounds on fairness loss . 3A panel distribution p implies a unique vector of marginals π as follows : fixing p , π , a pool member i ’ s marginal selection probability πi is equal to the probability of drawing a panel from p containing that pool member . For a more detailed introduction to the connection between panel distributions and marginals , we refer readers to Flanigan et al . [ 12 ] . all feasible panels all feasible panels Maximally fair distribution over panels ( output of LEXIMIN [ FGGHP21 ] ) Uniform lottery over m panels quantize p * p̄input output all pool members π * π̄ all pool members1/m ( ) Does there exist a uniform lottery over m panels that nearly preserves the fairness of the maximally fair unconstrained distribution over panels ? And , Algorithmically , how do we compute such a uniform lottery ? Results and contributions . After describing the model in Section 2 , in Section 3 we prove that it is possible to round an ( essentially ) arbitrary distribution over panels to a uniform lottery while preserving all individuals ’ selection probabilities up to only a small bounded deviation . These results use tools from discrepancy theory and randomized rounding . Intuitively , this bounded change in selection probabilities implies bounded losses in fairness ; we formalize this intuition in Section 4 , showing that there exists in general a uniform lottery that is nearly maximally fair , with respect to multiple fairness objectives . Although we would ideally like to give such bounds for the Leximin fairness objective due to its use practice , this objective can not be summarized by a single expression . Thus , we give bounds for the closely-related egalitarian objective , Maximin [ CELM07 ] . We additionally give upper bounds on the loss in Nash Welfare [ Mou03 ] , a similarly well-established fairness objective that has also been implemented in panel selection tools [ HG20 ] . Finally , in Section 5 , we consider the algorithmic question : given a maximally fair distribution over panels , can we find a near-maximally fair uniform lottery , as our bounds suggest should exist ? To answer this question , we implement two standard rounding algorithms , along with near-optimal ( but more computationally-intensive ) integer programming methods , for finding uniform lotteries . We then evaluate the performance of these algorithms in 11 real-world panel selection instances . We find that in all instances , we can compute uniform lotteries that nearly exactly preserve not only fairness with respect to both objectives , but entire sets of Leximin-optimal marginals , meaning that from the perspective of individuals , there is essentially no difference between using a uniform lottery versus the optimal distribution used by the latest algorithms . We discuss these results , their implications , and how they can be deployed directly into the existing panel selection pipeline in Section 6 . 2 Model Panel Selection Problem . First , we formally define the task of panel selection for citizens ’ assemblies . Let N = [ n ] be the pool of volunteers for the panel—individuals from the population who have indicated their willingness to participate in response to an invitation . Let F = { ft } t denote a fixed set of features of interest . Each feature ft : N ! ⌦t maps each pool member to their value of that feature , where ⌦t is the set of ft ’ s possible values . For example , for feature ft = “ gender ” , we might have ⌦t = { “ male ” , “ female ” , “ non-binary ” } . We define individual i ’ s feature vector F ( i ) = ( ft ( i ) ) t 2 Q t ⌦t to be the vector encoding their values for all features in F . As is done in practice and in previous research [ FGGP20 , FGG+21 ] , we impose that the chosen panel P must be a subset of the pool of size k , and must be representative of the broader population with respect to the features in F . This representativeness is imposed via quotas : for each feature f and corresponding value v 2 ⌦ , we may have lower and upper quotas lf , v and uf , v . These quotas require that the panel contain between lf , v and uf , v individuals i such that f ( i ) = v. In terms of these parameters , we define an instance of the panel selection problem as : given ( N , k , F , l , u ) —a pool , panel size , set of features , and sets of lower and upper quotas—randomly select a feasible panel , where a feasible panel is any set of individuals P from the collection K : K : = P 2 ( Nk ) : lf , v | { i 2 P : f ( i ) = v } | uf , v for all f , v . Maximally Fair Selection Algorithms . A selection algorithm is a procedure that solves instances of the panel selection problem . A selection algorithm ’ s level of fairness on a given instance is determined by its panel distribution p , the ( possibly implicit ) distribution over K from which it draws the final panel . Because we care about fairness to individuals , we evaluate the fairness of p in terms of the individual selection probabilities , or marginals , that p implies.3 We denote the vector of marginals implied by p as ⇡ , and we will sometimes specify a panel distribution as p , ⇡ to explicitly denote this pair . We say that ⇡ is realizable if it is implied by some valid panel distribution p. Maximally fair selection algorithms are those which solve the panel selection problem by sampling a specifically chosen p : one which implies marginals ⇡ that allocate probability as fairly as possible 3Any distribution over panels p implies a selection probability for each pool members : A pool member ’ s selection probability , per p , is equal to the probability of drawing a panel from p containing that pool member . 3 ( ) Does there exi t a uniform lottery over m panels that nearly preserves the fairness of the maximally fair unconstrained distribution over panels ? And , Algori hmically , how do we compute such a uniform lottery ? Results and ontributions . After describing the model in Section 2 , in Section 3 we prove that it is possible to round an ( essentially ) arbitrary distribution over panels to a uniform lottery while preservi g all individuals ’ select on probabiliti s p to only a small bounded deviation . These results use tools from discrepancy theory an randomized rounding . Intuitively , this bounded change in selection probabilities implies bounded losses in fairness ; we formalize this intuition in Section 4 , showing that there exists in gene al a uniform lottery that is nearly maximally fair , with respect to multiple fairness objectives . Although we would ideally like to give such bounds for the Leximin fairness objective due to its use practice , this objective can not be summarized by a single expression . Thus , we give bounds for the closely-relat d egalitarian objective , Maximin [ CELM07 ] . We addition lly give upper bounds on the loss in Nash Welfare [ Mou03 ] , a similarly well-established fairness objective that has also been implemented in panel selection tools [ HG20 ] . Finally , in Section 5 , we consider the algorithmic question : given a maximally fair distribution over panels , can we find a near-maximally fair uniform lottery , as our bounds suggest should exist ? To answer this question , we implement two standard rounding algorithms , along with near-optimal ( but ore computationally-intensive ) integer programming methods , for finding uniform lotteries . We then evaluate the erformance of these algorithms in 11 real-world panel selection instances . We find that in all instances , we can compute u iform lotteries that nearly exactly preserve not only fairness with respec to both objectives , but entire sets of Leximin-optimal marginals , meaning that from the persp ctive of individuals , there is essentially no difference between using a uniform lottery versus the optimal distribution u ed by the latest alg rithms . We discuss these results , their implications , and how they can be deployed directly i to the existing panel selection pipeline in Section 6 . 2 Model Panel Selection Problem . First , we formally define the task of panel selection for citizens ’ assemblies . Let N = [ n ] be the pool of volun eers for the panel—individuals from the population who have indicated their willingness to participate in response to an invitation . Let F = { ft } t denote a fixed set of features of interest . Each featu e ft : N ! ⌦t maps each pool member to their value of that feature , where ⌦t is the set of ft ’ s possible values . For example , for feature ft = “ gender ” , we might have ⌦t = { “ male ” , “ female ” , “ non-binary ” } . We define individual i ’ s feature vector F ( i ) = ( ft ( i ) ) t 2 Q t ⌦t to be the vector encoding their values for all features in F . As is done in practice and in pr vious r search [ FGGP20 , FGG+21 ] , we impose that the chosen panel P must be a subs t of the p ol of size k , and must be representative of the broader population with re pect to th features in F . This representativeness is imposed via quotas : for each feature f and corresponding value v 2 ⌦ , we may have lower and upper quotas lf , v and uf , v . These quotas require that the panel contain between lf , v and uf , v individuals i such that f ( i ) = v. In terms of these parameters , we define an instance of the panel selection problem as : given ( N , k , F , l , u ) —a pool , panel size , set of features , and sets of lower and upper quotas—randomly select a easible p nel , where a feasible panel is any set of individuals P from the collection K : K : = P 2 ( Nk ) : lf , v | { i 2 P : f ( i ) = v } | uf , v for all f , v . Maximally Fa r Selection Algorithms . A selection algorithm is a procedure that solves instances of t e pan l selection problem . A selection algorithm ’ s level of fairness on a given instance is determined by it panel distribution p , the ( possibly implicit ) distribution over K from which it draws the final panel . Beca se we care about fair ess to individuals , we evaluate the fairness of p in terms of the individual election probabiliti s , or marginals , that p implies.3 We denote the vector of marginals im lied by p as ⇡ , and we will sometimes specify a panel distribution as p , ⇡ to explicitly denote this pair . We say that ⇡ ealizable if it is implied by some valid panel distribution p. Maximally fair s ection algorithms are those which solve the panel selection problem by sampling a specifically chosen p : one which mplies marginals ⇡ that allocate probability as fairly as possible 3Any distribution over panels p implies a selection probability for each pool members : A pool member ’ s selectio prob bi ity , per p , is equal to the probability of drawing a panel from p containing that pool member . 3 Figure 1 : The quantization task takes s inp t a maximally fair p nel distribution p∗ ( implyi g marginals π∗ ) , and outputs a 1/m-quantized panel distribution p̄ ( implying marginals π̄ ) .

Summary: The paper discusses transparency in the context of sortition - selecting citizens to participate in a “Citizens’ Assembly” for the purpose of making policy recommendations. They seek to design algorithms which are approximately fair (with respect to three metrics - Maximin, Leximin, and NW) while being more interpretable/verifiable than disclosing selection probabilities. Their proposed solution is to design a uniform lottery, which selects a panel uniformly at random from a precomputed set of panels. They bound the fairness loss and deviation in marginals due to restricting to uniform lotteries, discuss LP rounding algorithms to find such lotteries, and run experiments on 11 real-world instances comparing the algorithms.