Is Sortition Both Representative and Fair?
Authors: Soroush Ebadian
Gregory Kehne
Evi Micha
Ariel D. Procaccia
Nisarg Shah
Venue: NeurIPS 2022 Accept (2022)
Keywords: computational social choice
sortition
fairness

Abstract: Sortition is a form of democracy built on random selection of representatives. Two of the key arguments in favor of sortition are that it provides representation (a random panel reflects the composition of the population) and fairness (everyone has a chance to participate). Uniformly random selection is perfectly fair, but is it representative? Towards answering this question, we introduce the notion of a representation metric on the space of individuals, and assume that the cost of an individual for a panel is determined by the $q$-th closest representative; the representation of a (random) panel is measured by the ratio between the (expected) sum of costs of the optimal panel for the individuals and that of the given panel. For $k/2 < q \le k-\Omega(k)$, where $k$ is the panel size, we show that uniform random selection is indeed representative by establishing a constant lower bound on this ratio. By contrast, for $q \leq k/2$, no random selection algorithm that is almost fair can give such a guarantee. We therefore consider relaxed fairness guarantees and develop a new random selection algorithm that sheds light on the tradeoff between representation and fairness. 


Is Sortition Both Representative and Fair?
Soroush Ebadian
Gregory Kehne
Department of Computer Science University of Toronto soroushccs. toronto......
School Engineering and Applied Sciences Harvard University gkehne@g.. harvard.
Evi Micha Department of Computer Science University of Toronto emi www.acc.tooootoooooooooooooo.oo. toronto.... edu
Ariel D. Procaccia School of Engineering and Applied Sciences
Harvard University
arielpro@seas harvard. edu
Nisarg Shah
Department of Computer Science University of Toronto nisargêcs toronto. eorarress.torontoooooooooo.o.oo. .edu
Abstract
Sortition is form of democracy built on random selection of representatives. Two the key arguments favor of sortition are that it provides representation random panel reflects the composition of the population) and fairness (everyone has chance participate). Uniformly random selection perfectly fair, but is it representative? Towards answering this question, we introduce the notion of representation metric on the space of individuals, and assume that the cost of an individual for panel is determined by the q-th closest representative; the representation of a (random) panel is measured by ratio between the (expected) sum of costs of the optimal panel for the individuals and that of the given panel. For k/2 q S (2k), where is the panel size, we show that uniform random selection indeed representative by establishing constant lower bound this ratio. By contrast, for q I k/2, no random selection algorithm that is almost fair give such a guarantee. We therefore consider relaxed fairness guarantees and develop new random selection algorithm that sheds light on the tradcoff between representation and fairness.
Introduction
Most people think of democracy synonymous with elections. But that has always been the case: from the inception of democracy in ancient Athens until the American and French revolutions, democracy had typically been associated with random selection of representatives [1], paradigm known as sortition.
These days, sortition is mainly practiced in the form of citizens' groups of people who deliberate on central questions, with the goal of generating recommendations and informing policy. The impact and prevalence of citizens' assemblies around the world have motivated computational work on how to fairly and transparently select assembly members [2- 4]. But there are signs that sortition becoming even more widely accepted, including its recent institutionalization in Belgium, where permanent sortition-based bodies are now working alongside the parliaments of the German- speaaing region and the Brussels region. In light of this progress,
cotemblley-rnnddnii selected
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

may only be a matter of time until one of the many blueprints for sortition-based democracy [5] is implemented at the level of an entire country.
The excitement about sortition is driven by several appealing qualities, which are seen as providing solutions to some of the problems plaguing electoral democracy. We briefly present two of them in the context of uniform selection, which selects a uniformly random panel and is considered to be the ideal sortition method [6].
Descriptive representation: A panel selected uniformly at random is likely to reflect the composition of the population from which it was drawn. Representation lends legitimacy to the process [7, 8), as individuals are able to identify some panelists who are similar to themselves."
Fairness: Under uniform selection, each individual has an equal chance to participate. Political theorists have argued that this quality realizes philosophical ideals like equality of opportunity and allocative justice [10].
By any reasonable measure of the fairness of selection Sitaabilities-e... the minimum selection probability of any individual [2] uniform selection achieves perfect fairness, as selection probabili- ties are equalized. We ask: Is uniform selection also representative in rigorous sense? If we had an analogous measure of representation. we would be able to explore whether this is the case. But quantifying representation poses conceptual challenge.
Our approach. We address this challenge by assuming that there exists representation metric on individuals, which measures to what degree one individual represents another (smaller distance means better representation). Readers familiar with the algorithmic fairness literature will no doubt make the connection to the similarity metric of Dwork et al. [11], which has been criticized on the grounds that it difficult to explicitly construct [12]: a major obstacle is that the question of whether certain features should be used to determine similarity is domain-speciff and tied to legal interpretation. By contrast, the prospect of constructing a representation metric is perhaps more tractable for practitioners. This is because the organizers of a citizens assembly (say) can choose it to be a function of a common set of features which are already routinely used by practitioners for the purpose of guaranteeing the representation of selection rules. Moreover, some of our main results, which pertain to uniform selection, are fully independent of the metric for these results it suffices to suppose that such a metric exists.
Note, however, that distance metric (on individuals) does not directly define a measure of the degree to which an individual is represented by panel. Following very recent work by Caragiannis et al. [13], we assume that the cost of a panel for an individual is determined by the q-th closest member of the panel, and our results are parameterized by g.
We can now define representation by taking page from the literature on distortion in social choice [14]. Specifically, for a given selection algorithm, we measure its representation via the ratio between the social cost (sum of costs) of the optimal panel and that of the panel chosen by the algorithm, in the worst case over underlying representation metrics.
Our results. Returning the question of whether uniform selection is also representative, and, more generally, the eponymous question of whether sortition is both representative and fair, our answer is that "it depends" -oon on the value of q.
When k/2 < s k - (2k), where k is the size of the panel, we show that uniform selection (which is perfectly fair) achieves constant representation. Qualitatively, we view this providing positive answers to our questions in the regime of > k/2. Note that this regime has a natural interpretation: each individual wants majority of the panel to be representative of themselves. This is especially justifiable when the panel makes decisions or recommendations through voting, which is often the case in citizens' assemblies.
By contrast, for the regime of q S k/2, we prove that any selection algorithm that chooses each indi- vidual with probability somewhat higher than g/n, where is the number of individuals, must in the worst case have representation precisely 0. This result clearly applies to uniform selection, where
Another motive for representation is epistemic: [9] and others argue that a diversity of opinions leads to better decisions.

---

the minimum selection probability is k/n, and it motivates us to consider weaker faimess guarantees. We design an algorithm, RANDOMREPLACE, which selects each individual with probability least q/n and has nontrivial representation guarantee of 1/(1 + 1) for any value of q-
Finally, we evaluate the average-case representation of uniform selection and RANDOMREPLACE on inferred metrics derived from two demographic datasets. For g > k/2, our experiments show that uniform selection achieves representation consistently greater than 55%. Even for q S k/2. in contrast to our worst-case result, uniform selection achieves good average-case representation. These results suggest that, in practice, uniform selection may be a good choice for all q-cost distance functions. While RANDOMREPLAN representation (at the expense of fairness), its advantage is small for values of q around k/ and close to k which means it does not offer a good representation-fune -inttaatiooo-inesssss fairness tradeoff in those regimes. Lastly, for both algorithms, we observe a spike in representation at q = k/2, which demonstrates that the chasm between provable guarantees for g S k/2 versus g > k/2 is not merely theoretical curiosity.
outperforms uniform selection across the board in terms of
Related work. The design of practical, fair and transparent algorithms for selecting citizens' assemblies was explored in several previous papers [2-4]-two of which appeared in previous NeurlPS conferences. Assemblies are required to be representative of the population with respect to features like gender, age, ethnicity, education and geography. This is generally done by setting quotas on individual features; for example, a panel of 100 people might be required to include at least 48 men, at least 48 women, and at least 2 people who identify as non-binary. The challenge that an assembly is selected from a pool of volunteers, and this pool is typically unrepresentative of the population due to self-
bias. Uniformly random selection from the pool, therefore, would likely itself result in an unrepresentative panel. Instead, the primary selection algorithm advocated by Flanigan et al. [2] computes a distribution over comta-co panels that (roughly speaking) maximizes the minimum selection probability of any volunteer, thereby maximizing fairness subject to these hard demographic constraints. By contrast, like Benadè et al. [15] and the political theory literature, we take longer-term view: We are interested in random selection directly from the population, which is a hallmark of some plans for sortition-based democracy [5]. In addition, we take fundamentally different, and arguably more nuanced, view of representation. Indeed, our framework can theoretically accommodate considcrations of intersectionality (to what degree is a rural, college-educater man represented by rural, collge--duucttd woman?) and could also capture a more holistic analysis of the composition of the panel (to what degree is a rural, college-educated man represented by panel that includes 10 rural, college-edus delleee-ducated men and 90 urban women with no college education?). Lang and Skowron [16], Celis et al. [17], and Do et al. [18] also study the problem of creating committees assemblies in which the features of the participants satisfy some desired quotas, but in the absence of an underlying representation metric and any fairness constraints.
Our approach to evaluating representation through metric is rooted in spatial theories of voting from political theory [19, 20). The idea of measuring how poorly a panel represents an individual by the distance of the q-th closest panel member to the individual was introduced by Caragiannis et al. [13] in the context of committee elections. Finally, aiming to minimize the total misrepresentation across all people and comparing that to the optimal panel makes our representation measure the inverse of distortion in voting theory [21, 22]. In voting, it is assumed that we only have partial access to the metric in the form of voters' ranked preferences over the candidates, which are induced by distance comparisons. In contrast, our results for uniform selection use no knowledge of the underlying metric, while our other algorithms assume complete access. When selecting a single candidate as the winner, it is known that the best distortion achievable by deterministic selection is 3 (which maps to 1/3 representation in our formulation) [23]. Our setting is closer committee selection, where a committee of h candidates is selected. Here, Caragiannis et al. [13] show that when each voter measures her distance to the q-th closest committee member, there is trichotomy: the best possible distortion is infinite when S k/3, linear in the number of voters when q E (k/33, k/2]. and 3 for deterministic selection when g k.22. Sortition may be viewed randomized committee elections, in which the set of candidates is the same as the set of voters. Hence, all the positive results from Caragiannis et al. [13] carry over in the absence of any fairness constraints. However, our results show that when (perfect) fairness in selection probabilities is sought in conjunction with representation, the distortion becomes infinite (zero representation) for all q < 2 but constant distortion can still be achieved for > k/2. The idea of the set of voters acting as the set of candidates was explored by Cheng et al. [24, 25]. However, they model infinitely many voters using continuous distribution over the metric space.
a special case of
3

---

2 Preliminaries
{1. ..... f). Let N [n] be the set that indexes the underlying population. A panel P is a subset of the population. Let Sk (N) denote the set of all subsets of N of size k. (We omit N when it is clear form the context.) The population lies in metric space with distance d. which we view as the representation metric discussed earlier. For each i. j E N, d(t, j) denotes the distance between and j: d is metric if the following properties are satisfied: (a) d(i.j) 2 0, and d(i, if and only if i = j. (b) d(i,,)) = d(j, i), and (c) for each i,j. E N, d(i, + d(e,j) 2 d(i, f). The last property is known as the triangle inequality. An instance of our problem is given by the underlying population along with distances as defined by d: hereinafter, we simply denote such an instance by d.
For all t N, define [t]
0
Given a panel P of size and a positive integer q E [b), the q-cost of individual for P. denoted by (i, P: d), is equal to the distance of i from her g-th closest representative in P. Note that for 1, we have ei (i, P; d) = minjer d(i..)) j) and for q = l we get Cx P:d) = max d(i. j). Let top, (i. P: d) be the set of q closest members of P to (with ties broken arbitrarily). The q-social cost of panel P is given by (P;((:d) individuals. (Observe that if q then (P; d) is the standard k-medians clustering objective evaluated for centers P.) We omit d from the notation when it is clear from the context.
Eien Cg d.) i.e., the sum of the q-costs over all
In this setting, selection algorithm At.g parameterized by k and takes as input the metric d and outputs distribution over all panels of size k. We are especially interested in the uniform selection algorithm, denoted by Udg that always outputs uniform distribution over Sg, regardless of In other words, does not take into account the underlying metric space or q. but instead outputs committee of size k chosen uniformly at random. We now formally introduce the faimess and representation of a selection algorithm.
Fairness: One appealing property of uniform selection is that each individual is selected to be part of the panel with probability exactly equal to k/n, i.e., Pr[i E Un] = k. particular, all individuals have equal chance of being chosen. We call this property perfect fairess; in general an algorithm Ak.q provides perfect fairness when for each instance d, it ensures that minge Pr(t E Ax.q] = k/n.
When perfect fairness is too restrictive, we relax this constraint by allowing individuals to be selected with probability less than k/n. In this case, the fairness of an algorithm Ab. is the worst-case ratio of the minimum probability of an individual to be selected by the algorithm and the ideal selection probability of k/n. Formally,
(Ak..) inf mingeN Prii E Ak. k/n
Representation: The other key property we would like to measure is representation. To do so, consider a panel to be a good representative of the whole population when its q-social cost is not much larger than the best possible. In other words, a selection algorithm Ak.g that outputs a distribution over the different committees of size k provides good representation when the expected q-social cost of the panel is similarly small. More formally, we define the representation of a selection algorithm At.g as the worst-case ratio of the minimum possible social g-cost of any panel and the expected g-social cost of the panel chosen by As.g over all possible instances; i.e.,
repr, (At.g(A inf pri (P':
(c)
3 Representation with Perfect Fairness for q > k/2
We begin by considering the case that g > k/2. We show that in this case uniform selection is asymptotically optimal with respect to representation among all selection algorithms that are perfectly fair. Moreover, the representation of uniform selection is constant for any q c k for 1/2 < < 1.
Theorem 1. For > k/2, uniform selection satisfies repr, (epr,(24)) 2 1-9+1
A crucial property that we exploit in this section is that the g-costs of the individuals satisfy the triangle inequality when g > k/2. This observation was first made by Caragiannis et al. [13]; we present the lemma below for completeness.

---

Lemma 1. For > 2/2, individuals i,J E N, and panel P. (ydi, + cgli.Pdd) 2 d(i,)).
Proof. Let T; top, (i, P) and T top, (i, P) be the g closest neighbors of and j. respectively, in the panel P. As [ = [T}] > k/2, 2, there exists an individual h: T, nTz. Therefore,
d(i j) - d(i, k) + d(k,j) S (t P) + cy
We this observation to lower bound the social cost of the optimal committee. Lemma 2. For > k/2, the q-social cost of the optimal panel P is at least
Σ :P;(P": Z 2(n EN (EN(())
1
d(i,g).
Proof. By applying Lemma 1 for all pairs of individuals (i, we get
Σ Σ +co
Σ
Σ
d(i.)).
iEN JEN((
JEN(()
The q-cost of each person appears exactly 2(n - 1) times on the left hand side. Thus,
(P";
Σ
C1, PT) 2 2(n 1)
Σ
Σ
d(i,5).
iEN
IE JEM((G
We are now ready prove the theorem.
Proof of Theorem 1. For any committee P of size k
Cg (i.
min d'E(g.....))
(i,
Σ (i,
q
d'Elg,k)
Σ
(i, P)
Σ E(i,))
h
('E(IAA]
JEP
where in the first inequality the minimum is upper bounded by the average. Therefore, the expected social cost of uniform selection is at most
(45C = E
Epaet legli, P)]
iEN
Ep-zi
Σ d(
h:
iEN
jEP
d(i,j)
Pre (i E Pl
k q +1
SEN JEN((()
Σ
Σ
d(i,j)
n
iEN JEN((O
By Lemma 2 and the upper bound shown above, we have
repr,(Ufs) 2 2(-11) 1J EiEN EjEN(()) d(i.g) x-9+I
1
k
q
1 k
+1
K EIENEEENN d(i,j) 2 n
2
k
In the proof above, the only property of uniform selection we use is that the marginal probabilities are equal to k/n. Hence, this lower bound also holds for any perfectly fair selection algorithm.
We next establish an upper bound on the representation any perfectly fair selection algorithm. shows that the lower bound of Theorem is tight up to factor of 4.
Theorem 2. For any > k/2, every selection algorithm Ar, with Sairness(Ae..) = 1 satisfies
repr, (Ax,) 2 8-9+1
5

---

Proof. First, note that if k+1 the statement trivially holds, since repr, ( 2. R-q+l
k+1 which is true for any algorithm. Thus, we assume q k+11 Consider an instance with n = k + individuals where h q + individuals are located at 0 and q individuals are at 1, denoted by No and N1, respectively. Any committee size leaves one person out of the committee, and for each individual i E N, this happens with probability of
2
2
Pro-Ac, (8 $ PJ Prp-An. E
A+I
Individuals in No will always have q-cost of 1, because k 11 k+1 < q individuals are located there. Therefore, Er-AA. E EiEN, Cg(i, P)] |Nol- For individuals N1, their g-cost is 1 if and only if less than individuals are selected from N1, i.e., the single person left out of the committee is located at 1. This event happens with probability
Pr [UEN, (i & PJ]= EVEN, Pre~Ar. ( PI
[Nd k+1
where the first equality comes from the fact that the events are disjoint (which holds because any committee leaves out exactly one individual). Therefore, Ep-Ax. [SiEN, c, P)] = [N R+I and the expected social cost of any perfectly fair algorithm is
[Nd]
E (P; No! + [M]
q k+1
No.
1 |N
The optimal committee would leave out person from No and achieve a social cost of Nol. Therefore, the representation of any algorithm with perfect fairness is most
[oo
|.Vol
k -- +1
I
N
+
|
NNo
IN
k+1
4 Representation with Relaxed Fairness for q S k/22
In stark contrast to the case of g > k/2. uniform selection and, more generally, any perfectly fair selection algorithm, cannot obtain bounded representation when k/2, In fact, the following theorem shows that selection algorithms with fairness strictly more than 2+(k ql (which itself is upper bounded by (2q 1)/k) suffer from unbounded representation. The proof is in Appendix A.
&
Theorem 3. For S k/2, > O, and any selection algorithm Ar. with fairness (As.g) 2
(+ mod q) (epr,(AA.ur
k
Although bounded representation is not feasible with fairesss slightly larger than g. we design a selection algorithm that, given an a-representative panel, can achieve representation of at least v+I with faimess of z. In particular, starting from the optimal panel (a 1), it achieves 9+T representation. Although finding the optimal panel given the metric space an NP-hard problem for most metric spaces, computing a constant-factor approximation (i.e., constant a)) is feasible in polynomial time [26], we explain in Section 5.
Our Algorithm, RANDOMREPLACE, is given in Algorithm 1. starts from a panel P. randomly selects panel S of g individuals, and replaces individuals in P with individuals in S as follows. First, individuals in S n remain in the final panel. Then, for i E S \ swap with one of its g-closest neighbors in the optimal panel top,(1. P) that has not been replaced by the algorithm yet. The next theorem establishes the fairness and representation guarantees RANDOMREPLACE.
Theorem 4. For any q E [k] and any panel P with repr, = a, we have that repr, repr, (RANDOMREPLACCE, Z 9+I
a and fairness RANDOMRELLOEE
g
Proof. Let S N be a set of size g chosen uniformly at random. We denote with Ps the panel that is returned from the algorithm. First, we show that Line 4 of the algorithm is valid. The algorithm reaches this line when considers i therefore is not included in P. Hence, top, (i, P) \ cannot be empty since top,(i. P) q. SS| and there exists i in S but not in top, top, P).
Ps. meaning that i not included in the panel Ps and

---

ALGORITHM 1: RANDOMREPLACE,
Input: Panel with repr, (P)
Output: Randomly selected panel by replacing most individuals of P
1: Pick E Sy uniformly at random
2: Set Ps <- P and 5 + - P
3: for E S do
4: Pick an arbitrary j: E top, (8, PS
5: Ps P (i.
6: end for
7:
return Ps
We see that every individual in S' is included in Ps as in Line 5 the algorithms ensures that each such individual included and never excluded afterwards. Hence, as each individual is chosen in with probability at least g/nt, we can see that Bairness(RANDORR) ANDOMRELLACE,, 2 g/k. Now, we prove that for any S Sy and any individual i' E N.
Cg(i', Ps) S cg(t', P) + max cg(i, P) (f Cc,(,. iES
SES
The second inequality holds the maximum is at most the sum. Therefore, we focus on the first inequality. If ci (i', , Ps) S (i, then it trivially holds. Otherwise, Cg (i', Ps) > Ci', (i',1 P). In this case, we can show that there exists some je E top, , P). First, note if for each E S P", je does not belong in top, (z', P), then it is not possible that Ps) > (i', P), since top, (i', ()',P) P) C Ps. Next, suppose for contradiction that for every that was included in Ps when j: E top, (i'P) was excluded from it, it holds that d("",)
S P such that d(i', i) > Cg (g((, Ps) and
i) < (i', P). Then, in Ps there are top,, (i', P) Usesy (((x) noop, (i', P))] individuals that have distance at most Cy ( P) Cgli', Ps) from i' and {i E S P jr E top, (t', P)H individuals that have distance less than (d Ps) from 1'. Note that
U ((ii) top,, (i', P)) = Ki P: E top, P))
and hence we get that there are least top, (i', q individuals in Ps with distance strictly less than Cg (i', Ps) from i', which is contradiction.
From the above observation, we have that
(i, Ps) d(i', d(f'.j)) +(i, (V, (1, S (i,,,, P) max c P)
iES
where the penultimate inequality holds because j. (i', and j. E (i, (i, P) from the definition of js- This proves (1).
Summing (1) over all E N, we have (Ps) S SC2(P) + n Eies (g, P). Taking the expectation this equation with respect to S. and using the fact that Pr(i E S] = g/n, we have
E[SC, (C) n Σ E c... (q + 1) SC))))
n
that is, g+I [ESC < (P). Combined with the assumption that P is a-representative, i.e.. SC₂ min SC₂ (P'), get repr, (BR. (P)) #
The next theorem shows that when g S((k), RANDOMREPLACE, attains an asymptotically optimal representation-fainne fairnesss tradeoff.
Theorem For any S k/2 such that l mod g = 0, every selection algorithm Ak.g with fairness(At..) 2 g/k satisfies repr, (Atg) S k/q².
Proof. Let m k//g- Consider an instance with n > 2k individuals on the real line, where one individual in is located at 0, [n/m] people are at 1, and at least [n/m] people are located at
7

---

K each position j E (2......, m). This way, there are at least n/m 1= (n/k) 9 2 2q - individuals located at each j E [m]]
Any optimal panel would include individuals from each position j E [m]] which results in a g-cost of 1 for and a q-cost of 0 for the rest. Hence, (P*)
However, any selection algorithm with fairness of at least g/k selects in with probability of at least q/ When in is selected, there must exist a group j E [m]] from which the algorithm selects at most g - people, incurring a g-cost of 1 for at least n m people (person ip and at least n//m- - people at position j). Hence,
q ng. q²
E| (Av.a)
n
k
which completes the proof.
From the above theorem, it follows that Algorithm achieves the highest possible fairness of q/k: subject to positive representation.
Experiments
Next, complement the worst-case analysis of the selection algorithms we have considered so far, conduct an empirical comparison between the average-case q-social cost and representation (repr,) ,) of the different selection methods.
Datasets. Data on the metric-structure preferences of groups in their full richness are difficult to come by, but it is reasonable expect that the extent to which individuals feel well-represented by one another is at least partly a function of their relative characteristics along some observable features. We therefore begin with two datasets that express the characteristics of populations along range of features, and randomly construct synthetic metric preferences from these feature signatures.
Adult dataset. Our first source of demographic data is the UCT Adult dataset, which was derived from the 1994 Current Population Survey of the US Census Bureau, and is made available by the UCI Machine Learning Repository under CC BY 4.0 license [27]. It contains range of demographic variables principally related to employment. Our experiments do not require Adult to be representative of any actual population, nor should this an assumption be made lightly [28]. For Adult we choose the features workclass, education on, marital status and sex. Adult contains 30162 individuals with values for each feature, who may be viewed distribution over the 721 unique feature vectors which they collectively hold.
ESS dataset. Our second source of demographic data is the European Social Survey (ESS) [29], which made available by the Norwegian Centre for Research Data under a CC BY 4.0 license. We use the ESS Round 9 (2018) data, which consists of 46,276 people in 27 countries, and contains 1445 features regarding socioeconomic demographic, political beliefs, geographical region, house-hold composition, personal values, media use and trust, etc. Most of the features are country-specific, which leaves roughly 250 features available per country, while each country has between 78 1 and 2745 entries (with a mean of 1713). Each entry is assigned an analysis weight which is aimed to correct the differential selection probabilities. In contrast to our experiments with Adult. we use all of the available features available in ESS. We report the experiment results based the data of the United Kingdom (2204 entries). Similar results were obtained using the data from other countries.
Metric construction. We choose some set of features along which to evaluate the individuals in the population. These are the features which will inform our metric. For each feature 8, F the set of possible values that this feature can take. Each individual is then represented as vector of feature values, where f is the value that i has for feature s. For categorical features (c.g., marital status, sex), we define d) jis) = If; f/l/ h1///maxyr,, ,f' lff - ff, where the normalization makes d(i. j: 8) E [0, 1). Then, for each feature sample weight w, UOO 1] uniformly at random from the interval [o, 1]. Finally the distance between individuals and j and our metric d is defined to be d(i,j) == Σ. Ww s))
I{ }j}. For range (e.g., income) features we define
For the empirical evaluation of our selection algorithms on these randomly generated metrics, we suppose that this is in fact the distribution for a population large enough that there are least individuals with any given feature vector. As these metrics represent populations, the q-social costs in Figures| and 2 are normalized by population size. Figures! to 3 depict data averaged over 100 random metries constructed in this manner, and the error bars show the standard error of the mean.
8

---

opt
opt unii seleect
3.5
1.4
2.3
2.5
12
2.0
8 10-
1.5
1.5
1.
0.5
Q-15
0.0
(a) UNIFORMSELECTION
(b) OPTPROXY
Figure 2: OPTPROXY finds the optimal panel (C,(P*. d) on small instances k = 10, based on Adult.
Figure 1: The q-social cost of UNIFORMSELECTION and OPT-
PROXY for E [1,20] and a selection of q. based on Adult.
Proxy for the optimum. In evaluating UNIFORMSELECTION and RANDOMREPLACE we use proxy for the optimal g-social cost, since and l are too large to support finding SC₃ (P" d) exactly, This selection algorithm OPTPROXY (indicated as "opt_apx" in legends) an implementation the fault-tolerant metric k-medians algorithm of Kumar et al. [26], which guarantees constant-facton metric k-medians algorithm as a primitive; we implement the local search algorithm of Arya et al. [30] with single swaps. More specifically, OPTPROXY works as follows. finds the (approximately) optimal |mediaan solution and picks individuals from each of these [k/g] positions, and then selects the remaining mod q people uniformly at random.
1P".d. approximation to SC, (P, .d). This algorithm uses a constant-
To evaluate the fidelity of OPTPROXY we compare it with SC, d) for 100 metrics d constructed by drawing 30 randomly chosen feature vectors from the supports of Adult instances described above. For panels of size k = 10 and all values of we find that OPTPROXY recovers SC,, [P* exactly (Figure 2). The fact that the g-social cost attained by OPTPROXY for fixed g and g E (k/2, appears constant, on the other hand, is not universally true of SC, (P*.d). .d. These plateaus are due to the way OPTPROXY selects panels by returning & copies the optimal 1-median location. Since P need not be of this form, it is interesting that exhibits this same step-like behavior in Figure 2.
Experimental results. In Figure la, we see UNIFORMSELECTION behaving as expected. For fixed k. the q-social cost of a uniformly random panel is reliably higher for larger q. and for fixed it decays smoothly as we increase the panel size Under the same conditions, OPTPROXY behaves similarly (Figure 1b) but the decay is not as smooth due to the reasons explained above. Here, all of our approximately optimal panels Pg start the same value of SC, (Pg: d) when = and decay as q remains fixed and the size of the panel & increases.?
Figure shows the ratio between the q-social cost obtained by different selection algorithms and the q g-social cost of OPTPROXY, which is approximation of their representation, averaged over 100 ran- domly generated metrics. Specifically, we evaluate UNIFORMSELECTION and RANDOMREPLACE (indicated unni_sselec' and hand_replace_r"" in legends, respectively), the latter being a general- ization of R ANDOM EEPLACE, that replaces r individuals instead of in the input panel and achieves representation at least a a/(r+ + 1) (when the input panel has representation We compare these algorithms for a range of and g. on the Adult dataset (Figure 3a) with panels of size k 20 and on the ESS (Figure 3b) dataset with panels of size 40.
and fairness at least r r/k.
Representation of UNIFORMSELECTION. As expected, the representation of UNIFORMSELECTION is the lowest for all k and q. However, it still achieves consistently good representation across the board, with representation dipping below 50% only on the Adult dataset with close to k.
We find it especially interesting that, on both datasets, UNIFORMSELECTIR sentation the regime of y S k/2, as this contrasts with our worst-case result. We conclude that, in real-world datasets, UNIFORMSELECTIN may be quite representative even with respect to the
exhibits good repre-
"This first property is to be expected: when = A and points in the metric may appear on the panel multiple times (as is the case with these distributions), an optimal panel P consists of & copies of a (carefully chosen) single point in the metric. For these panels Pq. OPTPROXY is simply finding that point.

---

10
0.8-
0.88
0.6-
selest opt_apx
0.7 snie
0.5
replace
rand_ rend_replace__2
rand replace_5
0.6
rand, replace_
rand replace
replace_
16
20
(a) Adult with 20.
(b) ESS UK with & 40.
Figure Comparison of different algorithms for fixed k. where RANDOMREPLACE, applied to the panel selected by OPTPROXY. The y-axis shows the average ratio of the q-social cost of OPTPROXY to that of different algorithms.
g-social cost measure when q I k/2. Overall, view these results providing empirical support for the use of UNIFORMSELECTION in practice.
Representation of RANDOMREPLACE, Figure 3 shows that RANDOMREPLACE achieves significant improvements in representation compared UNIFORNSSEECTTIN for small values of r- at the cost of fairness, of course. This tradeoff may be justified for some values of q, but the advantage of RANDOMREPLACE is much smaller around g = k/2 and when q approaches k, so for these regimes RANDOMREPLACE should not be in the rumning.
Shift at q = k/22. both Figures 3a and 3b, there is a sudden improvement in the representation of UNIFORMSELECTION
k/2. This provides empirical support for the peculiar transition in the q-social cost occurring q k/2, which predicted by the theory and is the reason we separated Sections 3 and 4.
6
Discussion
Our results in Section show that in some cases, relaxing fairness requirements allows improving representation dramatically. More generally, it is interesting understand the tradeoff between representation and fairness, and to chart the Pareto frontier. In Appendix B, we take some first steps in this direction. One observation is the aforementioned generalization of RANDOMREPLACC that replaces r individuals instead of We also show that a random-dictatonn wwreiddiiicttttiiiiiee nontrivial fairness and representation guarantees in the regime of q k/2. However, there remain several open questions: for example, when < k/2, what level of fairness can be achieved if we seek constant representation?
algorithm gives
We focused our attention on the g-cost formulation of Caragiannis et al. [13], in which each individual measures their distance to the q-th closest panel member. One analyze the perrseettaioon-aimmee tradeoff with other cost functions. For example, what if different individuals use different values of g? Another appealing choice is when each individual measures the average distance to all panel members: in Appendix C, we show that for this cost function, uniform selection achieves representation of at least 1/2.
More broadly, one can use measures of representation other than the total cost to all individuals. For example, one may wish to use selection algorithm that minimizes the maximum cost to any individual, or strikes a balance between maximum cost and total cost. We hope that answers to some of these questions will lead to better understanding of the strengths of sortition, and new ways of realizing this democratic paradigm.
Acknowledgementss. This work was partially supported by the National Science Foundation under grants IIS IIS-21477 1477177, CCF-2007080, IIS-2024287, and CCF-1733556; by the Office of Naval Research under grant 1-12----------- and by NSERC Discovery Grant.
10

---

References
[1] D. Van Reybrouck. Against Elections: The Case for Democracy. Random House, 2016.
[2] B. Flanigan, P. Gölz, A. Gupta, B. Hennig. and A. D. Procaccia. Fair algorithms for selecting citizens' assemblies. Nature, 596:548-553 2021.
[3] B. Flanigan, P. Gölz, A. Gupta, and A. D. Procaccia. Neutralizing self-selection bias in sampling for sortition. In Proceedings of the 34th Anmual Conference Neural Information Processing Systems (NeurIPS), pages 6528-6539. 2020.
[4] B. Flanigan, G. Kehne, and A. D. Procaccia. Fair sortition made transparent. In Proceedings of the 35th Annual Conference on Neural Information Processing Systems (NeurIPS), pages 25720-25731, 2021.
[5] Gastil and E. O. Wright, editors. Legislature by Lot: Transformative Designs for Deliberative Governance. Verso, 2019.
[6] F. Engelstad. The assignment of political office by lot. Social Science Information, 28(1):23-50, 1989.
[7] J. M. Parker. Randomness and Legitimacy in Selecting Democratic Representatives. PhD thesis, University of Texas Austin, 2011.
[8] Fishkin. Democracy When the People Are Thinking: Revitalizing Our Politics Through Public Deliberation. Oxford University Press, 2018.
[9] H. Landemore. Deliberation, cognitive diversity, and democratic inclusiveness: An epistemic argument for the random selection of representatives. Synthese, 190:1209-123,, 2013.
[10] P. Stone. The Luck of the Draw: The Role of Lotteries in Decision Making. Oxford University Press, 2011.
[11] C. Dwork, M. Hardt, T. Pitassi, O. Reingold, and R. S. Zemel. Fairness through awareness. In Proceedings of the 3rd Innovations in Theoretical Computer Science Conference (ITCS), pages 214-226, 2012.
[12] M.-F. Balcan, T. Dick, R. Noothigattu, and A. D. Procaccia. Envy-free classification. In Pro- ceedings of the 33rd Annual Conference on Neural Information Processing Systems (NeurIPS), pages 1238 1248, 2019.
[13] Caragiannis, N. Shah, and A. Voudouris. The metric distortion of multiwinner voting. Artificial Intelligence, 313:103802, 2022.
[14] E. Anshelevich, A. problems: The first 15 years and beyond. Proceedings of the 30th International Joint Conference Artificial Intelligence (IJCAI) (Survey Track), pages 4294 4301, 2021.
N. Shah, and A. A. Voudouris. Distortion in social choice
[15] G. Benadè, P. Gölz, and A. D. Procaccia. No stratification without representation. In Proceedings of the 20th ACM Conference on Economics and Computation (EC), pages 281-314, 2019.
[16] J. Lang and P. Skowron. Multi-attribute proportional representation. Artificial Intelligence, 227: 74-106, 2018
[17] I.. E. Celis, L. Huang, and N. K. Vishnoi. Multiwinner voting with fairness constraints. In Proceedings of the 27th International Joint Conference on Artificial Intelligence (IJCAI), pages 144 151, 2018
[18] V. Do, J. Atif, J. Lang, and N. Usunier. Online selection of diverse committees. In Proceedings of the 30th International Joint Conference on Artificial Intelligence (IJCAI), pages 154-160, 2021.
[19] James M Enelow and Melvin Hinich. The spatial theory of voting: An introduction. CUP Archive, 1984.
11

---

[20] Kenneth Arrow, Advances in the spatial theory of voting. Cambridge University Press, 1990.
[21] A. D. Procaccia and J. S. Rosenschein. The distortion of cardinal preferences in voting. In Proceedings of the 10th International Workshop on Cooperative Information Agents (CIA), pages 317-331, 2006.
[22] E. Anshelevich, O. Bhardwaj, E. Elkind, J. Postl, and P. Skowron. Approximating optimal social choice under metric preferences. Artificial Intelligence, 264:27-51, 2018.
[23] V. Gkatzelis,
D. Halpern, and N. Shah. Resolving the optimal metric distortion conjecture. In Proceedings of the s Symposium Foundations of Computer Science (FOCS), pages 1427-1438, 2020.
[24] Y. Cheng. S. Dughmi, and Kempe. Of the people: voting is more effective with representative candidates. In Pruceedings of the 18th ACM Conference Economics and Computation (EC), pages 305-322, 2017.
[25] Y. Cheng. S. Dughmi, and Kempe. On the distortion of voting with multiple representative candidates. In Proceedings of the 32nd AAAI Conference Artificial Intelligence (AAAI), pages 973-980, 2018.
[26] N. Kumar and B. Raichel. Fault tolerant clustering revisited. In Proceedings of the 25th Canadian Conference on Computational Geometry, CCCG 2013, Waterloo, Ontario, Canada, August 8-10, 2013. Carleton University, Ottawa, Canada, 2013.
[27] D. Dua and C. Graff. UCI machine learning repository, 2017. URL http:///archiveeiess uci. edu/ml.
[28) F. Ding, M. Hardt, J. Miller, and L. Schmidt. Retiring Adult: learning. In Proceedings of the 34th Annual Conference on Neural Information Processing Systems (NeurlPS), pages 6478-6490, 2021.
New datasets for fair machine
[29] ESS Round 9: European Social Survey (2021): ESS-9 2018 Documentation Report. Edition 3.1. bergen, european social survey data archive, nsd norwegian centre for research data for ESS ERIC. 2021. doi: 10.21338/NDD-ESS- 1 2018. URL https: www.eerrrooassocaasssveeeee org/data/.
[30] V. Arya, N. Garg, R. Khandekar, A. Meyerson, K. Munagala, and V. Pandit. Local search heuristics for k-median and facility location problems. SIAM J. Comput., (3(3):544-566) 2004.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you deseribe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [N/A]
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [Yes]
12

---

(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [N/A]
4. If you using existing assets (e.g., code, data, models) curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you'le 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "q1VmeoMfFjE", "reviewer": "Reviewer_MDMp", "summary": "The paper addresses the problem of sortition, which is a democratic process consisting in randomly selecting a subset of citizens from a population. There are two important criteria in this selection process: *fairness*, meaning that citizens should have an equal chance to participate, and *representation*, meaning that the chosen panel should well represent the large population. The authors introduce a representation metric which is inspired by distortion-based analysis of voting over metric spaces, and which depends on a parameter $q$. Depending on the value of $q$, the paper provides results on whether a perfectly fair algorithm achieves good representation. In the regime where there is a necessary fairness-representation trade-off, an algorithm with guarantees on the two criteria is presented.\nThe results are illustrated with experiments on the Adult dataset, as well as the European social survey.\n", "questions": "* Can you please respond to the concerns raised above? \n* Related to the main weakness I described, what are the approximate runtimes of the algorithms? What happens for larger values of $k,N$, e.g. $k=100$ as for citizens assemblies?\n\n\n", "limitations": "The social context of this work is well described.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The paper is clear, well-motivated and well-written.\n* The idea of measuring representation based on the distance of an individual to the $q$-closest member is clever and intuitive.\n* The theoretical results are not technically difficult, but they are intuitive, simple to understand and definitely interesting.\n* The experimental evaluation is convincing enough. \n* I appreciated that the authors anticipated the need for a better understanding of the fairness-representation trade-off in the Appendix.\n\nWeaknesses:\n* Algorithm 1, Line 1: There is no explanation on how to compute the minimizer over all possible subsets of size $k$ among the set of size $N$. The minimization problem is likely intractable; therefore the authors should describe a computationally efficient method to solve this. I found no details in Section 4, neither in the experimental section, nor the Appendices. \n\nMinor comments:\n* Topic: The paper is not related to machine learning: neither the techniques, nor the application problem. However, even if the paper falls outside NeurIPS usual topics, I do think sortition is an important problem.\n* Additional references on representation constraints when selecting a panel of representatives from a set of candidates (if you think they are off-topic for your work, please explain why). In particular, [2,3] also use a representation metric in that context. \n\n[1] L. Elisa Celis, Lingxiao Huang, and Nisheeth K. Vishnoi. 2018. Multiwinner voting with fairness constraints. In Proceedings of the 27th International Joint Conference on Artificial Intelligence (IJCAI'18). AAAI Press, 144–151.\n\n[2] Lang, J., & Skowron, P. (2016). Multi-Attribute Proportional Representation. Proceedings of the AAAI Conference on Artificial Intelligence, 30(1). https://doi.org/10.1609/aaai.v30i1.10024\n\n[3] V. Do, J. Atif, J. Lang, N. Usunier. Online Selection of Diverse Committees. IJCAI 2021, Aug 2021, virtual, Canada.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "MOYgn8tciTM", "reviewer": "Reviewer_fAHs", "summary": "The authors study how representative are panels chosen via sortition, i.e., by selecting randomly individuals from a population. The authors measure representation as follows. First, they assume that there is some metric that measures similarity between members of the population and assume that each person would like to be represented by someone as similar as possible. Then, they say that the q-cost of a panel for a person is the distance of the q-th closest panel member from that person (according to the metric). The cost of a panel is the sum of the q-costs. Finally, representation is the worst-case ratio between the minimum panel cost and the expected one (we look at the expectation because panels are drawn by a randomized algorithm). The authors show that if q is greater than half of the size of the panel then good representation values are possible, and otherwise they are not. ", "questions": "Why is it important to measure the distance between the candidates, instead of assuming that each agent can give a boolean value indicating if another person does or does not represent him/her?", "limitations": "I do not believe this question really applies to the theoretical part of the paper. For the experimental part, it is not clear to me what the actual point is, so I cannot really answer.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- theoretical results are nice and complete\n\nWeaknesses\n- The added value of the paper, as compared to previous publication, is nice but not as large as one might expect\n- The value of the experiments is very unclear. I could not really tell what the point of the experiment is. The authors crunch some numbers and show plots, but I am missing actual high-level conclusions that go beyond analyzing the shapes of the plots\n- Plots in Section 5 are missing any sort of standard deviation/variation information\n\nMinor issues:\nl. 99: \"99 urban\" --> \"90 urban\" ?\nIt might be useful to explain in lines 143-151 that correlations in the randomized selection process are allowed (assuming they are).\nl. 221: \"that that\"\nl. 222: \"individuals\" -> \"individual\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "D5Fnuivvxf2", "reviewer": "Reviewer_6NwB", "summary": "The paper considers the fairness and representativeness of sortition, a randomized method of group selection. Typically sortition involves uniformly random selection of k participants. This work asks whether other methods of selection may lead to a more representative panel with minimal impact on fairness. Fairness here is the goal of selecting each participant with at least probability k/n while representation is the ability of selecting a panel such that each non-member is similar to a member. In particular, they make use of the \"q-social cost\" metric -- the sum of distances from each individual to their q-th closest panel member.\n\nThe paper shows that when q > k/2 uniform random selection provides optimal representation. For smaller q the paper shows that good representation is not possible with only a tiny loss in fairness but develops an algorithm that modifies a low social cost panel to improve representation. Experiments are performed showing the efficacy of the algorithms they have developed. For both Uniform Selection and RandomReplace, several examples are given demonstrating various q-social costs for different values of k and q.", "questions": "1. Is it the case that RandomReplace is meant to provide a good balance of fairness and representation as compared with the optimal panel? Assuming so, is that evident in the experiments? I am unclear on what the experiments say about RandomReplace other than that it is (quite understandably) not as low-cost as the optimal panel.\n\n2. Section 3 begins by alluding to multiple algorithms that are perfectly fair. Are there multiple such algorithms? Aren't they all necessarily equivalent to Uniform Selection?", "limitations": "The authors have provided only very little discussion of the potential societal impact of their work. However, the work relatively far removed from meaningful real-world impact; I do not feel such discussion is missing, only absent.", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I find the idea of the q-social cost, once I got some intuition for it, quite a fitting idea for measuring the quality of a panel formed via sortition. Focusing on metrics beyond just fairness is useful and relatively novel in sortition. The paper is generally quite well written although it would be nice to have a little bit more emphasis on giving natural, intuitive explanations of your definitions of representation (primarily) and fairness (fairly clear already). I found the proofs of the theorems in the main paper generally clear and they seem correct. \n\nDue to the simplicity/natural usage of the Uniform Selection algorithm adding to theoretical knowledge on the mechanism is a useful contribution.  The RandomReplace algorithm is interesting although it seems destined to remain theoretical; finding an optimal panel is obviously a very complex task in a real-world setting. Discussing the importance of the optimality of the panel would be a useful contribution -- e.g. I'm unclear on what properties you would expect if you applied RandomReplace to a uniformly selected panel.\n\nI greatly appreciate the experiment section and some visualization of how q-social cost changes with k and q. Some parts of the experiements section could be clarified (see below).\n\n\nMinor Issues\n\nLine 127: Should it read \"denotes the distance between i and j, and...\"?\nLine 209: Is P* only defined in Algorithm 1? Mentioning it in text here or in the Preliminaries would be ideal.\nLine 218: \"The algorithm reaches this line when *it* considers ...\"\nLine 224: This is the only place you say \"agent\"; consistency in terms is ideal.\n\nExperiment section:\nIs there a reason not to standardize the y-axis? It makes comparison between plots much simpler\nWhat is opt_apx?\nIn general, more \"prettified\" legend terms would be better.\nIn Figure 1c is opt directly under opt_apx? This may be what you mean on lines 297-298 but it could be more clear.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept