Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper discusses transparency in the context of sortition - selecting citizens to participate in a “Citizens’ Assembly” for the purpose of making policy recommendations. They seek to design algorithms which are approximately fair (with respect to three metrics - Maximin, Leximin, and NW) while being more interpretable/verifiable than disclosing selection probabilities. Their proposed solution is to design a uniform lottery, which selects a panel uniformly at random from a precomputed set of panels. They bound the fairness loss and deviation in marginals due to restricting to uniform lotteries, discuss LP rounding algorithms to find such lotteries, and run experiments on 11 real-world instances comparing the algorithms.
main_review: The paper is well-written, clearly explaining the problem and techniques used. The theoretical results are basically applications of known randomized rounding algorithms over a natural linear program for this problem. The empirical results seem to indicate that imposing their definition of transparency is not harmful in terms of performance, and their methods seem implementable in practice. I liked the premise of the paper to make randomization more "transparent" by preselecting a set of $m$ panels, each of which could be a feasible panel, and then providing a uniform lottery on these to make the process look transparent. The main drawback to the paper is that the theoretical contribution is small—the LP rounding techniques used are not quite original. 

- I wonder though, if the set of m panels that are computed could be modified in a way to actually bias any uniformly selected panel. What's the guarantee that the $m$ panels selected are all possible panels? In fact, moving the upper and lower bounds slightly could have a significant impact on the kind of panels obtained by uniformly sampling the feasible set. 
limitations_and_societal_impact: Societal impact and limitations are adequately addressed.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: Additional to the maximally-fair consideration, this paper concerns the transparency in assembly selection problem. Specifically, the paper introduces the notion of transparency in panel selection as people can easy understand the probabilities with which each individual will be chosen for the panel and verify that individuals are actually selected with these probabilities. To achieve this, the paper studies the $m$-uniform lottery where the selection probability of each feasible panel must be multiples of $1/m$.
The paper shows that there exists a uniform lottery over $m$ panels that can nearly preserve the fairness of the maximally-fair unconstrained distribution over panels.  

Furthermore, the paper uses the fairness loss and marginal discrepancy to quantify the closeness of such lottery and the lottery obtained in an unconstrained (no transparency requirement) setting. The paper characterizes several upper bounds for these measures and further strengthen these bounds in more structured settings. 

Lastly, the paper conducts experiments on real-world panel selection instances to demonstrate the viability of the uniform lottery approach as a method of selecting assemblies both fairly and transparently.


main_review: Originality, quality, clarity, and significance

I should first mention that I feel I'm not a suitable reviewer for this paper, as the main topics are outside my areas of expertise – I have indicated this in my low confidence for my score. Although I cannot judge its novelty adequately, the problem studied in this paper is well motivated and exhibits solid applications in real-world. The theoretically results around the problem setting appear correct and the experiments seem to be comprehensive. 
In terms of the presentation, the paper is overall well written. I would like to appreciate author's efforts in introduction section to make the motivation of the problem well defined. 

Questions and comments: 

1. are there any connections between $|\mathcal{K}|$ and $m$? I'm a bit confusing here, as in Line 90, you mentioned "uniform lottery over $m$ panels", and $\mathcal{K}$ is the set of all feasible panels by definition in Line 129. 

2. I found it a bit difficult to parse the definition of Maximin, it might simply because that I'm not in this area. Seems that Maximin can not bound the difference of marginal of individuals, which is usually the focus of individual fairness. Could author explain this a little bit? 

3. The fairness objectives considered in the paper are more from the point of view of individuals. I'm wondering whether the current results still hold if you replace the objectives to a group fairness measure? 

4. Does $k\le m$ hold for sure? If $k>m$, Theorem 3.2 seems to be meaningless?  


Others:

Line 248: "be most natural start with" -> starting

Line 281, you probably want to unify Nash-Welfare optimal and NW-optimal. 

Line 291, it should be "Claim B.6"?

==== Post Rebuttal =====

Thank the authors for the clear and detailed feedback. Taking everything into consideration, I'd like to keep my rating as-is.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper gives a method for selecting panels via sortition that offers transparency that can be theoretically guaranteed and empirically audited. Their method is based on uniform lotteries and aims to achieve Maximin and Nash Welfare notions of fairness.
main_review: While previous works present the merits of sortition algorithms and give methods to better sample for sortition based committee selection, this work furthers these goals by analyzing uniform lottery with fairness constraints. 

The worst-case marginal discrepancy bounds seem reasonable and it is great to see a tighter upper bound for n >> k^2 case since that is likely applicable in most applications. 

I am not sure I fully understand the argument of why a worst-case Maximin guarantee is a Leximin guarantee. It seems like uniform lottery does not necessarily preclude Leximin solutions. 

It is nice that the authors consider both Maximin and Nash Welfare when bounding the fairness difference between the p* and \bar{p}. What explains the extra factor of k between the Maximin and Nash upper bounds?

Considering that sortition is increasingly used in the real world, I believe this paper makes an important contribution in analyzing fairness costs of a transparent sortition process: uniform lottery. 

limitations_and_societal_impact: The paper briefly acknowledges that the notion of transparency suggested in the paper looks at only the final stage of the panel selection process. I agree that thinking about transparency throughout the selection process is important from recruitment to final selection. 

Another major consideration is that quotas may not be legally compatible in certain domains. Transparency may lead to legal challenges that violate equal protection laws (particularly in the United States). What does this mean for comparing against quota systems of sortition-based selection or stratified sampling?
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper develops and characterizes methods for selecting a citizens' panel that is representative (satisfying upper and lower quotas on demographics' representation), fair (each eligible citizen has as equal a chance as possible of being selected), and transparent (where the final panel is chosen uniformly at random from a set of candidate panels in a public lottery). 
main_review: While I am not familiar with the sortition literature and did not check the mathematical details of this paper, I found it interesting and reasonably easy to follow. I'll let other reviewers to determine whether the problem studied (in particular, the transparency requirement) is important. 
limitations_and_societal_impact: This paper is grounded in a read-world problem and is obviously concerned with the values of the people involved in and affected by sortition. The limitations of the proposed method (notably, that outside observers cannot distinguish between an individual's high selection probability being necessary for representation and being a way to rig the panel to include them) are clearly explained.
review_title: null

Meta-review: Thanks for the strong submission; the reviewers unanimously enjoyed it. 
Decision: Accept (Poster)