SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Subset selection tasks. There are various types of selection tasks [59, 4, 30]. We consider the simplest instantiation where there are  $n$  items, each item  $i$  has a true utility  $v_{i} \geq 0$ , and the goal is to select a size-  $k$  subset  $S_{\mathcal{D}}$  maximizing  $\sum_{i \in S} v_{i}$ . If  $V = (v_{1}, \ldots, v_{n})$  is known, then this problem is straightforward: select  $k$  items with the highest utility. However, typically  $V$  is unknown and is estimated via a (human or algorithmic) evaluation process that outputs a possibly skewed/noisy estimate  $X = (x_{1}, \ldots, x_{n})$  of  $V$  [125, 152, 110, 118, 36]. Hence, the outputs is  $S_{\mathcal{E}} := \operatorname{argmax}_{|S| = k} \sum_{i \in S} x_{i}$ , which may be very different from  $S_{\mathcal{D}}$  and possible has a much lower true utility:  $\sum_{i \in S_{\mathcal{E}}} v_{i} \ll \sum_{i \in S_{\mathcal{D}}} v_{i}$ .
Interventions to mitigate bias. Several interventions have been proposed to counter the adverse effects of bias in selection, including, representational constraints, structured interviews, and interviewer training. Each of these interventions tackles a different dimension of the selection task. Representational constraints require the selection to include at least a specified number of individuals from unprivileged groups [30, 37, 135]. Structured interviews reduce the scope of unintended skews by requiring all interviewees to receive the same (type of) questions [30, 123, 68]. Interviewer training aims to improve efficiency: the amount of (accurate) information the interviewer can acquire in a given time [30]. Which intervention should a policymaker enforce?
Studying the effectiveness of interventions. A recent and growing line of work [90, 40, 28, 61, 67, 38, 107] evaluates the effectiveness of representational constraints under specific models of bias: they ask, given  $\mathcal{C}$  of subsets satisfying some constraint, when does the constraint optimal set,  $S_{\mathcal{E},\mathcal{C}} := \operatorname*{argmax}_{S \in \mathcal{C}} \sum_{i \in S} x_i$ , have a higher utility than the unconstrained optimal  $S_{\mathcal{E}}$ , i.e., when is  $\sum_{i \in S_{\mathcal{E},\mathcal{C}}} v_i > \sum_{i \in S_{\mathcal{E}}} v_i$ ? Based on their analysis [90, 40, 28, 61, 67, 38, 107] demonstrate the benefits of different constraints including, equal representation (ER), which requires the output  $S$  to satisfy  $|S \cap G_1| = |S \cap G_2|$  and, proportional representation (PR), which requires  $S$  to satisfy  $|S \cap G_1| / |G_1| = |S \cap G_2| / |G_2|$ . A feature of our model is that its parameters  $\alpha$  and  $\tau$  have a physical interpretation, which enables the study of other interventions: for instance, structured interviews aim to reduce skew in evaluation, which corresponds to shifting  $\alpha$  closer to 1, and interviewer-training affects the information-to-resource trade-off, i.e., reduces  $\tau$ .
Using our model we compare ER and PR with two new interventions: change  $\alpha$  by  $50\%$  ( $\alpha$ -intervention) and change  $\tau$  by  $50\%$  ( $\tau$ -intervention). Here,  $50\%$  is an arbitrary amount for illustration.
Setup. We consider a selection scenario based on the JEE 2009 data: we fix  $f_{\mathcal{D}} = f_{G_1}$ ,  $\alpha'$ ,  $\tau'$  to be the best-fit parameters on the JEE 2009 data (by TV distance), and  $|G_1| = 1000$ . Let  $f_{\alpha}$  and  $f_{\tau}$  be the densities obtained after applying the alpha intervention and the tau intervention respectively. (Formally,  $f_{\alpha} = f_{\alpha'/2,\tau'}^{\star}$  and  $f_{\tau} = f_{\alpha',3\tau'/2}^{\star}$ .) We vary  $|G_2| \in \{500, 1000, 1500\}$ . For each  $i \in G_1$ , we draw  $v_i \sim f_{\mathcal{D}}$  and set  $x_i = v_i$  (no bias). For each  $i \in G_2$ , we draw  $v_i \sim f_{\mathcal{D}}$ ,  $x_i \sim f_{\alpha,\tau}^{\star}$ ,  $x_i^{\alpha} \sim f_{\alpha}$ , and  $x_i^{\tau} \sim f_{\tau}$  coupled so that the CDFs of the respective densities at  $v_i, x_i, x_i^{\alpha}$ , and  $x_i^{\tau}$  are the same. We give ER and PR the utilities  $\{x_i\}_i$  as input, we give  $\alpha$ -intervention utilities  $\{x_i^{\alpha}\}_i$  as input, and the  $\tau$ -intervention utilities  $\{x_i^{\tau}\}_i$ . For each  $|G_1|$  and  $|G_2|$ , we vary  $50 \leq k \leq 1000$ , sample utilities and report the expected utilities of the subset output by each intervention over 100 iterations (Figure 1). Here, 1000 is the largest value for which ER is satisfiable across all group sizes  $|G_1|$  and  $|G_2|$ .
(a)  $|G_{1}| = 1000$  and  $|G_{2}| = 500$
Figure 1: Effectiveness of different interventions on the selection-utility-as estimated by our model: The  $x$ -axis shows  $k$  (size of selection) and the  $y$ -axis shows the ratio of the (true) utility of the subset output with an intervention to the (true) utility of the subset output without any intervention. The main observation across the figures is that, for each intervention, there is a choice of  $k$ , and group sizes  $|G_1|$  and  $|G_2|$ , where the intervention outperforms all other interventions. Hence, each intervention has a different effect on the latent utility of selection and a policymaker can use our model to study their effect and decide which intervention to enforce. Error bars represent the standard error of the mean over 100 repetitions.
(b)  $|G_{1}| = 1000$  and  $|G_{2}| = 1000$
(c)  $|G_1| = 1000$  and  $|G_2| = 2000$
Observations. Our main observation is that there is no pair of interventions such that one always achieves a higher utility than the others. In fact, for each intervention, there is a value of  $k$ ,  $|G_1|$ , and  $|G_2|$ , such that the subset output with this intervention has a higher utility than the subsets output with other interventions. Thus, each intervention has a very different effect on the latent utility of the selection and a policymaker can use our model to study the effects in order to systematically decide which interventions to enforce; see Appendix L.1 for a case study of how a policymaker could potentially use this model to study bias-mitigating interventions in the JEE context.
5 Conclusion, limitations, and future work
Acknowledgments. This project is supported in part by NSF Awards CCF-2112665 and IIS-2045951.
[1] Is Income Implicit in Measures of Student Ability? https://budgetmodel.wharton.upenn.edu/issues/2021/9/28/is-income-implicit-in-measures-of-student-a bility#:~text=Summary%3A%20Measures%20of%20student%20ability,in%20fam ily%20income%20across%20students, 2021.
[2] Supreme Court guts affirmative action, effectively ending race-conscious admissions. https://www.npr.org/2023/06/29/1181138066/affirmative-action-supreme-court-decision, 2023.
[3] Social Security Administration. Beyond the Top 1000 Names, 2018. https://www.ssa.gov/oact/babynames/limits.html.
[4] Rakesh Agrawal, Sreenivas Gollapudi, Alan Halverson, and Samuel Leong. Diversifying search results. In Proceedings of the Second ACM International Conference on Web Search and Data Mining, WSDM '09, page 5-14, New York, NY, USA, 2009. Association for Computing Machinery.
[5] William Aiello, Fan R. K. Chung, and Linyuan Lu. A random graph model for massive graphs. In STOC, pages 171-180. ACM, 2000.
[6] Julia Angwin, Madeleine Varner, and Ariana Tobin. Machine bias: Facebook enabled advertisers to reach 'jew haters'. ProPublica, Sept, 2017.
[7] Another PIL seeks entrance tests in Gujarati, October 2011. https://web.archive.org/web/20120106121134/http://articles-timesofindia.indiatimes.com/2011-10-21/ahmedabad/30306229_1 Entrance-tests-regional-language-gujarati-1 language.
[8] Michael Argyle. The psychology of interpersonal behaviour. Penguin UK, 1994.
[9] S. Arora, E. Hazan, and S. Kale. The multiplicative weights update method: a meta-algorithm and applications. Theory of Computing, 8(1):121-164, 2012.
[10] Kenneth J. Arrow. Aspects of the Theory of Risk-Bearing. YrjöJahnsson lectures. YrjöJahnssonin Säätö, Helsinki, 1965.
[11] Kenneth J. Arrow. The Theory of Discrimination, pages 1-33. Princeton University Press, Princeton, 1974.
[12] Kenneth J. Arrow. What has economics to say about racial discrimination? Journal of Economic Perspectives, 12(2):91-100, June 1998.
[13] L. Babcock and S. Laschever. Why Women Don't Ask: The High Cost of Avoiding Negotiations - And Positive Strategies for Change. Little, Brown Book Group Limited, 2009.
[14] Stijn Baert. Hiring a gay man, taking a risk?: A lab experiment on employment discrimination and risk aversion. Journal of Homosexuality, 65(8):1015-1031, 2018. PMID: 28841095.
[15] Katherine Baldiga. Gender Differences in Willingness to Guess. Management Science, 60(2):434-448, 2014.
[16] British Business Bank and Oliver Wyman. Alone together: Entrepreneurship and diversity in the UK, October 2020. https://www.british-business-bank.co.uk/wp-content/uploads/2020/10/Alone-together-Entrepreneurship-and-diversity-in-the-UK-FINAL.pdf.
[17] Albert-László Barabási and Réka Albert. Emergence of scaling in random networks. Science, 286(5439):509-512, 1999.
[126] Walter Rudin. Real and Complex Analysis. McGraw-Hill, 1986.
[127] Melody S Sadler, Joshua Correll, Bernadette Park, and Charles M Judd. The world is not black and white: Racial bias in the decision to shoot in a multiethnic context. Journal of Social Issues, 68(2):286-313, 2012.
[128] Sahitya parishhad demands entrance tests in gujarati, September 2011. https://web.archive.org/web/20120106140750/http://articles-timesofindia.indiatimes.com/2011-09-21/education/30184151_1 regional-languages-entrance-tests-raghu veer-chaudhary.
[129] Jad Salem and Swati Gupta. Closing the gap: Group-aware parallelization for the secretary problem with biased evaluations. Available at SSRN 3444283, 2019.
[130] Howard Schuman, Charlotte Steeh, Lawrence Bobo, and Maria Krysan. Racial Attitudes in America: Trends and Interpretations. Social trends in the United States. Harvard University Press, 1985.
[131] Deepa Seetharaman. Facebook Is Testing the 'Rooney Rule' Approach to Hiring. The Wall Street Journal, June 2015.
[132] Herbert A Simon. Bounded rationality and organizational learning. Organization science, 2(1):125-134, 1991.
[133] Mohit Singh and Nisheeth K. Vishnoi. Entropy, optimization and counting. In Proceedings of the 46th Annual ACM Symposium on Theory of Computing, pages 50-59. ACM, 2014.
[134] Aarzoo Snigdha. JEE Main 2021: Over 45,000 Students Writing Exam In Regional Languages. NDTV, February 2021. https://www.ndtv.com/education/jee-main-2021-over-45000-students-writing-exam-in-regional-languages-2378496.
[135] Thomas Sowell. Affirmative Action Around the World: An Empirical Study. Yale University Press, 2008.
[136] Damian Straszak and Nisheeth K. Vishnoi. Maximum entropy distributions: Bit complexity and stability.  $COLT$ , 2019.
[137] Damian Straszak and Nisheeth K. Vishnoi. Iteratively reweighted least squares and slime mold dynamics: connection and convergence. Mathematical Programming, 194(1):685-717, 2022.
[138] R.H. Thaler. Misbehaving: The Making of Behavioral Economics. W. W. Norton, 2015.
[139] The Cost of a Bad Hire, February 2019. https://www.northwestern.edu/hr/about/news/february-2019/the-cost-of-a-bad-hire.html.
[140] The Utter Uselessness of Job Interviews, April 2017. https://www.nytimes.com/2017/04/08/opinion/sunday/the-utter-uselessness-of-job-interviews.html.
[141] UP govt free coaching for JEE, NEET aspirants begins today, February 2021. https://www.livemint.com/education/news/up-govt-free-coaching-for-jee-neet-a spirants-begins-today-11613449583134.html.
[142] AD Van Knippenberg, AP Dijksterhuis, and Diane Vermeulen. Judgement and memory of a criminal act: The effects of stereotypes and cognitive load. European Journal of Social Psychology, 29(2-3):191-201, 1999.
[143] Nisheeth K. Vishnoi. Algorithms for Convex Optimization. Cambridge University Press, 2021.
[144] J. von Neumann, O. Morgenstern, H.W. Kuhn, and A. Rubinstein. Theory of Games and Economic Behavior: 60th Anniversary Commemorative Edition. Princeton Classic Editions. Princeton University Press, 2007.
[145] Joseph Walker. Meet the New Boss: Big Data, September 2012. https://www.wsj.com/ar  
ticles/SB10000872396390443890304578006252019616768.
[146] Thomas E. Weisskopf. Impact of Reservation on Admissions to Higher Education in India. Economic and Political Weekly, 39(39):4339-4349, 2004.
[147] Christine Wennerås and Agnes Wold. *Nepotism and Sexism in Peer-Review*. Nature, 387(6631):341–343, May 1997.
[148] Wikipedia contributors. Maximum entropy probability distribution — Wikipedia, the free encyclopedia. https://en.wikipedia.org/w/index.php?title=Maximum_entropy(probability_distribution&oldid=1154992719, 2023. [Online; accessed 16-May-2023].
[149] Joan C Williams. Double jeopardy? an empirical study with implications for the debates over implicit bias and intersectionality. Harvard Journal of Law & Gender, 37:185, 2014.
[150] Sarah Wood. Colleges With The Highest Application Fees, January 2022. https://www.usnews.com/education/best-colleges/the-short-list-college/articles/colleges-with-the-highest-application-fees.
[151] Ruixun Zhang, Thomas J Brennan, and Andrew W Lo. The origin of risk aversion. Proceedings of the National Academy of Sciences, 111(50):17777-17782, 2014.
[152] Jonathan C. Ziegert and Paul J. Hanges. Employment discrimination: The role of implicit attitudes, motivation, and a climate for racial bias. Journal of Applied Psychology, 90(3):553-562, 2005.
1 Introduction 1
2 Model 3
3 Theoretical results 5
4 Empirical results 8
5 Conclusion, limitations, and future work 10
A Other related work 22
B Specific examples of our model 23
C Characterization of optimal solution to the optimization problem 24
C.1 The primal optimization problem and assumptions 24
C.2 Dual formulation and weak duality 26
C.3 Strong duality 27
C.4 Properties of the integral  $I(x)$  29
C.5 Positivity of the optimal dual variable 31
C.6 Optimality conditions and uniqueness of optimal solution 33
C.7 Conditions under which optimal solution is  $f_{\mathcal{D}}$  34
D Derivation of output density for a single individual 35
E Connection to the Gibbs equation 36
F Effect of changing the resource-information parameter  $\tau$  36
F.1 Effect of decreasing  $\tau$  36
F.2 Effect of increasing  $\tau$  38
G Effect of changing the risk-averseness parameter  $\alpha$  39
H Gaussian density 39
I Pareto density 42
J Exponential density 45
K Laplace density 47
L Implementation details and additional empirical results 48
L.1 Case Study: Evaluating bias-mitigating interventions in IIT-JEE admissions 48
L.2 Additional plots for simulations in Section 4 51
L.3 Implementation details 51
L.3.1 Our framework and other models 51
L.3.2 Computational resources used 53
L.3.3 JEE-2009 Scores 53
L.3.4 Semantic Scholar Open Research Corpus 54
A Other related work
Models of bias in Economics. There are two prominent models of discrimination in the Economics literature: taste-based discrimination and statistical discrimination [114]. These capture different types of biases [117, 45, 49, 12, 22, 41, 11] (also see [66, 25, 114]). Taste-based discrimination [22] models explicit biases (e.g., racism and sexism) and, in the vanilla taste-based discrimination model, individuals are divided into two groups and a decision-maker pays an additional cost for interacting with individuals in the disadvantaged group. This additional additive cost diminishes the value of disadvantaged individuals for the decision-maker. While we do not model explicit biases, such an additive bias also arises in our model of evaluation processes with specific parameter choices, suggesting that additive biases may also arise due to resource constraints and risk averseness (Appendix C). Statistical discrimination models how group-wise disparities in the noise in the inputs to a Bayesian decision-maker propagate to systematic disparities in decisions [117, 11]. Our mathematical model can be viewed as giving an explanation of why such disparities may arise in the input.
Implicit biases in Psychology. There is a long and rich history of the study of implicit (and explicit) biases in Psychology, e.g., [85, 101, 74, 104, 69, 127, 149]. This body of works proposes various theories about why implicit biases arise [69] and their relation to real-world stimuli [130, 63, 42]. We refer the reader to [73, 87] for an overview. [69] explains that the ease of categorizing individuals into categories (defined by, e.g., color, race, or gender) provides an incentive to the evaluator to use their prior (possibly biased) knowledge, and this leads to implicit biases. In contrast, we show that even when the evaluator has the same prior estimate for all social groups, biases can arise in evaluation processes when the information-to-resource trade-off or the degree of risk averseness is different for different groups. Further, since resource constraints and risk averseness are not specific to the setting of a single evaluator, our model of evaluation processes also models scenarios where the evaluator denotes a group or an organization.
Optimization and human behavior. The use of optimization to model human behavior dates back to (at least) von Neumann and Morgenstern's work that showed that, under a small number of assumptions, the behavior of an evaluator is as if they are optimizing the expected value of a "utility function" [144]. Since then, numerous apparent deviations from this theory were discovered [84]. These deviations, broadly referred to as irrational behavior or cognitive biases, laid the foundation of Behavioral Economics [138]. Several theories have been proposed to account for the deviations in human behavior from utility maximization. Including prospect theory that models risk averseness of individuals – the empirical observation that humans process losses and gains of equal amounts (monetary or otherwise) asymmetrically [84] – bounded rationality that reconciles irrational human behavior by proposing that humans solve underlying optimization problems approximately (instead of optimally) [132], and resource rational analysis that proposes that humans trade-off the utility of with the costs (e.g., such as effort and time) required to find a solution with higher utility [100]. These works model hidden costs and constraints on humans that lead to deviations from the traditional "rational" utility maximization. Like this work, these works also use optimization to explain human behavior, but while they focus on irrational behaviors and cognitive biases, our work focuses on biases with respect to socially-salient attributes in evaluation processes.
Other entropy-based models. Maximum entropy distributions have been widely deployed in machine learning [56] and theoretical computer science [133]. Maximum entropy distributions have been shown to be "stable" [136]. The maximum-entropy framework over discrete distributions has been used to preprocess data to debias it [39]. In our optimization program, we use entropy to measure the amount of "information" in a distribution — this appears as a constraint in our program. An entropy-constrained viewpoint is also prevalent in explanations of other phenomena. For instance, various optimization algorithms can be captured using an optimization- and entropy-based viewpoint [9, 96, 143], it is also known to arise in explanations of biological phenomena [43], and leads to Page rank and other popular random-walk-based procedures [105]. Finally, taking an optimization viewpoint when studying a dynamical system also has additional benefits, such as providing a "potential function" that functions that gives an efficient and interpretable method of tracking the progress of a complex dynamical system [137].
(a)  $\ell_{\alpha}(x,0)$  when  $\ell (x,v) = (x - v)^2$
(b)  $\ell_{\alpha}(x,5)$  when  $\ell (x,v) = \ln x - \ln v$
Figure 2: Plots of risk-averse loss function  $\ell_{\alpha}(x,v)$  for different  $\alpha$  and  $\ell$ .
B Specific examples of our model
In this section, we present some specific examples of mechanisms by which information constraints and risk aversion lead to bias. One context is college admissions. It is well known that SAT scores are implicitly correlated with income (and, hence, test preparation) in addition to student ability [1]. While the true ability may be  $v$ , the score is skewed depending on the amount/quality of test preparation, which depends on socioeconomic status that may be correlated to socially-salient attributes. The parameter  $\alpha$  in our model can be used to encode this. As for  $\tau$ , while an evaluator may know what a GPA means at certain universities well known to them, they may not understand what GPA means for students from lesser-known schools. This lack of knowledge can be overcome, but takes effort/time, and without effort entrenches the status quo.
Another example is the evaluation of candidates using a standardized test. In time-constrained settings, a high value of the resource-information parameter  $\tau$  for the disadvantaged group indicates that such candidates may not be able to comprehend a question as well as someone from an advantaged group. This could be due to various factors including less familiarity with the language used in the test or the pattern of questions, as opposed to someone who had the resources to invest in a training program for the test. Similarly, a high value of the risk-averseness parameter captures that an evaluator, when faced with a choice of awarding low or high marks to an answer given by a candidate from the disadvantaged group, is less likely to give high marks. More concretely, suppose there are several questions in a test, where each question is graded either 0 or 1. Assume that a candidate has true utility  $v \in [0,1]$ , and hence, would have received an expected score  $v$  for each of the questions if one were allowed to award grades in the continuous range  $[0,1]$ . However, the fact that the true scores have to be rounded to either 0 or 1 can create a bias for the disadvantaged group. Indeed, the probability that an evaluator rounds such an answer to 1 may be less than  $v -$  the risk-averseness parameter measures the extent to which this probability gets scaled down.
Most of the prior works on interventions in selection settings have focused on adding representational constraints for disadvantaged groups. Such constraints, often framed as a form of affirmative action, could be beneficial but may not be possible to implement in certain contexts. For instance, in a landmark 2023 ruling, the US Supreme Court effectively prohibited the use of race-based affirmative action in college admissions [2]. Our work, via a more refined model of how bias might arrive at a population level in evaluation processes, allows for evaluating additional interventions that focus on procedural fairness; this allows working towards diversity and equity goals without placing affirmative-action-like constraints.
In this framing, we can consider decreasing either  $\alpha$  or  $\tau$ . Improving either would work towards equity, but which ones to target or to what extent and via which method would be context-dependent and vary in cost. A decrease in  $\alpha$  can be achieved by reducing risk-averseness in the evaluation process; e.g. by investing in better facilities for disadvantaged groups, or making the evaluation process blind to group membership. A reduction in  $\tau$  may follow by allocating additional resources to the evaluation process, e.g., by letting a candidate choose an evaluation in their native language. Our framework allows a policymaker to study these trade-offs, and we discuss specific examples in Appendix L.
C Characterization of optimal solution to the optimization problem