SlowGuess/NeurIPS_2023_OCR huggingface dataset:
L. Elisa Celis  
Yale University
Amit Kumar IIT Delhi
Anay Mehrotra  
Yale University
Nisheeth K. Vishnoi  
Yale University
Biases with respect to socially-salient attributes of individuals have been well documented in evaluation processes used in settings such as admissions and hiring. We view such an evaluation process as a transformation of a distribution of the true utility of an individual for a task to an observed distribution and model it as a solution to a loss minimization problem subject to an information constraint. Our model has two parameters that have been identified as factors leading to biases: the resource-information trade-off parameter in the information constraint and the risk-averseness parameter in the loss function. We characterize the distributions that arise from our model and study the effect of the parameters on the observed distribution. The outputs of our model enrich the class of distributions that can be used to capture variation across groups in the observed evaluations. We empirically validate our model by fitting real-world datasets and use it to study the effect of interventions in a downstream selection task. These results contribute to an understanding of the emergence of bias in evaluation processes and provide tools to guide the deployment of interventions to mitigate biases.
Evaluation processes arise in numerous high-stakes settings such as hiring, university admissions, and fund allocation decisions [20, 30, 90, 122]. Specific instances include recruiters estimating the hireability of candidates via interviews [121, 30], reviewers evaluating the competence of grant applicants from proposals [147, 18], and organizations assessing the scholastic abilities of students via standardized examinations [99, 19]. In these processes, an evaluator estimates an individual's value to an institution. The evaluator need not be a person, they can be a committee, an exam, or even a machine learning algorithm [51, 122, 145]. Moreover, outcomes of real-world evaluation processes have at least some uncertainty or randomness [30, 18, 79]. This randomness can arise both, due to the features of the individual (e.g., their test scores or grades) that an evaluator takes as input [31, 76, 124], as well as, due to the evaluation process itself [50, 30, 140].
Biases against individuals in certain disadvantaged groups have been well-documented in evaluation processes [147, 74, 104, 110, 30]. For instance, in employment decisions and peer review, women receive systematically lower competence scores than men, even when qualifications are the same [147, 110], in standardized tests, the scores show higher variance in students from certain genders [21, 112], and in risk assessment-a type of evaluation-widely used tools were twice as likely to misclassify Black defendants as being at a high risk of violent recidivism than White defendants [6]. Here, neither the distribution of individuals' true evaluation depends on their socially-salient attributes nor is the process trying to bias evaluations, yet biases consistently arise [147, 74, 104, 110, 30]. Such evaluations are increasingly used by ML systems to learn or make decisions about individuals, potentially exacerbating inequality [51, 122, 145]. This raises the question of explaining the emergence of biases in evaluation processes which is important to understand how to mitigate them, and is studied here.
Related work. A wide body of work has studied reasons why such differences may arise and how to mitigate the effect of such biases [75, 57, 54, 32, 88, 28]. For one, socioeconomic disadvantages (often correlated with socially-salient attributes) have been shown to impact an individual's ability to perform in an evaluation process, giving rise to different performance distributions across groups [55, 16]. Specifically, disparities in access to monetary resources are known to have a significant impact on individuals' SAT scores [55]. Moreover, because of differences between socially-salient attributes of individuals and evaluators, the same amount of resources (such as time or cognitive effort) spent by the evaluator and the individual, can lead to different outcomes for individuals in different groups [64, 95, 8]. For instance, it can be cognitively more demanding, especially in time-constrained evaluation processes, for the evaluator to interact with individuals who have a different cultural background than them, thus impacting the evaluations [95, 86, 70, 58, 142, 115, 113]. Further, such biases in human evaluations can also affect learning algorithms through biased past data that the algorithms take as input [75, 57, 54, 28].
Another factor that has been identified as a source of bias is "risk averseness:" the tendency to perceive a lower magnitude of increase in their utility due to a profit than the magnitude of decrease in their utility due to a loss of the same magnitude as the profit [84, 144, 151]. Risk averseness is known to play a role in high-stakes decisions such as who to hire, who to follow on social networks, and whether to pursue higher education [71, 23, 14]. In evaluations with an abundance of applicants, overestimating the value of an individual can lead to a downstream loss (e.g., because an individual is hired or admitted) whereas under-estimating may not have a significant loss [139, 65]. Thus, in the presence of risk averseness, the outputs of evaluation processes may skew the output evaluations to lower or higher values. The same skew can also arise from the perspective of individuals [13, 33, 111]. For instance, when negotiating salaries, overestimating their salary can lead to adverse effects in the form of evaluators being less inclined to work with the individual or in extreme cases denying employment [13, 33]. Moreover, these costs have been observed to be higher for women than for men, and are one of the prominent explanations for why women negotiate less frequently [13, 33].
A number of interventions to mitigate the adverse effects of such biases in evaluation processes have been proposed. These include representational constraints that, across multiple individuals, increase the representation of disadvantaged and minority groups in the set of individuals with high evaluations [46, 135, 131, 19, 77, 27, 116], structured evaluations which reduce the scope of unintended biases in evaluations [123, 68, 147, 15], and anonymized evaluations that, when possible, blind the decision makers to the socially-salient attributes of individuals being evaluated [72].
Mathematically, some works have modeled the outcomes of evaluation processes based on empirical observations [12, 22, 90, 61]. For instance, the implicit variance model of [61] models differences in the amount of noise in the utilities for individuals in different groups. Here, the output estimate is drawn from a Gaussian density whose mean is the true utility  $v$  (which can take any real value) and whose variance depends on the group of the individual being evaluated: The variance is higher for individuals in the disadvantaged group compared to individuals in the advantaged group. Additive and multiplicative skews in the outputs of evaluation processes have also been modeled [90, 22] (also see Appendix A). [90] consider true utilities  $v > 0$  distributed according to the Pareto density and they model the output as  $v / \rho$  for some fixed  $\rho \geq 1$ ; where  $\rho$  is larger for individuals in the disadvantaged group. These models have been influential in the study of various downstream tasks such as selection [90, 61, 38, 129, 67, 106, 108, 29], ranking [40], and classification [28] in the presence of biases.
Our contributions. We propose a new optimization-based approach to model how an evaluation process transforms an (unknown) input density  $f_{\mathcal{D}}$  representing the true utility of an individual or a population to an observed distribution in the presence of information constraints or risk aversion. Based on the aforementioned studies and insights in social sciences, our model has two parameters: the resource-information parameter  $(\tau \in \mathbb{R})$  in the information constraint and the risk-averseness parameter  $(\alpha \geq 1)$  in the objective function; see (OptProg) in Section 2. The objective measures the inaccuracy of the estimator with respect to the true density  $f_{\mathcal{D}}$ , and involves a given loss function  $\ell$  and the parameter  $\alpha - \alpha$  is higher (worse) for individuals in groups facing higher risk aversion. The constraint places a lower bound of  $\tau$  on the amount of information (about the density of the true value  $v$ ) that the individual and evaluator can acquire or exchange in their interaction  $-\tau$  is higher for individuals in groups that require more resources to gain unit information. We measure the amount of information in the output density by its differential entropy. Our model builds on the maximum-entropy framework in statistics and information theory [80] and is derived in Section 2 and can be viewed as extending this theory to output a rich family of biased densities.
In Section 3, we show various properties of the output densities of our model. We prove that the solution to (OptProg) is unique under general conditions and characterize the output density as a function of  $f_{\mathcal{D}}, \ell, \tau$ , and  $\alpha$ ; see Theorem 3.1. By varying the loss function and the true density, our framework can not only output standard density functions (such as Gaussian, Pareto, Exponential, and Laplace), but also their appropriate "noisy" and "skewed" versions, generalizing the models studied in [90, 22, 61]. Subsequently, we investigate how varying the parameter  $\tau$  affects the output density in Section 3. We observe that when  $\tau \rightarrow -\infty$ , there is effectively no constraint, and the output is concentrated at a point. For any fixed  $\alpha$ , as  $\tau$  increases, the output density spreads its variance and/or mean increases. We also study the effect of increasing  $\alpha$  on the output density. We observe that when the true density is Gaussian or Pareto, the mean of the output density decreases as  $\alpha$  increases for any fixed  $\tau$ . Thus, individuals in the group with higher  $\alpha$  and/or  $\tau$  face higher noise and/or skew in their evaluations as predicted by our model.
Empirically, we evaluate our model's ability to emulate biases present in real-world evaluation processes using two real-world datasets (JEE-2009 Scores and the Semantic Scholar Open Research Corpus) and one synthetic dataset (Section 4). For each dataset, we report the total variation (TV) distance between the densities of biased utilities in the data and the best-fitting densities output by our framework and earlier models. Across all datasets, we observe that our model can output densities that are close to the density of biased utilities in the datasets and has a better fit than the models of [61, 90]; Table 1. Further, on a downstream selection task, we evaluate the effectiveness of two well-studied bias-mitigating interventions: equal representation (ER) and proportional representation (PR) constraints, and two additional interventions suggested by our work: decreasing the resource-information parameter  $\tau$  and reducing the risk-averseness parameter  $\alpha$ . ER and PR are constraints on the allowable outcomes,  $\tau$  can be decreased by, e.g., training the evaluators to improve their efficiency, and  $\alpha$  can be decreased using, e.g., structured interviews [30]. We observe that for each intervention, there are instances of selection, where it outperforms all other interventions (Figure 1). Thus, our model can be used as a tool to study the effectiveness of different types of interventions in downstream tasks and inform policy; see also Appendix L.1 and Appendix B.
The evaluation processes we consider have two stakeholders—an evaluator and an individual—along with a societal context that affects the process. In an evaluation process, an evaluator interacts with an individual to obtain an estimate of the individual's utility or value. We assume that each individual's true utility  $v$  is drawn from a probability distribution. This not only captures the case that the same individual may have variability in the same evaluation (as is frequently observed in interviews, examinations, and peer-review [31, 50, 76, 124, 30, 140, 18]) but also the case that  $v$  corresponds to the utility of an individual drawn from a population. For simplicity, we consider the setting where  $v$  is real-valued and its density is supported on a continuous subset  $\Omega \subseteq \mathbb{R}$ . This density gives rise to a distribution over  $\Omega$  with respect to the Lebesgue measure  $\mu$  over  $\mathbb{R}$ . For instance,  $\Omega$  could be the set of all real numbers  $\mathbb{R}$ , the set of positive real numbers  $\mathbb{R}_{>0}$ , an open interval such as  $[1, \infty)$ , or a closed interval  $[a, b]$ . Following prior work modeling output densities [90, 40, 61], we assume that the true utility of all individuals is drawn from the same density  $f_{\mathcal{D}}$ .
We view an evaluation process as a transformation of an (unknown) true density  $f_{\mathcal{D}}$  into an observed density  $f_{\mathcal{E}}$  over  $\Omega$ . In real-world evaluation processes, this happens through various means: by processing features of an individual (e.g., past performance on exams or past employment), through interaction between the evaluator and the individual (e.g., in oral examinations), or by requesting the individual to complete an assessment or test [121, 99, 19]. We present an optimization-based model that captures some of the aforementioned scenarios and outputs  $f_{\mathcal{E}}$ . The parameters of this model encode factors that may be different for different socially-salient groups, thus, making  $f_{\mathcal{E}}$  group dependent even though  $f_{\mathcal{D}}$  is not group dependent. We derive our model in four steps.
Step 1: Invoking the entropy maximization principle. In order to gain some intuition, consider a simple setting where the utility of an individual is a fixed quantity  $v$  (i.e.,  $f_{\mathcal{D}}$  is a Dirac-delta function around  $v$ ). We first need to define an error or loss function  $\ell \colon \Omega \times \Omega \to \mathbb{R}$ ; given a guess  $x$  of  $v$ , the loss function  $\ell(x, v)$  indicates the gap between the two values. We do not assume that  $\ell$  is symmetric but require  $\ell(x, v) \geq 0$  when  $x \geq v$ . Some examples of  $\ell(x, v)$  are  $(x - v)^2, |x - v|, x / v$ , and  $\ln(x / v)$ . The right choice of the loss function can be context-dependent, e.g.,  $(x - v)^2$  is a commonly used loss function for real-valued data, and  $\ln(x / v)$  is sometimes better at capturing relative error for
heavy-tailed distributions over positive domains [83]. For a density  $f$  for  $x$ ,  $\mathbb{E}_{x\sim f}\ell (x,v)$  denotes the expected error of the evaluation process. One can therefore consider the following problem: Given a value  $\Delta$ , can we find an  $f$  such that  $\mathbb{E}_{x\sim f}\ell (x,v)\leq \Delta$ ? This problem is under-specified as there may be (infinitely) many densities  $f$  satisfying this constraint. To specify  $f$  uniquely, we appeal to the maximum entropy framework in statistics and information theory [81]: Among all the feasible densities, one should select the density  $f$  which has the maximum entropy. This principle leads to the selection of a density that is consistent with our constraint and makes no additional assumption. We use the notion of the (differential) entropy of a density with respect to the Lebesgue measure  $\mu$  on  $\mathbb{R}$ :
$$
\operatorname {E n t} (f) := - \int_ {x \in \Omega} f (x) \ln f (x) d \mu (x), \tag {1}
$$
where  $f(x) \ln f(x) = 0$  whenever  $f(x) = 0$ . Thus, we get the following optimization problem:
$$
\operatorname {a r g m a x} _ {f: \text {d e n s i t y o n} \Omega} \mathsf {E n t} (f), \quad s. t., \quad \mathbb {E} _ {x \sim f} \ell (x, v) \leq \Delta . \tag {2}
$$
This optimization problem is well-studied and it is known that by using different loss functions, we can derive many families of densities [102, 148]. For instance, for  $\ell(x,v) \coloneqq (x - v)^2$ , we recover the Gaussian density with mean  $v$ , and for  $\ell(x,v) \coloneqq \ln(x / v)$ , we obtain a Pareto density.
Step 2: Incorporating the resource-information parameter. We now extend the above formulation to include information constraints in the evaluation process. In an evaluation process, both the evaluator and the individual spend resources such as time, cognitive effort, or money to communicate the information related to the utility of the individual to the evaluator. For instance, in interviews, both the interviewer and the interviewee spend time and cognitive effort. In university admissions, the university admissions office needs to spend money to hire and train application readers who, in turn, screen applications for the university's admission program, and the applicants need to spend time, cognitive effort, and money to prepare and submit their applications [24, 150]. The more resources are spent in an evaluation process, the more additional information about  $v$  is acquired. We model this using a resource-information parameter  $\tau$ , which puts a lower bound on the entropy of  $f$ . Thus, we modify the optimization problem in Equation (2) in the following manner. We first flip the optimization problem to an equivalent problem where we minimize the expected loss subject to a lower bound on the entropy,  $\mathsf{Ent}(f)$ , of  $f$ . A higher value of the resource-information parameter  $\tau$  means that one needs to spend more resources to obtain the same information and corresponds to a stronger lower bound on  $\mathsf{Ent}(f)$  (and vice-versa) in our framework.
$$
\operatorname {a r g m i n} _ {f: \text {d e n s i t y} \Omega} \mathbb {E} _ {x \sim f} \ell (x, v), \quad s. t., \quad \operatorname {E n t} (f) \geq \tau . \tag {3}
$$
When  $\tau \rightarrow -\infty$ , the optimal density tends to a point or delta density around  $v$  (recovering the most information), and when  $\tau \rightarrow \infty$ , it tends to a uniform density on  $\Omega$  (learning nothing about  $v$ ). Since differential entropy can vary from negative to positive infinity, the value of  $\tau$  is to be viewed relative to an arbitrary reference point.  $\tau$  may vary with the socially-salient group of the individual in real-world contexts. For instance, in settings where the evaluator needs to interact with individuals (e.g., interviews), disparities can arise because it is less cognitively demanding for an evaluator to communicate with individuals who speak the same language as themselves, compared to individuals who speak a different language [95]. In settings where the evaluator assesses individuals based on data about their past education and employment (e.g., at screening stages of hiring or in university admissions), disparities can arise because the evaluator is more knowledgeable about a specific group's sociocultural background compared to others, and would have to spend more resources to gather the required information for the other groups [49, 62].
Step 3: Incorporating the risk-averseness parameter. We now introduce the parameter  $\alpha$  that captures risk averseness. Roughly speaking, risk averseness may arise in an evaluation process because of the downstream impact of the output. The evaluator may also benefit or may be held accountable for the estimated value, and hence, would be eager or reluctant to assign values much higher than the true utility [71, 23, 14]. Further, the individual may also be risk averse, e.g. during a hiring interview, the risk of getting rejected may prompt the individual to quote less than the expected salary [13, 33, 111]. To formalize this intuition, for a given  $\ell$ , we define a risk-averse loss function  $\ell_{\alpha}:\Omega \times \Omega \to \mathbb{R}$  that incorporates the parameter  $\alpha \geq 1$  in  $\ell$  as follows:
$$
\ell_ {\alpha} (x, v) := \alpha \cdot \ell (x, v) \text {i f} x \geq v \text {a n d} \ell_ {\alpha} (x, v) := \ell (x, v) \text {i f} x <   v. \tag {4}
$$
Not only does this loss function penalize overestimation versus underestimation, but in addition, the more the overestimation, the more the penalization is. This intuition is consistent with the theory
of risk averseness [10, 119]. Our choice is related to the notion of hyperbolic absolute risk aversion [78, 109], and one may pick other ways to incorporate risk averseness in the loss function [144, 98]. As an example, if  $\ell(x,v) = (x - v)^2$ , then the  $\ell_2^2$ -loss is  $\ell_{\alpha}(x,v) = \alpha \cdot (x - v)^2$ , if  $x \geq v$  and  $(x - v)^2$  otherwise. If  $\ell(x,v) = \ln(x / v)$ , then the log-ratio loss is  $\ell_{\alpha}(x,v) = \alpha \cdot \ln(x / v)$ , if  $x \geq v$ ;  $\ln(x / v)$  otherwise. Plots of these two loss functions are in Figure 2. $^1$  Thus, the analog of Equation (3) becomes
$$
\operatorname {a r g m i n} _ {f: \text {d e n s i t y o n} \Omega} \mathbb {E} _ {x \sim f} \ell_ {\alpha} (x - v, v), \quad s. t., \quad \operatorname {E n t} (f) \geq \tau . \tag {5}
$$
We note that, for the risk-averse loss function defined in (4), the following hold: For all  $\alpha \geq \alpha' \geq 1$ ,  $\ell_{\alpha}(x,v) \geq \ell_{\alpha'}(x,v)$  for all  $x,v \in \Omega$ , and  $\ell_{\alpha}(x,v) - \ell_{\alpha'}(x,v)$  is an increasing function of  $x$  for  $x \geq v$ . Beyond (4), one could consider other  $\ell_{\alpha}(x,v)$  satisfying these two properties in our framework; we omit the details. One can also incorporate the (opposite) notion of "risk eager," where values of  $x$  lower than  $v$  are penalized more as opposed to values of  $x$  higher than  $v$  by letting  $\alpha \in (0,1]$ .
Step 4: Generalizing to arbitrary  $f_{\mathcal{D}}$ . To extend Equation (5) to the setting when  $v$  comes from a general density  $f_{\mathcal{D}}$ , we replace the loss function by its expectation over  $f_{\mathcal{D}}$  and arrive at the model for the evaluation process that we propose in this paper:
$$
\begin{array}{l} \operatorname {a r g m i n} _ {f: \text {d e n s i t y o n} \Omega} \operatorname {E r r} _ {\ell , \alpha} \left(f _ {\mathcal {D}}, f\right) := \int_ {v \in \Omega} \left[ \int_ {x \in \Omega} \ell_ {\alpha} (x, v) f (x) d \mu (x) \right] f _ {\mathcal {D}} (v) d \mu (v), (\text {O p t P r o g}) \\ \text {s u c h t h a t} \quad - \int_ {x \in \Omega} f (x) \log f (x) d \mu (x) \geq \tau . \end{array}
$$
For a given  $\ell$  and parameters  $\alpha$  and  $\tau$ , this optimization framework can be viewed as transforming the true utility density  $f_{\mathcal{D}}$  of a group of individuals to the density  $f_{\mathcal{E}}$  (the solution to this optimization problem). It is worth pointing out that neither the evaluator nor the individual is solving the above optimization problem – rather (OptProg) models the evaluation process and the loss function  $\ell$ ,  $\alpha$ , and  $\tau$  depend on the socially-salient attribute of the group of an individual; see also Appendix B.
Characterization of the optimal solution. We first characterize the solution of the optimization problem (OptProg) in terms of  $f_{\mathcal{D}}$ ,  $\ell_{\alpha}$ ,  $\tau$ , and  $\alpha$ . Given a probability density  $f_{\mathcal{D}}$ , a parameter  $\alpha \geq 1$ , and a loss function  $\ell$ , consider the function  $I_{f_{\mathcal{D}},\ell,\alpha}(x) \coloneqq \int_{v \in \Omega} \ell_{\alpha}(x,v) f_{\mathcal{D}}(v) d\mu(v)$ . This integral captures the expected loss when the estimated utility is  $x$ . Further, for a density  $f$ , the objective function of (OptProg) can be expressed as  $\mathrm{Err}_{\ell,\alpha}(f_{\mathcal{D}},f) \coloneqq \int_{x \in \Omega} I_{f_{\mathcal{D}},\ell,\alpha}(x)f(x)d\mu(x)$ .