SlowGuess/NeurIPS_2023_OCR huggingface dataset:
finite  $\tau$ . (A1) and (A2) hold due to the choice of the loss function. (A3) can be shown to hold since  $f_{\mathcal{L}}$  is a Laplace density: To see this note that for any finite  $x \in \Omega$
$$
\begin{array}{l} \int_ {\Omega} | \ell (x, v) | f _ {\mathcal {L}} (v) d \mu (v) \leq \int_ {0} ^ {\infty} \left(| x - a | + | v - a |\right) f _ {\mathcal {L}} (v) d \mu (v) \\ = | x - a | + \frac {b}{2} \\ <   \infty . \\ \end{array}
$$
Thus, (A3) holds. (A4) holds with, e.g.,  $R = b\ln(2)$ . By Lemma K.1,
$$
I (x) = \left\{ \begin{array}{l l} \alpha b (w - 1) + b \frac {(\alpha - 1)}{2} e ^ {- w} & \text {i f} x \geq a \\ - b (w + 1) + b \frac {(1 - \alpha)}{2} e ^ {w} & \text {o t h e r w i s e} \end{array} , \right.
$$
where  $w = \frac{x - a}{b}$ . One can check that  $I(x)$  is continuous and differentiable at each  $x \in \Omega \setminus \{a\}$ . Moreover, for all  $x < a$ ,  $\frac{\partial I(x)}{\partial x} < 0$  and for all  $x \geq a$ ,  $\frac{\partial I(x)}{\partial x} \geq 0$ . Hence, it follows that  $I(x)$  has a unique global minimum at  $x = a$ . Therefore, (A5) holds. Since assumptions (A0)-(A5) hold, we invoke Theorem C.1 to deduce the form of the optimal density.
Theorem K.2 (Expression for  $f^{\star}$  when  $\alpha = 1$ ). Consider an instance  $\mathcal{I} = (\Omega, f_{\mathcal{L}}, \ell, 1, \tau)$  of PrimalOpt where  $\Omega = \mathbb{R}$ ,  $f_{\mathcal{D}}$  is the Laplace density with parameters  $a \in \mathbb{R}$  and  $b > 0$ , and  $\ell(x, v) = |x - a| - |v - a|$ . Let  $f^{\star}$  be the optimal solution of  $\mathcal{I}$ . Then  $f^{\star}$  is the Laplace density with parameters  $(a, e^{\tau - 1} / 2)$ .
Thus, for  $\alpha = 1$ , increasing  $\tau$  does not change the location parameter, but increases the "diversity" parameter of the output density.
Proof. Since  $\alpha = 1$ , Lemma K.1 implies that
$$
I (x) = | x - a | - \frac {b}{2}.
$$
$$
f ^ {\star} (x) \propto \exp \left(\frac {- | x - a | - \frac {b}{2}}{\gamma^ {\star}}\right) \propto \exp \left(\frac {- | x - a |}{\gamma^ {\star}}\right).
$$
where the proportionality constant and  $\gamma^{\star}$  are determined by  $\int_{\mathbb{R}}f^{\star}(x)d\mu (x) = 1$  and  $\mathrm{Ent}(f^{\star}) = \tau$ . Clearly,  $f^{\star}$  is a Laplace density with the diversity parameter  $\gamma^{\star}$ , its entropy is  $\mathrm{Ent}(f^{\star}) = 1 + \ln (2\gamma^{\star})$  [148]. On the other hand, since  $\mathrm{Ent}(f^{\star}) = \tau$ , the previous equality implies that  $\gamma^{\star} = \frac{1}{2} e^{\tau -1}$ . It follows that  $f^{\star}(x) = e^{1 - \tau}\cdot \exp \left(-\frac{2|x - a|}{\exp(\tau - 1)}\right)$  which is the Laplace density with parameters  $(a,\frac{1}{2} e^{\tau -1})$ .
L Implementation details and additional empirical results
In this section, we present additional discussions and evaluations of intervention in the JEE setting (Appendix L.1), plots omitted from Section 4 (Appendix L.2), and implementation details of our model (Appendix L.3). The code for this paper is available at https://github.com/AnayMehrotra/Bias-in-Evaluation-Processes.
L.1 Case Study: Evaluating bias-mitigating interventions in IIT-JEE admissions
In this section, we continue our study of the effectiveness of different interventions in a downstream selection task. Like in Section 4, we consider selection based on the JEE 2009 scores, but here consider representational constraints actually used in admissions to IITs. We also discuss additional interventions being implemented by the Indian state and central governments to reduce inequity in JEE scores.
Recall that, the Indian Institutes of Technology (IITs) are a group of engineering institutes in India. In 2009, there were 15 IITs and today this has grown to 23. Undergraduate admissions at IITs are decided based on the scores of candidates in the Joint Entrance Exam (JEE). JEE is conducted once every year. In 2009, the scores, (binary) genders, and birth categories of all candidates who appeared in JEE 2009 were released in response to a Right to Information application filed in June 2009 [91]. The birth category of the candidates is an official socioeconomic status label recognized by the government of India [135].
Here, we focus on two groups of candidates: the candidates in the general (GEN) category (the most privileged) and candidates not in the general category. We begin by discussing some of the interventions in place to reduce inequity in JEE scores and subsequent admissions at IITs.
Interventions used in IIT admissions. The Indian constitution allows the central government and state governments to enforce affirmative action in the form of quotas or lower bound constraints for official SES groups at educational institutes, employments, and political bodies [82, 146]. In 2005 lower-bound interventions were introduced in the admissions process at the IITs. Concretely, in 2009, out of the 7,440 seats, 3,688 (49.6%) were reserved for students who are not in the GEN category. This means that at least 3,688 out of the 7,440 students admitted into IITs must not be in the GEN category. Note that this allows more than 3,688 or even all admitted students to be outside the GEN category. We call this constraint the Reservation constraint and, in this section, we study its effectiveness compared to other forms of interventions.
Apart from reservations, a number of other interventions have also been proposed and/or implemented to reduce biases in the JEE. We discuss two other types of interventions next.
Interventions to reduce skew. Private coaching institutes that train students for JEE have been criticized for being exorbitantly expensive and, hence, inaccessible for students in low SES groups [44]. Lack of accessibility to training resources can reduce the scores of candidates in low SES groups—creating a skew in the scores. To improve accessibility to training, in 2022, the Delhi government established a new program that will provide free training to students enrolled in government-funded schools [53]. Similar programs have also been introduced in other states [52, 141] and by school education boards that span multiple states [26]. In the context of our model, these interventions can be seen as reducing this skew in the evaluation process.
Interventions to reduce information constraint. A criticism of JEE is that it is only offered in the English and Hindi languages. This is undesirable because only  $44\%$  of Indians report English or Hindi as their first language and, according to the 2011 census, less than  $68\%$  of Indians list one of these languages among the three languages they are most comfortable with [103]. IITs have been repeatedly criticized for not offering the exam in regional languages [128, 7, 47]. The main concern is that the current exam reduces the performance of students less familiar with English and Hindi. In the context of our model, this can be thought of as placing a stronger information constraint on candidates who do not speak English or Hindi as a first language: these students would need to spend a higher cognitive load to understand the questions. This constraint not only acts during the exam but also during the preparation period because students speaking regional languages (and not English or Hindi), have to devote additional time to learning either English or Hindi in addition to the technical material for the exam.
While the JEE exam itself has not been offered in regional languages yet. Recently, in 2021, the screening test that candidates have to clear before appearing in JEE was offered in 11 regional languages in addition to English and Hindi [134].
In this section, we compare the effectiveness of the above three interventions - Reservation for lower SES groups, interventions to reduce skew (change  $\alpha$  by  $\Delta_{\alpha}$  percent), and interventions to reduce information-constraint (change  $\tau$  by  $\Delta_{\tau}$  percent).
Setup (Group sizes and  $k$ ). Admissions into IITs are highly selective. For instance, in 2009, 384,977 applicants (232,334 from GEN;  $60\%$ ) took the exam and just 7,440 ( $2\%$ ) were admitted to IITs. The admission is based on the candidates' All India Rank (henceforth just rank)–which denotes the candidate's position in the list of candidates ordered in decreasing order of their scores in JEE. Let  $G_{1}$  be the group of students in GEN category and  $G_{2}$  be all other students. To study the impact of different interventions for admissions into IITs, we fix group sizes and  $k$  to match real numbers:  $|G_{1}| = 232,334$ ,  $|G_{2}| = 152,643$ , and  $k = 7,400$ . We focus on the set of candidates who scored at least 80 (out of 480) on the exam. (The threshold 80 ensures that at least 10k candidates outside
(a) Changing  $\alpha$  by  $\Delta_{\alpha}$  percent
(b) Changing  $\tau$  by  $\Delta_{\tau}$  percent
Figure 9: Effectiveness of different interventions on the selection-utility-as estimated by our model: We vary the strengths of the interventions  $(\Delta_{\alpha} \in [0,1]$  and  $\Delta_{\tau} \in [0,1])$  and report the expected utilities of the subset output by all three interventions. The  $x$ -axis shows the strength of the intervention changing  $\alpha$  (Figure 9(a)) or  $\tau$  (Figure 9(b)). The  $y$ -axis shows the ratio of the (true) utility of the subset output with an intervention to the (true) utility of the subset output without any intervention. Our main observation is that for each of the three interventions, there is a value of the percentage change in  $\alpha$  and  $\tau$  (i.e.,  $\Delta_{\alpha}$  and  $\Delta_{\tau}$  respectively) for which the intervention outperforms the other two interventions. Hence, depending on the amount of change a policymaker expects a specific intervention (e.g., providing free coaching) to have on the parameters  $\alpha$  and  $\tau$ , they can use our framework as a tool to inform their decision about which intervention to enforce. Error bars represent the standard error of the mean over 100 repetitions.
the GEN category are considered and this is significantly lower than the  $k$ -th highest score of 167). We fix  $f_{\mathcal{D}}$  to be the density of utilities of all candidates in  $G_{1}$  who scored at least 80. Since  $f_{\mathcal{D}}$  has a Pareto-like density (see Figure 10), we fix  $\ell(x, v) = \ln(x) - \ln(v)$ . We fix  $\Omega$  to be the set of all possible scores and  $f_{G_{2}}$  to be the density of all candidates in  $G_{2}$  who scored at least 80. As in Section 4, we select  $\alpha$  and  $\tau$  that lead to the density closest in TV distance to  $f_{G_{2}}$ . The rest of the setup is the same as in Section 4.
Unlike the main body, here, we only consider high-scoring candidates (those with a score of at least 80) because JEE is highly selective  $(k / n \leq 0.02)$  and, hence, to have meaningful results the estimated density  $f_{\mathcal{E}}$  should have a good fit to the density from the real-data on the top  $2\%$  quantile, i.e., the right tail. To ensure this, we specifically consider the right tail of the distribution (by dropping candidates with a score below 80).
Observations and discussion. We vary  $\Delta_{\alpha} \in [0,1]$  and  $\Delta_{\tau} \in [0,1]$  and report the expected utilities of the subset output by all three interventions over 100 iterations in Figure 9. Our main observation is that for each of the three interventions, there is a value of the percentage change in  $\alpha$  and  $\tau$  (i.e.,  $\Delta_{\alpha}$  and  $\Delta_{\tau}$  respectively) for which the intervention outperforms the other two interventions. Hence, depending on the amount of change a policymaker expects a specific intervention (e.g., providing free coaching) to have on the parameters  $\alpha$  and  $\tau$ , they can use our framework as a tool to inform their decision about which intervention to enforce. Further, we observe that, as expected, increasing  $\Delta_{\alpha}$  and  $\Delta_{\tau}$ , i.e., the percentage of change in  $\alpha$  and  $\tau$ , improves the utility achieved by the corresponding interventions.
Limitations and further discussion. Next, we discuss some of the limitations of our study. First, we note that interventions such as increasing the accessibility of education can not only reduce inequity
Figure 10: The densities of scores of students in GEN category (blue) and students not in GEN category (orange) in JEE-2009-only for students scoring at least 80 points out of 480.
in JEE but can also have positive effects on other exams and hiring. Hence, such interventions can have a larger positive (or negative) impact than suggested by our simulations. Studying these auxiliary effects is beyond the scope of this paper. Further, our study also does not model the response of the students, e.g., how do interventions affect the students' incentive to invest in skill development? Finally, our model only predicts the effect of  $\alpha$  and  $\tau$  on utility distributions. These predictions may not be accurate and a careful post-deployment evaluation may be required to accurately assess the effectiveness of different interventions.
L.2 Additional plots for simulations in Section 4
In this section, we present plots of the best-fit densities output by our framework on different datasets.
(a) Best-fit distribution  $(\alpha = 3 \cdot 10^{-4}$  and  $\tau = 1.51)$  with JEE-2009 (Birth category)
(b) Best-fit distribution  $(\alpha = 3\cdot 10^{-4}$  and  $\tau = 1.51$  with JEE-2009 (Gender)
Figure 11: Illustration of the best-fit distribution output by our framework for the JEE-2009 dataset. Captions of subfigures report the best fit  $\alpha$  and  $\tau$ .
L.3 Implementation details
L.3.1 Our framework and other models
In this section, we give implementation details of our model. Recall that our model outputs the density which is the optimal solution of the following optimization program.
Semantic Scholar Dataset -- Best Fits
Synthetic Network Dataset Best Fits
Figure 12: Illustration of the best-fit distribution output by our framework for the Semantic Scholar Open Research Corpus. Captions of subfigures report the best fit  $\alpha$  and  $\tau$ .
$$
\begin{array}{l} \operatorname {a r g m i n} _ {f: \text {d e n s i t y o n} \Omega} \operatorname {E r r} _ {\ell , \alpha} \left(f _ {\mathcal {D}}, f\right) := \int_ {\Omega} \left[ \int_ {\Omega} \ell_ {\alpha} (x, v) f (x) d \mu (x) \right] f _ {\mathcal {D}} (v) d \mu (v), (\text {O p t P r o g - A p p}) \\ \text {s u c h t h a t} \quad - \int_ {\Omega} f (x) \log f (x) d \mu (x) \geq \tau . \end{array}
$$
An instance of this program is specified by the following parameters.
1. A domain  $\Omega \subseteq \mathbb{R}$  (e.g.,  $\Omega = \mathbb{R}$  and  $\Omega = [1,\infty)$ );
2. A true density  $f_{\mathcal{D}}$  over  $\Omega$  with respect to the Lebesgue measure  $\mu$ ;
3. A loss function  $\ell \colon \Omega \times \Omega \to \mathbb{R}$  (e.g.,  $\ell(x, v) = (x - v)^2$  and  $\ell(x, v) = \ln(x / v)$ );
4. A risk-averseness (or risk-eagerness) parameter  $\alpha > 0$ ; and
5. A resource-information parameter  $\tau > 0$ .
Recall that  $\ell_{\alpha}$  is a risk-averse loss defined by  $\ell$  and  $\alpha$  as in (4). For our simulations, we consider the shifted variant of  $\ell_{\alpha}$  mentioned in Section 2: given a shift parameter  $v_0 \in \mathbb{R}$ , a loss function  $\ell \colon \Omega \times \Omega \to \mathbb{R}$ , and parameter  $\alpha > 0$
$$
\ell_ {\alpha , v _ {0}} (x, v) = \left\{ \begin{array}{l l} \alpha \cdot \ell (x, v + v _ {0}) & \text {i f} x > v + v _ {0}, \\ \ell (x, v + v _ {0}) & \text {o t h e r w i s e}. \end{array} \right.
$$
Let  $f_{\alpha, \tau, v_0}^{\star}$  be the optimal solution to the instance  $\mathcal{I}_{\alpha, \tau, v_0} = (\Omega, f_{\mathcal{D}}, \ell, \alpha, \tau, v_0)$  of (OptProg).
Algorithmic task. Given a "target" density  $f_{\mathcal{T}}$  (denoting the density of biased utilities in the data), risk-averse loss function  $\ell_{\alpha}$ , and true density  $f_{\mathcal{D}}$ , the goal of our implementation is to find  $\alpha^{\circ}, \tau^{\circ}$  and  $v_0^\circ$  that minimize the total variation distance between  $f_{\mathcal{T}}$  and  $f_{\alpha, \tau, v_0}^\star$ :
$$
(\alpha^{\circ},\tau^{\circ},v_{0}^{\circ}):= \operatorname *{argmin}_{\alpha ,\tau ,v_{0}}d_{\mathrm{TV}}(f_{\alpha ,\tau ,v_{0}}^{\star},f_{\mathcal{T}}).
$$
Algorithmic approach and implementation. We perform grid-search over all three parameters  $\alpha, \tau$ , and  $v_0$ . Given a specific  $\alpha, \tau$ , and  $v_0$ , to solve the above problem, we use the characterization in Theorem 3.1 to find  $f_{\alpha, \tau, v_0}^{\star}$ . Recall that the optimal solution of (OptProg) is of the following form
$$
f _ {\alpha , \tau , v _ {0}} ^ {\star} (x) = C \cdot \exp (- I _ {\alpha , v _ {0}} (x) / \gamma^ {\star})
$$
where  $I_{\alpha, v_0}(x) \coloneqq \int_{\Omega} \ell_{\alpha, v_0}(x, v) f_{\mathcal{D}}(x) d\mu(x)$  and  $C, \gamma^\star > 0$  are constants that are uniquely specified by the following two equations
$$
\int_ {\Omega} f _ {\alpha , \tau , v _ {0}} ^ {\star} (x) d \mu (x) = 1 \quad \text {a n d} \quad - \int_ {\Omega} f _ {\alpha , \tau , v _ {0}} ^ {\star} (x) \log \left(f _ {\alpha , \tau , v _ {0}} ^ {\star} (x)\right) d \mu (x) = \tau .
$$
Alge-matically, finding  $C$  and  $\gamma^{\star}$  requires computing a double integral over  $\Omega$ . In all of the simulations in Section 4,  $\Omega$  is a discrete domain, so these integrals reduce to summations and we compute them exactly. We also provide an implementation of our algorithm for continuous domains. The implementation for continuous domains uses the quad function in scipy to compute the integrals. For the grid search itself, we varied  $\alpha$  over  $[10^{-4},10^{2}]$ ,  $\tau$  over  $[10^{-1},10]$ , and  $v_{0}$  over  $\Omega$ . We found this range to be sufficient for our simulation, but it would be interesting to design a principled way of specifying the ranges given other parameters and target density  $f_{\mathcal{T}}$ .
Implementation details of multiplicative bias model [90] and implicit variance model [61]. Recall that the multiplicative bias and the implicit variance models are specified by parameters  $\rho$  and  $\sigma$  respectively: given a fixed true value  $v\in \mathbb{R}$ , the output of the multiplicative bias model is  $v / \rho$  and the output of the implicit variance model is  $v + \zeta$  where  $\zeta$  is a zero-mean normal random variable with variance  $\sigma^2$ . In addition, we allow both models to introduce a shift  $v_{0}$ . For the multiplicative bias model, given a true density  $f_{\mathcal{D}}$  and a target density  $f_{\mathcal{T}}$ , we compute  $(\rho^{\circ},v_0^{\circ})$  that solves  $\mathrm{argmin}_{\rho ,v_0}d_{\mathrm{TV}}(f_{\rho ,v_0},f_{\mathcal{T}})$  where  $f_{\rho ,v_0}$  is the density of  $(v / \rho) + v_{0}$  for  $v\sim f_{\mathcal{D}}$ . For the implicit variance model, given a true density  $f_{\mathcal{D}}$  and a target density  $f_{\mathcal{T}}$ , we compute  $(\sigma ,v_{0})$  that solves  $\mathrm{argmin}_{\sigma ,v_0}d_{\mathrm{TV}}(f_{\sigma ,v_0},f_{\mathcal{T}})$  where  $f_{\sigma ,v_0}$  is the density of  $v + v_{0} + \zeta$  for  $v\sim f_{\mathcal{D}}$  and a zero-mean normal random variable  $\zeta$  with variance  $\sigma^2$ . For both models, we compute the optimal parameters using grid search: we vary  $v_{0}$  over  $\Omega$ ,  $(1 / \rho)$  over  $[0,1]$ , and  $\sigma$  over  $[10^{-2},10]$ .
L.3.2 Computational resources used
All simulations were run on a MacBook Pro with 16 GB RAM and an Apple M2 Pro processor.
L.3.3 JEE-2009 Scores
Additional discussion of the dataset. The JEE-2009 test scores were released in response to a Right to Information application filed in June 2009 [91]. This dataset contains the scores of all students from JEE-2009 (384,977 total) [91]; we used the version available provided by [40]. In addition to the scores, for each student, the data contains their self-reported (binary) gender and their birth category. The birth category of a student is an officially designated indicator of their socioeconomic group, where the general (GEN) category is the most privileged; see [135, 19] for more details.
We observe that students not in the GEN category have significantly lower average scores than students in the GEN category (18.2 vs. 35.1); this may not imply that students not in the GEN category would perform poorly if admitted. Indeed, among students of equal true "potential," those from underprivileged groups are known to perform poorer on standardized tests [60]. In the Indian context, this could be due to many reasons, including that in India, fewer students outside the GEN category attend primary school compared to students from the general category, and on average a lower amount of money is spent on the education of students in the non-general category compared to the general category [92].
Figure 13: Distribution of scores in the JEE dataset for different protected groups based on birth category. See Section 4 for a discussion of the dataset.
L.3.4 Semantic Scholar Open Research Corpus
Cleaning and predicting author names. We follow the procedure used by [40]. Concretely, we remove papers without publication year (1.86% of total) and predict author gender using their first name from a publicly available dataset [3], containing first names and gender of everyone born between 1890 to 2018 and registered with the US social security administration (USSSA). We remove authors whose first name has 2 or fewer characters, as these names are likely to be abbreviations (retaining 75% of the total), and then categorize an author as female (respectively male) if more than  $\phi = 0.9$  fraction of the people of the same first name are female (respectively male) in the USSSA data. We drop all un categorized authors (32.25% of the remaining). This results in 3,900,934 women and 5,074,426 men (43.46% females). We present the tradeoff between the total number of authors retained and  $\phi$  in Figure 14.
Counting the number of citations. We aim to ensure that the citation counts we compute correspond to the total citations received by an author over their lifetime (so far). Since the dataset only contains citations from 1980 onwards, we remove authors who published their first paper before 1980 as the dataset does not have information about their earlier citations. This is the same as the cleaning procedure used by [40]. We present the resulting citation-distributions for male and female authors respectively in Figure 15.
Figure 14: The tradeoff between the threshold  $\phi$  used for clearing the Semantic Scholar Open Research Corpus and the number of authors retained. Details appear in Appendix L.3.4.
Figure 15: Distributions of total citations of men and women in the Semantic Scholar Open Research Corpus. Details appear in Appendix L.3.4.