Retiring Adult: New Datasets for Fair Machine Learning
Authors: Frances Ding
Moritz Hardt
John Miller
Ludwig Schmidt
Venue: NeurIPS 2021 Oral (2021)
Keywords: UCI Adult
datasets
benchmarks
Census data
fairness
archaeology
fair machine learning

Abstract: Although the fairness community has recognized the importance of data, researchers in the area primarily rely on UCI Adult when it comes to tabular data. Derived from a 1994 US Census survey, this dataset has appeared in hundreds of research papers where it served as the basis for the development and comparison of many algorithmic fairness interventions. We reconstruct a superset of the UCI Adult data from available US Census sources and reveal idiosyncrasies of the UCI Adult dataset that limit its external validity. Our primary contribution is a suite of new datasets derived from US Census surveys that extend the existing data ecosystem for research on fair machine learning. We create prediction tasks relating to income, employment, health, transportation, and housing. The data span multiple years and all states of the United States, allowing researchers to study temporal shift and geographic variation. We highlight a broad initial sweep of new empirical insights relating to trade-offs between fairness criteria, performance of algorithmic interventions, and the role of distribution shift based on our new datasets. Our findings inform ongoing debates, challenge some existing narratives, and point to future research directions.

Retiring Adult: New Datasets for Fair Machine Learning
Frances Ding" UC Berkeley
Moritz Hardt" UC Berkeley
John Miller UC Berkeley
Ludwig Schmidt" Toyota Research Institute
Abstract
Although the fairness community has recognized the importance of data, searchers in the area primarily rely on UCI Adult when it comes to tabular data. Derived from a 1994 US Census survey, this dataset has appeared in hundreds of research papers where it served as the basis for the development and comparison of many algorithmic fairness interventions. We reconstruct a superset of the UCI Adult data from available US Census sources and reveal idiosyncrasies of the UCI Adult dataset that limit its external validity. Our primary contribution is a suite of new datasets derived from US Census surveys that extend the existing data ecosystem for research on fair machine learning. We create prediction tasks relating to income, employment, health, transportation, and housing. The data span multiple years and all states of the United States, allowing researchers study temporal shift and geographic variation. We highlight broad initial sweep of new empirical insights relating to trade-offs between fairness criteria, performance of algorithmic interventions, and the role of distribution shift based our new datasets. Our findings inform ongoing debates, challenge some existing narratives, and point to future research directions.
Introduction
Datasets central to the machine learning ecosystem. Besides providing training and testing data for model builders, datasets formulate problems, organize communities, and interface between academia and industry. Influential works relating to the ethics and faimess of machine learning recognize the centrality of datasets, pointing to significant harms associated with data, as well as better data practices [17 21 25 27). While the discourse about data has prioritized cognitive domains such as vision, speech, or language, numerous consequential applications of predictive modeling and risk assessment involve bureaucratie, organizational, and administrative records best represented as
tabular data 8 15 26).
When it comes tabular data, surprisingly, most research papers algorithmic fairness continue to involve a fairly limited collection of datasets, chief among them the UCI Adult dataset [22]. Derived from the 1994 Current Population Survey conducted by the US Census Bureau, this dataset has made an appearance in more than three hundred research papers related to fairness where it served as basis for the development and comparison of many algorithmic faimess interventions.
Our work begins with a critical examination of the UCI Adult dataset-its origin, impact, and limitations. To guide this investigation we identify the previously undocumented exact source of the UCI Adult dataset, allowing us to reconstruct a superset of the data from available US Census records.
Authors ordered alphahetically
Email: (frances, hardt, f.allerrjoom)@beerklle.. edu Email: hudwigachmidtt20maall
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

This reconstruction reveals significant idiosyncrasy of the UCI Adult prediction task that limits its external validity.
While some issues with UCI Adult are readily apparent, such as its age, limited documentation, and outdated feature encodings, significant problem may be less obvious first glance. Specifically, UCI Adult has a binary target label indicating whether the income of a person is greater or less than fifty thousand US dollars. This income threshold of S50k US dollars corresponds to the 76th quantile of individual income in the United States in 1994, the 88th quantile in the Black population, and the 89th quantile among women. We show how empirical findings relating to algorithmic fairness are sensitive to the choice of the income threshold, and how UCI Adult exposes a rather extreme threshold. Specifically, the magnitude of violations in different fairness criteria, trade-offs between them, and the effectiveness of algorithmic interventions all vary significantly with the income threshold. In many cases, the S50k threshold understates and misrepresents the broader picture.
Turning to our primary contribution, we provide a suite of new datasets derived from US Census data that extend the existing data ecosystem for research on fair machine learning. These datasets are derived from two different data products provided by the US Census Bureau, One is the Public Use Microdata Sample of the American Community Survey, involving millions of US households each year. The other is the Annual Social and Economic Supplement of the Current Population Survey. Both released annually, they represent major surveying efforts of the Census Bureau that are the basis of important policy decisions, as well as vital resources for social scientists.
We create prediction tasks in different domains, including income, employment, health, transportation, and housing. The datasets span multiple years and all states of the United States, in particular, allowing researchers to study temporal shift and geographic variation. Alongside these prediction tasks, release a Python package called folktables which interfaces with Census data sources and allows users both access our new predictions tasks and create new tasks from Census data through simple 2
We contribute broad initial sweep of new empirical insights into algorithmic fairness based on new datasets. Our findings inform ongoing debates and in some cases challenge existing narratives about statistical fairness criteria and algorithmic fairness interventions. We highlight three robust observations:
1. Variation within the population plays major role in empirical observations and how they should be interpreted:
(a) Fairness criteria and the effect size of diffcrent interventions varies greatly by state. This shows that statistical claims about algorithmic fairness must be qualified carefully by context, even though they often are not.
(b) Training on one state and testing on another generally leads to unpredictable results. Accuracy and fairness criteria could change in either direction. This shows that algorithmic tools developed in one context may not transfer gracefully to another.
(c) Somewhat surprisingly, fairness criteria appear to be more stable over time than predic- tive accuracy. This is true both before and after intervention.
2. Algorithmic fairness interventions must specify a locus of intervention. For example, model could be trained on the entire US population, or on a state-by-state basis. The results differ significantly. Recognition of the need for such choice is still lacking, as is scholarship guiding the practitioner on how to navigate this choice and its associated trade-offs.
3. Increased dataset size does not necessarily help in reducing observed disparities. Neither does social progress measured in years passed. This is in contrast intuition from cognitive machine learning tasks where more representative data can improve metrics such error rate disparities between different groups
Our observations apply to years of active research into algorithmic fairness, and our work provides new datasets necessary to re-evaluate and extend the empirical foundations of the field.
'The datasets and Python package are available for download at https://github. cot.:/.oco/////iee /2y1s// folktables

---

2 Archaeology of UCI Adult: Origin, Impact, Limitations
Archaeology organises the past to understand the present. It lifts the dust-cod dust-cover off a world that we take for granted. It makes us reconsider what we experience as inevitable.
lan Hacking
Although taken for granted today, the use of benchmark datasets in machine learning emerged only in late 1980s [19]. Created in 1987, the UCI Machine Learning Repository contributed to this development by providing researchers with numerous datasets each with a fixed training and testing split [23]. As of writing, the UCI Adult dataset is the second most popular dataset among more than five hundred datasets in the UCI repository. An identical dataset is called "Census Income Data Set" and closely related larger dataset goes by "Census- Income (KDD) Data Set".
At the outset. UCI Adult contains 48,842 rows each apparently describing one individual with 14 attributes. The dataset information reveals that it was extracted from the "1994 Census database" according to certain filtering criteria. Since the US Census Bureau provides several data products, as we will review shortly, this piece of information does not identify the source of the dataset.
The fourteen features of UCI Adult include what the fairness community calls sensitive or protected attributes, such as, age, sex, and race. The earliest paper on algorithmic fairness that used UCI Adult to our knowledge is work by Calders al. 12 from 2009. The availability of sensitive attributes contributed to the choice of the dataset for the purposes of this work. An earlier paper in this context by Pedreschi al. (29] used the UCI German credit dataset, which smaller and ended up being less widely used in the community. Another highly cited paper on algorithmic fairness that popularized UCI Adult is the work of Zemel et al.
34 on learning fair representations (LFR). Published in 2013, the work introduced the idea of changing the data representation to achieve particular faimesss criterion, in this case, demographic parity, while representing the original data as well as possible. This idea remains popular in the community and the LFR method has become standard baseline.
Representation learning is not the only topic for which UCI Adult became the standard test case. The dataset has become broadly used throughout the area for purposes including the development of new fairness criteria, algorithmic interventions and fairness promoting methods, as well as causal modeling. Major software packages, such as AI Fairness 360 (7] and Fairlearn [9], expose UCI Adult as one of few standard examples. Indeed, based on bibliographic information available on Google Scholar there appear to be more than 300 papers related to algorithmic fairness that used the UCI Adult dataset at the time of writing.
2.1 Reconstruction of UCI Adult
Creating dataset involves a multitude of design choices that substantially affect the validity of experiments conducted with the dataset. To fully understand the context of UCI Adult and explore variations of its design choices, we reconstructed closely matching superset from the original Census sources. We now describe our reconstruction in detail and then investigate one specific design choice, the income binarization threshold, in Section 2.2
The first step in our reconstruction of UCI Adult was identifying the original data source. As mentioned above, the "1994 census database" description in the UCI Adult documentation does not uniquely identify the data product provided by the US Census Bureau. Based on the documentation of the closely related "Census-Incone Survey (CPS) data, specifically the Annual Social and Economic Supplement (ASEC) from 1994. We utilized the IPUMS interface to the CPS data [16] and hence refer to our reconstruction as IPUMS Adult.
(KDD) Data Set," we decided to start with the Current Population
The next step in the reconstruction was matching the 15 features in UCI Adult to the CPS data. This was a non-trivial task: the UCI Adult documentation does not mention any specific CPS variable names and IPUMS CPS contains more than 400 candidate variables for the 1994 ASEC. To address this challenge, we designed the following matching procedure that we repeated for each feature in UCI Adult: First, identify set of candidate variables in CPS via the IPUMS keyword search. For each candidate variable, use the CPS documentation to manually derive mapping from the CPS
Ron Kohavi is co-creator of both datasets.
3

---

11-11 Marc. P-,..-33 3) AP-1 1 Female, Y-11
IPUMS Adult
Demographic parily (DP) viclation
Equality of opporttunity (EO) violation
Pt. 11 p Af- 1 Femmlll)
0.55
0.4
0.90
0.55
0.2
0.1
0.00
0.75
0.1
0.0
0.70
0.0-
-0.1
0.05
-0.2
20000
10000
50000
20000
40000
60000
20000
40000
60000
Income thowshold
ticome threshoidd
Income threshold
GDM GBM W/ LFR
CBM w/ Postprocessing (DP)
GBM w/ ExpGred (DP)
UCI Adult threshold
Figure 1: Fairness interventions with varying income threshold on IPUMS Adult. We compare three methods for achieving demographic parity: pre-processing method (LFR), an in-training method based on Agarwal al. [2] (ExpGrad), and post-processing adjustment method [20). We apply each method using gradient boosted decision tree (GBM) as the base classifier. Confidence intervals are 95% Clopper-Pearson intervals for accuracy and 95% Newcombe intervals for DP.
encoding to the UCI Adult encoding. Finally, match each row in UCI Adult to its nearest neighbor in the partial reconstruction assembled from previous exact variable matches.
We only included a candidate variable if the nearest neighbor match was exact, i.e., we could find an exact match in the IPUMS CPS data for each row in UCI Adult that matched both the candidate variable and all earlier variables also identified via exact matches. There were only two exceptions to this rule. We discuss them Appendix A. After completing the variable matching, our reconstruction has 49,531 rows when we use the same inclusion criteria as UCI Adult to the extent possible, which slightly more than the 48,842 rows in UCI Adult. The discrepancy likely stems from the fact that UCI Adult used the variable "fnlwgt" in its inclusion criteria and we did not due to the lack of an exact match for this variable. This made our inclusion criteria slightly more permissive than those of UCI Adult. The fact that we found exact matches for 13 of the 15 UCI Adult variables and a very close match for "native-country" evidence that our reconstruction of UCT Adult accurate.
2.2 Varying income threshold
The goal in the UCI Adult dataset to predict whether an individual earns greater than 50,000 US dollars a year. The choice of the 50,000 dollar threshold is idiosyncratic and potentially limits the external validity of UCI Adult as a benchmark for algorithmic fairness. In 1994, the median US income was 26,000 dollars, and 50,000 dollars corresponds to the 76th quantile of the income distribution, and the 88th and 89th quantiles of the income distribution for the Black and female populations, respectively. Consequently, almost all of the Black and female instances in the dataset fall below the threshold and models trained on UCI adult tend to have substantially higher accuracies on these subpopulations. For instance, standard logistic regression model trained on UCI Adult dataset achieves 85% accuracy overall, 91.4% accuracy on the Black instances, and 92.7% on Female instances. This is rather untypical situation since often machine learning models perform more poorly on historically disadvantaged groups.
To understand the sensitivity of the empirical findings on UCI Adult to the choice of threshold, we leverage our IPUMS Adult reconstruction, which includes the continuous, unthresholded income variable, and construct new collection of datasets where the income threshold varies from 6,000 to 70,000. For each threshold, we first train a standard gradient boosted decision tree and evaluate both its accuracy and its violation of two common fairness criteria: demographic parity (equality of positive rates) and equal opportunity (equality of true positive rates). See the text by Barocas et al. [6] for background. The results are presented in Figure where we see both accuracy and the magnitude of violations of these criteria vary substantially with the threshold choice.
We then evaluate how the choice of threshold affects three common classes of faimess interventions: the preprocessing method LFR 34 mentioned earlier, an in-processing or in-Iraining method based on the reductions approach in Agarwal et al. (2), and the post-processing method from Hardt et al. 20 In Figure [ we plot model accuracy after applying each intervention to achieve demographic parity well violations of both demographic parity and equality of opportunity the income

---

Table 1: New prediction task details instantiated on 2018 US-wide ACS PUMS data
Constant
Task
Features
Datapoints
GBM acc predictor acc
LogReg acc
ACSIncome
10
1,599,229
63.1%
77.1%
79.7%
ACSPublicCoverage
19
1,127,446
70.2%
75.6%
78.5 %
ACSMobility
620,937
73.6%
73.7%
75.7%
ACSEmployment
17
2,320,013
56.7%
74.3%
78.5%
ACme 16
1,428,642
56.3%
57.4%
65.0%
threshold varies. Appendix A, we conduct the same experiment for methods achieve equality of opportunity. There are three salient findings. First, the effectiveness of each intervention depends on the threshold. For values of the threshold near 25,000, the accuracy drop needed to achieve demographic parity or equal opportunity is significantly larger than closer to 50,000. Second, the trade-offs between different criteria vary substantially with the threshold. Indeed, for the in-processing method enforcing demographic parity, as the threshold varies, the equality of opportunity violation monotonically increasing. Third, for high values of the threshold, the small number of positive instances substantially enlarges the confidence intervals for equality of opportunity, which makes it difficult to meaningfully compare the performance of methods for satisfying this constraint.
3 New datasets for algorithmic fairness
At least one aspect of UCI Adult is remarkably positive. The US Census Burcau invests heavily in high quality data collection, surveying methodology, and documentation based decades of experience. Moreover, responses to some US Census Bureau surveys are legally mandated and hence enjoy high response rates resulting in representative sample. In contrast, some notable datasets in machine learning are collected in an ad-hoc manner, plagued by skews in representation
[10 13 32 33,, often lacking copyright (4 consent from subjects [30], and involving unskilled or poorly compensated labor in the form of crowd workers (88
In this work, we tap into the vast data ecosystem of the US Census Bureau to create new machine learning tasks that we hope help to establish stronger empirical evaluation practices within the algorithmic fairness community.
As previously discussed, UCI Adult was derived from the Annual Social and Economic Supplement (ASEC) of the Current Population Survey (CPS). The CPS is monthly survey of approximately 60,000 US households. It's used to produce the official monthly estimates of employment and unemployment for the United States. The ASEC contains additional information collected annually.
Another US Census data product most relevant us are the American Community Survey (ACS) Public Use Microdata Sample (PUMS). ACS PUMS differs in some significant ways from CPS ASEC. The ACS is sent approximately 3.5 million US households each year gathering information relating to ancestry, citizenship, education, employment, language proficiency, income, disability, and housing characteristics. Participation in the ACS mandatory under federal law. Responses are confidential and governed by strict privacy rules. The Public Use Microdata Sample contains responses to every question from a subset of respondents. The geographic information associated with any given record is limited to a level that aims to prevent re-identification of survey participants. A number of other disclosure control heuristics are implemented. Extensive documentation is available on the websites of the US Census Bureau.
3.1 Available prediction tasks
We use ACS PUMS as the basis for the following new prediction tasks:
ACSIncome: predict whether an individual's 's income is above $50,000, after filtering the ACS PUMS data sample to only include individuals above the age of 16, who reported usual working hours of at least I hour per week in the past year, and an income of at least $100. The threshold of $50,000 was chosen so that this dataset can serve as a replacement UCI Adult, but we also offer datasets with other income cutoffs described in Appendix B.
5

---

ACSPublicCoveragee predict whether an individual is covered by public health insurance, after filtering the ACS PUMS data sample to only include individuals under the age of 65, and those with an income of less than $30,000. This filtering focuses the prediction problem on low-income individuals who are not eligible for Medicare.
predict whether an individual had the same residential address one year ago, after filtering the ACS PUMS data sample to only include individuals between the ages of 18 and 35. This filtering increases the difficulty of the prediction task, as the base rate of staying at the same address above 90% for the general population.
ACSMobility:
ACSEmployment: predict whether an individual is employed, after filtering the ACS PUMS data sample to only include individuals between the ages of 16 and 90.
predict whether an individual has a commute to work that longer than 20 minutes, after filtering the ACS PUMS data sample to only include individuals who are employed and above the age of 16. The threshold of 20 minutes was chosen as it is the US- USdwide median travel time to work in the 2018 ACS PUMS data release.
ACmmrave/Time:
All our tasks contain features for age, race, and sex, which correspond to protected categories in different domains under US anti-iiscrimination laws [5]. Further, each prediction task can be instantiated on different ACS PUMS data samples, allowing for comparison across geographic and temporal variation. We provide datasets for each task corresponding to 1) all fifty US states and Puerto Rico, and 2) five different years of data collection: 2014-2018 inclusive, resulting in a total of 255 distinct datasets per task to assess distribution shift. We also provide US-wide datasets for each task, constructed from concatenating each state's data. Table [ displays more details about each prediction task as instantiated on the 2018 USdwide ACS PUMS data sample. Our new tasks constitute a diverse collection of prediction problems ranging from those where machine learning achieves significantly higher accuracy than a baseline constant predictor to other potentially low- signal problems (ACSMobility) where accuracy improvement appears to be more challenging. We also provide the exact features included in each prediction task, and other details, in Appendix B. A datasheet [17] for our datasets is provided in Appendix E.
These prediction tasks are by no means exhausitive of the potential tasks one can construct using the ACS PUMS data. The folktables es package we introduce provides simple API that allows users to construct new tasks using the ACS PUMS data, and we encourage the community to explore additional prediction tasks beyond those introduced in this paper.
3.2 Scope and limitations
One distinction is important. Census data is often used by social scientists study the extent of inequality in income, employment, education, housing or other aspects of life. Such important substantive investigations should necessarily inform debates about discrimination in classification scenarios within these domains. However, our contribution is not in this direction. We instead use census data for the empirical study of algorithmic fairness. This generally may include performance claims about specific methods, the comparison of different methods for achieving given fairness metric, the relationships of different fairness criteria in concrete settings, causal modeling of different scenarios, and the ability of different methods to transfer successfully from one context to another. We hope that our work leads to more comprehensive empirical evaluations in research papers on the topic, at the very least reducing the overreliance UCI Adult and providing complement to the flourishing theoretical work on the topic. The distinction we draw between benchmark data and substantive domain-specifi investigations resonates with recent work that points out issues with using data about risk assessments tools from the criminal justice domain as machine learning benchmarks [4.
notable obvious limitation of our work is that is entirely US-centric. A richer dataset ecosystem covering international contexts within the algorithmic fairness community is still lacking. Although empirical work in the Global South central in other disciplines, there continues to be much need for the North American faimeess community to engage with it more strongly MI

---

GBM w/ LFR
w/ ExpGrad IDP)
GBM w/ Past-processing IDP)
0.3
0.3
0.3
0.2
0.2
0.2
PV- 1 [White] POY
0.0
0.0
0.0
0.78
0.80
0.82
0.84
0.78
0.80
0.82
0.84
0.78
0.80
0.82
0,84
State
State
State
The effect size of fairness interventions varies by state. Each panel shows the change in accuracy and demographic parity on the ACSIncome task after applying fairness intervention to an unconstrained gradient boosted decision tree (GBM). Each arrow corresponds to a different state distribution. The arrow base represents the (accuracy, DP) point corresponding to the unconstrained GBM, and the head represents the (accuracy, DP) point obtained after applying the intervention. The arrow for HI in the LFR plot is entirely covered by the start and end points.
Figure 2:
4 A tour of empirical observations
In this section, we highlight an initial sweep of empirical observations enabled by our new ACS PUMS derived prediction tasks. Our experiments focus on three fundamental issues in fair machine learning: (i) variation within the population of interest, e.g., how does the effectiveness of interventions vary between different states or over time?, (ii) the locus of intervention, e.g. should interventions be performed at the state or national level?, and (iii) whether increased dataset size or the passage of time mitigates observed disparities?
Our experiments not exhaustive and are intended highlight the perspective broader empirical evaluation with our new datasets can contribute to addressing questions within algorithmic fairness. The goal of the experiments is not to provide complete overview of all the questions that one can answer using our datasets. Rather, we hope to inspire other researchers to creatively use our datasets to further probe these question as well as propose new ones leveraging the ACS PUMS data.
4.1 Variation within the population
The ACS PUMS prediction tasks present two natural axes of variation: geographic variation between states and temporal variation between years the ACS is conducted. This variation allows us to both measure the performance of different fairness interventions on a broad collection of different distributions, as well study the performance of these interventions under geographical and temporal distribution shift when the test dataset differs from the one on which the model was trained.
Due to space constraints, we focus our experiments in this section the ACSIncome prediction task with demographic parity as the fairness criterion of interest. We present similar results for our other prediction tasks and faimeess criteria, as well as full experimental details in Appendix D.
Intervention effect sizes vary across states. The fifty US states which comprise the ACS PUMS data present a broad set of different experimental conditions on which to evaluate the performance of fairness interventions. At the most basic level, we can train and evaluate different fairness interven- tions on each of the states and compare the interventions' efficacy on these different distributions. Concretely, we first train an unconstrained gradient boosted decision tree (GBM) on each state, and we compare the accuracy and fairness criterion violation of this unconstrained model with the same model after applying one of three common fairness intervention: pre-processing (LFR), the in- processing fair reductions methods from Agarwal et al. 2 (ExpGrad), and the simple post- processing method that adjusts group-based acceptance thresholds to satisfy a constraint 20). [igure[]shows the result of this experiment for the ACSIncome prediction task for interventions to achieve demographic parity. For given method, performance can differ markedly between states. For instance, LFR decreases the demographic parity violation by 10% in some states and in other states the decrease close to zero. Similarly, the post-proes adjustment to enforce demographic parity incurs accuracy drops of less than 1 % in some states, whereas in others the drop is closer to 5%.
7

---

ACSIncome 2018
1-Year
1| [Whit)) -PIY 1| BBackk) PV- [White] =11 |Bback]
GBM, train on
GBM, train on
GBM, train on
GBM, train on NY
0.4
[CA] awatual evalixtion
0.4
TEA) wallzaioon evalintion
0.4
ID (IN) wvslustion 0G0 estiustion
0.4
0.3
0.3
0.3
0.3
0.2
0.2
0.2
0.2
0.1
0.1
0.1
0.1
0.0
0.0
0.0
0.0
oop exoluation
-0.1
-0.1
-0.1
-0.1
0.65
0.75
0.85
0.65
0.75
0.85
0.65
0.70
0.75
0:85
0.70
0.75 0.80 Accuracy
Accuracy
Accuracy
Accuracy
Postprocess (DP) CA
Postpracess (DP) TX
Postprocess (DP)
Postpracess (DP) on NY
0.4
(CA) ovsluation on
0.4
ID (h0 ovaliction
0.4
D (IN evalustion
0.4
ID INYI ovaluation 000 evaluation
OOD evalustion
000 avaluation
0CD eva evtl
0.2
0.2
0.2
0.2
0.1
0.1
0.1
0.1
0.0
-0.1
-0.1
-0.1
-0.1
0.65
0.70
0.75 Accuracy
0.80
0.85
0.65
0.70
0.75
0.80
0.85
0.65
0.70
0.75
0.80
0.85
0.65
0.70
0.75
0.80 0.85
Accuracy
Accuracy
Accuracy
Figure 3: Transfer from one state to another gives unpredictable results in terms of predictive accuracy and fairness criteria. Top: Each panel shows an unconstrained GBM trained on a particular state on the ACSIncome task and evaluated both in-distribution (ID) on the same state and out-of- distribution (OOD) on the 49 other states in terms of accuracy and demographic parity violation. Bottom: Each panel shows an GBM with post-processing to enforce demographic parity on the state on which it was trained and evaluated both ID and OOD on all 50 states. Confidence intervals are 95% Clopper- Pearson intervals for accuracy and 95% Newcombe intervals for demographic parity.
Training and testing on different states leads to unpredictable results. Beyond training and evaluating interventions on different states, we also use the ACS PUMS data to study the performance of interventions under geographic distribution shift, where we train a model on one state and test it on another. In Figure 3 we plot accuracy and demographic parity violation with respect to race for both an unconstrained GBM and the same model after applying post-processing adjustment to achieve demographic parity on a natural suite test sets: the in-distribution (same state test set) and the distir-bittiiiiioonnnn test sets for the 49 other states. For both the unconstrained and proseproeessed model, model accuracy and demographic parity violation varies substantially across different state test sets. In particular, even when method achieves demographic parity in one state, it may longer satisfy the fairness constraint when naively deployed on another.
Fairness criteria are more stable over time than predictive accuracy. In contrast to the unpre- dictable results that occur under geographic distribution shift, the fairness criteria and interventions we study are much more stable under temporal distribution shift. Specifically, in Figure 4 we plot model accuracy and demographic parity violation for GBM trained on the ACSIncome task using US-wide data from 2014 and evaluated on the test sets for the same task drawn from years 2014-2018 Perhaps unsurprisingly, model accuracy degrades slightly over time. However, the associated fairness metric is stable and essentially constant over time. Moreover, this same trend holds for the faireess interventions previously discussed. The same base GBM with pre-processing (LFR), in-processing (ExpGrad), or post-processing to satisfy demographic parity in 2014, all have a similar degradation accuracy, but the fairness metrics remain stable. Thus, a classifier that satisfies demographic parity on the 2014 data continues to satisfy the constraint on 2015-22 2015-2018 data.
4.2 Specifying a locus of intervention
On the ACSPUMs prediction task, fairness interventions can be applied either on state-by-state basis or on the entire US population. In Table [ we compare the performance of LFR and the post-processing adjustment method applied at the US-level with the aggregate performance of both methods applied on a state-by-state basis, using GBM as the base classifier. In both cases, applying the intervention on state-by-state improves USdw accuracy while still preserving demographic parity (post-processing) or further mitigating violations of demographic parity (LFR).
8

---

ACSIncome US-Wide (Trained 2014
0.80
Black]
0.175
0,130
0.84
0.125
Accuracy
0.100
[hite)
0.075
0.80
0.050
0.73
0.023
6,000
0.75
2018
2014
Evaluation year
Evaluation year
GBM
GBM ExpGrad (DP)
- GBM w/ Pestprocessing (DP)
GBM wT
Figure 4: Fairness criteria on US-wide ACSIncome with and without fairness interventions to achieve demographie parity and evaluated data in subsequent years suffer drop in accuracy violation of demographic parity remains essentially constant over time. Confidence intervals are 95% Clopper-Pearson intervals for accuracy and 95% Newcombe intervals for demographic parity.
stable over time than accuracy. Left: Models trained in 2014
time. Right: However, the
Table Comparison of two different strategies for applying an intervention achieve demographic parity (DP) on the US-wide ACSIncome task. US-level corresponds to training one classifier and applying the intervention on the entire US population. State-level corresponds to training classifier and applying the intervention separately for each state and then aggregating the results over all states. Here, DP refers to p(ś = 1 White) - P(Ý 1 Black). Confidence intervals are 95% Clopper-Pearson intervals for accuracy and 95% Newcombe intervals for DP.
US-level
State-level DP violation
US-Ievel acc
State-level acc
DP violation
Unconstrained GBM GBM w/ LFR
81.7 + 0.1 %
17.7 + 0.2%
82.8 + 0.1 %
16.9 + 0.2%
78.7 + 0.1 %
16.6 + 0.2%
79.4 + 0.1%
14.0 = 0.2%
GBM w/ post-processing (DP)
79.2 + 0.1
0.3 + 0.3 %
80.2 + 0.1%
-0.6 + 0.3%
4.3 Increased dataset size doesn't necessarily mitigate observed disparities
To mitigate disparities in error rates, commonly suggested remedies include collecting a) larger datasets and b) more representative data reflective of social progress. For example, response to research revealing the stark accuracy disparities of commercial facial recognition algorithms, particularly for dark-skinned females [TI.. IBM collected a more diverse training set of images, retrained its facial recogmition model, and reported 10-fold decrease in еттог for this subgroup [T]] However, on our tabular datasets, larger datasets collected in more socially progressive times do not automatically mitigate disparities. Table 3 shows that unconstrained gradient boosted decision tree trained on a newer, larger dataset (ACSIncome vs. IPUMS Adult), does not improve disparities such as in true positive rate (TPR). A fundamental reason for this is the persistent social inequality that is reflected in the data. It is well known that given a disparity in base rates between groups, a predictive model cannot be both calibrated and equal in error rates across groups [4), except if the model has 100% accuracy. This observation highlights a key difference between cognitive machine learning and tabular data prediction the Bayes eиror rate is zero for cognitive machine learning. Thus larger and more representative datasets eventually address disparities by pushing error rates to zero for all subgroups. In the tabular datasets collect, the Bayes error rate of an optimal classifier is almost certainly far from zero, so some individuals will inevitably be incorrectly classified. Rather than hope for future datasets to implicitly address disparities, must directly contend with how dataset and model design choices distribute the burden of these errors.
5 Discussion and future directions
Rather than settled conclusions, our empirical observations are intended spark additional work on our new datasets. Of particular interest is broad and comprehensive evaluation of existing methods on all datasets. We only evaluated some methods far. One interesting question is if there is method

---

Table 3: Disparities persist despite increasing dataset size and social progress.
Dataset
Year
Datapoints
GBM acc
TPR White
TPR Black
TPR disparity
IPUMS Adult
1994
49,531
86.4% 80.8%
58.0% 66.5%
46.5 %
11.5%
ACSIncome
2018
1,599,229
51.7%
14.8%
for achieving either demographic parity error rate parity that outperforms threshold adjustment (based on the best known unconstrained classifier) on any our datasets? We conjecture that the answer is no. The reason is that we believe on datasets well-tuned tree-ensemble achieves classification error close to the Bayes error bound. Existing theory (Theorem 5.3 in [20]) would then show that threshold adjustment based this model is, in fact, optimal. Our conjecture motivates drawing a distinction between classification scenarios where nearly Bayes optimal classifier is known and those where there isn't. How close we tasks is a good question. The role of distribution shift also deserves more attention. Are there methods that achieve consistent performance across geographic contexts? Why does there appear to be more temporal than geographic stability? What does the sensitivity to distribution shift say about algorithmic tools developed in one context and deployed in another? Answers to these questions seem highly relevant to policy-making around the deployment of algorithmic risk assessment tools. Finally, our datasets are also interesting test cases for causal inference methods, which we haven't 't yet explored. How would, for example, methods like invariant risk minimization geographic contexts?
Bayes optimal any of new prediction
[ perform on different
Acknowledgements
We thank Barry Becker and Ronny Kohavi for answering our many questions around the origin and creation of the UCI Adult dataset. FD and JM are supported by the National Science Foundation Graduate Research Fellowship Program under Grant No. DGE 1752814 14. FD is additionally supported by the Open Philanthropy Project AI Fellows Program.
10

---

References
[1] R. Abebe, K. Aruleba, A. Birhane, S. Kingsley, G. Obaido, S. L. Remy, and S. Sadagopan. Narratives and counternarratives data sharing in africa. In Proc. of the ACM Conference on Fairness, Accountability, and Transparency, pages 329-341, 2021.
[2] A. Agarwal, A. Beygelzimer, M. Dudík, J. Langford, and H. Wallach. A reductions approach to fair classification. In International Conference Machine Learning, pages 60-69. PMLR, 2018.
[3] M. Arjovsky, L. Bottou, I. Gulrajani, and D. Lopez-Paz. Invariant risk minimization. arXiv preprint 1907.01997..000 02893, 2019.
[4] M. Bao, A. Zhou, S. Zottola, B. Brubach, S. Desmarais, A. Horowitz, K. Lum, and S. Venkata- subramanian. It's compaslicated: The messy relationship between rai datasets and algorithmic fairness benchmarks. arXiv preprint arXiv:2106.00 10.0:22006005498,, 2021.
[5] S. Barocas and A. D. Selbst. Big data's 's disparate impact. California Law Review, 104, 2016.
[6] S. Barocas, M. Hardt, and A. Narayanan. Fairness and Machine Learning. fairmlbook.org..
2019. http::/www.. www.fiirrooooooo. org
[7] R. K. Bellamy, K. Dey, M. Hind, S. C. Hoffman, Houde, K. Kannan, P. Lohia, Martino, S. Mchta, A. Mojsilović,, et al. Ai fairesss 360: An extensible toolkit for detecting and mitigating algorithmic bias. IBM Journal of Research and Development, 63(4/5):4 2019.
[8] R. Benjamin, Race after Technology. Polity, 2019.
[9] Bird, M. Dudík, R. Edgar, B. Horn, R. Lutz, V. Milan, M. Sameki, H. Wallach, and K. Walker. Fairlearn: A toolkit for assessing and improving fairness in ai. Microsoft, Tech. Rep. MSR-TR- 2020-32, 2020.
[10] T. Bolukbasi, K.-W. Chang, Y. Zou, V. Saligrama, and A. T. Kalai. Man to computer programmer as woman is to homemaker? debiasing word embeddings. Advances in Neural Information Processing Systems, 2016.
[11] Buolamwini and Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Fairness, Accountability and Transparency, pages 77-91, 2018.
[12] T. Calders, F. Kamiran, and M. Pechenizkiy. Building classifiers with independency constraints. In In Proc. JEEE ICDMW, pages 13-18, 2009.
[13] A. Caliskan, J. Bryson, and A. Narayanan. Semantics derived automatically from language corpora contain human-like biases. Science, (1166334)118-18888 2017.
[14] A. Chouldechova. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big data, 5(2):153-16)) 163, 2017.
[15] V. Eubanks. Automating inequality: How high-tech tools profile, police, and punish the poor.
St. Martin's 's Press, 2018
[16] S. Flood, M. King, R. Rodgers, S. Ruggles, and J. R. Warren. Integrated Public Use Microdata Series, Current Population Survey: Version 8.0 [dataset], 2020. Minneapolis, MN: IPUMS, https://doi. http:///ooo.oo/////1o
[17] T. Gebru, J. Morgenstern, B. Vecchione, J. W. Vaughan, H. Wallach, H. Daumé III, and K. Crawford. Datasheets for datasets. / 2018.
[18] M. L. Gray and S. Suri. Ghost work: how stop Silicon Valley from building a new global underclass. Famon Dolan Books, 2019.
[19] M. Hardt and B. Recht. Patterns, predictions, and actions: A story about machine learning.
https://mlstory: org 2021.
[20] M. Hardt, E. Price, and N. Srebro. Equality of opportunity in supervised learning. In Proc. 29th NIPS, pages 3315-3323, 2016.
[21] E. S. Jo and T. Gebru. Lessons from archives: strategies for collecting sociocultural data in machine learning. In Fairness, Accountability and Transparency, pages 306-316, 2020.
[22] R. Kohavi and B. Becker. Uci adult data set. UCI Meachine Learning Repository, 5, 1996.
[23] P. Langley. The changing science of machine learning, 201
11

---

[24] A. Levendowski. How copyright law can fix artificial intelligence's 's implicit bias problem. Wash. Rev., 93:579, 2018.
[25] M. Onuoha. The point of collection. Data & Society: Points, 2016.
[26] F, Pasquale. The black box society. Harvard University Press, 2015.
[27] A. Paullada, I. D. Raji, E. M. Bender, E. Denton, and A. Hanna. Data and its (dis) con-
tents: survey of dataset development and use in machine learning research. arXiv preprint arXiv:2012.05544 2020.
[28] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournaeea,, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
[29] D. Pedreschi, S. Ruggieri, and F. Turini. Discrimination-awanl data mining. In Proc. 14th SIGKDD. 2008.
[30] Prabhu and Birhane. Large image datasets: A pyrrhic win for computer vision? arXiv preprint arXiv:2006. 16923, 2020.
[31] R. Puri. Mitigating bias in artificial intelligence (ai) models ibm research, Feb 2019. URL https: :/www. imm.
bias- ai moodess [32] A. Torralba and A. A. Efros. Unbiased look at dataset bias. In CVPR 2011, pages 1521-1528. IEEE, 2011.
[33] K. Yang, K. Qinami, Fei-Fei, J. Deng, and O. Russakovsky. Towards fairer datasets: Filtering and balancing the distribution of the people subtree in the imagenet hierarchy. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pages 547-558, 2020.
[34] R. Zemel, Y. Wu, Swersky, Pitassi, and Dwork. Learning fair representations. In Proceedings of the 30th International Conference on International Conference on Machine Learning, pages III-325, 2013.
12

---

NeurIPS Paper Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 3.2
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 3.2
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material as URL)? [Yes] The code and data are included the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Please see Appendix C.
(c) Did you report eтror bars (e.g., with respect the random seed after running ex- periments multiple times)? [Yes] We report all results with appropriate confidence intervals accounting for random variation in estimating accuracies or demographic parity/equality of opportunity violations.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Please see Appendix C.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] We cite and discuss the UCI Adult dataset in Section 2
(b) Did you mention the license of the assets? [Yes] See Appendix B.
(c) Did you include any assets either in the supplemental material URL? [Yes] We include our new prediction tasks in the supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [Yes] See Section
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [Yes] See Section
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "fEJwIR7UnI", "reviewer": "Reviewer_hnqZ", "summary": "The authors present a suite of datasets derived from recent US Census survey data and covering a variety of prediction tasks based on tabular demographic data. One of their datasets is a superset of the popular UCI Adult dataset, which is commonly used in many algorithmic fairness papers. The authors show the effect of various dataset design choices (such as the threshold used to binarize income) on accuracy and fairness metrics, as well as on the conclusions one might draw about different fairness interventions. The authors also provide a preliminary analysis of the other datasets, showcasing vignettes such as the effect of distribution shifts over states and time, and the effect of increasing dataset size on fairness metrics.\n\nI thought this was an excellent, well-written paper that addresses an important need within the algorithmic fairness community, namely, the complete lack of high-quality datasets. Beyond the dataset contribution, the empirical analysis that the authors provide is interesting and highlights the potential of these datasets. Kudos to the authors for this thorough and careful work.\n\n------\n\nUpdate: Thanks to the authors for their detailed responses. I would like to keep my score: I think this is a strong and positive contribution.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "As mentioned above, I think this is a strong paper that will be a positive contribution to the field of algorithmic fairness, and more generally, to the development of ML methods for tabular data. (In hindsight, it is an unfortunate reflection of the state of the field of algorithmic fairness that it took this long to generate better datasets from the publicly-available US Census data.)\n\nI'll focus here on some suggestions and questions:\n\n1. Why are the 5 prediction tasks outlined in the paper useful? Is the argument that there are real-world settings where we would, for example, want to predict if an individual's income is above a threshold given other demographic information, or predict if an individual is covered by public health insurance? Given that the empirical phenomena (the behavior of fairness interventions, the effect of distribution shifts, etc.) can vary significantly from task to task, it would be helpful to understand why we should be keyed to these tasks in particular. \n\n2. If, as claimed, the models are already close to Bayes optimal on these datasets, then what is the point of these datasets from the perspective of ML research? It would be interesting to hear the authors' thoughts on this. One perspective, for example, is that given the impossibility results around finding the one true perfect classifier, if our models are already obtaining Bayes optimal error rates on these datasets, then it is a solved problem as far as ML research goes (in the sense that the right tradeoffs between various notions of accuracy and fairness are questions for society and policymakers to decide, as opposed to algorithmic ones). Should the fairness community focus on the \"cognitive machine learning\" tasks that the authors note?\n\n3. What is driving the difference between in-distribution and out-of-distribution accuracy? To what extent can this be solved, e.g., if an (oracle) model is trained on all or multiple states, does it close the gap? The discussion from the authors about how the models are close to Bayes optimal seems to suggest that the differences in performance between states might be unavoidable, in the sense that they simply reflect the properties of different data distributions in each state (for example, perhaps some states have data that's noisier; or perhaps label balance varies, so it always helps to learn the right bias for the in-distribution state).\n\nNit: typo in L1194, \"oover\"", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Overall, the authors have done a commendable job addressing the limitations and societal impact of their paper. My main feedback in this direction is that the authors try to divorce the application from the ML task, in the sense that it almost doesn't matter for the authors' purposes what the actual prediction task is or where the data comes from. However, I think this perspective has been a persistent limitation of the work on algorithmic fairness, in that it deals with an abstract notion of \"fairness\" that can be hard to translate into a real application. An ideal benchmark in algorithmic fairness would not only be high-quality from an ML point of view (lots of data, clean, etc.) but also compelling as an actual application. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "FM5GVvRMrAP", "reviewer": "Reviewer_7q2x", "summary": "This paper provides a new dataset on which to ask questions around fairness in machine learning, intended as a replacement for the frequently-used UCI Adult dataset. The authors attempt to reconstruct the Adult dataset from US Census data, provide a broader dataset based on the American Community Survey Public Use Microdata Sample with several predefined prediction tasks, and perform an initial analysis of the new dataset from a fairness perspective.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper makes a solid contribution to the field of fairness in machine learning. The authors show how it can be used to ask a variety of new questions around temporal and geographical differences, among others.\n\nOne minor point that I feel is worth including -- the ACS data includes some noise due to the Census Disclosure Avoidance System, and it's not fully known how this operated in 1994 (see, e.g., \"A History of the Current Population Survey and Disclosure Avoidance\" (McKenna, 2019)).\n\nOther than that, I don't have major comments -- the paper is well-written and makes an important contribution.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors make certain choices in constructing the datasets, in particular with respect to the prediction tasks they define. It may be worth elaborating on those choices from the perspective of what types of outcomes they're encouraging others to predict.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "9R9h6N-3ZQW", "reviewer": "Reviewer_vtAB", "summary": "They have reconstructed the famous UCI adult dataset to seemingly high accuracy, using the CPS data that provides granularity at the individual level. Furthermore, as their reconstructed dataset now spans multiple years and all states of the United States as opposed to the original adult dataset which I believe is specific to 1994, they were able to created other related datasets for different prediction tasks. \n\nThey have carried out a bunch of experiments that demonstrate how different seemingly innocuous decisions can lead to very different outcomes. For example, because they have the raw data for the income as opposed to the original adult dataset which indicates whether the income was above 50k or not, they can see how varying the income threshold can lead to different outcomes.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: \n\nI don't know of any prior work trying to reconstruct the adult dataset and try to understand some of the contexts behind the dataset. Also, the new datasets (as a result of being able to span multiple years and states) are novel.\n\nQuality:\n\nTheir discussion of the reconstruction methodology (e.g. section 2.1) is convincing in that I believe that they have reconstructed the dataset with very high accuracy.\n\nClarity:\n\nOverall, the paper reads very easily in terms of its presentation of the results. \n\nStrength: \n\nHaving the actual raw income data from which the adult data's feature (>50k, <= 50k) is derived is important. By choosing a threshold at different quantiles, it allows other researchers to induce different distributions and understand the effect of different algorithms in those different distributions. I think this point can be even more emphasized in the paper.\n\nWeakness: \n\nA lot of the conclusions derived from the experiments seem to all point to the fact that each state's distribution is very different from one another whereas there isn't that much distribution shift in time.  \n\nThe tradeoff between accuracy and fairness heavily depends on the distribution itself, so it is intuitive that different states see different effectiveness in intervention. If there's a large discrepancy between the training and test distribution, one cannot expect the classifier trained on the training dataset to behave nicely on the test distribution.  \n\nAlso, the experiment purporting to show that increased datasest doesn't mitigate observed disparities is a little weird in that it is looking at two different distributions (1994 vs 2018) of different sizes as opposed to the same distribution of different sizes (same year but one is subsampled from the other). \n\nSo, I think it may be more helpful to emphasize that these datasets allow one to see that there is indeed much variation in distributions across states and in some other dimensions — and the consequences of such variation — instead of making separate claims about the nature of the dataset.   \n\nQuestions/comments:\n\n-The description of CPS doesn't show up until section 3 (e.g. that it stands for Current Population Survey), but this acronym is used without much discussion already in section 2.1. Similarly, what does IPUMS stand for?\n\n-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\nThanks for answering the questions! Despite my hesitancy about technical findings of the new dataset, I agree that the new dataset alone is a valuable contribution. I will be bumping up the score to 7", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Is there any concern for privacy? For instance, is it possible that one can use the methodology used here to reconstruct the adult dataset to reconstruct dataset sets for more recent years? Or does CPS data not available for recent years for privacy reasons?  ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "6T6wf68x-J6", "reviewer": "Reviewer_raqf", "summary": "The authors focus on the adult dataset, which was originated from the US census data and has records of individuals with a binary label of whether the income is higher or lower than 50k. The adult dataset has been one of the gold standard datasets in the algorithmic fair ML community, especially to test for the performance of new fairness metrics and fair algorithms. The authors discuss few drawbacks of the adult dataset and reconstructed a superset of the UCI adult dataset from the available census datasets released in the same year, namely the Annual Social and Economic Supplement (ASEC). They refer to their reconstruction as IPUMS adult. Furthermore, the authors also looked into the American Community Survey (ACS) Public Use Microdata Sample (PUMS) and defined multiple prediction tasks based on the ACS PUMS dataset. \n\n-------------------------------------------------------------------------------------------------------------------------\n\nupdate: Thanks for authors' comments. I'll be increasing my score as I think this is an important contribution to the community, although I still think that the level of technical novelty is below what is expected of NeurIPS main track. ", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Like many other ML researchers, I worked with the adult dataset a few times. I also always wished that I had access not only to the binary labels for the income being higher/lower than 50k but also to the actual income values. This is a great contribution, looking more closely into the adult dataset, discussing how different fairness criteria and fair algorithms differ in their fairness and accuracy based on the threshold, discussing more generally how state-wide analysis could be different from the country-wide analysis and many other aspects. I’m just not sure, that given the technical contribution of the paper if the scope of technical novelty would fit what has been traditionally the acceptance bar for NeurIPS and whether NeurIPS is the correct venue for this work. But I’d like to see it published somewhere, and this would be a paper that I will look into in the future for data resources. \n\nMinor/representation:\n\n— Remind the reader that GBM denotes gradient boosted decision tree already in Figure 1, where it’s used for the first time.\n— The abbreviation CPS is used the first time on page 3 but only introduced later on page 5. \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Discussed in the main review.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Thanks for the strong submission. The reviewers were unanimous that the paper provided a valuable contribution, and disagreed only on whether it was submitted to the right track. Our job is to provide platform for good work, and since there was no disagreement on the quality of the work, we decided to accept. 
Decision: Accept (Oral)