Deep Ecological Inference
Authors: Nic Fishman
Colin McAuliffe
Venue:  (2021)
Keywords: ecological inference
representation learning
multi-task learning
bayesian deep learning

Abstract: We introduce an efficient approximation to the loss function for the ecological inference problem, where individual labels are predicted from aggregates. This allows us to construct ecological versions of linear models, deep neural networks, and Bayesian neural networks. Using these models we infer probabilities of vote choice for candidates in the Maryland 2018 midterm elections for 2,322,277 voters in 2055 precincts. We show that increased network depth and joint learning of multiple races within an election improves the accuracy of ecological inference when compared to benchmark data from polling. Additionally we leverage data on the joint distribution of ballots (available from ballot images which are public for election administration purposes) to show that joint learning leads to significantly improved recovery of the covariance structure for multi-task ecological inference. Our approach also allows learning latent representations of voters, which we show outperform raw covariates for leave-one-out prediction. 

Under review as a conference paper at ICLR 2021
DEEP ECOLOGICAL INFERENCE
Anonymous authors Paper under double-blind review
ABSTRACT
We introduce an efficient approximation to the loss function for the ecological inference problem, where individual labels are predicted from aggregates. This allows us to construct ecological versions of linear models, deep neural networks, and Bayesian neural networks. Using these models we infer probabilities of vote choice for candidates in the Maryland 2018 midterm elections for 2,322,277 voters in 2055 precincts. We show that increased network depth and joint learning of multiple races within an election improves the accuracy of ecological inference when compared to benchmark data from polling. Additionally we leverage data on the joint distribution of ballots (available from ballot images which are public for election administration purposes) to show that joint learning leads to significantly improved recovery of the covariance structure for multi-task ecological inference. Our approach also allows learning latent representations of voters, which we show outperform raw demographics for leave-one-our prediction.
INTRODUCTION
Ecological inference (EI), learning labels from label proportions, is the problem of trying to make predictions about individual units from observations about aggregates. The canonical case is voting. We cannot observe individual people's votes, but people live in precincts, and we know for each precinct what the final vote count was. The problem is to try to estimate probabilities that a particular type of individual voted for candidate.
Since we can not observe individual labels, but only sums of pre-specified groups of labels, non- identifiability is inherent the ecological inference problem. The possibility of interaction effects between any relevant demographics and the aggregation groups themselves also means that Simp- son's paradox type confounding an ever present risk. The most basic approach to this problem involves assuming total heterogeneity at the precinct level, and simply assigning the final distribu- tion of votes in a precinct to each person living in that precinct. However, typically people are sorted geographically along characteristies that are politically salient, and that variation can be leveraged to learn information about voting patterns based on those demographics. Classical ecological re- gressions aggregate demographies, but here have access to individual-level demographies via a commercial voter file with individual records, and therefore we construct our models at the individual level. There a number of advantages to using individual demographics for ecological inference, but note that while individual-level features are observed, individual-level labels still can not be observed, and therefore the fundamental challenges of non-identifiabiiity and aggregation paradoxes remain.
Related Work Classical ecological inference typically assumes an underlying individual linear model and constructs estimators for those model coefficients using aggregated demographics and labels King (1997). More recent work has used distribution regression for large-scale ecological inference incorporating Census microdata in nationwide elections in the US Flaxman et al. (2016). Aggregated labels represent substantial loss of information that could be used to constrai infer- ences, and all ecological methods rely on the analyst making assumptions which are not definitively empirically testable from those aggregates alone. Some research has been done on visual techniques for determining when some of these assumptions may have been violated Gelman et al. (200I).
Other work has sought to impose additional constraints the ecological problem by incorporating information from multiple elections Park et al. (2014), which also has the benefit of allowing for estimation of voter transitions, which themselves are of interest.

---

Under review as a conference paper at ICLR 2021
Our models are built from individual records of commercial voter file data, which leads to some im- mediate differences between classical methods. First, we do not need to determine the composition of the electorate from ecological model, since already know who from the total population voted from administrative records and we only need to use an ecological model for vote choice (who votes is public). Second, we directly specify model for vote choice the individual level and train this using a suitable ecological loss function, as opposed to specifying a model only on ag- gregates. This is convenient and allows the analyst great deal of flexibility modeling choices, and opens the possibility of using ecological inference for individual-level vote choice prediction as well as individual-level latent space modeling. However it does not automatically resolve the fundamental difficulties of ecological methods. We therefore introduce additional constraints on ecological problem by jointly training on multiple races a given election year in a manner analogous to existing methods, but adapted to our individual-level framework.
Contributions We develop loss function that allows us to approximate the poisson binomial and poisson multinomial loss that sits the center of the individually-orienn intractable to optimize directly. This approach allows to extend EI with deep learning, providing three key benefits: learning non-linear relationships in data, jointly learning multiple aggregated outcomes and learning low-dimensional nigh-information representations of individual voters. We apply these methods to data from the Maryland 2018 midterm election, to estimate vote propensity in the elections for the Governor, US Senator, and Attorney General. We validate these estimates using three datasets: first, post-election survey data on individuals' vote choices; second, data on the joint distribution of candidate support for these three races; and third we validate the learned representations by predicting individual responses to surveys using data linked to the representation.
approach to EI, but are too
2 AN APPROXIMATE ECOLOGICAL LOSs AND ARCHITECTURES
2.1 APPROXIMATING THE POISSON BINOMIAL AND MULTINOMIAL LOss
We model the choice of candidate (including abstaining from voting particular race) as real- ization from a categorical distribution, according an individually defined vector of probabilities p()), so that each individuals "vote" one-hot encoded vector, where the indicator represents the candidate the selected. If individuals are modeled as independent realizations of non-identically distributed categorical variables then precincts are distributed as poisson multinomial variables, pa- rameterized by matrix P where each row corresponds to individual's set of probabilities. The likelihood of the observed precinct-level counts then:
L(v1, uclP) U0.,,
log Σ Pe (f,c(9)EEA
II
Where Fc the set of all possible assignments (i, c()) of an individual to a vote choice c(i), subject to the constraint that the total number of individuals assigned to each candidate is u.,. so that |Fcl II ve!
- ve)! Even for the simplest possible case of single race with only two candidates,
this loss is not straightforward to compute Hong (2013). practice this loss is intractable to compute for the voting setting, where C > 3 (the United States has two party vote system, and people need not fill vote in every race at the ballot box) and N 20, 000. We propose to approximate this loss by assuming
2

---

Under review as a conference paper at ICLR 2021
L(v1, 02 02..,,
vc P) = log
Σ
II (i,c(i))E
)
P(u)
(2)
N
oii)
AEFo
log
p(n)
Σ
(3)
AEFo
C
N
[ Uc)! c= Do!
= log
II
p(n)
+ log
(4)
==1
Where (4) is the multinomial loss using the precinct-average of the predicted probabilities for each candidate. This approximation is correct for the first moment, but overestimates the variance com- pared to the variance of the poisson multinomial distribution. however extremely efficient to compute for a large number of precincts in parallel. For the poisson binomial we employ a similar strategy.
2.2
EXTENDING ECOLOGICAL INFERENCE
2.2.1 ARCHITECTURES
We consider two architectures, each of which were implemented using Bayesian and non- Bayessay neural networks. The first architecture is dense network, and the second is extension of varying-intercept varying-slopes model that has been used for modeling voting behavior Ghitza & Gelman (2013). A varying-intereepal varying-sloper model is typically written follows for binary model (extensions to multiple classes are straightforward)
logit(p:) an
axN
Xa
BES
(5)
Where we have K groups of random effects and L fixed effects. Including column of ones in X account for the intercept terms and absorbing the 3s into the as, we can rewrite this as linear model where the coefficients themselves are computed from another linear model as follows
L
logit(p.) = an
X
(6)
1=1
K
Here, the slopes for the fixed effects are the sum of additive contributions which depend on some categorical variables for unit i. Therefore, Bu can be interpreted features, which are learned in such a way that the when taken in a linear combination with the numerical features X, the results give good predictions tend to be sparse, and so the embedding reduces the dimensionality of those input features. This concept can be extended by predicting 8ą from a general embedding of matrix of categorical features X so that
embedding of the categorical
the probability scale. Categorical features
Ba = ((f)
Where in this study, f is given by fully connected neural network. The advantage of this architec- ture is that globally has very high capacity to complex data, but locally has linear structure. We call this architecture Deep multi-level model (MLM). We find that this leads to more reasonable estimates of partisan crossover voting, that is republican voters voting for Democratic candidates and vice versa, than the dense network, which tends to underestimate the degree of crossover rel- ative to what found in survey data. This architecture also provides advantages in interpretation because is fundamentally a linear model.
3

---

Under review as a conference paper at ICLR 2021
2.2.2 BAYESIAN LAYERS
For the Bayesian implementation for both architectures, we use the following multilevel formulation for each dense layer.
0 ~ Po = HalllalNomallll S)
(9)
w ~ Pw = Normal σ)
(10)
b pb Normal (0,
(11)
y activation +
(12)
for layer inputs x and outputs where S is a hyperparameter that controls the prior on the regular- ization strength. These multilevel layers help desensitize the model from our choice of hyperparam- eters, while allowing for different regularization strengths in each layer. We use fully factorized
mean-field variational approximation with the standard non-centered parameterization.
Bayesian neural nets often under perform as result of training difficulties that arise from increased variance in the stochastic gradients. To mitigate this, we train all Bayesian nets using KI. annealing. where the variational standard deviation parameters are set to small values and the KI. divergence term in the variational loss is initially zeroed out, and gradually increased over the course of training. This allows the model to find solution basin about as quickly non Bayesian net, essentially by training the variational mean parameters with the standard deviations fixed. Once inside basin we optimize the full objective, and this tends give good results.
3 APPLICATION TO THE MARYLAND 2018 MIDTERM ELECTIONS
We applied this method the statewide races in the Maryland 2018 midterm elections. Maryland typically votes for Democrats but had a Republican governor in 2018 who had high approval ratings going into the race and won reelection by little less than 300,000 votes. Maryland also had an attorney general and US senate race in that same election, where incumbent Democrats won by large margins. This gives useful test case for examining not correlations in voting patterns between races.
We demonstrate two practical use cases for out methods. The first is the classical case for ecological regression, where the analyst wants to learn demographic breakdowns of the vote share in the election. This involves fitting an ecological model and then cross tabulating the model scores by various quantities of interest. While it is impossible to determine the ground truth for this, we compare the cross tabulations from our ecological model estimates of support from a large sample survey which is weighted to be representative of Maryland voters. We additionally leverage a dataset we are not aware of having been used in this validation setting: the distribution of ballots is public information. This means if we predict multiple races can additionally check how well our recovers the joint distribution votes for candidates in different offices.
The second, which to our knowledge is new for ecological modeling, is to use ecological model to train a latent space for few shot learning model, which we then trained on survey data. We show that the features learned by the El model are more predictive survey data than the raw features, with significantly reduced dimensionality.
3.1
DATA
The data for these models come from two primary sources. The first Maryland board of elections, which publishes vote tallies at the voting precinct level. The second is a commercial voterfile, which contains administrative data such as addresses (which are used to determine voting precinct assign- ments), age, party registration, and whether or not each person cast ballot in the 2018 election. This file is also augmented with model scores provided by political data vendors, which attempt to estimate demographic characteristics such as race and education, well political characteristics such as strength of partisanship. We use both the administrative and modeled data as the features for the model, along with additional features consisting of aggregate characteristics of census tracts such as median income and percentage college educated. We use three other datasets for validat- ing the models. Data how each individual actually votes is obviously not available. We the
4

---

Under review as a conference paper at ICLR 2021
Cooperative Congressional Election Study get estimates of candidate support in the 2018 Mary- land election. This data has been shown to very accurately reflect the election results, making well suited for validation (Agadjanian & Robinson (2019). We also want to be able to examine the joint distribution over several races. Information on the joint distribution of ballots is actually public information, and political scientists have recently begun cleaning these data for in anal- ysis. We actually chose Maryland for this study because such data was available (Kuriwaki (2019); Agadjanian & Robinson (2019)) Finally, we want to be able to estimate the quality of learned latent representations. We use data from polling performed over the course of 6 months by the think tank Data for Progress, where we have 132 Maryland respondents linked to an individual in the voterfile mentioned above. We have a set of 24 questions across all these respondents pertaining to politi- cal questions, primarily about partisan affiliation, presidential approval, candidate preference the 2020 election, and what news sources respondents rely on.
3.2 RESULTS
We trained a number of model variants for this study. We trained standard and Bayes nets, dense and MLM architectures with depths ranging from 0 to 5 (specifically 0, 1, 3, 5). We trained all these models jointly and independently for all the races we consider, and with the binomial and multinomial losses. We trained each model variant 5 times and selected the model with the best hold out performance as the representative for that variant. Results presented here are for jointly- jointly-trained Bayes nets with binomial loss unless otherwise indicated. We also select the optimal depth on the test R² unless explicitly plotting over model depth.
First we compare the cross tabulated El model scores to a post-election survey from the Cooperative Congressional Election Study. This survey asked about the three races we examine here: Attorney General, Governor and US Senate. Generally we see performance improvements from the deep models, and more benefit from the MLM. We see benefits to Bayesian layers here as well, which clear from Table ! Individuals have a strong tendency to vote for their co-partisans, and so partisanship is highly predictive of voting. However survey data shows that partisan crossover vot- ing is quite common, and EI appears to have a tendency to underestimate the extent that partisans erossover. Crossover is essentially matter of capturing the interaction of smaller effects which af- fect crossover with that of a vary large effect, namely partisanship. We conjecture that the inductive biases of our architecture facilitate the estimation of such interactions.
A major benefit of the neural network framework is the ability to jointly train all the races in an elec- tion which share a common latent space. This allows for more accurate estimation of the covariance structure between votes in different races, which is of interest since like partisan crossover: these correlations can explain a great deal about an election. The benefits of joint training are very clear from Figure 3.2 Panel A shows example modeled covariances and the true covariance estimated from the joint ballot results. We see significant benefits to depth for learning the intra-race corre- lations, but only in the joint learning case. We also find that especially for the MLM joint training improves the demographic vote propensity. Interestingly joint training does not seem to improve demographics estimation for the dense net, which is bit counter-intuitive because it does improve covariance estimation. This means that at the level of granularity we can validate at the covariance and demographic support metries are somewhat orthogonal.
Training deep models also has the advantage of allowing us to use the learned latent space in future few shot learning applications. We apply these learned features in linear model on polling data from Maryland, and show the loo score for different architectures and depths. While our architecture had advantages for predicting partisan crossover, dense architecture is clearly preferred for the transfer learning application. This is likely because our architecture does not reduce the numerical features. Increased depth and use of Bayes nets has clear advantages for learning predictive features. We find that the Bayes nets tend to be sparser, which leads to better out of distribution predictions for our survey data.
CONCLUSION AND FUTURE WORK
We presented a framework for ecological inference for cases where individual records are avail- able but only aggregated labels are observed, which is the case for building models of election results from commercial voter file and voter precinet data. We propose binomial/multinomial
5

---

Under review as a conference paper at ICLR 2021
A
Modeled
County Levely-Level Senate
B
Attomey General Race Democrat Support
Govemoor Race Democrat Support
U.S. Senate Race Democrat Support
Age:
Party
Party Republican
Party independerts
Gender Male
Gender female
face Whde
face Back
Educatin College
Edscation:
-035
0.10
-.05
0.00
0.5
-025 -220 -0.5
-010 -005
610 605
610
-0.00 0.0
0:000 0:025
0.00
0925 0050 6375
6380
Dffexence Detween Modeled and Polling
Differncce Setween Modkle Polling
Oefeeene Between Modeled Polling
C
MSR Demograshiss
Party
0.2
805
VISOO
Baseline Dense
Deep Dense
Deep MLM
Figure 1: Panel A shows the modeled county-level results from the Deep MLM versus the true county-level results. Panel B shows the difference between modeled and true demographic support levels for three models: the baseline linear (dense) model, the deep dense model, and the deep MLM. Support is a (0- scale so that O. 15 corresponds to modeling 15 percentage points more Democratic support within demographic than is actually displayed. Panel C. is aggregated representation of Panel B. We show the MSE of support for all demographics and for the three party demographics (where much of the improvement coming from.

---

Under review as a conference paper at ICLR 2021
Baseline
Deep Independent
Deep joint
True
0.4
924
022
924
0.23
8:14
0:18
0.3
0.34
0.37
0.3
015
0.2
0.4
0.35
62
-022
0.2
0.13
0.18
025
0.33
-018
0.5
0.14
0.18
024
0.4
018
0.36
0.16
1036
0.22
0:22
023
0.12
0.12
0.12
012
AGG
SEN
ATG
MS2 Covariance Structure Indopendent vs Join Training
Nean Demographics
Independent joint
Support Over faces
Independent TWEro
Junt Tharing
Deep Daep
Figure 2: Here we demonstrate the benefits of joint training to learning the covariance structure over races. Panel A shows three modeled covariances: the baseline linear model using loss function, a deep model trained independently, and a deep model trained jointly. The deep models are the best performing dense nets. Panel B shows that the examples Panel A are representative. We plot the MSE of the modeled covariance against depth, where the solid lines indicate independent training and dotted lines joint training. We see that joint training leads significant gains in estimating the covariance. Panel C shows the mean MSE of modeled demographic support over all races. It shows that - especiall for the MI.M - joint training can improve demograhic-level support estimates.
Multinomial
Binomial R*2
Multinomall
Dencontt Grean indepetient
120
Model
MLM
0.58
053
Untrrianan Repuslican
Slandard Layors
Mann LO0 80
0.2
2.2
10
20
-10
10
Figure 3: The first two panels here show two dimensional PCA embeddings of the latent represen- tations of all Maryland voters from the best performing binomial and multinomial models, colored by the registered party. The binomial loss embeds voters along partisanship axis. The multinomial embedding is more three dimensional, reducing less to pure partisanship. The second two panels show the results of training linear models predict survey responses using the latent space, for latent spaces of various depths. performance of the latent space for predicting survey responses. The metric here is the leave-one-out R² over all 24 prediction tasks. The x-axis shows the number of hidden layers. The 0 hidden layer "latent space" is just the model input; the raw demographic variables. We see as depth increases so does the quality of the latent space for out of distribution prediction.

---

Under review as a conference paper at ICLR 2021
Model
Layer Type
Demo. MSE
Test R
Cov. MSE
LOO R
Baseline Dense
Standard
0.077576
0.988824
0.000858
0,498582
Baseline Dense
Bayes
0.073171
0,988441
0.001111
Deep Dense
Standard
0.056831
0.988227
0.000432
0.546267
Deep Dense
Bayes
0.054882
0,986031
0.000526
0.570651
Deep MLM
Standard
0.055656
0.984454
0.000447
0.517463
Deep MLM
Bayes
0.050220
0.985576
0.000579
0.526413
Table 1: Results summary table for joint models, comparing Dense and MLM architectures, Stan- dard and Dense layers. Demo. MSE is the mean squared error of the modeled support for the Democrat within a demographic (see Fig. 3.2) over races. Test R the R² of the predicted precinct level two-way vote share the modeled covariance structure. LOO R is the leave-one-out R linear models built using the latent space from the model. For baseline models the "latent space" simply the raw demographic variables.
held out test Cov. MSE is the squared error of
loss function with averaged rate parameters as an efficient approximation the poisson bino- mial/multinomial. This framework allows for straightforward application of a large family of mod- els, including deep neural networks. We also propose joint learning to mutually constrain predictions via parameter sharing. In particular, applied both Bayesian and non Bayesian deep networks ing fully connected architectures as well as our own Deep Multi-level Model architecture, which extends the carying-intercepts varying-slopes models which are frequently used for estimation of political opinions in small subgroups. We demonstrate that this method performs well in the clas- sical ecological inference use case, where the analyst interested in creating cross tabulations of voting patterns by demographic variables of interest, and show that joint training confers significant advantages for modeling the between-race correlations in voting behavior.
Our models are built on the individual level, allowing for individual-level label prediction as well as individual-level latent space modeling. Latent representations trained from ecological models were found to consistently add predictive value over the original feature space on prediction tasks from survey data. The latent space models particularly benefited from the use of deep and Bayesian networks, which resulted in sparser and more predictive representations.
Future work will focus on further developing transfer learning capabilities for building highly pre- dictive models from a combination of ecological and survey data. Since our loss function approxi- mation has increased variance relative to the true loss, tighter approximations based off sampling methods may increase the signal from data.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Alexander Agadjanian and Jonathan Robinson. Ground truth validation of survey estimates of split- ticket voting with cast vote records data. Available at SSRN 3443722, 2019.
Seth Flaxman, Dougal Sutherland, Yu-Xiang Wang, and Yee Whye Teh. Understanding the 2016 us presidential election using ecological inference and distribution regression with census microdata. arXiv:1611.03787, 2016.
Andrew Gelman, David Park, Stephen Ansolabehere, Phillip Price, and Lorraine Minnite. Models, assumptions and model checking in ecological regressions. J. R. Stafist. Soc. A, 164:101-118, 2001.
Yair Ghitza and Andrew Gelman. Deep interaction with mrp: Election turnout and voting patterns among small electoral subgroups. American Journal of Political Science, 57:762-776, 2013.
Yili Hong. On computing the distribution function for the poisson binomial distribution. Computa- tional Statistics and Data Analysis, 59:41-51, 2013.
Gary King. Reconstructing Individual Behavior from Aggregate Data: Solution the Ecological Inference Problem. Princeton University Press, Princeton, 1997.
Shiro Kuriwaki. Party loyalty on the long ballot: Is ticket splitting more prevalent in state and local elections? 2019.
Won-ho Park, Michael Hanmer, and Daniel Biggers. Ecological inference under unfavorable condi- tions: Strait and split-ticket voting in diverse settings and small samples. 36:192-203 2014.

---

Under review as a conference paper at ICLR 2021
A APPENDIX
If the election for U.S. Congress in your district was held today, which one of the following candidates are you most likely to vote for? Generally speaking, do you think of yourself Democrat, Republican, an Independent, something else?
Who would you prefer Joe Biden select his Vice President for the 2020 Presidential election? If the 2020 presidential election was held tomorrow and the candidates for president were Republican Donald Trump and Democrat Joe Biden, who would you vote for? Which of these statements comes closest to describing your feelings about the Bible?
Do you approve or disapprove the way Donald Trump is handling the economy?
If you don't it mind saying, who did you vote for in 2016?
Do you approve or disapprove of the way Donald Trump is handling the coronavirus pandemic?
If you don't t mind saying, who did you vote for in 2012?
How likely are you to vote the 2020 presidential election?
Are you someone else in your household member of labor union?
Generally speaking, would you say that most people can be trusted or that you can't be too careful in dealing with people?
Using the following scale, how would you describe yourself politically? How enthusiastic are you to vote in November 2020?
Please click on all of the sources of news that you have gotten news from about government and politics in the last week. If you're unsure, please DO NOT click it. None of these
Please click on all of the sources of news that you have gotten news from about government
and politics in the last week. If you're 'r unsure, please DO NOT click it. Twitter
Please click all of the sources of news that you have gotten news from about government
and politics in the last week. If you're re unsure, please DO NOT click it - MSNBC
Please click all of the sources of news that you have gotten news from about government
and politics in the last week. If you't re unsure, please DO NOT click it. Local television news Please click all of the sources of news that you have gotten news from about government and politics in the last week. If you're 're unsure, please DO NOT click it. - Fox News
Please click all of the sources of news that you have gotten news from about government and politics in the last week. If you're re unsure, please DO NOT click it. Broadcast news like ABC. CBS, NBC
Please click on all of the sources of news that you have gotten news from about government and politics in the last week. If you're unsure, please DO NOT click it. Facebook
Please click on all of the sources of news that you have gotten news from about government and politics in the last week. If you're 're unsure, please DO NOT click it. CNN Which of these arguments in support and opposition of Green New Deal do you find most persuasive?
Table Survey Questions used for Generalization Task
10


Reviews:
[{"review_id": "QTloYv5Y6l1", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper takes an approach to ecological inference inspired by deep learning. Ecological inference is the problem of learning individual labels when only large sets of aggregated data are available. It requires a way to estimate label propensities as a function of covariates. This paper proposes combining a multi-level model with deep learning to estimate voter propensities. The model is then applied to Maryland 2018 midterm election data, and is validated with demographic-level polling data (treating the polling data as ground truth) and with known vote correlations.\n\nSpecifically, the paper makes the assumption that distribution of vote counts for a precinct comes from a multinomial distribution, where global probability is averaged across all voters eligible to vote in the precinct (or to a binomial with the same probability averaging). The remaining problem is modeling these probabilities from the individual level covariates. The authors look at 3 models: a linear model, a feed-forward neural network, and a multi-level model where the varying slope coefficients are the output of a feed-forward neural network that takes as input an individual's covariates. The authors consider both frequentist and Bayesian versions of these models. In terms of experimental analysis, the paper argues: the deep models recover the closest demographic splits to the polling estimates, jointly training models across races recover vote covariance structure similar to the truth, and that representations from deep models are useful for few-shot prediction.\n\nI think it's important to connect fields like ecological inference and deep learning. This paper is a solid attempt to combine these disparate fields. The datasets it uses for its experiments are interesting and should be of value to the ML community. \n\nThe main problem with the paper is that the experiments are incomplete, unclear, and unconvincing of the significance of the proposed models. For one, the paper doesn't compare to a standard MLM as a baseline -- only the deep MLM, proposed in this paper. How do we know the improvements of a Deep MLM wouldn't be present in the standard method?\n\nBecause the main results of this paper are applied, strong analysis is crucial. A high-level issue is that the analysis doesn't go deep enough. There are claims like the following: the deep multi-level model \"leads to more reasonable estimates of partisan crossover voting, that is Republican voters voting for Democratic candidates and vice versa, than the dense network, which tends to underestimate the degree of crossover relative to what is found in survey data\". Why is this true? Can we interpret the results of the deep MLM that show why this behavior is happening? This is another example where training the regular MLM would be beneficial, so we could compare them and isolate the effects of deep learning. A similar claim is: \"Crossover is essentially a matter of capturing the interaction of smaller effects which affect crossover with that of a vary large effect, namely partisanship. We conjecture that the inductive biases of our architecture facilitate the estimation of such interactions.\" Can we quantitatively show what's happening instead of conjecturing?\n\nMoreover, the model that has the best test set R^2 is the standard linear regression. This is depicted in Table 1 but not discussed in the paper. It is not a make-or-break result, but the paper would benefit from explicitly answering the question: how can we reconcile the fact that the baseline has the best heldout performance but the proposed models perform best on the other tasks?\n\nThe paper also omits many key experimental details, which would hamper the reproducibility of results. How are train/test splits created? Are they on the precinct-level? What percent of the data/precinct are in each split? How many hidden units -- and which nonlinearities -- are used for the neural networks? What are the latent features used for the few-shot learning experiment? Is it just the beta's? The beta's concatenated with alphas? Or the entire logits? How do the experiments use the binomial loss for third-party candidates? Is it 1/0 Democrats/not Democrats? \n\nSome smaller comments about the graphs/tables:\n  * Where is the mean MSE for the dense baseline in Figure 2C?\n  * Why is there no LOO for the 0 hidden layer input for binomial R^2 in Figure 3, i.e. the model input? We only see it for the multinomial. \n  * Why is LOO R^2 for baseline dense missing in Table 1?\n  * I would remove Figure 1A to clear up space for analysis. Currently, the figure only appears to be showing the fit to the training data, which is not a very salient piece of information.\n\nOn a more minor note, there were quite a few typos throughout. Some examples:\n  * \"represents the candidate the selected\" on page 2 [should remove \"the\"]\n  * \"This gives us a useful test case for examining not correlations in voting patterns between races\" on page 4 [should remove \"not\"]\n  * \"We demonstrate two practical use cases for out method\" on page 4 [\"out\" should be \"our\"]\n  * \"various depths. performance of the latent space for predicting survey responses.\" on page 7 [second sentence is incomplete and not capitalized]\n  * in general, \"R\" in \"Republican\" should be capitalized\n\nIn summary, this is an interesting idea and an important research area, but for now it is preliminary work due to incomplete experiments/analysis.\n\nPros:\n- Important research area (combining political science with machine learning)\n- Valuable datasets introduced to community\n- Model proposed is intuitive\n\nCons:\n- Experiments missing key analysis and baselines\n- Experiments aren't reproducible\n- Clarity could be improved, typos throughout.", "extra_scores": {}, "extra_text": {}}, {"review_id": "LAfj7eL7Fc7", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper uses deep neural networks into the ecological inference problem and shows its effectiveness by using large-scale datasets of the Maryland 2018 midterm elections. Some experimental results have been shown.\n\nPros.\n1. Ecological inference is an important problem in political science for modeling individual-level voting behavior given only aggregate-level data.\n2. The authors attempt to apply the recently advanced technique (I.e., DNN) to the ecological inference problem.\n3. This paper provides a case study that analyzes real-world voting behavior by using various kinds of datasets.\n\nCons:\n1. The proposed approach is not new.\n2. Related work is not adequately cited.\n3. This paper is not well-written, especially the Experiment section is not readable.\n\nReasons for score: \nI think the task that the authors address is interesting and important. However, this paper does not present new technical contributions and related works are not listed adequately. The authors attempt to provide the case study of the Maryland 2018 midterm elections via deep learning approaches in ecological settings. But, the Experiment section is not well-organized, and I did not find insightful results from this manuscript. Accordingly, my opinion is that this paper is not ready for publication. \n\nDetailed comments:\n1. Important references are missing. So, It is not clear how far the proposed method has pushed the boundary of existing technology. Firstly, as the authors mentioned, the ecological inference is strongly related to the problem “learning with label proportions (LLP)”, the authors should add more discussion.  For example, in [R1], the ecological inference problem has been formulated as LLP. Also, ecological inference relates to other problems discussed in the machine learning community. The common aim is to learn the individual-level models from only aggregate-level data; for example, distribution regression [R2], multiple-instance learning [R3, R4], and collective graphical models [R5, R6]. It would be a great idea to survey and discuss the relationships with these methods.\n2. In the approximation procedure for the loss function, the assumption is not clear. Is that “each individual adopts one of the candidates based on the shared probabilities that is the average over $N$ individuals”? If that is the case, the loss is the logarithm of multinomial distribution (i.e., Eq. (4)); this is equivalent to that has been used in the Collective Graphical Models (e.g., [R6]). \n3. Section 3.2 is not readable. Does each paragraph in Section 3.2 correspond to which Figure? What is Figure 3.2? \n4. In experiments, the authors should specify the experimental settings. For example, what are few-shot settings? Also, the authors should clarify some hyper-parameters such as optimizer selection and learning rate.\n\n\n     [R1] Tao Sun, Dan Sheldon, Brendan O’Connor, “A probabilistic approach for learning with label proportions applied to the US presidential election”, in ICDM, pp. 445-454, 2017.\n\n     [R2] S. R. Flaxman, Y.-X. Wang, and A. J. Smola, “Who supported Obama in 2012?: Ecological inference through distribution regression,” in Proceedings of the 21st ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 2015, pp. 289–298.\n\n     [R3] H. Hajimirsadeghi and G. Mori, “Multi-instance classification by max-margin training of cardinality-based Markov networks,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016.\n\n     [R4] H. C. L. Law, D. Sejdinovic, E. Cameron, T. C. D. Lucas, S. Flaxman, K. Battle, and K. Fukumizu. Variational learning on aggregate outputs with Gaussian processes. In NeurIPS, pages 6084–6094, 2018.\n\n     [R5] D. R. Sheldon and T. G. Dietterich, “Collective graphical models,” in Advances in Neural Information Processing Systems, 2011, pp. 1161– 1169.\n\n     [R6] D. Sheldon, T. Sun, A. Kumar, and T. G. Dietterich, “Approximate inference in collective graphical models,” in International Conference on Machine Learning (ICML), vol. 28, no. 3, 2013, pp. 1004–1012.\n\nMinor comments:\n- Modify the mathematical symbols to the italic font;  for example, unit $i$ and features $X$ in the sentences of Section 2.2.1.\n- Letters in the figures are small and hard to see.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Clm8HLOAIB2", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a deep learning framework for approximating ecological inference for estimating voting propensities based on demographic aggregates. This is an important problem, as EI has become a court standard for evaluating racially polarized voting in gerrymandering cases for the Gingles factors. Additionally, the increased attention on building coalition districts and availability of individual level data means that this is a problem that is likely to have a large impact in the next redistricting cycle that begins next year. \n\nThe proposed methodologies seem natural once the approximation is constructed and this analysis explores some potential ways to incorporate it into various learning architectures. Additional work could be devoted to optimizing over the choices of hyperparameters and providing additional guidance about ways to choose which model would be appropriate based on available input data, since not all applications of these methods will have access to the full sets of surveys and validation measures that were available here. It would be nice to see the performance of these methods on some synthetic data as well and at least one comparison to one of the current state of the art methods on an aggregate version of the data would be useful. \n\nOverall, this paper is interesting and presents an approximation that is likely to be useful in practice for real world problems and given the space constraints appears to present sufficient work to be publishable. \n\n\nA couple of typos: \nLast sentence of paragraph 1 in Section 3 `not correlations' seems like a misnomer\n\nEnd of caption 1, missing close paren. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "26AICUAMRCQ", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### Summary\n\nThe paper discusses an interesting direction to efficiently approximate the loss function in the ecological inference problem, which enables extensions using linear models, deep neural networks, and Bayesian neural networks. The proposed approach was evaluated using Maryland 2018 midterm elections data on a range of tasks.\n\n#### Strengths\n- The paper tackles one important and practical problem of ecological inference: inferring labels from label proportions, which is applicable to a lot of settings, one of which is \"voting\" as studied in the paper\n- The paper discusses an interesting direction to approximate the loss function of the ecological inference problem in an efficient manner, which enables different extensions, especially using Bayesian neural networks.\n- The paper evaluates the proposed model using real Maryland 2018 midterm election data and produces interesting insights\n\n#### Weaknesses\n- The paper is not easy to follow. Apart from various typos (see details below), I think the structure of the paper could be improved significantly to make it more accessible. For example:\n  - (1) Poisson binomial/multinomial losses were not introduced early in Section 1, which makes it hard to follow and understand the \"Contributions\" described at the end of Section 1\n  - (2) Although described in text in Section 1, it's still pretty unclear what the input data are. I'd suggest discussing the input data formally at the beginning of Section 2 before describing the techniques in details\n  - (3) It's very unclear what the evaluation tasks are (especially for people who are not familiar with the data and/or domain) and the intuitions behind why the tasks are suitable to evaluate the effectiveness of the proposed methods\n\n- The paper lacks details on how the proposed methods (and baselines) are implemented. In addition, there are various baselines/methods included in the \"results\" section but it's unclear what they are in details.\n\n- In addition, there are various typos / minor writing problems. Here are some of them:\n  - Sec 3: \"This gives us a useful test case for examining not correlations in voting patterns between races.\" \"not correlations\"?\n  - Sec 3: \"We demonstrate two practical use cases for out methods. \" -> \"We demonstrate two practical use cases for our methods. \"\n  - References in many places are without parentheses \n  - poisson -> Poisson\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper is very interesting and timely, but as the reviewers note there is significant room for improvement in the clarity of the presentation and evaluation. In addition to the references mentioned by the reviewers, some other relevant references are the following: 

[1] Evan Rosenman, Nitin Viswanathan, "Using Poisson Binomial GLMs to Reveal Voter Preferences," https://arxiv.org/abs/1802.01053

[2] Law, H. C. L., Sutherland, D., Sejdinovic, D., & Flaxman, S. (2018, March). "Bayesian approaches to distribution regression." In International Conference on Artificial Intelligence and Statistics (pp. 1167-1176).

Decision: Reject