Deep Ecological Inference

1 INTRODUCTION . Ecological inference ( EI ) , or learning labels from label proportions , is the problem of trying to make predictions about individual units from observations about aggregates . The canonical case is voting . We can not observe individual people ’ s votes , but people live in precincts , and we know for each precinct what the final vote count was . The problem is to try to estimate probabilities that a particular type of individual voted for a candidate . Since we can not observe individual labels , but only sums of pre-specified groups of labels , nonidentifiability is inherent to the ecological inference problem . The possibility of interaction effects between any relevant demographics and the aggregation groups themselves also means that Simpson ’ s paradox type confounding is an ever present risk . The most basic approach to this problem involves assuming total heterogeneity at the precinct level , and simply assigning the final distribution of votes in a precinct to each person living in that precinct . However , typically people are sorted geographically along characteristics that are politically salient , and that variation can be leveraged to learn information about voting patterns based on those demographics . Classical ecological regressions use aggregate demographics , but here we have access to individual-level demographics via a commercial voter file with individual records , and therefore we construct our models at the individual level . There are a number of advantages to using individual demographics for ecological inference , but note that while individual-level features are observed , individual-level labels still can not be observed , and therefore the fundamental challenges of non-identifiability and aggregation paradoxes remain . Related Work Classical ecological inference typically assumes an underlying individual linear model and constructs estimators for those model coefficients using aggregated demographics and labels King ( 1997 ) . More recent work has used distribution regression for large-scale ecological inference incorporating Census microdata in nationwide elections in the US Flaxman et al . ( 2016 ) . Aggregated labels represent a substantial loss of information that could be used to constrain inferences , and all ecological methods rely on the analyst making assumptions which are not definitively empirically testable from those aggregates alone . Some research has been done on visual techniques for determining when some of these assumptions may have been violated Gelman et al . ( 2001 ) . Other work has sought to impose additional constraints on the ecological problem by incorporating information from multiple elections Park et al . ( 2014 ) , which also has the benefit of allowing for estimation of voter transitions , which themselves are of interest . Our models are built from individual records of commercial voter file data , which leads to some immediate differences between classical methods . First , we do not need to determine the composition of the electorate from an ecological model , since we already know who from the total population voted from administrative records and we only need to use an ecological model for vote choice ( who votes is public ) . Second , we can directly specify a model for vote choice at the individual level and train this using a suitable ecological loss function , as opposed to specifying a model only on aggregates . This is convenient and allows the analyst a great deal of flexibility in modeling choices , and opens the possibility of using ecological inference for individual-level vote choice prediction as well as individual-level latent space modeling . However it does not automatically resolve the fundamental difficulties of ecological methods . We therefore introduce additional constraints on the ecological problem by jointly training on multiple races in a given election year in a manner analogous to existing methods , but adapted to our individual-level framework . Contributions We develop a loss function that allows us to approximate the poisson binomial and poisson multinomial loss that sits at the center of the individually-oriented approach to EI , but are too intractable to optimize directly . This approach allows us to extend EI with deep learning , providing three key benefits : learning non-linear relationships in data , jointly learning multiple aggregated outcomes and learning low-dimensional high-information representations of individual voters . We apply these methods to data from the Maryland 2018 midterm election , to estimate vote propensity in the elections for the Governor , US Senator , and Attorney General . We validate these estimates using three datasets : first , post-election survey data on individuals ’ vote choices ; second , data on the joint distribution of candidate support for these three races ; and third we validate the learned representations by predicting individual responses to surveys using data linked to the representation . 2 AN APPROXIMATE ECOLOGICAL LOSS AND ARCHITECTURES . 2.1 APPROXIMATING THE POISSON BINOMIAL AND MULTINOMIAL LOSS . We model the choice of candidate ( including abstaining from voting on a particular race ) as a realization from a categorical distribution , according to an individually defined vector of probabilities p ( i ) , so that each individuals “ vote ” is a one-hot encoded vector , where the indicator represents the candidate the selected . If individuals are modeled as independent realizations of non-identically distributed categorical variables then precincts are distributed as poisson multinomial variables , parameterized by a matrix P where each row corresponds to an individual ’ s set of probabilities . The likelihood of the observed precinct-level counts is then : L ( v1 , v2 , . . . , vC |P ) = log ∑ A∈FC ∏ ( i , c ( i ) ) ∈A P ( i ) c ( i ) ( 1 ) Where FC is the set of all possible assignments ( i , c ( i ) ) of an individual i to a vote choice c ( i ) , subject to the constraint that the total number of individuals assigned to each candidate c is vc , so that |FC | = ( ∑C c=1 vc ) ! ∏C c=1 vc ! . Even for the simplest possible case of a single race with only two candidates , this loss is not straightforward to compute Hong ( 2013 ) . In practice this loss is intractable to compute for the voting setting , where C > 3 ( the United States has a two party vote system , and people need not fill vote in every race at the ballot box ) and N > 20 , 000 . We propose to approximate this loss by assuming as L ( v1 , v2 , . . . , vC |P ) = log ( ∑ A∈FC ∏ ( i , c ( i ) ) ∈A ( 1 N N∑ n=1 P ( n ) c ( i ) ) ) ( 2 ) = log ( ( C∏ c=1 ( 1 N N∑ n=1 P ( n ) c ) vc ) ∑ A∈FC 1 ) ( 3 ) = log ( C∏ c=1 ( 1 N N∑ n=1 P ( n ) c ) vc ) + log ( ( ∑C c=1 vc ) ! ∏C c=1 vc ! ) ( 4 ) Where ( 4 ) is the multinomial loss using the precinct-average of the predicted probabilities for each candidate . This approximation is correct for the first moment , but overestimates the variance compared to the variance of the poisson multinomial distribution . It is however extremely efficient to compute for a large number of precincts in parallel . For the poisson binomial we employ a similar strategy . 2.2 EXTENDING ECOLOGICAL INFERENCE . 2.2.1 ARCHITECTURES . We consider two architectures , each of which were implemented using Bayesian and non-Bayesian neural networks . The first architecture is a dense network , and the second is an extension of a varying-intercepts varying-slopes model that has been used for modeling voting behavior in Ghitza & Gelman ( 2013 ) . A varying-intercepts varying-slopes model is typically written as follows for a binary model ( extensions to multiple classes are straightforward ) logit ( pi ) = α0 + K∑ k=1 αk [ i ] + L∑ l=1 Xil K∑ k=1 βk [ i ] l ( 5 ) Where we have K groups of random effects and L fixed effects . Including a column of ones in X to account for the intercept terms and absorbing the βs into the αs , we can rewrite this as a linear model where the coefficients themselves are computed from another linear model as follows logit ( pi ) = α0 + L∑ l=1 Xilγil ( 6 ) γil = K∑ k=1 αk [ i ] l ( 7 ) Here , the slopes for the fixed effects are the sum of additive contributions which depend on some categorical variables for unit i . Therefore , βil can be interpreted as an embedding of the categorical features , which are learned in such a way that the when taken in a linear combination with the numerical features X , the results give good predictions on the probability scale . Categorical features tend to be sparse , and so the embedding reduces the dimensionality of those input features . This concept can be extended by predicting βil from a general embedding of a matrix of categorical features Xc so that βil = f ( X c i ) ( 8 ) Where in this study , f is given by a fully connected neural network . The advantage of this architecture is that globally it has very high capacity to fit complex data , but locally it has linear structure . We call this architecture a Deep multi-level model ( MLM ) . We find that this leads to more reasonable estimates of partisan crossover voting , that is republican voters voting for Democratic candidates and vice versa , than the dense network , which tends to underestimate the degree of crossover relative to what is found in survey data . This architecture also provides advantages in interpretation because it is fundamentally a linear model . 2.2.2 BAYESIAN LAYERS . For the Bayesian implementation for both architectures , we use the following multilevel formulation for each dense layer . σ ∼ pσ = HalfNormal ( 0 , S ) ( 9 ) w ∼ pw = Normal ( 0 , σ ) ( 10 ) b ∼ pb = Normal ( 0 , σ ) ( 11 ) y = activation ( wx+ b ) ( 12 ) for layer inputs x and outputs y , where S is a hyperparameter that controls the prior on the regularization strength . These multilevel layers help desensitize the model from our choice of hyperparameters , while allowing for different regularization strengths in each layer . We use a fully factorized mean-field variational approximation with the standard non-centered parameterization . Bayesian neural nets often under perform as a result of training difficulties that arise from increased variance in the stochastic gradients . To mitigate this , we train all Bayesian nets using KL annealing , where the variational standard deviation parameters are set to small values and the KL divergence term in the variational loss is initially zeroed out , and gradually increased over the course of training . This allows the model to find a solution basin about as quickly as a non Bayesian net , essentially by training the variational mean parameters with the standard deviations fixed . Once inside a basin we optimize the full objective , and this tends to give good results . 3 APPLICATION TO THE MARYLAND 2018 MIDTERM ELECTIONS . We applied this method to the statewide races in the Maryland 2018 midterm elections . Maryland typically votes for Democrats but had a Republican governor in 2018 who had high approval ratings going into the race and won reelection by a little less than 300,000 votes . Maryland also had an attorney general and US senate race in that same election , where incumbent Democrats won by large margins . This gives us a useful test case for examining not correlations in voting patterns between races . We demonstrate two practical use cases for out methods . The first is the classical use case for ecological regression , where the analyst wants to learn demographic breakdowns of the vote share in the election . This involves fitting an ecological model and then cross tabulating the model scores by various quantities of interest . While it is impossible to determine the ground truth for this , we compare the cross tabulations from our ecological model to estimates of support from a large sample survey which is weighted to be representative of Maryland voters . We additionally leverage a dataset we are not aware of having been used in this validation setting : the distribution of ballots is public information . This means if we predict multiple races we can additionally check how well our recovers the joint distribution of votes for candidates in different offices . The second , which to our knowledge is new for ecological modeling , is to use an ecological model to train a latent space for a few shot learning model , which we then trained on survey data . We show that the features learned by the EI model are more predictive on survey data than the raw features , with significantly reduced dimensionality .

Summary: This paper takes an approach to ecological inference inspired by deep learning. Ecological inference is the problem of learning individual labels when only large sets of aggregated data are available. It requires a way to estimate label propensities as a function of covariates. This paper proposes combining a multi-level model with deep learning to estimate voter propensities. The model is then applied to Maryland 2018 midterm election data, and is validated with demographic-level polling data (treating the polling data as ground truth) and with known vote correlations.