SlowGuess/ACL_2020_OCR huggingface dataset:
Maryam Davoodi
mdavoodi@purdue.edu
Eric Waltenburg
ewaltenb@purdue.edu
Dan Goldwasser
dgoldwas@purdue.edu
While national politics often receive the spotlight, the overwhelming majority of legislation proposed, discussed, and enacted is done at the state level. Despite this fact, there is little awareness of the dynamics that lead to adopting these policies. In this paper, we take the first step towards a better understanding of these processes and the underlying dynamics that shape them, using data-driven methods. We build a new large-scale dataset, from multiple data sources, connecting state bills and legislator information, geographical information about their districts, and donations and donors' information. We suggest a novel task, predicting the legislative body's vote breakdown for a given bill, according to different criteria of interest, such as gender, rural-urban and ideological splits. Finally, we suggest a shared relational embedding model, representing the interactions between the text of the bill and the legislative context in which it is presented. Our experiments show that providing this context helps improve the prediction over strong text-based models.
Despite the fact that state-level legislation is rarely discussed, it has a dramatic influence on the everyday life of residents of the respective states. The policies enacted at the state-level touch on all aspects, from mundane topics, such as trash removal and state mascots, to highly ideologically-charged topics such as education, religious liberties, and health-care access. Moreover, state-legislatures discuss and vote-on significantly more bills than their Federal counterparts, adding up to over 120,000 bills per year (King, 2019). Also, the lack of general interest, as well as the complexity of the processes that differ across states, often leads to public disengagement from local politics. This results in decisions being made with little understanding of
a) Failed
b) Competitive
c) Inverse-Competitive
Figure 1: Example of failure and party cleavages.
the processes that shape them and how they are likely to influence different demographics.
Similarly, most effort directed at understanding political processes using data was directed at the Federal level. In the NLP community, several works looked at analyzing political texts (Iyyer et al., 2014) and the resulting behaviors of legislators (Gerrish and Blei, 2011, 2012). The only exception is recent work (Eidelman et al., 2018), predicting whether a bill would pass the preliminary stage, legislative committee, to a full-body vote.
State-level demographic cleavages: Our goal in this paper is to take a first step towards understanding the processes and interests that underlie how decisions are passed using data-driven methods. Our main intuition is that the impact of bills on different demographics will be reflected in the behavior and voting patterns of their representatives. Thus, providing the ability to automatically identify bills, before they are put to a vote, that will have a positive or negative influence on a specific demographic can help inform public responses and increase engagement with local political processes.
To help achieve this goal, we define two novel text classification tasks, characterizing the breakdown of votes, based on different cleavages or demographic indicators such as gender, geography (i.e., rural vs. urban districts), party membership and ideological splits. With respect to each one of these splits, we define two aggregate-level proper
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5358-5368
ties of a vote, competitive and inverse-competitive cleavages. Both of these measures capture the lack of consensus in the legislature body around a specific bill, but in different ways. We say that a bill is competitive in a vote (Fig. 1b) if the majority of legislators from a logical group (e.g., democrats, women, urban districts, liberals) vote differently from the majority of legislators from the opposite group (e.g., republican, men, rural districts, conservatives). A bill is inverse-competitive (Fig. 1c) if there is a partial or complete tie within the legislators from the same group (e.g., women). To help explain these concepts, consider a bill restricting access to abortion clinics. This bill is likely to results in a competitive vote, based on ideology. On the other hand, a bill granting tax breaks for farmers might result in an inverse-competitive vote, based on ideology. In that case, a competitive vote, based on geography is more likely.
In Table 1, we provide examples of the different splits associated with real bills that were brought to a vote. Unsurprisingly, a "benign" bill, such as #1 is widely accepted and does not result in any contention. A contentious bill, such as #2, touching on the way religion is taught is split ideologically (i.e., the vote is almost unanimous inside each ideological group), but mixed based on economic and gender splits. Bill #4 addressing nepotism issues and regulating public contracts is contentious across all splits. Alerting the public when such bills are brought to a vote can help ensure that legislators take into account the opinions and voiced raised in their constituencies.
Technical Contributions Although a text classification scheme is a reasonable starting point to determine demographic cleavages of bills only based on their content, it is not sufficient. Our key insight in this paper is that the context or relations through which specific information is propagated among different players in the legislative process (e.g., money donors and legislators), can be leveraged to further improve the performance. Thus, we build a shared relational architecture that models the text of a bill and its context into a graph; Our model captures the behavior of individual legislators, language of bills, and influence of contributions on the decision to identify demographic cleavages. While there are different ways to realize our relational model, we chose to build on recent advances in the NLP space, Relational Graph Convolutional Network (RGCN) (Schlichtkrull et al., 2018) and pre
trained BERT transformers (Devlin et al., 2018). RGCN allows us to define multiple relations between each pair of entities (e.g., a legislator sponsorship and casting a vote on a bill) and BERT enables us to represent the textual information more efficiently. With the help of the attention-based architecture, BERT has been shown to outperform LSTM models. To operationalize our relational settings, we collected information from different sources and introduced a new dataset combining information about legislators, bills, donations, and donors as well as demographic information about the legislators and their districts. In our experiments, we analyze the implication of different relations on the performance and show that our shared architecture outperforms existing text and graph models.
Table 1: Competitive and inverse-competitive bills.
#	Bill Title	Gen.	Geo.	Ideo.	Party
1	A CONCURRENT RESOLU-TION congratulating the Pio-neer Junior-Senior High School football team on winning the Indiana High School Athletic As-sociation	None	None	None	None
2	Teaching of the origin of life	Inver.	Inver.	Comp.	Comp
3	Beer dealer permits	None	None	None	None
4	Officeholder qualifications,nepotism, and public contracts	Both	Inver.	Both	Both
Bill analysis at the state level has received little attention and our work, while conducting a new in-depth modeling and analysis, is inspired by the following works:
Classification of congress roll calls. (Eidelman et al., 2018) combines the text of the bill with partisan identity of the bill's sponsor(s) in a model predicting the likelihood of a member of the U.S. Congress voting in support of a non-unanimous congress bill or resolution. They find that the models that combine text with sponsorship data significantly outperform several alternative models. Similarly, (Gerrish and Blei, 2011) uses topics associated with congress bills to infer its location in ideological space and then uses ideal point models to predict the likelihood of a U.S. Senator or House member voting in support of a bill. They find that their model increases predictive accuracy by about  $4\%$  over a naïve baseline model. (Patil et al., 2019; Kraft et al., 2016; Karimi et al., 2019; Kornilova et al., 2018; Peng et al., 2016) extend this congress model to learn embeddings for legislators and congress bills using other sources of data
(e.g., Twitter, knowledge graphs). More recently, (Budhwar et al., 2018) evaluates different models for predicting roll-call votes based on verbal statements that legislators make during questioning.
Predicting progress of bills Rather than using bill text in models to explain the roll-call behavior of individual legislators, (Yano et al., 2012) include the legislation's text in a model that predicts whether a bill emerges from a standing committee, a point in the legislative process that most bills do not pass. In particular, they use features based on the urgency and importance of the issue being addressed by the bill as well as a set of features extracted from co-sponsors of the bill. Examining the fate of bills between the 103rd and 111th congresses, they find that including features of the bill drawn from the text improves the model's predictive accuracy over their baseline model. (Eidelman et al., 2018) repeat a similar analysis for the states and they show "that combining contextual information about the legislators and the legislatures with bill text consistently provides the best predictions". (Nay, 2017) examines the text of congressional to identify the text structure most associated with a congress bill's enactment and then embeds it using Word2Vec for the classification based on Random Forests; Nay concludes that the full text of a congress bill enables better prediction efficiency.
Demographic bill cleavages. Demographic bill cleavages is a well-studied topic in the political science space. Research has properly differentiated between the multiple ways demographic background of legislators can influence roll-call voting. (Pinney and Serra, 1999) finds that Congressional Black Caucus members vote more consistently with the caucus than they do with fellow partisans or with representatives from their state. (Jenkins, 2012) discusses gender moderates the effect of party and ideology in roll-call voting. Similarly, (Frederick, 2010) discusses gender influences the roll-call vote in the Senate by moderating the effect of partisanship for GOP women. (Broach, 1972) demonstrates that urban-rural cleavages structure vote in less partisan states and on bills that clearly divide urban and rural interests.
NLP applications of GCN. Recently, GCNs have been explored in different NLP tasks. Semantic role labeling (SRL) (Marcheggiani and Titov, 2017), relation classification in clinical narratives (Li et al., 2018), and machine translations (Bastings et al., 2017) are a few instances.
In such tasks, GCN is used to encode syntactic structure of sentences. In a similar context, some works explored the idea of graph neural networks (GNNs) (Peng et al., 2018; Henaff et al., 2015; Defferrard et al., 2016), where each part of a document (e.g., sentences) is collapsed into a graph of words or the citation relations (Kipf and Welling, 2016) creates a network among different documents.
We model the legislative process as a graph that consists of bills, legislators, and money donors in all states. Building a global graph captures contextual information and relationships that interconnect different states. For instance, money donation by a contributor to two legislators from different states could indicate they have a similar roll call behaviors on abortion bills. Given this intuition, after a brief overview of the legislative process in US states, we describe how we collapse it into a graph structure.
Figure 2: Bill-to-Law stages
3.1 Primer on State-Level legislative Process
Although there are some specific differences across state legislatures, a common process, shown in Figure 2, prevails. This process starts with one or more legislators (Representatives or Senators) who sponsor and file a bill. The idea of a bill could be original or come from a constituent, public official, or an interest group. Each state consists of two "chambers": the House of Representatives ("House") and the Senate. To become law, the bill goes through a reviewing process in the origin chamber, where it can "die" at different stages. If the bill gets a pass vote, it is sent to the other chamber and the same process repeats. Finally, the bill is reviewed by the state Governor for signature. In parallel to these efforts, external contributors, e.g., money donors and lobbyists, play an important yet indirect role in the process. By sourcing information and money into the process, they leave an impact on legislators, which can change the progression of a bill.
Within a chamber the process is as follows: if the leadership in the chamber chooses, the bill gets its
First Reading by title. Then, the chamber president may refer the bill to a committee for review. If the committee casts a vote on the bill, it can be defeated or advance to Second Reading by the full body of legislators. Next, the chamber leadership may decide to approve the bill for Third Reading, where it again comes to a vote by the full body of legislators and a majority vote can advance the bill.
Figure 3: Collapsing the legislative process into a heterogeneous multi-relational legislative graph.
3.2 Legislative Process in a Heterogeneous Multi-Relational Graph
A close look reveals that the legislative process cannot be captured in a simple graph as there can be multiple relations between a pair of nodes (e.g., sponsorship and vote between legislators and bills), and the graph consists of several nodes types with different attributes and labels (e.g., bills with competitive labels). Thus, we model the process using a heterogeneous multi-relational graph, as follows:
Node attributes: The nodes in our proposed legislative graph come with a rich set of features and information: (1) Bill nodes contain title, description, and full text of the house and senate state bills. (2) Legislator nodes contain diverse textual information abstracting the behavior of a legislator such as his biography, political interests, committee assignments, and demographic attributes (gender, party, and ideology and the district information). (3) Contributors nodes come with different information (in the textual format) on money donors such as their specific and general business interests, party, and their type (individual vs non-individual).
Relations: Based on the legislative process, we identify that legislator and bill nodes participate in three main relations: sponsorship  $(R1)$ , negative ("Nay") vote  $(R2)$ , and positive ("Yea") vote  $(R3)$ . Similarly, we establish two types of relations between contributors and legislators: positive donation edges  $(R4)$ , which are realized based on the real data, and negative or lack of donation edges  $(R5)$ , inferred when a contributor shows lack of in
terest in specific legislators (e.g., always donates to Democrats). In this case, we randomly sample such legislators and link them to the contributor. Based on our data analysis, more than  $62\%$  of unique contributors always contribute to one party in our dataset. We also conducted an ablation study, not included due to space constraints, and the donor information contributed between 2 to 11 F1 points.
3.3 Bill Inference Problems
For a bill and one of its roll calls in the legislative graph, we seek to predict if (1) it evinces identifiable voting cleavages or (2) it can advance by getting a pass. For voting cleavages, we defined four demographic attributes (gender, party, ideology, and the urban/rural nature of the district) to divide legislators into groups. We assign nine labels to each bill as follows: (1) Competitive labels: For an attribute (e.g., party), a voting round of a bill is defined as "competitive" if the majority of legislators from one group (e.g., Democrats) votes differently from the majority of the other group (e.g., Republicans). For example, in Figure 1b,  $70\%$  of Democrats vote Yea and  $80\%$  Republicans vote Nay on a roll call, then the bill is competitive and the disagreement between the groups is  $10\%$  ( $=80\%-70\%$ ). (2) Inverse-competitive labels: Similarly, for an attribute (e.g., party), we call a voting round as inverse-competitive if there is a partial or full cleavage among the legislators of the same group. For instance, consider a bill with  $55\%$  of Democrats voting Yea and  $45\%$  of them voting Nay (Figure 1c). In this case, the bill turns out to be inverse-competitive and the disagreement is  $45\%$  (the percentage of minority votes). (3) Survival label: Depending on the progress, a bill passes a certain voting round if it gets a majority vote (e.g., in 2nd/3rd Reading) or if two-thirds of legislators agree to it (e.g., in amendments).
4 Inference on Legislative Graph
We argue for a joint graph and text embedding model to represent the nodes and their textual attributes in the legislative graph, which is used for the roll-call prediction and aggregation. Embedding models that only leverage textual information ignore important relations in the legislative graph. Graph-based models make textual information less distinguishable at the classification stage, where it matters. At a high level, our approach combines the complementary strengths of both approaches.
Figure 4: Joint text-graph architecture for predicting relations in the legislative graph and aggregating vote (roll-call) of individual legislators, by leveraging text-attributed RGCN and BERT's pretrained embeddings.
Our architecture (Figure 4a) uses BERT's pretrained embedding to represent the textual information of nodes in the graph; and text-attributed RGCN to generate an embedding for them based on their relations. Finally, we combine them to build a representation of edges in the graph for our relation prediction and then aggregate vote relations.
4.1 Text Representation Layer
The lower half of our architecture is based on BERT, which leverages transformers and acts as an efficient replacement for sequential models. In our case, we use the BERT's pretrained embedding to form an initial representation for the textual information of the nodes in the legislative graph.
Bill representation: We represent a bill by averaging three different vectors (Figure 4b) corresponding to: (1) title, (2) description, and (3) body of the bill. For each of these components, we compute the average word vector based on BERT's pretrained word embedding. Thus, the bill representation is  $X_{bill} = Avg(e_{title} + e_{description} + e_{body})$ .
Legislator representation: To represent a legislator, we compute BERT's pretrained embedding for his textual information: (1) attributes, (2) biography, and (3) committee information. Finally, we take the average of these vectors,  $X_{\text{legislator}} = \text{Avg}(e_{\text{attributes}} + e_{\text{biography}} + e_{\text{cmte-info}})$ , as illustrated in Figure 4c.
Contributor representation: Similarly, We transform different pieces of textual information on a contributor, i.e., party- and type-related attributes, business information, and industry data, into separate vectors,  $e_{\text{attributes}}$ ,  $e_{\text{business}}$ ,  $e_{\text{industry}}$  and then take their average as the final representation,  $X_{\text{contributor}}$  (Figure 4d).
4.2 Relational Graph Convolutional Layers
We feed the text representation of the bill, legislator, and contributor nodes, as their initial representation, into Relational Graph Convolutional Network (RGCNs) to better represent them given the legislative graph structure. In parallel, a feed-forward neural network (FFNN) processes these text representations and takes them to a concatenation layer for the joint text-graph optimization. From the message passing perspective, each (non-relational) GCN layer performs two operations: propagation and aggregation. In the propagation phase, the neighborhood nodes send their feature/hidden representation to the node that needs to be updated. In the aggregation phase, the node sums up all the messages coming from its neighborhood with its properties. The aggregated message is passed through a non-linear activation function which forms the new representation of the node. If the graph edges are not typed, the hidden representation of each node  $i$ , at  $(l + 1)$ 'th layer, is computed by:
$$
h _ {i} ^ {l + 1} = \sigma \left(\sum_ {j \in N _ {i}} \frac {1}{c _ {i}} W ^ {l} h _ {j} ^ {l}\right) \tag {1}
$$
In which the weight matrix  $W^l$  is shared by all edges in layer  $l$ . Also,  $c_i$  is a normalization factor, which is often set to  $c_i = |N_i|$ . Relational GCN (RGCN) generalizes GCNs to handle different relations between any pair of nodes, and thus being a better fit for our problem. Unlike GCNs, RGCNs use a different weight matrix and normalization factors (e.g.,  $c_i^r = |N_i^r|$ ) for each relation type and thus the hidden representation of nodes in  $(l + 1)'$ th
Table 2: Statistics of the legislative graphs, aggregated over the 2011-2018 period.
State	Nodes	Relations
# Cont	# Bills	# Leg.	#Cont-Leg	#Leg-Bill
IN	274	4818	226	17729	217026
OR	462	4884	150	29213	102463
WI	175	1320	208	5924	88004
All	911	11022	584	52866	407493
layer is computed as:
$$
h _ {i} ^ {l + 1} = \sigma \left(W _ {0} ^ {l} h _ {i} ^ {l} + \sum_ {r \in R} \sum_ {j \in N _ {i} ^ {r}} \frac {1}{c _ {i , r}} W _ {r} ^ {l} h _ {j} ^ {l}\right) (2)
$$
By having a  $K$ -layer RGCN (stacking layers onto each other), we can capture  $k^{th}$ -order relations from a node in the graph. However, a 2-layer RGCN turns out to be sufficient in our case as it fully realizes the 2nd order relations between contributors and bills.
4.3 Roll-Call Classification and Aggregation
By combining the outputs of the RGCN and FFNN, we train a model for predicting relations in the legislative graph through FFNN+softmax. One could leverage DistMult scoring functions (Schlichtkrull et al., 2018; Yang et al., 2014) as well. Next, we post-process the roll-call relations and aggregate them to form the demographic and pass/fail vote breakdowns and determine the final class labels. In more detail, the representation of an edge or relation  $(s,d)$  is the dot product of  $e_s^{joint}$  and  $e_d^{joint}$ , which are the embedding of the corresponding nodes. The representation of a node comes from the concatenation of two components: (1) text embedding (hidden states) coming from the BERT layer after being fine-tuned through FFNN, and (2) the graph embedding (hidden state of the node) from the last RGCN layer.
Loss function: At a high level, our loss function is  $L = L_{Cls} + L_{Text} + L_{Graph}$  and jointly optimizes the text and graph embeddings as well as the relation prediction and roll-call aggregation.  $L_{Cls}$  is the cross-entropy loss of the relation prediction;  $L_{Graph}$  and  $L_{Text}$  are the L2 regularizations of RGCN's and FFNN's weights that generate the graph and text representations, respectively.
In this section, we describe our comprehensive legislative dataset, combining different sources of data (e.g., money donors data, diverse information on
Table 3: Legislators' attributes across the target states aggregated over the 2011-2018 period—UR: Urban, RU: Rural, C: Conservative, M: Moderate, L: Liberal.
State	Gender	Party	Geography	Ideology
F	M	D	R	UR	RU	C	M	L
IN	50	176	67	159	161	64	125	94	7
OR	47	103	83	67	133	17	28	61	61
WI	51	157	84	124	160	48	78	49	81
All	148	436	234	350	454	129	231	204	149
legislators). Table 2 shows the statistics of our dataset after pruning and forming the legislative graph (discussed in Section 3). Next, we focus on our joint embedding model and its great ability in outperforming existing prediction models.
5.1 Data Collection Methodology & Statistics