SlowGuess/ACL_2020_OCR huggingface dataset:
Bills and legislator data. From the LegiScan website (LegiScan, 2019), we collected data on the text and lower chamber disposition of all bills introduced in Indiana, Oregon, and Wisconsin from the 2011 through 2018 sessions. To do so, we developed a comprehensive crawler in Python that performs multiple operations. First, it uses the LegiScan API to collect legislative information on every bill that covers: (1) bill metadata that includes the bill type, title, description, sponsors, and links to its texts; (2) vote metadata that consists of the individual legislator's vote – “Yea,” “Nay,” “Absent,” or “NV”; and (c) legislator metadata containing party and district information. Then, our crawler accurately converts bill texts that are stored in the PDF format to text files, using open-source libraries. To identify the fine-grained progression of bills in the legislative process, our crawler downloads and processes the “History” section of each bill on the LegiScan Website, which consists of a series of events associated with a bill’s history (e.g., committee report, roll-call vote). Such information is not readily available in the LegiScan API. Overall, we collected 34443 bills introduced in the target states from 2011 to 2018. We studied  $58\%$  of the bills that had both the votes of individual legislators and full texts, which are necessary for determining vote breakdowns and cleavage labels; However, our focus in this paper is on the 2nd/3rd reading, in which all members of the chambers vote, so we selected  $32\%$  of the bills that reached this stage to build the legislative graph (Table 2).
Biography, ideology and geography data. Finally, our crawler uses Ballotpedia (Ballotpedia, 2019) to collect texts on each legislator's biography, political interests, and committee assignments.
Also, it aggregates other publicly available datasets to identify each legislator's attributes such as ideology, gender, and district nature (urban/rural). The ideology scores for legislators were taken (Shor and McCarty, 2011) and they were grouped into conservatives, moderates, and liberals. The district identifier was combined with GIS census data (Census, 2019) to categorize each legislator as representing either an urban or rural district. Table 3 shows the breakdown of legislators' party, gender, ideology, and district information in our target states. For less than  $10\%$  of legislators, Ballotpedia profiles were missing. Thus, we used other public textual information about them (e.g., Twitter).
Donors data: FollowTheMoney (FollowTheMoney, 2019) captures and keeps tracks of donations to legislators and candidates in the US states. Our crawler consumes the FollowTheMoney API to collect the information of donors for each legislator and cosponsors of our bills. This includes multiple textual attributes and information for each contributor: type that could be individual or non-individual, general party, and economic and business information. While the contributor data can be used in more sophisticated ways, in this work, we focused on major contributors by setting a donation threshold (\(10000) and removing those who contributed to a single legislator; We also separated between ideological contributors and pragmatic ones (donating to both parties) by inferring "negative" (lack of) donation relations (see Section 3); We set the fraction of negative donations to \(30\%\) of the positive ones extracted from the real data. Table 2 shows the final per-state statistics of contributors.
We build different graph and textual models on top of PyTorch, DGL (Deep Graph Library), and spaCy. In our joint text-graph model (Figure 4) and other baselines, the initial embedding dimension of both BERT ("bert-large-uncased") and the first-layer RGCN are set to 1024. The FFNN (fully connected layer) and the second-layer RGCN take the initial text and graph embeddings to a 256-dimensional space. We have also experimented with different settings, which while resulting in lower overall performance, retained the same trend when comparing the other models. We used Adam to optimize our model and for each observed relation (Table 2), we sampled a negative example.
Data splits. Our focus is on the bill cleavage and
survival and thus we split legislative graphs based on bill nodes. To evaluate different scenarios, we have three configurations: (1) random where we select  $20\%$  of the bills for testing and keep the rest for training and validation. (2) time-based where  $20\%$  of most recent bills are considered for testing; and (3) state-based where the test bills come from one specific state and train bills from the other states. The test bills and corresponding legislators appear in the test graph, and the difference of the original and test graphs is used for training. Note that vote relations of sponsoring legislators and a bill are known, and appear in training.
Metric. Given the highly skewed data when predicting bill survival and cleavages, we pick Macro F1 as the main metric over accuracy.
5.2.1 Baselines
To demonstrate the benefits of our joint text-graph embedding, we implement a series of text and graph embedding architectures as the baseline.
Category 1: text embedding models: We realize our bill encoder (Figure 4b) using three text embedding models and then train a logistic regression classifier to directly predict if a bill text shows a certain cleavage or passes/fails:  $(a)BoW$ , where unigram and bigram features (top 10K highest scoring ones using scikit-learn (Pedregosa et al., 2011)) used to represent bill texts.  $(b)GloVe$  (Pennington et al., 2014) that is a popular word embedding model using the square loss; We used the GloVe-840B-300D pre-trained word vectors in our experiments. (c) BERT (Devlin et al., 2018) that is a transformer based architecture capable of capturing contextualized embedding.
Category 2: featureless graph embedding models: We build a edge classifier over edge embeddings generated by models that assume nodes in the legislative graph are homogeneous and featureless, and then aggregate the roll call results: (a) DeepWalk (Perozzi et al., 2014) is an embedding model that generates node vectors by running Skip-Gram on random walks formed at different nodes in the graph. (b) GCN (Kipf and Welling, 2016) is the basic two-layer GCN model that uses a single weight matrix in each layer and begins with the random node features in the first layer. (c) RGCN (Schlichtkrull et al., 2018) is the relational version of the GCN that captures different relations in our legislative graph.
Category 3: text-attributed (TA) graph embedding models: We use the same edge classifier
Table 4: Macro-F1 in bill survival and cleavage prediction for the random split and known sponsors' relations.
Embedding	Pass/Fail	Competitive	Inverse-Competitive
Party	Gender	Ideology	Geography	Party	Gender	Ideology	Geography
Naive	Majority	0.47	0.44	0.46	0.44	0.46	0.48	0.47	0.45	0.47
Sponsor	0.51	0.43	0.43	0.41	0.43	0.44	0.45	0.41	0.45
Text-based	BoW	0.63	0.64	0.64	0.65	0.60	0.58	0.60	0.57	0.62
GloVe	0.65	0.67	0.66	0.67	0.61	0.57	0.62	0.60	0.63
BERT	0.68	0.70	0.72	0.69	0.66	0.58	0.64	0.62	0.67
Featureless Graph	DeepWalk	0.49	0.52	0.50	0.54	0.56	0.52	0.50	0.52	0.51
GCN	0.49	0.53	0.51	0.55	0.57	0.52	0.51	0.53	0.52
RGCN	0.57	0.57	0.53	0.55	0.59	0.54	0.52	0.55	0.56
Text Attributed Graph	TA-DeepWalk	0.66	0.67	0.64	0.68	0.60	0.53	0.62	0.55	0.71
TA-GCN	0.67	0.67	0.65	0.66	0.61	0.52	0.62	0.54	0.72
TA-RGCN	0.72	0.69	0.65	0.71	0.63	0.56	0.64	0.57	0.72
Joint Graph+Text	0.82	0.83	0.79	0.82	0.73	0.64	0.78	0.65	0.78
but use the graph models that can consume the text-based node features generated by our BERT-based node encoders: (a) TA-DeepWalk (Yang et al., 2015) that changes the graph factorization in DeepWalk to support node features. (b) TA-GCN (Kipf and Welling, 2016) is the original GCN that takes as input an initial node features. (c) TA-RGCN (Schlichtkrull et al., 2018) is a relational GCN that captures node features initialized by our text-based node encoders.
Category 4: naive baselines. We evaluate two other naive classifiers: (a) Majority: A baseline predicting the most frequent class in the training data; (b) Sponsor: A logistic regression classifier that directly predicts bill survival and cleavages based on the one-hot encoded sponsors' info. encoded.
Performance of different textual and graph models. Table 4 shows macro F1 for different bill cleavages and pass/fail. We first analyze the performance of different models in each category: (1) Among the naive models, the sponsor-based classifier improves the bill survival prediction compared to the majority model but has no positive impact on bill cleavages as expected intuitively. (2) In the textual models, we observe BERT improves the F1 performance by  $2\% -8\%$  compared to GloVe and BoW. By leveraging a bidirectional operation, BERT more efficiently captures the context of each word in the bill title, summary, and body. (3) In the featureless graph models, RGCN consistently outperforms the standard GCN and DeepWalk models as it treats each of the relations in the legislative graph (e.g., donation and voting) differently and does not mix their weight matrices with each other. This benefit of RGCN is entirely enabled by our new dataset that explicitly tracks
different legislative relations; (4) Unlike the second category, the text attributed graph models capture implicit relations between different nodes in the graph through their text features. By leveraging our node encoders, they begin with better initial representations of the nodes and relations (e.g., particularly votes) and thus provide an improvement by up to  $15\%$  in the performance compared to their featureless counterparts. (5) Finally, our proposed model by combining and jointly optimizing the graph and textual representations consistently provides a higher F1 score. Compared to the other models, it improves recall while maintaining high precision, e.g., in the case of the bill survival prediction, the macro precision and recall values for BERT, TA-RGCN, and our model are (0.72, 0.67), (0.92, 0.66), (0.82, 0.84), respectively.
Language and implications of different cleavages. We can make a few observations: it is slightly more challenging to identify inverse-competitive bills compared to competitive ones. This happens across different graph and text models, and thus indicating the language of these bills and the dynamics of relations behind them is rather complex. To help provide an intuition, we summarized in Table 6 the top bigrams and unigrams used in competitive and inverse-competitive bills across the different cleavages. Interestingly, the top n-grams of competitive bills align better with the cleavages (e.g., "abortion" is competitive both based on ideology and gender) compared to the top inverse-competitive n-grams, which often focus on mundane issues such as taxes and services, suggesting that when non-polarizing legislation is discussed, group agreement takes a secondary role.
From another angle, Figure 5 further illustrates the differences between these two categories of cleavages. Overall, there are  $10\% -20\%$  more com
Table 5: Macro F1 for bill survival and party cleavages for the best model in each category based on the state- and time-based data splits.
Embedding	State-based (Test: IN)	State-based (Test: OR)	Time-based (Test: 20%)
Pass/fail	Comp.	Inverse. Comp	Pass/fail	Comp.	Inverse Comp	Pass/fail	Comp.	Inverse Comp.
Naive (Majority)	0.47	0.44	0.45	0.46	0.45	0.44	0.48	0.45	0.46
Text-based (BERT)	0.63	0.64	0.53	0.61	0.64	0.54	0.67	0.67	0.57
Featureless Graph(RGCN)	0.52	0.52	0.50	0.51	0.50	0.51	0.54	0.54	0.52
Text-Attributed Graph (TA-RGCN)	0.60	0.62	0.53	0.62	0.61	0.52	0.67	0.68	0.55
Joint Graph+Text	0.70	0.72	0.58	0.70	0.70	0.58	0.73	0.76	0.61
Table 6: Most frequent unigrams and bigrams of competitive and inverse-competitive bills.
Type	Unigram/Bigram
Comp.	Party	law, fund, abortion, political subdivision, providing penalty, badger-care plus, parental choice
Gender	abortion, child, medical, school, providing penalty, motor vehicle, minimum wage, parental choice
Ideology	income, abortion, insurance, drugs, local government, retirement system, natural resources, political subdivision
Geography	county, service, commission, district, transportation, housing, residential, state financial, criminal history, restroom facility, greenhouse gas
Inv-comp.	Party	state, program, motor vehicle, real estate, study committee, education matters,
Gender	financial, emergency, permits, legislative council, economic development, criminal penalty
Ideology	tax, services, county, criminal, alcoholic beverages, board education, commission declaring
Geography	law, school corporation, property tax, unemployment insurance
Figure 5: Distribution of competitive and inverse-competitive bills before split over 2011-2018.
petitive bills compared to inverse-competitive ones under the party and ideology attributes, indicating cross-group disagreements (e.g., conservatives VS. moderates VS. liberals) are more likely than intra-group disagreement. This pattern is reversed for the gender and geography attributes.
Implication of state- and time-based data splits. For the pass/fail and party cleavages with the best model in each category, Table 5 shows a sharp drop in the F1 score for the state-based and time-based data split, particularly for graph-based models (RGCN and TA-RGCN). By training the model with the two states and testing it with an
other one, the graph-based embedding models are challenged with representing many unseen legislators. While GCN-based solutions are capable of creating such representations in the test time (using the same weight matrix), they are sub-optimal particularly in featureless GCN settings. One interesting observation is that when the model is tested with the OR data, the drop is even sharper as OR tends to be a democratic state; While WI and IN are often republican states. For the time-based data split, we observe a similar but slightly better performance as the number of unseen nodes are fewer. In all these different configurations, our joint model still improves the F1 score but it is limited on how the underlying graph model behaves.
In this paper, we take the first step towards understanding the dynamics of state-level legislative processes in the US through a data-driven approach. We proposed to collapse the legislative process into a heterogeneous multi-relational graph and suggest several tasks for capturing disagreement over several ideological and demographic cleavages, as well as predicting the outcome of the legislative process. We approach these problems by formulating them as aggregate roll-call prediction.
To fully realize the potential of graph-based modeling, we created a new dataset, used to characterize the real-world context in which the legislative process takes place, consisting of bills, donors, and legislators and their behavior. We model the rich relationship between these entities and the content of the bills using a joint text and graph prediction model on top of BERT and RGCN, outperforming each one of the models in isolation.
Ballotpedia. 2019. State-level political encyclopedia data. https://ballotpedia.org/.
Joost Bastings, Ivan Titov, Wilker Aziz, Diego Marcheggiani, and Khalil Sima'an. 2017. Graph convolutional encoders for syntax-aware neural machine translation. arXiv preprint arXiv:1704.04675.
Glen T Broach. 1972. A comparative dimensional analysis of partisan and urban-rural voting in state legislatures. The Journal of Politics, 34(3):905-921.
Aditya Budhwar, Toshihiro Kuboi, Alex Dekhtyar, and Foaad Khosmood. 2018. Predicting the vote using legislative speech. In Proceedings of the 19th Annual International Conference on Digital Government Research: Governance in the Data Age.
GIS Census. 2019. GIS census data. https://www.nhgis.org/.
Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. 2016. Convolutional neural networks on graphs with fast localized spectral filtering. In Advances in neural information processing systems, pages 3844-3852.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Vlad Eidelman, Anastassia Kornilova, and Daniel Ar-gyle. 2018. How predictable is your state? leveraging lexical and contextual information for predicting legislative floor action at the state level. arXiv preprint arXiv:1806.05284.
FollowTheMoney. 2019. State-level contributor data. https://www.followthemoney.org/.
Brian Frederick. 2010. Gender and patterns of roll call voting in the us senate. In Congress & the Presidency, volume 37, pages 103-124. Taylor & Francis.
Sean Gerrish and David M Blei. 2011. Predicting legislative roll calls from text. In Proceedings of the 28th international conference on machine learning (icml-11), pages 489-496.
Sean Gerrish and David M Blei. 2012. How they vote: Issue-adjusted models of legislative behavior. In Advances in Neural Information Processing Systems, pages 2753-2761.
Mikael Henaff, Joan Bruna, and Yann LeCun. 2015. Deep convolutional networks on graph-structured data. arXiv preprint arXiv:1506.05163.
Mohit Iyyer, Peter Enns, Jordan Boyd-Graber, and Philip Resnik. 2014. Political ideology detection using recursive neural networks. In Association for Computational Linguistics.
Shannon Jenkins. 2012. How gender influences roll call voting. Social Science Quarterly, 93(2):415-433.
Hamid Karimi, Tyler Derr, Aaron Brookhouse, and Jiliang Tang. 2019. Multi-factor congressional vote prediction. Advances in Social Networks Analysis and Mining (ASONAM).
Kevin King. 2019. State Legislatures Vs. Congress: Which Is More Productive? http://bit.ly/30YsKwT. [Online; accessed 19-July-2019].
Thomas N Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907.
Anastassia Kornilova, Daniel Argyle, and Vladimir Eidelman. 2018. Party matters: Enhancing legislative embeddings with author attributes for vote prediction. In Proceedings of ACL.
Peter Kraft, Hirsh Jain, and Alexander M Rush. 2016. An embedding model for predicting roll-call votes. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing.
LegiScan. 2019. State-level legislative data. https://legiscan.com/.
Yifu Li, Ran Jin, and Yuan Luo. 2018. Classifying relations in clinical narratives using segment graph convolutional and recurrent neural networks (SegGCRNs). Journal of the American Medical Informatics Association, 26(3):262-268.
Diego Marcheggiani and Ivan Titov. 2017. Encoding sentences with graph convolutional networks for semantic role labeling. arXiv preprint arXiv:1703.04826.
John J Nay. 2017. Predicting and understanding lawmaking with word vectors and an ensemble model. PLoS ONE, 12(5):e0176999.
Pallavi Patil, Kriti Myer, Ronak Zala, Arpit Singh, Sheshera Mysore, Andrew McCallum, Adrian Benton, and Amanda Stent. 2019. Roll call vote prediction with knowledge augmented models. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 574-581.
Fabian Pedregosa, Gaël Varouquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. 2011. Scikit-learn: Machine learning in python. the Journal of machine Learning research, 12:2825-2830.
Hao Peng, Jianxin Li, Yu He, Yaopeng Liu, Mengjiao Bao, Lihong Wang, Yangqiu Song, and Qiang Yang. 2018. Large-scale hierarchical text classification with recursively regularized deep graph-cnn. In Proceedings of the 2018 World Wide Web Conference, pages 1063-1072. International World Wide Web Conferences Steering Committee.
Tai-Quan Peng, Mengchen Liu, Yingcai Wu, and Shixia Liu. 2016. Follower-followee network, communication networks, and vote agreement of the us members of congress. Communication Research, 43(7):996-1024.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVE: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543.
Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. 2014. Deepwalk: Online learning of social representations. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining.
Neil Pinney and George Serra. 1999. The congressional black caucus and vote cohesion: Placing the caucus within house voting patterns. Political Research Quarterly, 52(3):583-607.
Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. 2018. Modeling relational data with graph convolutional networks. In European Semantic Web Conference, pages 593-607. Springer.
Boris Shor and Nolan McCarty. 2011. The ideological mapping of american legislatures. American Political Science Review.
Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2014. Embedding entities and relations for learning and inference in knowledge bases. arXiv preprint arXiv:1412.6575.
Cheng Yang, Zhiyuan Liu, Deli Zhao, Maosong Sun, and Edward Chang. 2015. Network representation learning with rich text information. In Twenty-Fourth International Joint Conference on Artificial Intelligence.
Tae Yano, Noah A Smith, and John D Wilkerson. 2012. Textual predictors of bill survival in congressional committees. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 793-802. Association for Computational Linguistics.
5368