UPPAM: A Unified Pre-training Architecture for Political Actor Modeling based on Language
Year: 2023

Abstract:
Modeling political actors is at the core of quantitative political science. Existing works have incorporated contextual information to better learn the representation of political actors for specific tasks through graph models. However, they are limited to the structure and objective of training settings and can not be generalized to all politicians and other tasks. In this paper, we propose a Unified Pre-training Architecture for Political Actor Modeling based on language (UPPAM). In UPPAM, we aggregate statements to represent political actors and learn the mapping from languages to representation, instead of learning the representation of particular persons. We further design structureaware contrastive learning and behavior-driven contrastive learning tasks, to inject multidimensional information in the political context into the mapping. In this framework, we can profile political actors from different aspects and solve various downstream tasks. Experimental results demonstrate the effectiveness and capability of generalization of our method.

heading: 1 Introduction
text: Political actors are shaping our attitudes, opinions, and decisions toward public issues. For instance, on social platforms, politicians can select and emphasize certain aspects of content to bias the discussion, through which they can derive an opinion climate from user engagement and acquire direct feedback from potential voters and opinion leaders (Bene, 2017; Heiss et al., 2019). Political actor modeling is essential for quantitative political science and has applications in various downstream tasks such as roll call vote prediction (Yang et al., 2020), frame detection (Johnson et al., 2017) and bias detection (Baly et al., 2020).
Data-driven approaches utilize different kinds of information to profile political actors, including public statements, legislative behaviors and social
∗Corresponding author.
networks (Figure 1). Early research analyzes roll call data to estimate the ideology of political actors. Ideal point model (Clinton et al., 2004) is one of the most widely used approaches for votebased analysis that reveals how cleavages between legislators reflect partisan affiliation. Researchers further incorporate texts of bills to enhance the ideal point model (Gerrish and Blei, 2011, 2012; Kraft et al., 2016) and develop multidimensional vectors to replace one-dimension points. Recently, more abundant information has been considered to learn effective representations for political actors, such as co-sponsorship network (Yang et al., 2020), relations of contributors (Davoodi et al., 2020), stakeholders (Davoodi et al., 2022), mention in documents (Pujari and Goldwasser, 2021), and expert knowledge (Feng et al., 2021, 2022).
Generally speaking, previous research aims to learn representations for a certain group of political actors using supervision from specific downstream tasks as objectives. Although they report positive results on target tasks, their models lack generalization ability in two aspects. (1) Representations
11996
are learned on labeled data from specific tasks, e.g., state-level vote prediction, therefore they cannot be easily transferred to other tasks or scenarios. (2) The model is limited to the training setting and can not be adapted to dynamic social contexts. In other words, it’s hard for the model to estimate new legislators, non-voting candidates and other political actors unseen.
Recently, large-scale pre-trained language models (PLMs) (Devlin et al., 2019; Liu et al., 2019b; Brown et al., 2020) have demonstrated a strong generalization ability and achieved excellent performance in many language modeling tasks. Motivated by PLMs, we explore representing political actors based on their statements and propose a Unified Pre-training Architecture for Political Actor Modeling based on language (UPPAM) 1. We employ a two-stage training procedure following the fashion of PLMs. Firstly, we pre-train our model to learn the mapping from statements to actor representation. We propose a multigranular method to represent political actors based on language, and information of political scenarios is further injected into our model via proposed structure-aware contrastive learning and behaviordriven contrastive learning tasks. Secondly, we fine-tune the model for downstream tasks using the corresponding supervised objectives.
UPPAM is novel in three points. (1) We learn the mapping from statements to the representation of political actors, instead of directly learning actor representations. By doing so, the mapping parameters can be transferred to any downstream tasks easily, learning representations for unseen political actors based on their statements. (2) We propose several self-training tasks to inject general knowledge in the political scenarios into mapping parameters in the pre-training stage. (3) We propose a multigranular actor representation model, that can capture nuances of both general ideology and specific preferences between different political actors. We evaluate our approach on three types of tasks in quantitative political science, i.e., profile of actors, prediction of behaviors and analysis of languages. UPPAM outperforms general PLMs and other political domain-specific PLMs on these tasks. Our task-agnostic model also achieved competitive results compared to the task-specific models that employ architectures crafted for the
1We have made our code publicly available at https:// github.com/xymou/UPPAM.
vote prediction task. Further analysis shows the effectiveness and robustness of UPPAM in few-shot settings and different aggregation settings.

heading: 2 Political Actors Modeling based on Language

heading: 2.1 Multigranular Actor Representation
text: Political actors manifest themselves in political activities in multiple granularities. On the one hand, they hold a general ideology or bias, which is long-term and stable. On the other hand, when discussing or taking action on different issues, they hold specific positions (Gerrish and Blei, 2012), which are the result of long-term bias and shorttime interests (Spell et al., 2020). Based on this, we propose to represent political actors in two granularities to model both broad ideology and specific preferences for various downstream scenarios.
General and Specific Statements Collection In practice, we use all statements a political actor has posted to get his general representation, characterizing the broad political leaning. Furthermore, issue-related content is adopted to help capture specific attitudes. Concretely, we use a handcrafted information retriever (see more details in Appendix A.2), to collect statements related to the queried policy area as input to encode the specific representation.
Statements Aggregator Since a political actor can post thousands of statements, the first challenge is how to aggregate one’s statements to get his representation. It is too expensive in time and computation cost to combine full sentences. Instead, we identify indicator words from statements for information aggregation. According to the framing theory (Entman, 1993), entities and subjective content an author uses can implicitly reflect his political leaning. Following this, we identify entities, frame and sentiment words as indicators. We sort them by TFIDF (Jones, 1972) scores and keep indicators with the highest values to form an indicator sequence. In this case, for each political actor, we can get two kinds of indicator sequences, given a query about policy area j:
Sgi = w g 1, w g 2, ...w g N (1)
S pj i = w pj 1 , w pj 2 , ...w pj M (2)
where Sgi is calculated from all the statements made by political actor i, Spji is calculated from content
related to policy area j, and we reserve top N and M indicators with highest TFIDF value, where N and M are pre-defined hyper-parameters.
In subsequent pre-training and downstream tasks, we use general sequences as input when the goal is to profile the characters broadly, e.g., estimating ideology. And we input both sequences and average the representation when specific attitudes are required in tasks, as shown in Figure 2. Note that even if the issue-related content can not be retrieved, we can use the general sequence as a substitute, to ensure input compatibility.

heading: 2.2 Multidimensional Pre-training for Political Actor Modeling
text: To inject general knowledge of the political landscape into the mapping from statements to representation, we construct self-supervised tasks based on structural and behavioral information.

heading: 2.2.1 Structure-aware Contrastive Learning (SCL)
text: In terms of structural information, we mainly focus on the relationship formed between political actors. Previous studies have revealed that homophily exists in political communities, where people with similar ideologies form a link with each other (Barberá, 2015). We use two parts of links, namely party affiliation and co-sponsorship in voting. We
treat party affiliation as a coarse relationship and cosponsorship as a fine relationship respectively. By doing this, the model can further capture nuances across parties as well as inside the same party.
Party Affiliation Link We compare statements of legislators from different parties. We choose a legislator as the anchor, and then take another legislator with the same party affiliation as the positive sample, while those from the opposite party are regarded as negative samples. By comparing general statement sequences of legislators from different parties, the model can learn the differences in the languages of different ideologies.
Co-sponsorship Link In the legislative process, a bill is initialized by a sponsor and several cosponsors. We assume that the more two legislators collaborate, the more they are alike since they reach agreements on many occasions (Yang et al., 2020; Mou et al., 2021). Given an anchor legislator, other legislators are divided into three categories based on the number of times they co-sponsored with the anchor legislator: G1 (the co-sponsorship times are above the average); G2 (the co-sponsorship times are below the average); G3 (they have never cosponsored). And we further sample positive and negative samples with the rule of G1 < G2 < G3.
Based on the triplets constructed in the above
two ways, the structure-aware contrastive objective is formulated as follows:
LSCL = ∑
t∈TSCL
[∥∥∥t(a) − t(p) ∥∥∥ 2 − ∥∥∥t(a) − t(n) ∥∥∥ 2 + δSCL ] +
(3)
where TSCL is the set of legislator triplets, t(a), t(p) and t(n) are actor representation encoded by general sequences of anchor, positive and negative sample in triplet t, δSCL is a hyperparameter and [·]+ is max(·, 0).
Notably, this task endows the model to capture general ideology of speakers from their languages.

heading: 2.2.2 Behavior-driven Contrastive Learning (BCL)
text: When it comes to behavioral information, we pay attention to the most common and important actions, i.e., voting. Specifically, we sample triplets consisting of an anchor bill and a pair of legislators, where the positive legislator p votes yea on the given bill and the negative one n votes nay. Different from the ideology cleavages modeled in Sec 2.2.1, the divergence of specific preferences is supposed to be reflected in the languages here. Thus, for each legislator, we extract statements about the policy area of the anchor bill as the specific sequence, input with the general sequence, as we mentioned in Sec 2.1. In this way, the behaviordriven contrastive objective is as follows:
LBCL = ∑
t∈TBCL
[∥∥∥t(a) − t(p) ∥∥∥ 2 − ∥∥∥t(a) − t(n) ∥∥∥ 2 + δBCL ] +
(4)
where TBCL contains all vote triplets, and δBCL is a hyperparameter. t(a) is the bill representation, t(p) and t(n) are the average of representation of the general sequence and the specific sequence, for the positive and negative legislators respectively.
It’s noticeable that this pattern is not limited to the roll-call vote scenarios, instead, it can be applied to model the preferences towards any bills, events, or targets with a text description.

heading: 3 Pre-training Process

heading: 3.1 Language Model Co-training
text: As mentioned in Sec 2.2.2, modeling political actors in political scenarios inevitably requires encoding textual information of the bills and issues they interact with, e.g., Equation 4. Meanwhile, it is important to understand their opinions in a
single discourse without context. Thus, we incorporate additional modules to model political texts. Specifically, as shown in Figure 2, we have two FFN layers in parallel in each transformer layer, to handle text and actor sequences separately. Given a sequence of input x = {x1, ..., xn}, the model first performs multi-head self-attention and then the corresponding module FNNk obtains the required representation:
hk = FNNk ( Self-Attention ({x1, . . . , xn})) (5) where k ∈ {0, 1} indicates the modules of actor and text respectively.
We adopt a masked language model objective to pre-train the language model. As mentioned before, political bias and framing effect are often reflected in the selection and mention of specific entities, subjective content, and emphasized frames. Thus, we take a masking strategy that upsamples entity tokens, sentiment words (Wilson et al., 2005) and frame indicators (Roy and Goldwasser, 2020) to be masked for the MLM objectives, with a 30% probability. More details can be found in Appendix B.

heading: 3.2 Overall Pre-training
text: Since the indicator sequence is not a normal sentence, we don’t train the MLM task with contrastive learning together. Instead, the pre-training process is divided into two stages. In the first stage, we adopt the MLM task on the original statement sentences and activate text modules, to urge the model to understand the political text. Then, based on this checkpoint, we further conduct the multidimensional pre-training for political actor modeling by combining the objectives:
LCL = α ∗ LSCL + (1− α) ∗ LBCL (6)
where α is hyperparameters.

heading: 4 Experiment Setup
text: We fine-tune our model on different kinds of downstream tasks in quantitative political science. We then compare it with prior general PLMs and political domain-specific PLMs.

heading: 4.1 Pre-training Datasets
text: Compared to other political actors, congress legislators are more typical and they generate massive content every day. Thus, we start with legislators to construct our pre-training datasets.

heading: 4.1.1 Public Statements of Legislators
text: We obtained the Twitter accounts of members of Congress from Mou et al.. On the basis of it, we further crawl data of legislators elected after 2020 and tweets of all legislators up to April 2022. Overall, we get 887 legislators and delete the meaningless tweets including self-promotion advertisements, notifications, etc., using regular expressions. Finally, the cleaned data contains 2,020,938 tweets, covering discussions of events in various areas. We keep 10K held-out tweets as the validation set.

heading: 4.1.2 Legislative Context
text: We collect the party affiliation, sponsorship lists of bills, bills, and corresponding voting records from VoteView2 and the website of U.S. Congress3. Each bill belongs to a specific policy area and has textual information of title and description. We get bills of 112th and 113th for pre-training and reserve those of 114th and 115th for the formulation of downstream tasks. In the pre-training stage, 1,045 bills and 375,440 voting records are involved.
To correlate legislators’ votes to their statements in the related policy area, we filtered each legislator’s tweets in each policy area by the handcrafted information retriever mentioned in Sec 2.1. We finally acquire 1,142,587 tweets, and the details can be found in Appendix A.2. The distribution of the policy agenda of bills and the percentage of legislators whose related tweets can be retrieved in each policy area are shown in Figure 3a and Figure 3b. Over 90% of legislators can be retrieved with relevant statements in most policy areas.

heading: 4.2 Implementation Details
text: UPPAM is produced via continued pre-training on RoBERTa-base model (Liu et al., 2019b), where we add parallel FFN modules in each transformer layer with the same initialization as the original one. In the first stage, the model is trained on tweets, to minimize the MLM loss with AdamW (Loshchilov
2https://voteview.com/ 3https://www.congress.gov/
and Hutter, 2018) optimizer. In the second stage, the model is further trained on indicator sequences and bill texts, to minimize the LCL . We evaluate the model every 200 training steps on the validation set and keep the best checkpoint. The pre-training procedure takes around 96 hours on 4 Tesla V100SXM2 GPUs. More details and hyperparameters can be found in Appendix B.

heading: 4.3 Downstream Tasks and Datasets
text: We evaluate the models on three types of tasks, namely actor profiling, behavior prediction and language analysis. Notably, datasets include not only congress legislators but also other political actors such as journalists, news media, and even anonymous users, to validate the model’s generalization capability.

heading: 4.3.1 Actor Profiling
text: This type of task can be formulated as a user-level classification task, where we aggregate multiple statements to predict the speaker’s attribute.
Ideology Detection is the main task to profile actors broadly, aiming to predict political leaning. Models are evaluated on the following datasets.
• CongS (Gentzkow et al., 2018) collects speeches from US congressional records. • celeb (Wojcieszak et al., 2022) contains tweets of celebrities (journalists, politicians and media). We convert the ideology scores into labels according to the signs. • Reddit (Kitchener et al., 2022) collects comments of common users in non-political subreddits, and labels the users with ideology in the economic dimension. • PEM (Xiao et al., 2022) collects tweets of legislators, news outlets and cabinet of President Obama and President Trump. • TIMME (Xiao et al., 2020) includes Twitter accounts with location information and selfidentified political-polarity labels. These accounts are not run by politicians.

heading: 4.3.2 Behavior Prediction
text: This type of task can be regarded as a relation prediction task, where we predict a political actor’s attitude or action towards a given target with a piece of text description.
Vote Prediction tasks aim to predict votes of legislators towards bills with stances of yea or nay. We follow two configurations in (Mou et al., 2021).
• VoteIn refers to the in-session setup, where we randomly split the bills in the same congress session, i.e., the 114th session. • VoteOut refers to the more challenging outof-session setup, where we use data in the 114th session for training and validation while testing on the 115th session.
Grade Prediction tasks are designed as classification tasks for ratings in a certain issue, given a politician’s statements and background description of the given issue. We include datasets as follows:
• NRA Grades (Pujari and Goldwasser, 2021) provides politicians’ grades {A, B, C, D & F} assigned by National Rifle Association and their statements on guns, as well as background information of guns from ontheissues.org. • LCV Grades (Pujari and Goldwasser, 2021) is similar to NRA Grades, but it’s about the scores in the environment area.

heading: 4.3.3 Language Analysis
text: In addition to the overall characterization of political actors, we also test models’ ability to understand individual discourses. We apply stance detection and frame detection as downstream tasks, which can be formulated as sentence-level classification tasks.
Stance detection tasks aim to predict one’s stance towards a given target. The tasks take a 3-way label (favor, against, and neutral) or binary label (favor, against). We test on these datasets.
• poldeb (Somasundaran and Wiebe, 2010) provides opinion–target pairs from several debating platforms covering different domains. • election (Kawintiranon and Singh, 2021) contains tweets related to the 2020 US presidential election, expressing stances towards President Trump and Biden.
• SEval (Mohammad et al., 2016) is a shared task to detect stances in public tweets.
Frame detection tasks aim to detect which frame dimensions are employed in a piece of text. It’s a multi-label classification task with a pre-defined label set. We test on these datasets.
• twitter (Johnson et al., 2017) annotates tweets of politicians with 17 general frames. • gvfc (Liu et al., 2019a) collects news headlines about gun violence, and annotates them with 9 issue-specific frame dimensions. • immi (Mendelsohn et al., 2021) collects immigration-related tweets posted by the public, annotated with 14 general frames.

heading: 5 Experiment Results

heading: 5.1 Main Results
text: The compared general PLMs include BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019b). We also compare our model with available PLMs for social science texts-SsciBERT (Shen et al., 2022), and for the political domain: POLITICS (Liu et al., 2022) and PoliBERTweet (Kawintiranon and Singh, 2022). We fine-tune all the PLMs in the same settings, and we select the best fine-tuned model on validation sets using macro F1. The implementation details and hyperparameters can be found in Appendix C.2. Table 1 presents macro F1 scores on the downstream tasks.
Actor Profiling Our model shows superior performance on various political actor modeling tasks. Results of ideology detection tasks indicate that our model can not only characterize the ideology of legislators but is also good at modeling other roles like journalists in the celeb dataset and cabinet in the PEM dataset, demonstrating the transferability of using languages to represent characters. The reason for not performing best on the Reddit dataset
may be the gap between the expression habits of common users and that of politicians. Nevertheless, we still outperform the majority of baselines.
Behavior Prediction All the models show excellent performance on vote prediction and grade prediction tasks, using languages to represent political actors. It indicates that it’s a feasible scheme to infer political actors’ behaviors from their languages. Among all the PLMs, our model is the best. We attribute the performance gain to our proposed behavior-driven pre-training task.
Language Analysis Moreover, our model also achieves competitive performance on tasks of analyzing individual text including stance detection and frame detection, indicating that the ability to understand political languages is preserved while the model is learning to profile actors, benefiting from the co-training process in Sec 3.1.

heading: 5.2 Ablation Study
text: To explore the effects of different components, we conduct ablation studies and results are reported in Table 2. Removing SCL or BCL mainly hurts the performance of actor profiling tasks. Removing the text modules results in the most loss in language analysis tasks, especially the frame detection task. This demonstrates the necessity of separate modules to guarantee the ability to model political text.

heading: 5.3 Further Analysis
text: Few-shot Learning We fine-tune PLMs on different numbers of samples. Figure 4 shows UPPAM outperforms the baselines on nearly all the
tasks. Benefiting from the pre-training stages, our model can better capture ideology and preference differences, even when using only 16 samples.
Compare with Task-specific Models Taking the vote prediction task as an example, we compare our model with previous task-specific models, where particular meta-data and structural information is crafted for the task. Table 3 shows that UPPAM achieves competitive results, indicating that we can deduce political actors’ votes from languages. Additionally, our method can be used to analyze nonvoting actors, relieving the cold-start problem.
Methods of Statements Aggregation We show the impact of statements aggregation methods on ideology detection in fine-tuning. We mainly compare our method with concat (Table 4) and mean pooling (Table 5). concat means to concatenate each speaker’s political statements into a flat sequence and then encode it. mean pooling encodes each sentence individually and uses the averaged representation as the final representation. We further discuss the impact of the number of aggregated sentences in Appendix C.2.2. Results illustrate that our model shows robustness in several settings and our aggregator is more effective and efficient.

heading: 5.4 Visualization
text: General Ideology We perform Principle Component Analysis (PCA) on political actor representation generated by our model for the CongS dataset.
As shown in Figure 5a, our method can well separate politicians of different ideologies.
Individual Specific Preferences We also visualize specific representation in different policy areas for individuals. Figure 5b shows the representation in several highly-discussed policy areas, learned by different models from the tweets of Rep. Rooney. We can observe that Rep. Rooney behaves conservatively in immigration, but expresses left-wing views on environment (Pujari and Goldwasser, 2021). While most of our baselines fail to capture this nuance, UPPAM can well compare the relative polarity in each area.

heading: 6 Related Work
text: Political Actor Modeling focuses on modeling attributes and behaviors of political actors, with special attention to estimating the ideology. Because of the publicity and typicality, politicians like legislators have been the research subject for most work. The most widely used approach to estimate the ideology of legislators is ideal point model (Clinton et al., 2004) that represents legislators and bills as points in a one-dimension latent space from the rollcall data. After that, researchers further incorporate texts of bills (Gerrish and Blei, 2011; Gu et al., 2014) to enhance the model, solving the problem of prediction on new bills. Some embedding methods are also proposed to promote learning of legislators (Kraft et al., 2016; Kornilova et al., 2018).
More recently, external information including cosponsorship (Yang et al., 2020), donors (Davoodi et al., 2020), relevant stakeholders (Davoodi et al., 2022) and expert knowledge (Feng et al., 2021, 2022) is used to better learn legislator representation. They follow a mixed structure of textual encoder and graph encoder, to explicitly combine textual and structural information. Despite outstanding performance on target tasks, these methods are limited to certain settings or data, behaving inefficient in dynamic political scenarios. Thus they are hard to be transferred to all actors. By contrast, methods relying on texts (Vafa et al., 2020) provide more possibility for generalization.
Domain-specific Pre-training Based on continued pre-training on domain-specific data, domainspecific Pre-trained Language Models have shown superiority on many NLP tasks. Domain-specific PLMs have been investigated in many areas including medical (Zhang et al., 2021) and financial (Araci, 2019) domains. However, little work has explored PLMs in the political domain. Li and Goldwasser pre-trained a hierarchical LSTM for political perspective identification. Kawintiranon and Singh followed BERTweet (Nguyen et al., 2020) to train a PoliBERTweet for stance detection in elections. Liu et al. recently proposed storylevel contrastive learning for ideology understanding. These researches pave the way for pre-training in the political domain, but they currently only consider training objectives at the text level and are not yet able to deal with more complex problems in this domain. Thus, our work is novel in dealing with multiple levels of practical problems.

heading: 7 Conclusion
text: In this paper, we propose to learn political actors from languages and inject multidimensional domain knowledge into the PLMs through structureaware contrastive learning and behavior-driven con-
trastive learning. Experimental results validate the effectiveness and generalization capability of our approach.
Limitations
Our work is the first step towards unified pretraining for political actor modeling and it is limited in two aspects. In terms of data, we focus on the typical political actors, i.e., the congress legislators, and their statements, without using a larger corpus like political news. But our method can be easily scaled to a larger corpus, where we can aggregate articles of different media and consider their structure information like page links for pre-training. In terms of method, in order to improve the retrieval efficiency in both pre-training and fine-tuning, we use simple methods rather than dynamic selection methods based on embeddings to query and aggregate statements, leaving much room for future exploration.
Ethics Statement
Data Collection and Privacy Our data collection is in compliance with Twitter’s terms of service and matches previous publications. Although tweets are public, when releasing data, we will share user id or tweet id rather than raw data, to minimize the privacy risk.
Political Leaning Since political identity is becoming increasingly important in American society, the models could come up with some risks if a user is mislabeled with an error affiliation, e.g., a user may be socially ostracized for their supposed political beliefs (Alkiek et al., 2022). However, the research subject in this paper is public political actors, which have been studied in political science for decades, rather than the common public. Instead, understanding the bias and behaviors of these characters can help our public avoid being polarized by their certain strategies, mitigating the potential risk.

heading: Acknowledgements
text: This work is partially supported by National Natural Science Foundation of China (No. 71991471) and Science and Technology Commission of Shanghai Municipality Grant (No.21QA1400600). We would also like to thank the anonymous reviewers for their constructive feedback.

heading: A Data Cleaning and Retrieval
text: In this section, we provide the details of data cleaning and how we implement a query mechanism to obtain relevant statements from the corpus.
A.1 Data Cleaning Steps
Remove Meaningless Tweets We have observed that some tweets of the legislators do not express opinions and are unrelated to this research, such as the self-promotion advertisements, and notifications. Thus, we delete these tweets using regular expressions. Some examples of filter patterns are shown in Table 6.
Clean Tweets We replace urls and user mentions with symbols [URL] and [MENTION].
A.2 Information Retriever
In order to improve the efficiency of the retrieval, we do not use dynamic methods such as sentence embedding similarity but based on previous work (Barberá et al., 2019; Pujari and Goldwasser, 2021) to implement manual rules for querying.
Firstly, we get all policy areas of US politics and their description and codebook from https://www.congress.gov/ and https://www. comparativeagendas.net/. We then extract nouns and adjectives and delete stopwords from the description of each policy area, to form the keywords for each policy area. Since the words used in the definition can be abstract such as economy, we further summarize hashtags for specific issues following (Pujari and Goldwasser, 2021). The process is divided into 3 steps. (1) We counted hashtags in the corpus and retained hashtags that at least 100 members used. (2) Then, we merge hashtags for the same event or issue based on co-occurrence. (3) At last, we mapped events or issues to policy areas through the wiki of events and the codebook of policy areas to form a mapping from hashtags to policy areas. Overall, we use the keywords and
hashtags to retrieve tweets of each member. Table 7 shows some examples.

heading: B Pre-training Details
text: This section illustrates some details in the continued pre-training part.
Training Details. UPPAM is produced via continued pre-training on RoBERTa-base model (Liu et al., 2019b), where we add parallel FFN modules in each transformer layer with the same initialization as the original one. In this way, our model contains about 153M parameters. Our implementation is based on the HuggingFace Transformers library4. The hyperparameters are listed in Table 8.
MLM Strategy We link entities using DBPedia spotlight5 with types of person, organization and event. We identify sentiment words and frame indicators using lexicons by (Wilson et al., 2005) and (Roy and Goldwasser, 2020). We mask these tokens with a 30% probability, and randomly mask the remaining tokens with a 15% probability. As done in (Devlin et al., 2019), the masked tokens are replaced with [MASK], random tokens and the original tokens with a ratio of 8:1:1.
Construction of Triplets When generating triplets using the co-sponsorship information, we may get member triplets with a pattern of "<D, R, D>" or "<R, D, R>", where D and R represent Democrat and Republican. These samples can contradict some samples generated according to party affiliation. Thus, we deleted samples in these formats.

heading: C Fine-tuning Details
text: C.1 Fine-tuning Datasets
This section lists more details of the datasets used in our downstream evaluation. Statistics are listed in Table 9.
4https://github.com/huggingface/transformers 5https://www.dbpedia-spotlight.org/
Hyperparameter Value
number of steps 9,600 for stage1; 2,890 for stage 2 batch size 2048 maximum learning rate 2e-5 learning rate scheduler linear decay with warmup warmup percentage 10% optimizer AdamW δSCL 1 δBCL 1 α 0.5 N 256 M 256
self-identified political-polarity labels (either Democratic or Republican).
• VoteIn & VoteOut (Mou et al., 2021): For the in-session setup, we randomly select 20% bills for testing, 10% is for validation and the rest for training. For the out-of-session settings, we train and validate on bills of the 114th congress and test on that of the 115th congress. For both settings, bills in the test are unseen in training.
• NRA & LCV (Pujari and Goldwasser, 2021): we use the formatted statements and tweets provided by the paper to predict the NRA and LCV rankings, which are originally National Rifle Association (NRA) scores and League of Conservation Voters (LCV) scores.
• poldeb (Somasundaran and Wiebe, 2010): covers debates in domains of the existence of god, healthcare, gun rights, gay rights, abortion and creationism.
• election (Kawintiranon and Singh, 2021): includes tweets expressing support or opposition towards Trump or Biden during the 2020 US election period.
• SEval (Mohammad et al., 2016): The dataset contains stances towards six targets: Atheism, Climate Change, Feminist, Hillary Clinton, Abortion, and Donald Trump.
• twitter (Johnson et al., 2017): Tweets are annotated with 17 frame dimensions, covering 6 issues, i.e., abortion, aca, guns, immigration, isis and lgbt.
• gvfc (Liu et al., 2019a): 1,300 headlines of news articles on gun violence, annotated with 9 issue-specific frames.
• immi (Mendelsohn et al., 2021): We use the tweets which are annotated with 14 general frames.
C.2 Fine-tuning Procedure
C.2.1 Fine-tuning in Main Experiments Ideology Detection. We aggregate general statements of the speakers using the method mentioned in Sec 2.1. Then we encode the sequence and use the representation of [CLS] token for classification. We only activate actor modules during fine-tuning.
Vote Prediction & Grade Prediction. We encode the bills or issues using text modules. And we aggregate both general statements and specific statements about the bill’s policy area or given issue to represent legislators using the actor modules. Then we calculate the dot product of the representation and apply an FFN for classification. Stance Detection. We formulate a simple input by concatenating the target and the text and use the [CLS] token for standard fine-tuning. We only activate text modules during fine-tuning. Frame Detection. We use the [CLS] token for standard fine-tuning of sequence classification (Devlin et al., 2019). The threshold for multilabel classification is set to 0.5 for all models. We only activate text modules during fine-tuning.
Fine-tuning hyperparameters are listed in Table 10.
C.2.2 Fine-tuning Experiments of Different Aggregation Methods
Concatenation Limited by the length of the input sequence of PLMs, we select political-related content using the method in Appendix A.2. And we will truncate the concatenated sequence if it has more than 512 tokens. Mean Pooling Due to computational resource constraints, instead of encoding all statements of a person, we randomly sample records of his politicalrelated statements as input. And the averaged sentence embeddings are used as actor representation. Impact of the Number of Aggregated Statements We further explore the impact of the number of statements when using mean pooling to acquire political actor representation in the fine-tuning process. Figure 6 illustrates the average macro F1 of different models on ideology detection tasks.
We can observe that utilizing more sentences can improve the performance, where aggregating 64 statements can achieve competitive results of our
proposed indicator sequence method. However, this method costs 64 times more training time than our method.
C.3 Fine-tuning Results Table 11, Table 12, Table 13 and Table 14 show the standard error of our 3 runs of fine-tuning on downstream tasks.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
We discuss the limitations in the "Limitation" section.
3 A2. Did you discuss any potential risks of your work? Please refer to the "Ethic Statement" section.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Please refer to the "Introduction" section.
7 A4. Have you used AI writing assistants when working on this paper? No.
B 3 Did you use or create scientific artifacts? Please refer to section 2&3.
3 B1. Did you cite the creators of artifacts you used? Please refer to section 4.
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? We use multiple existing open-source artifacts that are based on different licenses, making it difficult to summarize. We cite the resources of utilized artifacts where the license details can be found.
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Please refer to section 4.
3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Please refer to section 4 and Appendix C.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Please refer to section 4.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Please refer to section 4&5 and Appendix B&C.
C 3 Did you run computational experiments? Please refer to section 4&5.
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Please refer to section 4 and Appendix B&C.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Please refer to section 4 and Appendix B&C.
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Please refer to section 5 and Appendix B&C.
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Please refer to section 4 and Appendix A.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: In terms of method, in order to improve the retrieval efficiency in both pre-training and fine-tuning, we use simple methods rather than dynamic selection methods based on embeddings to query and aggregate statements, leaving much room for future exploration