SlowGuess/NAACL_2022_OCR huggingface dataset:
<sup>9</sup>We use 0.95 for politics pages and 0.9 for non-politics pages.
# article before downsample	Earliest date	Latest date
Daily Kos	235,244	2009-01-02	2021-06-30
HuffPost (HPO)	560,581	2000-11-30	2021-06-30
CNN	152,579	2000-01-01	2021-06-30
The Washington Post (WaPo)	461,032	2000-01-01	2021-06-30
The New York Times (NYT)	403,191	2000-01-01	2021-06-22
USA Today	174,525	2001-01-01	2021-06-30
Associated Press (AP)	285,685	2000-01-01	2021-06-30
The Hill (Hill)	337,256	2002-10-06	2021-06-30
The Washington Times (TWT)	336,056	2000-01-01	2021-06-30
Fox News (FOX)	457,550	2001-01-12	2021-06-25
Breitbart News (Breitbart)	285,530	2009-01-08	2021-06-30
Table A4: Statistics of BIGNEWS corpus. Media outlets are sorted by ideology from left to right.
Hyperparameter	Value
number of steps	2,500
batch size	2048
maximum learning rate	0.0005
learning rate scheduler	linear decay with warmup
warmup percentage	6%
optimizer	AdamW (Loshchilov and Hutter, 2019)
weight decay	0.01
AdamW beta weights	0.9, 0.98
δideo	0.5
δstory	1.0
Table A5: Hyperparameters for continued pretraining.
paragraphs, if they contain a sentence that matches such pattern.
Appendix B News Story Alignment
As shown in Equation 1, we combine text similarity and entity similarity to be the final story similarity score. Only title and the first five sentences are considered in the calculation. We further require aligned articles  $a$  and  $b$  to satisfy two constraints:
- The difference in publication dates of  $a$  and  $b$  is at most three days.
-  $a$  and  $b$  must contain at least one common named entity in the title or in the first three sentences.
We use CoreNLP to extract named entities in arti
cles (Manning et al., 2014). For the second constraint, we further apply Crosswikis to map each entity to a unique concept in Wikipedia (Spitkovsky and Chang, 2012). When calculating entity similarity, we split each entity into single words and remove stop words. After alignment, we use the procedure described in Appendix A to remove duplicate articles in the same story cluster. The hyperparameters are  $\alpha = 0.4$  and  $\theta = 0.23$ .
Evaluating Alignment Algorithm. We search the hyperparameters on the Basil dataset (Fan et al., 2019) and test the algorithm on the Allsides dataset collected in Cao and Wang (2021). The Allsides dataset consists of manually aligned news articles from 251 media outlets. After removing media outlets not in BIGNEWSBLN, we obtain 2,904 articles on 1,316 stories.
To evaluate the performance of the alignment algorithm, we add the evaluation dataset into BIGNewsBLN and treat each evaluation article as the anchor article for the alignment algorithm. We use the remaining evaluation articles in the same story as relevant articles, which becomes the target to be identified. The algorithm achieves 0.612 mean reciprocal rank (MRR) on the Basil dataset and 0.679 MRR on the Allsides dataset.
Appendix C Continued Pretraining and Fine-tuning
C.1 Continued Pretraining
We initialize all variants of POLITICS with a RoBERTa-base model (Liu et al., 2019), which contains about 125M parameters. Our implemen
Prompt	Verbalizer
p [SEP] The stance towards {target} is [MASK].	negative or positive
p [SEP] It reveals a [MASK] stance on {target}.	negative or positive
p [SEP] The speaker holds a [MASK] attitude towards {target}.	negative or positive
p [SEP] What is the stance on {target}? [MASK].	Negative or Positive
p [SEP] The previous passage [MASK] {target}.	opposes or favors
p [SEP] The stance on {target} is [MASK].	negative or positive
p [SEP] The stance towards {target}: [MASK].	negative or positive
p [SEP] The author [MASK] {target}.	opposes or favors
p [SEP] [MASK] {target}	oppose or favor
p [SEP] [MASK]. {target}	No or Yes
p [SEP] [MASK] {target}	No or Yes
Table A6: List of prompts used for stance detection tasks.  $p$  is the input text, and  $\{$  target  $\}$  is the target of interest. Verbalizer maps the label (e.g., against) to the token (e.g., negative). Some datasets use a third label (neutral).
Hyperparameter	Value
number of epochs	10
patience	4
maximum learning rate	0.00001 or 0.00002
learning rate scheduler	linear decay with warmup
warmup percentage	6%
optimizer	AdamW
weight decay	0.001
AdamW beta weights	0.9, 0.999
# FFNN layer	2
hidden layer dimension in FFNN	768
dropout in FFNN	0.1
sliding window size	512
sliding window overlap	64
tation is based on the HuggingFace transformers library (Wolf et al., 2020).<sup>10</sup> We train each model using 8 Quadro RTX 8000 GPUs for 2,500 steps. The total training time for POLITICS is 20 hours, with shorter time for other variants. Table A5 lists the training hyperparameters.
Training Details. For triplet loss objectives, we only consider triplets in each mini-batch. We skip a batch if it contains no triplet. For the MLM objective, we truncate the article if it has more than 512 tokens. When masking entities and sentiment words, we only consider those with at most five tokens. When both triplet loss and MLM objectives
Table A7: Hyperparameters used to fine-tune PLMs.
Hyperparameter	Value
kernel	linear
regularization strength	0.3, 1, or 3
features	unigram and bi-gram TF-IDF
minimum document frequency	5
maximum document frequency	0.7 * |D|
Table A8: Hyperparameters used to train SVM.  $|D|$  is the number of documents in the training set.
are enabled, we adopt alternating training strategy as in Ganin et al. (2016) to apply these two objectives for parameter updates in an alternating manner.
For both ideology prediction and stance detection tasks, we fine-tune each model for up to 10 epochs. We use early stopping and select the best checkpoint on validation set among 10 epochs. For ideology prediction tasks, we follow standard practice of using [CLS] token and feed-forward neural networks (FFNN) for classification. For stance detection tasks, we use prompts to fine-tune PLMs. We curate 11 prompts as shown in Table A6, and select the best prompt based on the performance of RoBERTa. Fine-tuning hyperparameters are listed in Table A7.
For the SVM classifier, we use the implementation of TF-IDF feature extractor and linear SVM classifier in scikit-learn (Pedregosa et al., 2011). The classifier's hyperparameters are listed in Table A8.
10 https://github.com/huggingface/ transformers.
Models	VAST
Ffavor	Fagainst	Favg
BERT-joint (Allaway and McKeown, 2020)	54.5	59.1	65.3
TGA Net (Allaway and McKeown, 2020)	57.3	59.0	66.5
BERT-base (Jayaram and Allaway, 2021)	64.3	58.1	69.2
prior-bin:gold (Jayaram and Allaway, 2021)	64.5	54.6	68.4
RoBERTa	67.2	71.4	76.5
POLITICS	68.0	72.2	77.0
Appendix D Downstream Evaluation Datasets
This section lists more details of the eight datasets used in our downstream evaluation as well as their processing steps.
D.1 Ideology Prediction
Congress Speech<sup>11</sup> (CongS; Gentzkow et al., 2018): We filter out speeches with less than 80 words and use the speaker's party affiliation as the ideology of the speech.
- AllSides<sup>12</sup> (AllS): We crawl articles from AllSides and use the media outlet's annotated ideology as that of the article.
- Hyperpartisan<sup>13</sup> (HP; Kiesel et al., 2019): We convert the benchmark into a 3-way classification task by projecting media-level ideology annotations to articles.
- YouTube (Wu and Resnick, 2021) contains cross-partisan discussions between liberals and conservatives on YouTube. In our experiments, we only keep controversial comments: 1) A video must have at least 1,500 comments and 150,000 views; 2) A comment must have at least 20 replies. The original dataset annotates users' ideology on a 7-point scale. We further convert it into a 3-way classification task for left, center, and right ideologies. For the comment-level prediction task on YT (cmt.), we use the provided user-level ideology annotation. For user-level prediction on YT (user), we concatenate all comments by a user.
- Twitter (TW; Preojiuc-Pietro et al., 2017): We crawl recent tweets by each user and remove replies and non-English tweets. We as
Table A9: Comparison with state-of-the-art results on the original VAST dataset. Best results are in bold.
Models	SemEval (Seen)
\(F_{favor}\)	\(F_{against}\)	\(F_{avg}\)
WKNN (Al-Ghadir et al., 2021)	84.49	68.36	76.45
PNEM (Siddiqua et al., 2019)	66.56	77.66	72.11
MITRE (Zarrella and Marsh, 2016)	59.32	76.33	67.82
pkudblab (Wei et al., 2016)	61.98	72.67	67.33
SVM-grams (Mohammad et al., 2016a)	62.98	74.98	68.98
Majority class (Mohammad et al., 2016a)	52.01	78.44	65.22
BERT	62.89	70.75	66.82
RoBERTa	67.33	75.52	71.43
POLITICS	67.36	75.29	71.33
Table A10: Comparison with the state-of-the-art models on SemEval. Prior work (top panel) trains five models (one per target label). On the contrary, in this work, we target a more generalizable approach, i.e., one unified classifier for all labels. Due to different setups, POLI-TICS and baselines like RoBERTa perform worse.
sume users' ideologies do not change after their self-report since prior work has shown that people's ideology is less likely to change across the political spectrum (Fiorina and Abrams, 2008). We sort all tweets from a user chronologically and concatenate them.
D.2 Stance Detection
- BASIL $^{14}$  (Fan et al., 2019): We convert the original dataset such that the new tasks are to predict the stance towards a target at two granularities: article (art.) and sentence (sent.) levels. The targets in the dataset can be a person (e.g., Donald Trump) or an organization (e.g., Justice Department).
- VAST $^{15}$  (Allaway and McKeown, 2020) predicts the stance of a comment towards a target. The targets in the dataset are noun phrases covering a broad range of topics (e.g., immigration, home schoolers). We notice the original dataset contains contradictory samples, where the same comment-target pair is annotated with opposite stances, and therefore remove duplicate and contradictory samples.
- SemEval<sup>16</sup> (SEval; Mohammad et al., 2016a) predicts a tweet's stance towards a target. The dataset contains six targets: Atheism, Climate Change, Feminist, Hillary Clinton, Abortion, and Donald Trump. Notably, the last target is not seen during training, and only appears in testing.
11https://data.stanford.edu/congress_text.
12https://www.allsides.com.
13https://webis.de/data/
pan-semeval-hyperpartisan-news-detection-19.  $^{16}$ https://alt.qcri.org/semeval2016/htmltask6/index.php?id=data-and-tools.
$^{14}$ https://github.com/marshallwhiteorg/emnlp19-media-bias.
15https://github.com/emilyallaway/ zero-shot-stance.
.  $^{16}$ https://alt.qcri.org/semeval2016/task6/index.php?id=data-and-tools.
Ideology Prediction	Stance Detection	All avg
YT(cmt.)	CongS	HP	AllS	YT 用户)	TW	Ideo. avg	SEval (seen)	SEval (unseen)	Basil (sent.)	VAST	Basil (art.)	Stan. avg
Baselines
SVM	65.34±0.00	71.31±0.00	61.25±0.00	52.51±0.00	66.49±0.00	42.85±0.00	59.96	51.18±0.00	32.89±0.00	51.08±0.00	39.54±0.00	30.77±0.00	41.09	51.38
BERT	64.64±1.92	65.88±1.13	48.42±1.44	60.88±0.83	65.24±1.53	44.20±2.03	58.21	65.07±1.02	40.39±0.53	62.81±3.95	70.53±0.43	45.61±3.92	56.88	57.61
RoBERTa	66.72±0.85	67.25±0.48	60.43±3.13	74.75±1.26	67.98±4.03	48.90±1.53	64.34	70.15±0.87	63.08±0.77	68.16±2.55	76.25±0.11	41.36±7.35	63.80	64.09
Baly et al. (2020)
with Original Data	65.42±0.56	66.74±1.33	58.37±1.63	72.89±0.50	70.47±1.77	44.95±1.02	63.14	68.66±0.50	56.29±2.07	61.30±2.41	75.57±0.67	37.98±3.43	59.96	61.69
with BIGNewsBLN	68.57±1.02	70.39±0.38	71.24±2.06	76.47±3.35	74.74±1.63	47.38±1.31	68.13	65.84±0.74	49.54±2.03	60.60±6.55	75.03±1.58	41.84±9.30	58.57	63.79
Our models with triplet loss objective only
Ideology Obj.	66.20±1.46	68.18±0.54	64.15±6.82	76.52±1.62	68.15±6.89	42.66±10.84	64.31	68.78±0.79	59.61±3.97	64.18±4.41	76.03±0.32	44.94±5.61	62.71	63.58
Story Obj.	66.09±1.05	69.11±1.21	56.70±2.64	74.59±1.68	68.89±3.18	46.53±3.29	63.65	69.02±0.38	63.54±1.19	67.21±2.51	76.66±1.29	53.16±6.76	65.92	64.68
Ideology Obj. + Story Obj.	68.91±0.44	69.10±0.71	63.08±3.10	76.23±2.96	77.58±2.83	48.98±1.42	67.31	69.66±0.45	63.17±1.92	64.37±1.58	76.18±1.13	47.01±7.55	64.08	65.84
Our models with masked language model objective only
Random	67.82±1.30	70.32±0.94	60.59±2.22	73.54±1.55	70.77±1.43	44.62±2.32	64.61	69.16±0.84	60.39±0.85	69.94±1.61	77.11±0.53	39.16±3.71	63.15	63.95
Upsamp. Ent.	69.06±1.00	70.32±0.39	60.09±0.98	70.89±1.81	71.40±2.23	47.16±1.07	64.82	69.81±0.61	63.08±1.90	69.49±1.85	76.76±1.01	46.46±5.56	65.12	64.96
Upsamp. Sentiment	67.41±1.12	70.03±0.96	56.05±5.68	72.35±1.09	74.93±2.70	48.15±1.30	64.82	70.09±0.51	60.81±1.22	71.28±2.31	76.61±0.62	44.42±4.91	64.64	64.74
Upsamp. Ent. + Sentiment	68.31±0.37	71.42±0.51	58.02±3.34	71.90±0.61	71.04±3.56	47.31±2.07	64.67	69.25±0.71	62.84±3.93	69.23±1.08	77.10±0.73	43.16±4.95	64.32	64.51
POLITICS	67.83*±0.49	70.86±0.31	70.25*±2.10	74.93±0.83	78.73*±1.15	48.92±2.19	68.59	69.41±0.36	61.26±1.23	73.41*±0.97	76.73*±0.60	51.94*±3.42	66.55	67.66
Table A11: Macro F1 scores on 11 evaluation tasks (average of 5 runs) with standard deviations. Tasks are sorted by text length, short to long, within each group. "All avg" is the average of all 11 tasks. Best results are in bold and second best are underlined. Our models with triplet-loss objectives that outperform RoBERTa are in blue. Our models with specialized sampling methods that outperform vanilla MLM (Random) are in green. POLITICS uses Ideology + Story Obj. and Upsamp. Ent. + Sentiment. Results where POLITICS outperforms all baselines are highlighted in red. POLITICS has the smallest standard deviations on 5 tasks, showing its stable performance.
Appendix E Task Property
This section introduces detailed definitions of four properties, based on which we divide tasks into two categories reported in Figure 3.
- Formality: Speech and news genres are considered as formal, and others are informal.
- Training set size: Datasets with more than 2,000 training samples are categorized as large, and small otherwise.
- Document length: Datasets with average document length larger than 500 are treated as "long", and others are short.
- Aggregation level: If a dataset is a collection of single articles/posts/tweets, then it is in the category of "Single". If posts are concatenated and aggregated at user level, then it is marked as "User". Specifically, only YouTube User and Twitter in Table 2 are in the "User" category.
Appendix F Comparison with Previous State-of-the-art Models
Here we compare POLITICS with previous state-of-the-art models on three selected datasets: Hyperpartisan (\$5.5), VAST (\$F.1), and SemEval (\$F.2). \$F.3 discusses why direct comparisons are not applicable on the other 5 datasets.
F.1 VAST
POLITICS outperforms all previous state-of-the-art models in the literature as well as the strong RoBERTa baseline, as can be seen in Table A9. Following Allaway and McKeown (2020) and Jayaram and Allaway (2021),  $\mathrm{F}_{avg}$  is defined as the
macro-averaged F1 over all three classes (favor, against, and neutral). The results are reported on the original VAST dataset which contains contradictory samples, where the same comment-target pairs are annotated with opposite stances so they are counted in both categories.
F.2 SemEval
Table A10 show the results of state-of-the-art models and POLITICS on SemEval. Following Mohammad et al. (2016a),  $F_{avg}$  is defined as the macro-averaged F1 over favor and against classes. State-of-the-art models train separate classifiers, one for each target, thus yield better results than POLITICS. Similar observation is made by Mohammad et al. (2016a), where one single SVM that is trained on all five targets performs worse than five one-versus-rest SVM classifiers.
F.3 Reasons for Inapplicable Comparisons
We are unable to directly compare with existing models on datasets other than Hyperpartisan, VAST, and SemEval for the following reasons:
- The original dataset either is used for different tasks that are not ideology prediction or stance detection: Congress Speech (Gentzkow et al., 2018), BASIL (Fan et al., 2019), and YouTube (Wu and Resnick, 2021).
- The dataset is newly collected (AllSides) or contains newly collected samples (Twitter; Preotiuc-Pietro et al., 2017).
Appendix G Visualize Attention Weights
Figure A1: Example 1 for attention visualization. Last layer attention weights between [CLS] token and other tokens in the input. We illustrate the first 85 tokens of the article.
Figure A2: Example 2 for attention visualization. Last layer attention weights between [CLS] token and other tokens in the input. We illustrate the first 85 tokens of the article.
Figure A3: Example 3 for attention visualization. Last layer attention weights between [CLS] token and other tokens in the input. We illustrate the first 85 tokens of the article.
Figure A4: Example 4 for attention visualization. Last layer attention weights between [CLS] token and other tokens in the input. We illustrate the first 85 tokens of the article.