Multilingual estimation of political-party positioning: From label aggregation to long-input Transformers
Authors: Dmitry Nikolaev
Tanise Ceron
Sebastian Padó
Venue: EMNLP 2023 Main (2023)
Keywords: computational social science
scaling analysis
political science
political party positioning
TL;DR: A comparison of two approaches to the estimation of the Standard Right–Left Scale from political-party manifestos in a multilingual setting

Abstract: Scaling analysis is a technique in computational political science that assigns a political actor (e.g. politician or party) a score on a predefined scale based on a (typically long) body of text (e.g. a parliamentary speech or an election manifesto). For example, political scientists have often used the left–right scale to systematically analyse political landscapes of different countries. NLP methods for automatic scaling analysis can find broad application provided they (i) are able to deal with long texts and (ii) work robustly across domains and languages. In this work, we implement and compare two approaches to automatic scaling analysis of political-party manifestos: label aggregation, a pipeline strategy relying on annotations of individual statements from the manifestos, and long-input-Transformer-based models, which compute scaling values directly from raw text. We carry out the analysis of the Comparative Manifestos Project dataset across 41 countries and 27 languages and find that the task can be efficiently solved by state-of-the-art models, with label aggregation producing the best results.

Multilingual estimation of political-party positioning: From label aggregation to long-input Transformers
Dmitry Nikolaev Tanise Ceron Sebastian Padó
Institute for Natural Language Processing, University of Stuttgart
dnikolaev@fastmail.. com {tanise ceron, padoo@ime umi-ssuttgart.de
Abstract
was developed in the framework of the Mani- festo Research on Political Representation project (MARPOR), formerly known as the Comparative Manifestos Project (CMP), which collects, anno- tates, and makes available a large collection of party platforms from different countries. The RILE score is deductive, firs-princiipl--bsse method for describing party positions geared towards the widest possible applicability across time and coun- tries (Budge, 2013). For this very reason, it rather conservative and inflexible and has been repeatedly criticised (see, c.g., Flentje et al., 2017). Despite this, it is widely used in computational political sci- ence for model validation (Rheault and Cochrane, 2020), as dependent variable in regression analy- ses (Greene and O'Brien, 2016), or basis for party-stance analysis (Däubler and Benoit, 2022). A major practical drawback of the RILE score the fact that is computed based on the labels manually assigned by MARPOR annotators to all statements in party manifestos (see Section 2 for details). This procedure is expensive and time con- suming, which raises the question of whether we can adequately upproximate the RILE score using hotural-anngage-proossiiii methods, especially in multilingual setting, This will make it possible efficiently perform analyses of political texts that have not yet been covered by the MARPOR project due to timing constraints (e.g. manifestos from up- coming elections), accidental gaps (c.g.. Indonesia and the Philippines are not part of the dataset, and coverage of many countries, such as South Africa, is incomplete), lack of resources (there very few annotated manifestos from before 2000).
Scaling analysis is a technique computa- tional political science that assigns politi- cal actor (e.g. politician or party) a score on a predefined scale based on a (typically long) body of text (e.g. parliamentary speech or an election manifesto). For example, political scientists have often used the left-right scale to systematically analyse political landscapes different countries. NLP methods for au- tomatic scaling analysis can find broad appli- cation provided they (i) are able to deal with long texts (ii) work robustly across domains and languages. In this work, we implement and compare two approaches to automatic seal- ing analysis of political-party manifestos: la- bel aggregation, pipeline strategy relying annotations of individual statements from the manifestos, and long-input-Ti mang--inpu-Trrnffrmmme Transformer-baser models, which compute scaling values directly from raw text. We carry out analysis the Comparative Manifestos Project dataset across 41 countries and 27 languages and find that the task can be efficiently solved by state-of- sate-of-thhe-rr tht-art models, with label aggregation producing the best results.
1 Introduction
A widely used tool in computational political sci- ence is the so-called 'scaling analysis': set of methods for representing political platforms numbers on certain scale, such left- right, authoritarian-libertaaan or conservative- progressive (Laver et al., 2003; Slapin and Proksch, 2008; Diermeier et al., 2012; Lauderdale and Clark, 2014; Barberá, 2015). A wide variety of scales have been proposed in the literature, some based on tolitical-theoretic considerations (Jahn, 201 1), others more data-driven (Gabel and Huber, 2000; Albright, 2010; Rheault and Cochrane, 2020).
This work is a first step in this direction. Our contributions are the following:
Previous works computational analysis of party positioning targeted a limited number of texts from single country or several coun-
One well-established scoring scheme of this kind is the Standard Right-Lot Left Scale, also known as the RILE score (Budge, 2013; Volkens et al., 2013).
http:////aniffstttooojjjeeec www.jjjett.bbbteee....... wzb.eu/

---

tries. We scale the analysis up to 41 countries and 27 languages, including comparatively low-resource languages (such as Georgian and Armenian) that have not been tackled before.
Freedom and Human Rights, or subtypes thereof, are assigned 'favourable mentions of importance of personal freedom and civil rights' (Volkens al., 2020, 12). Some categories form binary opposi- tions (e.g. Constitutionalism: Positive vs. Consti- Negative), and some are purely one- sided (e.g. Freedom and Democrucy have positive loadings and do not have negative counterparts). As result, it is possible derive inferences about political stances of different parties from category counts alone. This provides a straightforward op- crationalization of the political-scient notion of issue salience, which is commonly used to analyse political positioning (Lipstein and Segal, 2000) the number of occurrences of category correlates with how important it is for party.
2. We contrast the label-aggeegttion approach, based on a statement-level classifier mimick- ing the work of a human annotator, with using long-input Transformer models predicting the scores directly from raw manifesto texts.
tutionalism:
3. In the label-aggregation setting, we further compare the performance of multilingual- modelling- and machine-tranie translation- based approaches. While the former more straightforward in the sense that a single base model can be directly used without any pre- processing, MT systems are easier to train for less-resourc languages, and only a single-language classifier is needed for predic- tions.
In total, there are 143 different categories, with 56 major categories, 32 sub-categories of the ma- jor categories, 54 additional categories, and the residual category 0.3
4. We evaluate the generalisability of models re- garding two dimensions: local (moving to new countries) and temporal (moving from the past to the future). These correspond to different real-life research scenarios. We show that our methods deal reasonably with both cases.
Right-Left scale A prominent way of analysing party positioning is the Standard Right-Left Scale, a.k.a. RILE score (Budge, 2013; Volkens et al., 2013). Originally developed in the framework of the MARPOR project, it has been consistently used in its publications and remains a standard reference scale for party positioning, despite a number of pro- posals to improve or replace it, both using theory- based and data-driven approaches (cf. Cochrane, 2015; Mölder, 2016; Flentje et al., 2017).
The paper is structured as follows: In $ 2, we provide more information on the MARPOR anno- tations and on how the RILE score is computed. The exact problem statement, different operational- ization strategies, and the experimental setup are presented in $ 3, while the results of the study given in 4. Additional discussion is provided in $ 5. Section surveys related work. Section con- cludes the paper and discusses directions for future research.
R L RILE R 1
(1)
Eq. 1 shows the formula for computing the RILE score based on sets of categories defined by MAR- POR as right- wing and left-wing, respectively. The categories belonging to the right and left sets are shown in Table 1.
2 MARPOR categories and political scales
R(right) and L(ett) are the percentages of state- ments labelled with categories from the two sets, and O the percentage of other statements. The range of RILE is [-1, 1]. Large absolute values in- dicate extreme left and right programs, and values close to zero correspond to centrist manifestos with a balanced program.*
Categories The annotations of the manifesto crc- ated in the framework of the Comparative Mani- festos Project follow the project codebook (Volkens et al., 2020). Each statement of given manifesto is annotated with a category representing a spe- cific policy domain (e.g. Military or Sustainabiliyy. These categories can be identified via their names and numbers (e.g., 103, Anti-lmperialismm.
*In some manifestos, special label 'H' attached head- ings. As we cannot reliably automatically identify headings in new texts, H's were converted to O's throughout.
A key feature of MARPOR categories is that they are not stance-neutral. Thus, category 201,
"They could also arise from political programs where most statements are associated with neither left nor right, but such programs are rare in practice.
"See Appendix B for the major categorics with numbers.

---

Military; Positive, Preedom, Human Rights, Constitutionalsmm: Positive,
Political Authority, Free Enterprise, Economic Incentives, Protectionism:
Right emphasis
Negative, Economic Orthodoxy. Social Services Limitation, National Way
of Life: Positive, Traditional Morality: Positive, Law and Order, Social Harmony Decolonisation, Anti-mmperialism Military: Negutive, Peace, Internationaism: Positive, Democracy, Regulate Capitalism, Market, Economic Planning, Positive, Controlled Economy, Nationalisation, Social Services: Expansion, Education: Expansion, Labour Groups: Positive
Left emphasis
Protectionism:
Table I: The MARPOR categories used for calculating the RILE score.
3 Methods
whole text, but averages of RILE scores for 4095- token chunks of manifestos nearly perfectly corre- late with gold manifesto-level scores (Spearman's ss r > 0.99), which makes by-chunk estimation a good proxy.
3.1 Operationalization
Label aggregation As outlined above, we aim at automatically estimating positions of political parties the Left- Right scale. An approach that closely mirrors the traditional MARPOR procedure would be to automatically label the sentences in the manifestos with MARPOR categories and aggre- gate them according to Eq. 1. Unfortunately, clas- sifying the sentences is difficult, as we will show below. Reasons include the large number of labels, their uneven distribution, and the country-specific nature of manifestos.
An additional motivation to pursue this avenue is provided by the fact that it not only removes the need to classify the labels of individual statements but also saves researchers the effort identify state- ments in the first place. This is non-trivial prob- lem as, according to the MARPOR codebook, any sequence of words with a distinct meaning can be considered a statement. E.g., a sentence All well- well-meaning citizens should strive to maintain the world peace can be construed as single example of the category Peace, or all well-meaning citi- zens can be assigned its own label of Civic minded- ness. line with previous work, our aggregation- based approach assumes that statement boundaries are known, but in practice they will have to be predicted together with the labels, or the coding scheme must be simplified, e.g. by assigning a sin- gle 'majority' label to each sentence. By virtue of working with raw text spans, LITs do not have to make such compromises.
However, the predicted categories arguably do not have to perfect - may be sufficient for high- quality scaling analysis if the mistakes are uncorre- lated so that, for example, the number of sentences mistakenly classified as left-leaning neutral is close to the number of sentences mistakenly classi- fied as right-leaning neutral. One way further raise the signal-to-noise ratio is predict more hege-level labels. To compute the RILE score, we do not require specific categories, but only a three- way classification (R[ight], L[cft], O[ther]), which is much more tractable. This approach can be eas- ily mapped into other dimensions as long as there is a list of categories from MARPOR belonging to both poles of the scale.
3.2 Problem settings
We consider two settings, corresponding to two different research scenarios. In the LEAVE--NEE- COUNTRY-OUT (X-COUNTRY) setting, we train the model on all data from n 1 countries (split into training and development sets), and evaluate it one held-out country. This corresponds to the situation when manifestos from a country not yet covered by the MARPOR project, such as Indonesia, need to be analysed. This is repeated for all countries.
Direct prediction As an alternative, we can de- fine a function T + [-1,1] 1] that directly maps a text to its RILE score, and approximate it with a neural regression model. Until recently, such an ap- proach was infeasible due to the restrictions on the input length in the state-of- ttate-of-the-art embedding mod- els: 512 or 1024 tokens depending on the model size, which is not enough to analyse longer texts. However, a new generation of long-input Trans- formers (LIT')) based on lightweight variants of the self-attention mechanism increased the input limit to 4096 tokens or more (Tay et al., 2021). This still does not give us a way to compute a score for a
In the OLD-VS.-NEW (X-TIME) setting, we train the model on all data from before 2019 and evaluate it on the data from 2019-202 1021.12 This corresponds to the situation when new data from an already covered country become available.'
Another application for this setting is the analysis of man-

---

3.3 Dataset
the classification head is 2-layer MLP with the inner dimension of 1024 and tanh activation after the first layer.
We use the annotated subset of the latest release of the MARPOR dataset (version 2022a; Lehmann et al., 2022a) augmented with the separately cu- rated South American dataset (Lehmann et al.. 2022b). We excluded manifestos annotated before the year 2000 obtain a more uniform training dataset. Furthermore, to ensure comparability be- tween two approaches to cross-lingual modelling - preprocessing using machine translation and us- ing a multilingual encoder (see & 3.4 below) - we excluded languages for which no pretrained free NMT system was readily available. This leaves us with 1314 manifestos from 41 different countries in 27 different languages.
In the X-COUNTRY setting, the model was then repeatedly trained for two epochs using cross- entropy loss and the AdamW optimiser (Loshchilov and Hutter, 2019) with the learning rate of 10-5, In the X-TIME setting, the general setup is the same but the model was trained for five epochs with checkpoint selected based on the dev-set accuracy. For the MACHINE- TR ANSLATION TRACK (MT), all manifestos are translated into English, for which the best MT systems and arguably the best pre- trained encoders are available. The current MT systems, however, are still rather noisy, especially for non-WEIRD (Henrich et al., 2010) languages, which offsets the benefits of stronger base model. We use the EasyNMT toolkit 10 giving access the Opus-MT models (Tiedemann and Thottingal, 2020). cursory inspection of the translated sen- tences shows that the translation quality does vary across languages. However, even for manifestos whose source languages are difficult to translate (e.g. Georgian) the results produced by the classi- fier are still acceptable.
In the X-COUNTRY setting, the rolling test set includes all of the data, while in the X-TIME setting it is much smaller (163,714 vs. 1,062,302 state- ments in the training set, i.e. around 13%) and has a weaker geographical coverage: only 18 countries have manifestos from 2019 and later.
The data for LITs have the same train-test gen- eral splits, but sentences in them were consecu- tively concatenated into text chunks of size no more than 4095 tokens (see Section 3.1), with a RILE score computed for each chunk based on its gold MARPOR labels. Chunks of size less than 1000 tokens were discarded.?
The translated sentences are encoded using pooled representations from all all-mpnet-base-2 version of MPNet (Song et al., 2020) fine-tuned following the SBERT methodology (Reimers and Gurevych, 2019) and available on HuggingFace. The same classification head was then used as in the XTMM-NNC approach, as well as the same training parameters.
3.4 Models
The MARPOR dataset is multilingual, which raises the challenge of language transfer. The two current approaches in this case are using a multilingual encoder or machine translating all the data into the pivot language, usually English (Litschko et al., 2022; Srinivasan and Choi, 2022).
For each model, aggregate the labels across manifesto sentences and compute its RILE score according to Eq.
Direct prediction We experiment with two long- input encoder models: Longformer (Beltagy et al., 2020) and BigBird (Zaheer et al., 2020). 12 They are only available for English, and we apply them to the translated dataset. We use the embedding of
Label aggregation Here we experiment with both options. For the MULTILINUUALEENCDDER TRACK (XLM-ENC), we extract the representation of the CLS token from M-oM-RaE base (in the X-COUNTRY setting) and RoBRTaa base and large (in the X-TIME setting), 8 Throughout,
the large model.
"The code for training and evaluating the mod- els can be found at http:://gttubb.com//eeeegggn con/nacleginn/ party tetttypppsiiiinn-ccocc "https://giit httt::////gttuub..co/kkkkabbbba "nttps://huggingfaas co/ sentencee trannncee-rrrneeie formers 111-mpnet-base-v2 Preliminary experiments showed, in agreement with the results of Ceron et al. (2022), that it consistently outperforms RoBERTa in monolingual settings. "Pretrained models were downloaded from Hug- gingFace: long baseer-bbse--eee 4896 and
ifestos of smaller parties that did not win any seats in previous elections and were not included in the dataset. The converse NEW-VS. -O-DD would permit running a historical analysis of party positioning within a country. We have not addressed this scenario due to the scarcity of annotations from before 2000.
'All data available the project web page: https: /mmaniff inme fes toproject... wzb. eu/datasets.
Stattsstiss of the datasets are shown in Tables 8 (by coun- ury) and 9 (by language) in Appendix
https://hun httgss//ggggngacc....om wtt.aaccccccoccaceaaoooooooaa wtt.c///aaaaaaaaaaa allenai/
"The necessity to train 41 different models on the full datasct in the X- C-COUNTRY setting made it impractical to use
co/ google/bigbirdd roberiggrrrrrrrrrrraaaeerreeeeee moberta-basee

---

the last layer's CLS token as input to a regression head. In the training slep, each chunk receives a gold label computed from its sentences using Eq. I. The final RILE score of each manifesto is the average of regression values of its chunks. The regression head is similar to the classification head described above with the final softmax layer replaced with single node with tanh activation mapping the output into the 1,1 range. The systems are trained using MSE loss.
X-COUNTRY
X-TIME
XLM
MT
MAJ
XLM
MT
MAJ
Ace
0.46
0,47
0.10
0.54
0.48
0.10
CMP
F1
0.44
0.44
0.02
0.55
0.48
0.02
Acc
0.70
0.71
0.59
0.77
0.74
0.63
RILE
F1
0.70
0.70
0.44
0.77
0.74
0.49
Table 2: The accuracies and wlash- F1 scores of predicting all 143 MARPOR/CMP categories and RILE-specific categories (left. right, other) in the leave-one-countyy (X-COUNTRY) and old-vs.-new (X-TIME) settings using multilingual encoder (XLM- ENC) or preprocessing via machine translation (MT). MAJ is the majority-class baseline for each setting.
3.5 From regression to classification with LITs
A possible concern about the direct computation of RILE scores, as we frame the task for LITs, is that the models may fail to implicitly recreate the aeeeling-and-aggeegatiin pipeline and instead learn spurious shortcuts by observing correlations between properties of texts and their RILE scores, which will then hurt test performance.
RILE (CMP)
RILE (3-way)
XLM
0.73
0.72
MT
0.71
0.72
X-COUNTRY
0.55
LF
0.16
0.88
0.9
MT 0.84
0.88
To address this concern, we carry out an addi- tional experiment where we make the models' task more comparable to what a human political analyst would do. We train the LITs in a binned-regression setting: the range of RILE scores is split into five regions, corresponding to hard left [-1,-0.6) -0.6), cen- tre left [-0.6, -0.2), centrist 0-.., 0.2), centre right [0.2, 0.6), and hard right [0.6, 1]. The models are then trained to predict these classes instead of real-valued RILE scores using cross-entropy loss.
X-TIME
0.71
0.35
Table 3: The results (Spearman correlations) comput- ing RILE via predicting all MARPOR/CMP sentence- level categories (CMP), RILF-specific categories (3- way), or using I.IT's (BB: BigBird; LF: Longformer).
4.1 Predicting MARPOR categories
As Table 2 shows, predicting the fine fine-grained MAR- POR categories directly a very hard task, both in the X- X-COUNTRY and X-TIME settings. Our mod- els easily beat the majority-class baseline but only achieve an accuracy above 50% in the X-TIME set- ting with the XLM-ENC encoder.
3.6 Evaluation metrics
For the lagel-aggn models, first diagnose the performance of the label classifiers using the weighted macro-averaged Fl score.
We then evaluate both the lagel-aggregation and the dirrectppredcctie models on the target task of predicting RILE score. We use Spearman's 's corre- lation coefficient, which shows if our scores are monotonically related those computed from gold annotations using Eq. Additionally, we look at absolute values of errors and their directionality.
Aggregating labels into the three RILE-relevant classes makes the task predictably simpler: the baseline F1 score rises from nearly zero to 0,44/0.49 49 (Other becomes the dominant category), but so does the performance of the models, to ac- curacies and Fl scores of 0.7 and above. However, there is still ample room for improvement. Inter- estingly, while using machine translation leads consistent improvements in the X-COUNTRY set- ting, the X-TIMR setting better served with the multilingual encoder." 13
We evaluate the performance of the LIT-based classifiers in the binned-ree regression setting using accuracy and Fl score.
4 Results
The main results of the experiments are sum- marised in Tables 2 and 3. Sections 4.1 and 4.2 discuss the results while $ 4.3 provides some detail about the strengths and weaknesses of the models.
I Thee results of using XLM XLM-RoBERTa base in the X-TIME setting are as follows: CMP labels: accuracy - 00511 Fl - 0.51, r - 0.87: 3 labels: accuracy - 0.76, Fl 0.75, r 0.888.

---

X-COUNIRY
X-TIME
CL
CR
Acc
0..93
0.74
L
3
2
0
0
BigBird
F1
0.68 0.771
0.72
CL
133
135
1
Acc
0.59/0.66
0.58 0.64
C
69
708
41
Longformer
F1
0.56/0063
/
CR
0
70
28
R
1
Table Performance (on the chunk/mmniffetoo level) of long-input Transformers on the task of 5-way political- stance classification. Fl scores are macro averaged and weighted by the frequency of the gold classes.
Table 5: Confusion matrix for the party stance predicted by the Big based classifier in the X X-COUNTRY set- ting. L: left, CL: centre left, C: centrist, CR: center right, R: right.
4.2 Computing RILE scores
4.3 Error analysis
Label aggregation In agreement with our work- ing hypothesis, Table 3 shows that even noisy la- bels can be used to calculate manifesto-win wide scale values that are largely in agreement with gold val- ues. When predicting RILE via label aggregation the best results are attained by using the multilin- gual encoder, both in the X-COUNTRY and in the X-TIME setting.
4.3.1 Regression to the mean
The distributions of gold RILE scores and those predicted the X-COUNTRY setting by the best- performing label-aggregattion pipeline and the best- performing LIT are shown in Figure 1. 14 The plots make it clear that both models are very conserva- tive: predicted values cluster closer to the mean RILE score than in the gold data. BigBird is espe- cially affected by this, which take to indicate that it suffers from lack of training data: the train- ing dataset was big enough to correctly estimate the mean of the distribution but not big enough to approximate the correct dispersion.
Somewhat surprisingly, aggregating the labels, even though this leads small number of surface- level classification mistakes, does not improve the eventual RILE scores the X-COUNTRY setting (r= 0.72 from aggregated labels vs. 0.73 from all labels) and gives only modest boost in the X-TIME setting (0.9 vs. 0.88).
The predictions of the aabel-aggregation model based on XLM-ENC approximate the dispersion much better. However, the model still fails to account for the heavy right tail in the gold data and presents more symmetric picture. In terms of RILE scores, this corresponds to left skew: the model often presents right-leaning manifestos (those with positive RILE scores) as more centrist.
Long-input Transformers The performance of LIT's is vastly uneven. In the X-COUNTRY setting, both models struggle: by-chunk RILEs from Long- former are essentially uncorrelated with gold ones, while Bird's predictions show non-negligible correlation (0.55), which is still much worse than the label aggregation results. In the X-TIME setting, while Longformer's predictions are still extremely noisy (r 0.35), BigBird's s ones are comparable to what the label aggregation approach achieves in the X-COUNTRY setting (0.71). ). As we discuss be- low, however, this correlation is somewhat mislead- ing: while producing scores that are monotonically aligned with correct ones, BigBird predicts values that are very close to zero and thus differ greatly in their absolute values from the gold scores.
A more detailed picture of the relationship be- tween the gold RILE scores and those predicted by the label-aggregation model is shown, for both set- tings, in Figure 2, which also presents the density of the prediction errors. Consistently with Figure 1, the density of the X-COUNTRY error distribution has slightly heavier left tail. To characterize this behavior, we can look at the cases where the sign of the prediction is flipped, i.e. the upper-left and the lower-right quadrants of the scatterplot. While the UL quadrant is nearly empty, the LR quadrant is populated not only near the x asymptote, but also further the right. This suggests that in the cross-country and cross-lingual setting, the hardest aspect of the problem correct identification of right-wing statements across countries.
LIT-based classifiers The results of the applica- tion of the better-performing LIT, BigBird, to the task of 5-way stance classification are shown in Table Unlike RILE scores, by-chunk stance la- bels cannot be averaged, for the final prediction each manifesto is assigned its majority class. The performance of the BigBird-base based model in this setting is reasonable, with Fl scores ss 0.7.
(Thhe situation in the X-TIME setting is similar. The corre- sponding plots are presented in Appendix D.

---

Gold
XL M--E-EC with label aggregation
Sighind with chunk waraging
40-
20.
10
10-
10-
00
-035
-6.50
alo
6.0
636
100
21.0
-6.3
-3.5 1:00
21 az
-675
-225
6.00
63s
100
Figure I: The distributions of gold and predicted RILE scores the X-COUNTRY setting.
Right
Left
Other
Right
46
20
33
04
Left 8
66
26
02-
Other
9
16
75
0.5
Table 6: Confusion matrix of coarse- -rrineer labels used compute the RILE score based on all MARPOR labels (the XLM-ENC + X-COUNTRY setting). True labels are in the rows, predicted labels in the columns.
-32
nitude of errors in the X-TIME setting is consider- ably lower, with only handful of sign-flip errors. This indicates that when model has access in- country data, the estimation of political positioning becomes easier, and the identification of right-wing tendencies not a major hurdle any more.
8-
-32
The 5-way LIT-based party-stance classifier also suffers from the regression-to-the-mne as can be seen in Table 5: the centrist category is overpredicted, while two extreme categories, which are rare in the data, are never predicted correctly.
04
problem,
-35
-22
0.5 25.
05
05
Figure 2: Gold vs. predicted RILE scores and his- tograms with density contours of the prediction errors in the X-COUNTRY (top) and X-TIME (bottom) settings with XLM-ENC and label aggregation.
4.3.2 Classifier errors and scaling analysis
One of the surprising results in Tables 2 and 3 is that low accuracy of the models trying pre- dict all MARPOR labels directly does not translate into low quality respective RILE scores the X- X-COUNTRY setting. This seems to suggest that errors of the models are not random: the models rather substitute, e.g., another Left-category label for a true Left-category label than replace a label from the Left set with label from the Right set. A confusion matrix for the 3 coarse-grained labels (computed based on the fine-grainee labels pre- dicted by XI-M-ENC in the X-COUNTRY setting) shown in Table 6 demonstrates that this is indeed the case.
One of the challenges associated with right-wing labels are their differing distributions across coun- tries. While the variation in the cumulative share of left-wing labels in manifestos is bounded roughly between 0.2 and 0.3, with the same labels dominant everywhere, the variability of right-wing labels is much higher and their share is lower on average. See Figure 4 in Appendix E for details and Lachat (2018); Fielitz and Laloire (2021); Jahn (2022) for more in-depth analyses.
As the bottom panel of Figure 2 shows, the mag-

---

5 Discussion
ality they reduce using PCA; they then interpret one of the resulting axes as the left-right scale. Vafa et al. (2020) instead develop a methodology for identifying the political position of lawmakers on the poogressive-to-modere dimension with a bag fag--of-words-basee topic-modelling approach. The use of contextualised embeddings for politi- cal analysis has not yet become mainstream. Aber- crombie et al. (2019) test wide range of meth- ods, from unigram statistics to BERT-based clas- sifiers, for assigning MARPOR labels to classify debate motions from the UK parliament. Dayanik et al. (2022) use several pre-trained single-language BERT models for the task of political-statemmnn classification in five languages. Facing the same issues of label-freuuency imbalance and rare labels, they mitigate them to some degree by using the hierarchical organisation of MARPOR labels; they do not try to compute RILE scores.
Our results show that multi-lingual automatic anal- ysis of paltt positioning is at least par- tially feasible. It is possible to provide a high-level overview of the party system in new country with reasonable degree of precision, and even better results can be achieved with some amount of in- country data: the RILE scores computed using our method demonstrate a remarkably high correlation with the gold scores. Interestingly. the main obsta- cle to the success of our method seems to be not the language barrier, which is bridged well by either the off-the-shee shelf MT systems or the multilingual encoder, but the differences in the political culture across countries: the models struggle to correctly identify right-wing statements in the manifestos.
In practical terms, using long-input Transform- ers instead of sentence-level classifiers offers a way to greatly simplify the analysis and obviate the problems of subsentence identification in the input, as such models are able to make holistic judge- ments about long spans of text. In terms of perfor- mance, LITs struggle on the task of directly estimat- ing RILE, compared to lahel-aggregation models, with the best model only upproaching a reason- able level of performance. However, this must be taken with a grain of salt, since the label aggrega- tion models have the advantage of gold-statement boundaries. Furthermore, our binned-regression experiment shows that LITs are promising candi- dates for coarse-grained party positioning analysis in terms of political 'camps'. For all models, the tails of the distribution remain hard identify, with extreme categories rarely predicted correctly and centre left/centre right labels often mistaken for centrist.
Ceron et al. (2022) introduce sentence transform- (Reimers and Gurevych, 2019) into the prob- lem space and fine-tune the embedding model it- self in order to learn a politically informative dis- tance measure between manifesto texts. Ceron al. (2023) further extend this method to analyse inter- party differences with regard to major policy do- mains, such as Law and Order or Sustainability and Agriculture.
More generally, our work falls into the domain of zero-shot classification with test data coming from a country or a time period not covered by the training data. The question of whether machine translation (Schäfer et al., 2022) or multilingual encoders (Litschko et al., 2022) is better suited for cross-lingual transfer is still actively debated, and explore both options. From another perspective, the task of identifying and characterising political positions from textual data abuts larger fields of stance detection and argument mining (Küçük and Can, 2020; Reimers et al., 2019).
6 Related work
The work cal documents traditionally employs bag-of-words methods, such as those popularised by Laver et al. (2003) and Slapin and Proksch (2008). Glavaš et al. (2017) introduce distributional semantics in the left- right analysis by using multilingual word alignment in the embedding space and graph-based score- propagation algorithm. This approach is then built upon by Nanni et al. (2022).
computational analysis of politi-
Conclusion
In this paper, we have proposed the first series models that generalise the task of political-party positioning across countries and election cycles. We showed that the main challenge predicting MARPOR labels across countries and election cy- cles with high accuracy is, surprisingly, not a real barrier on the way to a highly precise multi- lingual scaling analysis. We experimented with the Standard Right-Left Scale (RILE score), which
Rheault and Cochrane (2020) adapt the word2vec methodology to the analysis of parlia- mentary speeches in a single-language setting via the use of trained party vectors, whose dimension-

---

is widely discussed in the political-science litera- ture, and demonstrated that party manifestos can be effectively characterized in these terms using state-of-the-ar multilingual modeling techniques applied to sentence-level classification with subse- quent label aggregation and that even better results can be achieved via task-specific label clustering.
best possible performance in real-world set- tings. Using LIT' removes this issue, but their performance is not competitive.
Acknowledgments
We acknowledge partial support by Deutsche Forschungsgemeinschatt (DFG) for project MARDY 2 (375875969) within priority program RATIO.
We further experimented with replacing the label-aggregation approach with long-input Trans- formers- both using regression and classification formulations - order to obviate the task of iden- tifying spans of statements from manifestos. These models demonstrate promising performance but still underperform the more traditional pipeline mimicking manual analysis.
References
Gavin Abercrombie. Federico Nanni, Riza Batista- Navarro, and Simone Paolo Ponzetto. 2019. Policy preference detection in parliamentary debate motions. In Proceedings of the 23rd Conference on Computa- tional Natural Language Learning (CoNLL), pages 249-259, Hong Kong, China. Association for Com- putational Linguistics.
Bridging the gap between long-input models and political analysis is an important avenue for fu- ture work, together with tackling other political dimensions and further widening the scope of the analysis.
Jeremy Albright. 2010. The multidimensional nature of party competition. Party Politics, 16(6):69-771,
Limitations
Pablo Barberá. 2015, Birds of the same feather tweet gether: Bayesian idcal point estimation using Twitter data. Political analysis, 23(1):76-99
The main limitations of our work are twofold, and both stem from our dependence on the cate- gorics and annotations produced by the MARPOR project:
Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document Transformer. 2rXiv:2004.051.3
1. The RILE scale that we target is computed based on the MARPOR category labels, and we do not test if our methodology ily projected other categorisation schemes. However, given the important role of the MARPOR codebook in the political-science literature and the amount of annotated data al- ready available, we hope that our work makes valuable contribution to the debate.
Ian Budge. 2013. The standard Right-Left scale. Tech- nical report, Comparative Manifesto Project.
be eas-
Tanise Ceron, Nico Blokker, and Sebastian Pad6. 2022. Optimizing text representations capture (dis)similarity between political parties. In Proceed- ings of the 26th Conference Computational Nat- ural Language Learning (CoNLL), pages 325-338. Abu Dhabi, United Arab Emirates (Hybrid). Associa- tion for Computational Linguistics.
Tanise Ceron, Dmitry Nikolaev, and Sebastian Padó. 2023. Additive manifesto decomposition: A pol- icy domain aware method for understanding party positioning. Findings of the Association for Com- putational Linguistics: ACL 2023, pages 7874-7890, Toronto, Canada. Association for Computational Lin- guistics.
2.
In label-aggregation pipeline, we are depen- dent not only on the labels themselves but also on the way they are applied manifestos: fol- lowing previous work (Dayaník et al., 2022; Ceron et al., 2022), we use the sub-sentence boundaries selected by MARPOR annotators in order to assign a single category to each statement. In the manifesto texts, sentences therefore sometimes can be associated with several labels. There are several possible ways to address this issue (e.g.. selecting 'major- ity" label for each sentence in the training data, training a multi-label classifier, or learning splits together with labels from the training set), and they need to be explored to obtain
Christopher Cochrane. 2015, Left and right: The small world of political ideas. McGill-Queen's i's University Press.
Thomas Däubler and Kenneth Benoit. 2022, Scaling hand-coded political texts to learn more about left- right policy content. Party Politics, 28(5):834 844.
Erenay Dayanik, Andre Blessing, Nico Blokker, Se- bastian Haunss, Jonas Kuhn, Gabriella Lapesa, and Sebastian Pado. 2022. Improving neural political

---

statement classification with class hicrarchical infor- mation. In Findings of the Association for Compu- national Linguistics: ACI. 2022, pages 2367-2382, Dublin, Ireland. Association for Computational Lin-
Michael Laver, Kenneth Benoit, and John Garry. 2003, Extracting policy positions from political texts using words as data. American political science review, 97(2):111-311
guistics.
Pola Lehmann, Tobias Burst, Theres Matthieß, Sven Regel, Andrea Volkens, Bernhard Weßels, and Lisa Zehnter. 2022a. The manifesto data collection. Mani- festo project (MRG/CMP/MARPOR) version 2022a.
Daniel Diermeier, Jean-François Godbout, Bei Yu, and Stefan Kaufmann. 2012. Language and ideology in Congress. British Journal of Political Science,
42(1):31-55.
Pola Lehmann, Tobias Burst, Theres Matthieß, Sven Regel, Bernhard Weßels, and Lisa Zehnter. 2022b. The manifesto data collection: South America. Ver- sion 2022a.
Lee Epstein and Jeffrey A. Segal. 2000. Measuring issue salience. American Journal of Political Science, 44(1):66-83.
Maik Fielitz and Laura Lotte Laloire, editors. 2021. Trouble the far right. Political Science (COL). Transcript Verlag.
Robert Litschko, Ivan Vulić, Simone Paolo Ponzetto. and Goran Glavaš. 2022. On cross-lingual retrieval with multilingual text encoders. Information Re- trieval Journal, 25(2):149-15)
Jan-Erik Flentje. Thomas König. and Moritz Marbach. 2017. Assessing the validity of the manifesto com- mon space scores. Electoral Studies, 47:25-35.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled
weight decay regularization. Proceedings of the 7th International Conference on Learning Represen- tations, New Orleans, 6-9 May 2019.
Matthew J Gabel and John D Huber. 2000. Putting par- ties in their place: Inferring party left-right ideologi- cal positions from party manifestos data. American Journal of Political Science, pages 103.
Martin Mölder. 2016. The validity of the RILE left- right index as a measure of party policy. Party Poli- rics, 22(1):37-48.
Goran Glavaš, Federico Nanni, and Simone Paolo cross-lingual of political texts. Proceedings of the 15th Con- ference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Pa- pers, pages 688-693, Valencia, Spain. Association for Computational Linguistics.
Ponzetto. 2017. Unsupervised cross- lingual scaling
Federico Nanni, Goran Glavaš, Ines Rehbein, Si- mone Paolo Ponzetto, and Heiner Stuckenschmidt. 2022. Political text scaling meets computational se- mantics. ACM/IMS Transactions on Data Science (TDS), 2(4):1-27.
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLP), pages 3982-3992, Hong Kong, China. Association for Com- putational Linguistics.
Zachary Greene and Diana Z O'Brien. 2016. Diverse parties. diverse agendas? Female politicians and the parliamentary party's ' role in platform formation. Eu- ropean Journal of Political Research, 55(3):435 453..
Joseph Henrich, Steven J. Heine, and Ara Norenza- yan. 2010. Most people are not WEIRD. Nature, 466(7302):29-29
Nils Reimers, Benjamin Schiller, Tilman Beck, Jo- hannes Daxenberger, Christian Stab, and Tryna Gurevych. 2019. Classification and clustering of arguments with contextualized word embeddings. In Proceedings of the 57th Anmual Meeting of the As- sociation for Computational Linguistics, pages 567- 578, Florence, Italy. Association for Computational Linguistics.
Detlef Jahn. 201 I. Conceptualizing Left and Right in comparative politics: Towards deductive approach. Party Politics, 1766)775-7755
Detlef Jahn. 2022. The changing relevance and meaning of left and right in 34 party systems from 1945 to 2020. Comparative European Politics.
Dilek Küçük and Fazli Can. 2020. Stance detection: survey, ACM Computing Surveys (CSUR), 53 37.
Ludovic Rheault and Christopher Cochrane. 2020. Word embeddings for the analysis of idcological placement in parliamentary corpora. Political Analy- sis, 28( 12-1133.
Romain Lachat. 2018 Which way from left to right? On the relation between voters' issue preferences and left-right orientation in West European democra- cics. International Political Science Review / Revue internationale de science politique. 39(4):419-435
Henning Schäfer, Ahmad Idrissi- Idrissi-Yaghir, Peter Horn, and Christoph Friedrich. 2022. Cross-language trans- fer of high-quality annotations: Combining neural machine translation with roos-linguistic span align- ment to apply NER to clinical texts in a low-resource language. In Proceedings of the 4th Climical Natural Language Processing Workshop, pages 53-62, Seat- tle, WA. Association for Computational Linguistics.
Benjamin E Lauderdale and Tom S Clark. 2014. Scal- ing politically meaningful dimensions using texts and votes. American Journal of Political Science, 58(3):554-771

---

Jonathan Slapin and Even-Oliver Proksch. 2008. A scaling model for estimating time-series party po- sitions from texts. American Journal of Political Science, 52(3)700-722.
B Names of MARPOR categories
referenced by a number in the text
103 Anti- Imperiapeia
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie- Yan Liu. 2020. MPNet: Masked and permuted pre- training for language understanding. arXiv preprint arXiv:2004.09297.
104 Military: Positive
105 Military: Negative
106 Peace
Anirudh Srinivasan and Eunsol Choi. 2022. TyDiP:
A dataset for politeness classification in nine typolog- ically diverse languages. In Findings of the Associ- ation for Computational Linguistics: EMNLP 2022, pages 3723-5738, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
107 Internationalism: Positive
201 Freedom and Human Rights
201.1 Freedom
Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang. Sebastian Ruder, and Donald Metzler. 2021. Long range arena A benchmark for efficient transformers. In International Conference on Learning Representa- tions.
201.2 Human Rights
202 Democracy General
202.1 Democracy General: Positive
203 Constiutiinalism: Positive
Jörg Tiedemann and Santhosh Thottingal. 2020, OPUS-
MT Building open translation services for the World. In Proceedings of the 22nd Annual Confer- enec of the European Association for Machine Trans- lation (EAMT), Lisbon, Portugal.
305 Political Authority
401 Free Market Economy
402 Incentives: Positive
Keyon Vafa, Suresh Naidu, and David Blei. 2020. Text- based ideal points. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 5345-5357, Online. Association for Computational Linguistics.
403 Market Regulation
404 Economic Planning
406 Protectionism: Positive
Andrea Volkens, Judith Bara, Ian Budge, Michael D. McDonald, Robin Best, and Simon Franzmann. 2013. Understanding and Validating the Righ Scale (RILE). In Mapping Policy Preferences From Texts: Statistical Solutions for Manifesto Analysts. Oxford University Press.
406.1 Anti-Growth Economy-Positive
407 Protectionism: Negative
412 Controlled Economy
413 Nationalisation
Andrea Volkens, Tobias Burst, Werner Krause, Pola Lehmann, Theres MathieB, Nicolas Merz, Sven Regel. Bernhard Weßels, and Lisa Zehnter. 2020. The Manifesto Project Dataset - Codebook. Man- ifesto Project (MRG / CMP MARPOR). Version 2020b. Wissenschaftszeetrumm Berlin für Sozial- forschung (WZB), Berlin.
414 Economic Orthodoxy
416 Anti-Growth Economy: Positive
501 Environmental Protection
502 Culture: Positive
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago On- tanon, Philip Pham, Anirudh Ravula, Qifan Wang. Li Yang, et al. 2020. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 13:172
504 Welfare State Expansion
505 Welfare State Limitation
506 Education Expansion
A Examples of the MARPOR categories
601 National Way of Life: Positive
See Table 7.
602 National Way of Life: Negative

---

Party
Text
Category
AfD
The principles of equality before the law.
Equality: Positive
CDU
We are explicitly committed NATO's 2% target.
Military: Positive Governm. and Ad- min. Efficiency
FDP
And with a state that strong because acts lean and modern instead of complacent, old-fashioned and sluggish.
SPD
There need to be alternatives to the big platforms with real opportunities for local suppliers.
Market Regulation
Grüne
We will ensure that storage and shipments are strictly monitored.
Law and Order:
Positive
Die Linke
Blocking periods and sanctions are abolished without exception.
Labour groups:
Positive
Table Translated examples of sentences from German federal election manifestos (2021) with their categories
annotated by the Comparative Manifesto Project.
603 Traditional Morality: Positive
604
Traditional Morality: Negative
605 Law and Order
605.1 Law and Order: Positive
606
Civic Mindedness: Positive
607
Multiculturalism: Positive
608
Multiculturalism: Negative
701
Labour Groups: Positive
705
Unprivileged Minority Groups
706
Non-economic Demographic Groups
C Dataset breakdown by country and by language
See Tables 8 and 9.
D Distributions of predicted RILEs in the X-TIME setting
See Figure 3.
E Cumulative share of left and right categories across countries
See Figure 4.

---

Gold
Cumulative of Right XLM ENC with label aggregation
BigBird with dhunk averaging
10
6-
6-
2.25
1.00
0.30
3.2
0.00
3.23
Figure 3: The distributions gold and predicted RILE scores the X-TIME setting.
Cumulative Left labels
103
Argentina
412
203
Armenia
402
Australia
Austria
506
401
Belgium
107
201
Bulgaria
403
Canada
105
104
Czech Republic
601
Denmark
106
Estonia
406
407
Finland
504
603
Georgia
505
Germany
Hungary
Iceland
Ireland
Japan
Latvia
Luxembourg
Mexico
Netherfands
North Macedonia
Poland
Slovekia
South Africa
Spain
Sweden
Swilzerland
Turkey
Ukraine
kingdom
United States
Uruguay
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.35
0.40
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.35
0.40
Figure 4: Cumulative shares of left-wing and right-wing labels in manifestos from different countries. See
Appendix B for the explanation of label codes.

---

Country
manifestos
# sentences
Argentina
29
10983
Armenia
22
1623
Australia
30
21683
Austria
32
39452
Belgium
43
154699
Bolívia
7718
Bulgaria
18
8945
Canada
23
28524
Chile
33988
Czech Republic
31
25986
Denmark
45
17073
Estonia
23
16524
Finland
33
22520
France
20
9347
Georgia
19
2610
Germany
35
81759
Hungary
26
45246
Iceland
34
8139
Ireland
23
30348
Israel
24
Italy
32
22091
Japan
3387
Latvia
30
2030
Luxembourg
17
30768
Mexico
48
49818
Netherlands
45
72610
New Zcaland
44
43869
North Macedonia
43
56719
Poland
30
27285
Russia
1350
Slovakia
33
25325
South Africa
24
12835
South Korea
6030
Spain
90
142878
Sweden
31
17293
Switzerland
50
20975
Turkey
23
54472
Ukraine
35
3099
United Kingdom
32
33211
United States
16262
Uruguay
16518
Table 8: Number of manifestos and number of sentences per country.

---

Language code
Language
# manifestos
# sentences
Countries
bg
Bulgarian
18
8945
Bulgaria
Catalan
18
32780
Spain
Czech
25986
Czech Republic
da
Danish
17073
Denmark
Austria, Germany, Italy, Luxembourg, Switzerland
German
123
163622
Australia, Canada, Ireland, Isracl,
en
English
177
171812
New Zealand, South Africa, United Kingdom, United States Argentina, Bolivia, Chile, Mexico. Spain, Uruguay Estonia
Spanish
174
223047
Estonian
23
16524
fi
Finnish
21313
Finland
Belgium, Canada, France, Luxembourg, Switzerland
French
52
105570
gl
Galician
6
6076
Spain
hu
Hungarian
26
45246
Hungary
hy
Armenian
22
1623
Armenia
is
Icelandic
34
8139
Iceland
it
Italian
33
21646
Italy, Switzerland
ja
Japanese
3387
Japan
Georgian
19
2610
Georgia
ko
Korean
6030
South Korea
lv
Latvian
30
2030
Latvia
mk
Macedonian
43
56719
North Macedonia
nl
Dutch
75
155807
Belgium, Netherlands
Polish
30
27285
Poland
Russian
4
1350
Russia
Slovak
33
25325
Slovakia
Swedish
18500
Finland, Sweden
tr
Turkish
54472
Turkey
uk
Ukrainian
35
3099
Ukraine
Table 9: Number manifestos and sentences per language and respective source countries.


Reviews:
[{"review_id": "gxrEpb6G7a", "reviewer": "Reviewer_MK6B", "summary": "", "questions": "I would appreciate if you could briefly answer my question under \"Reasons To Reject\", thanks!", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful comments. \n\nThe exact meaning of “scaling” varies widely across studies. E.g., Rheault and Cochrane (2020) project party embeddings learned using word2vec from speeches from British, American, and Canadian legislative assemblies on two principal components and then discuss the resulting 2D patterns. They interpret one of the resulting dimensions as the left–right scale and report its correlation with RILE values assigned to respective parties; the correlations are 0.63 (US), 0.68 (UK), and 0.77 (Canada). These numbers are incomparable to ours because R&C are not trying to assign any numbers to manifestos, while RILE values are not defined for parliamentary speeches. Glavas et al. and Nanni et al. also target parliamentary speeches, with both studies using different projection axes/scales. \n\nThe biggest methodological overlap with our work is that by Dayanik et al. (2022), who assign categorical labels to political statements from manifestos. However, there are several differences in setup: they do not compute RILE values based on this analysis; they cover texts in only 5 languages; they randomly split train/test data within languages (i.e. do X-time but not exactly) and use five different single-language BERT models instead of a multilingual setting. Thus, it is possible to compare the performance of our label classifiers (F1=0.55 X-time) with their models (F1 between 0.35 and 0.45), but it is unclear whether the comparison makes sense.\n\nIn sum, we believe that a quantitative comparison of our results to prior work is not well defined, and consequently we did not address it in the draft. We will be happy to provide more background detail in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- (+) Overall solid work using state-of-the-art models for a central task in CSS / poltext\n- (+) Novel approach when compared with previous approaches (i.e., multilingual RILE estimation as opposed to multilingual scaling)\n- (+) Thorough evaluation", "reasons_to_reject": "- (-) Limited/missing comparison with previous work. My only problem with this paper is its limited experimental comparison with previous work. The authors mention a few previous multilingual approaches for multilingual political text scaling (Glavas et al. 2017; Nanni et al., 2022; Rheault and Cochrane 2020): how good are your results in relation to theirs? I understand that their problem setting is slightly different than yours (e.g., vanilla scaling vs. predicting RILE, different evaluation measures), but at the very minimum, there should be a discussion of why you cannot compare (i.e., are RILE scores not comparable with scaling ones? If yes, why?).", "questions_to_authors": "I would appreciate if you could briefly answer my question under \"Reasons To Reject\", thanks!", "paper_topic_and_main_contributions": "This paper explores various methods for estimating RILE scores in a multilingual setting, including aggregation of predicted MARPOR category labels and direct estimation using transformer-based regression models. The methods are evaluated experimentally in both a diachronic and \"country-centric\" setting.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Previous work on transformer-based prediction of policy preferences using the categories from the Manifesto project (these should be the same as your MARPOR categories): https://aclanthology.org/K19-1024.pdf"}}, {"review_id": "wrBQF1kxFI", "reviewer": "Reviewer_VGTx", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments and we largely concur with the highlighted limitations of our work. However, in our point of view, they partly stem from the problem statement, which we still find practically useful. \n\nOur method provides generally reliable, if not very fine-grained, estimates of dominant political tendencies on the by-party and by-election basis, which is relevant for the analysis of short-term policy and public-opinion shifts between election cycles or the study of the influence of extraneous factors, such as economics, climate change, or large-scale international events, on the political landscape. We regard it as an open question whether social scientists will use the proposed approach given that they previously have not had access to this kind of tool, and will endeavour to publicise the tool in that community.\n\nAs for capturing of parties’ stances on particular topics, this is an important area of research, and there is recent work (such as Ceron et al. ACL Findings 2023) that tries to tackle it directly without providing whole-manifesto RILE estimates. However, the ostensibly simplistic question of analysing political texts in terms of left—right scaling has proven to be of immense interest to political scientists (cf. the overview in the monograph Left and Right: The Small World of Political Ideas by Christopher Cochrane), and we are confident that they will welcome computational developments in this area.\n\nFinally, regarding the fact that our model often fails to predict radical tendencies in manifestos, we agree that this is a limitation, but the main reason for it is the fact that texts exhibiting such tendencies are very infrequent. Investigating methods to improve their identification for the benefit of the scholars who study extremism is an interesting area for future work. We will add this to the draft.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper works with an interesting and relevant dataset and compares different approaches and technologies\n* The experimental setup is well described and the paper presents an insightful analysis of the errors that occurred\n* The paper can provide a starting point for interesting discussions and future work", "reasons_to_reject": "I do believe that from a social science perspective, the practical relevance of the results is rather low because of two main shortcomings:\n* The approach was only reliable in generating manifesto-level classifications on the more high-level RILE-scale. The approach does not provide insights into the specific stances of a party in different areas or an explanation for the final assessment. Therefore, I would assume that social scientists would benefit relatively little from this very high-level classification, partially also because the approach does not provide any explanations for the assessment and scientists would therefore would be unable to directly assess the reliability of the assessment without reading the whole manifesto.\n* As the analysis of the results shows, the performance of the classifier is particularly weak at the edges of the left-right-spectrum, because it tends towards the middle. Arguably, the area around the edges is particularly interesting for many political analyses.\n\nThat said, I do believe these (at least partially) negative results are valuable because the authors present such a clear analysis of the errors and the task at hand is difficult and the paper can provide a great starting point for future work and can spark a discussion about the task in the EMNLP community.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper presents a novel approach for classifying party manifestos based on the MARPOR framework and RILE categories. Two different approaches are used, a label aggregation approach that makes the final assessment based on the aggregation of statement-level labels, and a long-input transformer approach, that provides a direct classification based on the whole manifesto.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "bODKnQvlp0", "reviewer": "Reviewer_gyFa", "summary": "", "questions": "Could you shortly discuss cases in which your approach can be useful?\nDoes the red contour in the RILE scores of figure 2 refer to the Gold or the predicted RILE scores? \nDo you think that using automatically extracted sentences instead of the quasi-sentences used by the CPM would change the results significantly?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments. We disagree that our problem statement is unimportant because “in many countries parties' orientation is known”, for at least two reasons. First, this information may be inaccessible in other countries or across longer time spans. Second, pre-theoretical assessments of party positions are typically specific to individual countries, but party comparison across countries is a relevant field of investigation that ideally requires a country-agnostic operationalisation of such scales. The aim of the MARPOR project is to make this kind of information universally available, but – as we mention in the paper – its coverage is incomplete both in terms of countries and election cycles. This motivates our use of NLP for generalised analysis.\n\nAs for capturing of parties’ stances on particular topics, this is an interesting research question that is largely orthogonal to our focus on left–right scaling, which, we would like to stress, is accorded a lot of value by the political-science community (cf. https://doi.org/10.1177/13540688211026076). There is recent work that tries to compare parties based on their policy-domain-specific views (e.g. Ceron et al. ACL Findings 2023), and we consider it complementary to ours as it does not provide whole-manifesto RILE estimates.\n\n“Could you shortly discuss cases in which your approach can be useful?”\n\nExamples include the analysis of short-term policy and public-opinion shifts between election cycles (where we reduce the time lag between the release of the manifestos and their analysis) and the analysis of the influence of extraneous factors, such as economics, climate change, or large-scale international events, on the political landscape. \n\n“Does the red contour in the RILE scores of figure 2 refer to the Gold or the predicted RILE scores?”\n\nThe red contour is the density estimate corresponding to the histograms of prediction errors in two different experimental settings. We will make it clearer in the caption. \n\n“Do you think that using automatically extracted sentences instead of the quasi-sentences used by the CPM would change the results significantly?”\n\nOur general guess would be that it would not because there is no evident reason why any of the strategies for dealing with raw sentences (only using the first-statement label or the majority label, or learning the splits) should bias the method in either direction, such that the eventual RILE predictions become skewed. However, to answer this question rigorously we need to make estimates vis-à-vis different strategies.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The study is interesting, and it gives some promising results using long-input Transformer models on the label aggregation, while annotating the quasi-sentences with the many labels manually annotated in the CPM is, not surprisingly,  difficult. All steps of the experiments are well described, and Spearman's correlation coefficient is used for relating their scores to the scores from the gold dataset, showing that on the cross-country and multi-lingual setting is difficult to individuate right-wing statements.", "reasons_to_reject": "The utility of predicting  a party position as Left/Right/Other without considering on which specific policy issues the party has left/centre/right positions can be in a few cases interesting, however in many countries parties' orientation is known. I suggest the authors to motivate further their work. More interesting is to have a party’s positioning w.r.t. specific issues, which is the motivation behind the CPM’s manual annotations.", "questions_to_authors": "Could you shortly discuss cases in which your approach can be useful?\nDoes the red contour in the RILE scores of figure 2 refer to the Gold or the predicted RILE scores? \nDo you think that using automatically extracted sentences instead of the quasi-sentences used by the CPM would change the results significantly?", "paper_topic_and_main_contributions": "The paper describes NLP experiments for automatic scaling analysis (party positioning on the RILE scale proposed by the CMP, the Comparative Manifesto Project,) of party manifestos using label aggregation (Left/Right/Other) and long-input Transformer-based models which make the prediction from raw text. The experiments are run on 2 settings, leave-one-country-out and old-vs-new time.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "239 evaluate it one - -> evaluate it on"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper tackles two tasks: predicting the 143 different categories of the ManIfesto Research on Political Representation project (MARPOR) and RILE values, Standard Right–Left Scale that uses pre-defined right-wing and left-wing categories from the MARPOR project and a deterministic formula to get a RILE score between -1 and 1. They highlight that there needs to be more methods that deal with long texts and work robustly across domains and languages. 

Reviewers generally found the experimental set-up to be very sound. 

Reviewers had a few questions about the motivations and impact of the substantive task (the predicting MARPOR categories and RILE scores). However, the authors wrote a pretty convincing rebuttal that the tasks are “relevant for the analysis of short-term policy and public-opinion shifts between election cycles or the study of the influence of extraneous factors, such as economics, climate change, or large-scale international events, on the political landscape” and “pre-theoretical assessments of party positions are typically specific to individual countries, but party comparison across countries is a relevant field of investigation that ideally requires a country-agnostic operationalisation of such scales.” These do seem to be well-established tasks that social scientists would benefit from having improved predictive accuracy. 

Overall, it seems this is a sound and exciting paper that spans *both* the NLP and social science communities. I think both communities could potentially benefit from this paper.
Decision: Accept-Main