Introduction
The polarization of actors' expressed preferences is a fundamental concern for studies of legislatures, court systems, and international politics BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 . Because preferences are unobservable, scholars must look for signals in the empirical world. Recent progress has been made in parliamentary and court settings through the employment of textual data BIBREF4 and votes and texts in tandem BIBREF5 , BIBREF6 . Many of these advances rely on spatial, scaling, and item-response type models that are intuitive for settings where a small number of parties or ideological divisions influence outcomes. This is less intuitive for the study of state preferences, because international relations is marked by multiple dimensions that span ideological, economic, and security concerns, among others BIBREF0 .
This paper introduces a new approach to estimate preference polarization in multidimensional settings using texts and votes. First, a distributional representation of textual data is utilized to retain qualities of human speech that are otherwise discarded by a bag-of-words approach. Second, community detection in multiplex networks is used to uncover preference affinity blocs across multiple layers of votes and speeches. Just as scaling and spatial models attempt to explain variance along one or a few reduced dimensions, our approach identifies densely connected communities based on preference similarity that are important for explaining variations in observed outcomes.
We illustrate the utility of this approach with an empirical test of a core hypothesis in International Relations (IR): militarized conflict is less likely between states with more similar preferences BIBREF7 . Specifically, we extend a recently published network model of conflict onset. BIBREF8 utilize temporal exponential random graph models to infer the relationship between conflict onset and a battery of predictors, one of which is affinity communities located via spectral clustering on a graph of UN votes. This covariate provides a natural comparison to examine whether our proposed clustering approach can improve our ability to model conflict onset, as measured by out-of-sample predictive accuracy. We find that multiplex clusters based on country speeches and votes – which we refer to as affinity blocs – outperform clusters based on votes or speeches alone. The proposed framework enables IR scholars to better explain behavioral outcomes in international politics, and will be of use to any scholar interested in the measurement of preference polarization in multidimensional settings.
Political Polarization: Measurements and Models
Polarization in IR is defined as “the degree to which the foreign policies of nations within a single cluster are similar to each other, and the degree to which the foreign policies of nations in different clusters are dissimilar" BIBREF9 . Therefore, operationalizing a concept of preference polarization broadly involves two steps: an approach to estimate preferences from available data on states' observable behavior; and a method of detecting distinct communities of nations, such that nations belonging to the same community share similar preferences, and nations belonging to different communities have dissimilar preferences.
UN votes and speeches
The most widely used source for deriving preferences in IR is UN roll call data BIBREF10 . Voting behavior represent a valuable source of revealed preference information, comparable across states and over time. However, UN roll call votes tend to be a weak signal of underlying preferences in cases where states vote for ceremonial purposes, are constrained by agenda-setting power dynamics, or vote as cohorts to maximize their impact within the UN, such as with regional blocs BIBREF11 .
Similar limitations exist in the study of polarization in national legislatures where actors' votes seldom diverge from party lines. In response, an emerging literature turns to actors' speeches to better capture expressed positions and to measure polarization of these positions BIBREF5 , BIBREF6 . The employment of text data for the measurement of state preferences in world politics is intuitive, because outcomes are a function of multiple issue dimensions, such as topics ranging from human rights to nuclear proliferation policy. In particular, states' annual addresses in the UN General Debate (GD) provide a valuable source of data on state preferences. Governments use their annual GD speeches to discuss their positions on the issues in international politics they consider most important. As states face few institutional constraints during these speeches, they can express their positions on a wider range of issues compared to votes on agenda-set items.
An example of this difference between votes and speeches can be seen in the case of Greece and Turkey in 1974. Both countries were NATO members, however, Turkey's invasion of Cyprus in July of 1974 led to heightened tension and hostilities between the two nations. This included Turkish and Greek fighter jets engaging in a dogfight which resulted in the death of a Turkish pilot. Yet, the ideal points of Greece and Turkey based on UN votes that year were the most similar among NATO member states (0.68 and 0.42, respectively). While their votes indicate that they have broadly similar foreign policy preferences and provide useful signals of membership on an alliance dimension (e.g. they are spatially distant from Warsaw Pact members in 1974), they fail to reflect the significant tension between the countries. In contrast, their speeches in the 1974 General Debate clearly reveal the hostility between the two nations. Both Greece and Turkey discussed the Cyprus invasion at length in their speeches, with each blaming the crisis on the other.
We draw on a recently released corpus of state speeches delivered during the annual UN General Debate that provides the first dataset of textual output from states that is recorded at regular time-series intervals and includes a sample of all countries that deliver speeches BIBREF11 . There are limitations to both votes and speeches in the UN in deriving estimates of states' underlying preferences. However, it is not controversial to suggest that state speeches can valuably complement roll call data, and the use of speeches and votes together can reveal useful preference information beyond that contained in states' voting behavior or GD speeches alone. The question, rather, is how best to represent these texts and how best to theoretically model these data in tandem.
Word embeddings
In order to use texts together with votes to estimate preference polarization, we first consider how to better exploit the information contained in textual data. To do this, we introduce a new representation of textual data which more adequately captures dynamics of human language, namely unsupervised learned word embeddings. In the broader natural language processing (NLP) literature, there has been a surge of research devoted to the development of distributional representations of speech which retain syntactical language qualities in ways that the bag-of-words (BOW) approach typically used in political text analysis research is not equipped to retain. The hypothesis claims that words that occur in similar contexts tend to have similar meanings BIBREF12 . When operationalized, the unique intuition is that similar words and phrases, such as “atomic, weapons” and “nuclear, warheads” are found in relatively proximate vector space locations. Although the BOW performs surprisingly well, this example has no features in common, and a BOW representation would assign low similarity scores or high distances.
When results are projected onto a two dimensional space, language relationships surface, such as the clustering of synonyms, antonyms, scales (e.g. democracy to authoritarianism), hyponym-hypernyms (e.g. democracy is a type of regime), co-hyponyms (e.g. atom bombs and ballistic missiles are types of weapons), and groups of words which tend to appear in similar contexts like diplomat, envoy, and embassy. Mikolov and collaborators introduce an evaluation scheme based on word analogies that examines dimensions of difference in vector space BIBREF13 , BIBREF14 . They originally reached the surprising conclusion that simple vector addition and subtraction uncovers interesting linear substructures of human language, famously that INLINEFORM0 .
To locate vector space representations of our corpus, we utilize the Stanford NLP group's Global Vectors for Word Representation (GloVe) unsupervised learning algorithm BIBREF15 . GloVe is a popular log bilinear, weighted least squares model that trains on global word-word co-occurence counts to make efficient use of the corpus statistics. Because it factorizes a word-context co-occurrence matrix, it is closer to traditional count methods like latent semantic analysis or principle component analysis. Here, we present three analogical examples from the located embeddings: DISPLAYFORM0
where each INLINEFORM0 describes a vector space location of the given feature, and the cosine similarity between each vector space location is added or subtracted to find the closest vector offsets (with cosine similarity printed underneath). These analogies are interpreted, for example, as INLINEFORM1 is to INLINEFORM2 as INLINEFORM3 is to INLINEFORM4 . These examples appear to encode relations of cause-effect, threats/harm, and location/geography, respectively. As found in the wider NLP literature, the implication is that these vector space models are surprisingly effective at capturing different lexical relations, despite the lack of supervision.
To measure position similarities, we apply a new document-level distance measure to the embeddings: the (relaxed) Word Mover's Distance (RWMD) BIBREF17 . RWMD is described in greater detail in the Supplementary Materials, but in short, this measures the cumulative distance required to transform one state's speech point cloud into that of another state, ensuring that differences do not simply reflect the use of different words. States employ varied language and lexical patterns to describe similar topics. For example, if state A says “nuclear weapons are bad," and state B says “atom bombs are terrible," the only feature in common is the term “are," which leads to near-orthogonality in their BOW vectors and low similarity scores. If a third state C says “atom bombs are good," then B and C would exhibit the highest cosine similarity of the three, despite having the opposite expressed policy positions. Word embeddings and term-document matrices are located for each year in the corpus, 1970-2014, and state dyad RWMD distances are calculated, converted to similarity scores, and stored in an INLINEFORM1 matrix INLINEFORM2 for each year.
For texts to be considered as a useful complement to roll call data, we should see differences in the positions expressed in speeches versus votes. This would indicate that the two sources reveal different preference information and that using one over the other risks overlooking available preference signals. Further, for the proposed word embeddings approach to be considered useful, it should provide greater insight into intra-bloc position variation beyond what is available in the BOW, because intra-bloc variation is an important component of the definition of polarization used here, as well as in much of the recent text versus votes literature. The case of NATO and Warsaw Pact members is presented in [fig:diffs]Figure 1, which plots states' ideal points from BIBREF21 , BOW cosine similarities, and RWMD similarities, with the latter two scaled between INLINEFORM0 .
Visual assessment of intra-bloc similarities indicates that the RWMDs based on word embeddings yield higher variations in expressed positions compared to the cosine similarities between members' BOW vectors. This provides preliminary confirmation that the suggested word embeddings approach captures more interesting variation in the preferences of states compared to the BOW. Furthermore, while the ideal points clearly reflect a lessening of Cold War tensions, the RWMD similarities appear to detect greater intra-bloc position variations. Therefore, both appear to provide different and potentially valuable information for the estimation of state preferences.
Multiplex networks and community detection
Having outlined a novel approach to estimate states' preference similarity, the second step is to detect distinct communities of states. The network science literature on community detection is especially well-suited for this task. One common measure is modularity, a community detection heuristic that partitions a network such that the total number of intra-community edges is maximized relative to a baseline expectation from a null model (i.e. a random graph) BIBREF22 . The intuition is that a community should have more or stronger ties among the actors within the community compared to ties with actors in other communities. This common approach, however, can be misleading in dense networks, such as vote and speech similarity. Indeed, BIBREF22 use UN voting data to illustrate the challenges related to network clustering on data with high levels of agreement between observations.
To overcome this issue with density and to exploit the information found in votes and speeches in tandem, we instead turn to multiplex network community detection. Multilayer graphs consist of more than one layer and permit the search for communities across graph levels. Community detection methods for multilayer graphs, though, are still in their infancy and most current approaches posit the same community structure at different levels of the multigraph. As [fig:diffs]Figure 1 shows, however, votes and speeches appear to exhibit rather heterogeneous structures.
A recently proposed solution for this task is the Multilayer Extraction procedure BIBREF27 . The algorithm identifies densely connected vertex-layers in multilayer networks through a significance-based score that quantifies the connectivity of an observed vertex-layer set by comparison with a multilayer fixed degree random graph model. For our analysis, the clusters from voting data comprise one layer and the text-based clusters comprise the second layer of the graph for each year. The Supplementary Materials describes the data manipulation steps in detail, but in short, we follow Pauls and Cranmer ( BIBREF8 ) in performing 5-nearest neighbor clustering on the matrix of state speech similarities to find candidates for affinity communities and then assign ties above thresholds. These text-based clusters and their vote-based clusters are then used as two layers of a multilayer network, and the Multilayer Extraction algorithm is used to detect affinity blocs across the two layers. This process is illustrated in [fig:plots]Figure 2, which displays (a) the vote and text similarity matrices, (b) the single layer vote and speech clusters, and (c) the multiplex affinity blocs located across both layers.
Empirical Application: Affinity Blocs and Conflict Onset
A recently published network conflict onset model provides an ideal test for our proposed multilayer affinity blocs. BIBREF8 use spectral clustering to identify densely connected affinity communities based on UN roll call data and employ these clusters in temporal exponential random graph models (TERGMs) to infer the relationship between violent conflict onset and affinity communities in the UNGA, among other covariates of interest. The outcome network of interest is constructed from conflict onset data from the Correlates of War (COW) project's Militarized Interstate Dispute (MID) dataset (v4.1) BIBREF30 . The time period for the model we replicate spans 1965-2000, whereas our corpus of speeches covers 1970-2014. We limit our extension to the years of overlap, 1970-2000.
Our inferential analysis first replicates their full conflict onset model BIBREF8 , which we successfully do. This same model is estimated over the constrained time range of our analysis to ensure that significance and direction of the coefficients on the covariates do not substantially differ. We find that the signs remain the same and the coefficients do not change dramatically. These results are reported in [table:tergm]Table 1 as Model 1 and 2, respectively. We then use speech and multiplex blocs as substitutes for the original vote clusters and report these as Model 3 and 4, respectively.
The results of Model 3 in [table:tergm]Table 1 indicate that the text-based clusters exhibit a noticeably larger effect of preference similarity on conflict avoidance compared to the vote-based clusters. Interestingly, the significance of joint democracy falls off, as well as the effects of the security, trade, and economic dependency networks. This implies that membership in text-based affinity communities is associated with a substantial decrease in the likelihood of violent conflict onset and is a much larger effect compared to other covariates of interest. This stark difference between the votes- and speeches-cluster coefficients provides further indication of underlying heterogeneity in the network graphs. Both indicate that membership in affinity communities is associated with a decrease in the likelihood of conflict onset but appear to capture different manifestations of latent preferences.
The multiplex model displays coefficients closer to Models 1 and 2. The multiplex bloc indicates that membership in affinity communities as located across vote and speech graphs is associated with a decrease in the likelihood that a given pair of states will engage in armed conflict. To increase confidence in these results, however, we follow BIBREF8 in the assessment of out-of-sample predictive accuracy by training models on five year windows and then assessing predictions on the next year. The areas under the precision recall curves are then summed over the entire date range. The predictive capability of Model 3 outperforms their paper's baseline model with no preference networks but underperforms their date-adjusted model as measured by area under the precision recall curve (0.081 vs. 0.959). The multiplex model, however, outperforms their baseline paper model (with and without clusters), the date-adjusted model, and our textual extension model. The multiplex model exhibits a 20.5% increase in area under the precision recall curve compared to the original date-adjusted model (1.156 vs. 0.959). Constructing affinity blocs based on texts and votes in tandem thus leads to a more substantively intuitive model, as well as increased predictive performance. Although this is a relatively large gain in predictive accuracy, these substantively small quantities confirm that the prediction of violent conflict onset remains an enduring challenge for scholars of IR. Nonetheless, the ability to exploit revealed preference information in speeches and votes in tandem appears to promise fruitful potential gains in terms of methodological capability and theoretical soundness.
Conclusion
This paper introduces a novel approach to estimate preference polarization in multidimensional settings using votes and textual data, based on developments in the natural language processing and network science literatures. The approach helps to better exploit information found in textual data, and to locate dense clusters in complex and multilayered networks in ways that were previously not computationally possible. Drawing on a new dataset of state speeches in the UN General Debate, together with voting data from the UNGA, these tools were employed to better estimate revealed state preferences in international politics and to locate preference affinity blocs which exist across multiple layers of speech and roll call networks.
It is perhaps worth noting that these sources are relatively weak signals of true state preferences and propensity to engage in armed conflict. However, these methods significantly improve our ability to identify meaningful patterns of preference similarity amongst the noise. Furthermore, the approach can assist any political study that seeks to measure position similarities from textual data and detect dense clusters of affinity or antagonism across multiple relational datasets. These might include social media actors who operate across multiple platforms, as well as contexts like legislatures where complex relations exist across votes, speeches, committee memberships, and others. Therefore, the approach presented in this paper will be useful to all scholars broadly seeking to measure political preferences and polarization in multidimensional contexts.
Supplementary Materials
The following material complements the analysis in the main text. To our knowledge, this paper presents the first political science application of vector space representations of textual data, the Word Mover's Distance, and the Multilayer Extraction procedure. The Supplementary Materials are structured as follows. First, the data are discussed and applications which use a bag-of-words treatment are presented. Second, the vector space approach and distance measure are introduced and discussed. Finally, the network clustering approach and model evaluations are presented.
Data and Measurements
We draw on the newly released UN General Debate Corpus BIBREF11 which contains every country statement in the UN General Debate between 1970 and 2014. The General Debate (GD) takes place every September at the start of each new session of the UN General Assembly (UNGA). It provides all member states with the opportunity to address the UNGA and to present their perspective on key issues in world politics. Governments use their GD statements to put on the record their position on events that have occurred during the past year and on longer-term underlying issues in world politics related to issues such as conflict, terrorism, development, human rights, and climate change.
A principal difference between GD statements and UNGA voting is that the GD statements are not institutionally connected to decision-making in the UN. As a result, governments are free to discuss the issues they consider to be of greatest importance in world politics, regardless of whether an issue is on the formal agenda of the UNGA. Therefore, as BIBREF32 notes, the General Debate acts “as a barometer of international opinion on important issues, even those not on the agenda for that particular session.” In providing information about states' preferences on world politics, the GD provides a valuable data source for measuring polarization in International Relations. In addition to being the one major forum where states present their views on international politics free from external constraints, the fact that it takes place annually and includes all UN member states enables comparison over time and across countries. Readers are encouraged to consult BIBREF11 for a comprehensive introduction to the corpus.
As mentioned in the main text, we discuss an example where disagreement is obvious in states' GD speeches but less obvious in their voting behavior. Consider the following brief excerpts from the GD speeches of Greece and Turkey in 1974.
Greece: On 15 July a coup, condemned by all of us, was staged to overthrow Archbishop Makarios, the legitimate, elected President of the Republic. This coup was not directed against the Turkish Cypriot community of the island... During the fighting while the coup was in progress, not a single Turkish Cypriot was killed or injured. Yet five days later, large Turkish invasion forces were landing in Cyprus and the Turkish Air Force was launching indiscriminate attacks against unarmed civilians, under the flimsy pretext of protecting the Turkish Cypriot minority on the island, which, I repeat, had not been harmed in any way... Two hours later, the Turkish troops were on the move again, sowing death and destruction, killing United Nations troops, bombing hospitals and schools. Repeated cease-fire calls by the Security Council went unheeded. Turkey even ignored the ceasefire proclaimed by its own Prime Minister on 16 August 1974.
Turkey: Turkey has constantly had to face faits accomplis of increasingly serious scope, particularly since 1963. The most recent and the most serious of these faits accomplis was, as we all know, that of 15 July last: a foreign Power undertook a coup d'etat which it had long been fomenting and the purpose of which was to annex the island... The coup d'etat of 15 July was directed precisely against the Turkish community and was directly aimed at the annexation of the island to Greece... I have not, however, finished correcting all the false allegations and baseless charges made by my colleague. I reserve the right to do so when we speak on this matter before the General Assembly. My Greek colleague's speech, unfortunately, shows the nature of the atmosphere in which the debate will take place on the future of the two communities, Turkish and Greek, in the island.
The two representatives are outlining their positions on the controversy related to the Turkish invasion of Cyprus. Expressed disagreement on this topic is clearly present in the speeches, but as mentioned in the main paper, the two countries' voting ideal points for that year at the most similar amongst all NATO members. A further example is illustrated in the speeches and voting habits of India and Pakistan in 1999, the year the two countries went to war (the Kargil War). Consider the following excerpts from their General Debate statements that year:
Pakistan: The Kargil crisis was a manifestation of the deeper malaise spawned by the unresolved Kashmir problem and India's escalating repression of the Kashmiri people. India launched a massive military operation in Kargil and threatened a wider conflict by mobilizing its armed forces all along the Pakistan-India international border. Pakistan acted with restraint... India's repression in Jammu and Kashmir has killed thousands of Kashmiris, forced hundreds of thousands into exile, led to three wars between Pakistan and India and consigned the two countries to a relationship of endemic conflict and mistrust.
India: Premeditated aggression by regular forces was committed against India. Not simply was the Lahore Declaration violated, but so was the Simla Agreement, which had prevented conflict for more than a quarter of a century. In self-defence, yet with the utmost restraint, India took all necessary and appropriate steps to evict the aggressor forces from its territory.... We have been greatly disappointed by this compulsive hostility of Pakistan, because it is an aberration in our region today, where all the other South Asian Association for Regional Cooperation (SAARC) countries are at peace with each other, and are trying, bilaterally and through the SAARC mechanisms, to tackle together the great challenge of development.
Tensions are clearly present in the textual data of the respective countries. That same year, however, India and Pakistan casted very similar votes in the UN, with ideal points of -0.797 and -0.739, respectively. Therefore, both sources of data appear to provide useful signals of different aspects of underlying state preferences.
Before proceeding to the description of our word embeddings approach, it is useful to first explore the corpus through commonly used measures of disagreement, namely Wordscore and Euclidean distance. Further, the network polarization measure known as modularity is also used to explore the levels of polarization exhibited in roll call data versus speeches. First, a bag-of-words (BOW) representation of the speeches is obtained through tokenization, stemming, removal of stop words, conversion to lower case, and the removal of punctuation, symbols, and numbers. We keep only the features which appear at least 5 times in 3 documents and apply term frequency-inverse document frequency (TF-IDF) weighting to the matrix. This yields a document term frequency matrix for each year.
The most frequently used text scaling method in political science is Wordscore BIBREF33 . Word frequencies in the document are used to classify the document into one of two categories. With Wordscore, the learning is supervised using training documents that are known to belong to each of the two categories so that the chosen dimension is substantively defined by the choice of training documents. Here we apply the approach to UNGD statements to calculate levels of disagreement. In Figure FIGREF19 , Wordscore detects in both cases a decline in polarization towards the end of the Cold War and a modest increase after the end of the Cold War. This broadly aligns with expectations in international relations research.
Second, we compare this Wordscore disagreement to the Euclidean distances between the US and Russia and the US and China and present these for each session over time. These are calculated using: DISPLAYFORM0
where INLINEFORM0 and INLINEFORM1 are TF-IDF vectors for the individual states. The results of this measure of disagreement are presented in Figure FIGREF20 . The distances closely align with the Wordscore results for US-Russia and US-China dyads. Polarization broadly decreases towards the end of the Cold War and then increases afterwards.
Although these measures provide useful aggregate information, they offer little by way of high-resolution information on preference divergence. Further, unlike at the national legislative level, polarity in IR is more than simple disagreement. Polarization refers to “the degree to which the foreign policies of nations within a single cluster are similar to each other, and the degree to which the foreign policies of nations in different clusters are dissimilar" BIBREF9 . Continuing with the BOW representation of the speeches, it is instructive to explore preference polarization using a more sophisticated measure which captures this theoretical definition.
Modularity is a recently developed community detection heuristic in the network sciences which attempts to partition a network such that the total number of intra-community edges is optimized relative to a baseline expectation from an appropriate null model (i.e. a random graph) BIBREF22 . This aligns with our theoretical definition via its comparison of within-group ties with other subgroup ties. The intuition is that a community should have more or stronger ties among the actors within the community compared to ties with actors in other communities. Modularity is measured between [0,1] where higher levels of modularity indicate stronger divisions in a given network, and therefore higher levels of polarization. The algorithm can be expressed as: DISPLAYFORM0
where INLINEFORM0 is the total weight of the edges in the network, INLINEFORM1 is the weighted degree of the INLINEFORM2 th node, INLINEFORM3 is the community to which INLINEFORM4 belongs, and INLINEFORM5 = 1 if INLINEFORM6 and INLINEFORM7 belong to the same community, and 0 otherwise BIBREF34 . Since modularity optimization is an NP-complete problem, we utilize the greedy variant which is implemented through the igraph package.
This approach has enjoyed widespread network science employment by applied mathematicians and physicists BIBREF35 , and has surfaced in political science studies of international trade BIBREF36 and international law BIBREF37 . As a direct measure of polarization, this approach has been used to measure roll call polarization in the US Congress BIBREF23 , BIBREF24 , BIBREF25 , as well as the UNGA BIBREF22 . To bridge from roll call data to our speeches, it is instructive to replicate the findings of BIBREF22 using ideal point data based on votes and to compare this to a simple weighted graph based on the cosine similarity of the speeches between states over time. Cosine similarity is well established in the natural language processing literature but has also received recent attention in political science BIBREF38 . This provides a measure of similarity between two vectors of an inner product space which measures the cosine of the angle between them and is expressed as: DISPLAYFORM0
where INLINEFORM0 and INLINEFORM1 are vectors of attributes (i.e. term frequency vectors), and INLINEFORM2 and INLINEFORM3 are components of vectors INLINEFORM4 and INLINEFORM5 (i.e. term frequencies), respectively. In the case of text, similarity scores are bounded to positive space [0,1]. Finally, we normalize using: DISPLAYFORM0
The result is an INLINEFORM0 adjacency matrix INLINEFORM1 where INLINEFORM2 contains the normalized textual similarity score (i.e. weighted edge) for each pair of states INLINEFORM3 and INLINEFORM4 .
We replicate the results of BIBREF22 using ideal point data, which is represented as the gray line. The blue line represents the modularity calculated on the speeches. Information on the roll call data can be found in BIBREF21 and BIBREF10 . Our aim here is to establish a preliminary idea of how polarization changes over time in these two sources of data. To the extent that the patterns diverge, both sources of data provide potentially different information about underlying preferences.
In line with the simple measures presented above, both networks in Figure FIGREF24 exhibit generally decreasing levels of polarization towards the end of the Cold War and then subsequent increases or stagnation afterwards. Interestingly, the speech network displays slightly higher levels of polarization than the ideal point network. This indicates that more heterogeneity exists in speeches than votes, and aligns with Figure 1 and Figure 2a in the main body of the paper.
For NATO-Warsaw in Figure FIGREF25 , the speech network follows a similar trend as using roll call data alone: polarity decreases towards the end of the Cold War. It appears, however, that the speech weighted network picks up higher variation in preferences, which indicates that looking to ideal point data alone might over- or under-estimate levels of preference polarization at certain points in time. Finally, the West-Rest alignment provides further confirmation that polarity can be detected in the votes and speeches: West-Rest members exhibit higher levels of polarization compared to the Assembly-level. Polarization exhibits a downward trend in the post-Cold War era – the period in which scholars note an increased prominence of liberal, interdependent globalization. These results align with the trends found by BIBREF22 . These results indicate that weighted word counts provide useful information, but we aim to leverage the information in the speeches in a more sophisticated way and in a way that can increase our confidence that divergence is not simply due to differences in word choice. For this task, we move beyond the BOW approach and use vector space representations.
At present, the development of vector space models for textual data is one of the areas that has attracted the most research interest in natural language processing. This interest is motivated by the desire to move away from simple counts and weights of words, towards representations which can preserve word context and linguistic features of human speech. Studies find that these approaches are not only intuitively desirable, but also increase classification accuracy in machine learning tasks.
Vector space approaches involve the embedding of document features from one dimension per word space to a continuous, lower dimensional vector space. Each document feature is represented as a real-valued vector and it has been shown that these representations retain desirable syntactical qualities, such as context and structure of speech. Traditionally, dimensionality reduction could be obtained through latent semantic analysis (LSA) used to factorize the feature matrices, but two recently developed models have been introduced which rely on different logic: word2vec from Mikolov and the research group at Google, and the global vectors for word representations (GloVe) unsupervised learning algorithm from the Stanford Natural Language Processing group.
We use GloVe introduced by BIBREF15 because while LSA tends to maximize the statistical information used, it does not perform well on analogy tasks. Word2vec does better on the analogy test but does not utilize statistics of the corpus because it trains on local context windows. GloVe was introduced to help bridge this gap and combine both desirable qualities. It is a log bilinear, weighted least squares model that trains on global word-word co-occurence counts and thus makes efficient use of the statistics. BIBREF15 show that their approach yields state-of-the-art performance on the word analogy task. GloVe is sometimes criticized for scalability issues but given that we are working with a fixed size corpus this does not pose an issue for our analysis. Readers are encouraged to consult the GloVe paper for full model details, but we describe our approach and decisions here. The model is expressed as: DISPLAYFORM0
where INLINEFORM0 represents parameters, INLINEFORM1 is the vocabulary size, INLINEFORM2 and INLINEFORM3 are column and row word vectors, INLINEFORM4 is the co-occurrence matrix of all pairs of words that ever co-occur, and INLINEFORM5 is a weighting function which assigns lower weights to words which frequently co-occur. This lattermost term serves as a cap on very frequent words, for example articles like “the" which provide little predictive information. The algorithm seeks to minimize the distance between the inner product of the word vectors and the log count of the co-occurrence of the two words. Compared to skip-gram approaches which update at each context window, it is clear from the utilization of INLINEFORM6 that the model trains relatively quickly since it uses the known corpus statistic of word co-occurrences for the entire corpus at once.
We first stem, tokenize, and convert the words to lowercase. Unlike a BOW approach, however, the punctuation is retained. The model is trained on each individual year in the corpus with the vocabulary pruned to include a minimum term count of 4 across documents and the term must exist in 25% of the documents. These relatively stringent parameter levels are employed because we train on individual years in order to avoid language drift over time and to ensure that our estimated embeddings correspond to the exact policy language used in a given year. We employ a skip gram window of 4 and search using a word vector size of 50. At present, we follow the computer science literature suggestion of tuning these parameters until reasonable and reliable linear combinations of language are located. Future work should explore in greater detail how systematic tuning decisions for social science applications can be made.
Mikolov and collaborors introduce a new evaluation scheme based on word analogies that examines dimensions of difference in vector space, as opposed to scalar distances between vectors BIBREF13 , BIBREF14 . In order to validate the quality of the located embeddings, we thus follow current standard practice and assess whether reasonable linear combinations of words can be returned. The main body of the paper presents three examples, and a further example is presented here: DISPLAYFORM0
where INLINEFORM0 describes a vector space location of the given feature and the cosine distance between each vector space location is added or subtracted to find the closest vector space feature (presented on the right hand-side of the equality with the cosine similarity printed underneath). The example is interpreted as: if we add the location of INLINEFORM1 to INLINEFORM2 , we arrive at the vector space location of the word INLINEFORM3 . This, as well as the examples presented in the main body, provide some indication that intuitive and reasonable embeddings have been located.
Vector space representations are particularly promising for the measurement of polarization. In contrast to a BOW approach, we want to know not only that two actors use (dis)similar words, but we want high resolution insights into how and when they speak differently on different topics. For example, we would like to capture the dissimilarity of statements like "we oppose the proliferation of nuclear weapons" versus "the proliferation of atom bombs is necessary." The words “necessary" and “oppose" would be counted in a BOW matrix, but their usage in the context of nuclear weapons would be lost. Further, if one state says “nuclear weapon" and another says “atomic bomb," the two phrases have no words in common and the phrases will be thought to be distant or dissimilar, despite referring to the same thing. Although BOW approaches often perform very well, for the measurement of polarization in IR, we need to make sure that the differences uncovered are not simply due to different word usage or policy topics being discussed.
To measure expressed (dis)agreement in these speeches, it is necessary to derive a document-level representation of the learned embeddings. Although well-established measurements based on cosine similarity, Euclidean distance, or Pearson correlations could be applied to the word embeddings, we utilized the relaxed variant of a newly introduced document distance measure that exploits information contained in both the word embeddings and term-document matrices: the (relaxed) Word Mover's Distance (RWMD) BIBREF17 .
WMD innovates by leveraging the finding that embedding distances between word vectors are semantically meaningful. WMD represents text documents as a weighted point cloud of embedded words where the distance between two documents is the minimum cumulative distance that words from document A would need to travel to match exactly the point cloud of document B. This has been shown to yield state-of-the-art classification accuracy BIBREF39 . Although WMD is relatively fast to compute, we use the relaxed variant (RWMD), which results in tighter bounds and is shown to yield lower test error rates. In short, this relaxes the optimization problem through the removal of one of the two constraints. If we let INLINEFORM0 and INLINEFORM1 be the BOW representations of two documents in the INLINEFORM2 dimensional simplex of word distributions which we obtained above, we can express RWMD as: DISPLAYFORM0
where INLINEFORM0 is a sparse flow matrix where INLINEFORM1 denotes how much of word INLINEFORM2 in INLINEFORM3 travels to word INLINEFORM4 in INLINEFORM5 and INLINEFORM6 is defined to be the distance between the two documents as the minimum weighted cumulative cost required to move all words from INLINEFORM7 to INLINEFORM8 . Then, the optimal solution is found when each word in INLINEFORM9 moves all of its probability mass to the most similar word in INLINEFORM10 . This optimal matrix INLINEFORM11 is decided by: DISPLAYFORM0
where INLINEFORM0 is the distance of interest which we normalize and convert to a similarity score using equation 4 above and INLINEFORM1 , respectively. The result is a list of INLINEFORM2 matrices INLINEFORM3 for each year in the corpus where INLINEFORM4 is the speech similarity score between states INLINEFORM5 and INLINEFORM6 for the given year in the corpus with the diagonals of the matrices set to 0.
Network Analysis
Although we present a novel approach for leveraging position similarity information found in political text, it is not immediately obvious that this approach is useful for the task of political analysis. We posit that for this approach to be considered useful, it should be capable of providing information useful for the performance of inference on observed state behavior. We choose violent conflict onset because this is one of the enduring methodological challenges faced by the discipline. For this task, we aim to see whether or not these embedded speeches provide information on state preferences which improve upon current out-of-sample predictions relative to current models which employ UN roll call data. We choose to compare our approach to the recently published models of BIBREF8 , because their goal and applications which use UN roll call data closely parallel our research motivations. This section provides further information on the graph partitioning approach, the new Multilayer Extraction algorithm, and model performance assessments.
For clusters based on speeches alone, we follow Pauls and Cranmer's ( BIBREF8 ) approach through the performance of 5-nearest neighbor clustering on the matrix of RWMD state-state similarities, which yields candidates for membership in an affinity community. As we are dealing with textual data, and in contrast to their sign-test approach, we assign ties between affinity candidate state pairs INLINEFORM0 based on relatively strict similarity thresholds between 0.50 and 0.60. The result is a square INLINEFORM1 adjacency matrix INLINEFORM2 of unweighted ties for each year which contains all states who voted and delivered a GD statement. To locate multiplex blocs, we use the voting clusters found by Pauls and Cranmer ( BIBREF8 ) as one layer, and the speech clusters just described as a second layer. The Multilayer Extraction algorithm (described below) is then applied to these two layers, which returns community membership labels in vectors. These are transformed into adjacency matrices of unweighted ties for each year. This is a strong test for our hypothesized effects because multilayer communities must be detectable after individual layers have already been clustered.
The intuition of employing a threshold is that every node is connected to every other node in the network through a given text similarity, but this computationally becomes infeasible and unnecessary as many of the ties are low valued. Further, in order to be compatible with the TERGMs, this binarization is necessary, and so we follow thresholding guidelines outlined in the network science literature, see BIBREF40 , BIBREF41 , BIBREF42 , BIBREF43 . Because the choice of threshold is case dependent and must take into account the type of data under scrutiny, we turn to the NLP literature, which suggests that a threshold of INLINEFORM0 is considered to be a relatively common benchmark for discarding low-information similarity scores BIBREF44 , BIBREF45 , BIBREF46 . As different thresholds result in different graphs, we check and confirm the robustness of our results at different levels and report these results below.
Clusters based on speeches or votes in isolation provide one slice of information about state preferences, but we are motivated to locate an approach which exploits both sources of information in tandem. This is broadly aligned with other work in political science which aims to combine votes and speeches in statistical models. Specifically, we posit that valuable and perhaps different information might be garnered from the voting behavior and speeches made by states and we consider these as two elementary layers of a multilayer network. A two-layer network illustration is presented in Figure FIGREF32 . Extracting communities based on this multilayer network provides one strategy of holistically exploiting both sources of preference information in tandem and is one of the current forefronts of network science research BIBREF26 .
At present, most multilayer community detection heuristics require the community structure to be homogenous at each layer, but as displayed above, speech and voting patterns often diverge. A recently proposed solution to this problem is the Multilayer Extraction procedure BIBREF27 . The algorithm identifies densely connected vertex-layers in multilayer networks through a significance-based score that quantifies the connectivity of an observed vertex-layer set by comparison with a multilayer fixed degree random graph model. For our analysis, the clusters from voting data comprise the first layer and the text-based clusters comprise the second layer of the graph for each year.
This unique approach to community detection allows us to exploit preference similarity information across speeches and voting habits without requiring states to be similar on both dimensions. Formally, our vote-speech layered network is a node-aligned INLINEFORM0 undirected multilayer network INLINEFORM1 with no self-edges INLINEFORM2 . The multilayer network is node aligned because we add extraneous nodes such that INLINEFORM3 where typically the INLINEFORM4 . Such an approach is shown in the literature to yield mathematically desirable properties for the task of multilayer community detection BIBREF26 and is also required for the TERGM estimations.
The Multilayer Extraction procedure locates communities across the two network layers and provides a resulting membership identification vector which can be used to construct a one-mode representation of the cross-layer communities for a given year where an edge between INLINEFORM0 and INLINEFORM1 exists if the extraction procedure identifies INLINEFORM2 and INLINEFORM3 as belonging to the same community. If the extraction procedure fails to locate multilayer communities, we can conclude that no information is lost by using the more common aggregation procedure of considering the union of edges in the two layers INLINEFORM4 and consider an edge to exist between INLINEFORM5 and INLINEFORM6 if an edge exists in either of the two layers, see BIBREF26 . The result is a one mode network for each year based on the two sources of preference information. These multiplex blocs are employed as a substitute for votes- and text-based clusters in Model 4, and the results are reported in [table:tergm]Table 1 of the main paper.
For the results reported in the paper's main body, we found that Multilayer Extraction detected heterogenous community structure in the two layers of speeches and votes for 12 of the years. For the other 18 years, no multilayer communities were detected and therefore could be aggregated into a single network without the risk of ignoring multilayer community structure in the analysis. This helps to explain why some findings using UN roll call data alone come up with varying conclusions. Roll call data would perform better in years where complex speech-vote dependencies did not exist (i.e. when a given state's votes and speeches are both “similar" and therefore considering one source of information alone is analytically sufficient). However, in years where a given state's voting behavior and speeches diverge (e.g. a NATO member voting with the bloc but delivering a speech which contains position information which diverges from the rest of the bloc), then favoring one source of information over the other will likely yield misleading estimates of preference polarization. This multilayer community detection approach allows us to capture these potentially complex structures at both levels of preference information.
In the body of the main paper, we present the results of the original model and date-adjusted models of BIBREF8 , as well as our textual and multiplex cluster results. The readers can refer to the original paper of BIBREF8 for complete details on variable operationalization and model specification. We focus here on the performance of the models.
Increasing attention is devoted to the appropriate role of unsupervised methods in political science research, because they require human interpretation and lack well-defined criteria for accuracy and performance assessment. Both the word embeddings and multilayer community detection procedure are unsupervised methods, and so we posit that in order to validate that these located affinity blocs really do capture meaningful preference similarity in international politics, they should be able to improve upon our current ability to predict conflict onset out-of-sample. This would enable us to more adequately explain observed state behavior and provides a rigorous criteria for performance assessment. As reported in the main paper, clusters based on partitions of our speech graph and our speech-vote multiplex graph are both statistically significant predictors of conflict onset. Here, we assess the goodness-of-fit of our models, the out-of-sample predictive accuracy of the models, and robustness checks of our results at different tie similarity thresholds.
To assess the in-sample goodness-of-fit (GOF) of exponential random graph models, it is common to simulate network statistics which were not specified in the model to see how well the fitted model can simulate statistics on the outcome network. In our case, the outcome network is the conflict onset network, and we perform 50 simulations over each of the 30 time steps for 1,500 total simulations for each of the four models. The in-sample areas under the ROC and PR curves for the three models of primary concern are presented in Figure FIGREF36 and Figure FIGREF37 , respectively. Furthermore, the dyad-wise and edge-wise shared partners and modularities are presented for all four models in Figure FIGREF38 and Figure FIGREF39 . Although the original models from BIBREF8 and our textual and multiplex models all exhibit impressive GOFs, the multiplex model exhibits the best in-sample GOF as measured by areas under the ROC and PR curves. This increases our confidence in the model specifications, but it is necessary to assess out-of-sample predictive capability since all four models fit quite well.
To assess whether or not the inclusion of textual or multiplex clusters improves upon the existing models with roll call-based clusters alone, we follow BIBREF8 in training on five-year windows and attempt to predict the next year of conflict onset. The multiplex model exhibits a 20.5% increase in area under the precision recall curve compared to the original date-adjusted model (1.156 vs. 0.959). In contrast, the model with textual clusters alone underperforms the original date-adjusted model (0.081 vs. 0.959). We plot these out-of-sample performance results for the multiplex models alongside other tie thresholds as checks on the stability of the results in Figure FIGREF40 which displays the sum of the areas under the precision recall curves at various thresholds compared to the baseline (date-adjusted) model presented in BIBREF8 .
We find that several of the thresholds provide reliable and statistically significant estimates of conflict onset. In order to choose the final model, we select the model with the lowest variance in out-of-sample prediction capability (i.e. the area under the precision recall curve), that is, we select the model which yields the most consistent predictions over time. Because out-of-sample predictions on sparse networks (such as the conflict onset network) is a challenging task, this selection approach helps to eliminate models with predictions that vary widely from one year to the next. As previously mentioned, a threshold of .50 is common in the NLP literature and so we report every threshold between [0.50,0.60] at .01 increments. We select the .58 tie threshold because this model exhibits the lowest variance in predictions.
To ensure that we are not selecting a single model which vastly outperforms all other thresholds (i.e. to ensure that our results are robust), we consider the out-of-sample predictive accuracy for the models in which the multiplex clusters yielded a statistically significant relationship with conflict onset. This was the case for 7 out of the 11 thresholds tested. The sums of the areas under the precision recall curve for these models are plotted in Figure FIGREF41 . These box plots make clear that our multiplex models do indeed display increased out-of-sample predictive capability relative to the baseline model across various thresholds.
The above amounts to a toolkit for the exploitation of multidimensional information on positions and preferences in political research. We show how the information in textual data can be usefully exploited beyond simple word counts and weighted frequencies. We show how this information can in its own right be useful, but also how it can be exploited in tandem with other existing sources of preference information like votes through a multilayer network approach. Finally, we show that these measures are not only substantively reasonable, but can be used to extend current state-of-the-art network models which infer the impact of preferences on international conflict.

Question: Do they use number of votes as an indicator of preference?
Answer: No

Yes

Question: What does a node in the network approach repesent?
Answer: Unanswerable

Unanswerable

Question: Which dataset do they use?
Answer: corpus of state speeches delivered during the annual UN General Debate

corpus of state speeches delivered during the annual UN General Debate