Venue: NAACL 2019

# No Permanent Friends or Enemies:

Tracking Relationships between Nations from News

 Xiaochuang Han

College of Computing

Georgia Institute of Technology

Atlanta, GA

xc@gatech.edu &Eunsol Choi

Paul G. Allen School of CSE

University of Washington

Seattle, WA

eunsol@cs.washington.edu &Chenhao Tan

Dept. of Computer Science

University of Colorado Boulder

Boulder, CO

chenhao@chenhaot.com

###### Abstract

Understanding the dynamics of international politics is important yet challenging for civilians. In this work, we explore unsupervised neural models to infer relations between nations from news articles. We extend existing models by incorporating shallow linguistics information and propose a new automatic evaluation metric that aligns relationship dynamics with manually annotated key events. As understanding international relations requires carefully analyzing complex relationships, we conduct in-person human evaluations with three groups of participants. Overall, humans prefer the outputs of our model and give insightful feedback that suggests future directions for human-centered models. Furthermore, our model reveals interesting regional differences in news coverage. For instance, with respect to US-China relations, Singaporean media focus more on "strengthening" and "purchasing", while US media focus more on "criticizing" and "denouncing".

## 1 Introduction

In the context of growing globalization Baylis et al. (2017), understanding complex international relations is increasingly relevant to our daily life. Yet this is a challenging task due to the inherently dynamic nature of international relations. As Kissinger famously said, "Americ has no permanent friends or enemies, only interests." Staying informed becomes even harder in the continuous streams of information from news outlets and social media.

This very availability of such information, however, opens up exciting opportunities for natural language processing to support individuals in understanding international relations. Supervised extraction has been incredibly useful at identifying pre-defined relations and events Doddington et al. (2004); Mintz et al. (2009) but fails to capture emerging or complex information needs. Topic models and neural models have been proposed to explore relations between entities without supervision O'Connor et al. (2013); Chaney et al. (2016); Iyyer et al. (2016). In particular, Iyyer et al. (2016) introduces an unsupervised neural model for tracking relations between fictional characters, and this approach outperforms baselines from topic models and hidden Markov models. In this work, we incorporate linguistic insights into this model to track relation dynamics between nations from news articles.

Our model reconstructs textual information in the embedding space using relation embeddings, as proposed in Iyyer et al. (2016). We integrate simple yet effective linguistic insights: verbal predicates often describe the relationship between entities,1 while nouns and proper nouns provide the context of this relationship. For example, in "U.S. denounces Russia for its interference in the 2016 election", _denounce_ describes the relation, and _election_ and _interference_ provide the context. We show that this intuition leads the model to discover relation descriptors that are easier to interpret and less noisy.

Footnote 1: We use entities, countries, and nations interchangeably in this work.

Evaluating these exploratory models for subjective tasks poses a challenge as there are no gold labels. Along with the model, we propose new approaches for evaluation. We introduce a quantitative metric which aligns pre-annotated key events with the temporal trends of relationships produced by the models. Since this task requires careful analysis of complex international relations, we conduct in-person user studies with NLP researchers and undergraduate students recruited from political science and linguistics courses. Both quantitative evaluation and human evaluation indicate

resents the dynamic relationships between nations than the prior model (Iyyer et al., 2016): 75.9% of participants preferred our model for finding natural language words describing international relations and 85.5% preferred temporal trends generated by our model.

Finally, we qualitatively explore the context of relations provided by an attention-based mechanism and demonstrate a practical application of our model by studying regional differences in news coverage of relationships between two countries. We conclude with discussions on future directions for buildings models that can support individuals in navigating a large collection of news articles. Our code is available at [https://github.com/BoulderDS/LARN](https://github.com/BoulderDS/LARN).

## 2 Data

We start by introducing our dataset of news articles, the shallow linguistic information that we extract, and our annotation of key events.

News article collectionOur dataset is derived from the NOW corpus, the largest corpus of news articles that is available in full-text format.2 The NOW corpus collects the news articles that are available on Google News in 20 English-speaking countries and thus include news articles that span a wide variety of topics, ranging from politics, to sports, to celebrity news from 23K media outlets. In this work, we consider the news articles in recent years, i.e., from January 2016 to June 2018, to facilitate human evaluation.

Footnote 2: The dataset can be obtained from [https://www.corpusdata.org/now_corpus.asp](https://www.corpusdata.org/now_corpus.asp).

We consider 12 nations (U.S., Russia, China, UK, Germany, Canada, France, India, Japan, Iran, Israel, and Syria) and the 66 nation pairs between them. To identify mentions of each nation in news articles, we manually construct a set of aliases for each nation to cover common abbreviations and the names of political leaders (e.g., Trump, Putin). On average, each nation has 3.5 aliases. We then use these aliases to find sentences that contain a pair of nations under consideration and obtain 1.2M sentences associated with 634K articles.

Adding Shallow Linguistic InformationTo incorporate shallow linguistic knowledge, we process the news article collection for each nation pairs to extract (1) verbal predicates and (2) nouns and proper nouns from each sentence. Specifically, we use a dependency parser to detect verbal predicates and their subjects and objects, and only include predicates for which both subjects and objects were found. For sentences with such predicates, we find nouns and proper nouns using part of speech tags. All data processing was done in spaCy Honnibal and Montani (2017) and full details can be found in the appendix.

Key events annotationThe main goal of our model is to support the exploration of international relations, which is very challenging to evaluate. To derive quantitative evaluation measures and provide the basic context of international relations, we manually identify key events over the 30 months for 8 most frequently mentioned nation pairs (i.e. US-[China, Russia, UK, India, Canada, Japan, Syria], China-India) by reading through the top Google search results for each two countries and each month.3 We identified roughly five key events per nation pair. For example, one key event for US-China relation is Chairman Xi's visit to the US in April 2017. A complete list of key events is shown in the appendix. Table 1 summarizes the data statistics.

Footnote 3: This annotation is done by the first author and is thus inherently subjective. We add a robustness check in the appendix based on another set of independently annotated key events by the third author. Automatic evaluation shows a similar trend holds for the two sets of annotations.

## 3 Model

In this section, we formally introduce our model that builds on Relation Modeling Network (_RMN_) Iyyer et al. (2016). Our main contribution is to integrate shallow linguistic information (i.e., verbal predicates and nouns/proper nouns) and identify the context of relations.

### Overview

The intuition behind our model follows RMN, i.e., inferring relation embeddings by reconstruct

\begin{table}
\begin{tabular}{l r} \hline Total \# of months considered & 30 \\ Total \# of articles & 7.7M \\ Total \# of articles with valid nation pairs & 634K \\ \hline Avg. \# of articles per nation pair & 9.6K \\ Avg. \# of sentences per nation pair & 18.2K \\ Avg. \# of predicates per nation pair & 21.6K \\ Avg. \# of nouns \& proper nouns per nation pair & 201K \\ \hline Avg. \# of key events per nation pair (8 pairs) & 4.9 \\ \hline \end{tabular}
\end{table}
Table 1: Data statistics.

 

ing textual information in the embedding space. Specifically, we learn a fixed set of relation embeddings and use a convex combination of these relation embeddings to reconstruct information in sentences that mention both entities.

Our main hypothesis is that relation information is often encoded in verbal predicates and we can obtain more interpretable and robust relations if we focus on predicates. For each pair of entities, we extract information from the predicates in sentences with both entities and reconstruct these predicates using shared relation embeddings. In addition, we use nouns to provide the context for the relations. We refer to our model as Linguistically Aware Relationship Networks (_LARN_).

We now formally define our problem. The input of our model is a collection of news articles. For each entity pair \(e_{i},e_{j}\), we obtain a set of articles, \(\mathcal{A}_{e_{i},e_{j}}\), containing at least one sentence mentioning both entities. We identify sentences where both \(e_{i}\) and \(e_{j}\) occur based on any alias associated with an entity (nation). We extract all the verbal predicates from these sentences in article \(a\in\mathcal{A}_{e_{i},e_{j}}\) as \(\{p^{a,e_{i},e_{j}}_{1},\ldots,p^{a,e_{i},e_{j}}_{N}\}\), as well as the proper nouns or nouns as \(\{n^{a,e_{i},e_{j}}_{1},\ldots,n^{a,e_{i},e_{j}}_{M}\}\). Preprocessing details could be found in the appendix. We then use GloVe embeddings to represent these words, i.e., \(v_{p^{a,e_{i},e_{j}}_{k}}\) for \(p^{a,e_{i},e_{j}}_{k}\) and \(v_{n^{a,e_{i},e_{j}}_{k}}\) for \(n^{a,e_{i},e_{j}}_{k}\)Pennington et al. (2014). These word embeddings are static in the entire learning process.

Our model learns relation embeddings \(\mathbf{R}\in\mathbb{R}^{K\times d}\), where \(K\) is a hyperparameter for the number of relations and \(d\) is the dimension of the relation embedding as well as the word embedding. Following Iyyer et al. (2016), we obtain a list of natural language descriptors for each relation using the nearest neighbors of the relation embedding within the 500 most common predicates.4 The model also provides (1) a probability distribution over relations for each article, (2) a probability distribution over nouns for each relation between each entity pair, and (3) an embedding for each entity.

Footnote 4: Refer to the appendix for a version that include all words. The relation descriptors from _RMN_ required a manual filtering step in Iyyer et al. (2016), and become unintelligible without the 500 common words constraint. Note that our model produces intelligible descriptors even without the 500 most common predicates constraint.

Figure 1 describes the overall architecture. We will describe the construction of \(v^{a}_{\text{label}}\) in SS3.2, a reconstruction of \(v^{a}_{\text{label}}\) through a weighted sum of relation embeddings in SS3.3, and finally the learning in SS3.4.

### Modeling Text with Predicates

We compute the representation for each article to be reconstructed as the sum of bag-of-words embeddings. While Iyyer et al. (2016) considers all the words in a window in which both entities occur, we only consider predicates in the sentences where both entities show up:

\[v^{a,e_{i},e_{j}}_{\text{label}}=\sum_{k=1}^{N}v_{p^{a,e_{i},e_{j}}_{k}}.\]

\(v^{a,e_{i},e_{j}}_{\text{label}}\) depends on both the news article and the entity pair. In the rest of the paper, We omit \(e_{i},e_{j}\) in the superscript here for simplicity, i.e., \(v^{a}_{\text{label}}\).

### Reconstructing Text with Relation Embeddings

We represent each article as a weighted sum of relation embeddings. Assuming \(d^{a}\) represents a weight vector over the \(K\) relation embeddings that

Figure 1: Model overview of an article with respect to a pair of nations. Our model approximates the predicates from the input article \(v^{a}_{\text{label}}\) as a weighted sum of relation embeddings from **R**. The blue embeddings are fixed and the green ones are trained. The red cell indicates our attention mechanism. The shaded area is our extension: we use only predicates as the label and provide the context of relations via nouns.

 

sums to 1 for an article \(a\) with respect to a pair of entities \((e_{i},e_{j})\), we obtain \(r^{a}\) as follows:

\[r^{a}=\textbf{R}^{\top}d^{a}.\]

\(d^{a}\) can also be thought of as a distribution over relations. This distribution over relations depends on entity pair (\(v^{a}_{e}\)), predicate information (\(v^{a}_{p}\)), and noun information (\(v^{a}_{n}\)). While _RMN_ simply takes the average of all words in the sentence, _LARN_ focuses on verbal predicates and nouns to capture our intuition that predicates describe the main relations, whereas nouns provide background information to explain the relations. We now describe these three components in details.

Representing predicates and entity pair.We follow _RMN_ to construct embedding for each entity pair and for the predicates. The entity pair vector, \(v^{a}_{e}\), simply adds the embedding of the two entities. The predicate vector, \(v^{a}_{p}\), is equivalent to \(v^{a}_{\text{label}}\) except for word dropout during training, i.e., setting \(b^{a}_{k}\) to be 0 or 1 with a probability of 0.5.

\[v^{a}_{e} =v_{e_{i}}+v_{e_{j}},\] \[v^{a}_{\text{p}} =\sum_{k=1}^{N}b^{a}_{k}\cdot v_{p^{a}_{k}}.\]

Representing context with nouns.To understand relations between an entity pair in a sentence, nouns should be considered in addition to predicates. For example, _tariff_ is indicative of the relation between US and China in "Originally, Trump favoured the simple imposition of a tariff on products from selected countries, especially China and Mexico", despite the seemingly positive predicate _favour_. As nouns are much more common than predicates (see Table 1) and not all of them are meaningful for understanding international relations, we employ a weighted sum of noun vectors. We use an attention mechanism Conneau et al. (2017); Bahdanau et al. (2015) and consider each entity pair as a unique key to compute the attention weights, since the same noun can be interpreted differently across different entity pairs. To this end, we train an attention query embedding \(q_{e_{i},e_{j}}\) for each entity pair separately. We further encode the temporal information by concatenating a one-hot vector \(t^{a}\) that indicates the month when the article was published with the noun representation \(v_{n^{a}_{k}}\). This allows us to capture the shifts in a word's meaning over time.

\[h_{n^{a}_{k}} =\text{tanh}(W_{\text{proj}}\cdot[v_{n^{a}_{k}};t^{a}]),\] \[\alpha_{n^{a}_{k}} =\frac{\exp(hn^{a}_{k}\cdot q_{e_{i},e_{j}})}{\sum_{k^{\prime}=1} ^{M}\exp(hn^{a}_{h^{a}_{k^{\prime}}}\cdot q_{e_{i},e_{j}})},\] \[v^{a}_{n} =\sum_{k=1}^{M}\alpha_{n^{a}_{k}}h_{n^{a}_{k}}.\]

Finally, we concatenate the three representations as the input to a feedforward network and pass to a softmax layer to create the weight vector \(d^{a}\).

\[v^{a}_{\text{final}} =\text{ReLU}(W_{\text{cat}}\cdot[v^{a}_{p};v^{a}_{n}]),\] \[d^{a} =\text{Softmax}(W_{\text{final}}\cdot v^{a}_{\text{final}}).\]

This \(d^{a}\) is multiplied with the descriptor matrix **R** to get the final representation \(r^{a}\). Different from _RMN_, we do not consider temporal dependencies between time steps in our model because it is important to understand sudden shifts in international relations rather than assuming that the relations slowly evolve. We also found the temporal dependencies were not helpful empirically in our domain but rather computationally expensive.

### Learning Objective and Summary

The reconstruction objective pushes \(r^{a}\) to resemble \(v^{a}_{label}\). Our formulation is identical to _RMN_: the loss function consists of a contrastive max-margin loss term, \(J\), and an auxiliary loss term, \(X\), to encourage unique relation embeddings.

\[L(\theta)= J(\theta)+\lambda X(\theta),\] \[J(\theta)= \sum_{a\in\mathcal{A}_{e_{i},e_{j}}}\sum_{v^{a}_{\text{label}} \in\mathcal{N}}\max(0,\] \[1-\frac{r_{a}\cdot v^{a}_{\text{label}}}{||r_{a}||\cdot||v^{a}_{ \text{label}}||}+\frac{r_{a}\cdot v^{a}_{\text{label}^{\prime}}}{||r_{a}|| \cdot||v^{a}_{\text{label}^{\prime}}||}),\] \[X(\theta)= |RR^{\top}-I||,\]

where \(v^{\prime}_{\text{label}}\) is a randomly sampled negative example, \(\mathcal{N}\) is a collection of them, and \(\lambda\) is a hyperparameter for balancing two loss terms.

## 4 Evaluation

In this section, we compare our model to _RMN_. For both models, we fixed the number of descriptors to 30 following Iyyer et al. (2016). As tracking dynamic international relations requires careful analysis, we hosted onsite user studies for quality control and in-person feedback. We first describe the model outputs, and then present both quantitative and qualitative evaluation results 



### Understanding Model Outputs

Given a set of time-stamped news articles and a list of nations of interest, both models provide a set of relation descriptors, where each one **defines** a type of relation and a temporal trend analysis of these descriptors that shows how the relation **evolves**.

Relation descriptors.Table 2 shows the top five and bottom five descriptors from _LARN_ and _RMN_ sorted by the average weights over all news articles related to the most frequently mentioned eight nation pairs.5 By using predicates to describe relations, our descriptors seem to contain more semantically meaningful words. For instance, the top relation in _RMN_ consists of exclusively non-content words. Another interesting advantage of our model is that the five relations with the lowest weight have much higher weights in _LARN_ than in _RMN_. This suggests that _RMN_ tends to generate "useless" relations that do not show up in the data, while even bottom relations in _LARN_ remain useful for describing the data.

Footnote 5: We focus on the top eight nation pairs to be consistent with human evaluation.

Temporal trends.We visualize the temporal trends of the most prominent relations between nation pairs. We further provide our annotated key events as the context to interpret these temporal trends. Figure 2 gives an example for US and China. The top three relations based on _LARN_ are "denounce", "strengthen", and "leave",6 while the top three based on _RMN_ are "seem", "range", and "negotiate". We find our model generally aligns better with the key events: for instance, the "denounce" relation peaked around the time that Trump started issuing a series of tariffs based on

\begin{table}
\begin{tabular}{l l|l c} \hline \hline _LARN_ (Linguistically Aware Relationship Network) & Weight & _RMN_ (Relationship Modeling Network) & Weight \\ \hline denounce, undermine, condemn, punish, oppose leave, tell,ask,know, want differ, indicate, affect, regard, determine strengthen, enhance, improve, develop, host hit, cut, end, kick, beat & 5.46\% & seem, thing, though, too, especially range, information, supply, infrastructure, value & 11.79\% range, information, supply, infrastructure, value & 9.42\% \\ \hline  & & 4.64\% & theresa, emmanuel, yemen, poland, lebanon & 8.24\% \\ \hline  & & 4.56\% & negotiate, establish, impose, propose, negotiation & 7.92\% \\ \hline  & & 4.05\% & terrorism, militant, terror, terrorist, condemn & 7.54\% \\ \hline buy, manufacture, use, brand, purchase win, defeat, beat, title, match receive, express, praise, send, acknowledge offer, provide, deliver, feature, guarantee launch, announce, unveil, release, celebrate & 2.59\% & propose, woman, represent, indians, spend & 1.69e-09 \\ \hline  & & 4.6efense, kremlin, cite, pressure & 1.62e-09 \\ \hline  & & car, de, bomb, little, note & 5.39e-10 \\ \hline  & & cost, cite, compare, co, m & 2.94e-10 \\ \hline  & & aid, hand, small, public, round & 2.21e-10 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Relation descriptors of the most/least frequent five relations by _LARN_ and _RMN_ and their average weights in news articles. The relations generated from _LARN_ are more semantically meaningful.

Figure 2: Temporal trends of top three relations between US and China based on _LARN_ (Figure 1(a)), in comparison with results from _RMN_ (Figure 1(b)). We highlight key events during this time period under the x-axes. Similar visualizations are also used in human evaluations. More figures are provided in the appendix.



our model (Figure 1(a)), while there do not exist similar fluctuations in Figure 1(b) based on _RMN_.

### Quantitative Evaluation

We leverage our manually annotated key events to develop a novel automatic metric to evaluate how well the temporal trends from our model are aligned with key events. We define a change rate at each month \(\Delta_{t}\) as the weighted average of changes in relation weights in the top three relations:

\[\Delta_{t}=\sum_{i\in\text{top 3 relations}}w_{t,i}*\frac{|d_{t,i}-d_{prev, i}|}{d_{prev,i}},\]

where \(d_{t,i}\) is the average weight for the \(i\)-th relation in all the articles published at \(t\), \(d_{prev,i}\) is the average weight before \(t\) in a window of \(W\) preceding months (\(\frac{1}{W}\sum_{W=1}^{W}d_{(t-w),i}\)), and weight \(w_{t,i}\) is the normalized weight for top three descriptors (\(\frac{d_{t,i}}{\sum_{j\in\text{top 3 relations}}d_{t,j}}\)). Our results are robust to the choices of \(W\), and we set it to 6 for presentation.

We expect change rates to be greater when significant events happen in international politics. Figure 3 compares the change rate at months where key events occurred with other months for eight nation pairs for which we annotated key events. Both models present more abrupt changes when key events occurred. Unlike _RMN_, our model does not have temporal dependencies between relation distributions over time,7 and thus has a higher discontinuity in general. However, even in relative terms, our model fluctuated more substantially than _RMN_ when key events occurred. We also did a robustness check with another set of independently annotated key events and the results can be found in the appendix. This measure captures whether the model can detect the change points, but does not measure whether the model correctly captures the **semantics** of the key events, i.e., did a negative relation increase after hostile events such as war? To this end, we performed human evaluations.

Footnote 7: Please refer to Iyyer et al. (2016)’s Section 3.2.3 for more details on how they incorporated previous time steps.

### Human Evaluation

We hosted three human evaluations with participants from different demographics: undergraduate students from political science classes, graduate students from a computer science department (mostly in NLP), and undergraduate students taking a linguistics class. The total number of participants was 29, roughly equally divided among the three groups. The participants were shown outputs from _RMN_ and _LARN_, and asked to choose the output that better aligns with their intuitions. Each participant answered about 10 questions and provided justification for their answers to each question, taking roughly 30 minutes to an hour. Table 3 summarizes the results of our human evaluations.

Relation descriptor evaluationThe participants were shown a list of top five descriptors (as in Table 2) from two models, and prompted to select a set which adequately covers possible relationships that can occur between countries. 75.9% of participants preferred our model.

Temporal trends evaluation.We showed temporal trends between nation pairs annotated with key events, one from _RMN_ and the other from _LARN_ (as in Figure 2). We asked them to evaluate whether the temporal trends accurately reflect the dynamics in nation-to-nation relationships. Each participant evaluated four randomly chosen nation

Figure 3: Change rate at key event months vs. other months. Our model demonstrates a clearer relative difference between key event months and other months (66.9%) than _RMN_ (50.1%). For the random baseline, we randomly choose the same number of key events and we observe no differences as expected.

 

pairs. The temporal trend from our model was preferred more frequently (85.5% of total responses).

**Nation pair matching.** We designed a novel task where we showed the participants the other four temporal trends without annotated key events and asked them to match each trend with the corresponding nation pair from the four candidates, based on their world knowledge about nation-nation relations. Each participant did the matching twice, once for _RMN_ and once for _LARN_. The participants found correct temporal trends for 45.2% of entity pairs for _RMN_, and 38.0% of entity pairs for _LARN_, when random pairing would yield 25.0%. The difference between two models here is not statistically significant. Most participants found this task very challenging, as they did not know much about the relationship between certain entity pairs (e.g., a participant said "As an American, there's no way to know the relation between China and India."). Even political science students do not perform better than the other two groups.

Discussion.Overall the output from our model is preferred by the participants. We found that political science students paid more attention to detail, took a longer time to finish, and were more ambivalent between the performance of the two models. For example, for temporal trends, they preferred our model for 71.4% of the examples, compared to other groups which preferred our model for 90% of the examples. They also preferred _RMN_'s relation descriptors slightly (42.9% selected our model) and commented that a few concepts from _RMN_, like "infrastructure, supply, and value", are more concrete (e.g., a participant said "I chose the left one (_RMN_), because it is easy to determine and remember the positive of items such as infrastructure, value, and supply. Those have more positive undertones, while it is easy to gauge negative sentiments with 'terrorism,' 'condemn' and those."). As _LARN_ encodes the background context specific to each nation pair separately, our relation descriptors do not contain such "concrete" concepts. In the next section, we will discuss **contexts** for relations, where these concepts appear in _LARN_.

## 5 Further Exploration

We present additional qualitative results to showcase the applications of our model. First, we examine the context (nouns) associated with a relation between two nations based on an attention-based mechanism, which _RMN_ does not handle. Second, we perform an in-depth analysis to show how our model can reveal regional differences in news coverage on the same topic.

### Context for Relations

To help users better understand the inferred relations, we offer specific contexts that relate to the inferred relations based on the attention-based mechanism introduced in SS3. For each relation, we find articles that place the most weight in that relation, and rank the nouns and proper nouns in those articles by their average attention score (i.e., \(\alpha_{n^{a}_{i}}\) in SS3.3).

The top part of Figure 4 shows the top 10 words for the "denounce" relation in Figure 1(a), the temporal trend for US-China relations. Since _RMN_ does not support such mechanism, we show the most frequent nouns that occurred in the documents that mention both entities as a baseline

Figure 4: Top contextual words for US-China’s “denounce” relation derived from three approaches. _Top_: Showing word’s avg. attention score in all “denounce” articles each month. _Middle_: Showing word’s frequency in all “denounce” articles each month. _Bottom_: Showing word’s average attention score multiplied by \(\log(\text{appearance})\) in each month. All figures are normalized by the global maximum score in the figure.

 

(middle part).8 We find that the attended nouns from our model are more informative than frequent nouns: "tariff" is the most attended word; words such as "sanction", "treaty", and "pressure" also show up, while the frequency baseline centers around words like "China" and "president". As we noticed that the frequency baseline can capture alignment with key events, we incorporate attention score and frequency in the bottom part of Figure 4. This augmented version captures informative words (e.g., "tariff", "sanction", and "missile") 9 and closely aligns with the key events.

Footnote 8: We had a comparison between the top figure and the middle figure in the human evaluation, but we found an error in visualization and thus focus on qualitative comparisons.

Footnote 9: “misisile” points us to another event related to US deploying missile defense system in South Korea, which also impacts US-China relations.

### Regional Differences in News Coverage

To further demonstrate the utility of our model, we explore regional differences in news coverage, as "it is possible to build real knowledge by comparing perspectives from different social contexts."10 This also relates to the longstanding literature on framing in news coverage, i.e., "selecting some aspects of a perceived reality and make them more salient to promote problem definition/interpretation" Entman (1993); Chong and Druckman (2007).

Footnote 10: [https://www.publicbooks.org/why-an-age-of-machine-learning-needs-the-humanities/](https://www.publicbooks.org/why-an-age-of-machine-learning-needs-the-humanities/)

We picked two countries, Singapore and US, to study US-China relations.11 Using country source of media outlets in the NOW corpus, we found 10K articles from Singapore media and 5.7K articles from US media on US and China.

Footnote 11: China is not an English speaking country and is thus not in the NOW corpus. Singapore contained the most articles containing both US and China.

Table 4 shows the top five relations sorted by their absolute weight differences between US media outlets and Singapore mean media outlets. Singapore media more frequently use "positive" descriptors such as "strengthen" and "purchase", whereas US media report negative relations such as "denounce" and "criticize" more frequently. Table 5 shows two example sentences from articles with the most weight in the "denounce" relation. Even though two media sources are focusing on events leading to the same type of relation, a reader who mainly consume news articles in Singapore would get a clearly different impression of US-China relations from those who read US news.

## 6 Related Work

Prior work Chambers et al. (2015); Choi et al. (2016); Rashkin et al. (2017) studied entity-entity relations in terms of positive and negative sentiments between them. Similarly, literature on relation extraction Riedel et al. (2010); Gardner and Krishnamurthy (2017); Elson et al. (2010); Srivastava et al. (2016) focused on pre-defined relations between a pair of entities in the database schema. In comparison, our work discovers descriptors for relations between entity pairs instead of finding entity pairs matching pre-defined relation schema.

Topic modeling has been an important method to grasp important concepts from a large collection of documents in an unsupervised fashion Blei et al. (2003); Das et al. (2015); Chang et al. (2009); Schein et al. (2015). Similar to our work, O'Connor et al. (2013) incorporates linguistic insights with topic models to identify event classes and detect conflicts. Our work additionally models

\begin{table}
\begin{tabular}{l c c} \hline \hline Relation descriptor & US & Singapore \\ \hline strengthen, enhance, improve, develop, boost & 4.62\% & **5.42\%** \\ purchase, sell, pay, buy, cost denounce, undermine, condemn, punish, oppose & 3.18\% & **3.86**\% \\ wage, criticise, accuse, tell, criticize & **6.21\%** & 5.66\% \\ ice & **3.81**\% & 3.29\% \\ flee, destroy, invade, unmake, expel & **3.70**\% & 3.32\% \\ \hline \hline \end{tabular}
\end{table}
Table 4: Top five relations between US and China sorted by absolute weight differences between US media outlets and Singapore mean media outlets.

\begin{table}
\begin{tabular}{l l} \hline \hline US media: “President Donald Trump is preparing to _impose_ a _package_ of $60 billion in annual _tariffs_ against Chinese products, following through on a longtime _threat_ that he says will _punish_ China for intellectual property theft and _create_ more American jobs.” \\ \hline Singapore media: It does not look like just a _trade war_, but rather the US is trying to _bully_ China and the rest of the world in order to make China _concede_ economic resources and development _opportunities_ to the US and make the US forever big and strong.

\begin{table}
\begin{tabular}{l l} \hline \hline USmedia: “President Donald Trump is preparing to _impose_ a _package_ of $60 billion in annual _tariffs_ against Chinese products, following through on a longtime _threat_ that he says will _punish_ China for intellectual property theft and _create_ more American jobs.” \\ \hline Singapore media: It does not look like just a _trade war_, but rather the US is trying to _bully_ China and the rest of the world in order to make China _concede_ economic resources and development _opportunities_ to the US and make the US forever big and strong.

 

the context of relations through nouns and focuses on exploring the potential of neural models.

Most relevant to our work is Iyyer et al. (2016), which suggests _RMN_ better capture dynamic relationships in literature than hidden Markov model Gruber et al. (2007) and LDA Blei et al. (2003). Recent work extended and applied _RMN_ to other settings such as studying user roles in online communities Wang et al. (2016); Frermann and Szarvas (2017). Notably, Chaturvedi et al. (2017) suggests HMM with shallow linguistic features (i.e., frame net parses) and global constraints can outperform _RNN_ for modeling relations in literature. In this work, we incorporate linguistic insights with _RMN_ and apply it to news domain.

Last but not least, researchers have studied the dynamics of media coverage from a wide range of perspectives, ranging from framing Card et al. (2015); Field et al. (2018), to relationship between ideas Tan et al. (2017), to quotes of politicians Niculae et al. (2015); Tan et al. (2018); Leskovec et al. (2009). There is also significant effort for building event databases in political science Leetaru and Schrodt (2013), and assisting journalists with tools Handler and O'Connor (2017), and dating historical text Nicolae et al. (2014).

## 7 Conclusion

We investigate the promise of unsupervised neural models for automatically inferring relations between nations. We find that incorporating shallow linguistic information is a simple yet effective strategy for deriving robust and interpretable relations. We develop a novel quantitative evaluation metric for understanding international relations and in-depth human evaluation also confirms the effectiveness of our model. We further show that our models can provide the background of relations using attention score and reveal regional differences for future studies on media framing.

Meanwhile, our work suggests important future directions for using NLP technologies to support individuals in navigating a large collection of news articles. Our participants often find it challenging to infer information simply from temporal dynamics of our inferred relations based on natural language descriptors. It is thus important to incorporate human cognitive preferences in developing such models and provide narratives beyond words such as key events. Furthermore, different populations pay attention to different parts of information. We need to understand the diversity when developing NLP technologies for end users and provide helpful personalized hints to lower the barrier of benefiting from model outputs.

## Acknowledgments

We thank the anonymous reviewers, Dallas Card, and Mohit Iyyer for helpful feedback and discussion. We thank all the participants in our human evaluation from LING 3813 at Georgia Tech by Lelia Glass, students at X-lab at University of Washington, and undergraduate students at University of Colorado Boulder. We also thank Andy Baker and Sven Steinmo for recruiting students in their classes. Choi is supported by a Facebook Fellowship.

## References

* Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In _ICLR_.
* Baylis et al. (2017) John Baylis, Steve Smith, and Patricia Owens. 2017. _The globalization of world politics: an introduction to international relations_. Oxford University Press.
* Blei et al. (2003) David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent dirichlet allocation. _Journal of machine Learning research_, 3(Jan):993-1022.
* Card et al. (2015) Dallas Card, Amber E. Boydstun, Justin H. Gross, Philip Resnik, and Noah A. Smith. 2015. The media frames corpus: Annotations of frames across issues. In _ACL_.
* Chambers et al. (2015) Nathanael Chambers, Victor Bowen, Ethan Genco, Xisen Tian, Eric Young, Ganesh Harihara, and Eugene Yang. 2015. Identifying political sentiment between nation states with social media. In _EMNLP_.
* Daney et al. (2016) Allison June-Barlow Chaney, Hanna M. Wallach, Matthew Connelly, and David M. Blei. 2016. Detecting and characterizing events. In _EMNLP_.
* Chang et al. (2009) Jonathan Chang, Jordan L. Boyd-Graber, and David M. Blei. 2009. Connections between the lines: augmenting social networks with text. In _KDD_.
* Chaturvedi et al. (2017) Snigdha Chaturvedi, Mohit Iyyer, and Hal Daume. 2017. Unsupervised learning of evolving relationships between literary characters. In _AAAI_.
* Choi et al. (2016) Eunsol Choi, Hannah Rashkin, Luke S. Zettlemoyer, and Yejin Choi. 2016. Document-level sentiment inference with social, faction, and discourse context. In _ACL_ 

* Chong and Druckman (2007) Dennis Chong and James N Druckman. 2007. Framing theory. _Annual Review of Political Science_, 10:103-126.
* Conneau et al. (2017) Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In _EMNLP_.
* Das et al. (2015) Rajarshi Das, Manzil Zaheer, and Chris Dyer. 2015. Gaussian lda for topic models with word embeddings. In _ACL_.
* Doddington et al. (2004) George R Doddington, Alexis Mitchell, Mark A Przybocki, Lance A Ramshaw, Stephanie Strassel, and Ralph M Weischedel. 2004. The automatic content extraction (ace) program-tasks, data, and evaluation. In _LHEC_.
* Elson et al. (2010) David K. Elson, Nicholas Dames, and Kathleen McKeown. 2010. Extracting social networks from literary fiction. In _ACL_.
* Entman (1993) Robert M Entman. 1993. Framing: Toward clarification of a fractured paradigm. _Journal of communication_, 43(4):51-58.
* Fellbaum (1998) Christiane Fellbaum. 1998. _WordNet: An Electronic Lexical Database_. Bradford Books.
* Field et al. (2018) Anjalie Field, Doron Kliger, Shuly Wintner, Jennifer Pan, Dan Jurafsky, and Yulia Tsvetkov. 2018. Framing and agenda-setting in russian news: a computational analysis of intricate political strategies. In _EMNLP_.
* Frermann and Szervas (2017) Lea Frermann and Gyorgy Szervas. 2017. Inducing semantic micro-clusters from deep multi-view representations of novels. In _EMNLP_.
* Gardner and Krishnamurthy (2017) Matt Gardner and Jayant Krishnamurthy. 2017. Open-vocabulary semantic parsing with both distributional statistics and formal knowledge. In _AAAI_.
* Glorot and Bengio (2010) Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In _AISTATS_.
* Gruber et al. (2007) Amit Gruber, Yair Weiss, and Michal Rosen-Zvi. 2007. Hidden topic markov models. In _AISTATS_.
* Handler and O'Connor (2017) Abram Handler and Brendan T. O'Connor. 2017. Rookie: A unique approach for exploring news archives. In _Workshop on Data Science + Journalism at KDD_.
* Honnibal and Montani (2017) Matthew Honnibal and Ines Montani. 2017. spacy 2: Natural language understanding with bloom embeddings, convolutional neural networks and incremental parsing.
* Iyyer et al. (2016) Mohit Iyyer, Anupam Guha, Snigdha Chaturvedi, Jordan Boyd-Graber, and Hal Daume III. 2016. Feuding families and former friends: Unsupervised learning for dynamic fictional relationships. In _NAACL_.
* Leetaru and Schrodt (2013) Kaley Leetaru and Philip A Schrodt. 2013. Gdelt: Global data on events, location, and tone, 1979-2012. In _ISA annual convention_.
* Leskovec et al. (2009) Jure Leskovec, Lars Backstrom, and Jon Kleinberg. 2009. Meme-tracking and the dynamics of the news cycle. In _KDD_.
* Mintz et al. (2009) Mike Mintz, Steven Bills, Ron Snow, and Dan Juraffsky. 2009. Distant supervision for relation extraction without labeled data. In _ACL_.
* Niculae et al. (2015) Vlad Niculae, Caroline Suen, Justine Zhang, Cristian Danescu-Niculescu-Mizil, and Jere Leskovec. 2015. Quotus: The structure of political media coverage as revealed by quoting patterns. In _WWW_.
* Nicolae et al. (2014) Vlad Niculae, Marcos Zampieri, Liviu P. Dinu, and Alina Maria Ciobanu. 2014. Temporal text ranking and automatic dating texts. In _EACL_.
* O'Connor et al. (2013) Brendan O'Connor, Brandon M Stewart, and Noah A Smith. 2013. Learning to extract international relations from political context. In _ACL_.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In _EMNLP_.
* Rashkin et al. (2017) Hannah Rashkin, Eric Bell, Yejin Choi, and Svitlana Volkova. 2017. Multilingual connotation frames: A case study on social media for targeted sentiment analysis and forecast. In _ACL_.
* Riedel et al. (2010) Sebastian Riedel, Limin Yao, and Andrew McCallum. 2010. Modeling relations and their mentions without labeled text. In _ECML_.
* Schein et al. (2015) Aaron Schein, John Paisley, David M Blei, and Hanna Wallach. 2015. Bayesian poisson tensor factorization for inferring multilateral relations from sparse dyadic event counts. In _KDD_.
* Srivastava et al. (2016) Shashank Srivastava, Snigdha Chaturvedi, and Tom M. Mitchell. 2016. Inferring interpersonal relations in narrative summaries. In _AAAI_.
* Tan et al. (2017) Chenhao Tan, Dallas Card, and Noah A. Smith. 2017. Friendships, rivalries, and trysts: Characterizing relations between ideas in texts. In _ACL_.
* Tan et al. (2018) Chenhao Tan, Hao Peng, and Noah A Smith. 2018. " you are no jack Kennedy": On media selection of highlights from presidential debates. In _WWW_.
* Wang et al. (2016) Alex Wang, William L. Hamilton, and Jure Leskovec. 2016. Learning linguistic descriptors of user roles in online communities. In _NLP+CCSS@EMNLP_.

 

Appendix

### Data: Aliases and Key Events

We use a dictionary of nation aliases to find relevant sentences in the corpus that mentioned at least one nation pair.12 Table A1 shows the full alias dictionary we used. Table A5 shows the complete list of key events used in this paper.

Footnote 12: We did not use coreference resolution to avoid noisy linking.

#### Preprocessing

We use spaCy to preprocess all sentences Honnibal and Montani (2017). We then use a rule based extractor13 to get verbal predicates with detectable subjects and objects. When verbal predicate has a negation, we take its antonym in WordNet Fellbaum (1998) if it exists, ignored otherwise. We also extract nouns and proper nouns in sentences using spaCy's part-of-speech tagging.

Footnote 13: [https://github.com/NSchrading/intro-spacy-nlp/blob/master/subject_object_extraction.py](https://github.com/NSchrading/intro-spacy-nlp/blob/master/subject_object_extraction.py)

We follow the preprocessing steps in Iyyer et al. (2016) for _RMN_. The original _RMN_, which used a fiction dataset, removes the 500 most frequently occurring words and words that occur in fewer than 100 books. We also remove the 500 most frequent words for the input to _RMN_. However, since our news dataset doesn't have the notion of "books", we remove 5000 least common words (out of a 200K vocabulary) for _RMN_. Note that this is not done for our model _LARN_.

#### Implementation Details

We show all hyperparameters used in our study in Table A2. We generally use the same hyperparameter values as _RMN_, except a few model-specific hyperparameters in Table A2. We also use Xavier initialization Glorot and Bengio (2010) for all trainable layers as in the original _RMN_.

### Relation Descriptors Postprocessing

In addition to finding the nearest neighboring words to the relation embeddings, the orginal _RMN_ also requires a manual filtering process to achieve an interpretable definition of the relation descriptor. To prevent introducing bias in this manual filtering step, we define each relation descriptor directly as its five nearest neighboring words in the input vocabulary. However, since the full vocabulary would contain many uncommon words which could hinder the interpretation of relation descriptors, we limit both models to choose descriptor words from the most frequently occuring 500 words in their own processed input vocabulary (i.e., verb predicates only for _LARN_ and all words for _RMN_).

In Table A3, we present the descriptor set by _RMN_ and _LARN_ without the most common 500 words constraint on descriptor word selection. We find that the descriptor set of our model is robust to this change and does not suffer from a lowered interpretability as much as _RMN_.

### Additional Temporal Relation Trends: US-Russia, US-India, US-Syria

See Figure A2, Figure A3, and Figure A4 for three more examples of temporal relation trends between nation pairs.

### Additional Influential Background Words:US-Russia, US-India, US-Syria

In Figure A5, Figure A6, and Figure A7, we further show the corresponding attended background words for the top relation descriptor in each of the 

three nation pair relations shown in the previous section.

### Robustness Check: Another Set of Key Events Annotations

Apart from the key events annotated by the first author, which were used in the change rate analysis in the main paper, the third author also did an independent annotation of the key events as a robustness check. The details of this annotation could found in Table A6. Figure A1 shows the change rate evaluation with respect to this annotation. A similar trend holds for both sets of annotations.

### A Simpler Baseline: Term-frequency Trend of Verbal Predicates

In addition to the comparison with _RMN_, we also checked a simpler baseline by replacing the descriptor weights with term-frequency values of each verbal predicates for each document. Specifically, the term-frequency of a predicate \(p\) in document \(d\) is \(\dfrac{count(p)}{length(d)}\). Note that _document_ refers to all sentences where a nation pair is mentioned. We show some temporal trend examples in Figure A8 and also the change rate evaluation result in Table A4. Note that the scales of these results are different from _LARN_ and _RMN_, since the predicate words are much more sparse than the 30 descriptors used in _LARN_ and _RMN_.

 

* [1] A. B. K., and S. M. R. G., (2017-04) The impact of the social distancing on the human's disease in a social network. In _Proceedings of the IEEE Conference on Human Factors in Social Networks_, pages 359-368. Springer, Berlin, Heidelberg, Germany, pp. 1-10.

[MISSING_PAGE_POST]

 



Figure A5: Influential words for US-Russia’s “denounce” relation derived from three approaches. _Top_: Showing word’s average attention score in all “denounce” articles each month. _Middle_: Showing word’s appearance frequency in all “denounce” articles each month. _Bottom_: Showing word’s average attention score multiplied by \(\log(\text{appearance})\) in each month. All figures are normalized by the global maximum score in the figure.

 

Figure A7: Influential words for US-Syria’s “denounce” relation derived from three approaches. _Top_: Showing word’s average attention score in all “denounce” articles each month. _Middle_: Showing word’s appearance frequency in all “denounce” articles each month. _Bottom_: Showing word’s average attention score multiplied by \(\log(\text{appearance})\) in each month. All figures are normalized by the global maximum score in the figure.

 

* [10] C.-H. Chen, J.-C. Lee, and D.-J. Kim, "A new approach to the identification of a single human body in an urban environment: A review," (2016). IEEE Trans. Med. Imag. **38**, no. 4, pp. 579-580.
* [11] C.-H. Chen, J.-C. Lee, and D.-J. Kim, "The role of a human body in an urban environment: A survey," (2016). IEEE Trans. Med. Imag. **38**, no. 4, pp. 101-110.

[MISSING_PAGE_POST]