An Ordinal Latent Variable Model of Conflict Intensity
Year: 2023

Abstract:
Measuring the intensity of events is crucial for monitoring and tracking armed conflict. Advances in automated event extraction have yielded massive data sets of “who did what to whom” micro-records that enable datadriven approaches to monitoring conflict. The Goldstein scale is a widely-used expert-based measure that scores events on a conflictual– cooperative scale. It is based only on the action category (“what”) and disregards the subject (“who”) and object (“to whom”) of an event, as well as contextual information, like associated casualty count, that should contribute to the perception of an event’s “intensity”. To address these shortcomings, we take a latent variablebased approach to measuring conflict intensity. We introduce a probabilistic generative model that assumes each observed event is associated with a latent intensity class. A novel aspect of this model is that it imposes an ordering on the classes, such that higher-valued classes denote higher levels of intensity. The ordinal nature of the latent variable is induced from naturally ordered aspects of the data (e.g., casualty counts) where higher values naturally indicate higher intensity. We evaluate the proposed model both intrinsically and extrinsically, showing that it obtains good held-out predictive performance. https://github.com/niklasstoehr/ ordinal-conflict-intensity

heading: 1 Introduction
text: On a scale from −10 for conflictual to +10 for cooperative, which of the following events should be considered more “intense”: “Soldiers injured two civilians” or “Rebels detained fifty soldiers”?
Measuring the intensity of events is crucial for monitoring and tracking armed conflict. Advances in the automated collection and coding of events have produced massive and systematized data sets of micro-records that enable data-driven
approaches to monitoring conflict. While the “intensity” of a given event has traditionally been assessed by human expert raters, the tremendous quantity of events collected every day makes caseby-case analysis unmanageable. As a consequence, there is a strong demand for automated and modelbased methods to aggregate events into meaningful “conflict intensity” measures.
One of the most frequently used measures is the Goldstein scale (Goldstein, 1992). Major event datasets like IDEA (Bond et al., 2003), KEDS (Schrodt, 2008), GDELT (Leetaru and Schrodt, 2013), ICEWS (Boschee et al., 2015), Phoenix (Beieler, 2016) and NAVCO (Lewis et al., 2016) all rely on it. The Goldstein scale assigns intensity scores between −10.0 and +10
4817
on a conflictual–cooperative scale to the action categories defined by the Conflict and Mediation Event Observations (CAMEO) event coding scheme (Schrodt, 2012). CAMEO specifies 204 low-level event types which are summarized into 20 high-level action categories. The Goldstein scale ranks “use unconventional mass violence” and “fight” as the most conflictual of the 20 highlevel action categories (−10.0) and “provide aid” (+7.0) as the most cooperative; see Tab. 1.
Despite its usage, the Goldstein scale has many well-known shortcomings (King and Lowe, 2003; Schrodt, 2019). In particular, it applies only to action categories, and does not account for any contextual information of a given event, like which actors are involved, or how many fatalities resulted, among other bits of context that should contribute to the perception of an event’s “intensity”.
This paper takes a latent-variable based approach to measuring conflict intensity. We introduce a probabilistic generative model that assumes each observed event n is associated with a latent intensity class zn. A novel aspect of this model is that it imposes an ordering on the classes, such that higher values of zn denote higher levels of intensity. The ordinal nature of zn is induced from naturally ordered aspects of the data (e.g., casualty counts) where higher values naturally indicate higher intensity. The model effectively learns to interpolate the ordered (i.e., cardinal or ordinal) elements of the data while inferring correlation structure with the non-ordered (e.g., categorical) elements of the data (e.g., actor types).
We start with a discussion of the Goldstein scale and introduce a political event dataset annotated with Goldstein values in §2 and §3. Then, we propose our model with an ordinal latent variable in §4. We evaluate the performance of the model intrinsically (§5) and extrinsically (§6) and find that it improves over measures based on the original Goldstein scale or heuristics based on the raw data.

heading: 2 Limitations of the Goldstein Scale
text: The Goldstein scale is a widely-used measure of the conflictual versus cooperative nature of interactions between countries (Goldstein, 1992). The scale was created by a panel of international relations experts who ranked descriptions of interactions. It was initially created to score action categories in the WEIS event ontology (McClelland, 1984) and was later adapted to CAMEO (Schrodt, 2012).
The Goldstein scale applies only to the action category of an event (e.g., “fight” or “trade”). Thus, two different “fight” events, which might involve two different pairs of actors, occur at different times, or differ dramatically with respect to the number of associated fatalities, will still be assigned the same Goldstein value.The Goldstein scale is thus a poor measure of a conflict’s perceived “intensity”, as it ignores much of the information that contributes to that perception. Recent work in conflict studies, for instance, operationalizes “intensity” primarily using casualty counts (Chaudoin et al., 2017; Zhong et al., 2023), which the Goldstein scale ignores entirely.
In Tab. 1, we show the empirical distribution of assigned Goldstein values alongside the empirical distribution of casualty counts in a dataset of conflict events. The Goldstein scale is very coarsegrained; while it ostensibly ranges between −10.0 and +10.0, only a small number of discrete values ever occur, with many action categories assigned the same value. For the purpose of measuring conflict intensity, a finer-grained and more contextual scale is desirable.

heading: 3 Conflict Event Data
text: This paper considers the publicly available Nonviolent and Violent Campaigns and Outcomes (NAVCO) data collection (Chenoweth et al., 2018), specifically, the latest release NAVCO 3.0 from November 2019 which comprises N = 112,089 events between December 1990 and December 2012. An exemplary event description is “On 19 May 2012, soldiers injured two civilians in Afghanistan”. Each part of this description has been parsed by human coders into standardized, structural features. We color-code the features that correspond to the semantic roles subject, predicate, quantifier, object, which are the focus of our modeling approach. Each data point n thus consists of a four-element tuple {sn, pn, qn, on}. We note that events are further coded for their location (in this case, Afghanistan) and time (19 May 2012), among other bits of contextual information. Let us discuss each feature in more detail:
Subject sn. NAVCO contains columns termed “actor3”, “actor6” and “actor9” which code for the subject (or agent) of a given action. The actor types are defined by the CAMEO actor codebook. We first merge the higher-level categories “actor3” and “actor6”, resulting in 33 different actor types, and
then map all actor types into one of S = 4 classes: sn ∈ {civilian,military, governmental, political}. We present our 4-class actor type mapping in Tab. 3 of the appendix.
Predicate pn (Goldstein values). NAVCO codes1 each event description into one of the 20 CAMEO action categories in the column “verb10”, which is by extension associated with a Goldstein value pn. Throughout, we refer to the Goldstein value pn as an action’s “predicate”, since there is a one-to-one mapping between action categories and Goldstein values. We scale Goldstein values pn to a [0, 1] range and invert them (i.e., pn ← 1 − pn) so that higher values close to 1 represent more conflictual action categories.
Quantifier qn (casualty counts). Each event description is annotated with human-verified fatality and wounded counts. We add the two and
1NAVCO features a 21st action category which we exclude since it is not specified by the CAMEO taxonomy and thus has no Goldstein value.
refer to the resulting value qn ∈ N+0 as an event’s “quantifier” or its “casualty count”. In Tab. 1, we give the average number of casualties associated with each action alongside its Goldstein value—as intuition might suggest, actions that Goldstein scores as more conflictual (e.g., “fight” (−10.0)) coincide with more casualties on average.
Object on. Similar to its subject, NAVCO codes for the direct object or “target” of a conflict action using the CAMEO coding scheme; these codes are found in the columns “target3” and “target6”. We map those into the O = 4 classes so that on ∈ {civilian,military, governmental, political}. Contextual information: location and time. Finally, each event is further annotated with a timestamp and location, which we use to design extrinsic evaluation tasks in §6.

heading: 4 Ordinal Latent Variable Model
text: We operationalize conflict intensity as a latent variable that expresses the association between the observed variables subject (sn), predicate (pn), quantifier (qn) and object (on). Each data point is a tuple {sn, pn, qn, on} representing an event. Our Bayesian latent variable model is depicted in Fig. 2. We assume the following generative story. For each event n, we assume that its event intensity class zn ∈ {1, . . . , C} is a Categorical random variable
zn ∼ Categorical(π(z)) (1)
where π(z) is a C-dimensional discrete distribution over latent intensity classes. We place a Dirichlet prior over π(z)
π(z) ∼ Dirichlet(α(z)) (2)
with concentration parameter α(z) ∈ RC+. Conditioned on zn, we assume each of the observed sites per event tuple sn, pn, qn and on are then drawn as
sn | zn ∼ Categorical(π(s)zn ) (3) pn | zn ∼ Beta(ω(p)zn , κ(p)zn ) (4) qn | zn ∼ Zero-infl. Geom.(δ(q)zn , b(q)zn ) (5) on | zn ∼ Categorical(π(o)zn ) (6)
Here π(s)zn and π (o) zn are the discrete distributions for class zn over S subject and O object types, respectively. They are given as row vectors in the matrices Π(s) ∈ (0, 1)C×S and Π(o) ∈ (0, 1)C×O that zn
tors, as indicated by double-border nodes ( 1 ) which leads the latent zn to represent ordinal intensity classes.
indexes into. We place a simple symmetric Dirichlet prior over all rows—e.g., π(s)c ∼ Dirichlet(1S) for c ∈ {1, . . . , C}. The scalar parameters ω (p) zn , κ (p) zn , δ (q) zn , b (q) zn are similarly selected by zn from C-dimensional vectors ω(p),κ(p), δ(q),b(q) as discussed below.

heading: 4.1 Ordinal Latent Variable
text: We want the latent variable zn to be ordinal, such that higher-valued classes correspond to higher intensity levels. However, in the model thus described, zn is a Categorical random variable whose classes are not inherently ordered. So how does this model encode an ordinal zn?Unlike the subject sn and object on, which are categorical, the Goldstein value pn and casualty count qn are cardinal quantities whose magnitudes naturally indicate the “intensity” of a given event. To capture this intuition, we first assume that pn and qn are drawn from Beta (eq. (4)) and Zero-inflated Geometric distributions (eq. (5)), respectively, whose parameters are indexed by zn. We then impose an ordering on the parameters, such that higher classes (e.g., zn = c) correspond to higher-valued parameters (e.g., ω (p) c > ω (p) c−1) which in turn encourage larger values of the observed cardinal quantities (e.g., pn). 2
2We note that one might also impose ordering on casualty types (e.g., an event with civilian casualties might be consid-
Ordered Normal To flexibly impose ordering on vectors of parameters, we first define the Ordered Normal prior (Stoehr et al., 2023a). An ordered vector λ = (λ1, . . . , λC), where λc > λc−1, is a Cdimensional Ordered Normal random variable λ ∼ OrderedNormalC(µ, σ) if it is sampled according to the following generative process:
xc iid∼ Normal(µ, σ) for c ∈ {1, . . . , C} (λ1, . . . , λC)← Ord({x1, . . . , xC}) (7)
where Ord(·) takes in an unordered set of numbers {x1, . . . , xC}, and transforms it into a vector whose components are in strictly increasing order,
λc ← { x1 if c = 1 x1 + ∑c i=2 exp(xi) if c > 1 (8)
This transformation is invertible and differentiable and thus does not obstruct gradient-based inference of model parameters (App. B).
Ordered Beta. We model Goldstein values pn as Beta random variables. For intensity class zn = c, we assume pn ∼ Beta(ω(p)c , κ(p)c ), where ω(p)c ∈ (0, 1) is the mode and κ(p)c > 0 is the concentration parameter. We impose an ordering on the modes by positing a transformed OrderedNormal prior:
S−1(ω(p)) ∼ OrderedNormalC(µ, σ) (9)
where ω(p) = (ω(p)1 , . . . , ω (p) C ) is the Cdimensional ordered vector of modes and S−1(·) is the element-wise inverse sigmoid function. To ensure that each class-conditional Beta distribution is unimodal around its mode ω(p)c , we then mandate that the concentration parameter κ(p)c is greater than 2 by imposing a shifted Gamma prior.
(κ(p)c − 2) iid.∼ Gamma(k, η) (10)
where k and η are the shape and rate parameters. We note that while the elements of ω(p) are ordered, those of κ(p) are not.
Ordered Zero-inflated Geometric. We model the casualty counts qn as Zero-inflated Geometric random variables. For intensity class zn = c, we assume qn ∼ Zero-infl. Geom.(δ(q)c , b(q)c ) where δ (q) c ∈ (0, 1) is the “gate” parameter—i.e., the inflated probability of sampling a zero—and b(q)c ∈ ered more “intense” than one with military casualties). In this work, however, we focus just on learning scales from observations that are naturally cardinal.
(0, 1) is the success probability parameter under the standard Geometric distribution. We impose an ordering on both C-dimensional vectors of parameters δ(q) and b(q) using the same transformed Ordered Normal prior as in the previous subsection:
S−1(δ(q)) ∼ OrderedNormalC(µ, σ) (11) S−1(b(q)) ∼ OrderedNormalC(µ, σ) (12)
Importantly though, we reverse the ordering of λ such that for all c ∈ {1, . . . , C} : λc < λc−1. The reason is that higher δ(q)c and b (q) c correspond, on average, to lower values sampled from the Zeroinflated Geometric and thus lower casualty counts and lower event intensity.

heading: 4.2 Posterior Inference
text: We implement the model using the probabilistic programming framework Pyro (Bingham et al., 2018). Pyro provides an OrderedTransform function that implements eq. (8). We approximate the posterior distribution of our model’s parameters using the No-U-Turn Sampler (NUTS; Homan and Gelman, 2014), a variant of Hamiltonian Monte Carlo. NUTS is gradient-based and requires the parameters to be continuous. However, we explicitly model our latent variable zn to be ordinal. To solve this problem, Pyro provides parallel_enumeration which marginalizes out discrete variables numerically during inference.
We refer to all continuous parameters of our model as θ = {π(z),Π(s),ω(p),κ(p), δ(q),b(q),Π(o)} and to the full dataset of event tuples as X . NUTS produces samples of θ from the posterior θ(t) ∼ p(θ | X). We can further sample the ordinal latent z(t)n ∼ p(zn | θ(t), X) using Pyro’s infer_discrete. Based on the samples {z(t)n }Tt=1 per event n, we can then compute a point estimate of the event’s ordinal intensity either by taking the mean z̄n = 1T ∑T t=1 z (t) n or the mode (i.e., the most frequently sampled class) żn.
Label Switching. It is typically not meaningful to compute z̄n for mixture and admixture models due to the problem of label switching (Stephens, 2000), where the labels of the latent classes may switch between Markov chain Monte Carlo (MCMC) iterations. However, in the proposed model, the ordering transformation presented in eq. (8) represents an identifiability constraint that fixes the meaning of zn = c—i.e., c is the class that is more intense than c − 1 and less intense than c + 1. This alleviates the problem of label switching, and permits us to meaningfully average posterior samples z(t)n .
Practical Details. Our model has 13× C trainable parameters, where C is the number of latent classes. When fit to the full dataset introduced in §3, our implementation generates 1,000 posterior samples, for C = 5, on a CPU with 64 GB of RAM in less than 10 minutes. Throughout we set hyperparameters to uninformative values: µ = −1.0, σ = 1.0, k = 1.0 and η = 1.0.

heading: 5 Intrinsic Evaluation: Imputation
text: To evaluate the fit of our model, we conduct a series of predictive experiments. We randomly split the N = 112,089 events into a 70% training and 30% held-out set. We fit the model on the training set X and approximate the posterior distribution of continuous parameters θ based on T = 1,000 NUTS samples after discarding the first 200 burn-in samples. For evaluation on the held-out set Y , we report scaled pointwise predictive density (SPPD), mean squared error and weighted F1 score as detailed in App. A.

heading: 5.1 Number of Latent Classes C
text: The number of classes C that our latent variable zn ranges over can be regarded a hyperparameter. To identify the optimal setting of C, we repeatedly
fit the model on the training set and evaluate it on the held-out set. We do so for each class setting in C = [3, . . . , 7] and find that C = 5 yields the best model fit overall, as shown in Fig. 3.

heading: 5.2 Imputation Task
text: Procedure. Next, we conduct a data imputation task. On the held-out set, we remove one observed site from all event tuples and refer to it as yn — e.g., we remove all Goldstein values and obtain yn = {pn}. We use the remaining three-way event tuples yremainn = {sn, qn, on} jointly with the posterior of θ to make predictions of yn. In particular, we draw samples from the posterior predictive distribution ŷ (t) n ∼ p(yn | θ(t), yremainn ) and compute their mean ŷn = 1 T ∑T t=1 ŷ (t) n to get point estimates of yn.
Baselines. We compare the proposed model against three baselines, where the first two baselines are simple and we expect our proposed model to outperform them. For the first baseline, we fit four versions of the proposed model to only one of the observed sites {sn, pn, qn, on}. This naive baseline effectively just learns the empirical histogram of each site in the training set, and uses the relevant one to make predictions for yn — we refer to this as naive. For the second baseline, we use an unfitted version of the proposed model, whose parameters have been sampled from the prior. For the last baseline, we train four linear regression models (LR) to predict yn directly from yremainn .
Results. Fig. 4 reports the results of the imputation experiments, where the proposed model performs comparatively well, obtaining substantially better predictive performance over both the naive and non-naive baselines.

heading: 6 Extrinsic Evaluation: Time Series
text: In the last section, we intrinsically evaluated the proposed model by having it impute held-out data. In this section, we consider extrinsic evaluations of the proposed model, which seek to evaluate it on downstream predictive tasks for which it was not explicitly designed. In this section, we rely on features of events coded by NAVCO which our model does not access during training. In particular, NAVCO codes for the location ℓn and the timestamp τn of each event. We discretize time so that each event is associated with a specific month mn. Augmented with these extra characteristics, as well as with the mean intensity inferred by our model, each event n has the following attributes: {sn, pn, qn, on, ℓn,mn, z̄n}. In all of the tasks in this section, we split events according to their location ℓn = ℓ and examine monthly-aggregated time series of the cardinal and ordinal quantities {pn, qn, z̄n}. In particular, we ask two questions: 1) whether z̄n adds predictive information for forecasting pn and qn, and 2) whether the model-based measure of intensity z̄n correlates with Google Trends more than just pn and qn.
Event Time Series. We construct time series in the following way. We first run inference on a training dataset X to obtain posterior samples of the parameters {θ(t)}Tt=1. We then use the trained parameters to obtain intensity scores {z(t)}Tt=1 for both events in the training set as well as events in a test set that the model did not access. We then compute z̄n for each n and re-scale the z̄n estimates to cover a [0, 1] range as we did with Goldstein values. For each location ℓ and month m appearing in the data, we then construct the
posterior average intensity class z̄ℓ,m:
z̄ℓ,m = ∑N n=1 z̄n 1(ℓn = ℓ)1(mn = m)∑N n=1 1(ℓn = ℓ)1(mn = m) (13)
For a given location ℓ, we refer to the full time series over months as z̄ℓ and linearly interpolate the entries m corresponding to months for which there is no data. We similarly aggregate the observed sites per event to obtain time series of predicates p̄ℓ and quantifiers q̄ℓ.
Autoregressive Forecasting. Does knowledge of the inferred intensity time series z̄ℓ improve forecasting of the Goldstein p̄ℓ and casualty count q̄ℓ time series? To test this, we first consider autoregressive (AR) models that forecast monthly values of each time series (e.g., p̄ℓ,m) based on the previous values of only that same time series (e.g., p̄ℓ,1:(m−1)). We then consider vector autoregressive (VAR) models that use multiple time series (e.g., p̄ℓ,1:(m−1) and q̄ℓ,1:(m−1)) to form the same predictions. When incorporating values from our latent intensity time series z̄ℓ, we must be careful to avoid test set leakage. To do so, we fit our model to a subset of data X\ℓ that excludes any events in a particular location ℓ. We obtain posterior samples {θ(t)\ℓ }Tt=1 by fitting the proposed model to X\ℓ. We then use these parameters to obtain z̄n for all events n in both the training X\ℓ and test set Xℓ.
Before fitting any (vector) autoregressive models and performing the forecasting experiments, we apply a first-order differencing transform to all time series to remove potential linear trends, and verify each time series’ stationarity using an Augmented Dickey-Fuller test (Fuller, 1976). We then fit an autoregressive (AR) model to each individual time-
series and a vector autoregressive (VAR) model to all pairs of time series. We determine their optimal orders (lag) in months using Bayesian Information Criterion (BIC), and use cross validation to measure held-out forecasting error across 24 folds. In Tab. 2, we report results in mean squared error (MSE) on the held-out sets averaged over all 24 folds. Indeed, our results suggest that z̄ℓ contains (predictive) information on both p̄ℓ and q̄ℓ. For instance, in the VAR experiments for forecasting p̄ℓ, we may replace the additional time series q̄ℓ with z̄ℓ without any drop in performance. We also report Granger tests (Granger, 1969) which test the null hypothesis that forecasting a variable (e.g., p̄ℓ) using only its own history is no less accurate than also using an additional variable’s history (e.g., z̄ℓ).
Descriptive Analysis. A growing body of work analyzes the correlation between shifts in online behavior or media attention and conflict intensity (Chykina and Crabtree, 2018; Timoneda and Wibbels, 2022). Following this line of work, we download time series of Google search keywords using the Google Trends Anchorbank (G-TAB) (West, 2020). We constrain searches to the category of “World News” and use the country name of location ℓ as the search keyword. In contrast to the forecasting setting, we fit our model to the entire dataset X , including events associated with the respective location, and obtain an intensity time series z̄ℓ. Fig. 5 shows a comparison between four time series: the Goldstein p̄ℓ, casualty counts q̄ℓ, our latent z̄ℓ and the Google trends time series for Syria between 2004 and 2013.3 We observe that
3Fisher (2012) reports a correlation between Google search volume and activities in the Syrian Civil War for this period.
z̄ℓ nicely interpolates between the time series of p̄ℓ and q̄ℓ. It captures larger trends and fluctuations in both time series, while not exactly mirroring either. Further, in Fig. 6, we report Pearson correlation coefficients between all pairs of time series. While p̄ℓ and q̄ℓ are positively correlated, p̄ℓ is more strongly correlated with z̄ℓ than with q̄ℓ. Moreover, Google trends are more strongly correlated with z̄ℓ than either p̄ℓ or q̄ℓ. We hypothesize this may be due
2004 is the earliest date to query Google trends.
to the additional information on the subject sn and object types on that zn encodes, and which may contribute to how much attention is paid to different conflicts, as measured by Google Trends.

heading: 7 Related Work
text: There are a number of relevant papers that broadly seek to measure “latent concepts” (Douglass et al., 2022) pertaining to international relations, particularly using event data. Terechshenko (2020) uses a Bayesian item-response theory model to learn ordinal conflict intensity levels from observed event types. O’Connor et al. (2013) present an unsupervised, probabilistic topic model to learn an ontology of event data and use the Goldstein scale to evaluate it. Schein et al. (2015, 2016, 2019) decompose four-way tensors (senders, receivers, actions, time steps) to infer latent classes of CAMEO-coded events. Stoehr et al. (2023b) build on those models by further imposing an ordering on their latent space, which captures conflict– cooperation intensity. Another line of work models friend–enemy relationship trajectories using neural network-based (Han et al., 2019) or hidden Markov model-based (Chaturvedi et al., 2017) approaches. There also exists work on signed network representations of relationships that are extracted from text (Srivastava et al., 2016; Choi et al., 2016) or Wikipedia conflict articles (Stoehr et al., 2021).

heading: 8 Future Work
text: Assuming civilian casualties are “more intense” than military casualties, we could impose an additional ordering on object types on. We could also condition observed sites on each other—e.g., casualty counts qn are conditionally dependent on on or even pn under the model. We plan to further incorporate multiple latent variables to model multi-dimensional intensity concepts. Future models could also condition on location and include a temporal component to account for how surprisal may affect the perceived intensity of an event. We also note that the latent variable model presented in this work could also be extended to a more general framework for learning interpretable, ordinal scales from a set of mixed-type data that include cardinal or ordinal observations. We plan to explore generalizations of the proposed model and applications beyond of international relations.

heading: Acknowledgments
text: We would like to thank Giuseppe Russo and Benjamin Radford for helpful discussions. Moreover, we acknowledge the feedback received from the anonymous reviewers, the NLP groups at ETH Zurich and the DLAB at EPFL. NS acknowledges support from the Swiss Data Science Center (SDSC) fellowship. LTH acknowledges support from the Michael Athans fellowship fund.
Limitations
We discuss limitations of our modeling assumptions in §8. They are based on prior work in political science such as the CAMEO ontology and the Goldstein scale. On this account, they may replicate or potentially introduce biases. Hyperparameter search, settings and implementation details are provided in §4.2 and §5. All NAVCO event descriptions are limited to English language, but do not disclose individuals.
Impact Statement
We emphasize that our models are intended for research, analysis and monitoring purposes. They should not be blindly deployed for automatized decision-making processes. The notion of conflict intensity is intrinsically hard to quantify: it is strongly dependent on socio-cultural background and subjective experience.

heading: A Evaluation Metrics
text: To evaluate the posterior predictive distribution p(yn | θ(t)), we consider a scaled variant of the log pointwise predictive density (LPPD; Gelman et al., 1996, 2014), which we term SPPD:
exp
( 1
N
N∑
n=1
log ( 1 T T∑
t=1
p(yn | θ(t)) ) ) (14)
The term inside the log, 1T ∑T
t=1 p(yn | θ(t)), computes a Monte Carlo approximation to the posterior predictive density for a given yn. The rest, exp ( 1 N ∑N n=1 log(·) ) , then computes the geometric mean of the pointwise densities. We also compute point estimates via the posterior predictive mean 1T ∑T
t=1 E[yn | θ(t)] which allows comparing predicted and true values based on error metrics like weighted F1 score or meansquared error (MSE).
B Inverse of the Ordered Normal
We define the Ordered Normal distribution with the help of an ordering transformation in eq. (8). This transformation is a smooth bijection since its inverse is given by:
xc ← { λ1 if c = 1 log(λc − λc−1) if c > 1
(15)
Note that for all c, λc > λc−1, so the log is welldefined.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
section 9 and limitations section after conclusion
3 A2. Did you discuss any potential risks of your work? section 9, limitations section and impact statement after conclusion
3 A3. Do the abstract and introduction summarize the paper’s main claims? abstract and section 1
7 A4. Have you used AI writing assistants when working on this paper? –
B 3 Did you use or create scientific artifacts? section 3 and section 7
3 B1. Did you cite the creators of artifacts you used? section 3 and section 7
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? section 3, section 7 and appendix
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? section 1, section 3 and section 7
3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? section 3, section 7, limitations section and impact statement after conclusion
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? section 3 and appendix
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. section 3, section 6 and appendix
C 3 Did you run computational experiments? section 4, section 5 and section 6
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? section 4, section 5 and appendix
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? section 4, section 5 and appendix
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? section 2, section 4, section 5, section 6 and appendix
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. did not use existing packages
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: Future models could also condition on location and include a temporal component to account for how surprisal may affect the perceived intensity of an event. We also note that the latent variable model presented in this work could also be extended to a more general framework for learning interpretable, ordinal scales from a set of mixed-type data that include cardinal or ordinal observations. We plan to explore generalizations of the proposed model and applications beyond of international relations