Testing the Power of Arguments in Referendums
A Bradley-Terry ApproachPeter John Loewen∗

Daniel Rubenson†

Arthur Spirling‡

February 7, 2011

Abstract
How can we determine which arguments in a referendum are most persuasive? We show thatthe Bradley-Terry model has several features that make it well-suited to this task, and thus
preferable to other, more conventional approaches. Using a survey experiment conductedduring an electoral reform referendum in Ontario, Canada in October 2007, we demonstrate
how unstructured and structured Bradley-Terry models can be straightforwardly fitted andinterpreted. In doing so, we gain insight into the factors which determine support for electoral
reform. We identify a status quo bias and find that power varies with mention of fairness, local
control over candidate selection, and the role of political parties. We conclude by discussing
the limits, extensions and further applications of such models in electoral studies and politicalscience more broadly.

∗

University of Toronto. peter.loewen@utoronto.ca
Ryerson University. rubenson@ryerson.ca‡
Harvard University. aspirling@gov.harvard.edu†

1Introduction

Scholars of politics have long been centrally concerned with assessing the persuasive power ofarguments (see, e.g., Aristotle (322BC/1991) for an ancient treatment and e.g. Mutz, Sniderman,
and Brody (1996) for a more recent discussion). For example, does a call for greater social welfare
spending which appeals to an individual’s sense of equality elicit greater support than an appeal
based on self-interest? Or, in a more oppositional manner, if a liberal argument for higher taxes
is pitted against a conservative argument for lower taxation, which garners more support? By
exposing respondents to various arguments in a controlled setting, we can learn much about thefoundations of preferences, as well as rhetorical and political success in real world arenas.

Elections are perhaps the most obvious arena in which issues and arguments compete (see, e.g.,
Page 1978; Erikson and Wright 1997; Ansolabehere, Snyder, and Stewart 2001). However, in
this case, issues and arguments are often ‘bundled’ together such that it becomes diﬃcult todetermine the eﬀects of such messages independent of candidates. A more systematic discussion of positions is seen in referendums—formal plebiscites in which voters directly convey theirpreferences on a binary choice that may concern anything from a new policy commitment to potentially profound constitutional changes. Since so much is at stake in such national reform votes,
political scientists have an obvious interest in attempting to understand why citizens choose theway they do. In some senses, this exercise ought to be empirically easier for referendums since,
relative to elections, such votes are depersonalized: the candidate is an inanimate proposition,rather than an individual (or collection of individuals) (Pattie, Denver, Mitchell, and Bochel
1998; Tranter 2003). Voters thus receive more focused and explicit discussions of the issue atstake, with party and other (partisan) loyalties sometimes at cross-cutting angles.1 As a result,scholars have direct access to the explicit arguments and types of reasoning used to buttress aposition, rather than having to piece together the more rhetorical, nebulous and eclectic sources
1
Consider, for example, the UK’s EEC referendum vote of 1975: Prime Minister Wilson and most of his cabinet
united with the majority of Margaret Thatcher’s Conservatives to recommend a ‘yes’ vote. Meanwhile, a left-wing
contingent of Labour MPs—and some right-wing Tories—campaigned for a ‘no’.

2

of appeal that a particular party or politician uses. Using such political arguments and voter
responses—observationally or experimentally—analysts ought to be able to teach us much about
the outcomes of these events.

We present a method to measure the power of such arguments. By ‘power’, we are referringto the ability of an argument to elicit agreement with its advocated position on an issue in theface of an argument from the ‘other side’. Such a deﬁnition implies the arguments exist in opposition to one another, and that an argument’s power depends not on its ability to defeat anothersingle, particular argument, but its ability to best a range of arguments making the case foran alternative position. Unfortunately, analysts of referendums face pronounced methodologicalproblems when attempting to assess the persuasive ‘power’ of diﬀerent political arguments. First,
in observational studies, it is not always apparent precisely which arguments voters have received.
Precisely measuring probable exposure is diﬃcult and asking voters to self-report their exposure
to arguments is problematic. Another solution is to take an experimental approach within asurvey, and have subjects receive one of k total contrasting arguments, followed by a report onwhich argument they agreed with. For increasing values of k, however, the size of the relevantsample on which inferences are based is quickly reduced. Moreover, exposing each subject toa large number of arguments and then soliciting opinions on which is ‘most convincing’ risks alearning or cumulative eﬀect where responses are a function of previous and current treatmentsin the series, rather than the content of the argument per se.

These two problems are exacerbated if we wish to pit particular arguments against one another in order to mimic the way debates take place in practice. Consider the case where we havem arguments in favour of a position, and the same number against. Randomly assigning a totalof T respondents to receive one pro argument and one con argument yields just n = mT2 subjects
per treatment. For a sample of, say, 500 individuals picking between, say, 6 pairs of arguments,
n < 15: that is, there are less than 15 respondents per condition! With such small numbers, we

3

would expect low statistical power to be an issue when attempting to draw conclusions about
the relative ability of arguments. And, of course, assigning multiple treatments risks the learning/fatigue eﬀect noted above.

We suggest an innovative solution to this problem: a novel application of the Bradley-Terrymodel of pairwise comparisons (Bradley and Terry 1952). This model has a unique value proposition. It can give survey researchers a comparatively large amount of information in a small
amount of survey space. By way of example, we use a survey experiment of just 520 respondents
in which respondents are assigned one of six arguments for and six against electoral reform (creating 36 treatment groups). By conceptualizing arguments as contestants engaged in bouts withother arguments, we can use the model to estimate the probability that an argument wins. Thatis, we wish to compare the ‘abilities of players’—in our case, the ‘power of arguments’—in order
to assess whether and why some arguments are better able to persuade (or ‘win’) when pitted
against an opposing argument.

By applying the Bradley-Terry model, we can rank-order arguments according to their power
to persuade. Moreover, we can also determine the sources of an argument’s power, i.e. the components that make it more persuasive than another argument. We argue that the utility of thismethod exceeds more traditional approaches. Our principal motivation, then, is to introduce
these models to election researchers. A secondary motivation is to introduce these models topolitical scientists more generally. As we show in the ﬁnal section of our paper, these models canbe applied to a wide range of substantive problems which are characterized by contests between
individual actors.

Substantively, we are motivated by the question of which arguments for and against electoral
reform are most powerful. To this end, we conducted a survey experiment in the midst of aprovince-wide referendum on whether to change the electoral system in Ontario, Canada. This4

new data consists of respondents randomly assigned to receive one of six arguments in favor of
the existing First Past the Post (FPTP) electoral system and one of six arguments in favor of the
proposed Mixed Member Proportional (MMP) system. By knowing which arguments increase theappeal of one electoral system over another and by knowing the sources of arguments’ power we
can gain important insight into support or opposition to electoral reform. By understanding why
certain arguments are more powerful, we also contribute to more general knowledge in electoral
studies and political science more broadly. In doing so, we also take a route distinct from earlier
studies of referendums that focus on voters rather than arguments (e.g. Pattie, Denver, Mitchell,and Bochel 1998; Tranter 2003; Pammett and LeDuc 2001; Hobolt 2009). Such an approach canthus compliment work that focusses on the determinants of individuals’ decisions.

We proceed as follows. In the next section we describe our experimental design and data. In
Section 3 we introduce our novel statistical approach for pairwise comparison, the Bradley-Terrymodel. In Sections 4 and 5 we describe our results. We conclude by suggesting other questions
to which this model might be applied in political science.

2

Data and Experimental Design

Our experiment occurred within the context of a referendum on electoral reform conducted in theprovince of Ontario, Canada, in October 2007. The referendum was conducted concurrently witha provincial election. The Ontario MMP proposal lost the ensuing referendum, garnering just
38% of the vote and a majority in only ﬁve districts. It was not a loss directly attributable to the
proposing government, as they won the concurrent election rather overwhelmingly. Thus, thisloss was much to the consternation of electoral reform advocates who believed that the arguments
for electoral reform were clearly superior to those in support of the current system. Indeed, in thedays following the referendum, advocates claimed that had the public been better educated on
the proposed system, especially its underlying values, support would have been higher, perhaps

5surpassing the 60% threshold required for reform (Fenlon 2007).

To put claims that the arguments for the MMP system are more powerful upon exposure thanthose for the FPTP system to the test, we developed six arguments in favor of each position.
These are drawn from campaign materials produced by both sides of the campaign, conversations with advocates on both sides and academic literature on electoral systems (e.g. Blais and
Massicotte 2002; Powell 2000).

To assess the comparative power of arguments, we conducted an online survey experiment with520 voting-aged Ontarians in the last week of the referendum campaign. Our experimental
module was contained at the end of the survey, after three sections related to federal politics,
provincial politics and environmental issues. No questions prior to the experimental module wererelated to electoral reform. A proﬁle of our subjects is found in Appendix A.

In the experiment, subjects read: “As you may know, there will also be a referendum during the October 10th election. The purpose of the referendum is to determine whether Ontario
should change its electoral system from the current ﬁrst past the post system to a mixed member
proportional system. We’d like to present you with an argument for each system and then getyour view on which system you prefer.” Respondents were then presented with one of six arguments in favor of the existing system and one of six arguments in favor of the proposed system.
The order of the arguments was also randomized. After receiving the arguments, respondents
were then asked to indicate their preference between MMP and FPTP; they were not able togive a ‘Don’t know’ response and had to indicate a preference before proceeding to the end of
the survey. Notice that each respondent is ‘used’ only once in the study: they receive one MMP
argument, and one FPTP argument.

The six arguments for FPTP and MMP were:

6

FPTP
– A ﬁrst past the post system is better because it creates strong majority governmentsthat can implement their policies (FPTP1).
– A ﬁrst past the post system is better because it makes sure that every Member ofProvincial Parliament is elected from a constituency (FPTP2).
– A ﬁrst past the post system is better because one ballot is less confusing than two.
(FPTP3)– A ﬁrst past the post system is better because it allows local party members to choose
all of a party’s candidates. (FPTP4)– A mixed member proportional system is worse because it will lead to unstable coalition
governments. (FPTP5)– A mixed member proportional system is worse because it puts too much control in
the hands of parties and political elites. (FPTP6)

MMP– A mixed member proportional system is better because it makes sure that parties
that have support of some of the population still get some representation. (MMP1)– A mixed member proportional system is better because it lets voters indicate theirpreference for both a local representative and a party. (MMP2)– A mixed member proportional system is better because parties should get the same
share of seats as their share of the vote. (MMP3)– A mixed member proportional system is better because it will lead to more diversity
in the legislature. (MMP4)– A ﬁrst past the post system is worse because it gives some parties a share of the seats
much larger than their share of the vote. (MMP5)– A ﬁrst past the post system is worse because it shuts out small parties. (MMP6)
As the pro and con arguments were assigned randomly and independently, we have 36 approximately equal sized groups.2

We are not only interested in the power of arguments, but also their sources of power. In
other words, what is it about the content of arguments that makes some more powerful than oth2

A χ2 test indicates that the assignments are independent of one another (χ2 = 26.41 p = .39).

7

ers? We note that several arguments share common components. For example, some arguments
reference local candidates, others appeal to notions of proportionality or fairness, others reference
government stability and eﬀectiveness, others highlight the strengths or weaknesses of the ballot
structure in FPTP and MMP. Many make reference to political parties. Some arguments arephrased in positive terms (i.e. a ﬁrst past the post system is better because...), while others arephrased negatively (i.e. a mixed member proportional system is worse because...). And somearguments combine many of these features. Table 1 summarizes the characteristics of the arguments. We are thus not only interested in which arguments are more powerful, but also what
components make one argument more persuasive than another. Accordingly, we use a structured
Bradley-Terry model (in Section 5) to attempt to identify the most persuasive arguments andthe components which make them persuasive.

3

Measuring the Power to Persuade: A Statistical Model

We contend that arguments vary in their ‘power to persuade’ citizens who receive them. The
central econometric concern is to estimate this persuasion ‘power’, possibly as a function of
the characteristics of the arguments themselves. We begin by supposing that in any two-way
comparison in which an argument pertaining to FPTP is paired with one pertaining to MMP,
one argument makes a more convincing case than the other for its respective electoral system; itis thus more likely to persuade the respondent. For any given comparison or ‘contest’ between
arguments—or ‘players’—i and j let πi,j ∈ (0, 1) be the probability that the respondent ﬁnds imore compelling than j and thus prefers the electoral system (implicitly or explicitly) promoted
by i. Write the odds that argument i ‘beats’ argument j as a function of their ‘powers’ αi , αjwhich are latent (and thus unobserved) but positive valued:
πi,j
πi,j
αi
=
=
∀i, ∀j.
πj,i
1 − πi,j
αj

8

(1)

9

MMP1
MMP2
MMP3
MMP4
MMP5
MMP6

FPTP1
FPTP2
FPTP3
FPTP4
FPTP5
FPTP6

Argument

X
X

X

X

Fairness/
Proportionality

X

X

X

Local Members/
Control

X

X

Government Formation/
Stability/Eﬀectiveness

Table 1: Argument Characteristics

X

X

BallotStructure
X
X
X
X
X
X

FPTP

X
X

X
X
X

X

X

Political
Parties

X
X

X
X

NegativeImplicit in Equation (1) is the fact that πj,i ≡ 1 − πi,j . In other words, there can be no tied
contests: either i wins and j loses or j wins and i loses.3 The task is to estimate αi and αj .
Suppose that i and j compete against one another a total of Ni,j times and let ni,j be thenumber of times i beats j. If all these contests are independent, then it is natural to assume ni,jis distributed as a binomial (Ni,j , πi,j ). With t total arguments competing—i.e. all the speciﬁc
is and js that are actually compared in the survey—the likelihood is∏t
L(α|n) = ∏

ni,j
i αi

Ni,j
i<j (αi + αj ).

(2)Maximization of (2) yields estimates of the elements of α subject to the identiﬁcation restrictionthat some αi is set equal to one.4

As implied by the form of Equation (1) and Equation (2),πi,j =αi.
αi + αj(3)

The functional form for this ‘dominance’ relationship between the players i and j has particulartransitivity properties. To see this, ﬁrst denote the matrix of all such πi,j probabilities as P . Now
suppose one draws three players (a ‘triplet’) which we will write {i, j, k}. We say that P satisﬁes
strong stochastic transitivity (SST) if when πi,j ≥ 12 and πj,k ≥ 21 then πi,k ≥ max(πi,j , πj,i )(Suppes, Krantz, Luce, and Tversky 1990). Equation 3 implies that the corresponding P matrix
meets SST. Speciﬁcally, if
πi,j =1αi≥αi + αj2

πj,k =αj1≥ ,αj + αk2

and3Eﬀectively, this means not allowing ‘don’t know’ responses.
Note (a) that Bayesianestimators exist, but we do not discuss them here; (b) an alternative identiﬁcation∑
restriction is to set i αi = 1.
410then αi ≥ αj ≥ αk andπi,k =

αj
αi
αi
≥
≥
αi + αkαi + αjαj + αkwhich is stronger than SST requires. The key point is simply that ‘transitivity’ is implicitly
assumed when modeling the data generating process this way. A consequence of this assumption
is that arguments that never meet one another can be empirically compared post-estimation.5

Returning to the estimation details, note that the α may be estimated even if ni,j is zero forsome pairings, so long as there does not exist some subset of arguments that never meet withothers in competition.6 A logit linear form of the problem proceeds by deﬁning exp(λi ) ≡ αi andexecuting the maximization above. In this case, with rearrangement of Equation (1),exp(λi ),exp(λi ) + exp(λj )(4)]Pr(i beats j)= λi − λj .
Pr(j beats i)(5)πi,j =

and, in terms of log-odds,[
log

The intuitive message from Equations (4) and (5) is clear: the larger the value of λi relative to
λj , the more likely it is that argument i beats argument j in a pairwise contest.

Standard errors on player abilities can be obtained in the usual way: that is, via the squareroot of the diagonal of the inverted Hessian matrix, although recall that one ability (of the‘reference’ player) is set to zero and thus will be accompanied with no uncertainty estimate. The
presence of the ni,j and Ni,j terms in Equation (2) has a sensible consequence: players involved
in more contests will have smaller variances around their estimated abilities. Furthermore, theform of Equation (2) and Equation (3) implies that if i beats a relatively powerful argument j,
5

In principle, one could check every possible triple for transitivity violations empirically. This is diﬃcult in thecurrent case because a large number of the triples involve two sets of pairs that do not meet: e.g. i beats j, jbeats k but i and k do not meet. This has no eﬀect on the estimation per se, and only eﬀects the veriﬁcation of
the modeling assumption.
6
More technically, the design must be ‘connected’ though it need not be ‘complete’.

11ceteris paribus, player i will receive a larger boost to his estimated ability than if he defeats a
weak opponent: for a ﬁxed πi,j , reducing αj results in a smaller estimate of αi . As a corollary
to these two points, and the comment on transitivity, notice that the model does not necessarily require large numbers of subjects in any particular two player treatment group in order toobtain meaningful estimates of abilities in the tournament as a whole. Rather, information isleveraged from the full dataset of contests, and the relative performance of the arguments in eachpairwise interaction involving them. Hence, the statistical power ‘problem’ that more conventional approaches such as means tests of support might encounter—and that we noted in the
Introduction—is sidestepped.

The approach here may appear unusual to political scientists, but is well known to statisticians as the Bradley-Terry model7 for pairwise comparison (Bradley and Terry 1952) and has
been used by psychologists interested in subjects selecting items from choice sets for some time(see, for example, Luce 1959; Thurstone 1959).8 The model has seen use in other ﬁelds such as
biology (e.g. Stuart-Fox, Firth, Moussalli, and Whiting 2006), genetics (e.g. Sham and Curtis
1995), the investigation of journal citation patterns (e.g. Carter and Spirling 2008; Stigler 1994)
and sports science (e.g. Agresti 2002). Spirling (2007) considers an application to the UnitedStates Senate, however we can ﬁnd no other application of this model in political science.9

An important version of the Bradley-Terry model is provided by Springall (1973), in which‘player’-speciﬁc (here, argument-speciﬁc) variables xi1 , . . . , xip are used to predict the ‘power’ of
the players (the λ) directly. These independent variables enter the model via the linear predictor
7
The model is sometimes called the Bradley-Terry-Luce model. We rest with simpler naming, as Luce’s contributions to the model came some years after the model’s original exposition by Bradley and Terry.
8
In practice, Thurstone (1959) opts for a slightly diﬀerent, probit form of the model. Here we will discuss thelogistic regression case.
9
Indeed, a search of all major political science journals on JSTOR resulted in only one hit; an article mentioning
the Bradley-Terry model in an end note (Monroe 1995).

12

in the sense that
λi =

p
∑

βr xir ,(6)

r=1

and interest focuses on the estimated coeﬃcients β1 , . . . , βp . In the current context, these β inform the analyst as to the source, or ‘cause’, of the arguments’ power. A more complicated form of∑Equation (6) includes a per player (here, argument) random eﬀect such that λi = pr=1 βr xir +Ui ,where Ui ∼ N (0, σ 2 ). Below, we ﬁt both structured versions, though in practice the variance ofthe errors is estimated as zero, and thus the forms are equivalent.

In practical applications, a general concern is that a judge—a person making a decision as towhich of i and j wins—might not give independent responses from one ‘row’ of the data to thenext. For example, we might worry that a person might is generally inclined towards a particularelectoral system, before the survey begins, and that this ‘bias’ is propagated throughout all theirdecisions. In that case, one might consider a ﬁxed eﬀect vr for each given respondent such that
Equation (5) becomes[]
Pr(i beats j)log= vr + λi − λj .
Pr(j beats i)

This makes most sense when there are a moderate number of respondents making multiple decisions.10 In our particular case however, we have as many respondents (520) as their are comparisons, since each surveyed individual is responsible for one (and only one) decision. As aresult, ﬁxed eﬀects are not identiﬁed. Notice, in addition, that there in fact is no intercept
term in the standard Bradley-Terry set up (McCullagh and Nelder 1999, 272). To see why, recall that Equation (5) has logitpij = λi − λj . Suppose that there were a constant, such that
logitpij = α0 + λi − λj . Consider the complement of this quantity, logitpji , which simply involves
switching the indices around; thus, logitpji = β0 + λj − λi , where β0 is a constant term, possibly
taking a diﬀerent value to that given by α0 . Since logitpij = −logitpji by deﬁnition, we musthave α0 + λi − λj = −β0 − λi + λj , which clearly implies α0 = −β0 . Yet this equality requires
10

Alternatively, if there are a large number of respondents, one might try a random eﬀects route.

13that α0 = β0 = 0 and we have no intercept for our model.

In principle, diﬀerent respondents may judge the arguments depending on their (diﬀering) personal covariates. For example, older people may be more likely to respond positively to a FPTP
argument since it endorses the long-standing status quo. In the current context, such eﬀects are
diﬃcult to include formally. Generally speaking, to the extent that a particular contest-speciﬁc‘bias’ (such as the sex or age of the respondent) occurs, it eﬀects both players (here, the relevant
arguments) equally. While it is possible to use binary ‘order eﬀects’ where we have an a priori
reason to believe that one player is (dis)advantaged in some way, in the sense of Agresti (2002),
it is not obvious how to include more complicated covariates at the respondent level.

Turner and Firth (2010) devote considerable eﬀort to designing software for the ﬁtting of both
‘unstructured’ models of the form given in Equation (5), and ‘structured’ versions as noted in
Equation (6). Our work below utilizes this implementation—BradleyTerry2—in conjunction
with the R language and environment (R Development Core Team 2006).

4

The Power of Arguments: The Bradley-Terry Method

Recall that the initial goal is to report a rank-ordering of arguments according to their ability
to convince voters. Two approaches may seem more obvious at this point. First, researchers
could rank the arguments according to the proportion of contests in which argument i beat thealternatives it was drawn against. As an empirical example, FPTP2 is preferred over its opponent (whatever that may be) 58 times out of 99 times it is paired in the survey. Hence, itwins approximately 59% of its contests. By contrast, MMP6 wins 43 of its 84 contests, andthus its ‘mean’ support might be reported as 51%. There are two interrelated problems withthis procedure. First, there is uncertainty in the estimates which is not reported. That is, thesemeans should come with standard errors that reﬂect sampling variation. This point is impor-14

tant because though the contests are plausibly independent, this was not strictly a ‘round-robin’
tournament in which all players met a set number of times: for example, MMP2 met FPTP2 on
19 occasions, while MMP5 met FPTP1 just 15 times. However, the arguments are of diﬀerent
abilities: stacking up wins against a relatively strong opponent should presumably count more
towards the ranking than drubbing a very weak one. Yet by just comparing proportions of victories that subtlety is lost.

An alternative approach might be to estimate a logistic regression wherein the dependent variable is support for, say, the MMP option and the various FPTP options are dummied out on theright hand side. One could then ﬂip the operation, instead looking at FPTP victories predictedby MMP argument dummies. While uncertainty would (as usual with such approaches) be estimated along with the coeﬃcients, the problem with this strategy is that it moves away fromthe core quantity of interest: the speciﬁc abilities of the arguments relative to one another, suchthat arguments that did not meet—and did not meet the same sets of opponents in the same
numbers of contests—can be sensibly compared.

4.1

Unstructured Model Results

As noted above, the Bradley-Terry model will facilitate direct estimation of abilities, allow for
uncertainty statements and incorporate information on the relative abilities of contestants interms of the estimated ‘power’ of those that win and lose against them. To show this we begin
with the ‘unstructured’ model results in Table 2.

Unstructured model results consist of a power coeﬃcient for each argument, λi , which can beused to compare its power with that of other arguments. The reported coeﬃcient for FPTP2, themost powerful argument, is set to 0. All other coeﬃcients (and their p-values), are reported incomparison to the power of this FPTP argument. The arguments are ordered from most to least
powerful. By examining the coeﬃcients we can determine which arguments are estimated as less15

Argument
FPTP2
FPTP1
FPTP3
MMP6
FPTP4
MMP3
FPTP5
MMP5
MMP1
MMP2
MMP4
FPTP6

Power (λ)
0.000
-0.137-0.151-0.197-0.203-0.224-0.278-0.302-0.343-0.422-0.593-0.605p-value
—
0.64
0.63
0.50
0.49
0.45
0.34
0.30
0.23
0.12
0.04
0.05

Quasi-variance
0.04
0.04
0.05
0.05
0.04
0.05
0.04
0.05
0.05
0.04
0.04
0.05

Table 2: Bradley-Terry Model of Argument Powerpowerful than others. Moreover, by examining the p-values, we can determine which arguments
are signiﬁcantly less powerful than FPTP2 (i.e. MMP4 and FPTP6). Though we do not report
it here, with such a rank ordering the Bradley-Terry model does not diﬀer much from a meansupport type approach outlined above and the order of the arguments appears approximately
the same. But, of course, this approach does oﬀer the distinct advantage that we can easily
compute the estimated probability that one argument dominates another. Recall Equation (4);if we wanted to know the probability that FPTP2 defeated MMP2, we enter the power estimates
into the equation and ﬁnd that:

π2,2 =

exp(0.00)= 0.60.
exp(0.00) + exp(−0.422)(7)

Similarly, we could calculate the probability that FPTP2 beats the more evenly matched MMP6
at 0.55. In Figure 1 we report our results in a slightly diﬀerent way. Here, the black dots are
the ability point estimates, while the bars around them are conﬁdence intervals based on their‘quasi-variances’ (in the sense of Firth and De Menezes 2004).11 The quasi-variances reported
11

Though the term may be unfamiliar to readers, there is a straightforward analogy. In the regression context,
quasi-variances are used to approximate the standard error between estimated coeﬃcients for a categorical variable
such as education, which might be coded into, say, four mutually exclusive, and jointly exhaustive bins: ‘less thanhigh school’, ‘high school graduate’, ‘some college’, ‘post-graduate education’. Because all standard errors (andp-values) are reported relative to some ‘base’ or ‘reference’ category, one cannot—without the variance-covariancematrix—assess the signiﬁcance of the diﬀerence between two categories that do not include the reference bin. But
this is precisely the situation here: we have a base category (FPTP2), yet we wish to compare other arguments to16

in Table 2 allow the analyst to compare arguments even though there is a base category whichhas (by construction) a standard error of zero. In particular, the t-statistic associated with acomparison between arguments is (well approximated as)t= √λ̂i − λˆjqv(λ̂i + qv(λˆj )where qv(·) is the relevant quasi-variance. So, for the example of F P T P1 and F P T P2 we havet = √−0.13−0= −0.47 which has an associated p-value of 0.63 (see Gayle and Lambert 2007,
0.04+0.04
for discussion of the calculations). Thus the diﬀerence is not statistically signiﬁcant. The quasivariances in Table 2 allow such a comparison for any such pair. Looking at Figure 1 it is clear thatevery interval overlaps every other, though in fact, there is a statistically signiﬁcant diﬀerence
between F P T P 1 and both F P T P 6 and M M P 4. Still, most arguments are not distinguishable,
and it not obvious that there is much of a diﬀerence between FPTP and MMP in terms of theirgeneral appeal.

5

The Sources of an Argument’s PowerIn the previous section, we estimated the power of twelve arguments for and against electoral
reform. Our task now is to estimate the sources of this power. In other words, what are thecharacteristics of an argument that make it able to overcome other arguments—in our case, foror against electoral reform? In Table 1 we summarized the attributes of each of the FPTP and
MMP arguments. By including these traits as covariates in a structured model we are able toestimate the sources of an argument’s power—or ‘ability’—to persuade.12

We began with an ‘all main eﬀects’ model in which all the argument characteristics reported
in Table 1 were coded as binary covariate vectors. Since the model is estimated via maximum
see if they are diﬀerent from one another.
12
This is Equation (6) in Section 3.

17

0.5
0.0
−1.0

−0.5

power

FPTP6

MMP4

MMP2

MMP1

MMP5

FPTP5

MMP3

FPTP4

MMP6

FPTP3

FPTP1

FPTP2

argument

Figure 1: Power of Arguments: quasi-standard errors (via quasi-variances) for Bradley-Terry
abilities. Notice that the ‘comparison intervals’ overlap for all arguments.

18

likelihood, standard procedures for stepwise ﬁnding of the ‘best-ﬁtting’ model are available via
the Akaike (AIC) or Bayes Information Criterion (BIC). We use the AIC and the model thus
selected is as reported in Table 3. As can be seen, we ﬁnish with four predictors. We take thisadvantage for some components over others as evidence that at least some of our sample was
responding to arguments systematically, rather than merely stating prior opinions. With the
exception of Parties, all the variables are signiﬁcant at the 10% level.

Table 3: Structured Bradley-Terry model for referendum argument data
FeatureEstimate

Std. Error

p-value

Fairness
Local
FPTP
Parties

0.53
0.32
0.28
-0.27

0.24
0.18
0.15
0.18

0.03
0.07
0.06
0.13

All else equal then, an argument in favor of the status quo appears systematically advantaged.
Local is a characteristic of arguments for both FPTP and MMP, as an MMP system does retain
local representation. However, MMP does not allow the local selection of all candidates, suggesting that in actual rhetoric the scope of local arguments favors FPTP. Fairness arguments clearlyadvantage MMP, as proportionality is a central component of this system and is never present inarguments for FPTP. Finally, the mention of political parties makes an argument less appealing.
This is likely to the disadvantage of MMP, as the central function of this system (and thus at least
some of the arguments in its favor) is to match seats to votes for political parties, rather than individual candidates. Parties are thus arguably more central to vote choice in MMP than FPTP.13

Using this model—and assuming for purposes of exposition that it is the only one we estimated—
13

As an aside, we note that the issue of ‘negative’ framing is a popular research topic in political science (see, forexample, Ansolabehere and Iyengar 1997). We found that ‘Negative’ was not a signiﬁcant predictor (on its own)
for the structured model, though it had a negative sign implying that voters are (all else equal) less convinced bysuch phrasing.

19

and the formula in Equation (4) we can estimate the probability that one argument dominates
another according to its characteristics.14 For example, we can estimate the probability that
FPTP2 (which mentions local control, and is in the FPTP group) dominates MMP6 (whichmakes a fairness and parties argument):

πF P T P 2,M M P 6 ==∑exp( βr xir )∑∑exp( βr xir ) + exp( βr xjr )exp(0.318 + 0.282)= 0.58.
exp(0.318 + 0.282) + exp(0.534 − 0.275)(8)

By contrast, if we pit FPTP2 against MMP4, an argument which has no signiﬁcantly powerfulattributes, the probability of FPTP dominance rises to 0.65.15

Figure 2 presents our ﬁndings in a slightly diﬀerent way. The ﬁgure compares the probability of an argument with various covariate proﬁles winning against the modal argument (that is,an argument which does not mention ‘fairness’, ‘local’ or ‘parties’, but is in the ‘FPTP’ group).
The arrows show the eﬀect of moving from one covariate proﬁle to another.

For example, in a contest with the modal argument, an argument that is identical but which
mentions fairness would do better: the probability of a win for this second argument increases
from 0.5 to around 0.64. By contrast, an argument that is identical but which does not mention
FPTP would do worse: the probability of a win for this second argument decreases from 0.5 toaround 0.45. On the left side of the plot, we report the predicted probabilities for the argumentsin the study based on their covariate proﬁles. Reading from top to bottom, we see that FPTP2
is once again the most powerful argument, in keeping with our unstructured results.
14

Of course, (a) the arguments have characteristics the coeﬃcients for which are not reported in this ‘best ﬁtting’model and (b) the coeﬃcients in this reduced model will diﬀer to those in the full model. Nonetheless, for purposesof example, we use the coeﬃcients in Table 3 in what follows.
15
An alternative, but∑
entirely equivalent way to conceive of the predicted powers of the arguments is via Equation(5): that is, λi − λj =βr (xir − xjr )

20

0.65
0.50

Pr(modal argument defeated)

FPTP2

FPTP4
MMP1,3,5,6

MODAL
ARGUMENT

fairness = 0

parties = 0

fptp = 1

P art i es

FPTP

local = 0

0.40

MMP2
FPTP6
MMP4

F ai rness

L ocalFeatureFigure 2: Power of arguments: structured results. The arrows show the eﬀect of moving from one
covariate proﬁle to another. On the left side of the plot, we report the predicted probabilities forthe arguments in the study based on their covariate proﬁles The modal arguments are FPTP1,
FPTP3, and FPTP5.

21

5.1

Structured or Unstructured Results?

Given structured and unstructured results, the question remains about which researchers should
prefer. We believe this is a matter of what questions a researcher wishes to answer. If we wanted tounderstand the power of exact arguments in the referendum, we would be well-served to consider
the unstructured results, as these arguments closely match those made during the campaign.
However, if we were looking forward to another campaign and wished to develop new arguments,
we could learn more from the structured results. Indeed, these would allow us to design optimalarguments which combined eﬀective components and avoided less eﬀective ones. Fortunately,
despite these models giving us diﬀerent types of information on power, the predictions which
result from them are very similar for our case. Table 4 reports the absolute diﬀerence in predictedprobabilities of FPTP dominance according to our unstructured and structured (best ﬁtting)models. The mean absolute diﬀerence in predicted probability is just two percentage points. Of
course, the situation that the typical scholar faces might be very diﬀerent and depends on the
degree to which the covariate proﬁles do a ‘good’ job of explaining the variation in the data.

Table 4: Diﬀerence in predicted probabilities of FPTP dominance by structured andunstructured models

6

MMP Argument⇓

FPTP Argument⇒
1
2

3

4

5

6

1
2
3
4
5
6

0.05
0.01
0.02
0.04
0.04
0.01

0.04
0.01
0.01
0.04
0.03
0.01

0.02
0.02
0.01
0.02
0.01
0.02

0.01
0.02
0.02
0.01
0.00
0.03

0.00
0.04
0.03
0.00
0.01
0.04

0.00
0.03
0.03
0.00
0.01
0.04

Further Applications and Conclusion

We have demonstrated that Bradley-Terry models can generate meaningful results from pairwise
matchings of arguments, even when the number of treatment groups is large and the sample

22

size small. Results of similar utility cannot be uncovered, for example, by more conventionalmodels that do not utilize the contest nature of the data and the fact that individual players (thearguments) are involved in multiple battles. Moreover, structured models can be used to identify
the source of an argument’s persuasive power. These models are thus well-suited to a situationin which a researcher wishes to test a large number of arguments on a small sample size. Moregenerally, these models are ideal for any question that features contest data.

Contest data is perhaps more common than we think. Indeed, we can identify a number offurther applications within electoral studies and political science more broadly. For example,
Bradley-Terry models could be deployed in experimental studies determining the attractivenessof candidates (e.g., Sigelman 1990). Likewise, such models could be deployed to determine which
factors lead voters to prefer one leader, candidate, or party over another. More generally, we
could measure the comparative power of frames over a small number of respondents without
the need to repeat frames (e.g., Chong and Druckman 2007). The models could also be used
to identify sources of persuasion in studies using a ‘jury method’ (e.g., London, Meldman, andLanckton 1970).

We can likewise identify applications in international relations and American electoral politics.
For example, the large and growing literature on ‘democratic eﬀectiveness’ is concerned with themodeling of ‘success’ in war as a function of polity characteristics (e.g. Lake 1992; Reiter andStam III 1998; Downes 2009). Meanwhile, the voluminous discussion of ‘incumbency bias’ in
US politics (e.g. Fiorina 1989; Ansolabehere and Snyder 2002) asserts that, for various reasons,contests between sitting oﬃcials and challengers are not ‘fair ﬁghts’. Indeed, any research design
with dichotomous outcomes and direct confrontation between units or actors is a candidate forthese models.

Our suggested approach is not without drawbacks. As discussed here, it requires pairwise com-23

parison and dichotomous outcomes. Accordingly, in the form expressed above, it cannot be used
if researchers are interested in measuring degrees of support for a position. On the other hand,as written (with the random eﬀect), it is in the form of a generalized linear mixed model, andso might be extended to normal-theory measurement situations. Certainly, if researchers wish to
make pairwise comparisons with dichotomous outcomes, this approach provides a powerful solution when a number of actors (be they arguments or something else) need to be tested, especially
if there are a small number of observations for each contest.

Given the large number of potential applications and the likely extensions of these models in
the near future, we conclude that the Bradley-Terry approach is a helpful addition to the political methodologist’s toolkit and we look forward to future research that utilizes it.

24

A

Sample, Subject Profile and Generalization

Our subjects were drawn from an internet panel containing a list of approximately 15000 speciﬁcally named individuals. The panel is managed by a commercial polling ﬁrm. Though the panel isnational, our experiment was provincial. At the time of the survey the panel had 3575 registeredpanelists in the province of Ontario. Respondents either self-select into the panel through thesurveying ﬁrm’s website, or they are recruited through solicitation by phone and email. Upon
registration in the panel respondents are assigned a number between 1 and 31. In each subsequentmonth they are invited to complete the survey in one of four weeks according to this number.

Our experiment occurred in the last week of the referendum campaign (October 2 to 9, 2007)
and was limited to those living in the province of Ontario. Of 3575 Ontarians registered in thepanel, 844 were invited to complete the survey during this week. 565 agreed to complete thesurvey. Of those 565, 520 agreed to participate in the Bradley-Terry experiment. This gives us
a response rate of 61.6%.

Table A-1 presents a proﬁle of the experimental subjects.
Assignment to treatment condition is unrelated to educational category (χ2 = 87.81 p = .08),
income category (χ2 = 138.6) p = .52), age category (χ2 = 145.02, p = .95) and gender(χ2 = 26.29 p = .86).

An obvious objection to our sample is that it is not a random sample of adult-aged Ontarians.
This certainly appears reasonable; compared to the general population (Canadian census, 2001),
our sample is both better educated (χ2 = 11.64, p = .00) and wealthier (χ2 = 3.79, p = .05).
Moreover, previous analysis with the panel suggests that respondents are more likely to identify
with a political party and appear highly interested in current aﬀairs. The panel is likely highlysophisticated politically. As a consequence, our ﬁndings may not generalize to the whole popula-25

Table A-1: Sample Demographics and Political Characteristics
Variable
Age

FemaleHousehold Income

EducationN% or Mean
19–24
25–34
35–44
45–54
55–64
65+
<$40000$40000 to $60000$60000 to $80000>$80000
RefusedHigh School or lessSome College
Some University
520

1.7%
9.6%
16.5%
26.5%
31.7%
13.9%
52.6%
12.89%
16.15%
17.12%
38.65%
15.19%
8.85%
28.46%
62.69%

tion of Ontario, endangering external validity (Shadish, Cook, and Campbell 2000). Objections
about generalization aside, nothing in our statistical model requires that we conduct our analysis
over a random sample of the population. As such, the methodological approach remains fecund
for more traditional randomly sampled surveys.

26

References
Agresti, A. (2002): Categorical Data Analysis. John Wiley and Sons, New York, 2 edn.
Ansolabehere, S., and S. Iyengar (1997): Going Negative: How Political Advertisements
Shrink and Polarize the Electorate. The Free Press, New York.
Ansolabehere, S., and J. Snyder (2002): “The Incumbency Advantage in U.S. Elections:
An Analysis of State and Federal Oﬃces, 1942-2000,” Election Law Journal, 1(3), 315–338.
Ansolabehere, S., J. M. Snyder, and C. Stewart (2001): “Candidate Positioning in U.S.
House Elections,” American Journal of Political Science, 45(1), 136–159.
Aristotle (322BC/1991):
Kennedy), New York.

Rhetoric. Oxford University Press (translation by George A.

Blais, A., and L. Massicotte (2002): “Electoral Systems,” in Comparing Democracies 2: New
Challenges in the Study of Elections and Voting, ed. by L. LeDuc, R. G. Niemi, and P. Norris.
Sage, London.
Bradley, R., and M. Terry (1952): “Rank analysis of incomplete block designs, I. The
method of paired comparisons.,” Biometrika, 39, 324–345.
Carter, D., and A. Spirling (2008): “Under the Inﬂuence? Intellectual Exchange in Political
Science,” PS: Political Science and Politics, 41(2), 375–8.
Chong, D., and J. N. Druckman (2007): “Framing Public Opinion in Competitive Democracies,” American Political Science Review, 101(4), 637–55.
Downes, A. (2009): “How Smart and Tough are Democracies? Reassessing Theories of Democratic Victory in War,” International Security, 33(4), 9–51.
Erikson, R. S., and G. C. Wright (1997): “Voters, Candidates, and Issues in Congressional
Elections,” in Congress Reconsidered, ed. by L. C. Dodd, and B. I. Oppenheimer. Congressional
Quarterly Press, Washington DC.
Fenlon, B. (2007): “Referendum on Electoral Reform an ‘Unmitigated Disaster’,” The Globe
and Mail, October 11, 2007, Online edition.
Fiorina, M. (1989): Congress: Keystone of the Washington Establishment. Yale University
Press, New Haven, 2 edn.
Firth, D., and R. De Menezes (2004): “Quasi-variances,” Biometrika, 91(1), 65–80.
Gayle, V., and P. Lambert (2007): “Using Quasi-variances to Communicate SociologicalResults from Statistical Models,” Sociology, 41(6).
Hobolt, S. B. (2009): Europe in Question: Referendums on European Integration. OUP,
OUPad.
Lake, D. (1992): “Powerful paciﬁsts: Democratic states and war,” The American Political
Science Review, pp. 24–37.
London, H., P. J. Meldman, and A. V. C. Lanckton (1970): “The Jury Method: How the
Persuader Persuades,” Public Opinion Quarterly, 34, 171–83.
27

Luce, R. D. (1959): Individual Choice Behaviour: A Theoretical Analysis. J. Wiley, New York.
McCullagh, P., and J. Nelder (1999): Generalized Linear Models. Chapman & Hall, New
York, 2 edn.
Monroe, B. L. (1995): “Fully Proportional Representation,” American Political Science Review, 89(4), 925–40.
Mutz, D., P. M. Sniderman, and R. Brody (eds.) (1996): Political Persuasion and Attitude
Change. University of Michigan Press, Ann Arbor.
Page, B. (1978): Choices and Echoes in Presidential Elections. University of Chicago Press,
Chicago.
Pammett, J. H., and L. LeDuc (2001): “Sovereignty, leadership and voting in the Quebec
referendums,” Electoral Studies, 20, 265–280.
Pattie, C., D. Denver, J. Mitchell, and H. Bochel (1998): “Partisanship, national identity and constitutional preferences: an exploration of voting in the Scottish devolution referendum of 1997,” Electoral Studies, 18, 305–322.
Powell, G. B. (2000): Elections as Instruments of Democracy: Majoritarian and Proportional
Visions. Yale University Press, New Haven CT.
R Development Core Team (2006): R: A Language and Environment for Statistical ComputingR Foundation for Statistical Computing, Vienna, Austria, ISBN 3-900051-07-0.
Reiter, D., and A. Stam III (1998): “Democracy and battleﬁeld military eﬀectiveness,” Journal of Conﬂict Resolution, pp. 259–277.
Shadish, W. R., T. D. Cook, and D. T. Campbell (2000): Experimental and QuasiExperimental Designs for Generalized Causal Inference. Houghton Miﬄin, Boston.
Sham, P., and D. Curtis (1995): “An Extended Transmission/Disequilibrium Test (TDT) for
Multiallele Marker Loci,” Annals of Human Genetics, 59(3), 323–36.
Sigelman, L. (1990): “Hair Loss and Electability: The Bald Truth,” Journal of Nonverbal
Behavior, 14(4), 269–83.
Spirling, A. (2007): “Power Tool: Measuring Power in Political Science—A New Methods with
Application to the United States Senate,” University of Rochester, Working Paper.
Springall, A. (1973): “Response surface ﬁtting using a generalization of the Bradley-Terrypaired comparisons model,” Applied Statistics, 22, 59–68.
Stigler, S. (1994): “Citation Patterns in the Journals of Statistics and Probability,” Statistical
Science, 9, 94–108.
Stuart-Fox, D. M., D. Firth, A. Moussalli, and M. J. Whiting (2006): “Multiple Signals
in Chameleon Contests: Designing and Analysing Animal Contests as a Tournament,” Animal
Behaviour, 71, 1263–1271.
Suppes, P., D. Krantz, D. Luce, and A. Tversky (1990): Foundations of Measurement:
Representation, Axiomatization, and Invariance. Academic Press, New York.
28

Thurstone, L. (1959): The Measurement of Values. University of Chicago Press, Chicago.
Tranter, B. (2003): “The Australian constitutional referendum of 1999: evaluating explanations of republic voting,” Electoral Studies, 22, 677–701.
Turner, H., and D. Firth (2010): Bradley-Terry models in R: The BradleyTerry2 packageRpackage version 0.9-4.

29