Proceedings of the 15th Conference on

Theoretical Aspects of Rationality andKnowledge
TARK 2015

EditorR. Ramanujam
The Institute of Mathematical Sciences, Chennai, India© by the authors. All rights reserved.

Please contact the authors directly for permission to reprint or to use this material in any form for any purpose.

ISBN number 978-0-692-46138-9

Chairs
Program Chair: R. Ramanujam (IMSc, Chennai)
Local Organizing Chair: Kevin T. Kelly (CMU)
Conference Chair: Joseph Y. Halpern (Cornell)

Program Committee
Eleonora Cresto
Clare Dixon
Edith Elkind
Amanda Friedenberg
Sujata GhoshAndreas Herzig
Bettina Klaus
Kevin Kelly
Yoram Moses
Andrés Perea
Sophie Pinchinat
R. Ramanujam (chair)Francesca Rossi
Olivier Roy
Burkhard Schipper
Hans van Ditmarsch
Yanjing Wang
Michael Wooldridge

Local Organizing Committee
David Danks
Teddy Seidenfeld
Mary Grace Joseph
Jacqueline Defazio
Rosemarie Commisso

Conference Program
Invited Papers
1 Quine’s Topiary: Coordination and Change in an Artificial Society
Robin Clark

1

3 The Problem of Analogical Inference in Inductive Logic
Simon M. Huttegger
11 Human-Agent Decision-making: Combining Theory and Practice
Sarit Kraus
21 Rationality and Beliefs in Dynamic Games: Revealed-Preference Foundations
Marciano Siniscalchi
Tutorial
23 Recent Methodological Advances in Causal Discovery and InferencePeter Spirtes and Kun Zhang

23

Contributed Papers
37 Resolving Distributed Knowledge
Thomas Ågotnes and Yi N. Wáng

37

47 Epistemic Protocols for Distributed Gossiping
Krzysztof R. Apt, Davide Grossi and Wiebe van der Hoek
57 Coordination Games on Directed Graphs
Krzysztof R. Apt, Sunil Simon and Dominik Wojtczak
65 On the Solvability of Inductive Problems: A Study in Epistemic Topology
Alexandra Baltag, Nina Gierasimczuk and Sonja Smets
75 Bayesian Games with Intentions
Adam Bjorndahl, Joseph Y. Halpern and Rafael Pass
83 How Many Levels Do Players Reason? Observational Challenges and a Solution
Adam Brandenburger, Alex Danieli and Amanda Friedenberg
87 Translucent Players: Explaining Cooperative Behavior in Social Dilemmas
Valerio Capraro and Joseph Y. Halpern
95 Announcement as effort on topological spaces
Hans van Ditmarsch, Sophia Knight and Aybüke Özgün103 Single-Peaked Consistency for Weak Orders Is Easy
Zack Fitzsimmons
111 Theory Choice, Theory Change, and Inductive Truth-Conduciveness
Konstantin Genin and Kevin T. Kelly
121 Do players reason by forward induction in dynamic perfect information games?
Sujata Ghosh, Aviad Heifetz and Rineke Verbrugge
131 Ceteris paribus logic in counterfactual reasoning
Patrick Girard and Marcus Anthony Triplett141 An Information-Gap Framework for Capturing Preferences About Uncertainty
Russell Golman and George Loewenstein
153 Deliberating between Backward and Forward Induction Reasoning: First Steps
Aleks Knoks and Eric Pacuit
163 Standard State Space Models of UnawarenessHarvey Lederman and Peter Fritz
173 An Axiomatic Approach to Routing
Omer Lev, Moshe Tennenholtz and Aviv Zohar
181 Preference at First Sight
Chanjuan Liu
191 The optimality of coarse categories in decision-making and information storage
Michael Mandler
195 Undecidable Cases of Model Checking Probabilistic Temporal-Epistemic Logic
Ron Van Der Meyden and Manas K. Patra
207 Relating Knowledge and Coordinated Action: The Knowledge of Preconditions Principle
Yoram Moses
217 Learning from unrealized versus realized prices
Kathleen Ngangoué and Georg Weizsäcker
219 Continuous, lexicographic context dependence in a binary choice setting
Patrick O’Callaghan
225 When Do Types Induce the Same Belief Hierarchy?
Andrés Perea
233 Rationalization and Robustness in Dynamic Games with Incomplete Information
Evan Piermont and Peio Zuazo-Garin
239 Parameterized Complexity Results for a Model of Theory of Mind based on Dynamic
Epistemic Logic
Iris van de Pol, Iris van Rooij and Jakub Szymanik
249 A Dynamic Epistemic Framework for Conformant Planning
Quan Yu, Yanjun Li and Yanjing Wang
261 Distinguishing Cause from Effect Based on Exogeneity
Kun Zhang, Jiji Zhang and Bernhard Schölkopf

Index of Authors273

Foreword

This volume consists of papers presented at the Fifteenth
conference on Theoretical Aspects of Rationality andKnowledge (TARK) held at Carnegie Mellon University,
Pittsburgh, USA from June 4 to 6, 2015.
It has been my pleasure to be part of the TARK community since the first conference at Asilomar, California, in
1986, principally due to the encouragement of Rohit Parikh,
one of the founders of TARK. This conference is uniquely
situated as one that brings together researchers from a wide
variety of fields, including Artificial Intelligence, Cryptography, Distributed Computing, Economics and Game Theory,
Linguistics, Philosophy, and Psychology. It has played an
important role in our understanding of interdisciplinary issues involving reasoning about rationality and knowledge.
This year we had 63 submissions out of which 18 were
accepted as contributed talks and 9 as poster presentations
for the programme. I am very grateful for having the cooperation and advice of 17 other members of the multidisciplinary program committee: Eleonora Cresto (CONICET,
University of Buenos Aires, Argentina), Clare Dixon (University of Liverpool, UK), Edith Elkind (Oxford University,UK), Amanda Friedenberg (Arizona State University, USA),
Sujata Ghosh (Indian Statistical Institute, India), Andreas
Herzig (IRIT, Toulouse, France), Bettina Klaus (University of Lausanne, Switerland), Kevin Kelly (Carnegie Mellon
University, USA), Yoram Moses (Technion, Tel Aviv, Israel),
Andrés Perea (Maastricht University, The Netherlands), Sophie Pinchinat (IRISA, Rennes, France), Francesca Rossi(University of Padova, Italy), Olivier Roy (Bayreuth University, Germany), Burkhard Schipper (University of California
at Davis, USA), Hans van Ditmarsch (LORIA, France), Yanjing Wang (Peking University, China) and Michael Wooldridge(Oxford, UK). I thank them for their hard work in providing careful reviews and for the detailed discussions about
the submissions. When papers are read across disciplines,
there can be keen differences in what is considered good andimportant; I thank the committee members for trying theirbest to listen to other viewpoints.
We have four eminent invited speakers in this year’s TARK:
Robin Clark of the University of Pennsylvania, USA, bringing a viewpoint from Psychology and Cognition to epistemic
reasoning; Simon Huttegger, of the University of California,
Irvine, USA, a philosopher’s look at observational process
and inductive logic; Sarit Kraus of Bar-Ilan University, Israel, on desiging computational agents for interacting withpeople, based on insights from game theory and logic; Marciano Siniscalchi, Northwestern University, USA, on foundations of rationality in sequential games. In addition, we havea tutorial on causal inference and causal discovery, jointly
by Peter Spirtes of Carnegie Mellon University, USA, and
Kun Zhang of Max Planck Institute for Intelligent Systems,Tübingen, Germany.
The organizing team at Carnegie Mellon University has

been doing excellent work for putting everything in place
for the conference, and I thank them for all the hard work.
I am extremely grateful to Kevin Kelly, the chair of theorganizing committee, for his terrific coordination job.
I thank the Easychair conference system for providing thisimportant service, easing the Programme committee’s job
truly easy. I thank my colleagues Vaishnavi Sundararajan
and S. P. Suresh of the Chennai Mathematical Institute forhelp with the Proceedings volume, and Anantha Padmanabha of my Institute for help with the conference web page.
I thank the Institute of Mathematical Sciences, Chennai, towhich I belong, for hosting the conference page and acting
as publisher.
Finally, I thank Joe Halpern, for his comforting presence
and guidance all along, providing inspiration to TARK.
R. Ramanujam
Institute of Mathematical Sciences, Chennai, India
Programme Chair, TARK 2015

Quine’s Topiary:
Coordination and Change in an Artificial Society
Robin Clark

University of Pennsylvania Philadelphia, USA

rclark@sas.upenn.edu

ABSTRACTThis talk reports the results of a large Agent-Based modelof phonetic variation; each agent in the society has its own
unique representation of the signal space, yet the agents areable to coordinate their signaling behavior. We show a number of results: First, agents in a segregated but egalitariansociety will blend their signals overtime if they signal to eachother; agents in a segregated, but bigoted, society will maintain stable variation. Second, if the artificial society containhigh status leaders—that is, the society is not egalitarian—then the signal space will actually move apart, creating variation where none existed before. We will analyze the sourceof this variation and show that it is a potential source of
language variation and language change. Finally, we will
discuss the relationship between private knowledge and social convention.

General Terms
Coordination

Keywords
Signaling behavior, artificial society, private knowledge

12The Problem of Analogical Inference in Inductive Logic[Extended Abstract]
Simon M. Huttegger

Department of Logic and Philosophy of Science
University of California, Irvine
Social Science Plaza A
Irvine, CA-92697, USA

shuttegg@uci.edu

ABSTRACT
We consider one problem that was largely left open by Rudolf
Carnap in his work on inductive logic, the problem of analogical inference. After discussing some previous attempts tosolve this problem, we propose a new solution that is basedon the ideas of Bruno de Finetti on probabilistic symmetries.
We explain how our new inductive logic can be developed
within the Carnapian paradigm of inductive logic—deriving
an inductive rule from a set of simple postulates about theobservational process—and discuss some of its properties.

Keywords
Inductive logic; Carnap; de Finetti; Analogy; Partial Exchangeability

1. INTRODUCTION
The logical empiricist movement is often associated withusing deductive logic to understand scientific reasoning. But
Rudolf Carnap actually favored an inductive approach, starting with his work on inductive logic in the 1940s. Carnapian
inductive logic can be thought of as a branch of probability theory that is especially concerned with predictive
probabilities—the probability of future observations given
past observations. Carnap spent much of the last thirty
years of his life on developing an inductive logic, but even in
his posthumously published works he considered the subject
to be wide open to further investigations. The open problem that I wish to consider in this paper is the problem of
analogical inference, which hasn’t received a satisfactory answer in Carnap’s original system. I shall review some of theattempts to develop an analogical inductive logic in §4. In
order to set the stage, I briefly describe Carnap’s programin §2 and point to its connections with de Finetti’s theoryof inductive inference in §3. Considering de Finetti is particularly important since he provides an alternative route toanalogical inference. In §5 I discuss an especially interesting
probabilistic symmetry that allows for a certain form of analogical inference. Finally, in §6 I introduce a new analogicalinductive logic based on that symmetry and discuss some ofits properties.

2. CARNAP’S PROGRAM
Carnap’s program for developing an inductive logic as described in his ‘Logical Foundations of Probability’ [2] wasbrought to a tentative conclusion in the posthumously published ‘A Basic System of Inductive Logic’ [4, 5]. Carnapian

3

inductive logic aims at finding rational foundations for thekind of inductive inferences that are used in scientific investigations. The classic example of such an inference in the
tradition of Bayes and Laplace is the predictive probabilityof events, such as future coin flips based on past observationsof coin flips. Carnap viewed all inductive inference problems
as being essentially reducible to this type of inference [2].1
Of particular importance for Carnap are predictive probabilities based on the relative frequencies of events. For
example, after observing a number of throws of a die, thepredictive probability of observing a six with the next throwusually is judged to be approximately equal to the relative
frequency of sixes. In his systems of inductive logic, Carnap
tries to explicate the foundations of this simplest kind of
inductive inference.
Independently of Carnap’s program, a similar approach
was developed more than two decades earlier by the Cambridge logician W. E. Johnson [21, 22]. Johnson’s main contribution was only published posthumously and contained
a number of gaps, which were closed by Sandy Zabell [33],
who also generalized Johnson’s approach to a theory that isessentially equivalent to Carnap’s basic system of inductive
logic. I’m going to follow Zabell’s elegant treatment because
it ties in neatly with the work of Bruno de Finetti (see thenext section).2
The basic postulate in this theory of inductive inference isa symmetry requirement known as ‘exchangeability’ (called
the ‘permutation postulate’ by Johnson). Suppose that there
is a finite sequence of random variables X1 , . . . , Xn representing observations (e.g. coin flips), and let their probability law be P. Like Carnap, we assume that the randomvariables can take on only a finite number of values. Then
P is exchangeable if it is invariant under permutations ofoutcomes; that is,P[X1 = x1 , . . . , Xn = xn ] = P[X1 = xσ(1) , . . . , Xn = xσ(n) ]for every permutation σ of {1, . . . , n}. This allows us to define exchangeable probabilities of infinite sequences X1 , X2 , . . .
as those for which every finite initial sequence is exchangeable. For simplicity, the sequence of random variables isoften called exchangeable without referring to its probability law.
Both Johnson and Carnap use a requirement for predic1
See [36] for an excellent overview for the development ofCarnapian inductive logic.
2
Kuipers [24] gives an overview of the mathematical aspectsof Carnap’s theory.

tive probabilities that is often called ‘Johnson’s sufficientness
postulate’. This postulate says that predictive probabilities
for i basically only depend on the past relative frequency ofi; i.e., there is a function f such that
P[Xn+1 = i|X1 , . . . , Xn ] = fi (ni , n).

(1)

Johnson’s sufficientness postulate judges information abouttypes other than i to be irrelevant for the predictive probability of i—a point that is going to be important for the
problem of analogical inference.
Finally, in order for conditional probabilities to be welldefined, a regularity postulate is assumed to the effect thateach finite initial sequence of outcomes has positive probability. It is then possible to show that the predictive probability of any outcome is equal to its relative frequency modulo
some prior parameters. More specifically, if trials are not independent, then there exist parameters αj for each outcome
j such that for all n and iP[Xn+1 = i|X1 , . . . , Xn ] =ni + αiP.
n + j αjcause it captures one of the classic situations of statistics—
i.i.d. trials with unknown parameters. This is what is shownby de Finetti’s representation theorem. Suppose, for example, that Xi records whether the ith toss of a coin flip cameup heads or tails, and that the infinite sequence X1 , X2 , . . .
is exchangeable. de Finetti proved that this is equivalent to
the probability of finite sequences of heads and tails beinga mixture of i.i.d. binomial trials with unknown bias of thecoin:3
Z 1
ph (1 − p)n−h dµ(p) (3)P[X1 = x1 , . . . , Xn = xn ] =0(Here, p is the bias for heads, µ is a uniquely determinedprior over biases and h is the number of heads in the firstn trials.) This theorem has profound consequences for thephilosophy of probability and for inductive inference [34].
Specifically, if the prior µ in the representation is a Beta
distribution (or, in the more gneral case of finitely many
types of outcomes, a Dirichlet distribution), then(2)

(If trials are independent, then there is no learning fromexperience.) Here ni is the number of times outcome i isobserved in the first n trials. The parameters αj are either
all positive or all negative; they must be positive if the sequence of observations is infinite exchangeable (see [33] fora thorough discussion). The rule given by (2) is called a‘generalized rule of succession’ (after Laplace’s special ‘ruleof succession’). A generalized rule of succession expresses amode of learning from experience. Experiences are given bypast observations of outcomes, and past observations leadto predictive probabilities for future outcomes.
The inductive logic given in (2) is equivalent to Carnap’s
mature basic system of inductive logic, also known as the‘λ − γ-continuum of inductive methods’. The system championed in his 1950 book is much more restricted [2]. It requires that all αj = 1, meaning that all outcomes are judged
to be equally probable prior to any observations. In his later‘A Continuum of Inductive Methods’ [3], Carnap generalized
this restricted system to one with a weight λ which regulates
the effect of the equally probable prior weights. The basic
system (2) extends this to arbitrary prior weights.
Especially in his early work on inductive logic, Carnap
thought of symmetry principles such as exchangeability as
requirements of rationality. The idea—familiar from justifications for Laplace’s principle of indifference—is that certain
probabilistic symmetries should hold whenever one does nothave any knowledge about the relevant underlying structure.
For instance, in the absence of any evidence concerning theorder of outcomes you should assume exchangeability. We
will see that interpreting symmetry principles in this way
puts significant constraints on how to include analogy effects into inductive logic, while the approach discussed in
the next section allows for a greater variety of inductive logics.

3. DE FINETTI’S PROGRAM
Bruno de Finetti is famous for his foundational work on
probability theory and inductive inference. The latter is of
special importance to us here. The most fundamental resultin this arena is de Finetti’s representation theorem for exchangeable sequences [9]. Exchangeability is important be-

P[Xn+1 = i|X1 , . . . , Xn ] =ni + αiP,n + j αjwhere αi , αj are nonnegative parameters determining theDirichlet distribution. This is equivalent to the Carnapian
inductive logic given in (2). One difference between the
two approaches lies in the underlying axiomatic foundations. In de Finetti’s case, it is given by (i) the assumptionof exchangeability and (ii) the assumption that the mixing
prior in the representation µ is a Dirichlet distribution. In
the Johnson-Carnap approach there is no appeal to the de
Finetti representation.
There is also an important interpretive issue that separates the early work of Carnap from de Finetti’s probabilistic epistemology (in his later work Carnap is closer to de
Finetti’s views). de Finetti did not view exchangeability orother symmetry requirements as postulates of rationality.
According to him, exchangeability is a personal judgementof an epistemic agent as to the basic structure of a learning
situation. Such a judgement does not arise from the lack ofknowledge but presupposes knowledge about an epistemic
situation.
This view of symmetry assumptions has two importantconsequences, one epistemological and one formal. In thefirst place, for de Finetti and his followers the justificationof generalized rules of succession is only a relative one. An
agent should make inductive inferences provided that she
assumes certain underlying symmetries about the learning
situation. This is unlike the objective Bayesian tradition—
which includes Bayes, Laplace, Keynes, the early work on
inductive logic by Carnap, and others—where symmetry assumptions themselves are viewed not just as assumptions
that one may make, but as principles every rational agent
has to make under certain conditions.
de Finetti’s probabilistic epistemology is thus distinctly
non-foundationalist. There is no bedrock of initial epistemic judgements that would endow all their consequenceswith full rationality because they are themselves requirements of rationality. For de Finetti, rationality is instead
to be found in the interplay of inductive assumptions, suchas Johnson’s sufficientness postulate or exchangeability, andrules for learning from observations. If you use such an
3

4

For finite forms of this result, see [13].

inductive rule but deny its underlying assumptions, you aresimply inconsistent. So, de Finetti requires a kind of relative
rationality: learning from experience should be compatible
with those inductive assumptions that are judged to be true.
The second consequence of de Finetti’s view of symmetry assumptions lifts constraints from inductive logic. If
assumptions such as exchangeability are not thought of as
requirements of rationality but as personal judgements, thenone might consider other kinds of symmetries whenever exchangeability does not seem appropriate. This led de Finetti
to study ‘partial exchangeability’ [10, 11, 13]. One kind ofpartial exchangeability, known as ‘Markov exchangeability’,allows outcomes to depend on previous trials [14, 16, 17, 25,
30, 35]. The type of partial exchangeability most relevant toour analogical inductive logic was investigated by de Finetti
himself [10, 11]. Consider a situation where outcomes can beof different types; e.g., coin flips with two coins, or medical
trials with men and women. Then one may not be willingto judge outcomes to be exchangeable across types but only
within types. There is a representation theorem for this kindof partial exchangeability, from which predictive conditional
probabilities can be derived [10]. The representation is very
similar to (3). Probabilities are again mixtures of independent trials, but now trials need not be identically distributed;
they are identically distributed within types, but need not
be so be across types.
de Finetti viewed partial exchangeability as a type of analogical inference. Take the example of flipping two coins.
The coins are judged to be similar but not indistinguishable
from each other. Because of the analogy between the two
coins, observations from one coin should have some influenceon predictions for the other coin. The analogy comes fromparticular prior distributions on the chances in the mixtureof the representation theorem. The biases of the two coins
may be chosen dependently, but then trials are independent.
Thus this kind of analogy influence does not persist for very
long. This is also a feature of some analogical inductive
logics considered in the next section.

4.

THE PROBLEM OF ANALOGICAL INDUCTIVE INFERENCE

Carnap’s basic system of inductive logic can express analogical influences only to a limited degree [6, 28, 29]. There
have been many attempts to extend Carnap’s original system, and the literature on analogical inductive logic includes
many valuable contributions [1, 5, 8, 15, 18, 19, 23, 26, 27,
12, 28, 29, 31, 32]. I am going to discuss some of thosecontributions in order to motivate my own.
The biggest obstacle to analogical inference in Carnap’s
system is Johnson’s sufficientness postulate (1). Johnson’s
sufficientness postulate makes it impossible that counts nk
of outcomes k other than i influence the predictive probability of i. Skyrms [31] suggests an extension of Carnapian
inductive logic that keeps exchangeability but drops Johnson’s sufficientness postulate. Skyrms’ proposal is further
studied and extended in [15] and [19], and a similar model
is developed for a different context (two families of predicates) in [29]. The basic idea is to use mixtures of inductive methods (2) in order to account for initial analogies
between outcomes. This is equivalent to considering mixtures of Dirichlet distributions instead of Dirichlet distributions in the de Finetti representation. Skyrms discusses

5

this idea in terms of a wheel of fortune, where observationsof an outcome should also increase the predictive probability of nearby outcomes. Using an appropriate mixture of
Dirichlet priors makes this possible. The resulting probability distributions are exchangeable but violate Johnson’s
sufficientness postulate.
The analogy influence exhibited by these kinds of inductive systems is transient. This is due to the fact that the corresponding prior probabilities are exchangeable. Exchangeability implies that the counts of one outcome can only havean indirect effect on the predictive probabilities of other outcomes. To see this, suppose that an outcome k is followedby an outcome i. Then exchanging i with some arbitrary
outcome in the past does not affect the joint probability.
Thus, the effect of counts of k outcomes affects the probability of i outcomes indirectly via the initial parameters in
the mixture of Carnapian inductive logics.
In order to get systems that exhibit a more permanent
analogy influence, exchangeability has to be dropped in addition to Johnson’s sufficientness postulate. The inductive
systems of Costantini [8], Kuipers [23], Niiniluoto [28] and,to a certain extent, Spohn [32] develop inductive logics ofthis type. In these models, the predictive probabilities foroutcome i do not just contain the counts ni but may also
have terms with counts nk of other outcomes. Each of thesesystems is interesting in its own right, but for none of themis it clear what the underlying symmetry assumptions are,or whether they exhibit interesting symmetries at all, andthus they seem a bit ad hoc.
Another criticism of some of these inductive methods was
put forward by Spohn [32] and is also expressed by Costantini [8]. Because counts of all outcomes may explicitly influence the predictive probabilities of an outcome i, thecorresponding inductive logics generally violate a postulate
known as ‘Reichenbach’s axiom’. Reichenbach’s axiom saysthat predictive probabilities have to converge to limiting relative frequencies of sample outcomes, provided that the limit
exists. That is, if X1 , X2 , . . . is an infinite sequence of outcomes such that the limit ni /n exists as n → ∞, thenlim P[Xn+1 = i|X1 , . . . , Xn ] = p.

n→∞Besides Spohn’s own system, Carnap’s basic system andSkyrms’ analogical system meet Reichenbach’s axiom.
I think this critique misses the point of certain forms
of inductive inference. The inductive logics of Costantini
and Niiniluoto may be appropriate when there are underlying probabilistic dependencies between the outcomes. If
these dependencies are persistent, then Reichenbach’s axiom should not hold. The dependencies will not be reflected
in relative frequencies of outcomes, while predictive probabilities should make use of known dependencies. I discuss
this point further in the context of our analogical inductive
logic.
It is not known whether the inductive methods discussed
so far can be derived from a set of axioms analogous tothose underlying the Johnson-Carnap system. This is a significant gap in our knowledge. The set of axioms from which
the Johnson-Carnap continuum of inductive methods (2) isderivable completely specifies inductive assumptions at theobservational level, making it easy to determine whether
one’s priors conform to them. None of the above models of
inductive inference has been treated within this Carnapian
paradigm. Maher’s inductive logic is something of an excep-

tion [26, 27]. He presents a set of axioms for an inductive
logic with two families of predicates. Maher himself discusses problems for the extension of the inductive logic topredicate families containing more than two predicates [27],so I confine my attention to the case of two predicates. I
think that already in this case one point is in need of clarification.
Here is a brief overview of Maher’s proposal. Suppose wehave two families with two predicates. In the language ofrandom variables, this means that we have two sequencesof random variables V1 , V2 , . . . and W1 , W2 , . . ., where eachrandom variable can take on two different values (the possible values being different for the V ’s and the W ’s). For
instance, the first sequence might record whether the coinlands heads and tails, and the second sequence may state
whether the coin is flipped with the right or the left hand.
Maher then considers the so-called ‘Q-predicates’ (‘state descriptions’ in Carnap’s terminology). The Q-predicates are
all possible combinations of basic predicates from the two
families. Again in the language of random variables, thismeans that we consider the sequence of pairs Zn = (Vn , Wn ).
The random vector Zn takes on pairs of values. Since therandom variables Vn and Wn are binary, Zn can take on four
values.
Maher [26] assumes that the infinite sequence Z1 , Z2 , . . .
is exchangeable. It follows from this that its probability
distribution has a de Finetti representation. Maher’s basic idea can then be described as follows. The de Finetti
representation implies that we can construct the probability distribution of Z1 , Z2 , . . . by putting a prior distribution
over the set of possible chances. Since the Zn can take onfour different values, the set of possible chances is the threedimensional simplex ∆4 = {(x1 , . . . , x4 ) ∈ R4 |x1 , . . . , x4 ≥0, x1 + . . . + x4 = 1}. Following an idea by Carnap [6], Maher considers the subset of probability distributions in ∆4
where the two families of predicates are probabilistically independent. This is the set of all (x1 , . . . , x4 ) ∈ ∆4 such thatx1 = (x1 + x2 )(x1 + x3 ), which defines a two-dimensionalsurface in ∆4 that is known as the ‘Wright manifold’ in
population genetics.4
If the prior on ∆4 is a Dirichlet distribution, as in Carnap’s basic system, then any two-dimensional surface in ∆4
has probability zero, since the Dirichlet distribution is absolutely continuous with respect to Lebesgue measure on∆4 . Thus, the Wright manifold has probability zero. Now,
Carnap and Maher propose to look at a mixture between a
Dirichlet distribution and a distribution that puts full weighton the Wright manifold. The resulting inductive logic is a
mixture of Carnap’s basic system on the random variables
Zn and the product of Carnap’s basic systems on the random variables Vn and Wn . The former terms correspond tothe hypothesis that the two predicate families are dependentand the product of the latter two terms to the hypothesisthat they are independent. Using the de Finetti representation, Maher also provides an axiomatic basis from which
this inductive method can be derived. He also shows withthe help of examples that the resulting system seems to leadto plausible numerical results that capture certain analogy
influences.
What type of analogy influences is this model supposed to4
After the population geneticist Sewall Wright. The Wright
manifold is the set of probabilities that make the alleles atdifferent genetic loci independent.

capture? Maher wants to say that some of the Q-predicatesare more similar than others, namely those that share atleast one underlying predicate from the two families. If
we denote the four combinations of values by Q1 = (0, 0),
Q2 = (1, 0), Q3 = (0, 1) and Q4 = (1, 1) then Q1 is similar
to Q2 and Q3 , Q2 to Q1 and Q4 , Q3 is similar to Q1 and
Q4 , and Q4 is similar to Q2 and Q3 . Maher’s goal is to havean inductive logic that respects the analogies based on thesesimilarities. But it is difficult to see the reason why placing positive prior probability on the Wright manifold should
achieve this. There is no straightforward relationship between considering the two predicate families as independentand the intended analogies.
The one reason I can see is the following. The similarity relationships between the Q-predicates described in theprevious paragraph yield four edges in ∆4 between the vertices that are considered similar. These edges are part of
the Wrigth manifold. If one wishes to reflect the analogies
between the Q-predicates in one’s prior, then one’s priordistribution over ∆4 should, presumably, place a sufficientamount of probability weight close to the four edges. One
way to achieve this is by distributing probabilities in an appropriate way on the Wright manifold. But this is neither
necessary nor sufficient. We can endow the Wright manifold
by assigning positive probability only to the barycenter of∆4 (which is an element of the Wright manifold) and probability zero to all the other points in the Wright manifold.
In this case, the overall prior over ∆4 may not place therequired probability weight close to the four edges. On theother hand, we may do exactly that without having to assign positive probability to the Wright manifold. Thus, eventhough it may work in some cases, assigning positive probability to the Wright manifold does not seem to be a principled solution to the analogy problem, which would characterize priors over ∆4 that assign a sufficient probability
weight to the four edges between analogous Q-predicates.

5.

EXTENDING PARTIAL EXCHANGEABILITY

The brief discussion in the previous section should makeit clear that there are many forms of analogical inference.
Each form of analogical inference merits study, and existing inductive logics vary in their degree of solving analogicalinference problems successfully. In the remainder of this paper I would like to propose one form of analogical inductive
inference that is based on de Finetti’s ideas about partial exchangeability and that can be solved within the Carnapian
paradigm.
Recall that partial exchangeability looks at situations withoutcomes of different types. This inductive situation can beillustrated with an example that Achinstein used to criticize Carnap’s original inductive logic [1]. In this example
we observe whether or not different types of metal conduct
electricity. We might, for instance, look at osmium, platinum and rhodium. These three metals are the types in de
Finetti’s setup. Each type may or may not conduct electricity. This defines two outcomes. The analogy between
types comes from the fact that they share certain significant
chemical properties. Because of the analogy between types,
it is reasonable to think that instances where osmium andrhodium where observed to conduct electricity are relevant
for predictions of whether platinum conducts electricity. In

6this case, de Finetti’s theory of partial exchangeability may
be applied with a prior that reflects these analogies.
Partial exchangeability has a similar effect on analogicalinferences as exchangeability: analogy is transient and vanishes in the limit. This makes sense in the example of flipping two coins. The similarity between the two coins may
influence one’s early judgements, but if there are no underlying dependencies between the coins the influence of similarity judgements will diminish. This is reflected by thefact that Reichenbach’s axiom holds for predictive probabilities. But what if there are persistent dependencies between
types? This might arguably be the case in the example
of whether different metals conduct electricity, since there
presumably is an underlying common cause for the relevant
outcome. Another example can be constructed by considering the success of medical trials among males and females.
The types are male and female, and the outcomes (in thesimplest case) are whether the trial was successful or not.
Now, there might be an underlying chancy dependency between types that is influenced by environmental and otherfactors. If this dependency is permanent, this should bereflected in the analogical inductive logic.
How might such an inductive logic look like? The basic
setup has a sequence of outcomes X1 , X2 , . . . and a sequenceof types Y1 , Y2 , . . .. Suppose, for simplicity, that there are
only two types. Predictive probabilities concern future outcomes and not future types. The predictive probability ofobserving outcome i given that it is of type 1 and given pastobservations may be given by
ni1 + βni2 + αi1P.
N1 + βN2 + j αj1(4)
In this formula, Xn = (X1 , . . . XN ), YN = (Y1 , . . . , YN ) are
the past observations of outcomes and types; nij is the number of outcomes i of types j; and N1 and N2 are the total
number of observations of type 1 and 2. The α parameters have the same meaning as in Carnap’s basic system (2).
The β parameter expresses the analogy influence of observations of type 2 on observations of type 1. If β is positive,then i observations of type 2 will have a positive influenceon the predictive probability. This indicates a judgementof positive analogy between types. Moreover, analogy ispermanent—since β is a constant, the analogy influence oftype 1 on type 2 does not vanish as n increases.
There are many ways in which the qualitative features ofthe predictive probability (4) could be formalized. Is (4) justa formula that exhibits some resemblance to Carnap’s original system? Or is there some underlying rationale? To see
what is going on, notice, in the first place, that de Finetti’s
notion of partial exchangeability will not in general allow
predictive probabilities to be of the form as given in (4).
Partial exchangeability implies the following. Suppose that
XN +1 = i, XN +2 = k, XN +3 = j. The predictive probabilityof this sequence of outcomes, given the past and the sequenceof types YN +1 = 1, YN +2 = 2, YN +3 = 1, is equal to thepredictive probability of the sequence XN +1 = j, XN +2 =k, XN +3 = i (in order to get from the first sequence of outcomes to the second we only exchange two outcomes within
the same type). Now suppose that k = j. Then the first
sequence of outcomes is XN +1 = i, XN +2 = j, XN +3 = j
and the second is XN +1 = j, XN +2 = j, XN +3 = i. It isdifficult to see how in this case counts of outcome j of type
P[XN +1 = i|XN , YN , YN +1 = 1] =7

2 can have a constant influence on the predictive probabilityof outcomes j of type 1. If it had, its effect would have to bebalanced exactly against the joint probability for the second
sequence, which may not work in general.5
The same issue does not arise if k 6= i, j. Thus, it seems
reasonable to weaken partial exchangeability in order to allow for persistent analogical influences. We let pn
ikj,st =P[XN +1 = i, XN +2 = k, XN +3 = j|Xn , Yn , YN +1 = s, YN +2 =t, YN +3 = s]. Then generalized partial exchangeability requires, in the first place, thatnpn
ikj,st = pjki,stwhenever k 6= i, j (if k = i or k = j, equality may but neednot hold). Furthermore, let pn
ij,s = P[XN +1 = i, XN +2 =j|Xn , Yn , YN +1 = s, YN +2 = s]. Then generalized partial
exchangeability requires, in the second place, thatnpnij,s = pji,sThe next section is devoted to showing that generalized
partial exchangeability, together with some further assumptions, leads to an interesting analogical inductive logic.

6.

A NEW ANALOGICAL INDUCTIVE LOGIC

The most important additional assumption that we needis a modification of Johnson’s sufficientness postulate:
P[XN +1 = i|XN , YN , YN +1 = j] = fij (ni1 , ni2 , N1 , N2 )(5)
For simplicity, we continue assuming that there are only
two types (for a generalization to a finite number of types,
see [20]). The modified sufficientness postulate says thatpredictive probabilities for an outcome i depend on i, its
type, as well as on the observed counts of i outcomes ofboth types. This is a natural way to allow for analogicalinfluences between types.
We also need two technical postulates. The first one is aregularity assumption to the effect that all finite sequencesof types and outcomes have positive probability; i.e., every
finite pair of sequences X1 , . . . , XN , Y1 , . . . , YN has positive
probability. Finally, we assume that future types do notgive information about the outcome of the next trial. Morespecifically,P[XN +1 = i|X1 , . . . , XN , YN +1 = j](6)

= P[XN +1 = i|X1 , . . . , XN , YN +1 = j, YN +2 = k]
= P[XN +1 = i|X1 , . . . , XN , YN +1 = j, YN +2 = k, YN +3 = l].
This condition is a significant restriction for the applicability of our inductive logic. For example, think of types as
different medical treatments (as in a bandit problem) and ofoutcomes as success or failure. Then a success on the next
trial might not be probabilistically independent of future
treatments.
Suppose now that X1 , X2 , . . . and Y1 , Y2 , . . . are two infinite sequences of outcomes and types for which the foregoing assumptions hold (generalized partial exchangeability,
modified sufficientness postulate, regularity, and conditional
independence (6)). Suppose, in addition, that trials within
types are not independent, and that there are at least threeoutcomes.6 Then the following theorem is true:5
For a precise statement, see my [20], especially Corollary
2.
6
Assuming independence has the same reason as in the case

Theorem 1. There exist positive constants αijPand nonnegative constants
β,
γ
such
that
N
+
βN
+
1
2
i αi1 6=
P0, N2 + γN1 + i αi2 6= 0 and
ni1 + βni2 + αi1PN1 + βN2 + i αi1
ni2 + γni1 + αi2
P
P[XN +1 = i|XN , YN , YN +1 = 2] =N2 + γN1 + i αi2

P[XN +1 = i|XN , YN , YN +1 = 1] =for all N and all 0 ≤ nij ≤ Nj .

This theorem follows from a more general result in my [20]where I prove these assertions for more than two types andallow the total number of trials to be finite.
The sequence of predictive probabilities can be generated
by an urn model (just like the predictive probabilities ofCarnap’s basic system are generated by a Polya urn). Since
the predictive probabilities of our new inductive logic do not
fix the probabilities of types, we may first choose a sequenceof types at random from a distribution that assigns positive
probability to each finite sequence of types. Assume that
we also have an urn for each type containing balls labelledby the outcomes. The initial distribution of balls in urn jdepends on the prior parameters αij . We now start choosing
balls from urns following the sequence of types. Whenever
we choose a ball from an urn, we put it back together withanother label. If the urn is of type 1, we put a ball withweight β into the urn associated with type 2.
The most important difference between our new inductive
logic and Carnap’s basic system (2) are the parameters β, γ.
Are there any good reasons to think that they are analogy
parameters? Let me mention two. First, it can be shownthat β is positive if
P [X2 = i|X1 = i, Y1 = 2, Y2 = 1] > P[X1 = i|Y1 = 1].
Furthermore, β increases as P [X2 = i|X1 = i, Y1 = 2, Y2 =1] approaches 1.7 This means that we have analogy effects oftype 2 on type 1 if observing an outcome of type 2 makes it
sufficiently more likely to observe the same outcome of type1. This is what one would expect of an analogical inference.
The second reason becomes relevant if there are more than
two types. Consider the analogy parameters of two types
with respect to a third one. If one parameter is larger thanthe other, then observing outcomes of the former type raises
the probability of outcomes of the third type more thanobserving outcomes of the second type.8
The inductive logic of Theorem 1 is open to various interpretations. If we interpret the parameters β and γ as analogy parameters, then it is plausible to require that β, γ ≤ 1since, arguably, every type is maximally analogous to itself.
This idea can be captured by another postulate:outcome when it is of type j than observing an outcome i ofanother type. It is easy to see that this forces the analogy
parameters β, γ to be between zero and one.
But we may also think of types in terms of different information sources that are used to predict probabilities ofoutcomes. In this case, β and γ express judgements about
the reliability of the two sources. Consequently, if β > 1 theagent believes that the second information source is moretrustworthy than the first one and that, accordingly, information from type 2 observations should have more weight.
One feature of the inductive logic of Theorem 1 was already discussed earlier in a different context. Our new inductive logic violates Reichenbach’s axiom whenever the analogy parameters β and γ are positive. In this case, predictive
probabilities converge to a convex combination of relative
frequencies of outcomes of the two different types. As remarked earlier, if the underlying process is not assumed tobe essentially independent, this is what one should expect.
Our inductive logic allows types to be probabilistically dependent throughout the process of observation, and so observations from other types don’t necessarily cease to be
relevant for predictive probabilities of one particular type.
Thus, Reichenbach’s axiom should not be postulated for this
case.

7.

CONCLUSION

One of the biggest advantages of our inductive logic isthat there is a precise set of conditions from which it can bederived. These conditions can be thought of as the inductive
assumptions that make the use of our analogical inductive
logic adequate, provided that they are thought to be true.
For most other analogical inductive logics the underlying
assumptions are not as clear, which makes it difficult toapply them.
What I wish to emphasize is that there are different ways
to reason analogically. Accordingly, there is going to bea variety of legitimate analogical inductive logics, and notjust the one inductive logic that fully captures analogicalreasoning. One basic distinguishing feature is suggested by
the foregoing discussion. There are, on the one hand, inductive logics where analogies reflect initial similarities but
are washed out with increasing information. On the other
hand, there are permanent analogical inferences such as in
our inductive logic. Here, analogy persists with increasing
information. Which type of analogy is appropriate depends
on one’s inductive assumptions.

8.

REFERENCES[1] P. Achinstein. Variety and analogy in confirmation
theory. Philosophy of Science, 30:207–221, 1963.
[2] R. Carnap. Logical Foundations of Probability.
University of Chicago Press, Chicago, 1950.
[3] R. Carnap. The Continuum of Inductive Methods.
University of Chicago Press, Chicago, 1952.
[4] R. Carnap. A basic system of inductive logic, part 1.
In Rudolf Carnap and Richard C. Jeffrey, editors,
Studies in Inductive Logic and Probability I, pages
33–165. University of California Press, Los Angeles,
1971.
[5] R. Carnap. A basic system of inductive logic, part 2.
In R. C. Jeffrey, editor, Studies in Inductive Logic andP[X2 = i|X1 = i, Y1 = j, Y2 = j]
≥ P[X2 = i|X1 = i, Y1 = k, Y2 = j]
This says that an observation of an outcome i of type jnever has a lower effect on the predictive probability of thatof the Johnson-Carnap continuum—independence meansthat there is no inductive learning. Since the sufficientness
postulate is empty if there are only two outcomes, this casehas to be treated separately, for example by assuming additivity of predictive probabilities. An alternative approach is
proposed in [7].
7Similar relations hold for γ; see [20].
8
See Proposition 1 in [20].

8Probability II, pages 7–155. University of California
Press, Los Angeles, 1980.
[6] R. Carnap and W. Stegmüller. Induktive Logik undWahrscheinlichkeit. Springer, Wien, 1959.
[7] D. Costantini. The relevance quotient. Erkenntnis,
14:149–157, 1979.
[8] D. Costantini. Analogy by similarity. Erkenntnis,
20:103–114, 1983.
[9] B. de Finetti. La prevision: ses lois logiques ses
sources subjectives. Annales d l’institut Henri
Poincaré, 7:1–68, 1937. Translated in Kyburg, H. E.
and Smokler, H. E., editors, Studies in Subjective
Probability, pages 93–158, Wiley, New York, 1964.
[10] B. de Finetti. Sur la condition d’equivalence partielle.
In Actualités Scientifiques et Industrielles No. 739:
Colloques consacré à la théorie des probabilités,
VIième partie, pages 5–18. Paris, 1938. Translated inJeffrey, R. C., editor, Studies in Inductive Logic andProbability II, pages 193–205, University of California
Press, Los Angeles, 1980.
[11] B. de Finetti. La probabilita e la statistica nei raporti
con l’induzione, secondo i dwersi punti di vista. In
Corso C.I.M.E su Induzione e Statistica. Cremones,
Rome, 1959. Translated in de Finetti, B, Probability,
Induction and Statistics, chapter 9, Wiley, New York,1974.
[12] M. C. di Maio. Predictive probability and analogy bysimilarity in inductive logic. Erkenntnis, 43:369–394,
1995.
[13] P. Diaconis and D. Freedman. De Finetti’s
generalizations of exchangeability. In R. C. Jeffrey,
editor, Studies in Inductive Logic and Probability II,
pages 233–249. University of California Press, Los
Angeles, 1980.
[14] P. Diaconis and D. Freedman. De Finetti’s theorem forMarkov chains. Annals of Probability, 8:115–130, 1980.
[15] R. Festa. Analogy and exchangeability in predictive
inferences. Erkenntnis, 45:89–112, 1997.
[16] S. Fortini, L. Ladelli, G. Petris, and E. Regazzini. On
mixtures of distributions of Markov chains. Stochastic
Processes and their Applications, 100:147–165, 2002.
[17] D. Freedman. Mixtures of Markov processes. Annalsof Mathematical Statistics, 33:114–118, 1962.
[18] M. Hesse. Analogy and confirmation theory.
Philosophy of Science, 31:319–324, 1964.
[19] A. Hill and J. Paris. An analogy principle in inductive
logic. Annals of Pure and Applied Logic, 64:1293–1321,
2013.
[20] S. M. Huttegger. Analogical predictive probabilities.
Manuscript, University of California at Irvine, 2015.
[21] W. E. Johnson. Logic, Part III: The LogicalFoundations of Science. Cambridge University Press,
Cambridge, UK, 1924.
[22] W. E. Johnson. Probability: The deductive andinductive problems. Mind, 41:409–423, 1932.
[23] T. Kuipers. Two types of inductive analogy bysimilarity. Erkenntnis, 21:63–87, 1984.
[24] T. A. F. Kuipers. Studies in Inductive Probability and
Rational Expectation. D. Reidel, Dordrecht, 1978.
[25] T. A. F. Kuipers. Inductive analogy by similarity andproximity. In D. H. Helman, editor, Analogical

9

Reasoning. Kluwer, Dordrecht, 1988.
[26] P. Maher. Probabilities for two properties. Erkenntnis,
52:63–91, 2000.
[27] P. Maher. Probabilities for multiple properties: Themodels of Hesse and Carnap and Kemeny. Erkenntnis,
55:183–216, 2001.
[28] I. Niiniluoto. Analogy and inductive logic. Erkenntnis,
16:1–34, 1981.
[29] J.-W. Romeijn. Analogical predictions for explicit
similarity. Erkenntnis, 2006:253–280, 2006.
[30] B. Skyrms. Inductive logic for Markov chains.
Erkenntnis, 35:439–460, 1991.
[31] B. Skyrms. Analogy by similarity in hyper-Carnapian
inductive logic. In J. Earman, A. I. Janis, G. Massey,
and N. Rescher, editors, Philosophical Problems of theInternal and External Worlds, pages 273–283.
University of Pittsburgh Press, Pittsburgh, 1993.
[32] W. Spohn. Analogy and inductive logic: A note on
Niiniluoto. Erkenntnis, 16:35–52, 1981.
[33] S. L. Zabell. W. E. Johnson’s “sufficientness”
postulate. The Annals of Statistics, 10:1091–1099,
1982.
[34] S. L. Zabell. The Rule of Succession. Erkenntnis,
31:283–321, 1989.
[35] S. L. Zabell. Characterizing Markov exchangeable
sequences. Journal of Theoretical Probability,
8:175–178, 1995.
[36] S. L. Zabell. Carnap and the logic of inductive
inference. In Handbook of the History of Logic, pages
265–309. Elsevier, Amsterdam, 2011.

Human-Agent Decision-making: Combining Theory andPractice[Extended Abstract]
Sarit Kraus

Dept. of Computer Science
Bar-Ilan University
Ramat-Gan, Israel

sarit@cs.biu.ac.il

ABSTRACT
Extensive work has been conducted both in game theory andlogic to model strategic interaction. An important questionis whether we can use these theories to design agents forinteracting with people? On the one hand, they provide
a formal design specification for agent strategies. On theother hand, people do not necessarily adhere to playing inaccordance with these strategies, and their behavior is affected by a multitude of social and psychological factors. In
this paper we will consider the question of whether strategies implied by theories of strategic behavior can be used byautomated agents that interact proficiently with people. We
will focus on automated agents that we built that need tointeract with people in two negotiation settings: bargaining
and deliberation. For bargaining we will study game-theory
based equilibrium agents and for argumentation we will discuss logic-based argumentation theory. We will also consider
security games and persuasion games and will discuss thebenefits of using equilibrium based agents.

Categories and Subject Descriptors
I.2 [ARTIFICIAL INTELLIGENCE]: Miscellaneous

Keywords

Another approach for the development of automated agentsis the (non-classical)-logic approach. The agent is givena logical representation of its environment and its desired
goals, and it reasons logically in order to generate its activities. When interacting with people, the environment consists also of the human model. Yet, modeling people’s behavior is a big challenge. We have incomplete information
about the person’s preferences, and we have to cope with theuncertainties inherent in human decision-making and behavior. Human behavior is diverse, and cannot be satisfactorily
captured by a simple abstract model. In particular, humandecision-making tends to be very noisy: a person may makedifferent strategic decisions in similar situations.
In this paper we survey briefly a few of the agents that webuilt over the years that interact proficiently with people. In
most of the cases, deploying only a game-theory approachor logical-based approach was not beneficial. Heuristics andmachine-learning techniques were augmented into the formal models to lead to agents that interact proficiently withpeople. We will discuss three negotiation settings: multiissue negotiations, games where the players interleave negotiations with resource exchange while attempting to satisfy
their goals and argumentation settings. Finally, we will discuss security games.

2.

Intelligent Agents

1. INTRODUCTION
Agents that interact proficiently with people may be useful for training [27], supporting [23, 12, 11, 1] and even replacing people in many applications [10, 22].
We are considering the agent-human interactions as beinga strategic activity [14]. That is, we assume that when theautomated agent engages in the interaction, it should act
as best it can to realize its preferences. Game theory is themathematical theory of strategic decision-making [28] andthus it seems that game theory might be an appropriate analytical tool for understanding how a strategic agent can andshould act, and might also be useful in both the design of automated agents and protocols for the interactions. However,game theory assumes that all players will act as best they
can to realize their preferences. Unfortunately, humans tend
to make mistakes, and they are affected by cognitive, socialand cultural factors [8, 25, 4]. In particular, people’s observed behavior does not correspond to game theory-based
equilibrium strategies [13, 29].

11

MULTI-ISSUE NEGOTIATIONSOver the years we designed and implemented several automated agents for multi-issue negotiations. In multi-issue
negotiations the players need to reach an agreement on several issues. Each issue is associated with a set of possible
values and the players need to agree on a specific value foreach issue. The negotiations can end with the negotiators
signing an agreement or with one of the sides opting outof the negotiations. In addition, if the crisis does not end
within a pre-specified deadline then the status quo is implemented. Each outcome of the negotiations is associated witha utility score for both players. A summary of our agents ispresented in Table 1.

2.1

EQH agent

The first agent, EQH, that we developed was for crisis scenarios and the setting was quite complex [24]. In addition to
the message exchange in a semi-structured language, players
could take actions during the negotiations and agreements
were not enforceable. In particular, opting out in a crisis isa stochastic action and thus the agents are uncertain about

Table 1: Multi-issue negotiations
Settings
Bilateral, single-issue, full information,complex actions, agreements not enforceable
Bilateral, uncertainty, multi-issue

Agent
Name
EQH
QO-agent
KBAgent
NegoChat

the result. In addition to the main issue of the negotiation or
opting out, there are various other parameters of an agent’s
action. These parameters influence the utility of the negotiators from the crisis. Time plays an important role in thecrisis [42]. The specific scenario we used for the experimental study was a fishing dispute between Canada and Spain.
We formalized the crisis scenario as a game and identified
a subgame-perfect equilibrium. We ran preliminary experiments when the automated agent followed its subgame perfect equilibrium strategy. However, the human negotiators
who negotiated with it became frustrated and the negotiation often ended with no agreement. The frustration of
the human negotiators was mainly due the lack of flexibilityof the agent. Since the proposed and accepted agreementsof the subgame perfect negotiation did not change over thenegotiation time, the agent did not compromise.
To address this limitation of the equilibrium-based agent,
we incorporated several heuristics to the EQH agent. We
allowed the owner of the agent to determine the way theagent will deviate from the equilibrium strategies by determining parameters that influence the agent’s behavior which
are instantiated before the beginning of negotiations. In order to provide the agent with some flexibility when playing
against people, we allowed the agent to consent to agreements that have a lower utility than it would have obtained
according to the relevant subgame perfect equilibrium strategy agreement. Therefore we added the margin parameterthat determines the largest number of points lower than thedesired utility value to which the agent will agree.
An additional parameter is the number of negotiation
units by which the agent will increase or decrease its firstoffer from the agreement specified in its equilibrium strategy. Human negotiators usually begin negotiations with anoffer higher (or lower, depending on the negotiator’s role)
than the value they would eventually like to reach at theend of negotiations. This leaves bargaining space and ouragent uses this type of strategy. Another parameter indicates whether the agent will send the first message in the negotiation or will wait for its opponent to make the first offer.
The default value of this parameter, following some literature recommendations [16], was that the agent will send thefirst offer, since we wanted a trigger to initiate negotiations
with the other agent.
Another heuristic concerns opting out. Given our assumptions, while rational agents will not opt out, people may opt
out. If the agent’s expected utility from opting out is higher
than its expected utility from its opponent opting out, it will

Agent Properties

Scenarios

SPE with manuallydesigned heuristics
Qualitative decision-making
Non-deterministic behavior

fishing
disputejob interview

Machine learning,
qualitative decision-making
non-deterministic behavior
KBagent algorithms, AAT
Anchoring, NLP module

Significancevs people
One role
One role

tobaccojob interview
tobaccoBoth roles

job interview

Both roles

try to predict whether its opponent is going to opt out. If
so, it will opt out first. The heuristic for the prediction ofwhether an opponent will opt out is based on the messages
sent by the opponent. For example, when a threatening
message is received, or when a comment message indicatingthat the negotiations are heading in a dangerous direction
is received, the estimation that the opponent may opt out
increases.
We ran extensive experiments for evaluating the equilibrium agent with the heuristics (EQH agent) [24]. We compared the results of the humans to those of the agents andconcluded that the EQH agent received a higher utility score
playing both roles, but the results were only statistically significant when the agent played just one of the roles. Furthermore, when an agent participates in a negotiation, thesum of the utilities are significantly higher than when two
humans play since the agent always proposes Pareto-optimal
offers while people reach agreements that are not.
While the EQH agent was based on the subgame perfectequilibrium strategies, it required the introduction of many
heuristics, and its success compared with people was only in
one role. The main open question is whether it is possible
to provide formal methodology that will lead to an agent
that is similar to the EQH without the need to manuallydesign the EQH heuristics. Furthermore, we are aiming foran agent that can achieve a significantly higher utility scorethan people in both roles. Toward this challenges, we next
tried to use a qualtative approach, to introduce incomplete
information into the environment and to improve the agent’s
results in both roles.

2.2

QOagent and KBagent

The QOagent was designed to interact with people inenvironments of bilateral negotiations with incomplete information when the agreements consist of multiple issues[26]. With respect to incomplete information, each negotiator keeps his preferences private, though the preferences
might be inferred from the actions of each side (e.g., offers
made or responses to offers proposed). Incomplete information is expressed as uncertainty regarding the utility preferences of the opponent, and it is assumed that there is a
finite set of different negotiator types. These types are associated with different additive utility functions (e.g., one typemight have a long term orientation regarding the final agreement, while the other type might have a more constrainedorientation). Lastly, the negotiation is conducted once witheach opponent. The experiments were run on two distinct

12domains. In the first domain, England and Zimbabwe negotiate in order to reach an agreement evolving from theWorld Health Organization’s Framework Convention on Tobacco Control, the world’s first public health treaty. In thesecond domain a negotiation takes place after a successful
job interview between an employer and a job candidate.
We first formalized the scenario as a Bayesian game andcomputed the Bayesian Nash equilibrium. Though we did
not run simulations of the Bayesian Nash equilibrium agent
against human negotiators, we ran two humans negotiations.
We found out that the opponent’s utility score from the offers suggested by the equilibrium agent are much lower than
the final utility values of the human negotiations. By also
analyzing the simulation process of the human negotiations,
we deduced that without incorporating any heuristics into
the equilibrium agent, the human players would not haveaccepted the offers proposed by it which will lead to low
utility scores for the equilibrium agent, similar to the lowscore of the equilibrium agent in the fishing dispute.
Therefore, we developed the QOAgent. For the decisionmaking process, the approach used by the QOAgent tries
to take the utility of both sides into consideration. While the
QOAgent’s model applies utility functions, it is based on a
non-classical decision-making method, rather than focusing
on maximizing the expected utility: the maximin function
and a qualitative valuation of offers. Using these methods,
the QOAgent generates offers and decides whether to accept or reject proposals it has received. As for incomplete
information, the QOAgent tackles this problem using asimple Bayesian update mechanism. After each action, thismechanism tries to infer which negotiator type best suits the
opponent.
The effectiveness of this method was demonstrated throughextensive empirical experiments by [26].
The results of the experiments showed that the automatedagent achieved higher utility scores than the human counterpart. This can be explained by the nature of our agent
both in reference to accepting offers and generating offers.
Using the decision-making mechanism we allow the agent topropose agreements that are good for it, but also reasonable
for its opponent. In addition, the automated agent makes
straightforward calculations. It evaluates the offer based on
its attributes, and not based on its content. In addition, it
also places more weight on the fact that it loses or gains astime advances. This is not the case, however, when analyzing the logs of the people. It seems that people put moreweight on the content of the offer than on its value. This wasmore evident in the Job Candidate domain with which thehuman subjects could more easily identify. Yet, this does
not explain why, in both domains, similar to the EQH agent
experiments, these results are significant only for one of thesides. In the England-Zimbabwe domain, the results are significant when the agent played the role of England, while in
the Job Candidate domain these results are significant whenit played the role of the job candidate.
In order to improve the QOAgent, we extended it byusing a generic opponent modeling mechanism, which allows
the agent to model its counterpart’s population and adapt its
behavior to that population [32]. The extended agent, called
KBAgent, is an automated negotiator that negotiates witheach person only once, and uses past negotiation sessions ofothers as a knowledge base for generic opponent modeling.
The database containing the a relatively small number of

13

Figure 1: The negotiation system’s interface for NegoChat.
past negotiation sessions is used to extract the likelihood of
acceptance of proposals and which proposals may be offeredby the opposite side. The performance of KBAgent interms of its counter-offer generation and generic opponent
modeling was tested against people in the Tobacco and the
Job interview domains.
The results of these tests indicate that the KBAgent negotiates proficiently with people and even achieves higher
utility score values than the QOAgent. Moreover, theKBAgent achieves significantly better agreements, in termsof utility score, than the human counterparts in both roles.
These results indicate that integrating general opponent modeling into qualtative decision-making is beneficial for automated negotiations.

2.3

NegoChat AgentAll the agents we discussed so far negotiated with the human counterpart either using a structured language or using
a menu-driven interaction. They lack the natural language
processing support required to enable real world types ofinteractions. To address this challenge we first developed
an NLP module that translates the free text of the human player to the agent’s formal language. We modified
the KBagent by adding this module without changing the
KBagent strategy and ran an experiment in which the modified KBagent played with people in a chat-like environment(see Figure 1 for the negotiation system’s interface for chat-based negotiations). We found that simply modifying the
KBagent to include an NLP module is insufficient to create a good agent for such settings and the revised agent
achieved relatively low utility scores. The main observation was that people in chat-based negotiations make andaccept partial agreements and follow issue-by-issue negotiations while the KBagent proposes full offers and has difficulties reaching partial agreements. To address this limitation,
we developed NegoChat, which extended the KBagent focusing on strategies that allow for partial agreements andissue-by-issue interactions. NegoChat’s algorithm is basedon bounded rationality, specifically anchoring and Aspiration Adaptation Theory (AAT). The AAT was used for deciding on the order in which the issues will be discussed.
The agent begins each negotiation interaction by proposing
a full offer based on the KBagent’s strategy, which serves as
its anchor. Assuming this offer is not accepted, NegoChat
then proceeds to negotiate via partial agreements, proposing the next issue for negotiation based on people’s typicalurgency (according to AAT).
We evaluated the NegoChat agent in extensive experiments negotiating with people in the job interview domain.
We compared its performance to the performance of theKBAgent that also negotiated with (different) people using the same NLP module. The NegoChat agent achieved
significantly better agreements (i.e., higher utility score) inless time. However, people playing against KBAgent, on
average, did better. This implies that some of NegoChat’s
success is evidently at the cost of the person’s score andconsequently the social welfare score of this agent is not significantly better than that of KBAgent. As our goal is tomaximize the agent’s utility score this should not be seen
as a fault. However, future generations of automated agentsmay decide to implement different strategies to maximizesocial welfare.

3.

NEGOTIATIONS AND ACTIONS INTERLEAVING

In most situations, negotiation is not done in isolation but
is associated with agreement implementation and other activities. We developed agents that can interact with people
in such settings. These studies were carried out in a configurable system called Colored Trails (CT)1 . It is a gameplayed by two or more participants on a board of coloredsquares. CT is an abstract, conceptually simple but highly
versatile game in which players negotiate and exchange resources to enable them to achieve their individual or group
goals. It provides a realistic analogue to multi-agent task domains, while not requiring extensive domain modeling [15,18]. A summary of the agents we developed are specified inTable 2.

3.1

Revelation games

We considered negotiation settings in which participants
lack information about each other’s preferences, often hindering their ability to reach beneficial agreements [34]. Specifically, we studied a particular class of such settings we call“revelation games”, in which two players are given the choice
to truthfully reveal private information before commencing
two rounds of alternating negotiation. Revealing this information narrows the search space of possible agreements
1

See http://www.eecs.harvard.edu/ai/ct.

and may lead to agreement more quickly, but may also cause
players to be exploited by others (see examples of such gamesin Figure 2). Revelation games combine two types of interactions that have been studied in the past in the economics
literature: Signaling games [39], in which players choose
whether to convey private information to each other, andbargaining [31], in which players engage in multiple negotiation rounds.
We were hopeful that, for revelation games, equilibriumbased agents will interact well with people since behavioral
economics work has shown that people often follow equilibrium strategies [7] when deciding whether to reveal private
information to others. The question is whether this observation will be stronger than our previous observations reported above that people’s behavior in bargaining settings
does not adhere to equilibrium strategies. We formalized thesetting as a Bayesian game and computed two types of perfect Bayesian equilibrium: a separating equilibrium where
both players reveal their type, and a pooling equilibrium
where none of the players reveal their types.
We compared the equilibrium agents with people playing
with other people and with the Sigmoid Acceptance Learning Agent (SIGAL) that we developed [34]. The SIGAL
agent used classical machine learning techniques to predict
how people make and respond to offers during negotiation,
how they reveal information and their response to potential
revelation actions by the agent. This model is integrated
into the agent’s decision tree. We conducted an extensive
empirical study spanning hundreds of human subjects.
Results show that the SIGAL agent was able to outperform people and the equilibrium agents. Furthermore, people outperformed the equilibrium agents. It turned out thatthe negotiation part of the game was more important (withrespect to the utility score) than the revelation part. The
equilibrium agent made very selfish offers in the last roundof the negotiations. Most of these offers were rejected. In
the first round, it made offers that were highly beneficial topeople and most of these offers were accepted, but the small
benefit it incurred in these proposals did not aid its performance. The SIGAL agent, on the other hand, (i) learned
to make offers that were beneficial to people while not compromising its own benefit; and (ii) incrementally revealed
information to people in a way that increased its expected
performance. We were able to adjust SIGAL to new, similar
settings that varied rules and situational parameters of thegame without the need to accumulate new data. However,moving to a completely new setting requires a lot of workcollecting data and adjusting the machine learning module
to the new setting.

3.2

Non-binding agreements

We also studied CT settings of two players in which bothparticipants needed to complete their individual tasks byreaching agreements and exchanging resources, the numberof negotiation rounds were not fixed in advance, and the negotiation protocol was an alternating offers protocol that allowed parties to choose the extent to which they kept each of
their agreements during the negotiation [19]. That is, there
are three phases in each round of the game: negotiation,transfer and movement. The negotiation phase consisted of
two rounds of alternating offers in which the players could
reach an agreement on resource exchange. After each phaseof negotiations, the game moved to the “transfer phase” in14

(a) Symmetric Board Game(b) Asymmetric Board (c) A possible proposal
Game

Figure 2: Two CT revelation games
Table 2: CT games
Settings
Bilateral, Uncertainty, Revelation Game
two-phases: revelation, bargaining
Bilateral, full information, agreements
not enforceable, multiple rounds, three phases:
bargaining, resource exchange, movement
Contract game, three players
two-phases: bargaining, movement

Agent Name
PBE agent
SIGAL
PAL

Agent Properties
Bayesian perfect equilibrium
decision theory, machine learning
decision theory: Influence Diagrammachine learning

Significance vs people
No rolesBoth roles
Both roles

SPE agent
SP-RAP

subgame-perfect equilibrium
subgame-perfect equilibriumbounded rational model of opponent
risk averse

CS roleFigure 3: An example of a CT Board for multiple negotiation
games with unenforceable agreements.

which both players could transfer resources to each other.
The transfer action was done simultaneously, such that neither player could see what the other player transferred until
the end of the phase. A player could choose to transfer moreresources than it agreed to, or any subset of the resources it
agreed to, including not transferring any resources at all. In
the “movement phase” both players could move their icons

15

SP roleon the board one step towards the goal square, provided
they had the necessary resources. Then, the game moved
to the next round, beginning again with negotiation phase.
The game ends when one of the players reaches his goal ordoes not move for two rounds (see an example of one suchgame in Figure 3).
The most important decision of a player in such settings
is whether or not to keep the agreements. Another important decision is whether to accept an offer given by the otherplayer. In subgame perfect equilibrium, the players should
not keep the agreements. Different equilibria may specify
various strategies for the acceptance decision. We ran preliminary experiments and observed that such strategies are
not beneficial when the equilibrium agent interacts with people. Most of the time the agent was not able to reach its
goal, yielding a low utility score.
Galit et al. [19] present the Personality Adaptive Learning(PAL) agent for negotiating with people from different cultures for the CT game where agreements are not enforceable.
The methodology was similar to that of SIGAL (Section 3.1),
combining a decision-theoretic model using a decision treewith classical machine learning techniques to predict how
people respond to offers, and the extent to which they fulfill
agreements.
PAL was evaluated empirically in the Colored Trails (CT)ple outperformed the EQ agent when it played the role ofone of the service providers. We believe that this is mainly
due to people playing the customer role not reaching thegoal even when they have all the needed resources to do so.
To face this problem we then developed an agent termed
SP-RAP which extended the EQ agent in the following two
ways to handle the uncertainty that characterizes humanplay in negotiation: First, it employed a risk averse strategy
using a convex utility function. Second, it reasoned about apossibly bounded rational customer (CS) player by assigning a positive probability p > 0 for the customer player not
Figure 4: An example of a CT Board for the Contract game.
reaching the goal. We assigned a separate value for p foreach country by dividing the number of times the CS player
environment by playing with people in three countries: Lebanon, reached the goal by the total number of games played. Consequently, SP-RAP outperformed people playing the SP role
the U.S., and Israel, in which people are known to varyin all three countries.
widely in their negotiation behavior. The agent was able tooutperform people in all three countries.

3.3

Contract GameWe studied commitment strategies in a three-player CT
game. The game is called Contract Game and is analogous to a market setting in which participants need to reachagreements over contracts and commit to or renege fromcontracts over time in order to succeed [20]. The game comprises three players, two service providers and one customer.
The service providers compete to make repeated contract offers to the customer consisting of resource exchanges in thegame (see an example of one such game in Figure 4). We
formally analyzed the game to compute subgame perfectequilibrium strategies for the customer and service provider
in the game that are based on making contracts containing
commitment offers. To evaluate agents that use the equilibrium strategies, we conducted extensive empirical studies
in three different countries, the U.S., Israel and China. We
ran several configurations in which two human participants
played a single agent participant in various role configurations in the game. Our results showed that the computer
agent using subgame Nash equilibrium strategies for the customer role was able to outperform people playing the samerole in all three countries and obtained statistically significant, higher utility scores than the humans. This was very
surprising since it was the first EQ agent after trying manyequilibrium agents that was able to achieve such results.
In particular, the customer agent made significantly morecommitment type proposals than people did, and requested
significantly more resources from service providers than did
people. It was quite surprising that people playing with itaccepted these offers; in other settings (such as the revelation games) such unfair offers were rejected by people. We
hypothesize that the competition between the two service
providers made such offers more acceptable. In addition,
while in the revelation games the EQ agent had only one
opportunity to make an offer, in the contract game it could
make offers several times (off the equilibrium path) which
we believe also increased the acceptance rate. Also, the customer agent reached one of the goals in all its games andwas able to reach the goal significantly more often than people. This is again quite surprising since at the beginning of
the game the customer has enough resources to reach bothgoals. Since reaching one of the goals is very beneficial tothe customer it is difficult to understand why human players
hadn’t always reached the goal.
While the customer EQ agent outperformed people, peo-

4.

ARGUMENTATION AGENT

An automated agent can help a human when engaging inan argumentative dialog by utilizing its knowledge and computational advantage to provide arguments to him. Argumentation was studied extensively using the well-establishedArgumentation Theory (see [41] for a summary). Therefore,
in the first step in the development of an automated agent
that advised people in such settings, we considered the abilities of Argumentation Theory to predict people’s arguments.
In [38] we presented extensive studies in three experimental settings, varying in complexity, which show the lack ofpredictive power of the existing Argumentation Theory. Second, we used Machine Learning (ML) techniques to provide
a probability distribution over all known arguments given a
partial deliberation. That is, our ML techniques provided
the probability of each argument to be used next in a given
dialog. Our model achieves 76% accuracy when predicting
people’s top three argument choices given a partial deliberation. Last, using the prediction model and the newly
introduced heuristics of relevance, we designed and evaluated the Predictive and Relevance based Heuristic agent(PRH). Through an extensive human study, we showed that
the PRH agent outperforms other agents that propose arguments based on Argumentation Theory, predicted arguments without heuristics or only the heuristics on both axes
we examined: people’s satisfaction from agents and people’s
use of the suggested arguments.

5.

SECURITY GAMESThe last several years have witnessed the successful application of Bayesian Stackelberg games in allocating limited
resources to protect critical infrastructures. These interesting efforts have been led by Prof. Milind Tambe from
USC. The first application is the ARMOR system (Assistantfor Randomized Monitoring over Routes) that has been deployed at the Los Angeles International Airport (LAX) since2007 to randomize checkpoints on the roadways entering the
airport and canine patrol routes within the airport terminals[33, 35]. Other applications include IRIS, a game-theoretic
scheduler for randomized deployment of the US Federal Air
Marshal Service (FAMS) requiring significant scale-up in underlying algorithms, which has been in use since 2009 [40];and PROTECT, which requires further scale up, is deployed
for generating randomized patrol schedules for the US Coast
Guard in Boston, New York, Los Angeles and other ports

16around the US [2, 3]. Furthermore, TRUSTS has been evaluated for deterring fare evasion, suppressing urban crimeand counter-terrorism within the Los Angeles Metro System [44, 21, 9] and GUARDS was earlier tested by the US
Transportation Security Administration (TSA) for security
inside the airport [37].
The evaluation of these systems could be quite limited.
The only system that was truly evaluated in the field is
TRUSTS. We were able to conduct controlled experimentsof our game theoretic resource allocation algorithms. Before this project, the actual evaluation of the deployed security games applications in the field was a major open
challenge. The reasons were twofold. First, previous applications focused on counter-terrorism, therefore controlled
experiments against real adversaries in the field were notfeasible. Second, the number of practical constraints related
to real-world deployments limited the ability of researchers
to conduct head-to-head comparisons
In TRUSTS we were able to address this challenge and run
the largest scale evaluation of security games in the field interms of duration and number of security officials deployed.
We evaluated each component of the system (Fare Evasion,
Counter Terrorism and Crime algorithms) by designing andrunning field experiments. In the context of fare evasion, we
ran an extensive experiment, where we compared schedules
generated using game theory against competing schedules
comprised of a random scheduler, augmented with officers
providing real-time knowledge of the current situation. Our
results showed that our schedules led to statistically significant improvements over the competing schedules, despite
the fact that the latter were improved with real-time knowledge.
In addition, extensive human experiments in the lab wereconducted [36, 30]. These experiments showed that incorporating bounded rational models of the adversaries to the
Stackelberg games improves the performance of the defenders. These results were observed both when the role of adversaries was played by novices and when it was played by
security experts.

6.

PERSUASION GAMESA persuasion game involves two players: a sender who attempts to persuade another agent (the receiver) to take acertain action [17]. Persuasion games are similar to both
negotiation games and security games [43]. They are similar to negotiation and argumentation games since one player
tries to convince another player to do something, as in negotiations. They are also similar to security games in theasymmetry between the players: the sender and the defender
are trying to influence the activities of the receiver and theattacker, respectively. So, it is interesting to check if equilibrium strategies will be beneficial in persuasion games. Furthermore, the incorporation of a bounded rational modelof the receiver will be beneficial to the sender as the incorporation of bounded rational models of the attacker was
beneficial to the defender in security games.
We focused on information disclosure games with twosided uncertainty [5, 6]. This is a special type of persuasion game in which an agent tries to lead a person to takean action that is beneficial to the agent by providing him
with truthful, but possibly partial, information relevant tothe action selection. We first computed the subgame perfect
Bayesian Nash equilibrium of the game assuming the human17

receiver is fully rational. We developed a sender agent that
follows the equilibrium strategy (GTBA agent).
We also developed a machine learning-based model that
effectively predicts people’s behavior in these games and wecalled it Linear weighted-Utility Quantal response (LUQ).
The model we provide assumes that people use a subjective
utility function which is a linear combination for all given
attributes. The model also assumes that while people use
this function as a guideline, they do not always choose theaction with the greatest utility value, however, the higher anaction’s utility value is, the more likely they are to choosethat action. We integrated this model into our persuasion
model and built the LUQA agent.
We ran an extensive empirical study with people in two
different games. In a multi-attribute road selection gamewith two-sided uncertainty, the LUQA agent obtained significantly higher utility points than the GTBA agent. However, in the second game, the Sandwich game, there was no
significant advantage to the machine learning-based model,
and using the game theory-based agent, GTBA, which assumes that people maximize their expected monetary values
is beneficial. We hypothesize that these different results are
due to the nature of the domains. The monetary result plays
an important role in the sandwich game. This is because
the game is played in an environment where a person’s goal
is to make a profit. However, in the road selection gamethe utility scores are associated with time. Thus, it seems
that maximizing expected monetary utility is easier for people than maximizing utility scores that are associated with
time.

7.

DISCUSSIONSThe state-of-the-art agent, NegoChat, for multi-issue negotiations integrates methods from several disciplines: qualitative decision-making, machine learning and heuristics basedon psychological theories. None of the equilibrium agentsthat were developed were successful when interacting withpeople. The reliance on heuristic and machine learning makes
the transfer of NegoChat from one scenario to the other andfrom one culture to the other problematic. This was evident
recently when we tried to run experiments with NegoChat,which was developed based on data collected in Israel andEgypt. We had to spend a lot of time and effort until thistransfer was possible.
Similarly, in most of the cases, the equilibrium agent was
not successful in the CT game settings. The only exception
is the contract game. We believe that the success of theequilibrium agent in the contract game has to do with thespecifics of the game: the competition between the two SPs.
In the contract game, it was extremely difficult to predict
people’s behavior, thus the success of the equilibrium agent
is even more significant.
The same observations were seen in argumentation – theargumentation theory-based agent was not very successful.
Therefore, in all these cases the development of new negotiation agents to new settings requires the collection of data
and the adjustment of the agent to the new settings. Therefore, we strongly believe that the development of theoreticalmodels for the design and implementation of agents that negotiate in multi-issue negotiation settings can be very useful.
However, this is still an open question.
On the other hand, it seems that in security games the
deployment of Stackelberg equilibrium is beneficial (possi-

bly with the incorporation of a bounded rational model ofthe attacker) and similarly in persuasion games where using
subgame perfect Baysian equilibrium is beneficial (possiblywith the incorporation of a bounded rational model of thereceiver).
We hypothesize that this is the case since in security
games and persuasion games the interactions between theagent and the human is quite limited. The attacker or
the receiver needs to choose one action compared to manydecision-making activities that are required from a humannegotiator. Nevertheless, even in security games and to someextent in persuasion games it was shown that taking the limitations of the other player into consideration is beneficial.

8.

ACKNOWLEDGMENT
This work was supported in part by ERC grant #267523.

9.

REFERENCES[1] O. Amir, B. Grosz, E. Law, and R. Stern.
Collaborative health care plan support. In Proceedingsof the Eleenth International Joint Conference on
Autonomous Agents and Multiagent Systems(AAMAS-2013), pages 793–796, St Paul, MN, 2013.
[2] B. An, M. Jain, M. Tambe, and C. Kiekintveld.
Mixed-initiative optimization in security games: A
preliminary report. In AAAI Spring Symposium: Helpme help you: Bridging the gaps in human-agent
collaboration, 2011.
[3] B. An, F. Ordóñez, M. Tambe, E. Shieh, R. Yang,
C. Baldwin, J. DiRenzo III, K. Moretti, B. Maule, and
G. Meyer. A deployed quantal response-based patrol
planning system for the us coast guard. Interfaces,
43(5):400–420, 2013.
[4] D. Ariely. Predictably Irrational. Harper Collins, 2008.
[5] A. Azaria, Z. Rabinovich, C. V. Goldman, and
S. Kraus. Strategic information disclosure to people
with multiple alternatives. ACM Transactions onIntelligent Systems and Technology (TIST), 5(4):64,
2014.
[6] A. Azaria, Z. Rabinovich, S. Kraus, and C. V.
Goldman. Strategic information disclosure to people
with multiple alternatives. In Proceedings of theTwenty-Fifth AAAI Conference on Artificial
Intelligence, AAAI 2011, San Francisco, California,
USA, August 7-11, 2011, 2011.
[7] J. Banks, C. F. Camerer, and D. Porter. Experimental
tests of Nash refinements in signaling games. Gamesand Economic Behavior, 6:1–31, 1994.
[8] M. H. Bazerman and M. A. Neale. Negotiatorrationality and negotiator cognition: The interactiveroles of prescriptive and descriptive research. In H. P.
Young, editor, Negotiation Analysis, pages 109–130.
The University of Michigan Press, 1992.
[9] F. M. Delle Fave, A. X. Jiang, Z. Yin, C. Zhang,
M. Tambe, S. Kraus, and J. P. Sullivan.
Game-theoretic patrolling with dynamic execution
uncertainty and a case study on a real transit system.
Journal of Artificial Intelligence Research, pages
321–367, 2014.
[10] E. Durenard. Professional Automated Trading: Theory
and Practice. John Wiley & Sons, 2013.

[11] A. Elmalech, D. Sarne, and B. J. Grosz. Problem
restructuring for better decision making in recurring
decision situations. Autonomous Agents andMulti-Agent Systems, 29(1):1–39, 2015.
[12] A. Elmalech, D. Sarne, A. Rosenfeld, and E. S. Erez.
When suboptimal rules. In Proceedings of theTwenty-Ninth AAAI Conference on Artificial
Intelligence, pages 1313–1319, 2015.
[13] I. Erev and A. E. Roth. Predicting how people playgames: Reinforcement learning in experimental gameswith unique, mixed strategy equilibria. American
Economic Review, 88(4):848–881, 1998.
[14] S. Fatima, S. Kraus, and M. Wooldridge. Principles ofAutomated Negotiation. Cambridge University Press,
2014.
[15] Y. Gal, B. Grosz, S. Kraus, A. Pfeffer, and S. Shieber.
Agent decision-making in open mixed networks.
Artificial Intelligence, 174(18):1460–1480, 2010.
[16] A. D. Galinsky and T. Mussweiler. First offers asanchors: the role of perspective-taking and negotiatorfocus. Journal of personality and social psychology,
81(4):657, 2001.
[17] J. Glazer and A. Rubinstein. A study in the
pragmatics of persuasion: a game theoretical
approach. Theoretical Economics, 1(4):395–410, 2006.
[18] B. J. Grosz, S. Kraus, S. Talman, B. Stossel, and
M. Havlin. The influence of social dependencies on
decision-making: Initial investigations with a new
game. In Proceedings of the Third International Joint
Conference on Autonomous Agents and Multiagent
Systems-Volume 2, pages 782–789. IEEE Computer
Society, 2004.
[19] G. Haim, Y. Gal, M. Gelfand, and S. Kraus. A
cultural sensitive agent for human-computer
negotiation. In Proceedings of the EleventhInternational Joint Conference on Autonomous Agentsand Multiagent Systems (AAMAS-2012), pages
451–458, Valencia, Spain, 2012.
[20] G. Haim, Y. K. Gal, S. Kraus, and B. An.
Human-computer negotiation in three-player market
settings. In Proc. of ECAI14, pages 417–422, 2014.
[21] A. X. Jiang, Z. Yin, C. Zhang, M. Tambe, and
S. Kraus. Game-theoretic randomization for security
patrolling with dynamic execution uncertainty. In
Proceedings of the 2013 international conference on
Autonomous agents and multi-agent systems, pages
207–214. International Foundation for Autonomous
Agents and Multiagent Systems, 2013.
[22] K. Kauppi, A. Brandon-Jones, S. Ronchi, and E. van
Raaij. Tools without skills: Exploring the moderating
effect of absorptive capacity on the relationshipbetween e-purchasing tools and category performance.
International Journal of Operations & Production
Management, 33(7):828–857, 2013.
[23] G. Kersten and H. Lai. Negotiation support ande-negotiation systems: an overview. Group Decision
and Negotiation, 16(6):553–586, 2007.
[24] S. Kraus, P. Hoz-Weiss, J. Wilkenfeld, D. R.
Andersen, and A. Pate. Resolving crises throughautomated bilateral negotiations. Artificial
Intelligence, 172(1):1–18, 2008.
[25] D. A. Lax and J. K. Sebenius. Thinking coalitionally:18

party arithmetic, process opportunism, and strategic
sequencing. In H. P. Young, editor, Negotiation
Analysis, pages 153–193. The University of Michigan
Press, 1992.
[26] R. Lin, S. Kraus, J. Wilkenfeld, and J. Barry.
Negotiating with bounded rational agents inenvironments with incomplete information using an
automated agent. Artificial Intelligence,
172(6):823–851, 2008.
[27] R. Lin, Y. Oshrat, and S. Kraus. Investigating thebenefits of automated negotiations in enhancing
people’s negotiation skills. In Proceedings of the EighthInternational Joint Conference on Autonomous Agentsand Multiagent Systems (AAMAS-2009), pages
345–352, Budapest, Hungary, 2009.
[28] M. Maschler, E. Solan, and S. Zamir. Game Theory.
Cambridge University Press: Cambridge, England,
2013.
[29] R. D. McKelvey and T. R. Palfrey. An experimental
study of the centipede game. Econometrica,
60(4):803–836, 1992.
[30] T. H. Nguyen, R. Yang, A. Azaria, S. Kraus, and
M. Tambe. Analyzing the effectiveness of adversarymodeling in security games. In Proc of AAAI 2013,2013.
[31] M. J. Osborne and A. Rubinstein. A course in gametheory. MIT press, 1994.
[32] Y. Oshrat, R. Lin, and S. Kraus. Facing the challengeof human-agent negotiations via effective general
opponent modeling. In Proceedings of the EighthInternational Joint Conference on Autonomous Agentsand Multiagent Systems (AAMAS-2009), pages
377–384, Budapest, Hungary, 2009.
[33] P. Paruchuri, J. P. Pearce, J. Marecki, M. Tambe,
F. Ordonez, and S. Kraus. Playing games for security:
an efficient exact algorithm for solving bayesian
stackelberg games. In Proceedings of the 7thinternational joint conference on Autonomous agentsand multiagent systems-Volume 2, pages 895–902.
International Foundation for Autonomous Agents andMultiagent Systems, 2008.
[34] N. Peled, K. Gal, and S. Kraus. A study ofcomputational and human strategies in revelation
games. Autonomous Agents and Multi-Agent Systems,
29(1):73–97, 2015.
[35] J. Pita, M. Jain, J. Marecki, F. Ordóñez, C. Portway,
M. Tambe, C. Western, P. Paruchuri, and S. Kraus.
Deployed armor protection: the application of a gametheoretic model for security at the los angelesinternational airport. In Proceedings of the 7thinternational joint conference on Autonomous agentsand multiagent systems: industrial track, pages
125–132. International Foundation for Autonomous
Agents and Multiagent Systems, 2008.
[36] J. Pita, M. Jain, M. Tambe, F. Ordóñez, and
S. Kraus. Robust solutions to stackelberg games:
Addressing bounded rationality and limited
observations in human cognition. Artificial
Intelligence, 174(15):1142–1171, 2010.
[37] J. Pita, M. Tambe, C. Kiekintveld, S. Cullen, andE. Steigerwald. Guards - innovative application ofgame theory for national airport security. In IJCAI

19

Proceedings-International Joint Conference on
Artificial Intelligence, page 2710, 2011.
[38] A. Rosenfeld and S. Kraus. Providing arguments indiscussions based on the prediction of humanargumentative behavior. In Proc. of AAAI, 2015.
[39] A. M. Spence. Market signaling: Informationaltransfer in hiring and related screening processes,
volume 143. Harvard Univ Pr, 1974.
[40] J. Tsai, C. Kiekintveld, F. Ordonez, M. Tambe, and
S. Rathi. Iris-a tool for strategic security allocation intransportation networks. In Proc. of AAMAS09, 2009.
[41] D. Walton. Argumentation theory: A very short
introduction. In Argumentation in artificial
intelligence, pages 1–22. Springer, 2009.
[42] J. Wilkenfeld, S. Kraus, K. M. Holley, and M. A.
Harris. Genie: A decision support system for crisis
negotiations. Decision Support Systems,
14(4):369–391, 1995.
[43] H. Xu, Z. Rabinovich, S. Dughmi, and M. Tambe.
Exploring information asymmetry in two-stage
security games. In Proc of AAAI, 2015.
[44] Z. Yin, A. X. Jiang, M. Tambe, C. Kiekintveld,
K. Leyton-Brown, T. Sandholm, and J. P. Sullivan.
Trusts: Scheduling randomized patrols for fareinspection in transit systems using game theory. AI
Magazine, 33(4):59, 2012.

Rationality and Beliefs in Dynamic Games:
Revealed-Preference Foundations[Extended Abstract]
Marciano Siniscalchi

Economics Department, Northwestern University
2001 Sheridan Rd
Evanston, IL 60611, USA

marciano@northwestern.edu

General Terms
Game Theory

Keywords
Conditional Probability Systems, Decision Theory, Game
Theory, Sequential Rationality

1. EXTENDED ABSTRACT
Sequential rationality is the prevalent notion of best response for dynamic games; it is an essential part of the definition of sequential equilibrium [9], perfect Bayesian equilibrium [6], and extensive-form rationalizability [11]. Abstracting from notational and other minor differences, a strategy
si of player i is sequentially rational if, beginning at any
information set where i moves, si specifies a sequence of actions that is optimal given the beliefs that i holds at that
information set about the play of her opponents.
While this notion is central to the theory of dynamic
games, it raises both practical and methodological concerns.
From a practical standpoint, it is not obvious how to reliably
ascertain which strategy a player follows in a given dynamic
game, and a fortiori whether it is or isn’t sequentially rational. Consider a three-stage Centipede game [13], playedby Ann (who moves at the first and third nodes) and Bob.
Suppose that, as predicted by backward induction, Ann ends
the game at the first node by moving Down. Then, we areunable to observe Bob’s intended choice at the second node.
Furthermore, to determine whether such a choice would havebeen optimal, had Ann chosen Across instead, we need toconsider Bob’s beliefs conditional upon an event that Bob
does not expect to occur—a zero-probability event. While
formally we can represent such beliefs, it is not clear how anexperimenter might elicit them in practice.
Reinhard Selten’s strategy method [16] is a widely used
experimental procedure that is intended to elicit players’
intended strategy choices in dynamic games. There is evidence that the strategy method is an effective elicitation
procedure: see, e.g., [3]. But this finding actually raises further questions. The strategy method essentially asks players
to simultaneously commit to a strategy, which is then implemented by the experimenter without possibility of subsequent intervention by the subjects. This reduces the original
dynamic game to one in which players face non-trivial moves
only in the initial stage; furthermore, such moves are simultaneous. Standard solution concepts such as sequential or21

perfect Bayesian equilibrium predict that players will maximize ex-ante expected payoffs when the strategy method
is employed; therefore, such solution concepts do allow subjects to commit to strategies that are not sequentially rational in the original game. Refinements that incorporate
the notion of invariance [8] do imply that subjects will only
commit to strategies that are sequentially rational in theoriginal game. However, there is ample experimental evidence that contradicts the invariance hypothesis [5, 15, 4,
7]. Thus, the received theory cannot at the same time explain the effectiveness of the strategy method, and accountfor violations of the invariance hypothesis.
These practical issues hint at a deeper methodological concern. Economics has long embraced the revealed-preference
approach: assumptions about agents’ tastes and beliefs shouldbe testable, or elicitable, on the basis of observable choices insuitably designed problems. To date, rationality and beliefs
in dynamic games have not been subject to analysis fromthe revealed-preference perspective. Formal definitions ofsequential rationality build upon expected-utility maximization. However, the revealed-preference foundations for expected utility [14, 1] concern atemporal, or one-shot choices.
Furthermore, extensions of expected-utility theory to dynamic choice problems are wholly silent about behavior conditional upon ex-ante zero probability events. Of course, theanalysis of intended choices following unexpected moves isat the heart of dynamic game theory. Thus, the received
decision theory is insufficient to provide foundations to theanalysis of dynamic games.
The objective of this project is to provide such a foundation. This entails two contributions. The first is to define a novel choice criterion for dynamic decision problems
and games, sequential preference, so as to satify two criteria. First, the proposed criterion implies sequential rationality. Second, it allows preferences over strategies to beelicited from ex-ante choices, using a version of the strategy
method. In particular, sequential preferences provide a theoretical rationale for the use of this common experimental
procedure, as well as a method to elicit conditional beliefs
following zero-probability events.
Building on the finding that sequential preferences are indeed elicitable, the second contribution of this project is toprovide a behavioral, or axiomatic, characterization of the
proposed choice criterion. This is based on a suitable adaptation of the Anscombe-Aumann [1] axioms.
Finally, in the analysis of sequential preferences, there arenatural connections to conditional probability systems [12,10] and lexicographic probability systems [2]. These are explored, and their implications for game-theoretic analysis
are discussed.
The project is carried out in two papers. The first, [18], introduces sequential preferences, and analyzes elicitation andthe strategy method. The second, [17], provides behavioral
foundations.

2.

[17] M. Siniscalchi. Foundations for sequential preferences.
mimeo, Northwestern University, 2015.
[18] M. Siniscalchi. Sequential preferences and sequential
rationality. mimeo, Northwestern University, 2015.

REFERENCES[1] F. J. Anscombe and R. J. Aumann. A definition ofsubjective probability. Annals of Mathematical
Statistics, 34:199–205, 1963.
[2] L. Blume, A. Branderburger, and E. Dekel.
Lexicographic probabilities and choice under
uncertainty. Econometrica, 59:61–79, 1991.
[3] J. Brandts and G. Charness. The strategy versus the
direct-response method: a first survey of experimental
comparisons. Experimental Economics, 14(3):375–398,
2011.
[4] D. J. Cooper and J. B. Van Huyck. Evidence on theequivalence of the strategic and extensive form
representation of games. Journal of Economic Theory,110(2):290–308, 2003.
[5] R. Cooper, D. V. DeJong, R. Forsythe, and T. W.
Ross. Forward induction in the battle-of-the-sexes
games. American Economic Review, 83(5):1303–1316,
1993.
[6] D. Fudenberg and J. Tirole. Perfect Bayesianequilibrium and sequential equilibrium* 1. Journal of
Economic Theory, 53(2):236–260, 1991.
[7] S. Huck and W. Müller. Burning money and (pseudo)
first-mover advantages: an experimental study on
forward induction. Games and Economic Behavior,
51(1):109–127, 2005.
[8] E. Kohlberg and J. Mertens. On the strategic stabilityof equilibria. Econometrica: Journal of the
Econometric Society, 54(5):1003–1037, 1986.
[9] D. Kreps and R. Wilson. Sequential equilibria.
Econometrica: Journal of the Econometric Society,50(4):863–894, 1982.
[10] R. B. Myerson. Axiomatic foundations of bayesian
decision theory. Discussion Paper 671, The Center for
Mathematical Studies in Economics and Management
Science, Northwestern University, January 1986.
[11] D. G. Pearce. Rationalizable strategic behavior andthe problem of perfection. Econometrica,
52:1029–1050, 1984.
[12] A. Rényi. On a new axiomatic theory of probability.
Acta Mathematica Hungarica, 6(3):285–335, 1955.
[13] R. Rosenthal. Games of Perfect Information,
Predatory Pricing and the Chain-Store Paradox.
Journal of Economic Theory, 25(1):92–100, 1981.
[14] L. Savage. The foundations of statistics. Dover Pubns,
1972.
[15] A. Schotter, K. Weigelt, and C. Wilson. A laboratory
investigation of multiperson rationality andpresentation effects. Games and Economic behavior,
6(3):445–468, 1994.
[16] R. Selten. Ein oligopolexperiment mit preisvariation
und investition. Beiträge zur experimentellen
Wirtschaftsforschung, ed. by H. Sauermann, JCB
Mohr (Paul Siebeck), Tübingen, pages 103–135, 1967.

22

Recent Methodological Advances in Causal Discovery andInference[Invited Paper]Peter Spirtes

Department of Philosophy
Carnegie Mellon University

ps7z@andrew.cmu.edu

Kun Zhang

MPI for Intelligent Systems
72076 Tübingen, Germany &Info. Sci. Inst., USC
4676 Admiralty Way, CA 90292

kzhang@tuebingen.mpg.de
ABSTRACTThis paper aims to give a broad coverage of central concepts and principles involved in automated causal inference
and emerging approaches to causal discovery from i.i.d data
and from time series. After reviewing concepts including
manipulations, causal models, sample predictive modeling,causal predictive modeling, and structural equation models,
we present the constraint-based approach to causal discovery, which relies on the conditional independence relationships in the data, and discuss the assumptions underlying its
validity. We then focus on causal discovery based on structural equations models, in which a key issue is the identifiability of the causal structure implied by appropriately defined structural equation models: in the two-variable case,under what conditions (and why) is the causal direction between the two variables identifiable? We show that the independence between the error term and causes, together withappropriate structural constraints on the structural equation, makes it possible. Next, we report some recent advances in causal discovery from time series. Assuming thatthe causal relations are linear with non-Gaussian noise, we
study two problems which are traditionally difficult to solve,
namely, causal discovery from subsampled data and that in
the presence of confounding time series. Finally, we list a
number of open questions in the field of causal discovery andinference.

Categories and Subject Descriptors
H.4 [Information Systems Applications]: Miscellaneous;
I.2.4 [Artificial Intelligence]: Knowledge Representation
Formalisms and Methods—Miscellaneous

General Terms

have (i.e., to find a generative model), and to predict what
the values of those variables would be if the naturally occurring mechanisms in a population1 were subject to outside
manipulations. For example, a randomized experiment is
one kind of manipulation, which substitutes the outcome ofa randomizing device to set the value of a variable, such aswhether or not a particular diet is used, instead of the naturally occurring mechanism that determines diet. In nonexperimental settings, biologists gather data about the geneactivation levels in normally operating systems, and seek tounderstand which genes affect the activation levels of which
other genes, and seek to predict what the effects of intervening to turn some genes on or off would be; epidemiologists
gather data about dietary habits and life expectancy in thegeneral population and seek to find what dietary factors affect life expectancy and to predict the effects of advisingpeople to change their diets. Finding answers to questions
about the mechanisms by which variables come to take on
values, or predicting the value of a variable after some othervariable has been manipulated, is characteristic of causal
inference. If only observational (non-experimental) data isavailable, predicting the effects of manipulations typicallyinvolves drawing samples from one density (of the unmanipulated population) and making inferences about the valuesof a variable in a population that has a different density (ofthe manipulation population).
Many of the basic problems and basic assumptions remain the same across domains. In addition, although there
are some superficial similarities between traditional supervised machine learning problems and causal inference (e.g.,both employ model search and feature selection, the kindsof models employed overlap, some model scores can be used
for both purposes), these similarities can mask some very
important differences between the two kinds of problems.

1.1

Algorithms, Theory

Keywords
Causal inference, causal discovery, structural equation model,
conditional independence, statistical independence, identifiability

1. INTRODUCTION
The goal of many sciences is to understand the mechanisms by which variables came to take on the values they

23

History

Traditionally, there have been a number of different approaches to causal discovery. The gold standard of causal
discovery has typically been to perform planned or randomized experiments [10]. There are obvious practical and ethical considerations that limit the application of randomized
1Here, the “population” is simply a collection of instantiations of a set of random variables. For example, it could consist of a set of satellite readings and rainfall rates in different
locations at a given time, or the readings of a single satellite
and rainfall rate over time, or a combination of these.

experiments in many instances, particularly on human beings. Moreover, recent data collection techniques and causal
inference problems raise several practical difficulties regarding the number of experiments that need to be performed inorder to answer all of the outstanding questions [8, 9].

2.

1. Each manipulation succeeds, i.e., if the manipulation
is designated as setting the density to p0 (X|O), thenthe post-manipulation density is p0 (X|O).

MANIPULATING AND CONDITIONING

Conditioning maps a given joint density, and a given subpopulation (typically specified by a set of values for randomvariables) into a new density. The conditional density is afunction of the joint density over the random variables, anda set of values for a set of random variables.2 The estimation of a conditional probability is often non-trivial because
the number of measurements in which the variables conditioned on that take on a particular value might be small.
A large part of statistics and machine learning is devoted
to estimating conditional probabilities from realistic sample
sizes under a variety of assumptions.
More generally, suppose the goal is to find a “good” predictor of the value of some target variable Y from the valuesof the observed covariates O, for a unit. We will refer to thisas Problem 1, described more formally below. Ultimately,
the prediction of the value of Y is performed by some prediction function Ŷn (O). One traditional measure of how good
the predictor Ŷn (O) is in predicting Y is the mean squared
prediction error (MSPE), which is equal to E[(Y − Ŷn (O))2 ],where the expected value is taken with respect to the densityp(O, Y ) [1].3
Problem 1: Sample predictive modeling
Input: i.i.d. samples from a population with densityp(O, Y ), background assumptions, and a target variable
Y whose value is to be predicted.
Output: Ŷn (O), a predictor of Y from O that has a small
MSPE.
In addition to predicting future values of random variablesfrom the present and past values, conditional probabilities
are also useful for predicting hidden values at the current
time.

2.1

ferent actions under consideration are manipulations.4 We
designate the density of a set of variables V after a manipulation M as p(V||M ). Each manipulation is assumed to be
an ideal manipulation in the following senses:

Manipulated Probabilities

A manipulated density results from taking action on agiven population – it may or may not be equal to any observational conditional density, depending upon what thecausal relations between variables are. Manipulated probability densities are the appropriate probability densities touse when making predictions about the effects of taking actions (“manipulating” or “doing”) on a give population (e.g.,assigning satellite readings), rather than observing (“seeing”)
the values of given variables. A manipulation M specifies a
new conditional probability density for some set of variables.
If X and O are sets of variables with density p(X|O), a
manipulation M changes the density to some new density
p0 (X|O). Manipulated probabilities are the probabilities
that are implicitly used in decision theory, where the dif2
In order to avoid technicalities, we will assume that the setof values conditioned on do not have measure 0.
3
Other measures of prediction error, such as the absolutevalue of prediction error or optimizing certain decision problems could be used, but would not substantially change thegeneral approach taken here.

2. There is no fat hand, i.e., each manipulation directly
affects only the variables manipulated.
A probability model specifies a density over a set of random variables O. A causal model specifies a set of densities
over a set of random variables O, one for each possible manipulation M of the random variables in O, including thenull manipulation. Hence a probability model is a memberof a causal model.
Given a set of variables V, the direct causal relations
among the variables can be represented by a directed graph,
where the variables in V are the vertices, and there is anedge from A to B if A is a direct cause of B relative to V.
We will refer to the problem of estimating manipulated
densities given a sample from a marginal unmanipulated
density, a (possibly empty) set of samples from manipulated
densities, and background assumptions, as Problem 2; it isstated more formally below. In contrast to conditional probabilities, which can be estimated from samples from a population, typically the gold standard for estimating manipulated densities is an experiment, often a randomized trial.
However, in many cases experiments are too expensive, toodifficult, or not ethical to carry out. This raises the questionof what can be determined about manipulated probability
densities from samples from a population, possibly in combination with a limited number of randomized trials. The
problem is even more difficult because the inference is madefrom a set of measured random variables O from samples
that might not contain variables that are causes of multiple
variables in O.
Problem 2 is usually broken into two parts: finding a
set of causal models from sample data, some manipulations(experiments) and background assumptions, and predicting
the effects of a manipulation given a causal model. Here, a“causal model” (Section 3) specifies for each possible manipulation that can be performed on the population (including
the manipulation that does nothing to a population) a postmanipulation density over a given set of variables.
Problem 2: Statistical causal predictive modeling
Input: i.i.d. samples from a population with densityp(O, Y ), a (possibly empty) set of i.i.d. samples from manipulated densities p(O, Y ||M1 ), ..., p(O, Y ||Mn ), a manipulation M , background assumptions, and a target variable Y whose post-manipulation value is to be predicted.
Output: Ŷ (O||M ), a predictor of the value of Y from O
after manipulation M that has a small MSPE.
4Here, p0 is not a derivative of p; the prime after the p merely
indicates that a new function has been introduced. The useof manipulated probability densities in decision theory isoften not explicit. The assumption that the density of statesof nature are independent of the actions taken (act-state
independence) is one way to ensure that the manipulated
densities that are needed are equal to observed conditional
densities that can be measured.

24

Problem 2a: Constructing Causal Models from
Sample Data
Input: i.i.d. samples from a population with density p(O),a (possibly empty) set of i.i.d. samples from manipulated densities p(O||M1 ), ..., p(O||Mn ), and backgroundassumptions.
Output: A set of causal models that is as small as possible,and contains an approximately true causal model.
Problem 2b: Predicting the Effects ofManipulations from Causal Models
A set C of causal models over a set of variables O and Y ,a manipulation M , and a target variable Y .
Output: Ŷ (O||M ) if one exists, and an output of “nofunction” otherwise.
The reason that the stated goal for the output of Problem
2a is a set of causal models, rather than a single causal modelis that it is in some cases it is not possible to reliably finda true causal model given the inputs. Furthermore, in contrast to predictive models, even if it a true causal model canbe inferred from a sample from the unmanipulated population, it generally cannot be validated on a sample from theunmanipulated population, because a causal model contains
predictions about a manipulated population that might not
actually exist. This has been a serious impediment to theimprovement of algorithms for constructing causal models,
because it makes evaluating the performance of such algorithms difficult. It is possible to evaluate causal inference
algorithms on simulated data, to employ background knowledge to check the performance of algorithms, and to conduct
limited (due to expense, time, and ethical constraints) experiments, but these serve as only partial checks how algorithms perform on real data in a wide variety of domains.

3.

STRUCTURAL EQUATION MODELSThe set of random variables in a structural equation model(SEM) can be divided into two subsets, the “error variables”
or “error terms,” and the substantive variables (for which
there is no standard terminology in the literature). The substantive variables are the variables of interest, but they are
not necessarily all observed. Each substantive variable X isa function of other substantive variables V, and a uniqueerror term εX ; i.e., X := f (V, εX ). We use an assignment
operator, rather than an equality operator because the equations are interpreted causally; manipulating a variable in V
can lead to a change in the value of X.
Each SEM is associated with a directed graph whose vertices include the substantive variables, and that represents
both the causal structure of the model and the form of thestructural equations. There is a directed edge from A to B
(A → B) if the coefficient of A in the structural equationfor B is non-zero. In a linear SEM, the coefficient bB,A of A
in the structural equation for B is the structural coefficientassociated with the edge A → B. In general, the graph of
a SEM may have cycles (i.e., directed paths from a variable to itself), and may explicitly include error terms withdouble-headed arrows between them to represent that theerror terms are dependent (e.g., εA ↔ εB ); if no such edge
exists in the graph, the error terms are assumed to be independent. If a variable has no arrow directed into it, then it
is exogenous; otherwise it is endogenous. In SEM K(θ) de-25

picted in Figure 1(i) (where θ is the set of parameter values
for K) A is exogenous and B and R are endogenous. If thegraph has no directed cycles and no double-headed arrows,
then it is a directed acyclic graph (DAG).
Given independent error terms in SEM K, for each θ,
SEM K entails both a set of conditional independence relations among the substantive variables, and that the joint
density over the substantive variables factors according to
the graph, i.e., the joint density can be expressed as the product of the density of each variable conditional on its parents
in the graph. For example, p(A, B, R) = p(A)p(B|A)p(R|A)
for all θ. This factorization in turn is equivalent to a set of
conditional independence relations among the substantive
variables [21].
Ip (X, Y|Z) denotes that X is independent of Y conditional on Z in density p, i.e., p(X|Y, Z) = p(X|Z) for allp(X|Z) 6= 0. (In cases where it does not create any ambiguity, the subscript p will be dropped). If a SEM M with parameter values θ (represented by M (θ)) entails that X is independent of Y conditional on Z, we write IM (θ) (X, Y|Z). If
a SEM with fixed causal graph M entails that IM (Θ) (X, Y|Z)
for all possible parameter values Θ we write IM (X, Y|Z). In
that case we say that M entails I(X, Y|Z). It is possible todetermine whether IM (X, Y|Z) from the graph of M using
the purely graphical criterion, “d-separation” [26].
A Bayesian network is a pair hG, pi, where G is a directed
acyclic graph and a p is a probability density such that if X
and Y are d-separated conditional on Z in G, then X andY are independent conditional on Z in G. If the error terms
in a SEM with a DAG G are jointly independent, and p(V)is the entailed density over the substantive variables, thenhG, p(V)i is a Bayesian network.

3.1

Representing Manipulations in a SEM

Given a linear SEM, a manipulation of a variable Xi in a
population P
can be described by the following kind of equation: Xi = Xj ∈PA(Xi ) bi,j Xj +εi , where all of the variables
are the post-manipulation variables, PA(Xi ) is a new set ofcauses of Xi (which are included in the set of non-effects of
Xi in the unmanipulated population). A simple special caseis where Xi is set to a constant c.
In a causal model such as SEM K(θ), the post-manipulation
population is represented in the following way, as shown in
Figure 1. The result of modifying the set of structural equations in this way can lead to a density in the randomized
population that is not necessarily the same as the density
in any subpopulation of the general population. (For moredetails see [27, 36].) See Figure 1 for examples of manipulations to SEM K.
A set S of variables is causally sufficient if every variable
H that is a direct cause (relative to S ∪ {H}) of any pair ofvariables in S is also in S. Intuitively, a set of variables S iscausally sufficient if no common direct causes (relative to S)
have been left out of S. If SEM K is true then {A, B, R} iscausally sufficient, but {B, R} is not because A is a common
direct cause of B and R relative to {A, B, R} but is not in{B, R}. If the observed set of variables is not causally sufficient, then the causal model is said to contain unobserved
common causes, hidden common causes, or latent variables.

4.

ASSUMPTIONS

The following assumptions are often used to relate causal

Figure 1: (i) Unmanipulated causal graph K; (ii) B
Manipulated to 5; (iii) A Manipulated to 5

relations to probability densities.

4.1

The Causal Markov Assumption

Causal Markov Assumption: For causally sufficient
sets of variables, all variables are independent of the their
non-effects (non-descendants in the causal graph) conditionalon their direct causes (parents in the causal graph) [36].
The Causal Markov Assumption is an oversimplification
because it basically assumes that all associations between
variables are due to causal relations. There are several otherways that associations can be produced.
First, conditioning on a common descendant can produce
a conditional dependency. For example, if sex and intelligence are unassociated in the population, but only the mostintelligent women attend graduate school, while men witha wider range of intelligence attend graduate school, thensex and intelligence will be associated in a sample consistingof graduate students (i.e., sex and intelligence cause graduate school attendance, which has been conditioned on in
the sample.) See [37] for a discussion of selection bias. Second, logical relationships between variables can also produce non-causal correlations (e.g., if GDP yearly is defined
to be the sum of GDP January, GDP F ebrurary, etc.,
GDP yearly will be associated with these variables, but
not caused by them.) For a discussion of logical relations
between variables, see [38]. Third, it does not have any way
of dealing with instantaneous symmetric interactions (likeclassical theories of gravity).

4.2

The Causal Faithfulness Assumption

Consider SEM O in Figure 2. Suppose we have IK (B, R|A),
where SEM K is shown in Figure 1(i), whereas it is not thecase that IO (B, R|A). However, just because O does not entail IO (B, R|A) for all sets of parameter values β, that does
not imply that there are no β for which IO(β) )(B, R|A). For
example, if the variances of R, A, and B are all 1, for any β
for which covO(β) (A, B) · covO(β) (A, R) = covO(β) (B, R),it follows that covO(β) (B, R|A) = 0. This occurs when(bB,R · bA,R + bA,B ) · (bB,R · bA,B + bA,R ) = bR,B . So if
Ip (B, R|A) is true in the population, there are at least two
kinds of explanation: any set of parameter values for SEMs
K (in Figure 1(i)), L, or M (in Figure 2) on the one hand,or any parameterization of SEM O for which (bB,R · bA,R +bA,B ) · (bB,R · bA,B + bA,R ) = bR,B . There are several arguments why, although O with the special parameter values

is a possible explanation, in the absence of evidence to thecontrary, K, L, or M should be the preferred explanations.
First, K, L, and M explain the independence of B and R
conditional on A structurally, as a consequence of no directcausal connection between the variables. In contrast O explains the independence as a consequence of a large direct
effect of B on R cancelled exactly by the product of large
direct and indirect effects of B and R on A.
Second, this cancellation is improbable (in the Bayesian
sense that if a zero conditional covariance is not entailed,the measure of the set of free parameter values for any DAG
that lead to such cancellations is zero for any “smooth” priorprobability density,5 such as the Gaussian or exponential
one, over the free parameters).
Finally, K, L, and M are simpler than O. K, L, and M
have fewer free parameters than O.
The assumption that a causal influence is not hidden bycoincidental cancellations can be expressed for SEMs in thefollowing way. A density p is faithful to the graph G of a
SEM if and only if every conditional independence relation
true in p is entailed by G.
Causal Faithfulness Assumption: For a causally sufficient set of variables V in a population P , the population
density pP (V) is faithful to the causal graph over V for
P [36].
The Causal Faithfulness Assumption requires preferring
K, L, and M to O, because parameter values β for which
IO(β) )(B, R|A) would violate the Causal Faithfulness Assumption. Recently, there have been a number of searchalgorithms that are consistent, but have substituted otherkinds of assumptions in place of the Causal Faithfulness Assumption.

4.3

The Output of a Search for Causal Models

The following sections describe several different possible
alternatives that can be output by a reliable search algorithm.

4.3.1

Markov Equivalence Classes

A trek between A and B is either a directed path from
A to B, a directed path from B to A, or a path between A
and B that does not contain a subpath X → Y ← Z. SEMs
K, L, and M are Markov equivalent, in the sense that theirrespective graphs all entail the same set of conditional independence relations. If K is true, any SEM with a graphthat contains no path between A and R can be eliminatedfrom consideration by the Causal Markov Assumption (e.g.,
N in Figure 2). SEM P also violates the Causal Markov
Assumption. O is incompatible with the population conditional independencies by the Causal Faithfulness Assumption. However, neither of these assumptions implies L or
M is incompatible with the population conditional independencies.
Since K, L, and M entail the same set of conditional independence relations, it is not possible to eliminate L or M
as incompatible with the population conditional independence relations without either adding more assumptions orbackground knowledge, or using features of the probability
density that are not conditional independence relations. In
the case of linear SEMs with Gaussian error terms (and for
multinomial Bayesian networks) there are no other features
5

A smooth measure is absolutely continuous with Lebesgue
measure.

26

Figure 2: Alternative SEM modelsof the density that distinguish K from L or M . However,as we will illustrate later, for other families of distributions,
there are non-conditional independence constraints that canbe entailed by a graph that do distinguish K from L or M .

4.3.2

Distribution Equivalence

K and L are distribution equivalent if and only if for any
assignment of parameter values θ to K there exists an assignment of parameter values θ 0 to L that represents thesame density, and vice versa. If all of the error terms are
Gaussian with linear causal relations, then K and L are distribution equivalent as well as Markov equivalent. In suchcases, the best that a reliable search algorithm can do isto return the entire Markov equivalence class, regardless ofwhat features of the marginal density that it uses.
In contrast, for linear causal models with at most one
error term is non-Gaussian, SEMs K and L are Markov
equivalent, but they are not distribution equivalent.
When Markov equivalence fails to entail distribution equivalence, then using conditional independence relations alone
for causal inference is still correct, but it is not as informative
as theoretically possible. For example, assuming linearity,causal sufficiency, and non-Gaussian errors [32], conditional
independence tests can at best reliably determine the correct
Markov equivalence class, while using other features of thesample density can be used to reliably determine a uniquegraph [32].

4.4

The following is a brief summary of some important differences between the problem of predicting the value of anvariable in an unmanipulated population from a sample, andthe problem of predicting the post-manipulation value of avariable from a sample from an unmanipulated population.
In an unmanipulated population P , the predictor that minimizes the MSPE is the conditional expected value.
1. E(Y |O) (the expected value of Y conditional on O)
is a function of p(O,Y ), regardless of what the truecausal model is.6 In contrast, a manipulated expectedvalue is a function of p(O, Y ) and a causal graph.
2. In order to determine whether EP (Y ||p0 (O)) (the expected value of Y after a manipulation to p0 (O)) is afunction of p(O, Y ) and background knowledge, it isnecessary to find all of the causal models compatible
with p(O, Y ) and background knowledge, not simply
one causal model compatible with p(O, Y ) and background knowledge.
3. Determining which causal models are compatible withbackground knowledge and a p(O, Y ) requires making
additional assumptions connecting population densities to causal models (e.g., Causal Markov and Faithfulness).
4. Without introducing some simplicity assumptions aboutcausal models, for some common families of densities(e.g., Gaussian, multinomial), no EP (Y |O0 ||p0 (O)) are
functions of the population density without very strong
background knowledge.

Constraint-Based Search

The number of DAGs grows super-exponentially with thenumber of vertices, so even for modest numbers of variables it is not possible to examine each DAG to determine
whether it is compatible with the population density given
the Causal Markov and Faithfulness Assumptions. The PC
algorithm, given as input an oracle that returns answers
about conditional independence in the population and optional background knowledge about orientations of edges,
returns a graphical object called a pattern that represents a
Markov equivalence class (or if there is background knowledge a subset of a Markov equivalence class) on the basisof oracle queries. If the oracle always gives correct answers,
and the Causal Markov and Causal Faithfulness Assumptions hold, then the output pattern contains the true SEM,
even thought the algorithm does not check each DAG. In
the worse case, it is exponential in the number of variables,
but for sparse graphs it can run on hundreds of thousandsof variables [34, 35, 23].

5.

DIFFERENCES BETWEEN CLASSIFICATION AND REGRESSION AND CAUSALINFERENCE
27

5. The justification for using simple statistical models is
fundamentally different than the justification for using simple causal models. At a given sample size, theuse of simple statistical model can be justified even if
causal relations are not simple. However, the assumption that the simplest causal model compatible withp(O, Y ) and background knowledge is a substantive
assumption about the simplicity of mechanisms thatexist in the world.
6. For many families of densities (e.g., Gaussian, multinomial), there is always a statistical model without
hidden variables that contains the population density.
For those same families of densities, a causal modelthat contains both the population probability density
and the post-manipulation probability densities may
require the introduction of unobserved variables.
6

This ignores the problem of conditioning on sets of measurezero.

7. Given a population density, and the set of causal models consistent with the population density and background knowledge, calculating the effects of a manipulation can be difficult because:a. There may be unobserved variables (even if onlya single causal model is consistent with p(O, Y )and background knowledge).
b. There may be multiple causal models compatible
with p(O, Y ) and background knowledge.
8. For non-experimental data, a post-manipulation density is different than the population density from which
the sample is drawn. The post-manipulation values of
the target variable Y are not directly measured in thesample. Hence, it is not possible to estimate the error
in EP (Y |O0 ||p0 (O)) by comparing it to the values in asample from the p(O, Y ).

6. SEMS CAN HELP IN CAUSAL DISCOVERY FROM I.I.D. AND TIME SERIES DATAAs discussed in Section 4.4, the constraint-based approachto causal discovery involves conditional independence tests,
which would be a difficult task if the form of dependence isunknown. It has the advantage that it is generally applicable, but the disadvantages are that faithfulness is a strong
assumption and that it may require very large sample sizes
to get good conditional independence tests. Furthermore,
the solution of this approach to causal discovery is usually
non-unique, and in particular, it does not help on determining causal direction in the two-variable case, where no
conditional independence relationship is available.
What information can we use to fully determine the causal
structure? A fundamental issue is given two variables, how
to distinguish cause from effect. To do so, one needs to finda way to capture the asymmetry between them. Intuitively,
one may think that the physical process that generates effect from cause is more natural or simple in some way thanrecovering the cause from effect. How can we represent thisgenerating process, and in which way is the causal process
more natural or simple than the backwards process?
Recently several causal discovery approaches based on
structural equation models (SEMs) have been proposed. A
SEM represents the effect Y as a function of the direct causes
X and some unmeasurable error:
Y = f (X, ε; θ 1 ),(1)

where ε is the error term that is assumed to be independent
from X, the function f ∈ F explains how Y is generated
from X, F is an appropriately constrained functional class,and θ 1 is the parameter set involved in f . We assume thatthe transformation from (X, ε) to (X, Y ) is invertible, suchthat N can be uniquely recovered from the observed variables X and Y .
For convenience of presentation, let us assume that both
X and Y are one-dimensional variables. Without precise
knowledge on the data-generating process, the SEM should
be flexible enough such that it could be adapted to approximate the true data-generating process; more importantly,
the causal direction implied by the SEM has to be identifiable in most cases, i.e., the model assumption, especiallythe independence between the error and cause, holds for only

one direction, such that it implies the causal asymmetry between X and Y . Under the above conditions, one can thenuse SEMs to determine the causal direction between two
variables, given that they have a direct causal relationshipin between and do not have any confounder: for both directions, we fit the SEM, and then test for independence between the estimated error term and the hypothetical cause,and the direction which gives an independent error term isconsidered plausible.
Several forms of the SEM have been shown to be able toproduce unique causal directions, and have received practical applications. In the linear, non-Gaussian, and acyclic
model (LiNGAM [32]), f is linear, and at most one of the error term ε and cause X is Gaussian. The nonlinear additivenoise model [15, 44] assumes that f is nonlinear with additive noise (error) ε. In the post-nonlinear (PNL) causal
model [45], the effect Y is further generated by a postnonlinear transformation on the nonlinear effect of the cause
X plus error term ε:
Y = f2 (f1 (X) + ε),(2)where both f1 and f2 are nonlinear functions and f2 is assumed to be invertible.7 The post-nonlinear transformation
f2 represents the sensor or measurement distortion, which isfrequently encountered in practice. In particular, the PNL
causal model has a very general form (the former two are
its special cases), but it has been shown to be identifiable in
the generic case (except five specific situations given in [45]).
It is worth noting that it is not closed under marginalization, even if there are not confounders. In the subsequent
sections, we will discuss the identifiability of various SEMs,
how to distinguish cause from effect with the SEMs, and therelationships between different principles for causal discovery, including mutual independence of the error terms andthe causal Markov condition, respectively.
Another issue we are concerned with is causal discovery
from time series. According to [13], Granger’s causality
in time series falls into the framework of constraint-based
causal discovery combined with the temporal constraint thatthe effect cannot precede the cause. The SEM, together withthe above temporal constraint, has also been exploited toestimate time-delayed causal relations possibly with instantaneous effects [44]. Compared to the conditional independence relationships, the SEM, if correctly specified, is able
to recover more about the causal information. In this paper, when talking about causality in time series, we assume
that the causal relations are linear with non-Gaussian errors. In Section 10, after reviewing linear Granger causality
with instantaneous effects, we focus on two problems which
are traditionally difficult to solve. In particular, we present
the theoretical results which make it possible to discover
the temporal causal relations at the true causal frequency
from subsampled data [12], that is, one can recover monthly
causal relations from quarterly data or estimate rapid causal
influences between stocks from their daily returns. Moreover, even when there exist confounder time series, theoretical results suggested that one can still identify the causal
7

In [42] both functions f1 and f2 are assumed to to invertible; this causal model, as a consequence, can be estimatedby making use of post-nonlinear independent component
analysis (PNL-ICA) [39], which assumes that the observed
data are component-wise invertible transformations of linear
mixtures of the independence sources to be recovered.

28

relations among the observed time series as well as the influences from the confounder series [11].

7.

SEVERAL SEMS AND THE IDENTIFIABILITY OF CAUSAL DIRECTION

When talking about the causal relation between two variables, traditionally people were often concerned with thelinear-Gaussian case, where the involved variables are Gaussian with a linear causal relation, or the discrete case. It
turned out that the former case is one of the atypical situations where the causal asymmetry does not leave a footprintin the observed data or their joint distribution: the joint
Gaussian distribution is fully determined by the mean andcovariance, and with proper rescaling, the two variables arecompletely asymmetric w.r.t. the data distribution.
In the discrete case, if one knows precisely what SEM
class generated the effect from cause, which, for instance,
may be the noisy AND or noisy XOR gate, then under mild
conditions the causal direction can be easily seen from thedata distribution. However, generally speaking, if the precise functional class of the causal process is unknown, inthe discrete case it is difficult to recover the causal direction
from observed data, especially when the cardinality of thevariables is small. As an illustration, let us consider the situation where the causal process first generates continuous
data and discretizes such data to produce the observed discrete ones. It is then not surprising that certain propertiesof the causal process are lost due to discretization, making
causal discovery more difficult. In this paper we will focus
on the continuous case.

7.1

Causal Direction Is Not Identifiable without Constraints on SEMs

In the SEM (1), the error term is assumed to be independent from the cause. If for the reverse direction, one cannot
find a function to represent X in terms of the hypotheticalcause Y and an error term which is independent from Y ,then we can determine the true causal direction, or distinguish cause from effect. Unfortunately, this is not the case
if we do not impose any constraint on the function f , as
explained below.
According to [17], given any two random variables X andY with continuous support, one can always construct another variable, denoted by ε̃, which is statically independent
from X. In [47] the class of functions to produce such an
independent variable ε̃ (or called independent error term inour causal discovery context) was given, and it was shownthat this procedure is invertible: Y is a function of X andε̃.
This is also the case for the hypothetical causal direction
Y → X: we can also always represent X as a function of Y
and an independent error term. That is, any two variables
would be symmetric according to the SEM, if f is not constrained. Therefore, in order for the SEMs to be useful todetermine the causal direction, we have to introduce certain
constraints on the function f such that the independence
condition on the error and the hypothetical cause holds for
only one direction. Below we focus on the two-variable case,and the results can be readily extended to the case with anarbitrary number of variables, as shown in [28].

7.2

Linear Non-Gaussian Causal Model
29

The linear causal model in the two-variable case can be
written as
Y = bX + ε.

(3)

It is nice that if at most one of X and ε is Gaussian, thecausal direction is identifiable, due to the ICA theory [16],or more fundamentally, due to the Darmois-Skitovich theorem [20]. This is known as the linear, non-Gaussian, acyclic
model (LiNGAM [32]).

7.2.1 On the Ubiquitousness of Non-Gaussianity inthe Linear Case
According to the central limit theorem, under mild conditions, the sum of independent variables tends to be Gaussian as the number of components becomes larger and larger.
One may then challenge the non-Gaussianity assumption in
the LiNGAM model. Here we argue that in the linear case,non-Gaussian distributions are ubiquitous.
Cramér’s decomposition theorem states that if the sumof two independent real-valued random variables is Gaussian, then both of the summand variables much be Gaussian as well; see [6, page 53]. By induction, this meansthat if the sum of any finite independent real-valued variables is Gaussian, then all summands must be Gaussian. In
other words, a Gaussian distribution can never be exactly
produced by linear composition of variables any of which
is non-Gaussian. This nicely complements the central limit
theorem: (under proper conditions) the sum of independent
variable get closer to Gaussian, but it cannot be exactly
Gaussian, except all summand variables are Gaussian. This
linear closure property of the Gaussian distribution implies
the rareness of the Gaussian distribution and ubiquitousness of non-Gaussian distributions, if we believe the relations between variables are linear. However, the closer itgets to Gaussian, the harder it is to distinguish the direction. Hence, the practical question is, are the errors typically
non-Gaussian enough to distinguish causal directions in thelinear case?

7.3

Nonlinear Additive Noise Model

In practice nonlinear transformation is often involved in
the data generating process, and should be taken into account in the functional class. As a direct extension of LiNGAM,
the nonlinear additive noise model represents the effect as anonlinear function of the cause plus independent error [15]:Y = fAN (X) + ε.

(4)

It has been shown that the set of all p(X) for which the
backward model also admits an independent error term iscontained in a 3-dimensional affine space. Bearing in mind
that the space of all possible p(X) is infinite dimensional, one
can see that roughly speaking, in the generic case, if the data
were generated by the nonlinear additive noise model, thecausal direction is identifiable. This model is a special caseof the PNL causal model, which is to be discussed below,
and the identifiability results for the PNL causal model also
apply here.

7.4

Post-Nonlinear Causal Model

If the assumed SEM is too restrictive to be able to approximate the true data generating process, the causal discovery results may be misleading. Therefore, if the specificknowledge about the data generating mechanism is not available, to make it useful in practice, the assumed causal modelshould be general enough, such that it can reveal the data
generating processes approximately.
The PNL causal model takes into account the nonlinear influence from the cause, the noise effect, and the possible sensor or measurement distortion in the observed variables [45].
See (2) for its form; a slightly more restricted version of themodel, in which the inner function, f1 , is also assumed to be
invertible, was proposed in [42] and applied to causal analysis of stock returns. It has the most general form among all
well-defined SEMs according to which the causal direction
is identifiable in the general case. (The model used in [24]does not impose structural constraints but assumes a certain
type of smoothness; however, it does not lead to theoretical
identifiability results.) Clearly it contains the linear modeland nonlinear additive noise model as special cases. The
multiplicative noise model, Y = X · ε, where all involved
variables are positive, is another special case, since it can be
written as Y = exp(log X + log ε), where log ε is considered
as a new noise term, f1 (X) = log(X), and f2 (·) = exp(·).
The identifiability conditions of the causal direction according to the PNL causal model was established by a proof
by contradiction [45]. We assume the causal model holds
in both directions X → Y and Y → X, and show that this
implies some very strong conditions on the distributions andfunctions involved in the model. Suppose the data were generated according to the PNL causal model in settings otherthan those specific conditions; then in principle, the backward direction does not follow the model, and the causal
direction can be determined.
Assume that the data (X, Y ) are generated by the PNL
causal model with the the causal relation X → Y . This data
generating process can be described as (2). Moreover, let us
assume that the backwards direction, Y → X also follows
the PNL causal model with independent error. That is,
X = g2 (g1 (Y ) + εY ),(5)

where Y and εY are independent, g1 is non-constant, and g2
is invertible.
Equations (2) and (5) define the transformation from (X, ε)|to (Y, εY )| ; as a consequence, p(Y, εY ) can be expressed interms of p(X, ε) = p(X)p(ε). The identifiability results wereobtained based on the linear separability of the logarithm of
the joint density of independent variables, i.e., for a set ofindependent random variables whose joint density is twice
differentiable, the Hessian of the logarithm of their density
is diagonal everywhere [22]. Since Y and εY are assumed to
be independent, log p(Y, εY ) then follows such a linear separability property. This implies that the second-order partial
derivative of log p(Y, εY ) w.r.t. Y and εY is zero. It thenreduces to a differential equation of a bilinear form. Under certain conditions (e.g., p(ε) is positive on (−∞, +∞)),
the solution to the differential equation gives all cases inwhich the causal direction is not identifiable according to
the PNL causal model. Table 1 in [45] summarizes all fivenon-identifiable cases. The first one is the linear-Gaussiancase, in which the causal direction is well-known to be nonidentifiable. Roughly speaking, to make one of those cases
true, one has to adjust the data distribution and the involved nonlinear functions very carefully. In other words,
in the generic case the causal direction is identifiable if thedata were generated according to the PNL causal model.

8.

DETERMINATION OF CAUSAL DIRECTION BASED ON SEMS

A commonly used approach to distinguishing cause from
effect with nonlinear SEMs consists of two steps. First, onefits the model (e.g., the nonlinear additive noise model or the
PNL causal model) on the data for both hypothetical causal
directions. The second step is to do independence test between the estimated error term and hypothetical cause [15,45]. If the independence condition holds for one and only
one hypothetical direction, the causal relation between thetwo variables X and Y implied by the corresponding SEM
has been successfully found. If neither of them holds, thedata-generating process may not follow the assumed SEM,
or there exists some confounder influencing both X and Y .
If both hold, the cause and effect can not be distinguished by
the exploited SEM; in this case, additional information, suchas the smoothness of the involved nonlinearities, may help
find the causal model with a lower complexity. We adopted
the Hilbert Schmidt information criterion (HSIC) [14] forstatistical independence test in the first step. Below we discuss how to estimate the function as well as the error termin the first step.
For the nonlinear additive noise model, the function fAN
is usually estimated by performing Gaussian process (GP)regression [15]. For details on GP regression, one may refer
to [29].
Estimation of the PNL causal model (2) has several indeterminacies: the sign, mean, and scale of the error term
varepsilon, and accordingly, the sign, location, and scale offi1 are arbitrary. In the estimation procedure, one may impose certain constraints to avoid such indeterminacies in theestimate. However, we should note that in principle, we do
not care about those indeterminacies in the causal discovery
context, since they do not change the statistical independence or dependence property between the estimated error
term and the hypothetical cause.
It is well known that for linear regression, the maximum
likelihood estimator of the coefficient is still statistically consistent even if the error distribution is wrongly assumed to
the Gaussian. However, this may not be the case for general nonlinear models. As shown in [47, Section 3.2], if theerror distribution mis-specified, the estimated PNL causal
model (2) may not be statistically consistent, even when theabove indeterminacies in the estimate are properly tackled.
Therefore, the error distribution should be adaptively estimated from data, if the true one is not known a priori. It
has been proposed to estimate the PNL causal model (2)by mutual information minimization [45] with the involved
nonlinear functions represented by multi-layer perceptrons(MLPs). Later, in [47] the PNL causal model was estimatedby extending the framework of warped Gaussian processes
to allow a flexible error distribution, which is represented bya mixture of Gaussians (MoG).

9.

ON THE RELATIONSHIPS BETWEEN DIFFERENT PRINCIPLES FOR MODEL ESTIMATION

One usually use maximum likelihood to fit the SEM together with a directed acyclic graph (DAG) to the given
data. Not surprisingly, the negative likelihood (with the distribution of the error term adaptively estimated from data)30

is equivalent to the mutual information between the estimated error terms, as stated in Theorem 3 in [47]. The
higher the likelihood, the less dependent the estimated error terms. (Note that the root variables in the DAG are alsocounted as error terms.)
On the other hand, the constraint-based approach to causal
discovery exploits conditional independence relationships ofthe variables to derive (the equivalence class of) the causal
structure [36, 27]. How are these principles, including mutual independence of the estimated error terms and the causal
Markov condition, related to each other? Below we will answer this question, and the results in this section hold foran arbitrary number of variables.
Let us consider optimization over different DAG structures to find the causal structure. Assume the we optimally
fit the nonlinear functions fi according to the given candidate DAG structure. First consider the situation where we
fit the nonlinear additive noise model, i.e.,
Xi = fAN,i (PAi ) + εi ,(6)

to the data. It has been shown that mutual independence ofthe error terms and conditional independence between observed variables (together with the independence between
εi and PAi ) are equivalent. Furthermore, they are achieved
if and only if the total entropy of the disturbances is minimized [44]. More specifically, when fitting the model (6) witha hypothetical DAG causal structure to the given variables
X1 , · · · , Xn , the following three properties are equivalent:(i) The causal Markov condition holds (i.e., each variable
is independent of its non-descendants in the DAG conditioning on its parents), and in addition, the error
term in Xi is independent from the parents of Xi .
(ii) The error terms Ni are mutually independent.
P(iii) The total entropy of the error terms, i.e., i H(εi ), isminimized, with the minimum H(X1 , · · · , Xn ).
Let us then consider the PNL causal model. When one
fits the PNL causal model
Xi = fi2 (fi1 (PAi ) + εi ),(7)

to the data, the scale of the error terms as well as fi1 is arbitrary, since fi2 is also to be estimated. Consequently, unlike
for the nonlinear additive noise model, in the PNL causal
model context it is not meaningful to talk about the total entropy of the error terms (see condition (iii) above). However,as shown in [45], when fitting the PNL causal model with ahypothetical DAG causal structure to the data, we still havethe equivalence between conditions (i) and (ii) above.
Given more than two variables, one way to estimate thecausal model based on SEMs is to use exhaustive search:
for all possible causal orderings, fit SEMs for all hypothetical effects separately, and then do model checking by testing for independence between the estimated error and thecorresponding hypothetical causes. However, note that thecomplexity of this procedure increases super-exponentiallyalong with the number of variables. Smart approaches arethen needed.
The above result concerning the relationship between mutual independence of the error terms and the causal Markov
condition combined with the independence between eacherror term and its associated parents suggests a two-step31

method to find the causal structure implied by the PNL
causal model. One first uses the constraint-based approach
to find the Markov equivalent class from conditional independence relationships with proper nonparametric conditional independence tests (e.g., [46]). The PNL causal modelis then used to identify the causal directions that cannot be
determined in the first step: for each DAG contained in
the equivalent class, we estimate the error terms, and determine whether this causal structure is plausible by examining
whether the disturbance in each variable Xi is independent
from the parents of Xi . Consequently, one avoids the exhaustive search over all possible causal structures and highdimensional statistical tests of mutual independence of allerror terms.

10.

CAUSAL DISCOVERY FROM TIME SERIESBoth the constraint-based and SEM-based approaches tocausal discovery are directly applicable to find causal relations from time series; moreover, one can benefit from thetemporal constraint that the effect cannot precede the cause,
which helps reduce the search space of the causal structure.
Below we assume linearity of the causal relations and consider three problems, namely, linear Granger causal analysis
with instantaneous effects, causal discovery from systematically subsampled data, and that in the presence of hidden
time series.

10.1

Linear Granger Causality and its Extension with Instantaneous Effects

For Granger causal analysis in the linear case [13], one fits
the following VAR model [33] to the data:
Xt = AXt−1 + εt ,(8)where Xt = (X1t , X2t , ..., Xnt )| is the vector of the observed
data, εt = (ε1t , ..., εnt )| is the temporally and contemporaneously independent noise process, and causal transition
matrix A contains the temporal causal relations.
In practice it is found that after fitting the VAR model,
the residuals are often contemporaneously dependent. To
account for such dependence, the above VAR model has
been extended to allow instantaneous causal effects between
Xit [18]. Let B0 contains the instantaneous causal relations
between Xt . Equation (8) changes to
Xt = B0 Xt + AXt−1 + εt ,⇒(I − B0 )Xt = AXt−1 + εt ,⇒Xt = (I − B0 )−1 AXt−1 + (I − B0 )−1 εt .

(9)

To estimate all involved parameters in Granger causality
with instantaneous effects, two estimation procedures havebeen proposed in [18]. The two-step method first estimate
the errors in the above VAR model and then apply independent component analysis (ICA) [16] on the estimatederrors. The other is based on multichannel blind deconvolution, which is statistically more efficient [44].

10.2

Causal Discovery from Subsampled Data

Suppose the original high-resolution data were generatedby (8). We consider low-resolution data generated by subsampling (or systematic sampling) with the subsampling factor k. Here we are interested in finding the causal transitionmatrix A which generated the data from the subsampled
data. Traditionally, if one uses only the second-order information, this suffers from parameter identification issues [25],i.e., the same subsampled (low-frequency) model may disaggregate to several high frequency models, which are observationally equivalent at the low frequency.

assume that the (multivariate) measurements are a sample
of a multivariate random process Xt , which, together withanother random process Zt , forms a VAR process. That is,
  
 

Xt
B C
Xt−1
=
·
+ εt ,
(11)
Zt
D E
Zt−1

where Zt is not measured and can be considered as confounder time series, B is the causal transition matrix for
Suppose that due to low resolution of the data, there is an
the observed process Xt , and C contains the the influenceobservation every k time steps. That is, the low-resolution
from Zt to the observed process Xt . The theoretical issue isobservations X̃ = (X̃1 , X̃2 , , ..., X̃t ) are (X1 , X1+k , ..., X1+(t−1)k ); whether B and C are identifiable from solely the observed
here we have assumed that the first sampled point is Xx1 .
process Xt .
We then have

10.2.1

Effect of Subsampling (Systematic Sampling)

10.3.1 Practical Granger Causal Analysis Can Go
WrongX̃t+1 = X1+tk = AX1+tk−1 + ε1+tk= A(AX1+tk−2 + ε1+tk−1 ) + ε1+tk= ...
= Ak X̃t +

k−1
X
l=0

|

Al ε1+tk−l .
{z

,~
εt

(10)

}

According to (10), subsampled data X̃t also follows a vector autoregression (VAR) model with the error term ~εt , andone can see that as T → ∞, the discovered temporal causal
relations from such subsampled data are given by Ak . As
k → ∞, Ak tends to vanish, and the subsampled data will becontemporaneously dependent. (We have assumed that thesystem is stable, in that all eigenvalues of A have modulus
smaller than one.)

Misleading Granger causal relations in low-resolution
data: An illustration.


0.8 0.5. Consider the case where k =0 −0.8
2. The corresponding VAR model for the subsampled data
is


0.64
0
2
~
X̃t−1 + ~εt .
X̃t = A X̃t−1 + εt =00.64
Suppose A =That is, the causal influence from X2,t−1 to X1t is missingin the corresponding low-resolution data (with k = 2).

10.2.2

Identifiability of the Causal Relations at the
Causal Frequency

It has been shown that if the distributions pNi are nonGaussian and different for different i, together with othertechnical assumptions, the transition matrix associated withthe causal-frequency data, A, is identifiable from the subsampled data X̃. As a by-product, the result also indicatesthat the subsampled data, although contemporaneously dependent, actually do not follow the model of linear Granger
causality with instantaneous effects [12].
Let the distributions of the noise terms be represented
by the MoG. An EM algorithm and a variational EM (withmean field approximation) were then proposed to estimate
A from subsampled data.

10.3

Causal Discovery with Hidden Time Series (Confounders)

In practice it is usually difficult and even impossible tocollect all relevant time series when doing causal analysis
on given ones. We approach this problem as follows. We

In practical Granger causal analysis, one just performs
a linear regression of present on past on the observed Xt
and then interpret the regression matrix causally. While
making the ideal definition practically feasible, this may leadto wrong causal conclusions in the sense that it does notcomply with the causal structure that we would infer given
we had more information. Let us give an example for this.
Let Xt be bivariate and Zt be univariate. Moreover, assume




0.5
0.9 0
B C
0.8  ,
=  0.1 0.1
D E
0
0
0.9
and let the covariance matrix of εt be the identity matrix.
To perform practical Granger causal analysis, we proceed as
usual: we fit a VAR model on only the observable process
Xt , in particular calculate the VAR transition matrix by
0.89 0.35|−1|
BpG = E(Xt Xt−1 )E (Xt Xt ) =.
0.08 0.65(up to rounding) and interpret the coefficients of BpG as
causal influences. Although, according to B, the true timedelayed causal relations in Xt , X2t does not cause X1t , BpG
suggests that there is a strong causal effect X2,t−1 → X1t
with the strength 0.35. It is even stronger than the relation
X1,t−1 → X2t , which actually exists in the complete modelwith the strength 0.1.

10.3.2

Identifiability of B and Almost Identifiabilityof C

Assume that all components of εt are non-Gaussian andthat the dimensionality of the hidden process Zt is not higher
than that of the observed process Xt . Together with somefurther technical assumptions, it has been shown that B is
identifiable from Xt ; furthermore, the set of columns of C
with at least two non-zero entries is identifiable from up toscaling of those columns [11].
One can then use a MoG to represent the distributions of
the components of εt and develop a variation EM algorithm
to estimate B and C from solely Xt .

11.

CONCLUSION AND OPEN PROBLEMS

We have reviewed central concepts in and fundamental
methodologies for causal inference and discovery. The concepts include manipulations, causal models, sample predictive modeling, causal predictive modeling, structural equation models, the causal Markov assumption, and the faithfulness assumption. We have discussed the constraint-based

32

causal structure search and its properties. In the second partof the paper, we have given a survey of structural equation
models which enable us to fully identify causal structurefrom observational data. We focused on the two-variablecase, where the task is to distinguish cause from effect. We
have reviewed the linear non-Gaussian causal model, nonlinear additive noise model, and the post-nonlinear causal
model, listed from the most to the least restrictive. We addressed the identifiablility of the causal direction: for those
three models, in the generic case the backwards direction
does not admit an independent error term and, as a consequence, it is possible to distinguish cause from effect. We
have also briefly discussed the procedure to do so, which
consists of fitting the structural equation model and doing
independence test between the estimated error term and the
hypothetical cause.
In the last three decades, enlightening progress has been
made in the field of causal discovery and inference. However,there are still many fundamental questions to be answered:8• What new models are appropriate for different combinations of kinds of data, e.g., experimental and observational [4, 7, 40, 8, 41, 9]?
• What new models are appropriate for different kinds ofbackground knowledge, and different families of densities?
• What kind of scores can be used to best evaluate causal
models from various kinds of data? In a related vein,what are good families of prior distributions that capture various kinds of background knowledge?
• How can search algorithms be improved to incorporate
different kinds of background knowledge, search overdifferent classes of causal models, run faster, handle
more variables and larger sample sizes, be more reliable
at small sample sizes, and produce output that is as
informative as possible?
• For existing and novel causal search algorithms, what
are their semantic and syntactic properties (e.g., soundness, consistency, maximum informativeness)? What
are their statistical properties (pointwise consistency,
uniform consistency, sample efficiency)? What are their
computational properties (computational complexity)?
• What plausible alternatives are there to the Causal
Markov and Faithfulness Assumptions? Are there otherassumptions might be weaker and hold in more domains and applications without much loss about what
can be reliably inferred? Are there stronger assumptions that are plausible for some domains that mightallow for stronger causal inferences? How often arethese assumption violated, and how much do violationsof these assumptions lead to incorrect inferences?
• There are special assumptions, such as linearity, which
can improve the strength of causal conclusions thatcan be reliably inferred, and the speed and sample efficiency of algorithms that draw the conclusions. Whatother distribution families or stronger assumptions about
8
The content and organization of the following open questions are largely due to suggestions from Constantin Aliferis,whom I thank for his suggestions.

33

a domain are there that are plausible for some domains
and how can they be used to improve causal inference?
• Can various statistical assumptions be relaxed? For
example, what if the sample selection process is not
i.i.d., but may be causally affected by variables of interest [2, 37, 5, 3, 30]?
In addition, there is also a number of open problems concerning SEM-based causal discovery and the asymmetry between cause and effect.
• First, one can consider structural equation models as away to represent the conditional distribution of the effect given the cause. Can we then find hints as to thecausal direction directly from the data distribution?
In other words, can we find a general way to directly
characterize the causal asymmetry in light of certain
properties of the data distribution? If we managed to
do so, it would hopefully put the causal Markov condition, the independent noise condition (in the SEMs),
and the independent transformation condition in the
nonlinear noiseless case [19] under the same umbrella.
To this end, an attempt has been made by exploiting
the so-called “exogeneity” property of a causally sufficient causal system [48]. But it is not clear whether
this property is able to bring about computationallyefficient and widely applicable causal discovery methods.
• Secondly, note that nonlinear structural equation models are usually intransitive. That is, if both causal
processes X1 → X2 and X2 → X3 admit a particular type of structural equation model, say, the nonlinear additive noise model, the process X1 → X3 does
not necessarily follow the same model. (Linear models
are transitive.) This could be a potential issue withstructural-equation-model-based causal discovery: it
may fail to discover indirect causal relations. (Hereby direction causal relations, we mean the causal relations in which only a single noise variable is involved.)
On the other hand, this may be a benefit of using structural equation models for causal discovery, in that it ispossible to detect the existence of causal intermediatevariables and further recover them. But how to do so
is currently unclear.
• We have discussed how different types of independence,
including conditional independence in the causal Markov
condition and statistical independence between the error term and hypothetical cause in structural equations models, help to discovery causal information from
data. On the other hand, it has been demonstrated
that this type of independence (which is, loosely speaking, the independence between how the cause is generated and how the effect is generated from cause)is able to facilitate understanding and solving somemachine learning or data analysis problems. For instance, it implies that when the feature causes the label (or target), unlabeled data points will not help inthe semi-supervised learning scenario [31], and inspired
new settings and formulations for domain adaptationby characterizing what information to transfer [43]. It
is under investigation whether other machine learningmethods including “adaptive boosting” can be understood from the causal perspective. In addition, it isunclear whether the learning guarantees for supervised
learning actually depend on the causal relationship between the feature and target (or label), i.e., the causal
role of the feature w.r.t. the target.
• Finally, developing efficient methods for causal discovery of more than two variables based on structural
equation models is an important step towards largescale causal analysis in various domains including neuroscience and biology. To make causal discovery computationally efficient, one may have to limit the complexity of the causal structure, say, limit the number of
direct causes of each variable. Even so, a smart optimization procedure instead of exhaustive search is still
missing in the literature.

12.

REFERENCES[1] P. J. Bickel and K. A. Doksum. Mathematical
Statistics: Basic Ideas and Selected Topics, Vol I.
Prentice Hall, 2000. 2nd Edition.
[2] G. F. Cooper. Causal discovery from data in thepresence of selection bias. In Fifth International
Workshop on AI and Statistics, pages 140–150, 1995.
[3] G. F. Cooper. A Bayesian method for causal modelingand discovery under selection. In Uncertainty In
Artificial Intelligence, pages 98–106, 2000.
[4] G. F. Cooper and C. Yoo. Causal discovery from a
mixture of experimental and observational data. In
Uncertainty In Artificial Intelligence, pages 116–125,
1999.
[5] D. R. Cox and N. Wermuth. Multivariate
Dependencies: Models, Analysis and Interpretation(Monographs on Statistics and Applied Probability).
Chapman & Hall/CRC, 1996.
[6] H. Cramér. Random variables and probability
distributions. Cambridge University Press, Cambridge,
UK, 3rd edition, 1970.
[7] D. Danks. Learning the causal structure ofoverlapping variable sets. In Lect Notes Comput Sc,
2534, pages 178–191, 2002.
[8] F. Eberhardt, C. Glymour, and R. Scheines. On thenumber of experiments sufficient and in the worst casenecessary to identify all causal relations among nvariables. In 21st Conference on Uncertainty in
Artificial Intelligence, pages 178–184, 2005.
[9] F. Eberhardt, C. Glymour, and R. Scheines. 4 n-1
experiments suffice to determine the causal relations
among n variables. In D. E. Holmes and C. J. Lakhmi,
editors, Innovations in Machine Learning: Theory
And Applications, pages 97–112, 2006.
[10] F. Fisher. Ba correspondence principle forsimultaneous equation models. Econometrica,
38:73–92, 1970.
[11] P. Geiger, K. Zhang, M. Gong, D. Janzing, and
B. Schölkopf. Causal inference by identification ofvector autoregressive processes with hidden
components. In Proc. 32th International Conference
on Machine Learning (ICML 2015), 2015.
[12] M. Gong*, K. Zhang*, D. Tao, P. Geiger, and
B. Schölkopf. Discovering temporal causal relations

from subsampled data. In Proc. 32th International
Conference on Machine Learning (ICML 2015), 2015.
[13] C. Granger. Testing for causality: A personal
viewpoint. Journal of Economic Dynamics andControl, 2, 1980.
[14] A. Gretton, O. Bousquet, A. J. Smola, and
B. Schölkopf. Measuring statistical dependence with
Hilbert-Schmidt norms. In S. Jain, H.-U. Simon, andE. Tomita, editors, Algorithmic Learning Theory: 16th
International Conference, pages 63–78, Berlin,
Germany, 2005. Springer.
[15] P. Hoyer, D. Janzing, J. Mooji, J. Peters, and
B. Schölkopf. Nonlinear causal discovery with additivenoise models. In Advances in Neural Information
Processing Systems 21, Vancouver, B.C., Canada,
2009.
[16] A. Hyvärinen, J. Karhunen, and E. Oja. Independent
Component Analysis. John Wiley & Sons, Inc, 2001.
[17] A. Hyvärinen and P. Pajunen. Nonlinear independent
component analysis: Existence and uniqueness results.
Neural Networks, 12(3):429–439, 1999.
[18] A. Hyvärinen, K. Zhang, S. Shimizu, and P. Hoyer.
Estimation of a structural vector autoregression modelusing non-gaussianity. Journal of Machine Learning
Research, pages 1709–1731, 2010.
[19] D. Janzing, J. Mooij, K. Zhang, J. Lemeire,
J. Zscheischler, P. Daniuvsis, B. Steudel, and
B. Schölkopf. Information-geometric approach toinferring causal directions. Artificial Intelligence,
pages 1–31, 2012.
[20] A. M. Kagan, Y. V. Linnik, and C. R. Rao.
Characterization Problems in Mathematical Statistics.
Wiley, New York, 1973.
[21] S. L. Lauritzen, A. P. Dawid, B. N. Larsen, and H. G.
Leimer. Independence properties of directed markov
fields. Networks, 20:491–505, 1990.
[22] J. Lin. Factorizing multivariate function classes. In
Advances in Neural Information Processing Systems
10, pages 563–569, Cambridge, MA, 1998. MIT Press.
[23] C. Meek. Strong completeness and faithfulness inbayesian networks. In Proceedings of the Eleventh
Conference Annual Conference on Uncertainty in
Artificial Intelligence (UAI-95), pages 411–419, 1995.
[24] J. Mooij, O. Stegle, D. Janzing, K. Zhang, and
B. Schölkopf. Probabilistic latent variable models for
distinguishing between cause and effect. In Advancesin Neural Information Processing Systems 23 (NIPS
2010), Curran, NY, USA, 2010.
[25] F. C. Palm and T. E. Nijman. Missing observations in
the dynamic regression model. Econometrica,
52:1415–1435, 1984.
[26] J. Pearl. Probabilistic Reasoning in Intelligent
Systems: Networks of Plausible Inference. Morgan
Kaufmann, 1988.
[27] J. Pearl. Causality: Models, Reasoning, and Inference.
Cambridge University Press, Cambridge, 2000.
[28] J. Peters, J. Mooij, D. Janzing, and B. Schölkopf.
Identifiability of causal graphs using functionalmodels. In Proc. UAI 2011, pages 589–598, 2011.
[29] C. Rasmussen and C. Williams. Gaussian Processes
for Machine Learning. MIT Press, Cambridge,

34

Massachusetts, USA, 2006.
[30] T. Richardson and P. Spirtes. Ancestral graph markov
models. Ann Stat, 30:962–1030, 2002.
[31] B. Schölkopf, D. Janzing, J. Peters, E. Sgouritsa,
K. Zhang, and J. Mooij. On causal and anticausal
learning. In Proc. 29th International Conference on
Machine Learning (ICML 2012), Edinburgh, Scotland,
2012.
[32] S. Shimizu, P. Hoyer, A. Hyvärinen, and A. Kerminen.
A linear non-Gaussian acyclic model for causal
discovery. Journal of Machine Learning Research,
7:2003–2030, 2006.
[33] C. A. Sims. Macroeconomics and reality.
Econometrica, 48:1–48, 1980.
[34] P. Spirtes and C. Glymour. An algorithm for fast
recovery of sparse causal graphs. Social Science
Computer Review, 9:62–72, 1991.
[35] P. Spirtes, C. Glymour, and R. Scheines. Causation,
Prediction, and Search. Spring-Verlag Lectures in
Statistics, 1993.
[36] P. Spirtes, C. Glymour, and R. Scheines. Causation,
Prediction, and Search. MIT Press, Cambridge, MA,
2nd edition, 2001.
[37] P. Spirtes, C. Meek, and T. S. Richardson. Causal
inference in the presence of latent variables andselection bias. In Eleventh Conference on Uncertainty
in Artificial Intelligence, pages 499–505, 1995.
[38] P. Spirtes and R. Scheines. Causal inference of
ambiguous manipulations. Philos Sci,, 71:833–845,
2004.
[39] A. Taleb and C. Jutten. Source separation in
post-nonlinear mixtures. IEEE Trans. on Signal
Processing, 47(10):2807–2820, 1999.
[40] C. Yoo and G. F. Cooper. An evaluation of a systemthat recommends microarray experiments to perform
to discover gene-regulation pathways. Artif Intell Med,
31:169–182, 2004.
[41] C. Yoo, G. F. Cooper, and M. Schmidt. A control
study to evaluate a computer-based microarrayexperiment design recommendation system forgene-regulation pathways discovery. J Biomed Inform,
39:126–146, 2006.
[42] K. Zhang and L. Chan. Extensions of ICA for
causality discovery in the hong kong stock market. In
Proc. 13th International Conference on Neural
Information Processing (ICONIP 2006), 2006.
[43] K. Zhang, M. Gong, and B. Schölkopf. Multi-source
domain adaptation: A causal view. In Proceedings ofthe 29th AAAI Conference on Artificial Intelligence,
pages 3150–3157. AAAI Press, 2015.
[44] K. Zhang and A. Hyvärinen. Causality discovery withadditive disturbances: An information-theoreticalperspective. In Proc. European Conference on
Machine Learning and Principles and Practice ofKnowledge Discovery in Databases (ECML PKDD)
2009, Bled, Slovenia, 2009.
[45] K. Zhang and A. Hyvärinen. On the identifiability ofthe post-nonlinear causal model. In Proceedings of the25th Conference on Uncertainty in Artificial
Intelligence, Montreal, Canada, 2009.
[46] K. Zhang, D. Janzing, and B. Schölkopf. Kernel-based

35

conditional independence test and application in
causal discovery. In Proceedings of the 27th Conference
on Uncertainty in Artificial Intelligence (UAI 2011),
Barcelona, Spain, 2011.
[47] K. Zhang, Z. Wang, J. Zhang, and B. Schölkopf. On
estimation of functional causal models: General
results and application to post-nonlinear causal model.
ACM Transactions on Intelligent Systems andTechnologies, 2015. forthcoming.
[48] K. Zhang, J. Zhang, and B. Schölkopf. Distinguishingcause from effect based on exogeneity. In Proc. 15th
Conference on Theoretical Aspects of Rationality andKnowledge (TARK 2015), 2015.

Resolving Distributed Knowledge
Extended Abstract
Thomas Ågotnes

Yì N. Wáng

University of Bergen, Norway

Zhejiang University, China

thomas.agotnes@uib.noynw@zju.edu.cn

ABSTRACTDistributed knowledge is the sum of the knowledge in a group; whatsomeone who is able to discern between two possible worlds whenever any member of the group can discern between them, would
know. Sometimes distributed knowledge is referred to as the potential knowledge of a group, or the joint knowledge they could obtainif they had unlimited means of communication. In epistemic logic,
the formula DG ϕ is intended to express the fact that group G has
distributed knowledge of ϕ, that there is enough information in thegroup to infer ϕ. But this is not the same as reasoning about what
happens if the members of the group share their information. In
this paper we introduce an operator RG , such that RG ϕ means thatϕ is true after G have shared all their information with each other– after G’s distributed knowledge has been resolved. The RG operators are called resolution operators. Semantically, we say thatan expression RG ϕ is true iff ϕ is true in what van Benthem [8,p. 249] calls (G’s) communication core; the model update obtainedby removing links to states for members of G that are not linked
by all members of G. We study logics with different combinationsof resolution operators and operators for common and distributed
knowledge. Of particular interest is the relationship between distributed and common knowledge. The main results are sound andcomplete axiomatizations.

1. INTRODUCTION
In epistemic logic [3, 5, 12] different notions of group knowledge
describe different ways in which knowledge can be associated witha group. Common knowledge is stronger than individual knowledge: that something is common knowledge requires not only thateverybody in the group knows it, but that everybody knows that everybody knows it, and so on. Distributed knowledge, on the other
hand, is weaker than individual knowledge: distributed knowledge
is knowledge that is distributed throughout the group even if noindividual knows it.
More concrete informal descriptions of the concept of distributed
knowledge abound, but they are often inaccurate descriptions of theconcept as formalized in standard epistemic logic. A misconception is that something is distributed knowledge in a group if theagents in the group could get to know it after some (perhaps unlimited) communications between them1 . To see that this interpre1
Some examples of informal descriptions of distributed knowledge
from the literature include “A group has distributed knowledge of afact ϕ if the knowledge of ϕ is distributed among its members, sothat by pooling their knowledge together the members of the group
can deduce ϕ” [3]; “.. it should be possible for the members of
the group to establish ϕ through communication” [11, 7]; “.. theknowledge that would result of the agents could somehow ’combine’ their knowledge” [11]. These descriptions can at least give a37

tation must be incorrect, consider the formula D{1,2} (p ∧ ¬K1 p).
In this formula, DG ϕ and Ki ϕ mean that ϕ is distributed knowledge in the group G, and individual knowledge of agent i, respectively. Thus, the formula says that it is distributed knowledge
among agents 1 and 2 that p is true and that agent 1 does not know
p. This formula is consistent (also when we assume that knowledge
has the S5 properties). However, it is not possible that agents 1 and2 both can get to know that p is true and that agent 1 does not knowthat p is true (assuming the S5 properties of knowledge), no matter how much they communicate (or “pool” their knowledge). The“problem” here is that in a formula DG ψ, ψ describes the possible
states of the world as they were before any communication or otherevents took place, so a more accurate reading of D{1,2} (p∧¬K1 p)
would perhaps be that it follows from the combination of 1 and
2’s knowledge that p ∧ ¬K1 p were true before any communication or other events took place. More technically, the “problem” isdue to the standard compositional semantics of modal logic: in theevaluation of DG ϕ, the DG operator picks out a number of statesconsidered possible by the group G (actually the states considered
possible by all members of the group), and then ϕ is evaluated ineach of these states in the original model, without any effect of the
DG operator.
But we don’t really consider this a problem. There are otherinterpretations of distributed knowledge where the consistency ofthe mentioned formula makes perfect sense, such that distributed
knowledge being the knowledge of a third party, someone “outside
the system” who somehow has access to the epistemic states of thegroup members. It shows, however, that it does not make sense tointerpret distributed knowledge as something that is true after theagents in the group have communicated with each other – with thestandard semantics.
In this paper we introduce and study an alternative group modality RG , where RG ϕ means (roughly speaking) that ϕ is true after
the agents in the group have shared all their information with eachother. We call that resolving distributed knowledge, and the RG
operators are called resolution operators.
Semantically, we say that an expression RG ϕ is true iff ϕ is true
in what van Benthem [8, p. 249] calls (G’s) communication core;
the model update obtained by removing links to states for membersof G that are not linked by all members of G. See Fig. 1 foran illustration. In this paper we capture that model transformationby the new resolution operators, and study resulting logics. For
example, the formula R{1,2} (p ∧ ¬K1 p) will be inconsistent in the
resulting logics. R{1,2} (p ∧ K1 p) is true in state t in the model inFig. 1.
This model transformation abstracts away from the issue of how
reader the impression that distributed knowledge is about internal
communication in the group of agents.

•¬p
s

1

•pt
1,2

•pv

2

•¬p
u

•pt
1,2

•pv

1

•pw
Figure 1: Example taken from [8, p. 248]. Model on the left,
its communication core (for the set of all agents {1, 2}) on theright. Reflexivity, symmetry and transitivity are implicitly assumed.
the agents share their information; whether they communicate directly with each other and if so in which language, whether they are
informed by some outsider about the information other agents haveand if so how, and so on. As noted by van Benthem [8, p. 249],
the communication core cannot always be obtained by public announcements using the epistemic language. Similarly, as noted byseveral researchers [11, 4, 7], standard distributed knowledge does
not always follow logically from the knowledge of the individual
agents expressible in the epistemic language. Our model, like thatof standard distributed knowledge, is purely semantic: we assume
that if an agent can discern between two different worlds, then there
exists some mechanism that results in other members of the group
being able to make the same distinction. This is further discussed
Section 5.
This model transformation models a particular kind of internalgroup information sharing event. Exactly which kind depends on
what we assume about what other agents, i.e., agents that are notin the group G that resolve their knowledge, know about the factthat this event is taking place. In this paper we will assume that
it is common knowledge among the other agents that G resolve
their knowledge – but not what the agents in G actually learn. This
corresponds to a natural class of events: publicly observable private
resolution of distributed knowledge. An example is a meeting in aclosed room, where it is observed that a certain group meets in theroom to share information.
We want to make it clear that we do not consider distributed
knowledge with standard semantics to be “wrong”; the important
thing is to be clear about its meaning. In particular, the resolution
operators are not intended as a “replacement” of distributed knowledge operators, but as a complement: they express different things.
The logics we study contains both types of operators, as well as
common knowledge. The main results are sound and complete axiomatizations.
Technically, the model transformation, which amounts to removing certain edges, is similar to those found in the simplest dynamic
epistemic logics [12] such as public announcement logics [6]. [8]has also pointed out the close connection between the communication core and sequences of public announcements. Public announcement logics with distributed knowledge have been studied
recently [13]. In the absence of common knowledge, we get reduction axioms for public announcement logic with distributed knowledge. This turns out to be the case for resolution operators as well.
It is not the case in the presence of common knowledge, however.
There is a close connection between the communication core andcommon knowledge [8]. By studying complete axiomatizations oflogics with the resolution operators we make some aspects of thatconnection precise and give an answer to the question “when does

distributed knowledge become common knowledge?” – under certain assumptions.
The rest of the paper is organized as follows. In the next section we review some background definitions and results from theliterature, before we introduce logics with the new resolution operators in Section 3, where we also look at some properties of theoperators. In Section 4 we prove completeness of resulting logics;
the most interesting case being epistemic logic with common anddistributed knowledge and resolution operators. We discuss related
and future work and conclude in Section 5.

2.

BACKGROUND

In this section we give a (necessarily brief) review of the main
background concepts from the literature.
We henceforth assume a countable set of propositional variables
PROP and a finite set of agents AG . We let GR be the set of all
non-empty groups, i.e., GR = ℘(AG) \ ∅.
An epistemic model over PROP and AG (or just a model) M =(S, ∼, V ) where S is a set of states (or worlds), V : PROP → 2S
associates a set of states V (p) with each propositional variable p,and ∼ is a function that maps each agent to a binary equivalence
relation on S. We write ∼i for ∼(i).
s ∼i t means that agent i cannot discern between states s andt – if we are in s she doesn’t know whether we are in t, andvice versa. Considering the distributed knowledge of a group G– a keyTconcept in the following – we define a derived relation∼G = a∈G ∼a (it is easy to see that ∼G is an equivalence relation). Intuitively, someone who has all the knowledge of all themembers of G can discern between two states if and only if at least
one member of G can discern between them. We will also consider common knowledge. A similar relation modeling the common knowledge of a group is obtained by taking the transitive
cloS
sure of the union of the individual relations: vCG = ( i∈G ∼i )∗ .
D EFINITION 1. Below are several languages from the literature.
(ELD)
ϕ ::= p | ¬ϕ | ϕ ∧ ϕ | Ki ϕ | DG ϕ
(ELCD) ϕ ::= p | ¬ϕ | ϕ ∧ ϕ | Ki ϕ | DG ϕ | CG ϕ
(PACD) ϕ ::= p | ¬ϕ | ϕ ∧ ϕ | Ki ϕ | DG ϕ | CG ϕ | [ϕ]ϕ,where p ∈ PROP, i ∈ AG and G ∈ GR. We use
V the usual propositional derived operators, as well as EG ϕ for i∈G Ki ϕ.

ELD and ELCD are static epistemic languages with distributed
knowledge, and with distributed and common knowledge, respectively. These are the languages we will extend with resolution operators in the next section. We will also be interested in PACD, thelanguage for public announcement logic with both common knowledge and distributed knowledge, when we look at completenessproofs.
Satisfaction of a formula ϕ of any of these languages in a state
m of a model M, denoted M, m |= ϕ, is defined recursively by thefollowing clauses:
M, m |= p
iff m ∈ V (p)M, m |= ¬ϕ
iff M, m 6|= ϕ
M, m |= ϕ ∧ ψ iff M, m |= ϕ & M, m |= ψM, m |= Ka ϕ iff ∀n ∈ S. (m ∼a n ⇒ M, n |= ϕ)M, m |= DG ϕ iff ∀n ∈ S. (m ∼SG n ⇒ M, n |= ϕ)M, m |= CG ϕ iff ∀n ∈ S. (m( i∈G ∼i )∗ n ⇒ M, n |= ϕ)M, m |= [ψ]ϕ iff M, m |= ψ ⇒ M|ψ, m |= ϕ.

where R∗ denotes the transitive closure of R and M|ψ is the submodel of M restricted to {m ∈ M | M, m |= ψ}. Validity isdefined as usual: |= ϕ means that M, m |= ϕ for all M and m.
38

We now define some axiom schemata and rules. The classical“S5” proof system for multi-agent epistemic logic, denoted (S5),consists of the following axioms and rules:
(PC)
(K)
(T)
(4)
(5)
(MP)
(N)

instances of tautologies
Ki (ϕ → ψ) → (Ki ϕ → Ki ψ)
Ki ϕ → ϕ
Ki ϕ → Ki Ki ϕ
¬Ki ϕ → Ki ¬Ki ϕfrom ϕ and ϕ → ψ infer ψfrom ϕ infer Ki ϕ.

Also note that (∼|G )i = (∼|G ){i} .

3.1

P ROPOSITION 3. The following is valid, where i ∈ AG andϕ ∈ RCD: R{i} ϕ ↔ ϕ.

DG (ϕ → ψ) → (DG ϕ → DG ψ)
DG ϕ → ϕ¬DG ϕ → DG ¬DG ϕ
Ki ϕ ↔ Di ϕ
DG ϕ → DH ϕ, if G ⊆ H.

More interesting are the following properties.

Axioms and rules for common knowledge, denoted (CK):
(KC )
(TC )
(C1)
(C2)
(NC )

CG (ϕ → ψ) → (CG ϕ → CG ψ)
CG ϕ → ϕ
CG ϕ → E G CG ϕ
CG (ϕ → EG ϕ) → (ϕ → CG ϕ)from ϕ infer CG ϕ.

The system that consists of (S5) and (DK) over the language
ELD, denoted S5D, is a sound and complete axiomatization of all
ELD validities. The system that consists of (S5), (DK) and (CK)over the language ELCD is a sound and complete axiomatizationof all ELCD validities.

3.

RESOLVING DISTRIBUTED KNOWLEDGE

We want to model the event that G resolves their knowledge. An
immediate question is: whenever the group G is a proper subset ofthe set of all agents, what do the other agents know about the factthat this event takes place? Here we will model situations where it
is common knowledge among the other agents that the event takes
place, but not what the members of the group learn. As discussed in
the introduction, this corresponds to a natural class of information
sharing events, namely publicly observable private communication,such as a meeting in a closed room that is observed to be taking
place. This is captured by a global model update: in every state,remove a link to another state for any member of G whenever it is
not the case that there is a link to that state for all members of G.
Formally, given a model M = (S, ∼, V ) and a group of agents
G, the (global) G-resolved update of M is the model M|G where
M|G = (S, ∼|G , V ) and Tj∈G ∼j , i ∈ G,(∼|G )i =∼i ,otherwise.
We consider the following new languages with resolution operators.
D EFINITION 2(L ANGUAGES ).

(RD)
ϕ ::= p | ¬ϕ | ϕ ∧ ϕ | Ki ϕ | DG ϕ | RG ϕ
(RCD) ϕ ::= p | ¬ϕ | ϕ ∧ ϕ | Ki ϕ | DG ϕ | CG ϕ | RG ϕ,

where p ∈ PROP, i ∈ AG and G ∈ GR.

The interpretation of these languages in a pointed model is defined as usual, with the following additional clause for the resolution operator:
M, s |= RG ϕ

Some Validities

Let us start with a trivial validity: resolution has no effect for asingleton coalition.

Axioms for distributed knowledge, denoted (DK):
(KD )
(TD )
(5D )
(D1)
(D2)

T
A couple of observations. Recall that we write ∼H for i∈H ∼i .
Thus,


∼G , i ∈ G,
∼H ,G ∩ H = ∅,(∼|G )i =(∼|G )H =∼i , i ∈/ G.
∼G∪H , G ∩ H 6= ∅.

iff M|G , s |= ϕ.
39

P ROPOSITION 4 (R EDUCTION P RINCIPLES ). The following
are valid, where G, H ∈ GR, p ∈ PROP and ϕ ∈ RCD:
1. RG p ↔ p
2. RG (ϕ ∧ ψ) ↔ RG ϕ ∧ RG ψ
3. RG ¬ϕ ↔ ¬RG ϕ
4. RG Ki ϕ ↔ DG RG ϕ, when i ∈ G
5. RG Ki ϕ ↔ Ki RG ϕ, when i 6∈ G
6. RG DH ϕ ↔ DG∪H RG ϕ, when G ∩ H 6= ∅7. RG DH ϕ ↔ DH RG ϕ, when G ∩ H = ∅.
These properties are reduction principles, of the type known frompublic announcement logic: they allow us to simplify expressions
involving resolution operators. If we have such principles for the
combination of resolution with all other operators we can eliminate
resolution operators altogether. There are two cases missing above:
RG CH and RG RH 2 . We consider them next.

3.1.1

Common Knowledge

First, after the grand coalition have resolved their knowledge,
then all the distributed information in the system is common knowledge: there is no longer a distinction between distributed and common knowledge:
P ROPOSITION 5. For any ϕ ∈ RCD: RAG CAG ϕ ↔ RAG DAG ϕ.
P ROOF. 5. Given a model M = (S, ∼, V ) and s ∈ S,
iff
iff
iff
iff
iff

M, s |= RAG CAG ϕ
M|AG , s |= CAG ϕ∀t ∈ S. (s(∼|AG )CAG t ⇒ M|AG , s |= ϕ)∀t ∈ S. (s ∼AG t ⇒ M|AG , s |= ϕ) (†)
M|AG , s |= DAG ϕ
M, s |= RAG DAG ϕ,

where for (†) we show that (∼|AG )CAG =∼AG . This is easy: bydefinition we can
S verify that for all i ∈ AG, (∼|AG )i =∼AG ; hence(∼|AG )CAG = ( i∈AG (∼|AG )i )∗ = (∼AG )∗ =∼AG .
For the general case, as in the case of distributed knowledge, wehave that the resolution operators and common knowledge operators commute when the groups are disjoint:

P ROPOSITION 6. Let i be an agent, G and H groups of agentsand ϕ ∈ RCD. The following hold:
1. If G ∩ H = ∅, then |= RG CH ϕ ↔ CH RG ϕ
2
The lack of a reduction axiom for the general RG RH ϕ case does
not mean we cannot get a reduction in the language RD: we cansimply do the reduction “inside-out”.

2. If G ⊇ H and i ∈ G, then |= RG CH ϕ ↔ RG Ki ϕ ↔
DG RG ϕ.

follows:
δ0 =P ROOF. See the appendix.
However, this does not hold for overlapping groups G and H.
In general, we have that (see the proof of the proposition above)0
M, s |= RG CH ϕ iff M|G , t |= ϕ for any (s, t) ∈∼∗H , where
T
S
0
∗
∼H
= ( i∈G ∼i ∪ i∈H\G ∼i )∗ . This does not seem to be
reducible.

3.1.2

Iterated resolution

What about RG RH ϕ? In extreme cases, we have:
P ROPOSITION 7. The following are valid, where G, H ∈ GR
and ϕ ∈ RCD:
1. RG RH ϕ ↔ RH RG ϕ, if G ∩ H = ∅2. RG RG ϕ ↔ RG ϕ.
However, in the general case there does not seem to be a reduction axiom in this case. In particular, RG RH ϕ is not equivalent to
RG∪H ϕ.
Let us consider an example of iterated resolution.
E XAMPLE 8 (T RIPLE UPDATE ). Let M = (S, ∼, V ) and
M|G1 |G2 |G3 = (S, ∼|G1 |G2 |G3 , V ). For any agent i, for any
number x, we write Gx for “i ∈ Gx ”, and Gx for “i ∈/ Gx ”. Then
if G1 G2 G3 :
∼i




if
G
G
G
:
∼

1
2
3
G1




G
∩
G
=
∅
:
∼

1
2
G2

if G1 G2 G3
G1 ∩ G2 6= ∅ : ∼G1 ∪G2if G1 G2 G3 :∼G1 ∪G2(∼|G1 |G2 |G3 )i =

if
G
G
G
:
∼G3
1
2
3




:
∼G1 ∪G3
if G1 G2 G3 
G1 ∩ G2 = ∅ : ∼G2 ∪G3
if G1 G2 G3
G1 ∩ G2 6= ∅ : ∼G1 ∪G2 ∪G3
if G1 G2 G3 :∼G1 ∪G2 ∪G3
In general we get the following (the proof is straightforward fromthe semantic definition).

P ROPOSITION 9. Let M = (S, ∼, V ) and M |G1 | · · · |Gn =(S, ∼|G1 | · · · |Gn , V ). Then, following the notation of Example 8,
for any i ∈ AG,∼i ,if G1 · · · Gn∼G1 ∪Θ , if starting with G1 G2 and(∼|G1 | · · · |Gn )i =G1 ∩ G2 6= ∅



∼Θ ,
otherwiseδx =δ(H, G1 , . . . , Gn ) =D EFINITION 10 (δ FUNCTION ). Given an agent i, a group H,
and a sequence of groups G1 , . . . , Gn , we define a function δ as

Gn−x ∪ δx−1 ,δx−1 ,δn .

Gn ∩ H 6= ∅Gn ∩ H = ∅Gn−x ∩ δx−1 6= ∅Gn−x ∩ δx−1 = ∅P ROPOSITION 11. Let i ∈ AG, G1 , . . . , Gn , H ∈ GR, M =(S, ∼, V ) and M |G1 | · · · |Gn = (S, ∼|G1 | · · · |Gn , V ). Then,
1. |= RG1 · · · RGn Ki ϕ ↔ Dδ({i},G1 ,...,Gn ) RG1 · · · RGn ϕ;
2. |= RG1 · · · RGn DH ϕ ↔ Dδ(H,G1 ,...,Gn ) RG1 · · · RGn ϕ;
3. (∼|G1 | · · · |Gn )i =∼δ({i},G1 ,...,Gn ) ;4. (∼|G1 | · · · |Gn )H =∼δ(H,G1 ,...,Gn ) .

P ROOF. Straightforward: the recursive steps in the definition ofthe δ function matches exactly the reduction axioms. Note thatclauses 1 and 3 can be treated as special cases of clauses 2 and 4
respectively.

4.

AXIOMATIZATIONS

We construct sound and complete axiomatizations of the logics
for the two languages RD and RCD.

4.1

Resolution and Distributed Knowledge

Consider the language RD. Let RD be the system defined in
Figure 2, where (S5) and (DK) are found in Section 2 and (RR)
stands for the following reduction axioms for resolution:
(RA)
(RC)
(RN)
(RD1)
(RD2)

RG p ↔ p
RG (ϕ ∧ ψ) ↔ RG ϕ ∧ RG ψ
RG ¬ϕ ↔ ¬RG ϕ
RG DH ϕ ↔ DG∪H RG ϕ, if G ∩ H 6= ∅RG DH ϕ ↔ DH RG ϕ, if G ∩ H = ∅.

Note that (RR) contains most of the validities in Proposition 4, except for the reduction principles for individual knowledge – they
are provable with RD1, RD2 and D1. In addition, we need the rule
NR for making a reduction to S5D. With the rule NR we can easily
show that the rule of Replacement of Equivalents (RoE) is admissible in RD. RoE allows us to carry out a reduction even without
having a reduction axiom for iterated resolution.
(S5)
(DK)
(RR)
(NR )

Reduction Normal Form for Individualand Distributed Knowledge

As we see from the previous section, the reduction axioms forindividual knowledge and distributed knowledge both contain two
distinct cases, and the principles of iterated resolution become more
complicated. In this section we give a unique form for such reductions, which will be of use later when we prove completeness. We
shall call it reduction normal form for individual and distributed
knowledge.

Gn ∪ H,H,

Clearly δ(H, G1 , . . . , Gn ) ⊆ H ∪ G1 ∪ · · · ∪ Gn . We simply
write δ instead of δ(H, G1 , . . . , Gn ) when its parameters are clear
in the context.

where Θ is the union of all Gx such that i ∈ Gx .

3.2

classical proof system for multi-agent epistemic logic
characterization axioms for distributed knowledge
reduction axioms for resolution
from ϕ infer RG ϕ
Figure 2: Axiomatization RD.

T HEOREM 12. Any RD formula is valid if and only if it is provable in RD.

4.2

Resolution, Distributed and Common
Knowledge

Consider the language RCD. Let RCD be the system defined in
Figure 3, which extends RD with (CK), found in Section 2, and an
induction rule for resolved common knowledge (RRC ).

40

(S5)
(CK)
(DK)
(NR )
(RR)
(RRC )

classical proof system for multi-agent epistemic logic
axioms and rules for common knowledge
characterization axioms for distributed knowledge
from ϕ infer RG ϕ
reduction axioms for resolution
from ϕ → (EH ϕ ∧ RG1 · · · RGn ψ) infer
ϕ → RG1 · · · RGn CH ψ
Figure 3: Axiomatization RCD

4.2.1

Soundness

For soundness it suffices to show that the rule RRC preserves validity (we know that the other axioms/rules are valid/validity preserving from soundness results for the logics based on the sublanguages of RCD).

L EMMA 13 (RRC - VALIDITY PRESERVATION ). For all RCD
formulas ϕ and ψ, all G1 , . . . , Gn , H ∈ GR, if |= ϕ → (EH ϕ ∧RG1 · · · RGn ψ), then |= ϕ → RG1 · · · RGn CH ψ.

P ROOF. Suppose |= ϕ → (EH ϕ ∧ RG1 · · · RGn ψ). Given amodel M and a state s, suppose M, s |= ϕ, we must show thatM, s |= RG1 · · · RGn CH ψ, i.e., M|G1 | · · · |Gn , s |= CH ψ. Thus,
for all H-paths s0 (∼ |G1 | · · · |Gn )i0 · · · (∼ |G1 | · · · |Gn )ix−1 sx ,where s = s0 , we need to show that M|G1 | · · · |Gn , sx |= ψ.
From |= ϕ → (EH ϕ ∧ RG1 · · · RGn ψ) and M, s0 |= ϕ we get
M, s0 |= (EH ϕ ∧ RG1 · · · RGn ψ), which entails:
M, s1 |= ϕandM, s2 |= ϕandM, sy |= ϕand

M|G1 | · · · |Gn , s0 |= ψ.

From M, s1 |= ϕ we get M, s1 |= (EH ϕ∧RG1 · · · RGn ψ), which
entails:

Pseudo Semantics.
D EFINITION 15 (P RE - MODELS [13]). A pre-model is a tuple
M = (S, v, V ) where:• S is a non-empty set of states;• v is a function which maps every agent and every non-empty
group of agents to an equivalence relation; we write vi andvG for v(i) and v(G) respectively;• V : PROP → ℘(S) is a valuation.
SvCG is defined as the reflexive transitive closure of i∈G vi ,just as for a model.
A pre-model is technically a model with a bigger set of agents(all groups are treated as agents in a pre-model). More precisely,
if we make the set of agents A explicit in a pre-model, e.g., M =(A, S, v, V ), then M is in fact a “genuine” model (S, v, V ) where
the set of agents is A ∪ (℘(A) \ ∅).
D EFINITION 16 (P SEUDO MODELS [13]). A pseudo model isa pre-model M = (S, v, V ) such that for any agent i and anygroups G and H,• v{i} =vi , and• G ⊆ H implies vH ⊆vG .
A pointed pre-model (resp. pointed pseudo model) is a tuple (M, s)consisted of a pre-model (resp. pseudo model) M and a state s inM.
D EFINITION 17 (P SEUDO SEMANTICS ). Given a pre-modelM = (S, v, V ), let m be a state in M . Satisfaction at (M, s) isdefined as follows:

M|G1 | · · · |Gn , s1 |= ψ.

By similar reasoning, for all y = 0, . . . , x, we have

M|G1 | · · · |Gn , sy |= ψ,which entails M|G1 | · · · |Gn , sx |= ψ as we wish to show.
C OROLLARY 14 (S OUNDNESS ). For any RCD formula ϕ, if
ϕ is provable in RCD, then it is valid.

4.2.2

rule for resolved common knowledge (RRC ). No corresponding
rule is needed in the PACD completeness proof. The rule is usedin the proof of Lemma 29(8).

CompletenessAs already discussed, RCD is similar to PACD (axiomatization for public announcement logic with common and distributed
knowledge; see [13]): both logics extend epistemic logic with common and distributed knowledge with dynamic operators with update semantics that remove states. There does not seem, however,to be a trivial relationship between the two types of dynamic operators. We are nevertheless able to make heavy use of the completeness proof of PACD in [13] when proving completeness of RCD.
That proof is again based on the completeness proof for public announcement logic with (only) common knowledge found in [2, 12],extended to deal with the distributed knowledge operators (which
is non-trivial since intersection is not modally definable). In thefollowing completeness proof we tweak the PACD proof to deal
with resolution operators instead of public announcement operators. The general proof strategy is as follows: define a finite canonical pseudo model, where distributed knowledge operators are taken
as primitive, and then transform it to a proper model while preserving truth. For the last step we can use a transformation based on
unraveling and folding in [13] directly.
The most important difference to the PACD completeness proofin [13], and indeed the crux of the proof, is the use of the induction

41

M, s |=p pM, s |=p ¬ϕM, s |=p ϕ ∧ ψM, s |=p Ki ϕ
M, s |=p CG ϕ
M, s |=p DG ϕ
M, s |=p RG ψ

iff
iff
iff
iff
iff
iff
iff

s ∈ V (p)M, s 6|=p ϕM, s |=p ϕ & M, s |=p ψ(∀t ∈ M)(s vi n ⇒ M, t |=p ϕ)(∀t ∈ M)(s vCG t ⇒ M, t |=p ϕ)(∀t ∈ M)(s vG t ⇒ M, t |=p ϕ)
M|G , s |=p ϕ,

where M|G = (S, v|G , V ) such thatvG , i ∈ G
vH∪G , H ∩ G 6= ∅(v|G )i =and (v|G )H =vi , i ∈/GvH , H ∩ G = ∅Satisfaction in a pre-model M (denoted by M |=p ϕ) is definedas usual. We use |=p ϕ to denote validity, i.e. M, s |=p ϕ for any
pointed pre-model (M, s). We write |= instead of |=p when there isno confusion.
P ROPOSITION 18. Let M be a pseudo model, G a group ofagents. Then M|G is a pseudo model.
P ROOF. See the appendix.
P ROPOSITION 19. Propositions 9 and 11 still hold for pseudomodels.
When we regard a pre-model as a genuine model, classical (individual) bisimulation becomes an invariance relation. To makethis clear, we first elaborate the definition of bisimulation for premodels, and then introduce its invariance results.

D EFINITION 20 (P RE - MODEL BISIMULATION ). Let two premodels M = (S, v, V ) and M0 = (S 0 , v0 , V 0 ) be given. A nonempty relation Z ⊆ S × S 0 is called a bisimulation between M
and M0 , denoted by M  M0 , if for all τ ∈ AG ∪ GR, all s ∈ S
and s0 ∈ S 0 such that sZs0 , the following hold.
(at) For all p ∈ PROP, s ∈ V (p) iff s0 ∈ V 0 (p);(zig) For all t ∈ S, if s ∼τ t, there is a t0 ∈ S 0 such that s0 ∼0τ t0
and tZt0 ;(zag) For all t0 ∈ S 0 , if s0 ∼0τ t0 , there is a t ∈ S such that s ∼τ t
and tZt0 .
We say that pointed pre-models (M, s) and (M0 , s0 ) are bisimilar,
denoted (M, s)  (M0 , s0 ), if there is a bisimulation Z between
M and M0 linking s and s0 .
P ROPOSITION 21.Resolution preserves pre-model bisimulation.
I.e., for all pointed pre-models (M, s) and (M0 , s0 ), if (M, s) 
(M0 , s0 ) then (M|G , s)  (M0 |G , s0 ).
P ROOF. See the appendix.

C OROLLARY 22. For any pre-models M and M , if (M, s) 
(M0 , s0 ) then M, s |=p ϕ iff M0 , s0 |=p ϕ for any RCD formula ϕ.
0

As introduced in [13], we can also consider a kind of bisimulations between genuine models and pre-models.
D EFINITION 23 (T RANS - BISIMULATION [13]). Let a modelM = (M, ∼, V ) and a pre-model N = (N, v, ν) be given. A nonempty binary relation Z ⊆ M × N is called a trans-bisimulation
between M and N, if for all m ∈ M and n ∈ N with mZn:(at) m ∈ V (p) iff n ∈ ν(p) for all p ∈ PROP,(zigag ) For all m0 ∈ M and all i ∈ AG, if m ∼i m0 (and som ∼{i} m0 ), then there is an n0 ∈ N such that m0 Zn0 andn vτ0 · · · vτx n0 with each of τ0 , . . . , τx being “i” or “G”
such that i ∈ G;
(ziggr ) For all m0 ∈ M and all G ∈ GR with |G| ≥ 2, if
m ∼G m0 , then there is an n0 ∈ N such that m0 Zn0 andn vG1 · · · vGx n0 with G ⊆ G1 ∩ · · · ∩ Gx ;(zag) For all n0 ∈ N and all τ ∈ AG ∪ GR, if n vτ n0 , then there
is an m0 ∈ M such that m0 Zn0 and m ∼τ m0 .
We write Z : (M, m) T (N, n) if Z is a trans-bisimulation between M and N linking m and n. We say a pointed model (M, m)and a pointed pre-model (N, n) are trans-bisimilar, denoted by(M, m) T (N, n), if there is a trans-bisimulation Z such that
Z : (M, m) T (N, n).
To make the notation symmetric, we call Z a trans-bisimulation
between N and M if it is a trans-bisimulation between M and N,
and we regard Z : (N, n) T (M, m) just as Z : (M, m) T(N, n).
(Pseudo) satisfaction of RCD formulas is invariant under transbisimulation. We will not prove that directly at this point: it follows
from a stronger result we prove later (Lemma 33).
T HEOREM 24 (P SEUDO SOUNDNESS ). All theorems of RCD
are valid in the class of all pseudo models.
P ROOF. See the appendix.

Finitary Canonical Models.
D EFINITION 25 (C LOSURE ). Given a formula ϕ, the closureof ϕ is given by the function cl : RCD → ℘(RCD) which isdefined as follows:

1. ϕ ∈ cl(ϕ), and if ψ ∈ cl(ϕ), so are all of its subformulas;
2. If ϕ is not a negation, then ϕ ∈ cl(ϕ) implies ¬ϕ ∈ cl(ϕ);
3. Ki ψ ∈ cl(ϕ) iff D{i} ψ ∈ cl(ϕ);
4. CG ψ ∈ cl(ϕ) implies {Ki CG ψ | a ∈ A} ⊆ cl(ϕ);
5. RG1 · · · RGn ¬ψ ∈ cl(ϕ) implies RG1 · · · RGn ψ ∈ cl(ϕ);
6. RG1 · · · RGn (ψ ∧ χ) ∈ cl(ϕ) implies{RG1 · · · RGn ψ, RG1 · · · RGn χ} ⊆ cl(ϕ);
7. RG1 · · · RGn Ki ψ ∈ cl(ϕ) impliesDδ({i},G1 ,...,Gn ) RG1 · · · RGn ψ ∈ cl(ϕ);
8. RG1 · · · RGn DH ψ ∈ cl(ϕ) implies
Dδ(H,G1 ,...,Gn ) RG1 · · · RGn ψ ∈ cl(ϕ);
9. RG1 · · · RGn CH ψ ∈ cl(ϕ) implies all of the following:• Dδ(H,G1 ,...,Gn ) RG1 · · · RGn CH ψ ∈ cl(ϕ),• {Dδ({i},G1 ,...,Gn ) RG1 · · · RGn CH ψ | i ∈ H} ⊆ cl(ϕ),• RG1 · · · RGn ψ ∈ cl(ϕ).
It is not hard to verify that the closure of a formula is finite.
We use Γ as shorthand forformulas.

Vϕ∈Γ ϕ when Γ is a finite set of

D EFINITION 26 (C ANONICAL PSEUDO MODEL ). Let α be aformula. The canonical pseudo model Mc = (S, v, V ) for cl(α)
is defined below:• S = {Γ | Γ is maximal consistent in cl(α)};• Γ vi ∆ iff {Ki ϕ | Ki ϕ ∈ Γ} = {Ki ϕ | Ki ϕ ∈ ∆};• Γ vG ∆ iff {DH ϕ | DH ϕ ∈ Γ} = {DH ϕ | DH ϕ ∈ ∆}whenever H ⊆ G;
• V (p) = {Γ ∈ S | p ∈ Γ}.
P ROPOSITION 27. The canonical pseudo model for any cl(α)
is a pseudo model.
P ROOF. See the appendix.
L EMMA 28. Let S = {Γ | ΓWis maximal consistentWin cl(α)}
with α a formula. It holds that ` Γ∈S Γ and ` ϕ ↔ ϕ∈Γ∈S Γ
for all ϕ ∈ cl(α).

P ROOF. See [12, Exercise 7.16] for the first result (althoughcl(α) is different in our case the proof is exactly the same). We
give a proof of the second result in the appendix.

Let (S, v|G1 | · · · |Gn , V ) be an update of a canonical pseudomodel, and P = hΦ0 τ0 · · · τn−1 Φn i where  stands for
v|G1 | · · · |Gn and every τx is an agent or a group. If all agents inτ0 , . . . , τn−1 appears in H, we call P a hG1 · · · Gn i-resolved Hpath (from Φ0 ); if a formula ϕ is such that ϕ ∈ Φi for all 0 ≤ i ≤n, we call P a canonical ϕ-path.
L EMMA 29. If Γ and ∆ are maximal consistent in cl(α), then
1. Γ is deductively closed in cl(α), i.e., Γ ` ϕ ⇔ ϕ ∈ Γ for any
ϕ ∈ cl(α);
2. If ¬ϕ ∈ cl(α), then ϕ ∈ Γ ⇔ ¬ϕ ∈/ Γ;
3. If ϕ ∧ ψ ∈ cl(α), then ϕ ∧ ψ ∈ Γ ⇔ ϕ ∈ Γ & ψ ∈ Γ;
4. If Γ ∧ K̂i ∆ is consistent, Γ vi ∆; if Γ ∧ D̂G ∆ is consistent,
Γ vG ∆;
5. If Ki ϕ ∈ cl(α), then Ki Γ ` ϕ ⇔ Ki Γ ` Ki ϕ;
6. If DG ϕ ∈ cl(α), then DG Γ ` ϕ ⇔ DG Γ ` DG ϕ;
7. If CG ϕ ∈ cl(α), then CG ϕ ∈ Γ ⇔ ∀∆(Γ vCG ∆ ⇒ ϕ ∈ ∆);8. If RG1 · · · RGn CH ϕ ∈ cl(α), then RG1 · · · RGn CH ϕ ∈ Γ
iff every hG1 · · · Gn i-resolved H-path from Γ is a canonical
RG1 · · · RGn ϕ-path.
42

P ROOF. We give the proof of the clause 8 in the appendix. Other
clauses are the same as in [13, Lemma 49] which can be traced backto [12, Chapter 7].
L EMMA 30 (P SEUDO TRUTH ). Let Mc = (S, v, V ) be thecanonical pseudo model for cl(α). For all groups G1 , . . . , Gn , allΓ ∈ S, and all RG1 · · · RGn ϕ ∈ cl(α), it holds that
RG1 · · · RGn ϕ ∈ Γ iff

Mc |G1 | · · · |Gn , Γ |= ϕ.

P ROOF. We show this lemma by induction on ϕ.
• The base case. RG1 · · · RGn p ∈ Γ iff p ∈ Γ (Proposition 4(1))
iff Mc , Γ |= p iff Mc , Γ |= RG1 · · · RGn p iff
Mc |G1 | · · · |Gn , Γ |= p.
• The case for negation. RG1 · · · RGn ¬ψ ∈ Γ
iff ¬RG1 · · · RGn ψ ∈ Γ (note that ¬RG1 · · · RGn ψ ∈ cl(α)/Γ
by Definition 25(2,5)) iff RG1 · · · RGn ψ ∈iff Mc |G1 | · · · |Gn , Γ 6|= ψ iff Mc |G1 | · · · |Gn , Γ |= ¬ψ.
• The case for conjunction. RG1 · · · RGn (ψ ∧ χ) ∈ Γiff (RG1 · · · RGn ψ ∧ RG1 · · · RGn χ) ∈ Γ
iff {RG1 · · · RGn ψ, RG1 · · · RGn χ} ⊆ Γ (RG1 · · · RGn ψ and
RG1 · · · RGn χ are in cl(α))
iff Mc |G1 | · · · |Gn , Γ |= ψ and Mc |G1 | · · · |Gn , Γ |= χ
iff Mc |G1 | · · · |Gn , Γ |= ψ ∧ χ.
• The case for individual knowledge. From left to right.
RG1 · · · RGn Ki ψ ∈ Γ
iff Dδ RG1 · · · RGn ψ ∈ Γ where δ = δ({i}, G1 , . . . , Gn )iff ∀∆.(Γ vδ ∆ ⇒ Dδ RG1 · · · RGn ψ ∈ ∆)(TD )⇒ ∀∆.(Γ vδ ∆ ⇒ RG1 · · · RGn ψ ∈ ∆)iff ∀∆.(Γ vδ ∆ ⇒ Mc |G1 | · · · |Gn , ∆ |= ψ) (IH)
iff ∀∆.(Γ vδ ∆ ⇒ Mc , ∆ |= RG1 · · · RGn ψ)
iff Mc , Γ |= Dδ RG1 · · · RGn ψ(11(1), 19)
iff Mc , Γ |= RG1 · · · RGn Ki ψ
iff Mc |G1 | · · · |Gn , Γ |= Ki ψ.
From right to left. Suppose Mc |G1 | · · · |Gn , Γ |= Ki ψ. We
must show RG1 · · · RGn Ki ψ ∈ Γ. Suppose this is not the case.
Then ¬RG1 · · · RGn Ki ψ ∈ Γ. Hence Γ ∧ ¬RG1 · · · RGn Ki ψ
is consistent, and so is Γ ∧ D̂δ ¬RG1 · · · RGn ψ, where δ =δ({i}, G1 , . . . , Gn ). Let S be the set of all maximal consistentWsets in cl(α). By Lemma 28, Γ ∧ D̂δ ¬RG ···RG ψ∈Θ∈S Θ is1nconsistent. Since conjunction,Wresolution and the D̂δ -operator
all distribute over disjunction, ¬RG ···RG ψ∈Θ∈S (Γ ∧ D̂δ Θ)n1
is consistent. Therefore there must be a Θ ∈ S such that
¬RG1 · · · RGn ψ ∈ Θ and Γ ∧ D̂δ Θ is consistent.
From ¬RG1 · · · RGn ψ ∈ Θ we get RG1 · · · RGn ψ ∈/ Θ.
By the induction hypothesis Mc |G1 | · · · |Gn , Θ 6|= ψ, and so
Mc , Θ 6|= RG1 · · · RGn ψ. By Lemma 29(4) and that Γ ∧ D̂δ Θ
is consistent, Γ vδ Θ. But this contradicts the supposition
that Mc |G1 | · · · |Gn , Γ |= Ki ψ, since by the same reasoning
as in the proof of the other direction (see above), Mc , ∆ |=RG1 · · · RGn ψ for all ∆ such that Γ vδ ∆.
• The case for distributed knowledge: similar to the case for individual knowledge, but just use δ(H, G1 , . . . , Gn ) instead of
δ({i}, G1 , . . . , Gn ).
• The case for common knowledge. RG1 · · · RGn CH ψ ∈ Γ
iff all hG1 · · · Gn i-resolved H-paths from Γ are also canonical
RG1 · · · RGn ψ-paths. Namely, for all ∆ such that (Γ, ∆) ∈(v|G1 | · · · |Gn )CH , RG1 · · · RGn ψ ∈ ∆iff for all ∆ such that (Γ, ∆) ∈ (v|G1 | · · · |Gn )CH , it holds by
the induction hypothesis that Mc |G1 | · · · |Gn , ∆ |= ψ
iff Mc |G1 | · · · |Gn , Γ |= CH ψ.
• The case for RH ψ. RG1 · · · RGn RH ψ ∈ Γ
iff Mc |G1 | · · · |Gn |H , Γ |= ψ (IH applies to ψ)
43

iff Mc |G1 | · · · |Gn , Γ |= RH ψ.
C OROLLARY 31. Let Mc = (S, v, V ) be the canonical pseudomodel for cl(α). For all Γ ∈ S and all ϕ ∈ cl(α), it holds thatϕ ∈ Γ iff Mc , Γ |= ϕ.

L EMMA 32 (P SEUDO COMPLETENESS ). Let ϕ be an RCDformula. If ϕ is valid on all pseudo models, then it is provable in
RCD.

From Pseudo Completeness to Completeness.

By using unraveling and folding from [13, pp. 9–15], we cantransform the canonical pseudo model to a bisimilar pre-model andthen to a trans-bisimilar proper model. It remains to show that
this process preserves truth. We will use T to denote the transbisimulation relation.
L EMMA 33 (I NVARIANCE OF TRANS - BISIMULATION ). Let(M, m) be a pointed model, (N, n) a pointed pre-model, and (S, s)a pointed pseudo model. If (M, m) T (N, n)  (S, s), thenM, m |= ϕ iff N, n |=p ϕ for all formulas ϕ.

P ROOF. The lemma can be shown by induction on ϕ. Here we
only show the case for the resolution operators, proofs of othercases are exactly as in the proof of [13, Lemma 26].
Given a pointed model (M, m), a pointed pre-model (N, n) anda pointed pseudo model (S, s), such that Z : (M, m) T (N, n)
for some Z and (N, n)  (S, s), we have the following:M, m |= RG ψ

iff M|G , m |= ψ
iff N|G , n |=p ψ(∗)
iff N, n |=p RG ψ,

where to show (∗) it is sufficient to show that Z : (M|G , m) T(N|G , n), as (∗) is then guaranteed by the induction hypothesis(note that (N|G , n)  (S|G , s) by Proposition 21). Let M =(M, ∼, V ) and N = (N, v, ν).
• The case for (at) holds by Z : (M, m) T (N, n).
• As for (ziggr ), suppose m(∼|G )H m0 for some m0 ∈ M and|H| ≥ 2.
– If G ∩ H = ∅, (∼|G )H =∼H . By Z : (M, m) T (N, n)
there is an n0 ∈ N such that m0 Zn0 and n vH0 · · · vHx n0
with H ⊆ H0 ∩ · · · ∩ Hx . Let H0 = · · · = Hx = H.
Thus n vH · · · vH n0 . Since (v|G )H =vH , it holds thatn(v|G )H · · · (v|G )H n0 , and so (ziggr ) holds in this case.
– If G ∩ H 6= ∅, (∼|G )H =∼G∪H . By Z : (M, m) T (N, n)
there is an n0 ∈ N such that m0 Zn0 and n vH0 · · · vHx n0
with G ∪ H ⊆ H0 ∩ · · · ∩ Hx . Thus n vG∪H · · · vG∪H n0 .
Since (v|G )H =vG∪H , It holds that n(v|G )H · · · (v|G )H n0 .
(ziggr ) holds also in this case.
• The case for (zigag ) is analogous.
• The case for (zag). For all n0 ∈ N and all τ ∈ AG ∪ GR, ifn(v|G )τ n0 , then we must show that there is an m0 ∈ M suchthat m0 Zn0 and m(∼|G )τ m0 .
– If τ is an agent i. Then if i ∈ G, (v|G )i =vG , otherwise(v|G )i =vi . By Z : (M, m) T (N, n), we have m ∼G m0
if i ∈ G, or m ∼i m0 otherwise. Namely m(∼|G )i m0 ineither case.
– If τ is a group H. Then if G ∩ H = ∅, (v|G )H =vH , otherwise (v|G )H =vG∪H . By Z : (M, m) T (N, n), we havem ∼H m0 if G ∩ H = ∅, or m ∼G∪H m0 otherwise. Namelym(∼|G )H m0 in either case.

We have shown that the lemma holds for the case for resolution.
For other cases we refer to the proof of [13, Lemma 26].
T HEOREM 34 (C OMPLETENESS ). For any RCD formula ϕ,
if ϕ is valid then it is provable in RCD.
P ROOF. It suffices to show that any RCD-consistent formula issatisfiable. Let ϕ be consistent. Let Mc be the canonical pseudomodel for cl(ϕ). By the pseudo truth lemma (with n = 0, i.e.,
an empty list of resolution
operators), ϕ is satisfied in a state Γ inc
3
Mc .c Now let NM be the unraveling [13, Definition 18]
of Mc .
MMc ∗N
is a pre-model [13, Proposition 19]. Now let (N ) be theccfolding [13, Definition 22] of NM . (NM )∗ is a (proper) model[13, Definition 22]. From [13, Lemma 27] and [13, Lemma 28] wehave that unraveling preserves bisimulation and that folding preserves trans-bisimulation, in other words we have that4 (M, Γ) ccc
(NM , Γ) T ((NM )∗ , Γ). By Corollary 22, (NM , Γ) |=p ϕ.
c
By Lemma 33, ((NM )∗ , Γ) |= ϕ and we are done.

5.

DISCUSSION

In this paper we captured the dynamics of publicly observableprivate resolution of distributed knowledge. Resolution operators(using update semantics) are both an alternative and a complement
to the standard distributed knowledge operators (which use standard modal semantics).
Resolution operators let us reason about the relationship between
common knowledge and distributed knowledge in general, and inparticular about distributed knowledge as potential common knowledge – when can distributed knowledge become common knowledge? A naive idea would be that DG ϕ should imply that RG CG ϕ
– any information that is distributed can become common knowledge through resolution. This does not hold in general, however,
due to Moore-like phenomena – ϕ might even become false after
resolution (an example is the formula D{1,2} (p ∧ ¬K1 p) discussed
in the introduction). We do, however, have the following (Prop.
6(2) with G = H):
RG CG ϕ ↔ DG RG ϕ.
A fact can become common knowledge after the group have shared
their information if and only if it was distributed knowledge before
the event that the fact would be true after the event. This is exactly
the distributed knowledge that can become common knowledge (inour special case of publicly observable private resolution of distributed knowledge). If the grand coalition resolves its distributed
knowledge, there is no distinction between distributed and common
knowledge any more: RAG CAG ϕ ↔ RAG DAG ϕ (Prop. 5).
As discussed in the introduction, it has been argued that distributed knowledge in general does not comply with the following
principle of full communication [11]: if DG ϕ is true, then ϕ follows logically from the set of all formulas known by at least one
agent in the group. This is seen as a problem: namely that agents
can have distributed knowledge without being able to establish it“through communication” [11]. Several papers [11, 4, 7] have tried
to characterize classes of models on which the principle of fullcommunication does hold – the class of all such models is calledfull communication models [7]. This may seem related to the distinction between distributed knowledge and resolution operators:
the latter is intuitively related to internal “full” communication in
the group. However, this similarity is superficial: the notion of fullcommunication in the sense of [11] is about expressive power ofthe communication language and the limits that puts on the resulting possible epistemic states under certain assumptions about how
information is shared. The key point of the resolution operators,
on the other hand, compared with the standard distributed knowledge operators, is to make a distinction between before and after
the information sharing event. That distinction is not made in standard distributed knowledge – even restricted to full communication
models: it is easy to see that, e.g., D{1,2} (p ∧ ¬K1 p) is satisfiable also on full communication models. The two ideas, of limiting
models to full communication models and of modeling group information sharing events using model updates, are orthogonal, andthere is nothing against restricting logics with the resolution operators to full communication models. We leave that for future work.
Furthermore, it would be interesting to look at a combined variant:“update by full communication”, which takes the communication
language into account when defining the updated model.
A main interest for future work is expressive power. Can it beshown that RCD is strictly more expressive than ELCD? Another, related, natural question is the relative expressivity of RCD
and PACD: can the combination of public announcement operators (which eliminate states) and distributed knowledge operators(which pick out states considered possible by everyone) always be
used to “simulate” the resolution operators (which eliminate statesconsidered possible by everyone)?
Also of interest for future work is to look at other assumptions
about the other agents’ knowledge about the group communication event taking place. In this paper we only studied the case that
it is common knowledge that the event takes place (but not what
the agents in the group learn). That was naturally modeled using
a “global” model update: in every state, replace accessibility foreach agent in the group with the group accessibility (intersection).
An interesting and also natural alternative is doing only a “local”
model update: change accessibility in the same way, but only in thecurrent state. That would correspond to it being common knowledge that if this is the current state, then the group resolves theirknowledge.
When looking at the interaction of the resolution and common
knowledge operators one might be reminded of relativized common
knowledge [9, 10]. Here is an open question: can RG CH ϕ be expressed using relativized common knowledge, in combination withother operators?
Finally, there is a conceptual relationship to group announcement logic [1], where formulas of the form hGiϕ say that G canmake a joint public announcement such that ϕ will become true.
A difference to the resolution operators in this paper is that lattermodel private communication. Yet, the exact relationship between
these operators is interesting for future work.

6.

ACKNOWLEDGMENTSYì N. Wáng acknowledges funding support from the Scientific
Research Foundation for the Returned Overseas Chinese Scholars,State Education Ministry of P.R.C.

7.

REFERENCES[1] T. Ågotnes, P. Balbiani, H. van Ditmarsch, and P. Seban.
Group announcement logic. Journal of Applied Logic,
8(1):62–81, 2010.
[2] A. Baltag, L. Moss, and S. Solecki. The logic of publicannouncements, common knowledge, and private suspicions.
In Proc. of TARK VII, pages 43–56, 1998.
[3] R. Fagin, J. Y. Halpern, Y. Moses, and M. Y. Vardi.
Reasoning about Knowledge. MIT, 1995.

3
While unraveling is a standard general technique; here we mean
unraveling exactly in the sense of the mentioned definition.
4
Here Γ is any path in the unraveling starting with Γ.

44

[4] J. Gerbrandy. Bisimulations on Planet Kripke. Ph.D. thesis,
University of Amsterdam, 1999.
[5] J.-J. C. Meyer and W. van der Hoek. Epistemic Logic for AI
and Computer Science. Cambridge University Press, 1995.
[6] J. A. Plaza. Logics of public communications. In
Proceedings of ISMIS, pages 201–216, 1989.
[7] F. Roelofsen. Distributed knowledge. Journal of Applied
Non-Classical Logics, 16(2):255–273, 2007.
[8] J. Van Benthem. Logical dynamics of information andinteraction. Cambridge University Press, 2011.
[9] J. F. A. K. van Benthem. Information update as relativisation.
Technical report, ILLC, University of Amsterdam, 2000.
[10] J. F. A. K. van Benthem, J. van Eijck, and B. Kooi. Logics ofcommunication and change. Information and Computation,
204(11):1620–1662, 2006.
[11] W. van der Hoek, B. van Linder, and J.-J. Meyer. Groupknowledge is not always distributed (neither is it always
implicit). Mathematical Social Sciences, 38(2):215–240,
1999.
[12] H. van Ditmarsch, W. van der Hoek, and B. Kooi. Dynamic
Epistemic Logic. Springer, 2007.
[13] Y. Wáng and T. Ågotnes. Public announcement logic with
distributed knowledge: Expressivity, completeness andcomplexity. Synthese, 190(1):135–162, 2013.

APPENDIX
Proof of Proposition 6.

1. M, s |= RG CH ϕ iff M|G , s |= CH ϕ iff M|G , t |= ϕ for any
TS00(s, t) ∈∼∗H , where ∼∗H = ( i∈H ∼0i )∗ and ∼0i = j∈G ∼ j fori ∈ G and ∼0i =∼i for i 6∈ G. Thus, when G ∩ H = ∅, we get thatSS
∗0
= ( i∈H ∼i )∗ . M|G , t |= ϕ for anyS(s, t) ∈ ( i∈H ∼i )∗∼H∗holds iff M, t |= RG ϕ for any (s, t) ∈ ( i∈H ∼i ) iff M, t |=CH RG ϕ.
2.
M, s |= RG CH ϕ
iff
M|G , s |= CH ϕ
iff
M|G , t |= ϕ for all t s.t. (s, t) ∈∼G
CH
iff †
iff
iff

M|G , t |= ϕ for all t s.t. (s, t) ∈∼Gi
M|G , s |= Ki ϕ
M, s |= RG Ki ϕ

GFor the † step, note that when i ∈ G, ∼G
i =∼j for any j ∈ G(and actually also equal to ∼G ). Therefore,
[ G ∗
∗
G
∼G
∼i ) = (∼G
CH = (i ) =∼i .
i∈HThat RG Ki ϕ ↔ DG RG ϕ is valid is already shown in Proposition 4.

Proof of Proposition 18.

Let M = (S, v, V ). Clearly M|G = (S, v|G , V ) is a premodel. Moreover,
1. Given an agent i,v{i}∪G i ∈ G(v|G ){i} =v{i} ,i∈/G
vG i ∈ G=
vi , i ∈/G=(v|G )i .

45

2. Given two groups H and H 0 such that H ⊆ H 0 ,
vH 0 ∪G H 0 ∩ G 6= ∅(v|G )H 0 =vH 0 ,
H0 ∩ G = ∅vH∪G H ∩ G 6= ∅(v|G )H =vH ,H ∩G=∅

So we have:• when H ∩ G 6= ∅ (and therefore H 0 ∩ G 6= ∅), (v|G )H 0 =vH 0 ∪G ⊆ vH∪G = (v|G )H ;• when H 0 ∩ G = ∅ (and therefore H ∩ G = ∅), (v|G
)H 0 =vH 0 ⊆vH = (v|G )H ;• otherwise H 0 ∩ G 6= ∅ and H ∩ G = ∅, and in this case(v|G )H 0 = vH 0 ∪G ⊆ vH = (v|G )H .
M|G is a pre-model satisfying the two conditions above, whichshows it is a pseudo model.

Proof of Proposition 21.

Let M = (S, v, V ) and M0 = (S 0 , v0 , V 0 ). Thus M|G =(S, v|G , V ) and M0 |G = (S 0 , v0 |G , V 0 ). Suppose Z : (M, s) 
(M0 , s0 ), and we show Z : (M|G , s)  (M0 |G , s0 ):(at) This clearly follows from the (at) clause of Z : (M, s) 
(M0 , s0 ).
(zig) For all t ∈ S, if s(v|G )H t, then
– If G ∩ H = ∅, then (v|G )H =vH and (v0 |G )H =v0H . By
Z : (M, s)  (M0 , s0 ) there must be a t0 ∈ S 0 such thats0 (v0 |G )H t0 and tZt0 .
– If G∩H 6= ∅, then (v|G )H =vG∪H and (v0 |G )H =v0G∪H .
By Z : (M, s)  (M0 , s0 ) there must be a t0 ∈ S 0 such thats0 (v0 |G )H t0 and tZt0 .
If s(v|G )i t, we can prove analogously to the above.
(zag) This can be shown analogously to the case for (zig).

Proof of Theorem 24.

It is easy to verify that (S5), (CK), (DK), (NR ), (RA), (RC)and (RN) are all valid or admissible with respect to the class ofall pseudo models. Here we only show that i) (RD1) and (RD2)are valid in all pseudo models, and ii) (RRC ) preserves validity ofpseudo models.
Let M = (S, v, V ) be a pseudo model and s ∈ S. We show thefollowing:• M, s |=p RD1 and M, s |=p RD2, i.e.,– If G ∩ H 6= ∅, then M, s |=p RG DH ϕ ↔ DG∪H RG ϕ;– If G ∩ H = ∅, then M, s |=p RG DH ϕ ↔ DH RG ϕ.
M, s |=p RG DH ϕ
iff M|G , s |=p DH ϕ
iff M|G , t |=p ϕ for all t s.t. (s, t) ∈ (v|G )H
iff M, t |=p RG ϕ for all t s.t. (s, t) ∈ (v|G )H
iff † if G ∩ H 6= ∅, M, t |=p RG ϕ for all t s.t. (s, t) ∈v|G∪H ,if G ∩ H = ∅, M, t |=p RG ϕ for all t s.t. (s, t) ∈v|H
iff if G ∩ H 6= ∅, M, t |=p DG∪H RG ϕ, and
if G ∩ H = ∅, M, t |=p DH RG ϕ,
where the † step is by definition:
vH∪G ,(v|G )H =vH ,G ∩ H 6= ∅,G ∩ H = ∅.

• M, s |=p ϕ → RG1 · · · RGn CH ψ under the assumption |=pϕ → (EH ϕ ∧ RG1 · · · RGn ψ). The proof is similar to theproof for genuine models.

Proof of Proposition 27.

Suppose that M = (S, v, V ) is the canonical pseudo model for
cl(α). We need to show that M is a pseudo model. Namely,
1. S is non-empty, and
2. all vi ’s and vG ’s are equivalence relations, and
3. V is a valuation from PROP to ℘(S), and4. vi =v{i} for every agent i, and
5. vH ⊆vG if G and H are groups such that G ⊆ H.
Conditions 1–3 are the conditions for being a pre-model which areeasy to verify. Conditions 4 and 5 are additional conditions forbeing a pseudo model.
By Definition 25(3), Ki ϕ and Di ϕ must be in cl(α) both orneither. Thus, for any Γ, ∆ ∈ S,
iff
iff
iff

Γ vi ∆{Ki ϕ | Ki ϕ ∈ Γ} = {Ki ϕ | Ki ϕ ∈ ∆}{Di ϕ | Di ϕ ∈ Γ} = {Di ϕ | Di ϕ ∈ ∆} (Axiom DK1)
Γ v{i} ∆.

Γ vH ∆{DH 0 ϕ | DH 0 ϕ ∈ Γ} = {DH 0 ϕ | DH 0 ϕ ∈ ∆},for any group H 0 ⊆ H⇒ {DG0 ϕ | DG0 ϕ ∈ Γ} = {DG0 ϕ | DG0 ϕ ∈ ∆},for any group G0 ⊆ G
iff Γ vG ∆.

iff

This finishes the proof, and shows that the notion “canonical pseudomodel” is well-defined.

Proof of Lemma 28(2). W

Let ϕ ∈ cl(α). By ` ¬ϕ∈Γ∈S Γ → ¬ϕ and the first reW
W
W
sult (i.e., ` ¬ϕ∈Γ∈S ∨ ϕ∈Γ∈S ) we get ` ϕ → ϕ∈Γ∈S Γ.
W
For the converse direction, suppose 0 ϕ∈Γ∈S Γ → ϕ. ThenW
W¬( ϕ∈Γ∈S Γ → ϕ) is consistent. Namely ¬ϕ ∧ ϕ∈Γ∈S Γ isconsistent. But this is impossible.

we get ` RG1 · · · RGn CH ϕ → Dδ RG1 · · · RGn CH ϕ. HenceΓn ` Dδ RG1 · · · RGn CH ϕ. As Dδ RG1 · · · RGn CH ϕ ∈ cl(α),
we have Dδ RG1 · · · RGn CH ϕ ∈ Γn . Moreover, by Proposition
19, i =vδ . Thus Dδ RG1 · · · RGn CH ϕ ∈ Γn+1 by the definition of vδ , and so RG1 · · · RGn CH ϕ ∈ Γn+1 .
• Suppose τn is a group I (I ⊆ H). By Axioms C1, D1 andD2 we have ` CH ϕ → DI CH ϕ. By NR and RG -distribution,` RG1 · · · RGn CH ϕ → RG1 · · · RGn DI CH ϕ. By similar reasoning to the case above, we get the result RG1 · · · RGn CH ϕ ∈Γn+1 (we use δ(H, G1 , . . . , Gn ) instead of δ({i}, G1 , . . . , Gn )in this case).
In both cases we get RG1 · · · RGn CH ϕ ∈ Γn+1 as we wish to
show.
From right to left. Suppose that every hG1 · · · Gn i-resolved Hpath from Γ is a canonical RG1 · · · RGn ϕ-path. Let S0 be the set of
all maximal consistent sets ∆ in cl(α) such that every hG1 · · · Gn iresolved H-path from ∆ is a canonical RG1 · · · RGn ϕ-path. Now
consider the formula
_
λ=
∆
∆∈S0

We will show the following:
1. ` Γ → λ2. ` λ → (EH λ ∧ RG1 · · · RGn ϕ).
From the above and the reduction rule for resolved common knowledge we get ` Γ → RG1 · · · RGn CH ϕ which furthermore entails
RG1 · · · RGn CH ϕ ∈ Γ. We now continue with the proof of thetwo clauses.
1. This is trivial, as Γ is one of the disjuncts of λ.
2. Suppose towards a contradiction thatλ ∧ ¬(EH λ ∧ RG1 · · · RGn ϕ)is consistent, i.e., λ ∧ (¬EH λ ∨ ¬RG1 · · · RGn ϕ) is consistent.
Because λ is a disjunction there must be a disjunct Ξ of λ suchthat Ξ ∧ (¬EH λ ∨ ¬RG1 · · · RGn ϕ) is consistent. It follows
that either Ξ ∧ ¬EH λ or Ξ ∧ ¬RG1 · · · RGn ϕ is consistent.
If the former is consistent, then there must be an agenti ∈W
H such that Ξ ∧ ¬Ki λ is consistent, i.e., Ξ ∧ K̂i ¬ ∆∈S0 ∆W
is W
consistent. Since
W ` ∆∈S0 ∆ by Lemma 28, we have `¬ ∆∈S0 ∆ → ∆0 ∈S\S0 ∆ , and so there must be a Θ in

Proof of Lemma 29(8).

Let RG1 · · · RGn CH ϕ ∈ cl(α). It follows from the definitionof closure (Definition 25) that the following formulas:• Dδ({i},G1 ,...,Gn ) RG1 · · · RGn CH ϕ where i ∈ H• Dδ(H,G1 ,...,Gn ) RG1 · · · RGn CH ϕ
• RG1 · · · RGn ϕ and ¬RG1 · · · RGn ϕ
are all in cl(α).
From left to right. Suppose RG1 · · · RGn CH ϕ ∈ Γ, we continue by induction on the length of the path that every hG1 · · · Gn iresolved H-path from Γ is a canonical RG1 · · · RGn CH ϕ-path.
Then the left-to-right direction follows: by ` CH ϕ → ϕ, NR
and RG -distribution (which follows from RR axioms) we get `RG1 · · · RGn CH ϕ → RG1 · · · RGn ϕ, and by RG1 · · · RGn ϕ ∈cl(α) we have RG1 · · · RGn ϕ ∈ Γ.
Suppose the length of the hG1 · · · Gn i-resolved H-path is 0, i.e.,
the path is hΓi, we must show that RG1 · · · RGn CH ϕ ∈ Γ. This isguaranteed by the supposition.
Suppose the length of the hG1 · · · Gn i-resolved H-path is n + 1,
i.e., the path is hΓ0 τ0 · · · τn−1 Γn τn Γn+1 i with Γ0 = Γand every τx is either in H or a subset of H. By the induction
hypothesis we may assume that RG1 · · · RGn CH ϕ ∈ Γn .
• Suppose τn is an agent i (i ∈ H). By Axiom C1 we have
` CH ϕ → Ki CH ϕ. It follows that ` RG1 · · · RGn CH ϕ →RG1 · · · RGn Ki CH ϕ by the rules NR and RG -distribution. Letδ = δ({i}, G1 , . . . , Gn ). By the reduction axioms we move Ki
left, i.e., ` RG1 · · · RGn Ki CH ϕ → Dδ RG1 · · · RGn CH ϕ, so

S \ S0 such that Ξ ∧ K̂i Θ is consistent. By item 4 of this lemmaΞ vi Θ (where v is the relation in the canonical pseudo model
for cl(α)). But then Ξ cannot be in S0 for Θ ∈/ S0 . A contradiction!
If the latter is consistent, since ¬RG1 · · · RGn ϕ ∈ cl(α) andΞ is maximal, ¬RG1 · · · RGn ϕ ∈ Ξ. But RG1 · · · RGn ϕ ∈ Ξsince Ξ vH Ξ and every hG1 · · · Gn i-resolved H-path from Ξ
is a canonical RG1 · · · RGn ϕ-path. We reach a contradiction.

46

Epistemic Protocols for Distributed Gossiping
Krzysztof R. Apt

Centrum Wiskunde & Informatica
Amsterdam, The Netherlands

k.r.apt@cwi.nl

Davide Grossi

Wiebe van der Hoek

d.grossi@liverpool.ac.uk

wiebe@liv.ac.uk

University of Liverpool
Liverpool, UK

ABSTRACT
Gossip protocols aim at arriving, by means of point-to-point
or group communications, at a situation in which all theagents know each other’s secrets. We consider distributed
gossip protocols which are expressed by means of epistemic
logic. We provide an operational semantics of such protocols
and set up an appropriate framework to argue about theircorrectness. Then we analyze specific protocols for completegraphs and for directed rings.

Keywords
Gossip protocols, epistemic logic, distributed computing,knowledge-based programs

1. INTRODUCTION
In the gossip problem ([18, 4], see also [10] for an overview)
a number n of agents, each one knowing a piece of information (a secret) unknown to the others, communicate by
one-to-one interactions (e.g., telephone calls). The result ofeach call is that the two agents involved in it learn all secrets
the other agent knows at the time of the call. The problem
consists in finding a sequence of calls which disseminates all
the secrets among the agents in the group. It sparked alarge literature in the 70s and 80s [18, 4, 9, 5, 17] typicallyfocusing on establishing—in the above and other variants ofthe problem—the minimum number of calls to achieve dissemination of all the secrets. This number has been provento be 2n − 4, where n, the number of agents, is at least 4.
The above literature assumes a centralized perspective on
the gossip problem: a planner schedules agents’ calls. In
this paper we pursue a line of research first put forth in [3]by developing a decentralized theory of the gossip problem,
where agents perform calls not according to a centralized
schedule, but following individual epistemic protocols they
run in a distributed fashion. These protocols tell the agents
which calls to execute depending on what they know, or
do not know, about the information state of the agents in
the group. We call the resulting distributed programs (epistemic) gossip protocols.

Contribution of the paper and outline.
The paper introduces a formal framework for specifying epistemic gossip protocols and for studying their computationsin terms of correctness, termination, and fair termination(Section 2). It then defines and studies two natural protocols in which the interactions are unconstrained (Section 3)
and four example gossip protocols in which agents are posi-

47

University of Liverpool
Liverpool, UK

tioned on a directed ring and calls can happen only between
neighbours (Section 4). Proofs are collected in the appendix.
From a methodological point of view, the paper integrates
concepts and techniques from the distributed computing,see, e.g., [1, Chapter 11] and the epistemic logic literature[8, 15] in the tradition of [16, 14, 7].

2.

GOSSIP PROTOCOLS

We introduce first the syntax and semantics of gossip protocols.

2.1

Syntax

We loosely use the syntax of the language CSP (Communicating Sequential Processes) of [11] that extends the guardedcommand language of [6] by disjoint parallel compositionand commands for synchronous communication. CSP was
realized in the distributed programming language OCCAM(see INMOS [12]).
The main difference is that we use as guards epistemic
formulas and as communication primitives calls that do not
require synchronization. Also, the syntax of our distributed
programs is very limited. In order to define gossip protocols
we introduce in turn calls and epistemic guards.
Throughout the paper we assume a fixed finite set A ofat least three agents. We assume that each agent holds
exactly one secret and that there exists a bijection between
the set of agents and the set of secrets. We denote by P
the set of all secrets (for propositions). Furthermore, it isassumed that each secret carries information identifying theagent to whom that secret belongs.

2.1.1 Calls
Each call concerns two agents, the caller (a below) andthe agent called (b). We distinguish three modes of communication of a call:
push-pull , written as ab or (a, b). During this call thecaller and the called agent learn each other’s secrets,
push, written as a.b. After this call the called agent learns
all the secrets held by the caller,
pull , written as a / b. After this call the caller learns all
the secrets held by the called agent.
Variables for calls are denoted by c, d. Abusing notation wewrite a ∈ c to denote that agent a is one of the two agents
involved in the call c (e.g., for c := ab we have a ∈ c andb ∈ c). Calls in which agent a is involved are denoted by ca .

2.1.2

Epistemic guards

Epistemic guards are defined as formulas in a simple modal
language with the following grammar:
φ ::= Fa p | ¬φ | φ ∧ φ | Ka φ,where p ∈ P and a ∈ A. Each secret is viewed as a distinct
symbol. We denote the secret of agent a by A, the secretof agent b by B and so on. We denote the set of so defined
formulas by L and we refer to its members as epistemic
formulas or epistemic guards. We read Fa p as ‘agent a isfamiliar with the secret p’ (or ‘p belongs to the set of secrets
a knows about’) and Ka φ as ‘agent a knows that formula φ istrue’. So this language is an epistemic language where atoms
consist of ‘knowing whether’ statements about propositional
atoms, if we view secrets as Boolean variables.
Atomic expressions in L concern only who knows whatsecrets. As a consequence the language cannot express formally the truth of a secret p. This level of abstraction suffices for the purposes of the current paper. However, expressions Fa p could be given a more explicit epistemic readingin terms of ‘knowing whether’. That is, ‘a is familiar with p’
can be interpreted (on a suitable Kripke model) as ‘a knows
whether the secret p is true or not’. This link is establishedin [3].

2.1.3

Gossip protocols

Before specifying what a program for agent a is, let us
first define the language La with the following grammar:
ψ ::= Ka φ | ¬ψ | ψ ∧ ψ
1with φ ∈ L.
By a component program, in short a program, for an
agent a we mean a statement of the form∗[[]mj=1 ψj → cj ],where m > 0 and each ψj → cj is such that ψj ∈ La and ais the caller in cj .
Given an epistemic formula ψ ∈ La and a call c, we callthe construct ψ → c a rule and refer in this context to ψ asa guard .
We denote the set of rules {ψ1 → c1 , . . ., ψk → ck } as[[]kj=1 ψj → cj ] and abbreviate a set of rules {ψ1 → c, . . ., ψk →Wc} with the same call to a single rule ki=1 ψi → c.
Intuitively, ∗ denotes a repeated execution of the rules, one
at a time, where each time a rule is selected whose guard istrue.
Finally, by a distributed epistemic gossip protocol , inshort a gossip protocol , we mean a parallel compositionof component programs, one for each agent. In order not
to complicate matters we assume that each gossip protocol
uses only one mode of communication.
Of special interest for this paper are gossip protocols thatare symmetric. By this we mean that the protocol is a composition of the component programs that are identical modulo the names of the agents. Formally, consider a statement
π(x), where x is a variable ranging over the set A of agentsand such that for each agent a ∈ A, π(a) is a component
program for agent a. Then the parallel composition of the1
Alternatively, La could be defined as the fragment of L
consisting of the formulae of form Ka ψ. In logic S5, it iseasy to prove that each ψ ∈ La is logically equivalent to a
formula Ka φ ∈ L.

π(a) programs, where a ∈ A, is called a symmetric gossip
protocol .
Gossip protocols are syntactically extremely simple. Therefore it would seem that little can be expressed using them.
However, this is not the case. In Sections 3 and 4 we consider
gossip protocols that can exhibit complex behaviour.

2.2

Semantics

We now move on to provide a formal semantics of epistemic guards, and then describe the computations of gossip
protocols.

2.2.1

Gossip situations and calls

A gossip situation is a sequence s = (Qa )a∈A , where
Qa ⊆ P for each agent a. Intuitively, Qa is the set of secrets a
is familiar with in situation s. The initial gossip situationis the one in which each Qa equals {A} and is denoted byroot. The set of all gossip situations is denoted by S. We
say that an agent a is an expert in a gossip situation s if
he is familiar in s with all the secrets, i.e., if Qa = P. The
initial gossip situation reflects the fact that initially eachagent is familiar only with his own secret, although it is
not assumed this is common knowledge among the agents.
In fact, in the introduced language we have no means to
express the concept of common knowledge.
We will use the following concise notation for gossip situations. Sets of secrets will be written down as lists. e.g., theset {A, B, C} will be written as ABC. Gossip situations will
be written down as lists of lists of secrets separated by dots.
E.g., if there are three agents, root = A.B.C and the situation ({A, B}, {A, B}, {C}) will be written as AB.AB.C.
Each call transforms the current gossip situation by modifying the set of secrets the agents involved in the call arefamiliar with. More precisely, the application of a call to asituation is defined as follows.
Definition 2.1 (Effects of calls). A call is a function c : S −→ S, so defined, for s := (Qa )a∈A :c = ab c(s) = (Q0a )a∈A , where Q0a = Q0b = Qa ∪ Qb , Q0c =Qc , for c 6= a, b;
c = a . b c(s) = (Q0a )a∈A , where Q0b = Qa ∪ Qb , Q0a = Qa ,Q0c = Qc , for c 6= a, b;
c = a / b c(s) = (Q0a )a∈A , where Q0a = Qa ∪ Qb , Q0b = Qb ,Q0c = Qc , for c 6= a, b.
The definition formalizes the modes of communications we
introduced earlier. Depending on the mode, secrets are either shared between caller and callee (ab), they are pushedfrom the caller to the callee (a . b), or they are retrieved by
the caller from the callee (a / b).

2.2.2

Call sequences

A call sequence is a (possibly infinite) sequence of calls,
in symbols (c1 , c2 , . . . , cn , . . .), all being of the same communication mode. The empty sequence is denoted by . We
use c to denote a call sequence and C to denote the set of all
call sequences. The set of all finite call sequences is denotedC<ω . Given a finite call sequence c and a call c we denote byc.c the prepending of c with c, and by c.c the postpendingof c with c.

48

The result of applying a call sequence to a situation s isdefined by induction using Definition 2.1, as follows:[Base] (s) := s,[Step] (c.c)(s) := c(c(s)).
Example 2.2. Let the set of agents be {a, b, c}.
ab
A.B.C

ca
AB.AB.Cab
ABC.AB.ABC

ABC.ABC.ABC

The top row lists the call sequence (ab, ca, ab), while the bottom row lists the successive gossip situations obtained fromthe initial situation A.B.C by applying the calls in the sequence: first ab, then ca and finally ab.
2
By applying an infinite call sequence c = (c1 , c2 , . . . , cn , . . .)to a gossip situation s one obtains therefore an infinite sequence c0 (s), c1 (s), . . . , cn (s), . . . of gossip situations, where
each ck is sequence c1 , c2 , . . . , ck . A call sequence c is said
to converge if for all input gossip situations s the generated
sequence of gossip situations reaches a limit, that is, there
exists n < ω such that for all m ≥ n cm (s) = cm+1 (s). Since
the set of secrets is finite and calls never make agents forgetsecrets they are familiar with, it is easy to see the following.
Fact 2.3. All infinite call sequences converge.

guards of gossip protocols. Since such guards are relevant
only after finite sequences of calls, the domain of a gossip
model is taken to consist only of finite sequences. Intuitively,
those are the finite sequences that can be generated by agossip protocol. Let us turn now to the ∼a relation. This isdefined with the following intuitions in mind. First of all, no
agent can distinguish the empty call sequence from itself—
this is the base of the induction. Next, if two call sequences
are indistinguishable for a, then the same is the case if (i)we extend one of these sequences by a call in which a is not
involved or if (ii) we extend each of these sequences by a callof a with the same agent (agent a may be the caller or thecallee), provided a is familiar with exactly the same secrets
after each of the new sequences has taken place—this is theinduction step.2
The above intuitions are based on the following assumptions on the form of communication we presuppose: (i) At
the initial situation, as communication starts, each agent
knows only her own secret but considers it possible that theothers may be familiar with all other secrets. In other words
there is no such thing as common knowledge of the fact that‘everybody knows exactly her own secret’. (ii) In general,
each agent always considers it possible that call sequences(of any length) take place that do not involve her. Theseassumptions are weaker than the ones analyzed in [3].
We state without proof the following simple fact.

However, as we shall see, this does not imply that all gossip protocols terminate. In the remainder of the paper, unless stated otherwise, we will assume the push-pull mode ofcommunication. The reader can easily adapt our presentation to the other modes.

2.2.3

Fact 2.5.
(i) Each ∼a is an equivalence relation;(ii) For all c, d ∈ C if c ∼a d, then c(root)a = d(root)a ,but not vice versa.

Gossip models

The set S of all gossip situations is the set of all possiblecombinations of secret distributions among the agents. As
calls progress in sequence from the initial situation, agents
may be uncertain about which one of such secrets distributions is the actual one. This uncertainty is precisely theobject of the epistemic language for guards we introduced
earlier.
Definition 2.4. A gossip model (for a given set A) is atuple M = (C<ω , {∼a }a∈A ), where each ∼a ⊆ C<ω × C<ω isthe smallest relation satisfying the following inductive conditions (assume the mode of communication is push-pull):[Base]  ∼a ;[Step] Suppose c ∼a d.
(i) If a 6∈ c, then c.c ∼a d and c ∼a d.c.

(ii) If there exists b ∈ A and c, d ∈ {ab, ba} such thatc.c(root)a = d.d(root)a , then c.c ∼a d.d.
A gossip model with a designated finite call sequence is calleda pointed gossip model.
For the push, respectively pull, modes of communication
clause (ii) needs to be modified by requiring that for someb ∈ A, c = d = a . b or c = d = a / b, respectively.
For instance, by (i) we have ab, bc ∼a ab, bd. But we
do not have bc, ab ∼a bd, ab since (bc, ab)(root)a = ABC 6=
ABD = (bd, ab)(root)a .
Let us flesh out the intuitions behind the above definition.
Gossip models are needed in order to interpret the epistemic

49

This prompts us to note also that according to Definition 2.4
sequences which make a learn the same set of secrets may
well be distinguishable for a, such as, for instance, ab, bc, aband ab, bc, ac. In the first one a comes to know that b knows
a is familiar with all secrets, while in the second one, shecomes to know that c knows a is familiar with all secrets.
Relation ∼a is so defined as to capture this sort of ‘higherorder’ knowledge.

2.2.4

Truth conditions for epistemic guards

Everything is now in place to define the truth of the considered formulas.
Definition 2.6. Let (M, c) be a pointed gossip model with
M = (C<ω , (∼a )a∈A ) and c ∈ C<ω . We define the satisfaction relation |= inductively as follows (clauses for Boolean
connectives are omitted):(M, c) |= Fa p

(M, c) |= Ka φiff p ∈ c(root)a ,iff ∀d s.t. c ∼a d, (M, d) |= φ.

So formula Fa p is true (in a pointed gossip model) wheneversecret p belongs to the set of secrets agent a is familiar within the situation generated by the designated call sequence c
applied to the initial situation root. The knowledge operator is interpreted as customary in epistemic logic using theequivalence relations ∼a .
2

Notice that the definition requires a designated initial situation, which we assume to be root.

2.2.5Computations

Assume a gossip protocol P that is a parallel compositionaaaof the component programs ∗[[]m
j=1 ψj → cj ], one for eachagent a ∈ A.
Given the gossip model M = (C<ω , {∼a }a∈A ) we define
the computation tree CP ⊆ C<ω of P as the smallest setof sequences satisfying the following inductive conditions:[Base]  ∈ CP ;[Step] If c ∈ CP and (M, c) |= ψja then c.caj ∈ CP . In thiscase we say that a transition has taken place betweenc and c.caj , in symbols, c → c.caj .
So CP is a (possibly infinite) set of finite call sequences thatis iteratively obtained by performing a ‘legal’ call (according
to protocol P ) from a ‘legal’ (according to protocol P ) callsequence.
A path in the computation tree of P is a (possibly infinite)sequence of elements of CP , denoted by ξ = (c0 , c1 , . . . , cn , . . .),where c0 =  and each ci+1 = ci .c for some call c and i ≥ 0.
A computation of P is a maximal rooted path in the computation tree of P .3
The above definition implies that a call sequence c is aleaf of the computation tree if and only if(M, c) |=^a^ ma∈A j=1¬ψja .

We call the formula^a^ ma∈A j=1¬ψjathe exit condition of the gossip protocol P .
Obviously computation trees can be infinite, though they
are always finitely branching. Further, note that this semantics for gossip protocols abstracts away from some implementation details of the calls. More specifically, we assume thatthe caller always succeeds in his call and does not require
to synchronize with the called agent. In reality, the calledagent might be busy, being engaged in another call. To takecare of this one could modify each call by replacing it bya ‘call protocol’ that implements the actual call using somelower level primitives. We do not elaborate further on this
topic.
P
Let us fix some more terminology.
Wma Forac ∈ C , an agent
a is enabled in c if (M, c) |= j=1 ψj and is disabled
otherwise. So an agent is enabled if it can perform a call.
An agent a is selected in c if it is the caller in the call thatfor some c0 determines the transition c → c0 in ξ. Finally, acomputation ξ is called a fair computation if it is finite oreach agent that is enabled in infinitely many sequences in ξ
is selected in infinitely many sequences in ξ.
We note in passing that various alternative definitions offairness are possible; we just focus on one of them. An interested reader may consult [2], where several fairness definitions (for instance one focusing on actions and not on agents)
for distributed programs were considered and compared.
3
Note that while the sequences that are elements of the computation tree of a protocol are always finite (although possibly infinite in number), computations can be infinite sequences (of finite call sequences).

We conclude this section by observing the following. Ourdefinition of computation tree for protocol P presupposesthat guards ψja are interpreted over the gossip model M =(C<ω , {∼a }a∈A ). This means that when evaluating guards,
agents consider as possible call sequences that cannot be
generated by P . In other words, agents do not know the
protocol. To model common knowledge of the considered
protocol in the gossip model one should take as the domainof the gossip model M the underlying computation tree.
However, the computation tree is defined by means of theunderlying gossip model. To handle such a circularity an
appropriate fixpoint definition is needed. We leave this topic
for future work.

2.3

Correctness

We are interested in proving the correctness of gossip protocols. Assume a gossip protocol P that is a parallel comaaaposition of the component programs ∗[[]m
j=1 ψj → cj ].
We say that P is partially correct, in short correct, if in
all situations sequences c that are leaves of the computation
tree of P , for each agent a^(M, c) |=
Fa B,
b∈A

i.e., if for all situations sequences c that are leaves of thecomputation tree of P , each agent is an expert in the gossip
situation c(root).
We say furthermore that P terminates if all its computations are finite and that P fairly terminates if all its faircomputations are finite.
In the next section we provide examples showing that partial correctness and termination of the considered protocols
can depend on the assumed mode of communication andon the number of agents. In what follows we study various
gossip protocols and their correctness. We begin with thefollowing obvious observation.
Fact 2.7. For each protocol P the following implications(⇒) hold, where TP (x) stands for its termination and F TP (x)for its fair termination in a communication mode x:
TP (x) ⇒ F TP (x).
Protocol R3 given in Section 4 shows that none of these
implications can be reversed. Moreover, it is not the caseeither that for each protocol P :TP (.) ⇒ TP (push-pull),
TP (/) ⇒ TP (push-pull).
Example 2.8. Let A = {a, b, c} and define the following
expression:^_
A ⊂ C :=
(Fa I → Fc I) ∧(Fc I ∧ ¬Fa I)
I∈{A,B,C}

I∈{A,B,C}

Expression B ⊂ C can be defined analogously. Note that
we denote by I the secret of agent i. Intuitively, A ⊂ C
means that agent c is familiar with all the secrets that agent
a is familiar with, but not vice versa. So c is familiar witha supersetVof the secrets a is aware of. Further, let Expj
stand for I∈{A,B,C} Fj I.
Consider now the following component programs:• for agent a: ∗[Ka (¬(A ⊂ C) ∧ ¬Expa ) → a . c],50

• for agent b: ∗[Kb (¬(B ⊂ C) ∧ ¬Expb ) → b . c],• for agent c: ∗[[]i∈{a,b} Kc Expc ∧ ¬Kc Expi → c . i].
First note that in our logic, Ki (φ1 ∧ φ2 ) is equivalent toKi φ1 ∧ Ki φ2 .
This protocol is correct. Indeed, initially, it is not the casethat c knows to be an expert, hence the guard of c is false.
Likewise, the guards of a and b are true; a for instance knowsthat c is not familiar with more secrets than a, and that a
is not familiar with all secrets. So initially both a or b are
enabled. If the first call is granted to a, this agent will call c,yielding the situation A.B.AC. Note that now, the guard of
a is false (from a’s perspective, c may now well be familiar
with all secrets), the guard of b is true, and the guard ofc is still false. So now only b is enabled, which yields thesituation A.B.ABC. At this stage, only agent c is enabled
and after he calls both a and b all guards become false.
Moreover, this protocol terminates. Indeed, the only computations are the ones in which first the calls a . c and b . ctake place, in any order, followed by the calls c . a and c . b,
also performed in any order. However, if we use the pushpull mode instead of push, the call ac can be indefinitely
repeated, so the protocol does not terminate.
2

3. TWO SYMMETRIC PROTOCOLSIn this section we consider protocols for the case when theagents form a complete graph. We study two protocols. We
present them first for the communication mode push-pull.
(Partial) correctness of the considered protocols does notdepend on the assumed mode of communication.

Learn new secrets protocol (LNS).
Consider the following program for agent i:∗[[]j∈A ¬Fi J → (i, j)].
Informally, agent i calls agent j if i is not familiar with j’s
secret. Note that the guards of this protocol do not use theepistemic operator Ki , but they are equivalent to the ones
that do, as ¬Fi J is equivalent to Ki ¬Fi J.
This protocol was introduced in [3] and studied with respect to the push-pull mode, assuming asynchronous communication. As noted there this protocol is clearly correct.
Also, it always terminates since after each call (i, j) the sizeof {(i, j) ∈ A × A | ¬Fi J} decreases. The same argumentshows termination if the communication mode is pull.
However, if the communication mode is push, the protocol
may fail to terminate, even fairly. To see it fix an agent aand consider a sequence of calls in which each agent calls a.
At the end of this sequence a becomes an expert but nobody
is familiar with his secret. So any extension of this sequenceis an infinite computation.
Let us consider now the possible call sequences generated
by the computations of this protocol. Assume that there aren ≥ 4 agents. By the result mentioned in the introductionin each terminating computation at least 2n − 4 calls are
made.
The LNS protocol can generate such shortest sequences(among others). Indeed, let A = {a, b, c, d, i1 , . . ., in−4 } bethe set of agents. Then the following sequence of 2n − 4 calls(a, i1 ), (a, i2 ), . . ., (a, in−4 ),(a, b), (c, d), (a, c), (b, d),(i1 , b), (i2 , b), . . ., (in−4 , b)

(1)

51

corresponds to a terminating computation.
The guards used in this protocol entail that after a call(i, j) neither the call (j, i) nor another call (i, j) can takeplace, that is between each pair of agents at most one call
can take place. Consequently, the longest possible sequencecalls. Such a worst case can becontains at most n(n−1)
2generated by means of the following sequence of calls:[2], [3], [4], . . ., [n],where for a natural number k, [k] stands for the sequence(1, k), (2, k), . . ., (k − 1, k).4

Hear my secret protocol (HMS).
Next, we consider a protocol with the following program foragent i:∗[[]j∈A ¬Ki Fj I → (i, j)].
Informally, agent i calls agent j if he (agent i) does not knowwhether j is familiar with his secret. To prove correctnessof this protocol it suffices to note that its exit condition^
Ki Fj Ii,j∈AV

implies i,j∈A Fj I. To prove termination it suffices to notethat after each call (i, j) the size of the set {(i, j) | ¬Ki Fj I}
decreases.
If the communication mode is push, then the terminationargument remains valid, since after the call i . j agent j stilllearns all the secrets agent i is familiar with.
However, if the communication mode is pull, then the protocol may fail to terminate, even fairly. To see it fix an agent
j and consider the calls i / j, where i ranges over A \ {j},arbitrarily ordered. Denote this sequence by c. Consider
now an infinite sequence of calls resulting from repeating c
indefinitely. It is straightforward to check that such a sequence corresponds to a possible computation. Indeed, in
this sequence agent j never calls and hence never learns any
new secret. So for each i 6= j the formula ¬Ki Fj I remains
true and hence each agent i 6= j remains enabled. Moreover,after the calls from c took place agent j is not anymore
enabled. Hence the resulting infinite computation is fair.
When there are n ≥ 4 agents, the extreme cases in termsof the lengths of possible call sequences are the same as in thecase of the LNS protocol. Indeed, let A = {a, b, c, d, i1 , . . ., in−4 }be the set of agents. Then the sequence of (1) corresponds
to a terminating computation. Further, this protocol cancalls are made. The
generate computations in which n(n−1)
2argument is the same as for the LNS protocol.

4.

PROTOCOLS OVER DIRECTED RINGSIn this section we consider the case when the agents arearranged in a directed ring, where n ≥ 3. For convenience wetake the set of agents to be {1, 2, . . ., n}. For i ∈ {1, . . ., n},
let i ⊕ 1 and i 1 denote respectively the successor andpredecessor of agent i. That is, for i ∈ {1, . . ., n − 1}, i ⊕1 = i + 1, n ⊕ 1 = 1, for i ∈ {2, . . ., n}, i 1 = i − 1,
and 1 1 = n. For k > 1 we define i ⊕ k and i k byinduction in the expected way. Again, when reasoning about
the protocols we denote the secret of agent i ∈ {1, . . ., n} by
4Other longest sequences are obviously possible, for instance: 12, 13, ..., 1n, 23, 24, ..., 2n, 34, 35, .., 3n, ..., (n − 1)n.

I. We consider four different protocols and study them withrespect to their correctness and (fair) termination.
In this set up, a call sequence over a directed ring is a(possibly infinite) sequence of calls, all being of the samecommunication mode, and all involving an agent i and i ⊕ 1.
As before, we use c to denote such a call sequence and CDR
to denote the set of all call sequences over a directed ring.
In this section, unless stated otherwise, by a call sequence
we mean a sequence over a directed ring. The set of all such
finite call sequences is denoted C<ω
DR . A gossip model fora directed ring is a tuple MDR = (C<ωDR , {∼a }a∈A ), where<ω
each ∼a ⊆ C<ω
×
C
is
as
in
Definition
2.4. The truth
DRDR
definition is as before, and the notion of a computation<ωtree for directed rings CP
DR ⊆ CDR of a ring protocol
P is analogous to the notion defined before. Note that by
restricting the domain in MDR to C<ωDR , the ring network—and hence who is the successor of whom—becomes common
knowledge.
When presenting the protocols we use the fact that Fi J
is equivalent to Ki Fi J.

Ring protocol R1.
Consider first a gossip protocol with the following programfor i:n_∗[ (Fi J ∧ Ki ¬Fi⊕1 J) → i3i ⊕ 1],
j=1

where 3 denotes the mode of communication, so ., / orpush-pull.
Informally, agent i calls his successor, agent i ⊕ 1, if i isfamiliar with some secret and he knows that his successor is
not familiar with it.
Proposition 4.1. Let 3 = .. Protocol R1 terminates
and is correct.
Termination and correctness do not both hold for theother communication modes. Consider first the pull communication mode, i.e., 3 = /. Then the protocol does notalways terminate. Indeed, each call i / i ⊕ 1 can be repeated.
Next, consider the push-pull communication mode. We showthat then the protocol is not correct. Indeed, takec = (1, 2), (2, 3), . . ., (n − 1, n).
We claim that after the sequence of calls c the exit conditionof the protocol is true. To this end we consider each agent
in turn.
After c each agent i, where i 6= n is familiar the secretsof the agents 1, 2, . . ., i + 1. Moreover, because of the call(i, i + 1) agent i knows that agent i + 1 is familiar with these
secrets. So the exit condition of agent i is true.
To deal with agent n note that c ∼n c.(n − 2, n − 1).(n −3, n − 2).. . .(2, 3).(1, 2). After the latter call sequence agent
1 becomes an expert. So after c agent n cannot know thatagent 1 is not familiar with some secret. Consequently, afterc the exit condition of agent n is true, as well. However,after c agent 1 is not an expert, so the protocol is indeed
not correct.
In what follows we initially present the protocols assuming
the push-pull mode of communication.

Ring protocol R2.
Consider now a gossip protocol with the following programfor agent i:∗[¬Ki Fi⊕1 I

1 → (i, i ⊕ 1)],

where (recall) I 1 denotes the secret of agent i 1. Informally, agent i calls his successor, agent i ⊕ 1, if i does
not know that his successor is familiar with the secret of i’spredecessor, i.e., agent i 1.
Proposition 4.2. If |A| ∈ {3, 4} then protocol R2 is correct.
However, this protocol is not correct for five or more agents.
To see it consider the sequence of calls(1, 2), (2, 3), . . ., (n − 1, n), (n, 1), (1, 2)
where n ≥ 5. After it the exit condition of the protocol istrue. However, agent 3 is not familiar with the secret of
agent 5.
Note that the same argument shows that the protocol inwhich we use ¬Ki Fi⊕1 I∨¬Ki Fi⊕1 I 1 instead of ¬Ki Fi⊕1 I
1 is incorrect, as well.
Moreover, this protocol does not always terminate. Indeed, one possible computation consists of an agent i repeatedly calling his successor i ⊕ 1.

Ring protocol R3.
Next, consider the following modification of protocol R2 inwhich we use the following program for agent i:
∗[(¬

n
^

Fi J) ∨ ¬Ki Fi⊕1 I

j=1

1 → (i, i ⊕ 1)].

Informally, agent i calls his successor, agent i ⊕ 1, if i is
not familiar with all the secrets or i does not know that his
successor is familiar with the secret of his predecessor, agent
i 1.
This
V gossip
Vn protocol is obviously correct thanks to the factthat ni=1
j=1 Fi J is part of the exit condition. However,it does not always terminate for the same reason as the previous one.
On the other hand, the following holds.
Proposition 4.3. Protocol R3 fairly terminates.
The same conclusions concerning non termination and fairtermination can be drawn for the push and the pull modes ofcommunication. Indeed, for push it suffices to consider thesequence of calls i . i ⊕ 1, i ⊕ 1 . i ⊕ 2, . . ., i 1 . i after which
agent i 1 becomes disabled, and for pull the sequence of
calls i / i ⊕ 1, i 1 / i, . . ., i ⊕ 2 / i ⊕ 3 after which agent i ⊕ 2
becomes disabled.

Ring protocol R4.
Finally, we consider a protocol that is both correct and terminates for the push-pull mode. Consider the following program for i:∗[n_(Fi J ∧ ¬Ki Fi⊕1 J) → (i, i ⊕ 1)].

j=1

Informally, agent i calls his successor, agent i ⊕ 1, if i isfamiliar with some secret and he does not know whether
his successor is familiar with it. Note the similarity with
protocol R1.

52

Protocol
LNS
HMS
R3
R4

T
yes
yes
no
yes

FT
yes
yes
yes
yes

T for .
no
yes
no
yes

FT for .
no
yes
yes
yes

T for /
yes
no
no
no

FT for /
yes
no
yes
yes

Table 1: Summary of termination results.
Proposition 4.4. Protocol R4 terminates and is correct.
If the communication mode is push, then the terminationargument remains valid, since after the call i . i ⊕ 1 agenti ⊕ 1 still learns all the secrets that agent i is familiar withand hence the above set {(i, j) | ¬Ki Fi⊕1 J} decreases.
If the communication mode is pull, then the protocol may
fail to terminate, because after the first call i / i ⊕ 1 agenti ⊕ 1 does not learn the secret of agent i and consequently
the call can be repeated. However, the situation changes
when fairness is assumed.
Proposition 4.5. For the pull communication mode protocol R4 fairly terminates.
Table 1 summarizes the termination properties of the protocols considered in the paper.

Third, it will be interesting to analyze the protocols forthe types of calls considered in [3]. They presuppose someform of knowledge that a call took place (for instance thatgiven a call between a and b each agent c 6= a, b noted the callbut did not learn its content). Another option is to consider
multicasting (calling several agents at the same time).
Finally, many assumptions of the current setup could be
lifted. Different initial and final situations could be considered, for instance common knowledge of protocols could
be assumed, or common knowledge of the familiarity of allagents with all the secrets upon termination could be required. Finally, to make the protocols more efficient passingof tokens could be allowed instead of just the transmissionof secrets by means of calls.

Acknowledgments
We would like to thank Hans van Ditmarsch and the referees for helpful comments and Rahim Ramezanian for usefulcomments about Example 2.8. This work resulted from a
research visit by Krzysztof Apt to Davide Grossi and Wiebevan der Hoek, sponsored by the 2014 Visiting Fellowship
Scheme of the Department of Computer Science of the University of Liverpool. The first author is also a Visiting Professor at the University of Warsaw. He was partially supported by the NCN grant nr 2014/13/B/ST6/01807.

6.

5. CONCLUSIONS
The aim of this paper was to introduce distributed gossip
protocols, to set up a formal framework to reason about
them, and to illustrate it by means of an analysis of selected
protocols.
Our results open up several avenues for further research.
First, our correctness arguments were given in plain English
with occasional references to epistemic tautologies, such as
Ki φ → φ, but it should be possible to formalize them ina customized epistemic logic. Such a logic should have a
protocol independent component that would consist of thecustomary S5 axioms and a protocol dependent component
that would provide axioms that depend on the mode of communication and the protocol in question. An example ofsuch an axiom is the formula Ki Fi⊕1 I 1 → Fi I ⊕ 1 that
we used when reasoning about protocol R2. To prove thevalidity of the latter axioms one would need to develop a
proof system that allows us to compute the effect of thecalls, much like the computation of the strongest postconditions in Hoare logics. Once such a logic is provided thenext step will be to study formally its properties, including
decidability. Then we could clarify whether the provided
correctness proofs could be carried out automatically.
Second, generalizing further the ideas we introduced byconsidering directed rings, gossip protocols could be studied in interface with network theory (see [13] for a textbook
presentation). Calls can be assumed to be constrained by a
network, much like in the literature on ‘centralized’ gossip(cf. [10]) or even have probabilistic results (i.e., secrets are
passed with given probabilities). More complex propertiesof gossip protocols could then be studied involving higherorder knowledge or forms of group knowledge among neighbors (e.g., “it is common knowledge among a and her neighbors that they are all experts”), or their stochastic behavior(e.g., “at some point in the future all agents are experts withprobability p”).

53

REFERENCES[1] K. R. Apt, F. R. de Boer, and E. R. Olderog.
Verification of Sequential and Concurrent Programs.
Springer, 2009.
[2] K. R. Apt, N. Francez, and S. Katz. Appraising
fairness in distributed languages. Distributed
Computing, 2(4):226–241, 1988.
[3] M. Attamah, H. van Ditmarsch, D. Grossi, andW. Van der Hoek. Knowledge and gossip. In
Proceedings of ECAI’14, pages 21–26. IOS Press, 2014.
[4] B. Baker and R. Shostak. Gossips and telephones.
Discrete Mathematics, 2:197–193, 1972.
[5] R. Bumby. A problem with telephones. SIAM Journalof Algorithms and Discrete Methods, 2:13–18, 1981.
[6] E. W. Dijkstra. Guarded commands, nondeterminacy
and formal derivation of programs. Communicationsof the ACM, 18:453–457, 1975.
[7] R. Fagin, J. Halpern, Y. Moses, and M. Vardi.
Knowledge-based programs. Distributed Computing,
10:199–225, 1997.
[8] R. Fagin, J. Y. Halpern, Y. Moses, and M. Y. Vardi.
Reasoning about knowledge. The MIT Press,
Cambridge, 1995.
[9] A. Hajnal, E. C. Milner, and E. Szemeredi. A cure forthe telephone disease. Canadian Mathematical
Bulletin, 15:447–450, 1972.
[10] S. M. Hedetniemi, S. T. Hedetniemi, and A. L.
Liestman. A survey of gossiping and broadcasting incommunication networks. Networks, 18(4):319–349,
1988.
[11] C. A. R. Hoare. Communicating sequential processes.
Communications of the ACM, 21:666–677, 1978.
[12] INMOS Limited. Occam Programming Manual.
Prentice-Hall International, 1984.
[13] M. O. Jackson. Social and Economic Networks.
Princeton University Press, 2008.

[14] R. Kurki-Suonio. Towards programming withknowledge expressions. In Proceedings of POPL’86,
pages 140–149, 1986.
[15] J. Meyer and W. van der Hoek. Epistemic Logic for
AI and Computer Science, volume 41 of Cambridge
Tracts in Theoretical Computer Science. Cambridge
University Press, 1995.
[16] R. Parikh and R. Ramanujam. Distributed processing
and the logic of knowledge. In Logic of Programs,
LNCS 193, pages 256–268. Springer, 1985. Similar to
JoLLI 12: 453–467, 2003.
[17] A. Seress. Quick gossiping without duplicate
transmissions. Graphs and Combinatorics, 2:363–383,
1986.
[18] R. Tijdeman. On a telephone problem. Nieuw Archiefvoor Wiskunde, 3(XIX):188–192, 1971.

Further, the implication Ki Fi⊕1 I 1 → Fi I 1 is true inevery node of the computation tree (remember, the agentsareVn positioned on a directed ring). If n = 3, this proves that
j=1 Fi J is true.
If n = 4, we note that Ki Fi⊕1 I 1 implies that agent i ⊕1 learned I
1 through a call of agent i and hence theimplication Ki Fi⊕1 I 1 → Fi I ⊕ 1 is true in every node of
the computation tree, as well V(remember that the mode ispush-pull). We conclude that n
j=1 Fi J is true.
Proof of Proposition 4.3. First, note that the following three statements are equivalent for each node c of anarbitrary computation ξ and each agent i:• i is disabled at c,V• (MDR , c) |= ( nj=1 Fi J) ∧ Ki Fi⊕1 I• a sequence of calls (i⊕2, i⊕3), (i⊕3, i⊕4), . . ., (i, i⊕1)
(possibly interspersed with other calls) has taken placein ξ before c.

APPENDIX
Proof of Proposition 4.1.
Termination Given a call sequence c define the set
Inf (c) := {(i, j) | i, j ∈ {1, . . ., n} and (MDR , c) |= Fi J}.
After each enabled call i . i ⊕ 1 in c, the set Inf (c) increases,
which ensures termination since each set Inf (·) has at mostn2 elements.
Correctness Consider a leaf of the computation tree. Thenthe exit conditionn ^n^(¬Fi J ∨ ¬Ki ¬Fi⊕1 J)

i=1 j=1

is true. We proceed by induction to show that then each Fi J
is true, where i, j ∈ {1, . . ., n}, and where the pairs (i, j) areordered as follows:(1, 1), (2, 1), . . ., (n, 1),(2, 2), (3, 2), . . ., (1, 2),. . .,(n, n), (1, n), . . ., (n − 1, n).
So the ith row lists the pairs (j, i) with j ∈ {1, . . ., n} ranging
clockwise, starting at i.
Take a pair (i, j). If i = j, then Fi J is true by assumption.
If i 6= j, then consider the pair that precedes it in the above
ordering. It is then of the form (i1 , j), where i = i1 ⊕ 1.
By the induction hypothesis Fi1 J is true, so by the exit
condition ¬Ki1 ¬Fi J is true.
Suppose now towards a contradiction that ¬Fi1 ⊕1 J istrue. Then i1 ⊕ 1 6= j. Hence by virtue of the considered communication mode and Definition 2.4 it follows thatagent ii knows that ¬Fi1 ⊕1 J is true since the only way fori1 ⊕ 1 to become familiar with J is by means of a call fromi1 . So Ki1 ¬Fi J is true. This yields a contradiction. HenceFi J is true.
V
Vn
So we showed, as desired, that n
i=1
j=1 Fi J is true inthe considered leaf.
V
Proof of Proposition 4.2. To start with, ni=1 Fi I istrue in every
node
of
the
computation
tree.
Supposethe exitV
condition n
1 is true at a nodeVof the compui=1 Ki Fi⊕1 I
tation tree (in short, true). It implies that n
1i=1 Fi⊕1 I
is true. Fix i ∈ {1, . . ., n}. By the above Fi I 2 is true.

1,

Suppose now towards a contradiction that an infinite faircomputation ξ exists. We proceed by case distinction.
Case 1 Some agent becomes disabled in ξ.
We claim that if an agent i becomes disabled in ξ, then also
agent i ⊕ 1 becomes disabled in ξ. Indeed, otherwise by
fairness at some point in ξ after which i becomes disabled,
agent i⊕1 calls his successor, i⊕2, and by the above sequenceof equivalences in turn becomes disabled.
We conclude by induction that at some point in ξ allagents become disabled and hence ξ terminates, which yields
a contradiction.
Case 2 No agent becomes disabled in ξ.
By fairness each agent calls in ξ infinitely often his successor.
So for every agent i there exists in ξ the sequence of calls(i ⊕ 2, i ⊕ 3), (i ⊕ 3, i ⊕ 4), . . ., (i, i ⊕ 1) (possibly interspersedwith other calls). By the above sequence of equivalences
after this sequence of calls agent i becomes disabled, whichyields a contradiction.
Proof of Proposition 4.4.
Termination It suffices to note that after each call (i, i ⊕ 1)
the size of the set{(i, j) ∈ A × A | ¬Ki Fi⊕1 J}
decreases.
Correctness Consider a leaf of the computation tree. Thenthe exit conditionn ^n^(¬Fi J ∨ Ki Fi⊕1 J)
i=1 j=1

is true. As in the case of protocol R1 we prove that it implies
each Fi J is true by induction on the pairs (i, j), where i, j ∈{1, . . ., n}, ordered as follows:(1, 1), (2, 1), . . ., (n, 1),(2, 2), (3, 2), . . ., (1, 2),. . .,(n, n), (1, n), . . ., (n − 1, n).
Take a pair (i, j). If i = j, then Fi J is true by assumption.
If i 6= j, then consider the pair that precedes it in the above
ordering, so (i1 , j), where i = i1 ⊕ 1. By the induction
hypothesis Fi1 J is true, so by the exit condition Ki1 Fi J istrue and hence Fi J is true.

54

Proof of Proposition 4.5. Consider the following sequence of statements:(i) i is disabled at c,V(ii) (MDR , c) |= n
j=1 (Fi J → Ki Fi⊕1 J),(iii) (MDR , c) |= Ki Fi⊕1 ,(iv) a sequence of calls i 1 / i, i 2 / i 1, . . ., i / i ⊕ 1(possibly interspersed with other calls) has taken placein ξ before c.

It is easy to verify that these statements are logically related
in the following way:(i) ⇔ (ii) ⇒ (iii) ⇒ (iv) ⇒ (ii)for each node c of an arbitrary computation ξ and each agent
i. They are therefore equivalent. Suppose now towards acontradiction that an infinite fair computation ξ exists. As
in the proof of Proposition 4.3 we proceed by case distinction.
Case 1 Some agent becomes disabled in ξ.
We claim that if an agent i becomes disabled in ξ, then also
i 1 becomes disabled in ξ. Indeed, otherwise by fairness
at some point in ξ after which j becomes disabled, agent
i 1 calls his successor, i, and by the above sequence ofequivalences in turn becomes disabled.
We conclude by induction that at some point in ξ allagents become disabled and hence ξ terminates, which yields
a contradiction.
Case 2 No agent becomes disabled in ξ.
By fairness each agent calls in ξ infinitely often his successor.
So for every agent i there exists in ξ a sequence of calls
i 1 / i, i 2 / i 1, . . ., i / i ⊕ 1 (possibly interspersed withother calls). By the above sequence of equivalences, after
this sequence of calls agent i becomes disabled, which yields
a contradiction.

55

Coordination Games on Directed Graphs
Krzysztof R. Apt

Centrum Wiskunde &Informatica
Amsterdam, The Netherlands

k.r.apt@cwi.nl

Sunil Simon

Department of Computer
Science and Engineering
IIT Kanpur, Kanpur, India

simon@cse.iitk.ac.in

ABSTRACT
We study natural strategic games on directed graphs, which
capture the idea of coordination in the absence of globallycommon strategies. We show that these games do not needto have a pure Nash equilibrium and that the problem ofdetermining their existence is NP-complete. The same holds
for strong equilibria. We also exhibit some classes of games
for which strong equilibria exist and prove that a strong
equilibrium can then be found in linear time.

1. INTRODUCTION

In this paper we study a simple and natural class of strategic games. Assume a finite directed graph. Suppose thateach node selects a colour from a private set of colours
available for it. The payoff to a node is the number of(in)neighbours who chose the same colour.
These games are typical examples of coordination games.
Recall that the idea behind coordination in strategic games
is that players are rewarded for choosing common strategies.
The games we study here are specific coordination games in
the absence of globally common strategies.
Recently, we studied in [2], and more fully in [3], a very
similar class of games in which the graphs were assumed to
be undirected. However, the transition from undirected todirected graphs drastically changes the status of the games.
For instance, for the case of directed graphs Nash equilibria do not need to exist, while they always exist when thegraph is undirected. Consequently, in [2] and [3] we focusedon the problem of existence of strong equilibria. We also argued there that such games are of relevance for the cluster
analysis, the task of which is to partition in a meaningful
way the nodes of a graph. The same applies here. Indeed,once the strategies are possible cluster names, a Nash equilibrium naturally corresponds to a ‘satisfactory’ clusteringof the underlying graph.
The above two classes of games are also similar in thatboth are special cases of a number of well-studied types ofgames. One of them are polymatrix games introduced in[10]. In these games the payoff for each player is the sum of
the payoffs from the individual two player games he playswith each other player separately. Another are graphicalgames introduced in [11]. In these games the payoff of eachplayer depends only on the strategies of its neighbours in agiven in advance graph structure over the set of players.
In addition both classes of games satisfy the positive population monotonicity (PPM) property introduced in [12]that states that the payoff of each player weakly increases if
another player switches to his strategy. Coordination games

57

Dominik Wojtczak

University of Liverpool
Liverpool, U.K.

d.wojtczak@liv.ac.uk

on graphs are examples of games on networks, a vast research area recently surveyed in [9]. Other related references
can be found in [3].

1.1

Plan of the paper and the results

2.

PRELIMINARIESIn the next section we introduce preliminary definitions,following [3]. We define the coordination games on directed
graphs in Section 3. In Section 4 we exhibit a number ofcases when a strong equilibrium exists. Next, in Section 5
we study complexity of the problem of existence of Nashand strong equilibria and the problem of determining thecomplexity of finding a strong equilibrium in a natural casewhen it is known to exist. Finally, in Section 6 we discuss
future directions.
The main results are as follows. If the underlying graph isa DAG, is complete or is such that every strongly connectedcomponent (SCC) is a simple cycle, then strong equilibria
always exist and they can always be reached from any initial joint strategy by a sequence of coalitional improvement
steps. The same is the case when only two colours are used.
In general Nash equilibria do not need to exist and the
problem of determining their existence is NP-complete. The
same is the case for strong equilibria. We also show thatwhen every SCC is a simple cycle, then strong equilibrium
can always be found in linear time.

A strategic game G = (S1 , . . . , Sn , p1 , . . . , pn ) with n > 1
players, consists of a non-empty set Si of strategies and apayoff function pi : S1 × · · · × Sn → R, for each player i.
We denote S1 × · · · × Sn by S, call each element s ∈S a joint strategy and abbreviate the sequence (sj )j6=i to
s−i . Occasionally we write (si , s−i ) instead of s. We call a
strategy si of player i a best response to a joint strategy s−iof his opponents if for all s0i ∈ Si , pi (si , s−i ) ≥ pi (s0i , s−i ).
Fix a strategic game G. We say that G satisfies the positive population monotonicity (PPM) if for all joint
strategies s and players i, j, pi (s) ≤ pi (si , s−j ). (Note that(si , s−j ) refers to the joint strategy in which player j choosessi .) So if more players (here just player j) choose player
i’s strategy and the remaining players do not change theirstrategies, then i’s payoff weakly increases.
We call a non-empty subset K := {k1 , . . . , km } of the setof players N := {1, . . . , n} a coalition. Given a joint strategy s we abbreviate the sequence (sk1 , . . . , skm ) of strategies
to sK and Sk1 × · · · × Skm to SK . We also write (sK , s−K )instead of s. If there is a strategy x such that si = x for all
players i ∈ K, we also write (xK , s−K ) for s.

Given two joint strategies s0 and s and a coalition K, we
say that s0 is a deviation of the players in K from s ifK
K = {i ∈ N | si 6= s0i }. We denote this by s→s0 . If in
0
addition pi (s ) > pi (s) holds for all i ∈ K, we say that thedeviation s0 from s is profitable. Further, we say that theplayers in K can profitably deviate from s if there exists
a profitable deviation of these players from s.
Next, we call a joint strategy s a k-equilibrium, wherek ∈ {1, . . . , n}, if no coalition of at most k players can profitably deviate from s. Using this definition, a Nash equilibrium is a 1-equilibrium and a strong equilibrium, see[5], is an n-equilibrium.
Given a joint strategy s, we call the sum
SW(s) =Xpi (s)i∈N

the social welfare of s.
A coalitional improvement path, in short a c-improvement path, is a maximal sequence ρ = (s1 , s2 , . . . ) of joint
strategies such that for every k > 1 there is a coalition
K such that sk is a profitable deviation of the players in
K from sk−1 . If ρ is finite then by last(ρ) we denote thelast element of the sequence. Clearly, if a c-improvement
path is finite, its last element is a strong equilibrium. We
say that G has the finite c-improvement property (cFIP) if every c-improvement path is finite. Further, we
say that the function P : S → A, where A is a set, is a
generalized ordinal c-potential, also called generalized
strong potential, see [7, 8], for G if for some strict partial
ordering (P (S), ) the fact that s0 is a profitable deviationof the players in some coalition from s implies that P (s0 ) P (s).
If a finite game admits a generalized ordinal c-potential
then it has the c-FIP. The converse also holds, see, e.g.,[3]. We say that G is c-weakly acyclic if for every joint
strategy there exists a finite c-improvement path that starts
at it. Note that games that have the c-FIP or are c-weakly
acyclic game have a strong equilibrium.
We call a c-improvement path an improvement path if
each deviating coalition consists of one player. The notionsof a game having the FIP or being weakly acyclic, see[15, 13], are then defined by referring to improvement paths
instead of c-improvement paths.

3. COORDINATION GAMES ON DIRECTED
GRAPHS

We now introduce the class of games we are interested
in. Fix a finite set of colours M and a weighted directed
graph (G, w) without self loops in which each edge e has
a non-negative weight we associated with. We say that anode j is a neighbour of the node i if there is an edge j → i
in G. Let Ni denote the set of all neighbours of node i in
the graph G. By a colour assignment we mean a functionthat assigns to each node of G a finite non-empty set ofcolours. For technical reasons we also introduce the conceptof a bonus, which is a function β that to each node i and acolour c assigns a natural number β(i, c). (We allow zero asa natural number.)Given a weighted graph (G, w), a colour assignment A anda bonus function β we define a strategic game G(G, w, A, β)as follows:• the players are the nodes,• the set of strategies of player (node) i is the set ofcolours A(i); we refer to the strategies as colours andto joint strategies as colourings,• each payoff function is defined bypi (s) =X

wj→i + β(i, si ).

j∈Ni , si =sj

So each node simultaneously chooses a colour and the payoff to the node is the sum of the weights of the edges from its
neighbours that chose its colour augmented by the bonus to
the node from choosing the colour. We call these games coordination games on directed graphs, from now on justcoordination games. Because weights are non-negative
each coordination game satisfies the PPM.
In the paper we mostly consider the case when all weights
are 1 and all bonuses are 0. Then each payoff function is
simply defined bypi (s) := |{j ∈ Ni | si = sj }|.
Example 1. Consider the directed graph and the colour
assignment depicted in Figure 1.
7{a} {a,b}
:D 1
{b,c}

{a,b}

$

6

O

  {a,c}
2 d

{b,c}

: 3 od

9 {b}

4

z

5 {a,c}{c} 8Figure 1: A directed graph with a colour assignment.
Take the joint strategy s that consists of the underlined
strategies. Then the payoffs are as follows:• 0 for the nodes 1, 7, 8 and 9,• 1 for the nodes 2, 4, 5, 6,• 2 for the node 3.
Note that the above joint strategy is not a Nash equilibrium.
For example, node 1 can profitably deviate to colour a. 2
In what follows we study the problem of existence of Nashequilibria or strong equilibria in coordination games.
Finally, given a directed graph G and a set of nodes K,
we denote by G[K] the subgraph of G induced by K.

4.

STRONG EQUILIBRIAIn this section we focus on the existence of strong equilibria. To start with, we have the following positive result.

58

Theorem 2. Every coordination game whose underlying
graph is a DAG has the c-FIP and a fortiori a strong equilibrium. Further, each Nash equilibrium is a strong equilibrium.
Proof. Given a DAG G := (V, E), where V = {1, . . ., n},we fix a permutation π of 1, . . ., n such that for all i, j ∈ V
if i < j, then (π(j) → π(i)) 6∈ E.

(1)

So if i < j, then the payoff of the node π(i) does not dependon the strategy selected by the node π(j).
Then given a coordination game whose underlying directed graph is the DAG G we associate with each joint
strategy s the sequence pπ(1) (s), . . ., pπ(n) (s) that we abbreviate to pπ (s). We now claim that pπ : S → Rn is a generalized ordinal c-potential when we take for the partial ordering
 on pπ (S) the lexicographic ordering >lex on the sequencesof reals.
Suppose that some coalition K profitably deviates fromthe joint strategy s to s0 . Choose the smallest j such that
π(j) ∈ K. Then pπ(j) (s0 ) > pπ(j) (s) and by (1) pπ(i) (s0 ) =pπ(i) (s) for i < j. This implies that pπ (s0 ) >lex pπ (s), as
desired. Hence the game has the c-FIP.
To prove the second claim, take a Nash equilibrium s andsuppose it is not a strong equilibrium. Then some coalition
K can profitably deviate from s to s0 . Choose the smallest
j such that π(j) ∈ K. Then pπ(j) (s0 ) > pπ(j) (s) and by (1)the payoff of π(j) does not depend on the strategies selectedby the other members of the coalition K. Hence pπ(j) (s0 ) =pπ(j) (s0π(j) , s−π(j) ), which contradicts the assumption that s
is a Nash equilibrium.
The next result deals with a class of coordination games
introduced in [3]. Given the set of colours M , we say thata directed graph G is colour complete (with respect to acolour assignment A) if for every colour x ∈ M each component of G[Vx ] is complete, where Vx = {i ∈ V | x ∈ Ai }.
In particular, every complete graph is colour complete.
Theorem 3. Every coordination game on a colour complete directed graph has the c-FIP and a fortiori a strong
equilibrium.
Proof. In [3] it is proved that every uniform game has
the c-FIP, where we call a coordination game on a directed
graph G uniform if for every joint strategy s and for everyedge i → j ∈ E it holds: if si = sj then pi (s) = pj (s). (In [3]only undirected graphs are considered, but the proof remains
valid without any change.) Clearly every coordination gameon a colour complete directed graph is uniform.
It is difficult to come up with other classes of directed
graphs for which the coordination game has the c-FIP. Indeed, consider the following example.
Example 4. Consider a coordination game on a simple
cycle 1 → 2 → . . . → n → 1, where n ≥ 3 and such that thenodes share at least two colours, say a and b. Take the initialcolouring (a, b, . . ., b). Then both (a, b, b, . . ., b), (a, a, b, . . ., b)and (a, a, b, . . ., b), (b, a, b, . . ., b) are profitable deviations. (Toincrease readability we underlined the strategies that weremodified.) After these two steps we obtain a colouring thatis a rotation of the first one. Iterating we obtain an infinite
improvement path.
Hence the coordination game does not have the FIP and
a fortiori the c-FIP.
2

59

However, a weaker result holds, which, for reasons thatwill soon become clear, we prove for a larger class of games.
Theorem 5. Every coordination game with bonuses on asimple cycle is c-weakly acyclic, so a fortiori has a strong
equilibrium.
To prove it, we first establish a weaker claim.
Lemma 6. Every coordination game with bonuses on asimple cycle is weakly acyclic.
Proof. To fix the notation, suppose that the consideredgraph is 1 → 2 → . . . → n → 1. Below for i ∈ {2, . . ., n},i 1 = i − 1, and 1 1 = n.
Let MA(i) be the set of available colours to player i withthe maximal bonus, i.e., MA(i) = {c ∈ A(i) | β(i, c) =maxd∈A(i) β(i, d)}. Let
BR(i, s−i ) = {c ∈ MA(i) | colour c is a best responseof player i to s−i }be the set of best responses among the colours with the highest bonus only. The set BR(i, s−i ) is never empty becauseof the game structure and the fact that bonuses are natural
numbers. Indeed, if si 1 ∈ MA(i), then BR(i, s−i ) = {si 1 }and otherwise BR(i, s−i ) is a non-empty subset of MA(i).
Below we stipulate that whenever a player i updates ina joint strategy s his strategy to a best response to s−i , healways selects a strategy from BR(i, s−i ).
Consider an initial joint strategy s. We construct a finite
improvement path that starts with s as follows.
Phase 1. We proceed around the cycle and consider theplayers 1, 2, . . . , n−1 in that order. For each player in turn,if his current strategy is not a best response, we update it
to a best response respecting the above proviso. When thisphase ends the current strategy of each of the players 1, 2,. . . , n − 1 is a best response.
If at this moment the current strategy of player n is also
a best response, the current joint strategy s0 is a Nash equilibrium and the path is constructed. Otherwise we move tothe next phase.
Phase 2. We repeat the same process as in Phase 1, but
starting with s0 and player n and proceeding at most n steps.
From now on at each step at least n − 1 players have a best
response strategy. So if at a certain moment the current
strategy of the considered player is a best response, the current joint strategy is a Nash equilibrium and the path isconstructed. Otherwise, after n steps, we move to the final
phase.
Phase 3. We repeat the same process as in Phase 2. Now
in the initial joint strategy each player i has a strategy from
MA(i). Because of the definition of BR(i, s−i ) each player
can improve his payoff only if he switches to the strategy
selected by his predecessor. So after at most n steps thisphase terminates and we obtain a Nash equilibrium.
By Lemma 6 every coordination game on a simple cycle
has a Nash equilibrium. However, not every Nash equilibrium is then a strong equilibrium.
Example 7. Consider the directed graph depicted in Figure 2, together with the sets of colours associated with thenodes.
Clearly (a, b) is a Nash equilibrium. However, it is nota strong equilibrium since the coalition {1, 2} can profitably
deviate to (c, c), which is a strong equilibrium.
2{a,c}

1

i

)

2

{b,c}

Figure 2: Nash equilibria versus strong equilibria
On the other hand, the following observation holds.
Lemma 8. Consider a coordination game with bonuses ona simple cycle with n nodes. Then every Nash equilibrium
is a k-equilibrium for all k ∈ {1, . . ., n − 1}.

Proof. Take a Nash equilibrium s. It suffices to provethat it is an (n − 1)-equilibrium. Suppose otherwise. Thenfor some coalition K of size ≤ n − 1 and a joint strategy s0 ,K
s→s0 is a profitable deviation.
Assume k 1 = k − 1 if k > 1 and 1 1 = n. Take somei ∈ K such that i 1 6∈ K. We have pi (s0 ) > pi (s). Alsopi (s0i , s−i ) = pi (s0 ), since si 1 = s0i 1 . So pi (s0i , s−i ) > pi (s),which contradicts the fact that s is a Nash equilibrium.

Proof of Theorem 5. Take a joint strategy s. By Lemma 6
a finite improvement path starts at s and ends in a Nashequilibrium s0 . By Lemma 8 s0 is an (n − 1)-equilibrium.
If s0 is not a strong equilibrium, then a profitable deviationNs0 →s00 exists, where, recall, N is the set of all players. Because of the game structure the social welfare along eachc-improvement path weakly increases, while in the last step
the social welfare strictly increases. So SW(s00 ) > SW(s).
If s00 is not a strong equilibrium, we repeat the above procedure starting with s00 . Since each time the social welfare
strictly increases, eventually this process stops and we obtain a finite c-improvement path.
2
Using Theorem 5, we now show that every coordination
game in which all strongly connected components are simple
cycles is c-weakly acyclic. We first introduce some notations
and make use of the following well-known decomposition
result.
Theorem 9 ([6], page 92). Every directed graph is adirected acyclic graph (DAG) of its strongly connected components (SCCs).
Given a graph G = (V, E), let D = (VD , ED ) be thecorresponding DAG obtained by the above decomposition
theorem and let g : 2V → VD be the function that maps
each SCC in G to a node in D. Let g −1 (v) = X ⊆ V whereg(X) = v. Note that for each i ∈ V , there is a uniquev ∈ VD such that i ∈ g −1 (v), we denote this node by vi . Let|VD | = m and θ = (θ1 , θ2 , . . . , θm ) be a topological orderingof VD (this is well-defined since D is a DAG). We definea labelling function lD : VD → {1, . . . , m} as follows: forall v ∈ VD , lD (v) = j iff θj = v. We can extend lD toa function l : V → {1, . . . , m} in the natural way: for alli ∈ V , l(i) = lD (v) if i ∈ g −1 (v).
Note that for each node v ∈ VD , either g −1 (v) = {i} forsome i ∈ V or g −1 (v) = X ⊆ V with |X| ≥ 2 and thesubgraph of G induced by the set of nodes X forms an SCC.
Also, note that every v ∈ VD and a joint strategy s in G,
defines a coordination game with bonuses Gv on the graph
G(v, s) = (V 0 , E 0 ) which is the subgraph induced by the setof nodes V 0 = g −1 (v). For i ∈ V 0 and a ∈ A(i) we putβ(i, a) := |{j ∈ Ni \ V 0 | sj = a}|.

Theorem 10. Every coordination game on a directed graphG in which all strongly connected components of G are simple cycles is c-weakly acyclic and a fortiori has a strong equilibrium.
Proof. Consider a coordination game G on a graph G =(V, E) where all SCCs are simple cycles. Let D = (VD , ED )be the corresponding DAG with |VD | = m. Since all SCCs
in G are simple cycles, it follows that for all v ∈ VD , eitherg −1 (v) = {i} or g −1 (v) = X ⊆ V such that the induced
graph on X forms a simple cycle in G.
Let v ∈ VD such that the induced graph on g −1 (v) forms a
simple cycle in G. For a joint strategy t in G, consider the resulting game Gv on the graph (V 0 , E 0 ), the subgraph inducedby the set of nodes g −1 (v). Let s0 = tV 0 (the restriction of
the joint strategy t to nodes in V 0 ) and let ρ : s0 , s1 , . . . , sk
be a finite c-improvement path in Gv which is guaranteed toexist by Theorem
 5. Define CPath(Gv , t) as follows:
if tV 0 is a strong equilibrium in Gv ,CPath(Gv , t) =λt (s1 ), . . . , λt (sk )otherwise,
where for all h ∈ {1, . . . , k}, λt (sh ) is the joint strategy in
G defined as: for all i ∈ V , (λt (sh ))i = shi if i ∈ V 0 and(λt (sh ))i = ti if i 6∈ V 0 .
For a joint strategy t in G and v ∈ VD , if the underlying
graph of the coordination game Gv with bonuses consists ofexactly one node, then the game is trivially c-weakly acyclic
and we define CPath(Gv , t) analogously.
Let t0 be an arbitrary joint strategy in G. We define asequence of sequences of joint strategies as follows:• ρ0 = t 0 ,• for h ∈ {0, 1, . . . , m − 1}, let ρh+1 = ρh · CPath(Gv , th )where lD (v) = h + 1 and th = last(ρh ).
Let ρ = ρm . From the definition of ρm and CPath, itfollows that ρ is a finite sequence of joint strategies in G.
By induction on the length of ρ, we can show that for every
subsequent pair of joint strategies tk and tk+1 in ρ, there
is a coalition K ⊆ V for which tk+1 is a profitable deviation from tk . To complete the proof, it suffices to argue
that ρ is maximal, or equivalently, that last(ρ) is a strong
equilibrium.
Suppose last(ρ) is not a strong equilibrium. Then there
exists K ⊆ V and a joint strategy s such that there is a
profitable deviation of players in K from last(ρ) to s. Let dbe the least element of the set {l(i) | i ∈ K} and X = K ∩{i ∈ V | l(i) = d}. By definition of a profitable deviation,
we have that for all i ∈ X, pi (s) > pi (last(ρ)). Note thatfor all i ∈ X and for all j ∈ Ni \ g −1 (vi ), we have l(j) < d.
Therefore, (Ni \ g −1 (vi )) ∩ K = ∅. Also note that for allj ∈ g −1 (vi ), (last(ρd ))j = (last(ρ))j . But this implies thatthe coalition X has a profitable deviation from the joint
strategy (last(ρd ))X to sX in the game Gvi . This contradicts
the fact that last(ρd ) is a strong equilibrium in the gameGvi .
We conclude this section by considering another class ofcoordination games. Example 4 shows that even when only
two colours are used, the coordination game does not needto have the c-FIP. On the other hand, a weaker property
does hold.

Theorem 11. Every coordination game in which only twocolours are used is c-weakly acyclic and a fortiori has astrong equilibrium.

60

Proof. We prove the result for a more general class ofgames, namely the ones that satisfy the PPM. Call thecolours blue and red, that we abbreviate to b and r. Whena node selected blue we refer to it as a blue node, and thesame for the red colour.
Take a joint strategy s1 . Consider a maximal sequenceξ of profitable deviations of the coalitions starting in s inwhich the nodes can only switch to blue. At each step thenumber of blue nodes increases, so ξ is finite. Let s1 , . . ., sk ,where k ≥ 1, be the successive joint strategies of ξ.
If sk is a strong equilibrium, then ξ is the desired finite
improvement path. Otherwise consider a maximal sequenceχ of profitable deviations of the coalitions starting in sk
in which the nodes can only switch to red. χ is finite. Let
sk , sk+1 , . . ., sk+l , where l ≥ 1, be the successive joint strategies of χ.
We claim that sk+l is a strong equilibrium. Suppose othK
erwise. Then for some joint strategy s0 , sk+l →s0 is a profitable deviation of some coalition K. Let L be the set ofnodes from K that switched in this deviation to blue. By
the definition of sk+l the set L is non-empty.
Given a set of nodes M and a joint strategy s we denote by(M : b, s−M ) the joint strategy obtained from s by letting
the nodes in M to select blue, and similarly for the red
colour. Also it should be clear what joint strategy we denoteby (M : b, P \ M : r, s−P ), where M ⊆ P .
We claim that sk+l →(L : b, sk+l
−L ) is a profitable deviationof the players in L. Indeed, we have for all i ∈ L
Lk+l
pi (L : b, sk+l),
−L ) > pi (s(2)

0since by the PPM pi (L : b, sk+l
−L ) ≥ pi (s ) and by the as0
k+l
sumption pi (s ) > pi (s ).
Let M be the set of nodes from L that are red in sk .
Suppose that M is non-empty. We show that thenpM (M : r, L \ M : b, sk−L ) < pM (M : b, L \ M : b, sk−L ). (3)Indeed, we have for all i ∈ M

pi (M : r, L \ M : b, sk−L ) ≤ pi (M : r, L \ M : b, sk+l
−L )k+l
≤ pi (M : r, L \ M : r, sk+l
)
<
p
(M
:
b,
L
\
M
:
b,
s
i
−L
−L )
k
≤ pi (M : b, L \ M : b, s−L ),where the weak inequalities are due to the PPM and thestrict inequality holds by the definition of L.
But sk = (M : r, L \ M : b, sk−L ), so (3) contradicts thedefinition of sk . So M is empty, i.e., all nodes from L areblue in sk . We now have for all i ∈ L<

k+l
pi (L : r, sk−L ) ≤ pi (L : r, sk+l)
−L ) = pi (sk+lkpi (L : b, s−L ) ≤ pi (L : b, s−L ),where again the weak inequalities are due to the PPM andthe strict inequality holds by (2).
But (L : r, sk−L ) = sk , so we proved that sk →(L : b, sk−L )is a profitable deviation. This yields a contradiction withthe definition of sk .
L

The following example shows that when three colours areused, Nash equilibria, so a fortiori strong equilibria do notneed to exist.
Example 12. Consider the directed graph depicted in Figure 1 of Example 1, together with the sets of colours associated with the nodes. We argue that the coordination game61

associated with this graph does not have a Nash equilibrium.
Note that for nodes 7, 8 and 9 the only option is to selectthe unique strategy in its strategy set. The best response fornodes 4, 5 and 6 is to always select the same strategy as
nodes 1, 2 and 3 respectively. Therefore, to show that thegame does not have a Nash equilibrium, it suffices to consider the strategies of nodes 1, 2 and 3. We denote this by
the triple (s1 , s2 , s3 ). Below we list all such joint strategies
and we underline a strategy that is not a best response to the
choice of other players: (a, a, b), (a, a, c), (a, c, b), (a, c, c),(b, a, b), (b, a, c), (b, c, b) and (b, c, c).
2Call now a graph a coloured DAG (with respect to acolour assignment A) if for each available colour x the components of the subgraph induced by the nodes having colour
x are DAGs. In view of Theorem 3 it is tempting to try to
generalize Theorem 2 to coloured DAGs. However, the directed graph depicted in Figure 1 is a coloured DAG and,as explained in the above example, the coordination gameon this graph has no Nash equilibrium.

5.

COMPLEXITY ISSUES

Next, we study the complexity of the existence problems
and of the problem of finding strong equilibria.
Theorem 13. The Nash equilibrium existence problem incoordination games is NP-complete.
Proof. The problem is in NP, since we can simply guessa colour assignment and checking whether it is a Nash equilibrium can be done in polynomial time.
To prove NP-hardness we provide a reduction from the
3-SAT problem, which is NP-complete. Notice that an edge
with a natural number weight w can be simulated by adding
w extra players to the game. More precisely, an edge (i →j) with the weight w can be simulated by the extra setof players {i1 , . . . , iw } and the following 2 · w unweighted
edges: {(i → i1 ), (i → i2 ), . . . , (i → iw ), (i1 → j), (i2 →j), . . . , (iw → j)}. Given a colour assignment in the gamewith the weighted edges, we then assign to each of the nodes
i1 , . . ., iw the colour set of the node i.
Therefore we will assume that edges can have such weights
assigned to them, because this simplifies our construction.
Assume we are given a 3-SAT formulaφ = (a1 ∨ b1 ∨ c1 ) ∧ (a2 ∨ b2 ∨ c2 ) ∧ . . . ∧ (ak ∨ bk ∨ ck )with k clauses and n propositional variables x1 , . . . , xn , where
each ai , bi , ci is a literal equal to xj or ¬xj for some j. We
will construct a coordination game Gφ of size O(k) such thatGφ has a Nash equilibrium iff φ is satisfiable.
First, for every propositional variable xi we have a corresponding node Xi in Gφ with two possible colours > and ⊥.
Intuitively, for a given truth assignment, if xi is true then >should be chosen for Xi and otherwise ⊥ should be chosen.
In our construction we make use of the following gadget, denoted by Di (x, y, z), with three parameters x, y, z ∈ {>, ⊥}and i used just for labelling purposes, and presented in Figure 3. This gadget behaves similarly to the game without
Nash equilibrium analyzed in Example 12.
What is important is that for all possible parameters values, the gadget Di (x, y, z) does not have a Nash equilibrium.
Indeed, each of the nodes Ai , Bi , or Ci can always securea payoff 2, so selecting > or ⊥ is never a best response and{>, ⊥}{>, ⊥}{>, ⊥}{>, ⊥}

X1

X2

X3

X4

{•}

4

6

40

4

4

1

3

{•, •, >}
2

···

Xn

4

4
2

1

{>, ⊥}{•, •, ⊥}
1

2

{•}

1

{•, • >}
2{•, •, ⊥}
2

5

60

{•}

4

2

4

1

{•}

30

{•}

10

1

{•, •, >}
1
20

...

{•, •, ⊥}
2
{•}

50

Figure 4: The game Gφ corresponding to the formula φ = (x1 ∨ ¬x2 ∨ x3 ) ∧ (¬x3 ∨ x4 ∨ ¬xn ), where in each gadget
the nodes of indegree 1 are omitted.
•

{R}

2


9 C Ai {R,G,x}

1

2
{G,B}

{R,G}

•

1

O

%

1

1
{G,B,z}
Ci
2

:

•

2

  {R,B,y}
Bi d

1

oe

2
1

y

• {G}

• {R,B}2{B} •Figure 3: Gadget Di with three parameters x, y, z ∈{>, ⊥} and three distinguished nodes Ai , Bi , Ci .
hence in no Nash equilibrium a node chooses > or ⊥. The
rest of the reasoning is as in Example 12.
For any literal, l, letpos(l) :=>⊥

if l is a positive literalotherwise

For every clause (ai ∨bi ∨ci ) in φ we add to the game graphGφ the Di (pos(ai ), pos(bi ), pos(ci )) instance of the gadget.
Finally, for every literal ai , bi , or ci in φ, which is equal to
xj or ¬xj for some j, we add an edge from Xj to Ai , Bi , orCi , respectively, with weight 4. We depict an example gameGφ in Figure 4.
We claim that Gφ has a Nash equilibrium iff φ is satisfiable.

(⇒) Assume there is a Nash equilibrium s in the game Gφ .
We claim that the truth assignment ν : {x1 , . . . , xn } →{>, ⊥} that assigns to each xj the colour selected by thenode Xj in s makes φ true. Fix i ∈ {1, . . ., k}. We need toshow that ν makes one of the literals ai , bi , ci of the clause(ai ∨ bi ∨ ci ) true.
From the above observation about the gadgets it follows

that at least one of the nodes Ai , Bi , Ci selected in s thesame colour as its neighbour Xj . Without loss of generality
suppose it is Ai . The only colour these two nodes, Ai and
Xj , have in common is pos(ai ). So Xj selected in s pos(ai ),which by the definition of ν equals ν(xj ). Moreover, byconstruction xj is the variable of the literal ai . But ν(xj ) =pos(ai ) implies that ν makes ai true.
(⇐) Assume φ is satisfiable. Take a truth assignment ν :{x1 , . . . , xn } → {>, ⊥} that makes φ true. For all j, weassign to the node Xj the colour ν(xj ). We claim that thisassignment can be extended to a Nash equilibrium in Gφ .
Fix i ∈ {1, . . ., k} and consider the Di (pos(ai ), pos(bi ), pos(ci ))instance of the gadget. The truth assignment ν makes theclause (ai ∨ bi ∨ ci ) true. Suppose without loss of generality that ν makes ai true. We claim that then it is always
a unique best response for the node Ai to select the colour
pos(ai ).
Indeed, let j be such that ai = xj or ai = ¬xj . Notice thatthe fact that ν makes ai true implies that ν(xj ) = pos(ai ).
So when node Ai selects pos(ai ), the colour assigned to Xj ,its payoff is 4.
This partial assignment of colours can be completed to aNash equilibrium. Indeed, remove from the directed graphof Gφ all Xj nodes and the nodes that secured the payoff 4, together with the edges that use any of these nodes.
The resulting graph has no cycles, so by Theorem 2 thecorresponding coordination game has a Nash equilibrium.
Combining both assignments of colours we obtain a Nashequilibrium in Gφ .

Corollary 14. The strong equilibrium existence problem in coordination games is NP-complete.

Proof. It suffices to note that in the above proof the (⇒)
implication holds for a strong equilibrium, as well, while in
the proof of the (⇐) implication by virtue of Theorem 2
actually a strong equilibrium is constructed.
An interesting application of Theorem 13 is in the context
of polymatrix games. These are finite strategic form games
in which the influence of a pure strategy selected by any

62

player on the payoff of any other player is always the same,
regardless of what strategies other players select. Formally,
for all pairs of players i and j there exists a partial payoff
function aij such that for any joint strategy P
s = (s1 , . . . , sn ),the payoff of player i is given by pi (s) :=aij (si , sj ).
j6=i
In [14] we proved that deciding whether a polymatrix gamehas a Nash equilibrium is NP-complete. We can strengthen
this result as follows.
Theorem 15. Deciding whether a polymatrix game with
0/1 partial payoffs has a Nash equilibrium is NP-complete.
Proof. We can efficiently translate any coordination game
G(G, M, w, A, β) into a polymatrix game P with only 0/1
partial payoff as follows. The number of players in P will
be equal to the number of nodes in G and the set of strategies for each player will be M . We define aij (si , sj ) := 1 if
si = sj and j ∈ Ni , and aij (si , sj ) := 0 otherwise.
Notice that any joint strategy s = (s1 , . . . , sn ) in G is
also a joint strategy
M with exactly the same payoff,
P in ij
because pPa (si , sj ) = |{j ∈ Ni | si = sj }| =i (s) =j6=i
pGi (s). It follows that Nash equilibria in G and P coincide.
In particular, there exists Nash equilibrium in G if and only
if there exists one in P, but the former problem was shown
to be NP-hard in Theorem 13, so the latter is also NP-hard.
On the other hand, for any polymatrix game we can guessa joint strategy and check whether it is a Nash equilibriumin polynomial time, which shows this decision problem is infact NP-complete.
Next, we determine the complexity of finding a strong
equilibrium. We begin with the following auxiliary result.
Theorem 16. A strong equilibrium of a coordination gamewith bonuses on a simple cycle can be found in linear time.
Proof. Let n be the number of players in the game andC the number of possible colours. We assume adjacency
list representation for the game graph, binary representation of the bonuses and that the list of colours available toplayer i is given as a list of length |A(i)| of elements of sizelog C. Formally,P
the size of the input for player i only isΘ(|A(i)| log C + c∈A(i) log(β(i, c) + 1)); the sum of theseover i = 1, . . . , n gives the total size of the input.
First note that for any colour assignment, the best response of the i-th player can be found in time linear in
the size of her part of the input just by checking all possible colours in A(i). Second, each phase of the algorithm
in Lemma 6 looks for the best response (with a preference
given to colours with a higher bonus) of each player at mostonce, which will require time linear in the size of the whole
input. The algorithm requires at most three such phases
before a Nash equilibrium is found, so it runs in linear time.
Note that thanks to Lemma 8 we know that any NE insuch a game structure is already a (n − 1)-equilibrium, so
the only way this joint strategy is not a strong equilibrium
is when all n players can strictly improve their payoff. However, in any Nash equilibrium a player has to have her payoff
at at most one below the maximum possible one, becausethat is the minimum payoff for picking a colour with thehighest bonus. Moreover, player’s payoff can only be a natural number.
Therefore, the only possibility when a NE is not a strong
equilibrium is when there is a joint strategy which gives all63

the players their maximum possible payoff, i.e. each player
is assigned a colour with the highest possible bonus as wellas gets an extra +1 to her payoff for colour agreement withher only neighbour. The latter implies that all the players
need to pick the same colour in such a joint strategy.
To check whether such a joint strategy exists we do thefollowing. Let p = argmini |A(i)| be the player with the least
number of colours to choose from. We pick the set of hercolours with the maximal bonus and intersect it with the setof colours with the maximal bonus for every other player.
An intersection of two sets represented as lists of length aand b of elements of size K can be done in Θ(aK + bK)
time, so the total runningP
Pn time will be Θ(n|A(p)| log C +n|A(i)| log C) = Θ( i=1 |A(i)| log C), because |A(p)| ≤i=1|A(i)| for all i, which is linear. If the final set is empty thenany NE is a strong equilibrium and otherwise we know how
to construct one.
Corollary 17. A strong equilibrium of a coordination
game on a graph in which all strongly connected components
are simple cycles can be computed in linear time.
2

6.

CONCLUSIONSWe presented here a study of a simple class of coordination
games on directed graphs. We focused on the existence ofNash and strong equilibria. We also studied the complexityof checking for the existence of Nash and strong equilibria,
as well as the complexity of computing a strong equilibrium
in certain cases where it is guaranteed to exist.
A number of open problems remain. We showed that ingeneral Nash equilibria and strong equilibria are not guaranteed to exist. However, if the underlying graph is a DAG, iscolour complete or is such that every SCC is a simple cycle,
then strong equilibria always exist. It would be interesting
to identify other classes of graphs for which Nash or strong
equilibria exist.
The proof of Lemma 6 shows that in the case of a simple
cycle, starting from any initial joint strategy a Nash equilibrium can be found by an improvement path of length at
most 3n. Also, each step of such a path can be constructedin linear time. Additionally, the proof of Theorem 5 showsthat a strong equilibrium can be found by an improvement
path of length at most 3n + 1, possibly augmented by asingle profitable deviation of all players. It would be interesting to extend this analysis of bounds on the lengths ofimprovement paths to other cases when a Nash or a strong
equilibrium is known to exist.
In the future we plan to study the inefficiency of equilibria in coordination games on directed graphs. Also, weplan to study coordination games on finite directed weighted
graphs. While we already defined here these games, we usedweights solely as a means to simplify the argument in the
proof of Theorem 13. It should be noted that Lemma 6
does not hold for finite directed weighted graphs and, as aconsequence, Theorems 5, 10, and 16 do not hold either. A
counterexample to Lemma 6 can be constructed by modifying the game in Figure 1 as follows. Nodes 4, 5, 6 areremoved and replaced by assigning weight 2 to all the edges
in the cycle. Nodes 7, 8, 9 are also removed and replaced
by a +1 bonus to the colour of the node removed. It is easy
to see that the behaviour of this new game will mimic thegame in Figure 1. On the other hand, Theorem 2 and its
proof is still valid for finite directed weighted graphs as well

is Theorem 13, because checking whether a colour assignment is a Nash equilibrium can still be done in polynomial
time for them.
As an example of coordination games on weighted directed
graphs consider the problem of a choice of the trade treaties
between various countries. Assume a directed weighted graphin which the nodes are the countries and the weight on anedge i → j corresponds to the percentage of the overall import of country j from country i. Suppose additionally thateach country should choose a specific trade treaty, that theoptions for the countries differ (for instance because of its
geographic location) and that each treaty offers the sametax-free advantages. Then once the countries choose thetreaties, the payoff to each country is the aggregate percentage of its import that is tax-free.
The case of weighted directed graphs can be seen as a minor modification of the social network games with obligatory product selection that we introduced and analyzed in[4]. These are games associated with a threshold model of asocial network introduced in [1] which is based on weighted
graphs with thresholds. The difference consists of using
thresholds equal to 0. However, setting the thresholds to
0 essentially changes the nature of the games and cruciallyaffects the validity of several arguments.

Acknowledgments

Theory, 42(2):461–482, 2013.
[8] R. Holzman and N. Law-Yone. Strong equilibrium incongestion games. Games and Economic Behavior,21(1-2):85–101, 1997.
[9] M. Jackson and Y. Zenou. Games on networks. In
H. P. Young and S. Zamir, editors, Handbook of Game
Theory 4, pages 95–163. Elsevier, 2014.
[10] E. Janovskaya. Equilibrium points in polymatrix
games. Litovskii Matematicheskii Sbornik, 8:381–384,
1968.
[11] M. Kearns, M. Littman, and S. Singh. Graphicalmodels for game theory. In Proceedings of the 17th
Conference in Uncertainty in Artificial Intelligence(UAI ’01), pages 253–260. Morgan Kaufmann, 2001.
[12] H. Konishi, M. Le Breton, and S. Weber. Equivalenceof strong and coalition-proof Nash equilibria in games
without spillovers. Economic Theory, (9):97–113, 1997.
[13] I. Milchtaich. Congestion games with player-specificpayoff functions. Games and Economic Behaviour,
13:111–124, 1996.
[14] S. Simon and K. R. Apt. Social network games.
Journal of Logic and Computation, 1(25):207–242,
2015.
[15] H. P. Young. The evolution of conventions.
Econometrica, 61(1):57–84, 1993.

We are grateful to Mona Rahn and Guido Schäfer for useful
discussions and thank Piotr Sankowski and the referees forhelpful comments. First author is also a Visiting Professor
at the University of Warsaw. He was partially supported by
the NCN grant nr 2014/13/B/ST6/01807.

7. REFERENCES[1] K. R. Apt and E. Markakis. Diffusion in social
networks with competing products. In Proc. 4thInternational Symposium on Algorithmic Game
Theory (SAGT11), volume 6982 of Lecture Notes in
Computer Science, pages 212–223. Springer, 2011.
[2] K. R. Apt, M. Rahn, G. Schäfer, and S. Simon.
Coordination games on graphs (extended abstract). In
Proc. 10th International Workshop on Internet andNetwork Economics (WINE), volume 8877 of Lecture
Notes in Computer Science, pages 441–446. Springer,
2014.
[3] K. R. Apt, M. Rahn, G. Schäfer, and S. Simon.
Coordination games on graphs, 2015. Available fromhttp://arxiv.org/abs/1501.07388.
[4] K. R. Apt and S. Simon. Social network games withobligatory product selection. In Proc. 4thInternational Symposium on Games, Automata, Logics
and Formal Verification (GandALF ’13), volume 119,
pages 180–193. EPTCS, 2013.
[5] R. J. Aumann. Acceptable points in general
cooperative n-person games. In R. D. Luce and A. W.
Tucker, editors, Contribution to the theory of game
IV, Annals of Mathematical Study 40, pages 287–324.
University Press, 1959.
[6] S. Dasgupta, C. Papadimitriou, and U. Vazirani.
Algorithms. McGraw-Hill, 2006.
[7] T. Harks, M. Klimm, and R. H. Möhring. Strong
equilibria in games with the lexicographicalimprovement property. International Journal of Game64

On the Solvability of Inductive Problems:
A Study in Epistemic Topology
Alexandru Baltag

Institute for Logic, Language
and Computation
Science Park 107
Amsterdam, The Netherlands

a.baltag@uva.nl

Nina Gierasimczuk

Institute for Logic, Language
and Computation
Science Park 107
Amsterdam, The Netherlands

n.gierasimczuk@uva.nl

ABSTRACT
We investigate the issues of inductive problem-solving andlearning by doxastic agents. We provide topological characterizations of solvability and learnability, and we use them toprove that AGM-style belief revision is “universal”, i.e., thatevery solvable problem is solvable by AGM conditioning.

Keywords
Learning Theory, Topological Epistemology, Belief Revision

1. INTRODUCTION
When in the course of observations it becomes necessary
for agents to arrive at a generalization, they should declare,
along with their conjecture, the extent of their certainty.
The problem of induction seems formidable if a standard ofabsolute certainty is imposed on the learner. Indeed, as is
well-known in Philosophy of Science, the so-called problem
of empirical underdetermination (i.e., the fact that typicallythe data are compatible with more than one hypothesis)
rules out any chance of obtaining infallible knowledge inempirical research. But apart from the conclusions basedon absolute certainty (cf. [13, 10, 15]), learners can produce
hypotheses based on beliefs. It is thus strange that Formal
Learning Theory and Belief Revision Theory developed completely independently from each other, and that they havegenerally maintained their distance ever since.
However, there does exist a line of research that combines
belief revision with learning-theoretic notions, line pursued
by Kelly [21, 20], Kelly, Schulte and Hendricks [26], Martin
and Osherson [28], Osherson [29] and ourselves [13, 3, 4,14]. In this paper we continue this research program, using
topological characterizations and methods.
An inductive problem consists of a state space, a family of“potential observations”, and a “question” (i.e., a partition of
the state space). These observations provide data for learning. The problem is solvable if there exists a learner that,after observing “enough” pieces of data, eventually stabilizes on the correct answer. A special case of solvability islearnability in the limit, corresponding to the solvability of
the “ultimate” question: ‘What is the actual state of theworld?’. This notion matches the usual learning-theoretic
concept of identifiability in the limit [32, 16, 30].
The aim of the paper is twofold. First, we give topological characterizations of the notions of solvability (andlearnability), in terms of topological separation principles.
Intuitively, the ability to reliably learn the true answer to a

65

Sonja Smets

Institute for Logic, Language
and Computation
Science Park 107
Amsterdam, The Netherlands

S.J.L.Smets@uva.nl

question, is related to the ability to “separate” answers byobservations. The second goal is to use these topologicalresults to look at the “solving power” of well-behaved doxastic agents, such as the ones whose beliefs satisfy the usual
KD45 postulates of doxastic logic, as well as the standard
AGM postulates of rational belief-revision [1]. We look ata particularly simple and canonical type of doxastic agent,
who forms beliefs by AGM conditioning.
Our main result is that AGM conditioning is universal
for problem-solving, i.e., that every solvable problem can besolved by AGM conditioning. This means that (contrary tosome prior claims), AGM belief-revision postulates are notan obstacle to problem-solving. As a special case, it follows
that AGM conditioning is also “universal for learning” (every
learnable space can be learned by conditioning).1
The close connections between Epistemology and General
Topology have already been noticed long ago [33, 19]. Basedon these connections, Kevin Kelly started a far-reaching
program [19, 22] meant to import ideas and techniquesfrom both Formal Learning Theory and Topology into mainstream Epistemology, and show their relevance to the induction problem in Philosophy of Science. A further connectionis the one with Ockham’s Razor, that would(...) guarantee that always choosing the simplest
theory compatible with experience and hanging on
to it while it remains the simplest is both necessary and sufficient for efficiency of inquiry. [22]Simplicity has been claimed to have topologicalcharacteristics—the simplicity order should in someway follow the structure imposed on the uncertainty range
by possible tests and observations. It has also been linked
with the notion of minimal mind change, where the learning
agent keeps the conjecture changes to a minimum [19, 31].
Taken together, our results can be seen as a vindication
both of the general topological program in Inductive Epistemology [19, 22] and of the AGM Belief Revision Theory[1]. On the first front, our general topological characterizations of learning-theoretic concepts seem to confirm Kelly’s
long-standing claim that Inductive Epistemology can be seen
mathematically as a branch of General Topology. On thesecond front, our universality result seems to vindicate Be1
This special case is a topological translation of one of ourprevious results [3, 4]. However, the result about problemsolving universality is not only new and much more general,
but also much harder to prove, involving new topologicalnotions and results.

lief Revision Theory as a canonical form of learning.2

U

2. EPISTEMIC SPACES AND INDUCTIVE
PROBLEMS

Example 1. Let our epistemic space S = (S, O) be thereal numbers, with observable properties given by open intervals with rational endpoints: S := R, O := {(a, b) | a, b ∈Q, a ≤ b}, where (a, b) := {x ∈ R | a < x < b}. For instance,
observables may represent measurements of a physical quantity (such as a position along a one-dimensional line) thattakes real numbers as its possible values. In such case, forany state x ∈ R and any two sequences an , bn ∈ Q of rational
numbers, such that an ≤ x ≤ bn and both sequences converge
to x, the sequence (a0 , b0 ), . . . , (an , bn ), . . . is a (sound andcomplete) data stream for x.
2
And in the same time (if we adopt a “simplicity” interpretation of the prior), this last result can be seen as a vindicationof Ockham’s razor (in line with Kevin Kelly’s program).

stDefinition 1. An epistemic space is a pair S = (S, O)
consisting of a state space S and a countable (or finite) set
of observable properties (“data”) O ⊆ P(S). We denote by
by Os := {O ∈ O | s ∈ O} the set of all observable properties(holding) at a given state s.
One can think of the states in S as “possible worlds”, inthe tradition of Kripke and Lewis.The sets O ∈ O represent
properties of the world that are in principle observable: if
true, such a property will eventually be observed (although
there is no upper bound on the time needed to come toobserve it).
To keep things simple, we assume that at each step of thelearning process only one property is observed. As for thecountability of the set O, it is natural to think of observables
as properties which can be expressed by means of a language
or numerical coding system, generated from a grammar witha finite vocabulary. Any such family O will be (at most)countable.
We denote by O∩ the family of all finite intersectionsof observations from O, and by O∗ the family of all finite sequences of observations. Such a finite sequence σ =(O0 , O1 , . . . , Oi ) ∈ O∗ is called a data sequence, and its i-thcomponent is denoted by σi := Oi . It is easy to see thatboth O∩ and O∗ are countable.
~ = (O0 , O1 , . . .) ∈A data stream is a countable sequence O
Oω of data from O. (Here, ω is the set of natural numbers, so Oω is the set of all maps assigning an observable
property to every natural number.) We use the following~ n is the n-th element in O;~ O[n]~
notation: O
is the ini~ of length n, (O0 , . . . , On−1 ); set(O)~ :=tial segment of O
~ is the set of all data in O;~ ∗ is{O | O is an element of O}
the concatenation operator on strings.
The intuition is that at stage n of a data stream, the agent
observes the information in On . A data stream captures apossible future history of observations in its entirety, while
a data sequence captures only a finite part of such a history.
~ ∈Given a state s ∈ S, a dataTstream for s is a stream OOω such that Os = {O ∈ O | n
O
⊆
O
for
some
n
∈
ω}.
i
i=0
~ is true at s) andSuch a stream is “sound” (every data in O
“complete” (every true data is entailed by some finite set of~observations in O).

U

V

Q
u

V

P
v

s

t

Q
u

P
v

Figure 1: A problem P (left-hand side) and its refinement P0 (right-hand side), see Example 3
Other examples include standard n-dimensional Euclidean
spaces, e.g., S = R3 with O consisting of all open balls withrational radius and center.
Definition 2. An inductive problem is a pair P = (S, Q)
consisting of an epistemic space S = (S, O) together with a“question” Q, i.e., a partition3 of S. The cells Ai of the partition Q are called answers. Given s ∈ S, the unique A ∈ Q
with s ∈ A is called the answer to Q at s, and denoted As .
We say that a problem P0 = (S, Q0 ) is a refinement of another problem P = (S, Q) (or that the corresponding question
Q0 is a refinement of the question Q) if every answer of Q
is a disjoint union of answers of Q0 .
The most refined question concerns the identity of the realworld.
Example 2. The learning question on a space S is Q ={{s} | s ∈ S} (‘What is the actual state?’).
Example 3. Let S = (S, O), where S = {s, t, u, v),
O = {U, V, P, Q}, with U = {s, t}, V = {s},P = {u, v},Q ={u}. Take the problem P = (S, Q), given by the question Q = {{t, u}, {s, v}} depicted on the left-hand sideof Figure 1. This can obviously refined to obtained the
problem P0 = (S, Q0 ) given by the learning question Q ={{s}, {t}, {u}, {v}} for this space, as depicted on the righthand side of Figure 1.

3.

LEARNING AND PROBLEM-SOLVING

Definition 3. Let S = (S, O) be an epistemic space andlet σ0 , . . . , σn ∈ O. An agent (also known as a “learner”,
or a “learning method”) is a map L that associates to any
epistemic space S and any data sequence (σ0 , . . . , σn ) somefamily LS (σ0 , . . . , σn ) ⊆ P(S) of subsets of S, satisfying
a “consistency” condition: ∅ 6∈ LS (σ0 , . . . , σn ) wheneverTni=o σi 6= ∅.

Intuitively, after observing the data sequence ~σ =(σ0 , . . . , σn ), we can say that agent L believes a proposition
P after observing the data sequence ~σ = (σ0 , . . . , σn ), and~σwrite BLP iff P ∈ LS (σ0 , . . . , σn ). We can also interpret thisas a conditional belief, rather than as revised belief, the agent
believes every P ∈ LS (σ0 , . . . , σn ) conditional on σ0 , . . . , σn .
S
3This means that i∈I Ai = S, and Ai ∩Aj = ∅ for all i 6= j.
66

But in the end we are of course interested in the actual revised beliefs after observing the data, so the assumption inthis case is that conditional beliefs guide the agent’s revision
strategy: they “pre-encode” future belief revisions, to use aterm coined by J. van Benthem [6]. The above consistency
simply means that each of the agent’s beliefs is consistentwhenever the observed data are consistent.
A doxastic agent is one whose set LS (σ0 , . . . , σn ) of beliefs
forms a (proper) filter on S when observing consistent data;
in other words, her beliefs are (consistent when possible, andalso) inference-closed (i.e., if P ⊆ Q and P ∈ LS (σ0 , . . . , σn ),then Q ∈ LS (σ0 , . . . , σn )) and conjunctive (i.e., if P, Q ∈LS (σ0 , . . . , σn ) then (P ∩ Q) ∈ LS (σ0 , . . . , σn )). Hence, forany doxastic agent L and every consistent data sequence ~σ ,~σthe belief operator BL(as defined above) satisfy the usual
KD45 axioms of doxastic logic.
A standard agent is a doxastic agent L whose beliefs forma principal filter, i.e., all her beliefs are entailed by one“strongest belief”; formally, a doxastic agent L is standard iff
for every data sequence ~σ over any epistemic space S there
exists some set LS (~σ ), such that
LS (~σ ) = {P ⊆ S | LS (~σ ) ⊆ P }.
It
T is easy to see that in this case, we must have LS (~σ ) =LS (~σ ). Indeed, we canT equivalently define a doxastic agent
L to be standard iff LS (~σ ) ∈ LS (~σ ) holds for all data sequences ~σT. Standard agents are globally consistentwheneverT
possible: LS (σ0 , . . . , σn ) 6= ∅ whenever ni=o σi 6= ∅.
Traditional learning methods in Formal Learning Theory correspond to our standard agents, and they are typically identified with the map L (given by LS (σ0 , . . . , σn ) :=TLS (σ0 , . . . , σn )). From now on we follow this tradition,and refer to standard agents using the map L. But in general we do not restrict ourselves to standard agents.
An AGM agent is an agent L≤ who forms beliefs by AGM
conditioning, i.e., it comes endowed with a map that associates any epistemic space S some total preorder4 ≤S on S,
called “prior” plausibility relation; and whose beliefs after
observing any data sequence ~σ = (σ0 , . . . , σn ) are given byσ ) := {P ⊆ S | ∃s ∈
L≤
S (~

n
\

i=0

σi ∀t ∈n\i=0

σi (t ≤ s ⇒ t ∈ P )}.

Intuitively, t ≤ s means that t is at least as plausible ass (according to our agent). So, an AGM agent believes P
conditional on a data sequence ~σ iff P is true in all the states(consistent with the data) that are “plausible enough”.
It is easy to see that every AGM agent
Tn is a doxastic agent:
L≤
(~
σ
)
is
a
properfilterwheneverS
i=0 σi 6= ∅; hence, thebeliefs of an AGM agent satisfy the usual KD45 axioms ofdoxastic logic (when learning any consistent data sequence).
Moreover, it is well-known that in fact, the beliefs of AGM
agents satisfy all the so-called AGM axioms from Belief Revision Theory [1]: if, for any data sequence ~σ = (σ0 , . . . , σn ),we set T = L(σ0 , . . . , σn ), and for any new observationφ ∈ O we set T ∗ φ = L(σ0 , . . . , σn , φ), then the resulting
revision operator ∗ satisfies all the AGM postulates. In fact,~σ
for any AGM agent L, if we interpret the operator BL(as defined above) as representing a conditional belief B σ0 ∧...∧σn ,then the sound and complete logic of these conditional belief
4
A total preorder on S is a binary relation ≤ on S that isreflexive, transitive, and connected (i.e., for all s, t ∈ S, wehave either s ≤ t or t ≤ s).

67

operators is the so-called Conditional Doxastic Logic [8, 5](which is itself just a repackaging of the AGM postulates in
the language of conditional logic).
Observation 1. Given a total preorder ≤ on S and asubset A ⊆ S, set
M in≤ (A) := {s ∈ A | s ≤ t for all t ∈ A}
for the set of ≤-minimal states in A. Let
T ~σ = (σ0 , . . . , σn )∅. Thenbe any data sequence such that M in≤ ( ni=0 σi ) 6= T
n
L≤
(~
σ
)
is
the
principal
filtergenerated
by
M
in
(
≤
S
i=0 σi ),i.e., we haveL≤
S (σ0 , . . . , σn ) := {P ⊆ S | M in≤ (

n
\

i=0

σi ) ⊆ P }.

In general though, the filter L≤σ ) is not principal. So
S (~AGM agents are not necessarily standard agents. But there
is an important case when they are standard: whenever the
preorder ≤S is well-founded in every space S (i.e., there are
no infinite chains s0 > s1 > s2 . . . of more and more plausible
states). It is easy to see that the map L associated to a
standard AGM agent is given by the set of ≤-minimal statesconsistent with the data:
n
\
L≤
σi ).
S (σ0 , . . . , σn ) := M in≤ (i=0Intuitively, this means that a standard AGM agent believes
a proposition P iff P is true in all the “most plausible” statesconsistent with the data.
The original semantics of AGM belief was given using only
standard AGM agents. But this semantics was in fact borrowed by Grove [18] from Lewis’ semantics for conditionals[27], which did not assume well-foundedness.5

Definition 4. Let S be an epistemic space. An agent L
verifies a proposition A ⊆ S in the limit if, for every state~ for s, we have s ∈ A iff theres ∈ S and every data stream O~
exists some k ∈ ω such that A ∈ LS (O[n])for all n ≥ k. For~
standard agents, this means that LS (O[n])⊆ A for all n ≥ k.
A set A ⊆ S is verifiable in the limit if there exists someagent that verifies A in the limit.6
An agent L falsifies a proposition A ⊆ S in the limit if,~ for s,for every state s ∈ S and every data stream for O
we have s ∈/ A iff there exists some k ∈ ω such that Ac ∈~L(S, O[n])⊆ Ac for all n ≥ k. (Here, as in the rest of thiscpaper, X := S \ X stands for the complement of X.) For a~standard agent, this means L(S, O[n])⊆ Ac for all n ≥ k,
A proposition A ⊆ S is falsifiable in the limit if there
exists some agent that falsifies A in the limit.
A proposition A ⊆ S is decidable in the limit if it is bothverifiable and falsifiable in the limit.
An agent L solves a problem P = (S, Q) if, for every state~ for s, there exists some k ∈ ωs ∈ S and every data stream O~
such that As ∈ LS (O[n])for all n ≥ k. (Recall that As istrue answer to Q at s.) For a standard agent, this means~that LS (O[n])⊆ As for all n ≥ k. A problem is solvable (inthe limit) if there exists some agent that solves it.
5

Indeed, Lewis’ definition of conditionals has a similar shape
to our above definition of (conditional) beliefs for nonstandard AGM agents.
6
For a discussion of the relationship between verifiability
and learnability see, e.g., [19, 12].

An epistemic space S = (S, O) is learnable (by an agentL) if the (problem given by the) learning question QS ={{s} | s ∈ S} is solvable (by L).
All the above notions have a standard counterpart, e.g.,
A is standardly verifiable if there exist some standard agent
that verifies it; P is standardly solvable if it can be solved
by some standard agent, etc.
Note that standard learnability is essentially the same asGold’s identifiability in the limit [30, 17].
Examples and Counterexamples: An example of nonlearnable space S = (S, O) is obtained by taking four abstract states S = {s, t, u, w} and two observable properties
O = {V, U }, with V = {s, t, u} and U = {t, u, w}, as depicted in Figure 2. Since states s and t satisfy the sameobservable properties, no learning method will ever distinguish them.
V

U
s

u

t

w

Figure 2: A non-learnable space
But even spaces in which no two states satisfy the sameobservations can still be non-learnable, e.g., all the ndimensional Euclidean spaces from Example 1 are not learnable (though, as we will see, many questions are solvable andmany subsets are decidable over these spaces). Another example of non-learnable space is given in Figure 3: formally,S = (S, O), where S := {sn | n ∈ ω} ∪ {s∞ }, and O ={Oi | i ∈ ω}, and for any i ∈ ω, Oi := {si , si+1 , . . .} ∪ {s∞ }.

O0

O1
s0

O2
s1

O3
s2

s3

...

In contrast, an example of learnable space is in Figure 4: formally, S = {sn | n ∈ ω} consists of countably
many distinct states, with O = {On | n ∈ ω}, where
On = {s0 , s1 , s2 , . . . , sn }. A standard agent that can learn

O0

O1
s1

O2
s2

O3
s3

O4
s4

Proposition 1. Let S be an epistemic space, A ⊆ S a
proposition and P = (S, Q) an inductive problem. Then we
have the following:• A is verifiable (falsifiable, decidable) in the limit iff
it is standardly verifiable (falsifiable, decidable) in thelimit.
• P is solvable iff it is standardly solvable.
• S is learnable iff it is standardly learnable.

Proof. Let A ⊆ S be a set that is verifiable (falsifiable,
decidable) by an agent L on an epistemic space S. We construct a standard agent that does the same thing, by setting,for every data sequence ~σ ∈ O∗ : LS (~σ ) := A if A ∈ LS (~σ ),LS (~σ ) := Ac if A 6∈ LS (~σ ) but Ac ∈ LS (~σ ), and LS (~σ ) := Sotherwise. Also, on any other space S0 = (S 0 , O0 ), we set bydefault LS0 (~σ 0 ) := S 0 .
Similarly, let P = (S, Q) be a problem that is solvable
by L. Let ≤ be some arbitrary well-order of the set Q.
(Such a well-order exists, by the Well-Ordering Theorem.)
We construct a standard agent who also solves P, by setting
LS (~σ ) := A if A is the first answer in Q (according to ≤)such that A ∈ LS (~σ ) holds; and LS (~σ ) := S if no such answer
exists. (As before, we can extend our agent to any otherspace S0 = (S 0 , O0 ), by setting LS0 (~σ 0 ) := S 0 .)By applying this to the learning problem Q = {{s} | s ∈S}, we obtain the similar result for learnability.
In conclusion, everything that can be learned by any agent
can also be learned by some standard agent. However, thisis no longer true when we restrict to more canonical typesof agents (such as AGM agents).
Proposition 2. There exist spaces that are learnable, but
not learnable by standard AGM agents. (Hence, there exist
solvable problems that are not solvable by standard AGM
agents.)s∞

Figure 3: Another non-learnable spaces0

to
Tn be the maximum number (in the natural order) in
i=0 σi , whenever there
Tis such a maximum number, andsettingL(σ1 , . . . , σn ) := ni=0 σi otherwise.

...

Figure 4: A learnable space

Proof. Here is a counterexample from [13, 3, 4]. Take
the epistemic model from Figure 4. This space is learnable, and thus learnable by AGM conditioning, but it is notlearnable by standard conditioning. Indeed, this space islearnable by conditioning only with respect to the following
non-wellfounded prior: s0 > s1 > . . . > sn > sn+1 > . . .

4.

THE OBSERVATIONAL TOPOLOGY

In this section, we assume familiarity with the following
notions: topology τ (identified with its family of open subsets) over a set S of points, topological space (S, τ ), open
sets, closed sets, interior Int(X) and closure X of a set X,(open) neighborhood of a point s, base of a topology andlocal base (of neighborhoods) at a point. We use letters U ,U 0 , etc., for open sets in τ , and letters C, C 0 , etc., for closedsets.
A space is said to be second-countable if its topology hasa countable base. Given a topological space (S, τ ), the specialization preorder v ⊆ S × S is defined in the following
way: for any s, t ∈ S, we set
s v t iff ∀U ∈ τ (s ∈ U ⇒ t ∈ U ).

this space in the limit is given by setting L(σ1 , . . . , σn )68Separation Principles. In this paper we use four key topological separation notions. The first is the well-known separation axiom T 0, which will be satisfied by all the topologies
that arise in our setting. The second is the separation axiom T D. This condition (together with countability) will beshown to characterize learnable spaces. The next two notions are analogues of T D separation for questions. Insteadof asking for open sets that separate points (states), these
conditions require the existence of open sets that separate
answers (to the same question). The concept of locally closed
questions is a first analogue of T D, and it will be shown tocharacterize in some sense solvable problems. Finally, thenotion of linearly separated questions is a stronger analogueof T D for questions, which characterizes a stronger typeof solvability, what we will call direct solvability by (AGM)
conditioning.
Definition 5. A topological space (S, τ ) satisfies the separation axiom T 0 if the specialization preorder is actually apartial order, i.e., it is antisymmetric: s v t v s impliess = t. Equivalently, if s 6= t, then there exists some “separating” open U , such that either s ∈ U , t 6∈ U , or s 6∈ U ,t ∈ U.
The space (S, τ ) satisfies the separation axiom T D iff forevery point s ∈ S, there is an open Ux 3 x such that y 6v x
for all y ∈ Ox \ {x}. Equivalently: for every s ∈ S there isan open U ∈ τ such that {s} = U ∩ {s}.
Essentially, T 0 says that every two points s 6= t can beseparated (by an open U ) one way or another (i.e., either s ∈U , t 6∈ U , or s 6∈ U , t ∈ U ), while T D essentially says thatevery point s can be separated (by an open neighborhood)from all the points t 6= s that are inseparable from s.7
Definition 6. Given a topological space (S, τ ), a set A ⊆S is locally closed if it is the intersection A = U ∩ C of anopen set U with a closed set C. Equivalently, if it is of theform A = U ∩ A for some open U .
A set is ω-constructible if it is a countable union of locallyclosed sets.
A question Q (partition of S) is locally closed if all its
answers are locally closed. A problem P is locally closed if
its associated question is locally closed.
Essentially, locally closed questions are partitions with theproperty that every “answer” (i.e., partition cell) A can beseparated (by an open neighborhood) from all the non-Astates that are inseparable from A.8
Definition 7. A question Q is linearly separated if there
existsS some total order  on the answers in Q, such thatA ∩ BA B = ∅. In other words, every answer A can beseparated (by some open UA ⊇ A) from the union of all theprevious answers: UA ∩ B = ∅ for all B  A.
Essentially, a linearly separated question is one whose answers can be totally ordered by a “plausibility” (or “simplicity”) order, in such a way that every answer A can be separated (by an open neighborhood UA ⊇ A) from all answers
that are “more plausible” (or “simpler”) than A.
7

A point y is “inseparable” from x if every open neighborhood of y contains x, i.e. y and x are in the topologicalrefinement order y v x.
8Here, a state t is said to be “inseparable” from a set A if
there is no open neighborhood U 3 t that is disjoint from A.
69

Definition 8. The observational topology τS associatedwith an epistemic space S = (S, O) is the topology generated
by O (i.e., the smallest collection of subsets of S, that includes O ∪ {∅, S} and is closed under finite intersections andarbitrary unions).
From now on, we will always implicitly consider our epistemic spaces S to also be topological spaces (S, τS ), endowed with their observational topology τS . Every topological property possessed by the associated topological space
will thus be also attributed to the epistemic space.
Observation 2. Every epistemic space is T 0 and secondcountable. A (sound and complete) data stream for s is thesame as a local neighborhood base at s.
Proposition 3. Every ω-constructible set can be written
as a disjoint countable union of locally closed sets.
Proof. In order to prove this, we first recall some standard topological notions and results: A set is called constructible if it is a finite disjoint union of locally closed sets.
Obviously, all locally closed sets are constructible. It is
known that constructible sets form a Boolean algebra, i.e.,
the family of constructible sets is closed under complementation, finite unions,
S and finite intersections.
Suppose A = i∈ω Ai , where all Ai are locallyS closed.
Then we can rewrite A as a disjoint
union A = T i∈ω Bi ,S
where we have set Bi = Ai \ ( k<i Ak ) = Ai ∩ k<i Ack ,for every i. Since Bi ’s are generated from locally closed setsusing complementation and finite intersections, they must
be constructible. Hence, each Bi can beSwritten as disjoint
finite unions of locally closed sets Bi = 1≤j≤i Bij . Hence,SS
we can write A = i∈ω 1≤j≤i Bij as a disjoint countable
union of locally closed sets.
Definition 9. A pseudo-stratification is a finite or ωlong sequence of locally closed sets hAi | i < λi (where λ ∈ω ∪{ω}), which form a partition of S satisfying the following
condition:
if j < i then either Ai ∩ Aj = ∅ or Ai ⊆ Aj .
Proposition 4. Every countable locally closed question
can be refined to a pseudo-stratification.
Proof. Suppose Π = {Ai | i ∈ ω} is a countable locallyclosed question (partition of S). We first show the following:
Claim. There exists a family {(Πi , <i ) | i ∈ ω}, satisfying(1) each Πi is a finite partition of Ai into locally closed
sets;(2) each <i is a total order on Πi ;(3) if j < i, E ∈ Πj , B ∈ Πi , then either B ⊆ E or
c
B⊆E ;
c

(4) if B, E ∈ Πi , E <i B, then B ⊆ E .
Proof of Claim: We construct (Πn , <n ) by recursion: forn = 0, set Π0 := {A0 }, with <0 trivial. For the step n + 1:
assume given {(Πi , <i ) | i ≤ n} satisfying the above four
conditions (for i ≤ n). We set
Πn+1 := {Bf | f :

n
[

i=1

Πi → {0, 1}},

S
where for each function f : n
i=1 Πi → {0, 1} we have set\\ c
Bf := An+1 ∩ {E | E ∈ f −1 (0)} ∩ {E | E ∈ f −1 (1)}.

It is obvious that the Bf ’s are locally closed (given that An+1
is locally closed) and that they form a partition of An+1 . So
condition (1) is satisfied.
It is also easy to check condition (2) for i = n + 1: let
j < n + 1, E ∈ Πj and Bf ∈ Πn+1 . Then we have eitherf (E) = 0, in which case Bf ⊆ E (by construction of Bf ), orcelse f (E) = 1, in which case Bf ⊆ E .
To construct the order <n+1 , observe first thatSthere isa natural total order <(n) on the disjoint union n
i=1 Πi ,namely the one obtained by concatenating theSorders <0 ,<1 , . . . , <n . (More precisely, if, for every B ∈ n
i=1 Πi , we
set i(B) to be the unique index i ≤ n such that B ∈ Πi ,then the order <(n) is given by setting: B <(n) E iff either
i(B) < i(E), or else i(B) = i(E) and B <i(B) E.)
Now, the order <n+1 on Bf ’s is given by the lexicographic
order induced by <(n) on the functions f (thought as “words”
written with the letters 0 and 1). More precisely, we set:
Bf <n+1 BgS
iff there exists some set E ∈ n
i=1 Πi such that
∀E 0 <(n) E f (E 0 ) = g(E 0 ), but f (E) < g(E) ,where < is the usual order 0 < 1 on {0, 1}. Clearly, <n+1 is
a total order on Πn+1 , so condition (2) is satisfied.
Finally, we check condition (4) for n+1, let Bf , Bg ∈ Πn+1
such that Bf <n+1 Bg . By definition ofS the order <n+1 ,this means that there exists some E ∈ n
i=1 Πi such thatfor all E 0 <(n) E we have f (E 0 ) = g(E 0 ) but f (E) < g(E),i.e., f (E) = 0 and g(E) = 1. By the construction of Bf ’s,
f (E) = 0 implies that Bf ⊆ E, from which we get Bf ⊆ E,ccand thus E ⊆ Bf . Similarly, g(E) = 1 implies that Bg ⊆ccc
E . So we have Bg ⊆ E ⊆ Bf , and thus by transitivity ofcinclusion we get Bg ⊆ Bf . This completes the proof of ourClaim.
Given now the above Claim, we can prove our Lemma bytaking as our refined partition
[
Π0 :=
Πi .
i∈ω

0

Clearly, Π is a refinement of Π consisting of locally closed
sets. We now define a well-order <0 on Π0 as the concatenation of all the ≤i ’s.9 Obviously, <0 is a total order of type≤ ω on Π0 , so we get finite or ω-long sequence that enumerates Π0 . The above properties (3) and (4) ensure that thisis a pseudo-stratification.
Lemma 1. Given a pseudo-stratification hAi | i < λi (oflength λ ≤ ω), there exists a λ-long sequence of open setshUi | i < λi, satisfying:(1) Ui ∩ Ai = Ai ;(2) if j < i and Ui ∩ Aj 6= ∅, then Ai ⊆ Aj .
9Once again, one can specify this more precisely by first
defining i : Π0 → ω by choosing i(B) to be the unique index
i such that B ∈ Πi , and finally defining: B <0 E iff either
i(B) < i(E), or else i(B) = i(E) and B <i(B) E.

Proof. We know that each Ai is locally closed, so there
exists some open set U Ai ∈ τ such that U Ai ∩ Ai = Ai .
Now, for all i ∈ ω set\ccUi := U Ai ∩ {Aj | j < i, Ai ⊆ Aj }.

Let us first check that the sequence hUi | i < λi satisfies
condition (1):\ccUi ∩ Ai = (U Ai ∩ {Aj | j < i, Ai ⊆ Aj }) ∩ Ai
= (Ui ∩ Ai ) ∩= Ai ∩\\cc{Aj | j < i, Ai ⊆ Aj }cc

{Aj | j < i, Ai ⊆ Aj } = Ai

Second, let us check condition (2): Suppose that we havej < i and Ui ∩ Aj 6= ∅, but Ai 6⊆ Aj . Since (Ai )i<λ is a
pseudo-stratified sequence, from j < i and Ai 6⊆ Aj we cancderive Ai ⊆ Aj . By the construction of Ui , this implies thatcccUi ⊆ Aj , and hence that Ui ∩Aj ⊆ Aj ∩Aj ⊆ Aj ∩Aj = ∅,which contradicts the assumption that Ui ∩ Aj 6= ∅.
Lemma 2. Every pseudo-stratification is linearly separated.
Proof. Let Π = {Ai | i < λ} be a pseudo-stratification(with λ ≤ ω), and let hUi | i < λi be a sequence satisfying
the conditions of Lemma 1. It is clear that, in order to prove
our intended result, it is enough to construct a total order on the set {i ∈ ω|i < λ} = λ ⊆ ω, such thatUi ∩ Aj 6= ∅ ⇒ i  j.
For this, we first define a reflexive relation R on λ, bysettingiRj ⇐⇒ Ui ∩ Aj 6= ∅.
Claim: There are no non-trivial cycles
i1 R · · · in Ri1 (with distinct ik ’s).
Proof of Claim: Let i1 R · · · in Ri1 be a non-trivial cycle ofminimal length n ≥ 2. There are two cases:
Case 1: n = 2, i.e., i1 Ri2 Ri1 with i2 6= i1 . We must have
either i1 < i2 or i2 < i1 . Without loss of generality, we canassume i1 < i2 (otherwise, just swap i1 and i2 , and use thecycle i2 Ri1 Ri2 ). From i2 Ri1 , we get Ui2 ∩ Ai1 6= ∅. This
together with i1 < i2 , gives us Ai2 ⊆ Ai1 (by condition (2)from Lemma 2), and hence Ui1 ∩ Ai2 ⊆ Ui1 ∩ Ai1 = Ai1 .
From this, we get that Ui1 ∩ Ai2 = (Ui1 ∩ Ai2 ) ∩ Ai2 ⊆Ai1 ∩ Ai2 = ∅ (since i1 6= i2 , so Ai1 and Ai2 are different
answers, hence disjoint), so we conclude that Ui1 ∩ Ai2 = ∅.
But on the other hand, from i1 Ri2 we get Ui1 ∩ Ai2 6= ∅.
Contradiction.
Case 2: n > 2. Since all the ik ’s are distinct, there must
exist a (unique) smallest index in the cycle. Without lossof generality (since otherwise we can rearrange the indices,permuting the cycle), we can assume that i3 is the smallest
index. (Note that, since n > 2, there must be at least three
distinct successive indices i1 , i2 , i3 .) So i3 < i1 and i3 < i2 .
From i2 Ri3 we get Ui2 ∩ Ai2 6= ∅. Since i3 < i2 , it follows
that Ai2 ⊆ Ai3 (by Lemma 2). But on the other hand, i1 Ri2gives us Ui1 ∩Ai2 6= ∅. We hence obtain Ui1 ∩Ai3 6= ∅. This,
together with i3 < i1 , gives us Ai1 ⊆ Ai3 (again by Lemma
2). From this, we derive Ai1 ⊆ Ui1 ∩ Ai3 (since Ai ⊆ Ui for70all i). Let now s ∈ Ai1 be any state satisfying the answer
Ai1 ⊆ Ui1 ∩ Ai3 . So we have s ∈ Ui1 and s ∈ Ai3 , which
together imply that Ui1 ∩ Ai3 6= ∅ (since s ∈ Ai3 implies
that every open neighborhood of s intersects Ai3 ). Hence,
we have i1 Ri3 , which means we can shorten the cycle byeliminating i2 , we obtain contradiction.
Given the above Claim, it follows that the transitive closure R∗ is a partial order on λ (which obviously includes R).
By the Order Extension Principle, we can extend R∗ to a
total order  on λ, which still includes R.

5.

TOPOLOGICAL
CHARACTERIZATION OF SOLVABILITY

Definition 10. Let S = (S, O) be an epistemic space, L
be a standard agent, A ⊆ S, and s ∈ A. An A-locking
sequence for s (with respect to L) is a data sequence σ =(O1 , . . . , Ok ), such that:T(1) σ is sound for s, i.e., s ∈ 1≤i≤k Oi ;(2) if δ is any data sequence sound for s, then L(S, σ ∗δ) ⊆A.

For a given data sequence σ, we denote by LσA the set of
all states in A having σ as an A-locking sequence, i.e.,LσA := {s ∈ A | σ is an A-locking sequence for s wrt L}.
Lemma 3. If
S A is verifiable in the limit by a standard
agent L, then σ∈O∗ LσA = A.

Proof. Suppose
not. Let A be verifiable in the limit, but
Ssuch S
that A 6= σ∈O∗ LσA . Since all LσA ⊆ A, his means that
A 6⊆ σ∈O∗ LσA , i.e., there exists some state s ∈ A for which
there is no A-locking sequence. This means that every data
sequence σ that is sound for s can be extended to a sequenceδ that is also sound for s and has L(δ) 6⊆ A.
~ be a (sound and complete) data stream for
Let now O~ , by definings. We construct a new infinite data stream V~ in countably
increasingly longer initial segments δk of O,
many stages: we first set V0 = O0 , thus obtaining an initial
segment δ0 = (O0 ) = (V0 ); at the k + 1-th stage, given some
initial segment δk = (V0 , V1 , . . . , Vnk ) (of some length nk ),we built our next initial segment by taking any extension
δk+1 of the sequence σk = (V0 , . . . , Vnk , On+1 ) that is sound~ isfor s and has L(σk ) 6⊆ A. The resulting infinite stream V~a (sound and complete) stream for s (the completeness of Vwith respect to s follows the fact that this stream includes all~ ), but which contains arbitrarily long initialthe elements of U
segments σk with L(σk ) 6⊆ A. Since s ∈ A, this contradicts
the assumption that A is verifiable in the limit.
Lemma 4. If A ⊆ S is verifiable in the limit by a standard
agent L, then for every data sequence σ = (O1 , . . . Ok ), theset LσA is locally closed.
T
Proof. Let O := ki=1 Oi be the intersection of all theobservations in σ. We will show thatO ∩ LσA = LσA ,from which the desired conclusion follows.
(⊇) If s ∈ LσA , then σ is an A-locking
sequence for s, henceTσ is sound for s, and thus s ∈ n
i=1 Oi = O.
71(⊆) Suppose that s ∈ O ∩ LσA . We prove two claims:
Claim 1: For every data sequence δ that is sound for s andextends σ, we have LS (δ) ⊆ A.
Proof of Claim 1 : Let δ = (δ1 , . . . , δn ) be a data sequencethat is sound for s (i.e., s ∈ δi for all i = 1, . . . , n) andextendsσ, i.e., n ≥ k and Ui = Oi for all i ≤ k). Hence,Tnof s, and s ∈ LσA , so there
i=1 δi is an open neighborhood
T
δ
such
that
t ∈ LσA . Hence, t ∈ A
must exist some t ∈ nii=1and σ is an A-locking sequence for t. But δ extends σ andis sound for t, so (by the definition of σ being an A-locking
sequence for t), we have that L(δ) ⊆ A, which concludes theproof of Claim 1.
Claim 2: We have s ∈ A.
~ be a stream for s that extends
Proof of Claim 2 : Let V
σ (such a stream must exist, since σ is sound for s: justtake any stream for s and prefix it with σ). Then, for everyn ≥ k, the sequence δn = (V1 , . . . , Vn ) is sound for s andextends σ. Hence, by the above Claim, we must have that
LS (V1 , . . . , Vn ) ⊆ A for all n ≥ k. But we assumed that
A is verifiable in the limit, so we must have s ∈ A, whichconcludes the proof of Claim 2.
From Claims 1 and 2 together, we conclude that σ is an
A-locking sequence for s ∈ A, hence s ∈ LσA .
Theorem 1. Given an epistemic space (S, O), a set A ⊆S is verifiable in the limit iff it is ω-constructible.
S
Proof. (⇐) Assume A = n (Un ∩ Cn ) is a countabledisjoint union of (mutually disjoint) locally closed sets Un ∩Cn (with Un open and Cn closed). We define a standard
agent L for A on finite data sequences
T δ = (O1 , . . . , Ok ),by setting L(S, δ) = Ac , if we have j Oj 6⊆ Un for alln T∈ ω; L(S, δ) = Ac (where Ac is the complement of A),
if j Oj ⊆ Cnc holds for the first index n ∈ ω such thatT
j Oj ⊆ Un ; and L(S, δ) = A otherwise. Then it is easy to
see that L verifies A in the limit.
(⇒) Suppose that A is verifiable in the limit. By Proposition 1, it is then verifiable by a standard agent L. By Lemma
1, A is the union of all sets LσA for all finite data sequences
σ. But there are only countably many such sequences, so
this is a countable union. Moreover, by Lemma 2, each LσA
is locally closed. Hence A is a countable union of locallyclosed sets, i.e., an ω-constructible set.
Corollary 1. A is decidable in the limit iff both A and
Ac are ω-constructible.
Proof. Follows trivially from the above results.
Theorem 2. Let P = (S, Q) be an inductive problem on
an epistemic space S. The following are equivalent:(1) P is solvable (in the limit);(2) the associated question Q is an (at most) countablefamily of ω-constructible answers;(3) Q has an (at most) countable locally closed refinement.

Proof. (1) ⇒ (2) : Let P be a solvable problem. By
Proposition 1, there exists some standard agent that solves
it. Let L be such a standard agent that solves P.
Claim: Every answer A ∈ Q is verifiable in the limit.
Proof of Claim: Let A ∈ Q be an answer. We constructa standard agent LA that verifies it, by setting LA
S (σ) := Ac
iff LS (σ) ⊆ A, and LA
S (σ) := A otherwise. It is easy to seeAthat L verifies A.

Using the Claim and Lemma 3, we obtain that, for eachanswer A ∈ Q, there exists some data sequence σ ∈ O∗ suchthat LS (σ) ⊆ A. But O∗ is countable, so there can be only
countably many answers in Q.
By the claim above, Lemma 3 and Lemma 4, we obtainthat every answer A ∈ Q is a countable union of locallyclosed sets, hence it is ω-constructible.
(2) ⇒ (3) : By (2), Q is (at most) countable, say Q ={Ai | i ∈ ω}, and also each answer AI ∈ Q is ω-constructible,
hence it can be written
as a countable disjoint union of loS
cally closets A = k∈ω Aki (where all Aki ’s locally closed andmutually disjoint). Then the question {Aki | i ∈ ω, k ∈ ω} isa refinement of Q, which is countable and locally closed.
(3) ⇒ (1) : Let Q0 = {Bi | i ∈ ω} be a countable closed
refinement of Q0 . By Corollary 1, every answer B ∈ Q0 is
decidable, and so by Proposition 1, we can choose for eachBi ∈ Q some standard agent Li that decides Bi . We define
now a new standard agent L, by:[LS (σ) := {Bi | i ∈ ω such that Li (σ) ⊆ Bi }.
It is easy to see that this agent L solves Q0 , and since Q0 is
a refinement of Q, L also solves Q.

Corollary 2. An epistemic space S = (S, O) is learnable in the limit iff it is countable and satisfies the T D separation axiom.
Proof. Apply Theorem 2 to the learning question{{s} | s ∈ S}, noticing that the fact that all its answers
are ω-constructible is equivalent to all singletons being locally closed, which is just another formulation of the T D
axiom.

6.

UNIVERSALITY OF CONDITIONING

Our aim in this section is to show that AGM conditioning
is “universal”: every solvable problem can be solved by some
AGM agent. First, we introduce an auxiliary notion, thatof a problem being directly solvable by AGM conditioning.
Given a question Q on an epistemic space (S, O), any totalorder  ⊆ Q×Q on (the answers of) the question Q induces
in a canonical way a total preorder ≤⊆ S × S, obtained by:s ≤ t iff As  At
(where As is the unique answer As ∈ Q such that s ∈ As ).

Definition 11. A problem P = (S, Q) is directly solvableby conditioning if it is solvable by AGM conditioning withrespect to (a prior ≤ that is canonically induced, as explained
above, by) a total order  ⊆ Q × Q on (the answers of ) thequestion Q.
Direct solvability by conditioning essentially means thatthe problem can be solved by a conditioning agent who does
not attempt to refine the original question: she forms beliefs
only about the answers to the given question, and is thus
indifferent between states satisfying the same answer. Direct
solvability by conditioning is thus a very stringent condition,and unsurprisingly this form of conditioning is not universal.
Proposition 5. ( K. Genin, personal communication)
Not every solvable problem is directly solvable by conditioning.

Proof. Let P be the problem in Example 3, depicted on
the left-hand side of Figure 1. It is easy to see that this problem cannot be directly solvable by conditioning! (Indeed, if{t, u}  {s, v} then v is not learnable by -conditioning; if{s, v} < {t, u} then t is not learnable by -conditioning;
while if {t, u} and {s, v} are equally plausible, then neithert nor v are learnable.)
But P can be refined to a directly solvable problem,
namely the “learning question” P0 (depicted on the righthand side of Figure 1), which can be directly solvable (e.g.
if we set {t}  {s}  {v}  {u}). As a consequence, P canitself be solved by (non-direct) conditioning (with respect tothe order t < s < v < u).
This counterexample suggests a way to prove our intended
universality result: it is enough to show that every solvableproblem has a refinement that is directly solvable by conditioning. To do this, we first need a structural characterization of direct solvability.
Lemma 5. (Topological Characterization of Direct Solvability by Conditioning) A problem P = (S, Q) is directly
solvable by conditioning iff Q is linearly separated.

Proof. Left-to-right implication: Suppose that P is directly solvable by conditioning with respect to (a prior ≤that is canonically induced by) a total order  ⊆ Q × Q.
Then, for every s ∈ S choose some sound and complete data
~ S = (Ons )n∈ω for s (with Osn ∈ O ⊆ τS ). Direct
stream O
solvability by conditioning implies then that there existss) ⊆ As . Set Us :=some Ns such that M in≤ (O1s , . . . , ONs
TNs s
i=1 Oi ∈ τS , soSthat we have s ∈ Us and M in≤ Us ⊆ As .
Then set UA := s∈A Us ∈ τS for every answer A ∈ Q. We
claim that UA “separates” A from the union of all the answers B  A (as linear separation demands): indeed, by theconstruction of UA , it is obvious that (1) A ⊆ UA , and alsothat M in≤ UA ⊆ A. By unfolding the last clause in termsof , we obtain that: A  B holds for all B ∈ Q such thatUA ∩ B 6= ∅. Since  is a total order on Q, this is equivalentto: (2) UA ∩ B = ∅ for all B  A. By (1) and (2) together,
we obtain that Q is linearly separated.
Right-to-left implication: Suppose Q is linearly separated.
Let  be a total order on Q that linearly separates it. Thismeans that, for every answer A ∈ Q, there exists some open
set UA ∈ τS such that A ⊆ UA and UA ∩ B = ∅ for all B  A.
For each s ∈ S, we set Us := UAs (where As is the uniqueanswer As ∈ Q with s ∈ As ).
Let ≤ be the total preorder on S canonically induced by
the order  ⊆ Q × Q (by s ≤ t iff As  At ). We show
now that P is directly solvable by conditioning with respect~ = (On )n∈ω
to ≤. For this, let s ∈ S be any state, and O
be a sound and complete stream for s. Completeness of thestreamimplies that there must exist some N ∈ ω such thatTNO
⊆ Us .
ii=1
To conclude our proof, it is enough to show the following
Claim: For every n ≥ N , we haves ∈ M in≤ (

n
\

i=1

Oi ) ⊆ As .

First, let us see why this Claim is enough to give us
direct T
solvability by conditioning. TThe fact that s ∈nM in≤ ( ni=1 Oi ) implies that M in≤ ( i=1 Oi ) 6= ∅, for alln ≥ N . A previous observation tells us that, when applied72

to such data streams, the AGM agent L≤ produces a “principal filter”, given byL≤ (O1 , . . . , On ) = {P ⊆ S | M in≤ (

n
\

i=1

Oi ) ⊆ P }.

T
By the Claim above we have M in≤ ( ni=1 Oi ) ⊆ As , andhence we obtain As ∈ L≤ (O1 , . . . , On ), for all n ≥ N .
Proof of Claim: Let n ≥ N . To prove the Claim, it isenough to show the following two implications (for all statest):T(1) t ∈ ni=1 Oi ⇒ s ≤ t;
T(2) t ∈ M in≤ ( ni=1 Oi ) ⇒ At = As .
Tn
Then t ∈ Us (since
To show (1), let t ∈i=1 Oi .
TN
Tn
O
⊆
U
),
so
U
∩
At 6= ∅. Hence (by
O
⊆
i
s
s
i
i=1
i=1
linear separation) we must haveTAs  At , i.e., s ≤ t.
n
To show (2), let
Tnt ∈ M in≤ ( i=1 Oi ). This implies thatt ≤ s (since s ∈ i=1 Oi ). But by (1), we also have s ≤ t,and hence s ≤ t ≤ s. This means that As  At  As . But 
is a total order on Q, so it follows that At = As .
Theorem 3. AGM conditioning is a universal problemsolving method, i.e., every solvable problem is solvable bysome AGM agent.
Proof. Let P be a solvable problem. From Theorem 2,
Proposition 3 and Lemma 2, it follows that P has a linearly
separated refinement P0 . By Lemma 5, that refinement is(directly) solvable by an AGM agent L≤ . It is obvious (fromthe definition of solvability) that any doxastic agent whichsolves the more refined problem P0 solves also the original
problem P.
Corollary 3. AGM conditioning is a universal learning method, i.e., every learnable space is learnable by some
AGM agent.
Proof. Apply the previous result to the finest question
Q := {{s} | s ∈ S}.
In contrast, recall that the counterexample in Proposition 2 showed that standard AGM agents have a very limited problem-solving power. Standard conditioning is not auniversal learning method (while general AGM conditioning
is universal). This means that allowing prior plausibility orders that are non-wellfounded is essential for achieving universality of conditioning. Beliefs generated in this way may
occasionally fail to be globally consistent. (Indeed, note thatin the counterexample from Proposition 2, the beliefs of thenon-standard AGM agent who learns the space are initiallyglobally inconsistent. In conclusion, occasional global inconsistencies are the unavoidable price for the universality of
AGM conditioning.

7. CONCLUSIONS AND CONNECTIONSTO OTHER WORKThe general topological setting for problem-solving assumed here is a variation of the one championed by Kelly invarious talks [23] and in unpublished work [24, 25], though
until recently we did not realize this close similarity. Our
topological characterizations of verifiable, falsifiable and decidable properties are generalizations of results by Kelly [19],73

who proved characterizations for the special case of Bairespaces.10 Our result on learning-universality (Corollary 3)
is also a generalization of analogue results by Kelly [21, 20],and Kelly, Schulte and Hendricks [26]. But our generalization to arbitrary spaces is highly non-trivial, requiring theuse of the T D characterization. (In contrast, the Baire spacesatisfies the much stronger separation axiom T 1, which trivializes the specialization order, and so the proof of learninguniversality is much easier in this special case: any total
ω-like ordering of the space can be used for conditioning.)
Nevertheless, in a sense, this result is just a topological repackaging of one of our own previous results [13, 3, 4].
While writing this paper, we learned that our T D characterization of learnability (Corollary 2) was independently reproven by Konstantin Genin ([11], unpublished manuscript),soon after we announced its proof. This characterization is
actually a topological translation of a classical characterization of identifiability in the limit [2], and in fact it also
follows from a result by de Brecht and Yamamoto [9], whoprove it for so-called “concept spaces”.
Our key new results are far-reaching and highly nontrivial: the topological characterization of solvability (Theorem 2), and the universality of AGM condition for problemsolving (Theorem 3). They required the introduction of new
topological concepts (e.g., pseudo-stratifications and linearly
separated partitions), and some non-trivial proofs of new
topological results.
Philosophically, the importance of these results is that,on the one hand they fully vindicate the general topologicalprogram in Inductive Epistemology started by Kelly andothers [19, 31], and on the other hand they reassert thepower and applicability of the AGM Belief Revision Theory
against its critics.
To this conclusion, we need to add an important proviso:
our results show that, in order to achieve problem-solving
universality, AGM agents need to (a) be “creative”, by going beyond the original problem (i.e., finding a more refinedproblem that can be solved directly, and forming prior beliefs about the answer to this more refined question), and(b) admit non-standard priors, which occasionally will lead
to beliefs that are globally inconsistent (although still locallyconsistent). Such occasional global inconsistencies can giverise to a type of “infinite Lottery Paradox”. But this is theprice that AGM agents have to pay in order to be able tosolve every solvable question.
Whether or not this is a price that is worth paying is adifferent, more vague and more “ideological” question, although a very interesting one. But this question lies beyond
the scope of this paper.

8.

ACKNOWLEDGMENTS

We thank Johan van Benthem, Nick Bezhanishvili, Konstantin Genin, Thomas Icard and Kevin Kelly for their useful feedback on issues related to this paper. Johan helped
us place belief-based learning within the larger context oflong-term doxastic protocols [7], and beyond this he gave
10

In unpublished work [25] the authors claim a characterization of solvability in a general setting. Their characterization is sightly “looser” than ours, and can be easily obtained
from ours. Our tighter characterization is the one needed
for proving universality.

us his continuous support and encouragement for our workon this line of inquiry. Nick pointed to us the connections
between our work and the notions of TD-space and locallyclosed set. Konstantin pointed to us the connections to thenotion of stratification and gave the counterexample proving
Proposition 5. His critical feedback on our early drafts wasreally essential for clarifying our thoughts and cleaning up
our proofs, and so it’s fair to say that this paper in its current form owes a lot to Konstantin Genin. Thomas Icard’scomments on a previous draft and our friendly interactions
with him on related topics during our Stanford visits arevery much appreciated. Finally, Kevin Kelly’s work formsof course the basis and the inspiration for ours. Our frequent discussions with him in recent years influenced thedevelopment of our own perspective on the topic. He also
gave us excellent reference tips concerning the history of theconnections between topology and formal epistemology, as
well as concerning his more recent work on related issues.
Nina Gierasimczuk’s work on this paper was funded by
an Innovational Research Incentives Scheme Veni grant 27520-043, Netherlands Organisation for Scientific Research(NWO). Sonja Smets was funded in part by an Innovational Research Incentives Scheme Vidi grant from NWO,
and by the European Research Council under the European
Community’s Seventh Framework Programme (FP7/20072013)/ERC Grant agreement no. 283963.

9.

REFERENCES[1] C. Alchourrón, P. Gärdenfors, and D. Makinson. On
the logic of theory change: Partial meet contraction
and revision functions. JSL, 50:510–530, 1985.
[2] D. Angluin. Inductive inference of formal languages
from positive data. Inform. Control, 45(2):117–135,
1980.
[3] A. Baltag, N. Gierasimczuk, and S. Smets. Belief
revision as a truth-tracking process. In K. Apt, editor,
Proc. of TARK’11, pages 187–190. ACM, 2011.
[4] A. Baltag, N. Gierasimczuk, and S. Smets. Truth
tracking by belief revision. Technical report, ILLC
Report PP-2014-20, 2015. To appear in Studia Logica.
[5] A. Baltag and S. Smets. A qualitative theory ofdynamic interactive belief revision. In G. Bonanno,
W. van der Hoek, and M. Wooldridge, editors, Proc.
of LOFT’7, number 3 in Texts in Logic and Games,
pages 9–58. Amsterdam University Press, 2008.
[6] J. v. Benthem. Logical Dynamics of Information andInteraction. Cambridge University Press, 2011.
[7] J. v. Benthem, J. Gerbrandy, T. Hoshi, and E. Pacuit.
Merging frameworks for interaction. JPL,
38(5):491–526, 2009.
[8] O. Board. Dynamic interactive epistemology. Gamesand Economic Behavior, 49(1):49–80, 2004.
[9] M. de Brecht and A. Yamamoto. Topologicalproperties of concept spaces (full version). Inform.
Comput., 208(4):327–340, 2010.
[10] C. Dégremont and N. Gierasimczuk. Finite
identification from the viewpoint of epistemic update.
Inform. Comput., 209(3):383–396, 2011.
[11] K. Genin. Linearizing a countable TD space.
Unpublished manuscript, 2015.
[12] N. Gierasimczuk. Identification through inductive
verification. In Proc of TBiLLC’07, volume 5422 of

LNCS, pages 193–205. Springer, 2009.
[13] N. Gierasimczuk. Knowing One’s Limits. Logical
Analysis of Inductive Inference. PhD thesis,
Universiteit van Amsterdam, The Netherlands, 2010.
[14] N. Gierasimczuk, D. de Jongh, and V. F. Hendricks.
Logic and learning. In A. Baltag and S. Smets,
editors, Johan van Benthem on Logical andInformational Dynamics. Springer, 2014.
[15] N. Gierasimczuk and D. de Jongh. On the complexityof conclusive update. The Computer Journal,
56(3):365–377, 2013.
[16] E. M. Gold. Limiting recursion. JSL, 30(1):28–48,
1965.
[17] E. M. Gold. Language identification in the limit.
Inform. Control, 10:447–474, 1967.
[18] A. Grove. Two modellings for theory change. JPL,
17:157–170, 1988.
[19] K. T. Kelly. The Logic of Reliable Inquiry. Oxford
University Press, Oxford, 1996.
[20] K. T. Kelly. Iterated belief revision, reliability, andinductive amnesia. Erkenntnis, 50:11–58, 1998.
[21] K. T. Kelly. The learning power of belief revision. In
Proc. of TARK’98, pages 111–124. Morgan Kaufmann
Publishers Inc., 1998.
[22] K. T. Kelly. Ockham’s razor, truth, and information.
In P. Adriaans and J. van Benthem, editors, Handbookof the Philosophy of Information, pages 321–359.
Elsevier, 2008.
[23] K. T. Kelly. An erotetic theory of empirical simplicity
and its connection with truth. Unpublishedmanuscript, 2011.
[24] K. T. Kelly. Notes on a general topological paradigm.
Unpublished manuscript, 2015.
[25] K. T. Kelly and H. Lin. A simple theory of theoretical
simplicity. Unpublished manuscript, 2011.
[26] K. T. Kelly, O. Schulte, and V. Hendricks. Reliable
belief revision. In Proc. of the 10th ICLMPS, pages
383–398. Kluwer Academic Pub., 1995.
[27] D. Lewis. Convention. Cambridge: Harvard University
Press, 1969.
[28] E. Martin and D. Osherson. Scientific discovery basedon belief revision. JSL, 62(4):1352–1370, 1997.
[29] E. Martin and D. Osherson. Elements of Scientific
Inquiry. MIT Press, Cambridge, MA, USA, 1998.
[30] H. Putnam. Trial and error predicates and the solution
to a problem of Mostowski. JSL, 30(1):49–57, 1965.
[31] O. Schulte and C. Juhl. Topology as epistemology.
Monist, 79(1):141–147, 1996.
[32] R. Solomonoff. A formal theory of inductive inference.
Inform. Control, Part I, 7(1):1–22, 1964. Part II,7(2):224–254, 1964, 1964.
[33] S. Vickers. Topology Via Logic. Cambridge University
Press, 1996.

74

Bayesian Games with Intentions
Adam Bjorndahl

Joseph Y. Halpern

abjorn@andrew.cmu.edu

halpern@cs.cornell.edu

Carnegie Mellon University
Philosophy
Pittsburgh, PA 15213

Cornell University
Computer Science
Ithaca, NY 14853

ABSTRACTWe show that standard Bayesian games cannot represent
the full spectrum of belief-dependent preferences. However,by introducing a fundamental distinction between intended
and actual strategies, we remove this limitation. We define
Bayesian games with intentions, generalizing both Bayesian
games and psychological games [5], and prove that Nashequilibria in psychological games correspond to a specialclass of equilibria as defined in our setting.

1. INTRODUCTION
Type spaces were introduced by John Harsanyi [6] as aformal mechanism for modeling games of incomplete information where there is uncertainty about players’ payoff functions. Broadly speaking, types are taken to encode payoffrelevant information, a typical example being how each participant values the items in an auction. An important feature of this formalism is that types also encode beliefs abouttypes. Thus, a type encodes not only a player’s beliefs aboutother players’ payoff functions, but a whole belief hierarchy:a player’s beliefs about other players’ beliefs, their beliefs
about other players’ beliefs about other players’ beliefs, andso on.
This latter point has been enthusiastically embraced by
the epistemic game theory community, where type spaces
have been co-opted for the analysis of games of complete
information. In this context, types encode beliefs about
the strategies used by other players in the game as well as
their types. So again, types encode a belief hierarchy, but
one that describes players’ beliefs about other players’ beliefs. . . about the other players’ types and strategies. In thisframework, one can determine whether a player is rational
given her type and strategy; that is, whether her strategy
is such that she is making a best response to her beliefs,
as encoded by her type. Thus, rationality, common beliefof rationality, and so on can be defined as events in thespace of (profiles of) strategy-type pairs. This opens theway to epistemic analyses of solution concepts, among otherapplications [3]. In this setting, types do not encode payoffrelevant information; rather, they are simply a tool for describing belief hierarchies about other players’ (types and)strategies.
By contrast, in a Bayesian game, types are payoff-relevant
objects in that utility depends on them, though the payoffrelevant information they are taken to encode often includes
such things as characteristics of the players (strength, work
ethic, etc.), or more generally any relevant facts that may
not be common knowledge. There is typically assumed to75

Rafael Pass

Cornell University
Computer Science
Ithaca, NY 14853

rafael@cs.cornell.edu

be a prior probability on types (indeed, often a common
prior), so a type can still be viewed as encoding beliefs onother types in this setting (a type t encodes the probability
obtained by conditioning the prior on t), and thus a belief
hierarchy. However, the only aspect of this belief hierarchy
that is typically used in Bayesian games is the first-order belief about other players’ types (but not beliefs about beliefs,
and so on), which is needed to defined a player’s expectedutility. Nonetheless, it is possible to leverage the fact thattypes encode beliefs to define Bayesian games in which players’ preferences depend to some extent on the beliefs of theiropponents (see Example 2.2). This observation is the pointof departure for the present work.
The notion that a player’s preferences might depend on
the beliefs of her opponents (or on her own beliefs) is notnew. Psychological games [1, 5] model phenomena like anger,
surprise, and guilt by incorporating belief hierarchies directly into the domain of the utility functions. Languagebased games [2] model similar belief-dependent preferencesby defining utility over descriptions in a given language (inparticular, a language that can express the players’ beliefs).
Types play no explicit role in either of these frameworks; onthe other hand, the discussion above suggests that they may
be naturally employed to accomplish many of the same modeling goals. Since Bayesian games and, more generally, typespaces have become cornerstones of modern game theory, if
the modeling and analysis of psychological games could be
carried out in this familiar framework, it would unify theseparadigms and thereby amplify both the insights and theaccessibility of the latter. In this paper, we provide an extension of Bayesian games that allows us to do just this.
There is an obvious obstruction to capturing general beliefdependent preferences using types in the standard way: types
in Bayesian games encode beliefs about types, not about
strategies. This severely limits the extent to which preferences over types can capture feelings like surprise or guilt,
which are typically expressed by reference to beliefs about
strategies (e.g., my opponent is surprised if I do not play
the strategy that she was expecting me to play). It may
seem that there is a simple solution to this problem: allow types to encode beliefs about strategies. But doing thisleads to difficulties in the definition of Bayesian Nash equilibrium, the standard solution concept in Bayesian games;
this notion depends on being able to freely associate strategies with types. In Section 2, we give the relevant definitions
and make this issue precise.
In Section 3, we develop a modification of the standard
Bayesian setup where each player is associated with twostrategies: an intended strategy that is determined by her
type (and thus can be the object of beliefs), and an actual strategy that is independent of her type (as in standard Bayesian games). This gives us what we call Bayesian
games with intentions. We define a solution concept for suchgames where we require that, in equilibrium, the actual andintended strategies are equal. As we show, under this requirement, equilibria do not always exist.
In Section 4, we show that psychological games can beembedded in our framework. Moreover, we show that thenotion of Nash equilibrium for psychological games definedby Geanakoplos, Pearce, and Stachetti [5] (hereafter GPS)corresponds to a special case of our own notion of equilibrium. Thus, we realize all the advantages of psychologicalgames in an arguably simpler, better understood setting.
We do not require complicated beliefs hierarchies; these are
implicitly encoded by types.
The advantages of distinguishing actual from intended
strategies go well beyond psychological games. As we show
in the full paper, intended strategies can be fruitfully interpreted as reference points in the style of prospect theory [7].
One of the central insights of prospect theory is that the
subjective value of an outcome can depend, at least in part,on how that outcome compares to some “reference level”;
for example, whether it is viewed as a relative gain or loss.
The intended/actual distinction naturally implements theneeded comparison between “real” and “reference” outcomes.
Using this insight, we show that reference-dependent preferences, as defined by Kőszegi and Rabin [8], can be captured
using Bayesian games with intentions.

2. BAYESIAN GAMES
2.1

Definition

A Bayesian game is a model of strategic interaction amongplayers whose preferences can depend on factors beyond
the strategies they choose to play. These factors are often
taken to be characteristics of the players themselves, such aswhether they are industrious or lazy, how strong they are,or how they value certain objects. Such characteristics canbe relevant in a variety of contexts: a job interview, a fight,
an auction, etc.
A type of player i is often construed as encoding precisely
such characteristics. More generally, however, types can beviewed as encoding any kind of information about the worldthat might be payoff-relevant. For example, the resolutionof a battle between two armies may depend not only on what
maneuvers they each perform, but also on how large or welltrained they were to begin with, or the kind of terrain they
engage on. Decision-making in such an environment therefore requires a representation of the players’ uncertainty regarding these variables.
We now give a definition of Bayesian games that is somewhat more general than the standard definition. This will
make it easier for us to develop the extension to Bayesian
games with intentions. We explain the differences after wegive the definition.
Fix a set of players, N = {1, . . . , n}. A Bayesian game(over N ) is a tuple B = (Ω, (Σi , Ti , τi , pi , ui )i∈N ) where• Ω is the measurable space of states of nature;• Σi is the set of strategies available to player i;• Ti is the set of types of player i;• τi : Ω → Ti is player i’s signal function;• pi : Ti → ∆(Ω) associates with each type ti of player i a
probability measure pi (ti ) on Ω with pi (ti )(τi−1 (ti )) =1, representing type ti of player i’s beliefs about thestate of nature;1• ui : Σ × Ω → R is player i’s utility function.2
As we said above, this definition of a Bayesian game ismore general than what is presented in much (though notall) of the literature. There are two main differences. First,
we take utility to be defined over strategies and states ofnature, rather than over strategies and types (cf. [9] for asimilar definition). This captures the intuition that what isreally payoff-relevant is the way the world is, and types simply capture the players’ imperfect knowledge of this. Since
the type signal function profile (τ1 , . . . , τn ) associates witheach world a type profile, utilities can depend on players’
types. Of course, we can always restrict attention to thespecial case where Ω = T and where τi : T → Ti is theith projection function; this is called the reduced form, andit accords with a common conception of types as encoding
all payoff-relevant information aside from strategy choices(cf. [4]).
The second respect in which this definition is more generalthan is standard is in the association of an arbitrary probability measure pi (ti ) to each type ti . It is typically assumed
instead that for each player i there is some fixed probability
measure πi ∈ ∆(Ω) representing her “prior beliefs” about
the state of nature, and pi (ti ) is obtained by conditioning
these prior beliefs on the “private information” ti (or, more
precisely, on the event τi−1 (ti )).3 When π1 = π2 = · · · = πn ,we say that the players have a common prior ; this condition is also frequently assumed in the literature. We adopt
the more general setup because it accords with a standard
presentation of type spaces as employed for the epistemic
analysis of games of complete information [3], thus making
it easier for us to relate our approach to epistemic gametheory.
The requirement that pi (ti )(τi−1 (ti )) = 1 amounts to assuming that each player is sure of her own type (and hence,
her beliefs); that is, in each state ω ∈ Ω, each player iknows that the true state is among those where she is oftype ti = τi (ω), which is exactly the set τi−1 (ti ).

2.2

Examples

It will be helpful to briefly consider two simple examplesof Bayesian games, one standard and one a bit less so.
Example 2.1. First consider a simplified auction scenario
where each participant i ∈ N must submit a bid σi ∈ Σi =1

As usual, we denote by ∆(X) the set of probability measures on the measurable space X. To streamline the presentation, we suppress measurability assumptions here andelsewhere in the paper.
2
Given a collection (Xi )i∈N indexed by N , we adopt
theQ
usual convention of denoting
by X the product i∈N Xi
Q
and by X−i the product j6=i Xj .
3
To ensure this is well-defined, it is also typically assumed
that none of player i’s types are null with respect to πi ; thatis, for all ti ∈ Ti , πi (τi−1 (ti )) > 0.
76

R+ for a given item. Types here are conceptualized as encoding valuations of the item up for auction: for each ti ∈ Ti ,let v(ti ) ∈ R+ represent how much player i thinks the item
is worth, and define player i’s utility ui : Σ × T by
 v(ti ) − σi if σi = max σjj∈Nui (σ, t) =
 0otherwise.

Thus, a player’s payoff is 0 if she does not submit the highest bid, and otherwise is equal to her valuation of the item
less her bid (for simplicity, this model assumes that in theevent of a tie, every top-bidding player gets the item). Notethat the state space here is implicitly taken to be identicalto the space T of type profiles, that is, the game is presented
in reduced form. A type ti therefore tells us not only how
valuable player i thinks the item is (v(ti )), but also what beliefs pi (ti ) ∈ ∆(T ) player i has about how the other players
value the item (and what beliefs they have about their opponents, and so on). The condition that pi (ti )(τ −1 (ti )) = 1
then simply amounts to the assumption that each player issure of her own valuation (as well as her beliefs about otherplayers’ types).

Example 2.2. Next we consider an example where theBayesian framework is leveraged to model a player whose
preferences depend on the beliefs of her opponent. Consider
a game where the players are students in a class, with player
1 having just been called upon by the instructor to answer
a yes/no question. Assume for simplicity that N = {1, 2},
Σ1 = {yes, no, pass}, and Σ2 = {∗} (where ∗ denotes a vacuous move, so only player 1 has a real decision to make). LetΩ = {wy , wn , vy , vn }, where, intuitively, states with the subscript y are states where “yes” is the correct answer, whilestates with the subscript n are states where “no” is the correct answer. Let T1 = {t1 , t01 }, T2 = {t2 , t02 , t002 }, and define
the signal functions by
τ1 (wy ) = τ1 (wn ) = t1 , τ1 (vy ) = τ1 (vn ) = t01 , andτ2 (wy ) = τ2 (wn ) = t2 and τ2 (vy ) = t02 and τ2 (vn ) = t002 .
Finally, assume that all of the subjective probability measures arise by conditioning a common prior π ∈ ∆(Ω) on thetype of the player in question; assume further that π is theuniform distribution. It follows that in each state, player 1
is unsure of the correct answer. On the other hand, while instates wy and wn , player 2 is also unsure of the correct answer, in states vy and vn , player 2 knows the correct answer.
Moreover, in states wy and wn , player 1 is sure that player
2 does not know the correct answer, whereas in states vy
and vn , player 1 is sure that player 2 does know the correct
answer (despite not knowing it himself). We can therefore
use this framework to encode the following (quite plausible)
preferences for player 1: guessing the answer is preferable topassing provided player 2 does not know the right answer,
but passing is better than guessing otherwise. Set
u1 (yes, wy ) = u1 (yes, vy ) = u1 (no, wn ) = u1 (no, vn ) = 5,
representing a good payoff for answering correctly; set
u1 (pass, x) = −2 for all x ∈ Ω,
representing a small penalty for passing regardless of what77

the correct answer is; finally, set
u1 (yes, wn ) = u1 (no, wy ) = −5 andu1 (yes, vn ) = u1 (no, vy ) = −15,

representing a penalty for getting the wrong answer thatis substantially worse in states where player 2 knows thecorrect answer.
It is easy to check that if player 1 considers wy and wn tobe equally probable, then her expected utility for randomly
guessing the answer is 0, which is strictly better than passing(passing, of course, always yields an expected utility of −2).
By contrast, if player 1 considers vy and vn to be equallyprobable, then her expected utility for randomly guessing is−5, which is strictly worse than passing. In short, player 1’s
decision depends on what she believes about the beliefs ofplayer 2.
Example 2.2 captures what might be thought of as embarrassment aversion, which is a species of belief-dependent
preference: player 1’s preferences depend on what player 2
believes. It is worth being explicit about the conditions thatmake this possible:
C1. States in Ω encode a certain piece of information I(in this case, whether the correct answer to the given
question is “yes” or “no”).
C2. Types encode beliefs about states.
C3. Utility depends on types.
From C1–C3, we can conclude that preferences can depend
on what the players believe about I.
Not all kinds of belief-dependent preferences can be captured in the Bayesian framework. Suppose, for example,
that the goal of player 1 is to surprise her opponent by playing an unexpected strategy. More precisely, suppose that
Σ1 = {σ1 , σ10 } and we wish to define u1 in such a way that
player 1 prefers to play σ1 if and only if player 2 believes
he will play σ10 . In contrast to Example 2.2, this scenario
cannot be represented with a Bayesian game for the following simple reason: states do not encode strategies. In otherwords, condition C1 is not satisfied if we take I to be player
1’s strategy. Therefore, types cannot encode such beliefs
about strategies, so utility cannot be defined in a way that
depends on such beliefs.
This suggests an obvious generalization of the Bayesiansetting, namely, encoding strategies in states. Indeed, thisis the idea we explore in this paper; however, it is not quite
as straightforward a maneuver as it might appear, primarily
due to its interaction with the mechanics of Bayesian Nashequilibrium.

2.3

Equilibrium

Part of the value of Bayesian games lies in the fact thata generalized notion of Nash equilibrium can be defined inthis framework, for which the following notion plays a crucialrole: a behaviour rule for player i is a function βi : Ti → Σi .
In Bayesian games, we talk about behaviour rule profiles being in equilibrium, just as in normal-form games, we talkabout strategy profiles being in equilibrium. Intuitively,βi (ti ) represents the strategy that type ti of player i is playing, so a player’s strategy depends on her type.
From a technical standpoint, behaviour rules are important because they allow us to associate a payoff for eachplayer with each state, rather than strategy-state pairs. Sincetypes encode beliefs about states, this yields a notion of expected utility for each type. A Bayesian Nash equilibriumis then defined to be a profile of behaviour rules such thateach type is maximizing its own expected utility.
More precisely, observe that via the signal functions τi , abehaviour rule βi associates with each state ω the strategy
βi (τi (ω)). Thus, a profile β of behaviour rules defines an
induced utility function uβi : Ω → R as follows:uβi (ω) = ui ((βj (τj (ω)))j∈N , ω).

information is encoded in her type) votes Democrat.
We associate intended strategies with types rather thandirectly with states by analogy to behaviour rules, in keeping with the modeling paradigm where the personal characteristics of a player—including her beliefs, decisions, andintentions—are entirely captured by her type. Nonetheless,the composition si ◦ τi : Ω → Σi does associate strategies
with states and so satisfies condition C1 (again, with I being a player’s strategy); thus, players can have beliefs about
strategies. This, in turn, allows us to define utility so as tocapture preferences that depend on beliefs about strategies.

The beliefs pi (ti ) then define the expected utility for each
type: let Eti (β) denote the expected value of uβi with respect
to pi (ti ). Denote by Bi the set of all behaviour rules forplayer i. A behaviour rule βi is a best response to β−i if,for each ti ∈ Ti , βi maximizes Eti :3.2

Finally, a Bayesian Nash equilibrium of the Bayesiangame B is a profile of behaviour rules β such that, for eachi ∈ N , βi is a best response to β−i . A (mixed) BayesianNash equilibrium is guaranteed to exist when the strategy
and types spaces are finite (see [10] for a more general characterization of when an equilibrium exists).

Example 3.1. Consider a 2-player game in which player
1’s goal is to surprise her opponent. We take player 2 to be
surprised if his beliefs about what player 1 intends to play
are dramatically different from what player 1 actually plays.
For definiteness, we take “dramatically different” to meanthat his beliefs about player 1’s intended strategy ascribe
probability 0 to player 1’s actual strategy. Thus, we define
player 1’s utility function as follows:(
1 if p2 (τ2 (ω))([[σ1 ]]) = 0
u1 (σ, ω) =0 otherwise.

[[σi ]] = (si ◦ τi )−1 (σi ) = {ω ∈ Ω : si (τi (ω)) = σi }.

(∀βi0 ∈ Bi )(Eti (βi , β−i ) ≥ Eti (βi0 , β−i )).

3. INTENTION
3.1

Definition

Behaviour rules map types to strategies, but the underlying model does not enforce any relationship between types
and strategies (or between states and strategies). Thus, behaviour rules do not provide a mechanism satisfying condition C1 with I taken to be a player’s strategies, so they
do not allow us to express preferences that depend on beliefs about strategies. In order to express such preferences,
we must associate strategies with states in the model itself.
Note that once we do this, utility functions depend on strategies in two ways. Specifically, since ui is defined on the crossproduct Σ×Ω, players’ preferences depend on strategies both
directly (corresponding to the strategy-profile component of
ui ’s input) and as encoded in states (the second componentof ui ’s input). To keep track of this distinction, we call theseactual and intended strategies, respectively.
Formally, a Bayesian game with instantiated intentions (BGII) is a tuple I = (Ω, (Σi , Ti , τi , si , pi , ui )i∈N ),where si : Ti → Σi is player i’s intention function and theremaining components are defined as in a Bayesian game.
(The reason for this terminological mouthful will become
clear in Section 3.3, where we define Bayesian games withintentions.) Each si associates with each type ti of player
i an intended strategy si (ti ). Intuitively, we might thinkof si (ti ) as the strategy that a player of type ti “intends”
or “is planning” to play (though may ultimately decide notto); alternatively, it might be conceptualized as the “default”
strategy for that type; it might even be viewed as the “stereotypical” strategy employed by players of type ti . The former
interpretation may be appropriate in a situation where wewant to think of self-control; for example, a player who intends to exercise, but actually does not. The latter interpretation may be appropriate if we think about voting. Wealthypeople in Connecticut typically vote Republican, but a particular player i who is wealthy and lives in Connecticut (thisExamplesThe presentation of a BGII is made clearer by introducing
the following notation for the set of states where player i
intends to play σi :(Recall that p2 (τ2 (ω)) is a measure on states, which is why
we apply it to τ1−1 (s−11 (σ1 )), that is, the set of states ω where
player 1’s intended strategy, s1 (τ1 (ω)), is equal to σ1 .)Example 3.2. Next we consider an example introduced
by GPS [5] called the bravery game. This is a 2-player scenario in which player 1 has the only real decision to make:he must choose whether to take a bold action or a timid action, so Σ1 = {bold, timid} (and Σ2 = {∗}). The crux of thegame is the psychological factor, described by GPS as follows: player 1 prefers “to be timid rather than bold, unlesshe thinks his friends expect him to be bold, in which casehe prefers not to disappoint them” [5]. It is also stipulated
that player 2 prefers player 1 to be bold, and also prefers tothink of him as bold. Define q : T → [0, 1] to be the degreeof belief that type t2 of player 2 has in player 1 being bold:q(t) = p(t2 )([[bold]]).
Define q̃ : T → [0, 1] to be type t1 of player 1’s expectationof this degree of belief:q̃(t) = Et1 (q),where Eti (f ) denotes the expected value of f with respect
to the measure p(ti ). We can then represent the players’
preferences in a reduced-form BGII as follows:(
2 − q̃(t)if σ1 (t1 ) = boldu1 (σ, t) =3(1 − q̃(t)) if σ1 (t1 ) = timid,
u2 (σ, t) =78(

2(1 + q(t))

if σ1 (t1 ) = bold

1 − q(t)if σ1 (t1 ) = timid.

This representation closely parallels that given in [5], inwhich q and q̃ are understood not as functions of types,but (implicitly) as functions of belief hierarchies.4 But thismakes no difference to the preferences this game encodes.
For example, it is easy to see that player 2 prefers player 1
to be bold, and all the more so when q is high—that is, all
the more so when she believes with high probability that hewill be bold.5 Similarly, one can check that player 1 prefers
to be timid provided that q̃(t) < 21 ; in other words, provided
that his expectation of his opponent’s degree of belief in him
being bold is sufficiently low.
Why not define player 1’s preferences directly in termsof the beliefs of his opponent, rather than his expectationof these beliefs? GPS cannot do so because of a technicallimitation of the framework as developed in [5]; specifically,that a player’s utility can depend only on her own beliefs.
Battigalli and Dufwenberg [1] correct this deficiency. BGIIs
do not encounter such limitations in the first place. In particular, it is easy enough to redefine player 1’s utility as
follows:(
2 − q(t)if σ1 (t1 ) = bold0
u1 (σ, t) =3(1 − q(t)) if σ1 (t1 ) = timid.
In this case, we find that player 1 prefers to be timid provided q(t) < 21 , or in other words, provided that his opponent’s degree of belief in him being bold is sufficiently
low.
Observe that in neither of the preceding examples did weprovide a concrete BGII, in that we did not explicitly define
the type spaces, the intention functions, and so on. Instead,
we offered general recipes for implementing certain beliefdependent preferences (e.g., to surprise, to live up to expectations, etc.) in arbitrary BGIIs. Particular choices of typespaces and intention functions do play an important role inequilibrium analyses; however, as illustrated by the preceding two examples, at the modeling stage they need not beprovided up front.

3.3

Equilibrium

We now define a notion of equilibrium for this setting. Asa first step towards this definition, given a BGII I, we saythat a profile of behaviour rules β is an equilibrium of I
provided:(1) β is a Bayesian Nash equilibrium of the underlying Bayesian
game: that is, each βi is a best response to β−i in precisely the sense defined in Section 2.3;(2) for each player i ∈ N , βi = si .
This definition, and in particular condition (2), embodies
the conception of equilibrium as a steady state of play where
4
Additionally, GPS give the value of q, not by the probability that player 2 assigns to player 1 being bold, but by
player 2’s expectation of the probability p with which player
1 decides to be bold. We forgo this subtlety for the time
being.
5
It is not quite clear why GPS define player 2’s payoff inthe event that player 1 is timid to be 1 − q(t) rather than
1+q(t). This latter value preserves the preferences described
while avoiding the implication that, assuming that player 1
will be timid, player 2 also prefers to believe that he will
be timid—this stands in opposition to the stipulation that
player 2 prefers to think of her opponent as bold.

79

each player has correct beliefs about her opponents (and isbest responding to those beliefs). In a BGII, beliefs about
the strategies of one’s opponents are beliefs about intended
strategies (although, in equilibrium, a player will also havebeliefs about actual strategies). On the other hand, since behavior rules associate strategies with types and players havebeliefs over types, behaviour rules also induce beliefs about
strategies; in our terminology, these are beliefs about actual
strategies. Condition (2) implies that these two beliefs coincide in equilibrium; in equilibrium, each type of each player
actually plays the strategy she intended to play (which isexactly the strategy her opponents expected her to play).
Does condition (2) collapse the distinction between intended and actual strategies, thereby returning us to theclassical setting? It does not. First, in a standard Bayesiangame we could not even write down a model where players’
preferences depended on beliefs about strategies. In addition, although we demand that intended and actual strategies coincide in equilibrium, this restriction does not apply
to the evaluation of best responses. Recall that βi is a best
response to β−i if and only if(∀βi0 ∈ Bi )(Eti (βi , β−i ) ≥ Eti (βi0 , β−i )).

Crucially, βi0 need not be equal to si . In other words, for βito count as a best response, it must be at least as good as
all other behaviour rules, including those that recommend
playing a strategy distinct from that specified by si .
Example 3.3. Consider a 2-player reduced-form BGII with
Σ1 = {a, b}, Σ2 = {∗}, T1 = {x, x0 }, and T2 = {y, y 0 }, andwhere
p1 (x)({y}) = p1 (x0 )({y 0 }) = p2 (y)({x0 }) = p2 (y 0 )({x}) = 1.
Let u1 be defined as in Example 3.1, encoding player 1’s
desire to surprise her opponent:(
1 if p2 (t2 )([[σ1 ]]) = 0
u1 (σ1 , ∗, t) =0 otherwise.
Suppose that s1 (x) = s1 (x0 ) = a. Then, of course, p2 (y)([[a]]) =p2 (y 0 )([[a]]) = 1, and likewise p2 (y)([[b]]) = p2 (y 0 )([[b]]) = 0.
It follows immediately that the expected utility of playing
a for either type of player 1 is equal to 0 (since player 1 issure that this will not surprise her opponent), whereas the
expected utility of playing b for either type of player 1 isequal to 1 (since, in this case, player 1 is sure that this will
surprise her opponent). In particular, if β1 = s1 , then β1 is
not a best response. Thus, this particular BGII admits noequilibrium.
Now suppose that s1 (x) = a and s1 (x0 ) = b. This is,of course, a different BGII from the one considered in theprevious paragraph, but it differs only in the specificationof player 1’s intentions. Moreover, in this BGII it is not
hard to check that β1 = s1 is a best response and therefore
constitutes an equilibrium: type x is sure that player 2 isof type y; therefore, type x is sure that player 2 is surethat player 1 is of type x0 , and so is playing b; thus, a is abest response for x, since x is sure that it will surprise her
opponent; a similar argument shows that b is a best response
for x0 .
Example 3.3 demonstrates that the notion of best response
in a BGII—and therefore the notion of equilibrium—can besensitive to states of play where players are not playing theirintended strategies. But it also illustrates the pivotal roleof the intention functions si in determining the existence ofan equilibrium. Indeed, condition (2) implies that if a given
BGII I has an equilibrium at all, it is unique and equal tos. This suggests that BGIIs are not at the right “resolution”
for equilibrium analysis, since they come already equippedwith a unique candidate for equilibrium. Thus, rather thanrestricting attention to a single BGII, where the intentionfunction is specified and hard-coded into the model, we consider a more general model, where the intention function is
not specified, but still affects the utility. This is parallel tothe role of strategies in standard games, which are not hardcoded into the model, but of course the utility function still
depends on them. Essentially, we are moving the intentionfunction from the model to the utility function. As we shallsee, our earlier examples of BGIIs can be easily interpreted
as models in this more general sense.
In order to make this precise, we must first formally define
utility functions that take as arguments intention functions.
More precisely, taking ΣT = ΣT1 1 × · · · × ΣTnn (so that ΣT isthe set of intention function profiles), an explicit utility function is a map ũi : Σ × Ω × ΣT → R; these are just like theutility functions in a BGII except they explicitly take as input the associations between types and strategies provided
by intention functions. A Bayesian game with intentions (BGI) is a tuple Ĩ = (Ω, (Σi , Ti , τi , pi , ũi )i∈N ), where
the components are defined just as they are in a Bayesiangame, except that the functions ũi are explicit utility functions. We emphasize that a BGI, unlike a BGII, does notinclude players’ intention functions among its components;
instead, these functions show up as arguments in the (explicit) utility functions.
It is easy to see that all the examples of BGIIs that wehave considered so far can be naturally converted to BGIs.
For example, the utility function u1 (σ, t) in Example 3.1 becomes ũi (σ, t, s). The definition of ũi (σ, t, s) looks identicalto that of u1 (σ, t); the additional argument s is needed todefine [[σ1 ]].
A BGI induces a natural map from intention functions
to BGIIs: given Ĩ = (Ω, (Σi , Ti , τi , pi , ũi )i∈N ) and functions
si : Ti → Σi , letĨ(s1 , . . . , sn ) = (Ω, (Σi , Ti , τi , si , pi , ui )i∈N ),where ui : Σ × Ω → R is defined byui (σ, ω) = ũi (σ, ω, s1 , . . . , sn ).
Clearly Ĩ(s1 , . . . , sn ) is a BGII; we call it an instantiationof Ĩ. We then define an equilibrium of Ĩ to be a profile ofbehaviour rules β that is an equilibrium of the corresponding instantiation Ĩ(β). Here we make implicit use of the factthat both behaviour rules and intention functions are functions from types to strategies. Indeed, the profile β plays
two roles: first, it is used to determine the intentions of theplayers; then, in the context of the instantiated BGI withthese fixed intentions, we evaluate whether each βi (ti ) is abest response, just as in the definition of equilibrium for astandard Bayesian game.
Is this a reasonable notion of equilibrium? As we observed
above, in a BGII, the only possible equilibrium is “built in”
to the model in the form of the intention functions. In particular, the only possible equilibrium for the instantiationĨ(β) is β itself. Of course, β is not necessarily an equilibrium of this game; however, by quantifying over β andconsidering the corresponding class of BGIIs (i.e., those obtained as instantiations of Ĩ), we are essentially asking thequestion: “Is there a profile of intentions such that, assuming
those intentions are common knowledge, no player prefers todeviate from their intention?” If so, that profile constitutes
an equilibrium. This is a natural solution concept; in fact,
as we show in Section 4, the notion of equilibrium proposedby GPS for psychological games is a special case of our definition.
Example 3.4. In light of these definitions, Example 3.3
can be viewed as first defining a BGI Ĩ, and then considering
two particular instantiations of it. The equilibrium observations made then amount to the following: the behaviour ruleβ1 ≡ a (i.e., the constant function a) is not an equilibriumof Ĩ, but the behaviour rule β10 that sends x to a and x0 to bis. (As there is only ever one option for player 2’s behaviour
rule, namely β2 ≡ ∗, we can safely neglect it.)Example 3.5. Consider again the bravery game of Example 3.2. Under any particular specification of state space
and type spaces, this becomes a BGI Ĩ. It is not difficult to
see that each of the behaviour rules β1 ≡ timid and β10 ≡ bold
is an equilibrium of Ĩ.

3.4

Existence

Are equilibria of BGIs guaranteed to exist? Not necessarily. At least one obstacle to existence lies in the specificationof the underlying type space and the corresponding probability measures: as the following example shows, certain
kinds of belief that are necessary for best-responses may be
implicitly ruled out.
Example 3.6. Consider a 2-player reduced-form BGI Ĩ
where Σ1 = {a, b}, Σ2 = {∗}, T1 = {x, x0 }, and T2 = {y, y 0 },and where
p1 (x)({y}) = p1 (x0 )({y 0 }) = p2 (y)({x}) = p2 (y 0 )({x0 }) = 1.
Once again we consider a model where player 1 wishes tosurprise her opponent, and so define u1 as in Example 3.3:(
1 if p2 (t2 )([[σ1 ]]) = 0
u1 (σ1 , ∗, t) =0 otherwise.
Note that player 1 is certain that player 2 knows her type.
It follows that no matter what her intentions are, player 2knows them, and so (by definition of u1 ), player 1 can always
do better by deviating. In other words, no behaviour rule β1
is an equilibrium of Ĩ(β1 ) (since it is not a best response).
It follows immediately that Ĩ admits no equilibria.
This obstacle persists even if we extend our attention to
mixed strategies. More precisely, consider the class of BGIIs
where, for each player i, Σi = ∆(Ai ) for some finite set Ai(the set of player i’s pure strategies), and ui : Σ × Ω → R
satisfiesXui (σi , σ−i , ω) =σi (ai )ui (ai , σ−i , ω).
ai ∈AiIn other words, player i’s utility for playing σi is just the
expected value of her utility for playing her various purestrategies with the probabilities given by σi . As is standard,
we call elements of Σi mixed strategies, and the corresponding BGIIs mixed-strategy BGIIs. We can similarly define

80

Example 3.7. Consider a 2-player reduced-form BGI whereΣ1 = ∆({a, b}), Σ2 = {∗}, T1 = {x, x0 }, and T2 = {y, y 0 },and whereb1i ∈ ∆(Σ−i ) is player i’s first-order beliefs, b2i is player
i’s second-order beliefs (i.e., roughly speaking, her beliefs
about the beliefs of her opponents), and so on, such thatthe beliefs in this sequence satisfy certain technical conditions (roughly speaking, lower-order beliefs must agree withthe appropriate marginals of higher-order beliefs, and thisagreement must be common knowledge); see the full paper
for the complete definition.
Given a mixed-strategy BGII I and a type ti ∈ Ti , we candefine the first-order beliefs associated with ti by

p1 (x)({y}) = p1 (x0 )({y 0 }) = p2 (y)({x}) = p2 (y 0 )({x0 }) = 1.

ϕ1i (ti ) = (s−i )∗ (pi (ti ));mixed-strategy BGIs. Note that in this context, since theintention functions si map into Σi , intended strategies arealso mixed.
The next example shows that, in contrast to the classical setting, there are mixed-strategy BGIs with finite typespaces that admit no equilibria.

Set
u1 (a, ∗, t) =(1

if p2 (t2 )([[a]]) < 1

0otherwise

u1 (b, ∗, t) =(1

if p2 (t2 )([[a]]) = 10

otherwise,

andthat is, the pushforward of pi (ti ) from Ω to Σ−i by s−i . Note
that, in our terminology, these are beliefs about intended
strategies. The kth-order beliefs associated with ti , denoted
ϕki (ti ), can be defined inductively in a similar fashion; it isthen straightforward to show that the sequence
ϕi (ti ) = (ϕ1i (ti ), ϕ2i (ti ), . . .)and extend to all σ1 ∈ ∆({a, b}) by taking expectation:
u1 (σ1 , ∗, t) = σ1 (a)u1 (a, ∗, t) + σ1 (b)u1 (b, ∗, t).
Note that, following standard conventions, here we identify
the pure strategy a with the degenerate mixed strategy thatplaces probability 1 on a; likewise for b. Thus, for example, the condition p2 (t2 )([[a]]) < 1 amounts to the following:“type t2 is not absolutely certain that player 1 intends toplay the pure strategy a”, or equivalently, “type t2 considers it possible that player 1 intends to play a mixed strategy
that places positive weight on b”. The preferences defined by
u1 can be roughly summarized as follows: “player 1 prefers
to play a in the event that player 2 thinks she might place
positive weight on b, and prefers to play b if player 2 is surethat she’ll play a for sure”.
This game admits no equilibria. To see this, suppose that
β1 were an equilibrium: that is, set player 1’s intention function equal to β1 , and suppose that β1 is an equilibrium of theresulting BGII.6 First consider the case where β1 (x) ∈ Σ1
satisfies β1 (x)(b) > 0. Then it follows that p2 (y)([[a]]) = 0(i.e., type y is certain that player 1 is not playing the purestrategy a), and so, since type x is certain that player 2 isof type y, it follows by definition of u1 that type x’s best response is to play the pure strategy a. In particular, β1 (x) is
not a best response, so β1 cannot constitute an equilibrium.
Now consider the case where β1 (x)(b) = 0; in other words,
β1 (x) is the pure strategy a. Then we have p2 (y)([[a]]) = 1,
from which it follows that type x’s best response is to play
the pure strategy b. Thus, once again, β1 cannot constitutean equilibrium.

4.

PSYCHOLOGICAL GAMES

Psychological games can be captured in our framework.
A psychological game P consists of a finite set of players
N , together with mixed strategies Σi and utility functions
vi : B̄i × Σ → R for each player i, where B̄i denotes theset of “collectively coherent” belief hierarchies for player i.
Somewhat more precisely, an element bi ∈ B̄i is an infinite sequence of probability measures (b1i , b2i , . . .) where6

As before, we ignore player 2’s behaviour since he has no
choices to make.

81

is collectively coherent, and thus ϕi : Ti → B̄i (see the full
paper).
This correspondence provides a natural notion of equivalence between psychological games and BGIIs with respect
to the psychological preferences expressed in the former,
namely,∀i ∈ N ∀σ ∈ Σ ∀ω ∈ Ω(ui (σ, ω) = vi (ϕi (τi (ω)), σ)).
When a BGII I satisfies this condition with respect to a
psychological game P, we say that I and P are preferenceequivalent.
The notion of preference-equivalence lifts naturally to BGIs.
Observe that the functions ϕki depend on the profile of intention functions s; being explicit about this dependence, wewrite ϕki (ti ; s) rather than ϕki (ti ); we then say that Ĩ and P
are preference-equivalent provided that∀i ∈ N ∀σ ∈ Σ ∀ω ∈ Ω ∀s ∈ ΣT (ũi (σ, ω, s) = vi (ϕi (τi (ω); s), σ)).
It is easy to see that, given a psychological game P, we
can obtain a preference-equivalent BGI Ĩ simply by taking
the above condition as the definition of the utility functionsũi . Thus, we have the following:
Proposition 4.1. For every psychological game there exists a preference-equivalent BGI.
Note that even very simple BGIs (i.e., those with verysmall type/state spaces) can be preference-equivalent to psychological games; indeed, it is sufficient for the utility functions ũi to be of the formũi (σ, ω, s) = f (ϕi (τi (ω); s), σ),so that utility depends on states only to the extent thatstates encode belief hierarchies. In particular, although theutility functions in a psychological game have uncountable
domains (since they apply to all possible belief hierarchies),a BGI Ĩ can be preference-equivalent to a psychological
game P even if Ĩ has only finitely many states, since allthat matters is that the utility functions of Ĩ agree withthe utilitiy functions of P on the belief hierarchies encodedby the states of Ĩ. Given a psychological game, we canconstruct a preference-equivalent BGI with type spaces richenough that each ϕi is surjective: in other words, every belief hierarchy is realized by some type. However, in orderto capture equilibrium behaviour, such richness turns out tobe superfluous. We now show how the notion of equilibrium
defined by GPS for psychological games can be recovered as
equilibria in our setting.
Given σ ∈ Σ, let χi (σ) ∈ B̄i denote the unique belief
hierarchy for player i corresponding to common belief in σ.
A psychological Nash equilibrium of P is a strategy profile σsuch that, for each player i, σi maximizes the function
σi0 7→ vi (χi (σ), σi0 , σ−i ).
In particular, to check whether σ constitutes a psychologicalNash equilibrium, the only relevant belief hierarchies arethose corresponding to common belief of σ. This, in essense,is the reason we do not need rich type spaces in BGIs to
detect such equilibria.
Theorem 4.2. If P and Ĩ are preference-equivalent, thenσ is a psychological Nash equilibrium of P if and only if the
profile of (constant) behaviour rules β for which βi ≡ σi isan equilibrium of Ĩ.
Proof. When β is the profile of behaviour rules describedin this theorem, the corresponding instantiation Ĩ(β) has theproperty that, for each type ti , ϕi (ti ) = χi (σ). The rest of
the proof is essentially just unwinding definitions; see thefull paper for details.
Theorem 4.2 shows that equilibrium analysis in psychological games does not depend on the full space of belief hierarchies; it can be captured by particularly simple BGIs. It
also establishes an equivalence between psychological Nashequilibria and a certain restricted class of equilibria in BGIs;
namely, those consisting of constant behaviour rules. This
restriction is not surprising in light of the fact that psychological games do not model strategies as functions of types,
while BGIs do. Thus, BGIs are not merely recapitulationsof the GPS framework: they are a common generalizationof psychological games and Bayesian games.

5.

CONCLUSION

We have introduced BGIs, Bayesian games with intentions, which generalize Bayesian games and psychologicalgames in a natural way. We believe that BGIs will prove
much easier to deal with than psychological games, whileallowing greater flexibility.
When do equilibria exist? While Theorem 4.2 provides
sufficient conditions for the existence of equilibria in BGIs,
they are certainly not necessary conditions. We can show,for example, that there are BGIs that admit only equilibria
in which no behaviour rule is constant. Formulating moregeneral conditions sufficient for existence is the subject ofongoing work.
Perhaps the most exciting prospect for future research liesin leveraging the distinction between actual and intended
strategies. As we show in the full paper, this distinction
can be used to implement Köszegi and Rabin’s [8] model ofreference-dependent preferences; we believe that it will haveother uses as well, and perhaps lead to new insights into
solution concepts.

Bayesian games to capture belief-dependent preferences. Halpern is supported in part by NSF grants IIS-0911036 andCCF-1214844, by ARO grant W911NF-14-1-0017, and by
the Multidisciplinary University Research Initiative (MURI)
program administered by the AFOSR under grant FA955012-1-0040. Pass is supported in part by a Microsoft Research
Faculty Fellowship, NSF CAREER Award CCF-0746990,
AFOSR Award FA9550-08-1-0197, and BSF Grant 2006317.

6.

REFERENCES[1] P. Battigalli and M. Dufwenberg. Dynamic
psychological games. Journal of Economic Theory,
144:1–35, 2009.
[2] A. Bjorndahl, J. Y. Halpern, and R. Pass.
Language-based games. In Theoretical Aspects ofRationality and Knowledge: Proc. Fourteenth
Conference (TARK 2013), pages 39–48, 2013.
[3] E. Dekel and M. Siniscalchi. Epistemic game theory.
In H. P. Young and S. Zamir, editors, Handbook ofGame Theory with Economic Applications, Volume 4,
pages 619–702. North-Holland, 2015.
[4] D. Fudenberg and J. Tirole. Game Theory. MIT Press,
Cambridge, Mass., 1991.
[5] J. Geanakoplos, D. Pearce, and E. Stacchetti.
Psychological games and sequential rationality. Gamesand Economic Behavior, 1(1):60–80, 1989.
[6] J. Harsanyi. Games with incomplete information
played by ‘Bayesian’ players, parts I–III. Management
Science, 14:159–182, 320–334, 486–502, 1968.
[7] D. Kahneman and A. Tversky. Prospect theory, ananalysis of decision under risk. Econometrica,
47(2):263–292, 1979.
[8] B. Kőszegi and M. Rabin. A model ofreference-dependent preferences. The Quarterly
Journal of Economics, CXXI:1133–1165, 2006.
[9] M. J. Osborne and A. Rubinstein. A Course in Game
Theory. MIT Press, Cambridge, Mass., 1994.
[10] G. Tian. The existence of equilibria in games with
arbitrary strategy spaces and payoffs: a fullcharacterization. Available atwww.dklevine.com/archive/refs4814577000000000160.pdf.,
2009.

Acknowledgements
We are indebted to Aviad Heifetz for asking the questionthat first prompted us to explore the notion of extending82How Many Levels Do Players Reason?
Observational Challenges and a Solution∗[Extended Abstract]
Adam Brandenburger∗

Alex Danieli

NYU Stern School of Business
New York University
adam.brandenburger@stern.nyu.edu

alex.daniels85@gmail.com

Amanda Friedenberg

Department of Economics
Arizona State University
amanda.friedenberg@asu.edu

Keywords
Epistemic Game Theory, Bounded Reasoning, Level-m Reasoning, Context of the Game

1. INTRODUCTION
Interactive reasoning is an important aspect of how players
behave. To determine whether a particular course of action
is good or bad, Ann may need to form a theory about Bob’s
play of the game. In forming such a theory, she may reason
that Bob is ‘strategically sophisticated’ — if so, she may
reason that Bob forms a belief about her own play to determine whether a particular strategy is good or bad for him.
That is, Ann may want to form a second-order theory about
Bob’s play of the game. Of course, Ann may then reason
that Bob uses a second-order theory to choose his strategy.
In this case, Ann may want to form a third-order theory
about Bob’s play of the game. And so on.
How many levels of reasoning do players undertake? We
address this question for the case where the players’ processes of reasoning are not observable. Instead, the researcher observes only the behavior of the players — or,perhaps, only the outcome of the game. We ask: Can the
researcher use this information to identify — or provide
bounds on — levels of reasoning?

2. A MOTIVATING EXAMPLE
Figure 1 depicts the game of Battle-of-the-Sexes with an
Outside Option: Ann can either choose to exercise an outside option or choose to play Battle-of-the-Sexes with Bob.
∗
This abstract was prepared for TARK 2015. We thankJessy Hsieh, Terri Kneeland, Xiaomin Li, and Elliot Lipnowski for valuable comments. We also thank audiencesat the University of Bristol, the 2013 Asian Meeting of the
Econometric Society, National University of Singapore, andthe 2013 Workshop on Logical Structure of Correlated Information Change for important input. Adam Brandenburger
thanks the NYU Stern School of Business for financial support. Amanda Friedenberg thanks NSF SES-1358008 forfinancial support and the UCL Economics Department for
its unbounded hospitality.

83

The standard argument is that, if the players are ‘strategically sophisticated,’ then Ann will choose In-U and Bob
will choose L: The strategy In-D is dominated for Ann by
the outside option. Thus, if Ann does play In, Bob should
reason that she will play U , since In-U is undominated. In
this case, his best response is to play L. Ann should understand that Bob will reason this way and expect Bob to play
L. With this expectation, Ann should play In-U .
Ann
In

Out

3

L

*

Bob

R

U

4,2

0,0

D

0,0

2,4

Ann

Figure 1: Battle-of-the-Sexes with an Outside Option

Thus, there appears to be a clear prediction: Ann will play
In-U and Bob will play L. Yet, in the lab, Out is played withsignificant frequency. (See, inter alia, [4] and [2].) One mightthen draw the conclusion that there is limited reasoning: If
Bob engages in one level of reasoning, he may choose either
L or R depending on what he believes about Ann’s play. So,
if Ann engages in two levels of reasoning, she might choose
to play Out. It is only if Ann reasons three (or more) levels
that she would not play Out.
But, in fact, this behavior need not be an artifact of limited reasoning. There is another reason why Ann mightchoose to play Out — one based on the idea that there is a“context” to the game. In particular, suppose that it is commonly understood that “Bob is a bully” and, so, whenevera Battle-of-the-Sexes game is played, he attempts to go forhis best option and plays R. To be specific, suppose:

Types

Bully-1: at each information set, Ann believes that Bob
plays R,

R1 (T )
R2 (T )

Bully-2: at each information set, Bob believes “Bully1,”R3 (T )
R4 (T )

Bully-3: at each information set, Ann believes “Bully2.”..
.
If this is the context under which the game is played, Ann
may play Out, even if she reasons three levels. In fact, she
may play Out, even if she engages in common reasoning
about the play of the game.
Here is the basic idea: When the game is played in thiscontext, if Ann reasons at least one level, then she must
play Out. (Ex ante, she expects that Bob will play R, andso Out is the unique best choice.) If Bob reasons at least twolevels, then he will reason that Ann reasons at least one level,
provided he has not observed information that contradicts
this hypothesis. Thus, if Bob reasons at least two levels, hemust begin the game believing Ann plays Out. Conditional
upon Ann’s playing In, he is forced to reason that Ann does
not reason one level. So, Bob can reason two levels andconclude (after observing In) that Ann is playing In-U , but
he can also conclude that Ann is playing In-D. Thus, if Bob
reasons two levels, he can play either of L or R. It follows
that believing that Bob play will R is consistent with threelevels of reasoning for Ann.
In light of the above, it appears premature to conclude
that observing the play of Out indicates that Ann is necessarily only a level m-reasoner, for some m ≤ 2. This paper
focuses on the case where the researcher cannot observe players’ actual beliefs or which beliefs players consider possible.
Thus, we seek an answer to our question that is independentof the context of the game.

3. APPROACH AND CHALLENGE
In keeping with what we have just seen, we will need todescribe what beliefs players do vs. do not consider possible
in a particular game. The device we will use to describe
these beliefs is an epistemic type structure, denoted by
T . An epistemic type structure will consist of a set of types
for each player, where a type for a player will describe that
player’s hierarchies of beliefs about the play of the game.
Different type structures are associated with different events
which are commonly believed. For instance, for Battle-ofthe-Sexes with an Outside Option, there is a type structure
where the event “Bob is a bully” is commonly believed andthere are other type structures where it is not.
Call a pair (Γ, T ) an epistemic game. For a given epistemic game (Γ, T ), we can define the set of strategy-typepairs which are consistent with m levels of reasoning, to be
denoted Rm (T ). Refer to Figure 2, that illustrates these
sets. Specifically:(0) The set R0 (T ) is the set of all strategy-type pairs.
This set captures level-0 reasoning, since there is norequirement on reasoning.
(1) The set R1 (T ) is the set of strategy-type pairs where
the players’ strategies are optimal given their belief(i.e., type). This set captures level-1 reasoning.

Strategies

Figure 2: Level-m Reasoning..
.
(m) The set Rm+1 (T ) is the set of strategy-type pairs in
Rm (T ) where each player reasons that the other players engage in level-m reasoning. This set captureslevel-(m + 1) reasoning.
..
.
(∞) The set R∞ (T ) is the set of strategy-type pairs in
Rm (T ) for all m. This set captures level-∞ or common reasoning.
The sets Rm (T ) depend not only on the game Γ but also on
the type structure T . This fits with our informal analysis
of Battle-of-the-Sexes with an Outside Option, where thebehavior of a level-3 reasoner depended on whether or not
the event “Bob is a bully” is commonly understood.
Observe that the sets R0 (T ), . . . , Rm (T ), . . . are decreasing. This reflects the fact that, if players reason at least(m + 1) levels, then they reason at least m levels. As aconsequence, we will not be able to identify the minimum
number of levels of reasoning by observing behavior alone.1
Instead, we seek to identify the maximum number of levelsof reasoning consistent with observed behavior.
The goal then is to identify when a strategy is consistent
with m but not (m+1) levels of reasoning. Toward this end,
we seek to construct an ordered partition of the strategy set,
denoted by L = {L0 , L1 , . . . , Lm , . . . , L∞ }, that satisfies thefollowing criteria: When m is finite, s ∈ Lm if it is
1. consistent with level-m reasoning in some epistemic
game (Γ, T ∗ ), but
2. inconsistent with level-(m + 1) reasoning in any epistemic game (Γ, T ).
When m is infinite, s ∈ Lm if it is consistent with level∞ reasoning in some epistemic game (Γ, T ∗ ). We refer toplayers who choose s ∈ Lm as Level-m Reasoners (orLm -Reasoners).
1
As the “level-k” and “cognitive hierarchy” literatures makeclear, it may be possible to identify the minimum number oflevels of reasoning by making auxiliary assumptions about
behavior or beliefs. See the discussion below.

84For the case of a matrix, standard results give that s ∈ Lm
if and only if s is m but not (m + 1)-rationalizable. One
might conjecture that the same is true for the tree, where
we now take “rationalizability” to mean “extensive-form rationalizability” as in [8] and [1] (or, equivalently, “iterated
conditional dominance,” as in [9]). This is not the case.
Refer back to Battle-of-the-Sexes with an Outside Option(Figure 1). There, Out is consistent with two but not threerounds of extensive-form rationalizability. But, the “Bob isa bully” analysis showed that Out ∈ L∞ .
The main paper provides a novel procedure which servesto construct the partition L in a finite number of steps. A
major challenge in providing such an procedure is the factthat the definition of Lm makes reference to all type structures. For a given finite tree, there are infinitely many associated type structures and, therefore, searching across alltype structures would appear to be an infinite task. The
Main Theorem in the paper provides a way to side-step thisdifficulty for generic games. It characterizes the set of strategies consistent with m levels of reasoning as a property of
the game alone — without reference to any type structure.
It goes on to show how to implement the procedure in a
“simple” manner.

4. UPPER BOUND ON REASONING
The main paper identifies the maximum number of levelsof reasoning consistent with observed behavior. The focus
on this upper bound is inevitable, absent making auxiliary
assumptions on behavior and/or beliefs. (The “level-k” and“cognitive hierarchy” literatures, e.g., [7], [10], [5], [3], etc.,
obtain exact identification by imposing such restrictions.)
But, there are also good reasons why this upper bound is ofinterest. To the extent that the researcher is interested inusing the number of levels of reasoning as an empirical input,
the researcher may care only that the player acts ‘as if’ sheis an L2 -Reasoner — even if she is, in fact, an L1 -Reasoner
or L0 -Reasoner. A similar argument applies to higher levelsof reasoning.
For certain datasets, it may be possible to distinguish an
L2 -Reasoner from an L1 -Reasoner who acts as if she is an L2 Reasoner. With a large dataset, we should expect, on statistical grounds, to see more than just occasional in-sample
play of strategies inconsistent with higher levels of reasoning. In the spirit of the “level-k” and “cognitive hierarchy”
literatures (e.g., [7], [10], [5], [3], etc.), the researcher may
be able to design an experiment so that, even in small samples, we would expect to see play inconsistent with higher
levels of reasoning. For instance, the design in [6] is based on“ring games,” where the payoff to any player depends only
on his/her left-hand neighbor’s choice. So, changing payoffs
to a player two steps to the left of Ann should not affecther behavior if she is a L1 -Reasoner, but should affect her
behavior is she is a L2 -Reasoner. As a consequence, varying
payoffs across the experimental session should generate distinct behavior for L1 - and L2 -Reasoners, thereby allowing
the experimenter to identify the L2 -Reasoner.

5. REFERENCES[1] P. Battigalli. On Rationalizability in Extensive Games.
Journal of Economic Theory, 74(1):40–61, 1997.
[2] J. Brandts and C. Holt. Limitations of dominance andforward induction: Experimental evidence. Economics
Letters, 49(4):391–395, 1995.

85[3] C. Camerer, T. Ho, and J. Chong. A Cognitive
Hierarchy Model of Games. The Quarterly Journal of
Economics, 119(3):861–898, 2004.
[4] R. Cooper, D. DeJong, R. Forsythe, and T. Ross.
Forward induction in the battle-of-the-sexes games.
American Economic Review, 83(5):1303–1316, 1993.
[5] M. Costa-Gomes, V. Crawford, and B. Broseta.
Cognition and Behavior in Normal-Form Games: An
Experimental Study. Econometrica, 69(5):1193–1235,
2001.
[6] T. Kneeland. Testing Behavioral Game Theory:
Higher-Order Rationality and Consistent Beliefs.
http://terri.microeconomics.ca, 2013.
[7] R. Nagel. Unraveling in Guessing Games: An
Experimental Study. The American Economic Review,85(5):1313–1326, 1995.
[8] D. Pearce. Rationalizable Strategic Behavior and the
Problem of Perfection. Econometrica,
52(4):1029–1050, 1984.
[9] M. Shimoji and J. Watson. Conditional Dominance,
Rationalizability, and Game Forms. Journal of
Economic Theory, 83(2):161–195, 1998.
[10] D. Stahl and P. Wilson. On Players’ Models of Other
Players: Theory and Experimental Evidence. Gamesand Economic Behavior, 10(1):218–254, 1995.

Translucent Players: Explaining Cooperative Behavior in
Social Dilemmas
Valerio Capraro

Centre for Mathematics and Computer Science(CWI)
Amsterdam, 1098 XG, The Netherlands

V.Capraro@cwi.nl

ABSTRACT

Joseph Y. Halpern

Cornell University
Computer Science Department
Ithaca, NY 14853

halpern@cs.cornell.edu

cooperation can be observed also in one-shot anonymous interactions [Camerer 2003], where none of the five rules ofIn the last few decades, numerous experiments have shown
cooperation proposed by Nowak [2006] seems to be at play.
that humans do not always behave so as to maximize their
Here we propose a novel approach to explain cooperation,material payoff. Cooperative behavior when non-cooperation
based
on work of Halpern and Pass [2013] and Salcedo [2013],
is a dominant strategy (with respect to the material payoffs)
assuming what Halpern and Pass call translucent players.
is particularly puzzling. Here we propose a novel approach
Typically, players are assumed to be opaque, in the sense
to explain cooperation, assuming what Halpern and Passthat a deviation by one player in a normal-form game does[2013] call translucent players. Typically, players are asnot affect the strategies used by other players. But a player
sumed to be opaque, in the sense that a deviation by one
may believe that if he switches from one strategy to another,player in a normal-form game does not affect the stratethe fact that he chooses to switch may be visible to the othergies used by other players. But a player may believe that if
players. For example, if he chooses to defect in Prisoner’s
he switches from one strategy to another, the fact that heDilemma, the other player may sense his guilt. (Indeed, itchooses to switch may be visible to the other players. For
is well known that there are facial and bodily clues, such
example, if he chooses to defect in Prisoner’s Dilemma, theas increased pupil size, associated with deception; see, e.g.,other player may sense his guilt. We show that by assuming[Ekman and Friesen 1969]. Professional poker players aretranslucent players, we can recover many of the regularities
also very sensitive to tells—betting patterns and physicalobserved in human behavior in well-studied games such as
demeanor that reveal something about a player’s hand andPrisoner’s Dilemma, Traveler’s Dilemma, Bertrand Compestrategy.)1
tition, and the Public Goods game.
We use the idea of translucency to explain cooperation.
This may at first seem somewhat strange. Typical lab ex1. INTRODUCTION
periments of social dilemmas consider anonymous players,
In the last few decades, numerous experiments have shown
who play each other over computers. In this setting, therethat humans do not always behave so as to maximize their
are no tells. However, as Rand and his colleagues have armaterial payoff. Many alternative models have consequently
gued (see, e.g., [Rand et al. 2012, Rand et al. 2014]), behavbeen proposed to explain deviations from the money-maximization ior of subjects in lab experiments is strongly influenced byparadigm. Some of them assume that players are boundedly
their experience in everyday interactions. People internalize
rational and/or make mistakes in the computation of thestrategies that are more successful in everyday interactions
expected utility of a strategy [Camerer et al. 2004, Costaand use them as default strategies in the lab. We would arGomes et al. 2001, Halpern and Pass 2015, McKelvey andgue that people do not just internalize strategies; they also
Palfrey 1995, Stahl and Wilson 1994]; yet others assume thatinternalize beliefs. In everyday interactions, changing strateplayers have other-regarding preferences [Bolton and Ockengies certainly affects how other players react in the future.
fels 2000, Charness and Rabin 2002, Fehr and Schmidt 1999];
Through tells and possible leaks about changes in plans,
others define radically different solution concepts, assuming
it also may affect how other players react in current play.
that players do not try to maximize their payoff, but rather
Thus, we would argue that, in everyday interactions, peotry to minimize their regret [Halpern and Pass 2012, Renou
ple assume a certain amount of transparency, both because
and Schlag 2010], or maximize the forecasts associated to
it is a way of taking the future into account in real-world
coalition structures [Capraro 2013, Capraro et al. 2013], or
situations that are repeated and because it is a realistic asmaximize the total welfare [Apt and Schäfer 2014, Rong andsumption in one-shot games that are played in settings where
Halpern 2013]. (These references only scratch the surface; a1complete bibliography would be longer than this paper!)The idea of translucency is motivated by some of the same
Cooperative behavior in one-shot anonymous games isconcerns as Solan and Yariv’s [2004] games with espionage,
particularly puzzling, especially in games where non-cooperation but the technical details are quite different. A game withespionage is a two-player extensive-form game that extends
is a dominant strategy (with respect to the material payoffs):
an underlying normal-form game by adding a step where
why should you pay a cost to help a stranger, when no clear
player 1 can purchase some noisy information about player
direct or indirect reward seems to be at stake? Neverthe2’s planned move. Here, the information is free and all playless, the secret of success of our societies is largely due toers may be translucent. Moreover, the effect of the transluour ability to cooperate. We do not cooperate only withcency is modeled by the players’ counterfactual beliefs rather
than by adding a move to the game.
family members, friends, and co-workers. A great deal of87

players have a great deal of social interaction. We claim that
players then apply these beliefs in lab settings where they
are arguably inappropriate.
There is experimental evidence that can be viewed as providing support for players believing that they are transparent. Gilovich et al. [1998] show that people tend to overestimate the extent to which others can discern their internal
states. For instance, they showed that liars overestimate thedetectability of their lies and that people believe that their
feelings of disgust are more apparent than they actually are.
There is also growing evidence that showing people simple
images of watching eyes has a marked effect on behavior,ranging from giving more in Public Goods games to littering less (see [Bateson et al. 2013] for a discussion of someof this work and an extensive list of references). One wayof understanding these results is that the eyes are making
people feel more transparent.
We apply the idea of translucency to a particular class ofgames that we call social dilemmas (cf. [Dawes 1980]). A
social dilemma is a normal-form game with two properties:
1. there is a unique Nash equilibrium sN , which is a purestrategy profile;
2. there is a unique welfare-maximizing profile sW , again
a pure strategy profile, such that each player’s utility
if sW is played is higher than his utility if sN is played.
These uniqueness assumptions are not necessary, but they
make definitions and computations easier. Although theserestrictions are nontrivial, many of the best-studied gamesin the game-theory literature satisfy them, including Prisoner’s Dilemma, Traveler’s Dilemma [Basu 1994], Bertrand
Competition, and the Public Goods game. (See Section 3
for more discussion of these games.)
There are (at least) two reasons why an agent may beconcerned about translucency in a social dilemma: (1) his
opponents may discover that he is planning to defect andpunish him by defecting as well, (2) many other people inhis social group (which may or may not include his opponent) may discover that he is planning to defect (or has
defected, despite the fact that the game is anonymous) andthink worse of him.
For definiteness, we focus here on the first point and assume that, in social dilemmas, players have a degree of belief
α that they are transparent, so that if they intend to cooperate (by playing their component of the welfare-maximizing
strategy) and decide to deviate, there is a probability α that
another player will detect this, and play her component of
the Nash equilibrium strategy. (The assumption that cooperation acts as a default strategy is supported by experiments showing that people forced to make a decision undertime pressure are, on average, more cooperative than thoseforced to made a decision under time delay [Rand et al. 2012,
Rand et al. 2014]. Rand and his colleagues suggest that thisis due to the internalization of strategies that are successful
in everyday interactions.) We assume that these detections
are independent, so that the probability of, for example, exactly two players other than i detecting a deviation by i isα2 (1 − α)N −3 , where N is the total number of players. Of
course, if α = 0, then we are back at the standard gametheoretic framework. We show that, with this assumption,
we can already explain a number of experimental regularities
observed in social dilemmas (see Section 3). We can model

the second point regarding concerns about transparency inmuch the same way, and would get qualitatively similar results (see Section 6).
The rest of the paper is as follows. In Section 2, we formalize the notion of translucency in a game-theoretic setting. In Section 3, we define the social dilemmas that we
focus on in this paper; in Section 4, we show that by assuming translucency, we can obtain as predictions of theframework a number of regularities that have been observedin the experimental literature. We discuss related work in
Section 5. Section 6 concludes. Proofs are deferred to thefull paper, where we also discuss a solution concept that wecall translucent equilibrium, based on translucency, closely
related to the notion of individual rationality discussed by
Halpern and Pass [2013], and show how it can be applied insocial dilemmas.

2.

RATIONALITY WITH TRANSLUCENTPLAYERSIn this section, we briefly define rationality in the presenceof translucency, motivated by the ideas in Halpern and Pass[2013].
Formally, a (finite) normal-form game G is a tuple (P, S1 , . . . , SN ,u1 , . . . , uN ), where P = {1, . . . , N } is the set of players, Si
is the set of strategies for player i, and ui is player
Q i’s utility
function. Let S = S1 × · · · × SN and S−i = j6=i Sj . We
assume that S is finite and that N ≥ 2.
In standard game theory, it is assumed that a player i has
beliefs about the strategies being used by other players; iis rational if his strategy is a best response to these beliefs.
The standard definition of best response is the following.
Definition 2.1. A strategy si ∈ Si is a best response toa probability µi on S−i if, for all strategies s0i for player i,XXµi (s0−i )ui (si , s0−i ) ≥µi (s0−i )ui (s0i , s0−i ).
s0−i ∈S−i

s0−i ∈S−iut
Definition 2.1 implicitly assumes that i’s beliefs about what
other agents are doing do not change if i switches from si , thestrategy he was intending to play, to a different strategy. (Ingeneral, we assume that i always has an intended strategy,
for otherwise it does not make sense to talk about i switching
to a different strategy.) So what we really have are beliefs
s ,s0µi i i for i indexed by a pair of strategies si and s0i ; wes ,s0interpret µi i i as i’s beliefs if he intends to play si but
instead deviates to s0i . Thus, µsi i ,si represents i’s beliefs if
he plays si and does not deviate. We modify the standard
definition of best response by defining best response withs ,s0
respect to a family of beliefs µi i i .
Definition 2.2. Strategy si ∈ Si is a best response for
s ,s0
i with respect to the beliefs {µi i i : s0i ∈ Si } if, for all
strategies s0i ∈ Si ,
X
X
s ,s0
µisi ,si (s0−i )ui (si , s0−i ) ≥µi i i (s0−i )ui (s0i , s0−i ).

s0−i ∈S−i

s0−i ∈S−iut
We are interested in players who are making best responses to their beliefs, but we define best response in terms

88

of Definition 2.2, not Definition 2.1. Of course, the standard
notion of best response is just the special case of the notion
s ,s0
above where µi i i = µsi i ,si for all s0i : a player’s beliefs about
what other players are doing does not change if he switches
strategies.
Definition 2.3. A player is translucently rational if hebest responds to his beliefs in the sense of Definition 2.2. ut
Our assumptions about translucency will be used to des ,s0
termine µi i i . For example, suppose that Γ is a 2-player
game, player 1 believes that, if he were to switch from si to
s0i , this would be detected by player 2 with probability α,
and if player 2 did detect the switch, then player 2 woulds ,s0
switch to s0j . Then µi i i is (1 − α)µsi ,si + αµ0 , where µ0
assigns probability 1 to s0j ; that is, player 1 believes thatwith probability 1 − α, player 2 continues to do what hewould have done all along (as described by µsi ,si ) and, withprobability α, player 2 switches to s0j .

3. SOCIAL DILEMMASSocial dilemmas are situations in which there is a tension
between the collective interest and individual interests: every individual has an incentive to deviate from the common
good and act selfishly, but if everyone deviates, then they are
all worse off. Many personal and professional relationships,
the depletion of natural resources, climate protection, thesecurity of energy supply, and price competition in markets
can all be viewed as instances of social dilemmas.
As we said in the introduction, we formally define a social dilemma as a normal-form game with a unique Nashequilibrium and a unique welfare-maximizing profile, bothpure strategy profiles, such that each player’s utility if sW isplayed is higher than his utility if sN is played. While thisis a quite restricted set of games, it includes many that havebeen quite well studied. Here, we focus on the following
games:
Prisoner’s Dilemma. Two players can either cooperate(C) or defect (D). To relate our results to experimental results on Prisoner’s Dilemma, we think of cooperation as meaning that a player pays a cost c > 0 to givea benefit b > c to the other player. If a player defects,
he pays nothing and gives nothing. Thus, the payoffof (D, D) is (0, 0), the payoff of (C, C) is (b − c, b − c),and the payoffs of (D, C) and (C, D) are (b, −c) and(−c, b), respectively. The condition b > c implies that
(D, D) is the unique Nash equilibrium and (C, C) isthe unique welfare-maximizing profile.
Traveler’s Dilemma. Two travelers have identical luggage,
which is damaged (in an identical way) by an airline.
The airline offers to recompense them for their luggage.
They may ask for any dollar amount between L and H(where L and H are both positive integers). There is
only one catch. If they ask for the same amount, thenthat is what they will both receive. However, if they
ask for different amounts—say one asks for m and theother for m0 , with m < m0 —then whoever asks for m(the lower amount) will get m+b (m and a bonus of b),
while the other player gets m − b: the lower amountand a penalty of b. It is easy to see that (L, L) is
the unique Nash equilibrium, while (H, H) maximizessocial welfare, independent of b.

89

Public Goods game. N ≥ 2 contributors are endowedwith 1 dollar each; they must simultaneously decide
how much, if anything, to contribute to a public pool.
(The contributions must be in whole cent amounts.)
The total contribution pot is then multiplied by a constant strictly between 1 and N , and then evenly redistributed among all players.2 So the payoff of player i is
ui (x1 , . . . , xN ) = 1−xi +ρ(x1 +. . .+xN ), where xi denotes i’s contribution, and ρ ∈ N1 , 1 is the marginal
return. (Thus, the pool is multiplied by ρN before
being split evenly among all players.) Everyone contributing nothing to the pool is the unique Nash equilibrium, and everyone contributing their whole endowment to the pool is the unique welfare-maximizing profile.
Bertrand Competition. N ≥ 2 firms compete to sell theiridentical product at a price between the “price floor”
L ≥ 2 and the “reservation value” H. (Again, we assume that H and L are integers, and all prices must
be integers.) The firm that chooses the lowest price,
say s, sells the product at that price, getting a payoffof s, while all other firms get a payoff of 0. If there
are ties, then the sales are split equally among all firmsthat choose the lowest price. Now everyone choosing L
is the unique Nash equilibrium, and everyone choosing
H is the unique welfare-maximizing profile.3
From here on, we say that a player cooperates if he plays
his part of the welfare-maximizing strategy profile and defects if he plays his part of the Nash equilibrium strategy
profile.
While Nash equilibrium predicts that people should always defect in social dilemmas, in practice, we see a great
deal of cooperative behavior; that is, people often play (theirpart of) the welfare-maximizing profile rather than (theirpart of) the Nash equilibrium profile. Of course, there havebeen many attempts to explain this. Evolutionary theories may explain cooperative behavior among genetically related individuals [Hamilton 1964] or when future interactions among the same subjects are likely [Nowak and Sigmund 1998, Trivers 1971]; see [Nowak 2006] for a review ofthe five rules of cooperation. However, we often observe cooperation even in one-shot anonymous experiments among
unrelated players [Rapoport 1965].
Although we do see a great deal of cooperation in thesegames, we do not always see it. Here are some of the regularities that have been observed:• The degree of cooperation in the Prisoner’s dilemma
depends positively on the benefit of mutual cooperation and negatively on the cost of cooperation [Capraro
et al. 2014, Engel and Zhurakhovska 2012, Rapoport
1965].
2
We thus consider only linear Public Goods games. This
choice is motivated by the fact that our purpose is to compare the predictions of our model with experimental data.
Most experiments have adopted linear Public Goods games,
since they have much easier instructions and thus they minimize noise due to participants not understanding the rulesof the game.
3
We require that L ≥ 2 for otherwise we would not have aunique Nash equilibrium, a condition we imposed on Social
Dilemmas. If L = 1 and N = 2, we get two Nash equilibria:(2, 2) and (1, 1); similarly, for L = 0, we also get multiple
Nash equilibria, for all values of N ≥ 2.

• The degree of cooperation in the Traveler’s Dilemma
depends negatively on the bonus/penalty [Capra et al.
1999].
• The degree of cooperation in the Public Goods gamedepends positively on the constant marginal return[Gunnthorsdottir et al. 2007, Isaac et al. 1984].
• The degree of cooperation in the Public Goods gamedepends positively on the number of players [Barcelo
and Capraro 2015, Isaac et al. 1994, Zelmer 2003].
• The degree of cooperation in the Bertrand Competition depends negatively on the number of players[Dufwenberg and Gneezy 2002].
• The degree of cooperation in the Bertrand Competition depends negatively on the price floor [Dufwenberg
et al. 2007].

4. EXPLAINING SOCIAL DILEMMAS USING TRANSLUCENCY
As we suggested in the introduction, we hope to use translucency to explain cooperation in social dilemmas even whenplayers cannot see each other. We expect that people get
so used to assuming some degree of transparency in theireveryday interactions, which are typically face-to-face, thatthey bring these strategies and beliefs in the lab setting,even though they are arguably inappropriate.
To do this, we have to make assumptions about an agent’s
beliefs. Say that an agent i has type (α, β, C) if i intends
to cooperate (the parameter C stands for cooperate) andbelieves that (a) if he deviates from that, then each otheragent will independently realize this with probability α; (b)if an agent j realizes that i is not going to cooperate, thenj will defect; and (c) all other players will either cooperate
or defect, and they will cooperate with probability β.
The standard assumption, of course, is that α = 0. Our
results are only of interest if α > 0. The assumption that ibelieves that agent j will defect if she realizes that i is going
to deviate from cooperation seems reasonable; defection is
the “safe” strategy. We stress that, for our results, it does
not matter what j actually does. All that matters are i’sbeliefs about what j will do. The assumption that players
will either cooperate or defect is trivially true in Prisoner’s
Dilemma, but is a highly nontrivial assumption in the othergames we consider. While cooperation and defection are
arguably the most salient strategies, we do in practice see
players using other strategies. For instance, the distribution of strategies in the Public Goods game is typically trimodal, concentrated on contributing nothing, contributing
everything, and contributing half [Capraro et al. 2014]. We
made this assumption mainly for technical convenience: it
makes the calculations much easier. We believe that results
qualitatively similar to ours will hold under a much weakerassumption, namely, that a type (α, β, C) player believes
that other players will cooperate with probability β (without assuming that they will defect with probability 1 − β).
Similarly, the assumptions that a social dilemma has aunique Nash equilibrium and a unique social-welfare maximizing strategy were made largely for technical reasons. We
can drop these assumptions, although that would require
more complicated assumptions about players’ beliefs.

Our assumptions ensure that the type of player i deters ,s0mines the distributions µi i i . In a social dilemma with
N agents, the distribution µisi ,si assigns probability β r (1 −β)N −1−r to a strategy profile s−i for the players other than iif exactly r players cooperate in s−i and the remaining N −1−r players defect; it assigns probability 0 to all other strats ,s0
egy profiles. The distributions µi i i for s0i 6= si all have thePform J⊆{1,...,i−1,i+1,...,N } α|J| (1 − α)N −1−|J| µJi , where µJi

is the distribution that assigns probability β k (1 − β)N −|J|−k
to a profile where k ≤ N −1−|J| players not in J cooperate,
and the remaining players (which includes all the players inJ) defect. Thus, µJi is the distribution that describes whatplayer i’s beliefs would be if he knew that exactly the players
in J had noticed his deviation (which happens with probability α|J| (1 − α)N −1−|J| ). In the remainder of this section,
when we talk about best response, it is with respect to these
beliefs.
For our purposes, it does not matter where the beliefs α
and β that make up a player’s type come from. We do notassume, for example, that other players are (translucently)
rational. For example, i may believe that some players cooperate because they are altruistic, while others may cooperate because they have mistaken beliefs. We can think of βas summarizing i’s previous experience of cooperation whenplaying social dilemmas. Here we are interested in the impact of the parameters of the game on the reasonableness ofcooperation, given a player’s type.
The following four propositions analyze the four social
dilemmas in turn; the proofs can be found in the full paper. We start with Prisoner’s Dilemma. Recall that b is
the benefit of cooperation and c is its cost.
Proposition 4.1. In Prisoner’s Dilemma, it is translucently rational for a player of type (α, β, C) to cooperate if
and only if αβb ≥ c. ut
As we would expect, if α = 0, then cooperation is not abest response in Prisoner’s Dilemma; this is just the standard argument that defection dominates cooperation. But
if α > 0, then cooperation can be rational. Moreover, if we
fix α, the greater the benefit of cooperation and the smaller
the cost, then the smaller the value of β that still allows
cooperation to be a best response.
We next consider Traveler’s Dilemma. Recall that b is the
reward/punishment, H is the high payoff, and L is the low
payoff,
Proposition 4.2. In Traveler’s Dilemma, it is translucently rational for a player of (α, β, C) to cooperate if andonly if( (H−L)βif α ≥ 12
1−αβ


b≤
min (H−L)β, H−L−1
if α < 12 .
1−αβ
1−2α
u
t

Proposition 4.2 shows that as b, the punishment/reward,
increases, a player must have greater belief that his opponent
is cooperative and/or a greater belief that the opponent willlearn about his deviation and/or a greater difference between
the high and low payoffs in order to make cooperation abest response. (The fact that increasing β increases (H−L)β1−αβfollows from straightforward calculus.)90

We next consider the Public Goods game. Recall the ρ isthe marginal return of cooperating.

extent to which player i is averse to inequity in favorSof others, and bF
represents his aversion to inequityiin his favor. Consider the Public Goods game with N
players. The strategy profile (x, . . . , x), where all players contribute x gives player i a utility of (1−x)+ρN x.
If x > 0 and player i contributes x0 < x, then his
S0
payoff is (1 − x0 ) + ρ((N − 1)x + x0 ) − bF
i ρ(x − x ).
FS
0Thus, (x, . . . , x) is an equilibrium if bi ρ(x − x ) ≥S(1 − ρ)(x − x0 ), that is, if bF≥ (1 − ρ)/ρ. Thus,iFS
if bi ≥ (1 − ρ)/ρ for all players i, then (x, . . . , x)
is an equilibrium for all choices of x and all values of
N . While there may be other pure and mixed strategySequilibria, it is not hard to show that if bF< (1−ρ)/ρ,ithen player i will play 0 in every equilibrium (i.e., notcontribute anything). As a consequence, assuming, as
in our model, that players believe that there is a probability β that other agents will cooperate and thatthe other agents either cooperate or defect, Fehr andSchmidt [1999] model does not make any clear prediction of a group-size effect on cooperation in the public
goods game.

Proposition 4.3. In the Public Goods game with N players, it is translucently rational for a player of type (α, β, C)
to cooperate if and only if αβρ(N − 1) ≥ 1 − ρ. ut
Proposition 4.3 shows that if ρ = 1, then cooperation iscertainly a best response (you always get out at least as
much as you contribute). For fixed α and β, there is guaranteed to be an N0 such that cooperation is a best response
for all N ≥ N0 ; moreover, for fixed α, as N gets larger,
smaller and smaller βs are needed for cooperation to be a
best response.
Finally we consider the Bertrand competition. Recall that
H is the reservation value and L is the price floor.
Proposition 4.4. In Bertrand Competition, it is translucently rational for a player of type (α, β, C) to cooperate iff
β N −1 ≥ max(γ N −1 N (H − 1)/H, f (γ,N )LN/H), where γ =P −1 N −1
k N −k−1(1−γ)γ/(k+1).
(1−α)β and f (γ, N ) = N
k=0
k
u
t
PN −1 N −1
Note that f (γ, N ) =(1 − γ)k γ N −k−1 /(k +k=0kPN −1 N −1
k N −k1) ≥(1 − γ) γ/N = 1/N , so Proposik=0ktion 4.4 shows cooperation is irrational if β N −1 < L/H.
Thus, while cooperation may be achieved for reasonable values of α and β if N is small, a player must be more and morecertain of cooperation in order to cooperate in Bertrand
Competition as the number of players increases. Indeed,
for a fixed type (α, β, C), there exists N0 such that cooperation is not a best response for all N ≥ N0 . Moreover,
if we fix the number N of players, more values of α and β
allow cooperation as L/H gets smaller. In particular, if we
fix H and raise the floor L, fewer values of α and β allow
cooperation.
While Propositions 4.1–4.4 are suggestive, we need to makeextra assumptions to use these propositions to make predictions. A simple assumption that suffices is that there are a
substantial number of translucently rational players whosetypes have the form (α, β, C), and for each pair (u, v) and(u0 , v 0 ) of open intervals in [0, 1], there is a positive probability of finding someone of type (α, β, C) with α ∈ (u, v)
and β ∈ (u0 , v 0 ). With this assumption, it is easy to see that
all the regularities discussed in Section 3 hold.

5.

• McKelvey and Palfrey’s [1995] quantal response equilibrium (QRE) is defined as follows.4 Taking σi (s)to be the probability that mixed strategy σi assigns
to the pure strategy s, given λ > 0, a mixed strategy profile σ is a QRE if, for each player i, σi (s) =P

λEUi (s0 ,σ−i )i.

• Iterated regret minimization [Halpern and Pass 2012]
does not make appropriate predictions in Prisoner’s
Dilemma and the Public Goods game, because it predicts that if there is a dominant strategy then it will
be played, and in these two games, playing the Nashequilibrium is the unique dominant strategy.

COMPARISON TO OTHER APPROACHES

91

λEUi (s,σ−i )To see that QRE does not describe human behaviour
well in social dilemmas, observe that in the Prisoner’s
Dilemma, for all choices of parameters b and c in thegame, all choices of the parameter λ, all players i,and all (mixed) strategies s−i of player −i, we have
EUi (C, s−i ) < EUi (D, s−i ). Consequently, whatever
the QRE σ is, we must have σi (C) < 12 < σi (D), that
is, QRE predicts that the degree of cooperation cannever be larger than 50%. However, experiments showthat we can increase the benefit-to-cost ratio so as toreach arbitrarily large degrees of cooperation (close to
80% in [Capraro et al. 2014] with b/c = 10).

Here we show that approaches (that we are aware of) otherthan that of Charness and Rabin and possibly that of Bolton
and Ockenfels are not able to obtain all the regularities that
we mentioned in Section 3. We consider a number of approaches in turn.
• The Fehr and Schmidt [1999] inequity-aversion model
assumes that subjects play a Nash equilibrium of amodified game, in which players do not only care about
their monetary payoff, but also they care about equity.
Specifically, player i’s utility when strategy s is played
aF S Pis assumed to be Ui (s) = ui (s)− Ni−1 j6=i max(uj (s)−bF S Pui (s), 0)− Ni−1 j6=i max(ui (s)−uj (s), 0), where ui (s)SS
is the material payoff of player i, and 0 ≤ bF
≤ aF
i
i
FS
are individual parameters, where ai represents thee

s0 ∈Si ei• Capraro’s [2013] notion of cooperative equilibrium, while
correctly predicting the effects of the size of the groupon cooperation in the Bertrand Competition and the
Public Goods game [Barcelo and Capraro 2015], fails
to predict the negative effect of the price floor on cooperation in the Bertrand Competition.
• Rong and Halpern’s [2010, 2013] notion of cooperativeequilibrium (which is different from that of Capraro[2013]) focuses on 2-player games. However, the definition for games with greater than 2 players does notpredict the decrease in cooperation as N increases in
4

We actually define here a particular instance of QRE called
the logit QRE ; λ is a free parameter of this model.

Bertrand Competition, nor the increase as N increases
in the Public Goods Game.
• Bolton and Ockenfels’ [2000] inequity-aversion model
assumes that a player i aims at maximizing his or
her motivational function vi = vi (xi ,Pσi ), where xi is
i’s monetary payoff and σi = σi (x1 , j=1,...,N xj ) =Pxi / j=1,...,N xj . The motivational function is assumed
to be twice differentiable, weakly increasing in the firstargument, and concave in the second argument with amaximum at σi = N1 , but otherwise is unconstrained.
For each of the social dilemmas that we have considered, it is not hard to define a motivational functionthat will obtain the regularities observed. However,
we have not been able to find a single motivational
function that gives the observed regularities for all
four social dilemmas that we have considered. In any
case, just as with the Charness and Rabin model, once
we consider the interaction between social groups andtranslucency, we can distinguish our approach fromthis inequity-aversion model. Specifically, consider asituation where people are given a choice between giving $1 to an anonymous stranger, rather than burning
it. In such a situation, inequity aversion would predictthat people would burn the dollar to maintain equity(i.e., a situation where no one gets $1). However, perhaps not surprisingly, Capraro et al. [2014] found thatover 90% people prefer giving away the dollar to burning it. Of course, translucency (and a number of otherapproaches) would have no difficulty in explaining thisphenomenon.
The one approach besides ours that we are aware of thatobtains all the regularities discussed above is that of Charness and Rabin [2002]. Charness and Rabin, like Fehr andSchmidt [1999], assume that agents play a Nash equilibriumof a modified game, where players care not only about theirpersonal material payoff, but also about the social welfare
and the outcome of the least fortunate person. Specifically, player i’s utility is assumed to be (1 − aCR)ui (s) +
i
CR PN
CR
)
u
(s)).
Assummin
u
(s)
+
(1
−
b
(b
aCR
j
j=1,...,N
j
i
i
i
j=1
ing, as in our model, that agents believe that other players either cooperate or defect and that they cooperate withprobability β, then it is not hard to see that Charness andRabin [2002] also predict all the regularities that we havebeen considering.
Although it seems difficult to distinguish our model fromthat of Charness and Rabin [2002] if we consider only social dilemmas, the models are distinguishable if we look atother settings and take into account the other reason wementioned for translucency: that other people in their social group might discover how they acted. We can easily
capture this in the framework we have been considering by
doubling the number of agents; for each player i, we add
another player i∗ that represent’s i’s social network. Playeri∗ can play only two actions: n (for “did not observe player
i’s action) and o (for “observed player i’s action”).5 The
payoffs of these new players are irrelevant. Player i’s payoff
depends on the action of player i∗ , but not on the actions ofplayer j ∗ for j ∗ 6= i∗ . Now player i must have a prior probability γi about whether his action will be observed; in a
5
Alternatively, we could take player i’s payoff to depend on
the state of the world, where the state would model whether
or not player i’s action was observed.

social dilemma, this probability might increase to γi0 ≥ γi if
he intends to cooperate but instead deviates and defects. It
should be clear that, even if γi0 = γi , if we assume that player
i’s utilities are significantly lower if his non-cooperative action is observed, with this framework we would get qualitatively similar results for social dilemmas to the ones that
we have already obtained. Again, a player has beliefs about
the extent to which he is transparent, and we can set thepayoffs so that the effects of transparency are the same if aplayer’s social network learns about his actions and if otherplayers learn about his action.
The advantage of taking into account what your socialgroup thinks is that it allows us to apply ideas of translucency even to single-player games like the Dictator Game
[Kahneman et al. 1986]. To do so, we need to make assumptions about what a player’s utility would be if his socialgroup knew the extent to which he shared the pot. But
it should be clear that reasonable assumptions here would
lead to some degree of sharing. While this would still not
distinguish our predictions from those of the Charness-Rabinmodel, there is a variant of the Dictator Game that has recently been considered to show existence of hyper-altruism
in conflict situations [Crockett et al. 2014, Capraro 2014].
In the simplest version of this game, there are only two possible allocations of money: either the agent gets x and theother player gets −x, or the other player gets x and theagent gets −x. In this game, the Charness-Rabin approach
would predict that the agent will either keep x or be indifferent between keeping x and giving it away. But assuming
translucency allows for the possibility that some types of
agents would think that their social group would approve ofthem giving away x, so if the action were observed by theirsocial group, they would get high utility by giving away x.
However, recent results by Capraro [2014] show that a significant fraction (1/6) of people are hyper-altruistic: they
strictly prefer giving away x to keeping it [Capraro 2014].
Just to be clear, we do not mean to imply that translucency is the unique “right” explanation for cooperation insocial dilemmas and all the other explanations that we discussed above are “wrong”. There are probably a number offactors that contribute to cooperation. We hope in future
work to tease these apart.

6.

DISCUSSION

We have presented an approach that explains a number ofwell-known observations regarding the extent of cooperation
in social dilemmas. In addition, our approach can also be
applied to explain the apparent contradiction that people
cooperate more in a one-shot Prisoner’s dilemma when they
do not know the other player’s choice than when they do. In
the latter case, Shafir and Tversky [1992] found that mostpeople (90%) defect, while in the former case, only 63% of
people defect. Our model of translucent players predicts thisbehavior: if player 1 knows player 2’s choices then there is notranslucency, so our model predicts that player 1 defects forsure. On the other hand, if player 1 does not know player
2’s choice and believes that he is to some extent translucent, then, as shown in Proposition 4.1, he may be willingto cooperate. Seen in this light, our model can also be interpreted as an attempt to formalize quasi-magical thinking[Shafir and Tversky 1992], the kind of reasoning that is supposed to motivate those people who believe that the others’
reasoning is somehow influenced by their own thinking, even92

though they know that there is no causal relation between
the two. Quasi-magical thinking has also been formalized
by Masel [2007] in the context of the Public Goods gameand by Daley and Sadowski [2014] in the context of symmetric 2 × 2 games. The notion of translucency goes beyond
these models, since it may be applied to a much larger set
of games.
Besides a retrospective explanation, our model makes new
predictions for social dilemmas which, to the best of ourknowledge, have never been tested in the lab. In particular,
it predicts that• the degree of cooperation in Traveler’s dilemma increases as the difference H − L increases;• for fixed L and N , the degree of cooperation in Bertrand
Competition increases as H increases, and what reallymatters is the ratio L/H.
Clearly much more experimental work needs to be done
to validate the approach. For one thing, it is important tounderstand the predictions it makes for other social dilemmas and for games that are not social dilemmas. Perhaps
even more important would be to see if we can experimentally verify that people believe that they are to some extent
translucent, and, if so, to get a sense of what the value of
α is. In light of the work on watching eyes mentioned in
the introduction, it would also be interesting to know what
could be done to manipulate the value of α.
One feature of our approach is that, at least if we takethe concern with translucency to be due to an opponent discovering what you are going to do (rather than other members of your social group discovering what you are going todo), then, unlike many other approaches to explaining social
dilemmas, our approach does not involve modifying the utility function; that is, we can apply translucency while still
identifying utility with the material payoff. While this makeit an arguably simpler explanation, that does not necessarily make it “right”, of course. We do not in fact believe that
there is a unique “right” explanation for cooperation in social
dilemmas and all the other explanations that we discussed
above are “wrong”. There are probably a number of factorsthat contribute to cooperation. We hope in future work totease these apart.
Of course, we do not have to assume α > 0 to get cooperation in social dilemmas such as Traveler’s Dilemma or
Bertrand Competition. But we do if we want to consider
what we believe is the appropriate equilibrium notion. Suppose that rational players are chosen at random from a population and play a social dilemma. Players will, of course,then update their beliefs about the likelihood of seeing cooperation, and perhaps change their strategy as a consequence.
Will these beliefs stabilize and the strategies played stabilize? By stability here, we mean that (1) players are all best
responding to their beliefs, and (2) players’ beliefs about thestrategies played by others are correct: if player i ascribes
probability p to player j playing a strategy sj , then in facta proportion p of players in the population play sj . We
have deliberately been fuzzy here about whether we mean
best response in the sense of Definition 2.1 or Definition 2.2.
If we use Definition 2.1 (or, equivalently use Definition 2.2
and take α = 0), then it is easy to see (and well known)that the only way that this can happen is if the distribution of strategies played by the players represents a mixed

93

strategy Nash equilibrium. On the other hand, if α > 0 and
we use Definition 2.2, then we can have stable beliefs that
accurately reflect the strategies used and have cooperation(in all the other social dilemmas that we have studied). We
make this precise in the full paper, using the frameworkof Halpern and Pass [2013], by defining a notion of translucent equilibrium. Roughly speaking, we construct a model
where, at all states, players are translucently rational (so wehave common belief of translucent rationality), the strategies used are common knowledge, and we nevertheless havecooperation at some states. Propositions 4.1–4.4 play a keyrole in this construction; indeed, as long as the strategies
used satisfy the constraints imposed by these results, we geta translucent equilibrium.
In the full paper, we also characterize those profiles ofstrategies that can be translucent equilibria, using ideas similar in spirit to those of Halpern and Pass [2013]. While
allowing people to believe that they are to a certain extent
transparent means that the set of translucent equilibria is asuperset of the set of Nash equilibria, not all strategy profiles
can be translucent equilibria. For example, (C,D) is not atranslucent equilibrium in Prisoner’s dilemma. We have notfocused on translucent equilibrium in the main text, because
it makes strong assumptions about players’ rationality andbeliefs (e.g., it implicitly assumes common belief of translucent rationality). We do not need such strong assumptions
for our results.

7.

REFERENCES[2014] Apt, K. and G. Schäfer (2014). Selfishness level ofstrategic games. Journal of Artificial Intelligence
Research 49, 207–240.
[2015] Barcelo, H. and V. Capraro (2015). Group size
effect on cooperation in one-shot social dilemmas.
Scientific Reports 5.
[1994] Basu, K. (1994). The traveler’s dilemma: paradoxesof rationality in game theory. American Economic
Review 84 (2), 391–395.
[2013] Bateson, M., L. Callow, J. R. Holmes, M. L.
Redmond Roche, and D. Nettle (2013). Do images of“watching eyes” induce behaviour that is more pro-social
or more normative? A field experiment on littering.
PLoS ONE 8 (12).
[2000] Bolton, G. E. and A. Ockenfels (2000). ERC: A
theory of equity, reciprocity, and competition. American
Economic Review 90 (1), 166–193.
[2003] Camerer, C. F. (2003). Behavioral Game Theory:
Experiments in Strategic Interaction. Princeton, N. J.:
Princeton University Press.
[2004] Camerer, C. F., T.-H. Ho, and J.-K. Chong (2004).
A cognitive hierarchy model of games. Quarterly Journalof Economics 119, 861–897.
[1999] Capra, M., J. K. Goeree, R. Gomez, and C. A. Holt
(1999). Anomalous behavior in a traveler’s dilemma.
American Economic Review 89 (3), 678–690.
[2013] Capraro, V. (2013). A model of human cooperation
in social dilemmas. PLoS ONE 8 (8), e72427.
[2014] Capraro, V. (2014). The emergence of altruistic
behaviour in conflictual situations. Working Paper.
[2014] Capraro, V., J. J. Jordan, and D. G. Rand (2014).
Heuristics guide the implementation of social preferencesin one-shot Prisoner’s Dilemma experiments. ScientificReports 4.
[2014] Capraro, V., C. Smyth, K. Mylona, and G. A. Niblo(2014). Benevolent characteristics promote cooperative
behaviour among humans. PloS ONE 9 (8), e102881.
[2013] Capraro, V., M. Venanzi, M. Polukarov, and N. R.
Jennings (2013). Cooperative equilibria in iterated social
dilemmas. In Proc. Sixth International Symposium on
Algorithmic Game Theory (SAGT ’13), pp. 146–158.
[2002] Charness, G. and M. Rabin (2002). Understandingsocial preferences with simple tests. Quarterly Journal of
Economics 117 (3), 817–869.
[2001] Costa-Gomes, M., V. Crawford, and B. Broseta(2001). Cognition and behavior in normal form games:
An experimental study. Econometrica 69 (5), 1193–1235.
[2014] Crockett, M. J., Z. Kurth-Nelson, J. Z. Siegel,
P. Dayan, and R. J. Dolan (2014). Harm to others
outweighs harm to self in moral decision making.
Proceedings of the National Academy of
Sciences 111 (48), 17320–17325.
[2014] Daley, B. and P. Sadowski (2014). A strategic modelof magical thinking: Axioms and analysis. Available at
http://www.princeton.edu/economics/seminar-scheduleby-prog/behavioralf14/Daley Sadowski MT.pdf.
[1980] Dawes, R. M. (1980). Social dilemmas. Annual
Review of Psychology 31, 169–193.
[2002] Dufwenberg, M. and U. Gneezy (2002). Information
disclosure in auctions: an experiment. Journal of
Economic Behavior and Organization 48, 431–444.
[2007] Dufwenberg, M., U. Gneezy, J. K. Goeree, and
R. Nagel (2007). Price floors and competition. Special
Issue of Economic Theory 33, 211–224.
[1969] Ekman, P. and W. Friesen (1969). Nonverbal
leakage and clues to deception. Psychiatry 32, 88–105.
[2012] Engel, C. and L. Zhurakhovska (2012). When is therisk of cooperation worth taking? The Prisoner’s
Dilemma as a game of multiple motives. Working Paper.
[1999] Fehr, E. and K. Schmidt (1999). A theory offairness, competition, and cooperation. Quarterly
Journal of Economics 114 (3), 817–868.
[1998] Gilovich, T., K. Savitsky, and V. H. Medvec (1998).
The illusion of transparency: biased assessments ofothers’ ability to read one’s emotional states. Journal of
Personality and Social Psychology 75 (2), 332.
[2007] Gunnthorsdottir, A., D. Houser, and K. McCabe(2007). Dispositions, history and contributions in public
goods experiments. Journal of Economic Behavior andOrganization 62 (2), 304–315.
[2012] Halpern, J. Y. and R. Pass (2012). Iterated regret
minimization: a new solution concept. Games andEconomic Behavior 74 (1), 194–207.
[2013] Halpern, J. Y. and R. Pass (2013). Game theory
with translucent players. In Theoretical Aspects ofRationality and Knowledge: Proc. Fourteenth Conference(TARK 2013), pp. 216–221.
[2015] Halpern, J. Y. and R. Pass (2015). Algorithmic
rationality: Game theory with costly computation.
Journal of Economic Theory 156, 246–268.
[2010] Halpern, J. Y. and N. Rong (2010). Cooperativeequilibrium. In Proc. Ninth International Joint
Conference on Autonomous Agents and Multiagent
Systems, pp. 1465–1466.

[1964] Hamilton, W. D. (1964). The genetical evolution ofsocial behavior. i. Journal of Theoretical Biology 7, 1–16.
[1984] Isaac, M. R., J. M. Walker, and S. Thomas (1984).
Divergent evidence on free riding: an experimental
examination of possible explanations. Public
Choice 43 (1), 113–149.
[1994] Isaac, M. R., J. M. Walker, and A. W. Williams(1994). Group size and the voluntary provision of public
goods. Journal of Public Economics 54, 1–36.
[1986] Kahneman, D., J. Knetsch, and R. H. Thaler(1986). Fairness and the assumptions of economics.
Journal of Business 59 (4), S285–300.
[2007] Masel, J. (2007). A Bayesian model of quasi-magicalthinking can explain observed cooperation in the public
good game. Journal of Economic Behavior andOrganization 64 (1), 216–231.
[1995] McKelvey, R. and T. Palfrey (1995). Quantal
response equilibria for normal form games. Games andEconomic Behavior 10 (1), 6–38.
[2006] Nowak, M. A. (2006). Five rules for the evolution ofcooperation. Science 314 (5805), 1560–1563.
[1998] Nowak, M. A. and K. Sigmund (1998). Evolution ofindirect reciprocity by image scoring. Nature 393,
573–577.
[2012] Rand, D. G., J. D. Green, and M. A. Nowak (2012).
Spontaneous giving and calculated greed. Nature 489,
427–430.
[2014] Rand, D. G., A. Peysakhovich, G. T. Kraft-Todd,
G. E. Newman, O. Wurzbacher, M. A. Nowak, and J. D.
Greene (2014). Social heuristics shape intuitive
cooperation. Nature Communications 5, 3677.
[1965] Rapoport, A. (1965). Prisoner’s Dilemma: A Studyin Conflict and Cooperation. University of Michigan
Press.
[2010] Renou, L. and K. H. Schlag (2010). Minimax regret
and strategic uncertainty. Journal of Economic
Theory 145, 264–286.
[2013] Rong, N. and J. Y. Halpern (2013). Towards a
deeper understanding of cooperative equilibrium:characterization and complexity. In Proc. Twelfth
International Joint Conference on Autonomous Agentsand Multiagent Systems, pp. 319–326.
[2013] Salcedo, B. (2013). Implementation withoutcommitment in moral hazard environments. Working
paper.
[1992] Shafir, E. and A. Tversky (1992). Thinking throughuncertainty: Nonconsequential reasoning and choice.
Cognitive Psychology 24, 449–474.
[2004] Solan, E. and L. Yariv (2004). Games withespionage. Games and Economic Behavior 47, 172–199.
[1994] Stahl, D. and P. Wilson (1994). Experimental
evidence on players’ models of other players. Journal of
Economic Behavior and Organization 25 (3), 309–327.
[1971] Trivers, R. (1971). The evolution of reciprocal
altruism. Quarterly Review of Biology 46, 35–57.
[2003] Zelmer, J. (2003). Linear public goods experiments:
A meta-analysis. Experimental Economics 6, 299–310.

94

Announcement as effort on topological spaces
Hans van Ditmarschhans.vanditmarsch@loria.fr

Sophia Knightsophia.knight@gmail.com

Aybüke Özgün
aybuke.ozgun@loria.fr

LORIA, CNRS - Université de Lorraine

ABSTRACT
We propose a multi-agent logic of knowledge, public andarbitrary announcements, that is interpreted on topologicalspaces in the style of subset space semantics. The arbitrary announcement modality functions similarly to the effort modality in subset space logics, however, it comes withintuitive and semantic differences. We provide axiomatizations for three logics based on this setting, and demonstrate
their completeness.

Keywords
Topology, subset space logic, dynamic epistemic logic, arbitrary (public) announcements

1. INTRODUCTION
In [13], Moss et al. introduce a bi-modal logic with language
ϕ ::= p | ¬ϕ | ϕ ∧ ϕ | Kϕ | 2ϕ,called subset space logic (SSL), in order to formalize reasoning about sets and points together in one modal system.
The main interest in their investigation lies in spatial structures such as topological spaces and using modal logic andthe techniques behind for spatial reasoning, however, theyalso have a strong motivation from epistemic logic. While
the modality K is interpreted as knowledge, 2 intends tocapture the notion of effort, i.e., any action that results inincrease in knowledge. They propose subset space semantics for their logic. A subset space is defined to be a pair
(X, O), where X is a non-empty domain and O is a collection of subsets of X (not necessarily a topology), wherein
the modalities K and 2 are evaluated with respect to pairsof the form (x, U ), where x ∈ U ∈ O. According to subset
space semantics, given a pair (x, U ), the modality K quantifies over the elements of U , whereas 2 quantifies over all
open subsets of U that include the actual world x. Therefore,
while knowledge is interpreted ‘locally’ in a given observation set U , effort is read as open-set-shrinking where more
effort corresponds to a smaller neighbourhood, thus, a possible increase in knowledge. The schema 3Kϕ states thatafter some effort the agent comes to know ϕ where effort canbe in the form of measurement, observation, computation,
approximation [13, 8, 14, 5], or announcement [15, 1, 16].
The epistemic motivation behind the subset space semantics and the dynamic nature of the effort modality suggests
a link between SSL and dynamic epistemic logic, in particular dynamics known as public announcement [4, 5, 3,

95

19, 6]. The works [4, 5, 3] propose modelling public announcements on subset spaces by deleting the states or theneighbourhoods falsifying the announcement. This dynamic
epistemic method is not in the spirit of the effort modality:
dynamic epistemic actions result in global model change,
whereas the effort modality results in local neighbourhood
shrinking. Hence, it is natural to search for an ‘open-setshrinking-like’ interpretation of public announcements on
subset spaces. To best of our knowledge, Wang and Ågotnes[19] were the first to propose semantics for public announcements on subset spaces in the style of the effort modality,
although this is not necessarily on topological spaces. Bjorndahl [6] then proposed a revised version of the [19] semantics.
In contrast to the aforementioned proposals, Bjorndahl uses
models based on topological spaces to interpret knowledge
and information change via public announcements. He considers the language
ϕ ::= p | ¬ϕ | ϕ ∧ ϕ | Kϕ | int(ϕ) | [ϕ]ϕ,where int(ϕ) means ‘ϕ is true and can be announced’, andwhere [ϕ]ψ means ‘after public announcement of ϕ, ψ.’In [1], Balbiani et al. introduce a logic to quantify overannouncements in the setting of epistemic logic based on
the language (with single-agent version here)
ϕ ::= p | ¬ϕ | ϕ ∧ ϕ | Kϕ | [ϕ]ϕ | 2ϕ.
In this case, unlike above, 2ϕ means ‘after any announcement, ϕ (is true)’ so that 2 quantifies over epistemically definable subsets (2-free formulas of the language) of a given
model. In this case, 3Kϕ again means that the agent comesto know ϕ, but in the interpretation that there is a formulaψ such that after announcing it the agent knows ϕ. What
becomes true or known by an agent after an announcement
can be expressed in this language without explicit reference
to the announced formula.
Clearly, the meaning of the effort 2 modality and of the arbitrary announcement 2 modality are related in motivation.
In both cases, interpreting the modality requires quantification over sets. Subset-space-like semantics provides natural
tools for this. In [16], we extended Bjorndahl’s proposal [6]with an arbitrary announcement modality
ϕ ::= p | ¬ϕ | ϕ ∧ ϕ | Kϕ | int(ϕ) | [ϕ]ϕ | 2ϕand provided topological semantics for the 2 modality, andproved completeness for the corresponding single-agent logic
AP ALint .
In the current proposal we generalize this approach to a
multi-agent setting. Multi-agent subset space logics have

been investigated in [11, 12, 4, 18]. There are some challenges with such a logic concerning the evaluation of higherorder knowledge. The general setup is for any finite number of agents, but to demonstrate the challenges, consider
the case of two agents. Suppose for each of two agents iand j there is an open set such that the semantic primitive
becomes a triple (x, Ui , Uj ) instead of a pair (x, U ). Now
consider a formula like Ki K̂j Ki p, for ‘agent i knows thatagent j considers possible that agent i knows proposition p’.
If this is true for a triple (x, Ui , Uj ), then K̂j Ki p must betrue for any y ∈ Ui ; but y may not be in Uj , in which case(y, Ui , Uj ) is not well-defined: we cannot interpret K̂j Ki p.
Our solution to this dilemma is to consider neighbourhoods
that are not only relative to each agent, as usual in multiagent subset space logics, but that are also relative to eachstate. This amounts to, when shifting the viewpoint fromx to y ∈ Ui , in (x, Ui , Uj ), we simultaneously have to shift
the neighbourhood (and not merely the point in the actual
neighbourhood) for the other agent. So we then go from(x, Ui , Uj ) to (y, Ui , Vj ), where Vj may be different from Uj .
If they are different, their intersection should be empty.
In order to define the evaluation neighbourhood for eachagent with respect to the state in question, we employ atechnique inspired by the standard neighbourhood semantics[7]. We use a set of neighbourhood functions, determining theevaluation neighbourhood relative to both the given state
and the corresponding agent. These functions need to bepartial in order to render the semantics well-defined for the
dynamic modalities in the system.
In Section 2 we define the syntax, structures, and semantics of our multi-agent logic of arbitrary public announcements, AP ALint , interpreted on topological spaces equippedwith a set of neighbourhood functions. Without arbitrary
announcements we get the logic P ALint , and with neither
arbitrary nor public announcements, the logic ELint . In thissection we also show some typical validities of the logic, andgive a detailed example. In Section 3 we give axiomatizations for the logics: P ALint extends ELint and AP ALint
extends P ALint . In Section 4 we demonstrate completenessfor these logics. The completeness proof for the epistemic
version of the logic, ELint , is rather different from the completeness proof for the full logic AP ALint . We then compare
our work to that of others (Section 5) and conclude.

tution). They will be used in the axiomatization (Section
3).
Definition 2. Let ϕ ∈ LAP ALint . The necessity forms
are inductively defined asξ(]) := ] | ϕ → ξ(]) | Ki ξ(]) | int(ξ(])) | [ϕ]ξ(]).

It is not hard to see that each necessity form ξ(]) has aunique occurrence of ]. Given a necessity form ξ(]) and aformula ϕ ∈ LAP ALint , the formula obtained by replacing ]by ϕ is denoted by ξ(ϕ).
In the completeness proof (Section 4) we use a complexity measure on formulas based on the size and 2-depth offormulas where the size of a formula is a weighted count of
the number of symbols and 2-depth counts the number ofthe 2-modalities occurring in a formula. The measure was
first introduced in [2].
Definition 3. The size S(ϕ) of a formula ϕ ∈ LAP ALintis defined as: S(p) = 1, S(¬ϕ) = S(ϕ) + 1, S(ϕ ∧ ψ) =S(ϕ) + S(ψ), S(Ki ϕ) = S(ϕ) + 1, S(int(ϕ)) = S(ϕ) + 1,
S([ϕ]ψ) = S(ϕ) + 4S(ψ), and S(2ϕ) = S(ϕ) + 1.
The factor 4 in the clause for [ϕ]ψ is to ensure Lemma 7.
Although the choice of the number 4 might seem arbitrary,
it is the smallest natural number guaranteeing the desired
result (see the proof of Lemma 7).
Definition 4. The 2-depth of a formula ϕ ∈ LAP ALint ,denoted by d(ϕ), is defined as: d(p) = 0, d(¬ϕ) = d(ϕ),
d(ϕ ∧ ψ) = max{d(ϕ), d(ψ)}, d(Ki ϕ) = d(ϕ), d(int(ϕ)) =d(ϕ), d([ϕ]ψ) = max{d(ϕ), d(ψ)}, and d(2ϕ) = d(ϕ) + 1.
We now define three order relations on LAP ALint based on
the size and 2-depth of the formulas.
Definition 5. For any ϕ, ψ ∈ LAP ALint ,• ϕ <S ψ iff S(ϕ) < S(ψ)• ϕ <d ψ iff d(ϕ) < d(ψ)• ϕ <Sd ψ iff (either d(ϕ) < d(ψ), or d(ϕ) = d(ψ) andS(ϕ) < S(ψ))

We let Sub(ϕ) denote the set of subformulas of a given formula ϕ.
Lemma 6. For any ϕ, ψ ∈ LAP ALint ,1. <S , <d , <S
d are well-founded strict partial orders between formulas in LAP ALint ,2. ϕ ∈ Sub(ψ) implies ϕ <Sd ψ ,3. int(ϕ) <S
d [ϕ]ψ,4. ϕ ∈ LP ALint iff d(ϕ) = 0,
5. ϕ ∈ LP ALint implies [ϕ]ψ <S
d 2ψ.

2. THE LOGIC AP ALint
We define the syntax, structures, and semantics of ourlogic. From now on, Prop is a countable set of propositionalvariables and A a finite and non-empty set of agents.

2.1

SyntaxDefinition 1. The language LAP ALint is defined by
ϕ ::= p | ¬ϕ | ϕ ∧ ϕ | Ki ϕ | int(ϕ) | [ϕ]ϕ | 2ϕ
where p ∈ Prop and i ∈ A. Abbreviations for the connectives ∨, → and ↔ are standard, and ⊥ is defined as abbreviation by p ∧ ¬p. We employ K̂i for ¬Ki ¬ϕ, and 3ϕ for¬2¬ϕ. We denote the non-modal part of LAP ALint (without
the modalities Ki , int, [ϕ] and 2) by LP l , the part without
2 by LP ALint , and the part without 2 and [ϕ] by LELint .
Necessity forms [10] allow us to select unique occurrences ofa subformula in a given formula (unlike in uniform substi-

Lemma 7. For any ϕ, ψ, χ ∈ LAP ALint and i ∈ A,
1. ¬[ϕ]ψ <S
d [ϕ]¬ψ,
2. int([ϕ]ψ) <S
d [ϕ]int(ψ),
3. Ki [ϕ]ψ <S
[ϕ]K
i ψ,
d
4. [¬[ϕ]¬int(ψ)]χ <S
d [ϕ][ψ]χ.
Proof. We only prove Lemma 7.4. The proof demonstrates why in the [ϕ]ψ clause of Definition 3, 4 is the smallest natural number guaranteeing the result.
By Definition 3, we have that S([¬[ϕ]¬int(ψ)]χ) = S(ϕ)+4S(ψ) + 4S(χ) + 9 and that S([ϕ][ψ]χ) = S(ϕ) + 4S(ψ) +16S(χ). As for any χ ∈ LAP ALint , 1 ≤ S(χ), it follows that
4S(χ)+9 ≤ 4S(χ)+9S(χ) = 13S(χ) < 16S(χ). Further, weobserve that d([¬[ϕ]¬int(ψ)]χ) = max{d(ϕ), d(ψ), d(χ)} =d([ϕ][ψ]χ). (This is similar in the first three items.)

96

2.2

Background2.4

In this section, we introduce the topological concepts thatwill be used throughout this paper. All the concepts in thissection can be found in [9].
Definition 8. A topological space (X, τ ) is a pair consisting of a non-empty set X and a family τ of subsets of X
satisfying ∅ ∈ τ and X ∈ τ , and closed under finite intersections and arbitrary unions.
The set X is called the space. The subsets of X belonging
to τ are called open sets (or opens) in the space; the family
τ of open subsets of X is also called a topology on X. If forsome x ∈ X and an open U ⊆ X we have x ∈ U , we saythat U is an open neighborhood of x.
A point x is called an interior point of a set A ⊆ X if
there is an open neighborhood U of x such that U ⊆ A. The
set of all interior points of A is called the interior of A anddenoted by Int(A). We can then easily observe that for any
A ⊆ X, Int(A) is the largest open subset of A.
Definition 9. A family B ⊆ τ is called a base for a
topological space (X, τ ) if every non-empty open subset of X
can be written as a union of elements of B.
Given any family Σ = {Aα | α ∈ I} of subsets of X,
there exists a unique, smallest topology τ (Σ) with Σ ⊆ τ (Σ)[9, Th. 3.1]. The family τ (Σ) consists of ∅, X, all finite
intersections of the Aα , and all arbitrary unions of thesefinite intersections. Σ is called a subbase for τ (Σ), and τ (Σ)
is said to be generated by Σ. The set of finite intersectionsof members of Σ forms a base for τ (Σ).

2.3

Structures

In this section we define our multi-agent models based on
topological spaces.
Definition 10. Given a topological space (X, τ ), a neighbourhood function set Φ on (X, τ ) is a set of partial functions θ : X * A → τ such that for all x, y ∈ Dom(θ), for
all i ∈ A, and for all U ∈ τ :1. θ(x)(i) ∈ τ ,2. x ∈ θ(x)(i),3. θ(x)(i) ⊆ Dom(θ),
4. if y ∈ θ(x)(i) then θ(x)(i) = θ(y)(i),5. θ|U ∈ Φ,
where θ|U is the partial function with Dom(θ|U ) = Dom(θ) ∩ U
and θ|U (x)(i) = θ(x)(i) ∩ U . We call the elements of Φneighbourhood functions.
Definition 11. A topological model with functions (orin short, a topo-model) is a tuple M = (X, τ, Φ, V ), where(X, τ ) is a topological space, Φ a neighbourhood function set,
and V : P rop → X a valuation function. We refer to thepart X = (X, τ, Φ) without the valuation function as a topoframe.
A pair (x, θ) is a neighbourhood situation if x ∈ Dom(θ)
and θ(x)(i) is called the epistemic neighbourhood at x ofagent i. If (x, θ) is a neighbourhood situation in M we write(x, θ) ∈ M. Similarly, if (x, θ) is a neighbourhood situationin X we write (x, θ) ∈ X .
Lemma 12. For any (X, τ, Φ) and θ ∈ Φ, Dom(θ) ∈ τ .
97

Semantics

Definition 13. Given a topo-model M = (X, τ, Φ, V ) anda neighbourhood situation (x, θ) ∈ M, the semantics for the
language LAP ALint is defined recursively as:
M, (x, θ) |= p
M, (x, θ) |= ¬ϕM, (x, θ) |= ϕ ∧ ψM, (x, θ) |= Ki ϕM, (x, θ) |= int(ϕ)
M, (x, θ) |= [ϕ]ψM, (x, θ) |= 2ϕx ∈ V (p)not M, (x, θ) |= ϕM, (x, θ) |= ϕ and M, (x, θ) |= ψ(∀y ∈ θ(x)(i))(M, (y, θ) |= ϕ)x ∈ Int[[ϕ]]θ
M, (x, θ) |= int(ϕ) ⇒M, (x, θϕ ) |= ψiff (∀ψ ∈ LP ALint )(M, (x, θ) |= [ψ]ϕ)

iff
iff
iff
iff
iff
iff

where p ∈ Prop, [[ϕ]]θ = {y ∈ Dom(θ) | M, (y, θ) |= ϕ}and θϕ : X * A → τ such that Dom(θϕ ) = Int[[ϕ]]θ andθϕ (x)(i) = θ(x)(i) ∩ Int[[ϕ]]θ .
The updated neighbourhood function θϕ is the restriction ofθ to the open set Int[[ϕ]]θ , i.e., for all x ∈ X, θϕ (x)(i) =θ|Int[[ϕ]]θ (x)(i).
A formula ϕ ∈ LAP ALint is valid in a topo-model M, denoted M |= ϕ, iff M, (x, θ) |= ϕ for all (x, θ) ∈ M; ϕ is valid,denoted |= ϕ, iff for all topo-models M we have M |= ϕ.
Soundness and completeness with respect to topo-models
are defined as usual.
Let us now elaborate on the structure of topo-models andthe above semantics we have proposed for LAP ALint . Givena topo-model (X, τ, Φ, V ), the epistemic neighbourhoods ofeach agent at a given state x are determined by (partial)
functions θ : X * A → τ assigning an open neighbourhood
to the state in question for each agent. We allow for partial
functions in Φ, and close Φ under taking restricted functions θ|U where U ∈ τ (see Definition 10, condition 5), sothat updated neighbourhood functions are guaranteed to bewell-defined elements of Φ. As in the standard subset spacesemantics, by picking a neighbourhood situation (x, θ), we
first localize our focus to an open subdomain, in fact toDom(θ), including the state x and the epistemic neighbourhood of each agent at x determined by θ. Then the function
θ(x) designates an epistemic neighbourhood for each agent
i in A. It is guaranteed that every agent i is assigned aneighbourhood by θ at every state x in Dom(θ), since eachθ(x) is defined to be a total function from A to τ . Moreover,condition 2 of Definition 10 ensures that ∅ cannot be an epistemic neighbourhood, i.e., θ(x)(i) 6= ∅ for all x ∈ Dom(θ).
Finally, conditions 2 and 4 of Definition 10 make sure that
the S5 axioms for each Ki are sound with respect to all
topo-models.
We now provide some semantic results. As usual in thesubset space setting, truth of non-modal formulas only depends on the state in question.
Proposition 14. Give a topo-model M = (X, τ, Φ, V ),neighbourhood situations (x, θ1 ), (x, θ2 ) ∈ M, and a formulaϕ ∈ LP l . Then (x, θ1 ) |= ϕ iff (x, θ2 ) |= ϕ.
Proposition 15. Given M = (X, τ, Φ, V ), θ ∈ Φ andϕ ∈ LAP ALint . Then [[int(ϕ)]]θ = Int[[ϕ]]θ .
Proof.

[[int(ϕ)]]θ

= {y ∈ Dom(θ) | (y, θ) |= int(ϕ)}= {y ∈ Dom(θ) | y ∈ Int[[ϕ]]θ }= Int[[ϕ]]θ (since Int[[ϕ]]θ ⊆ Dom(θ))

A corollary is that Int[[int(ϕ)]]θ = IntInt[[ϕ]]θ = Int[[ϕ]]θ .
Proposition 16.
1. |= [ϕ]ψ ↔ [int(ϕ)]ψ
2. |= (int(ϕ) ∧ hϕiint(ψ)) ↔ hϕiint(ψ)
Proposition 17.
ϕ
1. [[ψ]]θ = [[hϕiψ]]θ
ϕ
2. θ = θint(ϕ)
3. (θϕ )ψ = θhϕiint(ψ)

2.5

ExampleWe illustrate our logic by a multi-agent version of Bjorndahl’s convincing example in [6] about the jewel in the tomb.
Indiana Jones (i) and Emile Belloq (e) are both scouring
for a priceless jewel placed in a tomb. The tomb could either contain a jewel or not, the tomb could have been rediscovered in modern times or not, and (beyond [6]), thetomb could be in the Valley of Tombs in Egypt or not. The
propositional variables corresponding to these propositions
are, respectively, j, d, and t. We represent a valuation ofthese variables by a triple xyz, where x, y, z ∈ {0, 1}. Given
carrier set X = {xyz | x, y, z ∈ {0, 1}}, the topology τ that
we consider is generated by the base consisting of the subsets{000, 100, 001, 101}, {010}, {110}, {011}, {111}. The idea
is that one can only conceivably know (or learn) about thejewel or the location, on condition that the tomb has been
discovered. Therefore, {000, 100, 001, 101} has no strict subsets besides empty set: if the tomb has not yet been discovered, no one can have any information about the jewel or
the location.
A topo-model M = (X, τ, Φ, V ) for this topology (X, τ )has Φ as the set of all neighbourhood functions that are
partitions of X for both agents, and restrictions of thesefunctions to open sets. A typical θ ∈ Φ describes complete ignorance of both agents and is defined as θ(s)(i) =θ(s)(e) = X. This corresponds most to the situation described in [6]. A more interesting neighbourhood situationin this model is one wherein Indiana and Emile have different knowledge. Let us assume that Emile has the advantage over Indiana so far, as he knows the location of thetomb but Indiana doesn’t. This is the θ0 such that for allx ∈ X, θ0 (x)(i) = X whereas the partition for Emile consists of sets {101, 100, 001, 000}, {111, 011}, {110, 010}, i.e.,θ0 (111)(e) = {111, 011}, etc.
We now can evaluate what Emile knows about Indiana at
111, and confirm that this goes beyond Emil’s initial epistemic neighbourhood. This situation however does not create any problems in our setting since Indiana’s epistemicneighbourhoods will be determined relative to the states inEmile’s initial neighbourhood. Firstly, Emile knows that thetomb is in the Valley of Tombs in Egypt
M, (111, θ0 ) |= Ke t

the tomb’s location
M, (111, θ0 ) |= Ke K̂i ¬(Ke t ∨ Ke ¬t)
Announcements will change their knowledge in different ways.
Consider the announcement of j. This results in Emile knowing everything but Indiana still being uncertain about thelocation.
M, (111, θ0 ) |= [j](Ke (j ∧ d ∧ t) ∧ Ki (j ∧ d) ∧ ¬Ki (t ∨ Ki ¬t))Model checking this involves computing the epistemic neighbourhoods of both agents given by the updated neighbour0
hood function (θ0 )j at 111. Observe that Int[[j]]θ = {111, 110}.
0 jθ00Therefore, (θ ) (111)(e) = Int[[j]] ∩ θ (111)(e) = {111} and0(θ0 )j (111)(i) = Int[[j]]θ ∩ θ0 (x)(i) = {111, 110}.
There is an announcement after which Emile and Indiana
know everything (for example the announcement of j ∧ t):
M, (111, θ) |= 3(Ke (j ∧ d ∧ t) ∧ Ki (j ∧ d ∧ t))
As long as the tomb has not been discovered, nothing will
make Emile (or Indiana) learn that it contains a jewel or
where the tomb is located:
M |= ¬d → 2(¬(Ke j ∨ Ke ¬j) ∧ ¬(Ke t ∨ Ke ¬t))

3.

AXIOMATIZATION

We now provide the axiomatizations of ELint , P ALint ,and AP ALint , and prove their soundness and completenesswith respect to the proposed semantics.
(P) all instantiations of propositional tautologies(K-K) Ki (ϕ → ψ) → (Ki ϕ → Ki ψ)(K-T) Ki ϕ → ϕ(K-4) Ki ϕ → Ki Ki ϕ(K-5) ¬Ki ϕ → Ki ¬Ki ¬ϕ(int-K) int(ϕ → ψ) → (int(ϕ) → int(ψ))(int-T) int(ϕ) → ϕ(int-4) int(ϕ) → int(int(ϕ))(Kint ) Ki ϕ → int(ϕ)(R1) [ϕ]p ↔ (int(ϕ) → p)
(R2) [ϕ]¬ψ ↔ (int(ϕ) → ¬[ϕ]ψ)(R3) [ϕ](ψ ∧ χ) ↔ [ϕ]ψ ∧ [ϕ]χ(R4) [ϕ]int(ψ) ↔ (int(ϕ) → int([ϕ]ψ))(R5) [ϕ]Ki ψ ↔ (int(ϕ) → Ki [ϕ]ψ)(R6) [ϕ][ψ]χ ↔ [¬[ϕ]¬int(ψ)]χ
(R7) 2ϕ → [χ]ϕwhere χ ∈ LP ALint(DR1) From ϕ and ϕ → ψ, infer ψ(DR2) From ϕ, infer Ki ϕ(DR3) From ϕ, infer int(ϕ)(DR4) From ϕ, infer [ψ]ϕ(DR5) From ξ([ψ]χ) for all ψ ∈ LP ALint , infer ξ(2χ)
Table 1: Axiomatizations ELint , P ALint , and AP ALintand he also knows that Indiana does not know thatM, (111, θ0 ) |= Ke ¬(Ki ¬t ∨ Ki t)
The latter involves verifying M, (111, θ0 ) |= K̂i t andM, (111, θ0 ) |= K̂i ¬t. And this is true because θ0 (111)(i) =X, and 000, 001 ∈ X, and while M, (001, θ0 ) |= t, we also
have M, (000, θ0 ) |= ¬t. We can also check that Emile knowsthat Indiana considers it possible that Emile doesn’t know

Definition 18. The axiomatization AP ALint is given in
Table 1. The axiomatization P ALint is the one without(DR5) and (R7). We get ELint if we further remove axioms (R1)-(R6) and the rule (DR4).
The parts (DR1) to (DR5) are the derivation rules andthe other parts are the axioms. A formula is a theorem of98

AP ALint , notation ` ϕ, if it belongs to the smallest set of
formulas containing the axioms and closed under the derivation rules. (Similarly for ELint and P ALint .)Lemma 19. Axiomatization AP ALint satisfies substitution of equivalents. If ` ϕ ↔ ψ, then ` χ[p/ϕ] ↔ χ[p/ψ].
Proof. In the above, χ[p/ϕ] means uniform substitutionof ϕ for p. The proof is not trivial but proceeds along similarlines as for public announcement logic, see [17].
Proposition 20. [ϕ]⊥ ↔ ¬int(ϕ) is a theorem of AP ALint .
Proposition 21. AP ALint is sound with respect to theclass of all topo-models.
Proof. Let M = (X, τ, Φ, V ) be a topo-model, (x, θ) ∈M and ϕ, ψ, χ ∈ LAP ALint . We show three cases.
(Kint ) Suppose (x, θ) |= Ki ϕ. This means, (y, θ) |= ϕfor all y ∈ θ(x)(i). Hence, θ(x)(i) ⊆ [[ϕ]]θ . By Definition 10,
θ(x)(i) is an open neighbourhood of x, therefore we obtainx ∈ Int[[ϕ]]θ , i.e., (x, θ) |= int(ϕ).
(R7) Let χ ∈ LP ALint and suppose (x, θ) |= 2ϕ. By thesemantics, we have (x, θ) |= 2ϕ iff (∀ψ ∈ LP ALint )((x, θ) |=[ψ]ϕ). Therefore, in particular, (x, θ) |= [χ]ϕ.
(DR5) Suppose ξ([ψ]χ) is valid for all ψ ∈ LP ALint .
The proof follows by induction on the complexity of ξ(]).
In case ξ(]) = ], we have ξ([ψ]χ) = [ψ]χ. By assumption,
we have that [ψ]χ is valid for all ψ ∈ LP ALint . This implies
M, (x, θ) |= [ψ]χ for all ψ ∈ LP ALint , all topo-models M,
and (x, θ) ∈ M. Therefore, by the semantics, M, (x, θ) |=2χ, i.e., M, (x, θ) |= ξ(2χ). All other, inductive, cases are
elementary.
Corollary 22. The axiomatizations ELint and P ALintare sound with respect to the class of all topo-models.

4. COMPLETENESS
We now show completeness for ELint , P ALint , and AP ALint
with respect to the class of all topo-models. Completeness ofELint is shown in a standard way via a canonical model construction and a Truth Lemma that is proved by induction on
formula complexity. Completeness for P ALint is shown by
reducing each formula in LP ALint to an equivalent formula of
LELint . The proof of the completeness for AP ALint becomes
more involved. Reduction axioms for public announcements
no longer suffice in the AP ALint case, and the inductive
proof needs a subinduction where announcements are considered. Moreover, the proof system of AP ALint has an
infinitary derivation rule, namely the rule (DR5), and given
the requirement of closure under this rule, the maximallyconsistent sets for that case are defined to be maximally consistent theories (see, Section 4.2). Lastly, the Truth Lemma
requires the more complicated complexity measure on formulas defined in Section 2. There, we need to adapt thecompleteness proof of [2] to our setting.

4.1

Completeness of ELint and P ALintFor LELint we define consistent and maximally consistentsets in the usual way, see e.g. [6] for details, and the multiagent aspect does not complicate the definition. Let X c be
the set of all maximally consistent sets of ELint . We define relations ∼i on X c as x ∼i y iff ∀ϕ ∈ LELint (Ki ϕ ∈x iff Ki ϕ ∈ y). Notice that the latter is equivalent to:99∀ϕ ∈ LELint (Ki ϕ ∈ x implies ϕ ∈ y) since Ki is an S5
modality. As each Ki is of S5 type, every ∼i is an equivalence relation, hence, it induces equivalence classes on X c .
Let [x]i denote the equivalence class of x induced by the
relation ∼i . Moreover, we define ϕb = {y ∈ X c | ϕ ∈ y}.
Observe that x ∈ ϕb iff ϕ ∈ x.
Lemma 23 (Lindenbaum’s Lemma). Each consistent
set can be extended to a maximally consistent set.
Definition 24. We define the canonical model
X c = (X c , τ c , Φc , V c ) as follows:• X c is the set of all maximally consistent sets;• τ c is the topological space generated by the subbase\ | x ∈ X c , ϕ ∈ LEL and i ∈ A};
Σ = {[x]i ∩ int(ϕ)
int• x ∈ V c (p) iff p ∈ x, for all p ∈ Prop;• Φc = {θ∗ |U | U ∈ τ c }, where we define θ∗ : X c →A → τ c as θ∗ (x)(i) = [x]i , for x ∈ X c and i ∈ A.
\ = X c , we have [x]i ∩ int(>)\ =Observe that, since int(>)[x]i ∈ Σ for each i. Therefore, each [x]i is an open subsetof X c . Moreover, the elements of Φc satisfy the required
properties given in Definition 10.
Lemma 25 (Truth Lemma). For every ϕ ∈ LELint andfor each x ∈ X c , ϕ ∈ x iff X c , (x, θ∗ ) |= ϕ.
Proof. Cases for the propositional variables and Booleans
are straightforward. We only show the cases for Ki and int.
Case ϕ := Ki ψ(⇒)
Suppose Ki ψ ∈ x and let y ∈ θ∗ (x)(i). Since∗y ∈ θ (x)(i) = [x]i , by definition of ∼i , we have Ki ψ ∈ y.
Then, by T-axiom for Ki , we obtain ψ ∈ y. Then, by IH,
X c , (y, θ∗ ) |= ψ. Therefore X c , (x, θ∗ ) |= Ki ψ.
(⇐) Suppose Ki ψ 6∈ x. Then, {Ki γ | Ki γ ∈ x} ∪ {¬ψ}is a consistent set. We can then extend it to a maximallyconsistent set y. As {Ki γ | Ki γ ∈ x} ⊆ y, we have y ∈[x]i meaning that y ∈ θ∗ (x)(i). Moreover, since ¬ψ ∈ y,
ψ 6∈ y. Therefore, we have a maximally consistent set y ∈θ∗ (x)(i) such that ψ 6∈ y. By (IH), X c , (y, θ∗ ) 6|= ψ. Hence,
X c , (x, θ∗ ) 6|= Ki ψ.
Case ϕ := int(ψ)\(⇒) Suppose int(ψ) ∈ x. Consider the set [x]i ∩ int(ψ)\\
for some i ∈ A. Obviously, x ∈ [x]i ∩ int(ψ) and [x]i ∩ int(ψ)\ Sinceis open (since it is in Σ). Now let y ∈ [x]i ∩ int(ψ).
\ int(ψ) ∈ y. Then, by (int -T), since y is maximaly ∈ int(ψ),
consistent, we have ψ ∈ y. Thus, by IH, we have (y, θ∗ ) |= ψ.
∗\ ⊆ [[ψ]]θ∗ .
Therefore, y ∈ [[ψ]]θ . This implies [x]i ∩ int(ψ)\ ∈ τ c , we have x ∈ Int[[ψ]]θ∗ , i.e.,And, since x ∈ [x]i ∩ int(ψ)∗(x, θ ) |= int(ψ).
∗(⇐) Suppose (x, θ∗ ) |= int(ψ), i.e., x ∈ Int[[ψ]]θ . Recall
that the set of finite intersections of the elements of∗Σ forms
a base, which we denote by BΣ , for τ c . x ∈ Int[[ψ]]θ implies∗
that there exists an open U ∈ BΣ such that x ∈ U ⊆ [[ψ]]θ .
Given the construction of BΣ , U is of the form
U=

\

i∈I1

[x1 ]i ∩ . . .

\

i∈In

[xk ]i ∩\\int(η)

η∈Formfin

where I1 , . . . , In are finite subsets of A, x1 . . . xk ∈ X c andFormfin is a finite subset of LELint . Since int is a normal

modality, we can simply write\\\[x1 ]i ∩ . . .
[xk ]i ∩ int(γ),
U=
V

where

i∈In

i∈I1

η∈Formfin

η := γ. Since x is in each [xj ]i with 1 ≤ j ≤k, we have [xj ]i = [x]i for all such j. Therefore, we have\\ ⊆ [[ψ]]θ∗ ,x ∈ U = ( [x]i ) ∩ int(γ)
i∈Iwhere I = I1 ∪ · · · ∪ In .
T
\ then ψ ∈ y.
This implies, for all y ∈ ( [x]i ), if y ∈ int(γ)
S i∈I
From this, we can say{Ki σ | Ki σ ∈ x} ` int(γ) → ψ.
i∈ISThen, there is a finite subset Γ ⊆{Ki σ | Ki σ ∈ x} suchi∈I
V
that `λ → (int(γ) → ψ). It then follows:λ∈Γ

1. ` int(V

λ → (int(γ) → ψ))

(DR3)

λ∈Γ
V
2. ` int(λ) → int(int(γ) → ψ))(int-K) and (DR1)V λ∈Γ
3. ` (
int(λ)) → int(int(γ) → ψ)) (int-K)λ∈Γ

ObserveSthat each λ ∈ Γ is of the form Kj α for some
Kj α ∈{Ki σ | Ki σ ∈ x} and we have ` Ki ϕ ↔ int(Ki ϕ).
i∈IVTherefore, ` (λ) → int(int(γ) → ψ)). Thus, sinceλ∈ΓVλ ∈ x (by Γ ⊆ x), we have int(int(γ) → ψ)) ∈ x. Then,λ∈Γ

by (int-K), (DR1) and since ` int(int(γ)) ↔ int(γ) and\ (i.e., int(γ) ∈ x) , we obtain int(ψ) ∈ x.
x ∈ int(γ)
Theorem 26. ELint is complete with respect to the classof all topo-models.
Theorem 27. P ALint is complete with respect to the classof all topo-models.
Proof. This follows from Theorem 26 by reduction in astandard way. The occurrences of the modality int on theright-hand-side of the reduction axioms (axioms (R1)-(R6))
should not lead to any confusion: extending the complexity
measure defined in [17, Definition 7.21 p. 187] to the language LP ALint by adding the same complexity measure forthe modality int as for Ki gives us the desired result.

4.2

Completeness of AP ALint

We now reuse the technique of [2] in the setting of topological semantics. Given the closure requirement under derivation rule (DR5) it seems more proper to call maximally consistent sets of AP ALint maximally consistent theories, as
further explained below.
Definition 28. A set x of formulas is called a theory
iff AP ALint ⊆ x and x is closed under (DR1) and (DR5).
A theory x is said to be consistent iff ⊥ 6∈ x. A theory x
is maximally consistent iff x is consistent and any set of
formulas properly containing x is inconsistent.
Observe that AP ALint constitutes the smallest theory. Moreover, maximally consistent theories of AP ALint posses theusual properties of maximally consistent sets:
Proposition 29. For any maximally consistent theory x,
ϕ 6∈ x iff ¬ϕ ∈ x, and ϕ ∧ ψ ∈ x iff ϕ ∈ x and ψ ∈ x.

In the setting of our axiomatization based on the infinitaryrule (DR5), we will say that a set x of formulas is consistent
iff there exists a consistent theory y such that x ⊆ y. Obviously, maximal consistent theories are maximal consistentsets of formulas. Under the given definition of consistency
for sets of formulas, maximal consistent sets of formulas arealso maximal consistent theories.
Definition 30. Let ϕ ∈ LAP ALint and i ∈ A. Then x +ϕ := {ψ | ϕ → ψ ∈ x} and Ki x := {ϕ | Ki ϕ ∈ x}.
Lemma 31. For any theory x of AP ALint andϕ ∈ LAP ALint , x + ϕ is a theory and it contains x and ϕ,
and Ki x is a theory.
Lemma 32. Let ϕ ∈ LAP ALint . For all theories x, x + ϕ
is consistent iff ¬ϕ 6∈ x.
Proof. Let ϕ ∈ LAP ALint and x be a theory. Then ¬ϕ ∈x iff ϕ → ⊥ ∈ x (as ¬ϕ ↔ ϕ → ⊥ is a theorem) iff ⊥ ∈ x+ϕ.
Therefore, x + ϕ is inconsistent iff ¬ϕ ∈ x, i.e., x + ϕ isconsistent iff ¬ϕ 6∈ x.
Lemma 33 (Lindenbaum’s Lemma [1]). Each consistent theory can be extended to a maximal consistent theory.
Lemma 34. If Ki ϕ 6∈ x, then there is a maximally consistent theory y such that Ki x ⊆ y and ϕ 6∈ y.
Proof. Let ϕ ∈ LAP ALint and x be such that Ki ϕ 6∈
x. Thus, ϕ 6∈ Ki x. Hence, by Lemma 32, Ki x + ¬ϕ isconsistent. Then, by Lemma 33, there exists a maximallyconsistent set y such that Ki x + ¬ϕ ⊆ y. Therefore Ki x ⊆ yand ϕ 6∈ y.
Lemma 35. For all ϕ ∈ LAP ALint and all maximally consistent theories x, 2ϕ ∈ x iff for all ψ ∈ LP ALint , [ψ]ϕ ∈ x.
Proof. Let ϕ ∈ LAP ALint and x be a maximally consistent theory.
(⇒) Suppose 2ϕ ∈ x. Then, by (R7) and (DR1), we
have [ψ]ϕ ∈ x for all ψ ∈ LP ALint .
(⇐) Suppose [ψ]ϕ ∈ x for all ψ ∈ LP ALint . Consider thenecessity form ]. By assumption, ]([ψ]ϕ) for all ψ ∈ LP ALint .
Then, since x is closed under (DR5), ](2ϕ) ∈ x, i.e., 2ϕ ∈ x
as well.
The definition of the canonical model for AP ALint is thesame as for ELint , except that the maximally consistentsets are maximally consistent theories. We now come to
the Truth Lemma for the logic AP ALint . Here we use thecomplexity measure ψ <S
d ϕ.
Lemma 36 (Truth Lemma). For every ϕ ∈ LAP ALint
and for each x ∈ X c , ϕ ∈ x iff X c , (x, θ∗ ) |= ϕ.

Proof. Let ϕ ∈ LAP ALint and x ∈ X c . The proof is by<S
d -induction on ϕ, where the case ϕ = [ψ]χ is proved by asubinduction on χ. We therefore consider 14 cases.
Case ϕ := px ∈ p iff x ∈ ν c (p)iff (x, θ∗ ) |= p
Induction Hypothesis (IH): For all formulas ψ ∈ LAP ALint ,c∗if ψ <S
d ϕ, then ψ ∈ x iff X , (x, θ ) |= ψ.
The cases negation, conjunction, and interior modality are
as in Truth Lemma 25 for ELint , where we observe that the100subformula order is subsumed in the <S
d order (see Lemma
6.2). We proceed with the knowledge operator, i.e., case
ϕ := Ki ψ, and then with the subinduction on χ for caseannouncement ϕ := [ψ]χ, and finally with the case ϕ := 2ψ.
Case ϕ := Ki ψThis case is also similar to the one in Truth Lemma 25 for
ELint , however, using maximally consistent theories in the
canonical model creates some differences. For the direction
from left-to-right, see Truth Lemma 25. For (⇐), suppose
Ki ψ 6∈ x. Then, by Lemma 34, there exists a maximallyconsistent theory y such that Ki x ⊆ y and ψ 6∈ y. By ψ <SdKi ψ and (IH), (y, θ∗ ) 6|= ψ. Since Ki x ⊆ y, we have y ∈ [x]imeaning that y ∈ θ∗ (x)(i). Therefore, by the semantics,
X c , (x, θ∗ ) 6|= Ki ψ.
Case ϕ := [ψ]p
[ψ]p ∈ x iff int(ψ) → p ∈ x
iff int(ψ) ∈6 x or p ∈ xiff (x, θ∗ ) 6|= int(ψ) or (x, θ∗ ) |= piff (x, θ∗ ) |= [ψ]p

(R1)
Prop. 29
(∗)
(R1)

S
(*): By (IH), int(ψ) <S
d [ψ]p and p <d [ψ]p (Lemma 6.3 andLemma 6.2).
Case ϕ := [ψ]¬η Use (R2) and (IH) and, by Lemma 6.3
S
and Lemma 7.1, int(ψ) <S
d [ψ]¬η and ¬[ψ]η <d [ψ]¬η.
Case ϕ := [ψ](η ∧ σ) Use (R3) and (IH), [ψ]η <S
d [ψ](η ∧σ) and [ψ]σ <S
d [ψ](η ∧ σ).
Case ϕ := [ψ]int(η) Use (R4) and (IH) and, by Lemmas
S
6.3, 7.2, int(ψ) <S
d [ψ]int(η) and int([ψ]η) <d [ψ]int(η).
Case ϕ := [ψ]Ki η Use (R5) and (IH) and, by Lemmas
S
6.3, 7.3, int(ψ) <S
d [ψ]Ki η and Ki [ψ]η <d [ψ]Ki η.
Case ϕ := [ψ][η]σ Use (R6) and (IH) and, by Lemma
7.4, [¬[ψ]¬int(η)]σ <S
d [ψ][η]σ.
Case ϕ := [ψ]2σ For all η ∈ LP ALint , [ψ][η]σ <S
d [ψ]2σ,as [ψ]2σ has one more 2 than [ψ][η]σ . Therefore, it suffices
to show [ψ]2σ ∈ x iff ∀η ∈ LP ALint , [ψ][η]σ ∈ x.
(⇐) Consider the necessity form [ψ]] and assume thatfor all η ∈ LP ALint , [ψ][η]σ ∈ x, i.e., for all η ∈ LP ALint ,[ψ]]([η]σ) ∈ x . As x is closed under (DR5), we obtain[ψ]](2σ) ∈ x, i.e., [ψ]2σ ∈ x.
(⇒) Suppose [ψ]2σ ∈ x. We have` 2σ → [η]σ, for all η ∈ LP ALint` [ψ](2σ → [η]σ) for all η ∈ LP ALint` [ψ]2σ → [ψ][η]σ, for all η ∈ LP ALint

(R7)
(DR4)
(DR1), (R1-R3)

Therefore, for all η ∈ LP ALint , [ψ][η]σ ∈ x. As [ψ][η]σ <Sd[ψ]2σ for all η ∈ LP ALint , by (IH), we have for all η ∈LP ALint , (x, θ∗ ) |= [ψ][η]σ. Then, by the semantics, we obtain (details omitted) that (x, θ∗ ) |= [ψ]2σ.
Case ϕ := 2ψ Again note that for all η ∈ LP ALint ,[η]ψ <Sd 2ψ, as 2ψ has one more 2 than [η]ψ (see Lemma
6.4 and Lemma 6.5). Therefore, we obtain2ψ ∈ x iff
iffiff(∀η ∈ LP ALint )([η]ψ ∈ x)(∀η ∈ LP ALint )(x, θ∗ ) |= [η]ψ(x, θ∗ ) |= 2ψ

Lemma 35
(IH)
semantics

Theorem 37. AP ALint is complete with respect to theclass of all topo-models.
Proof. Let ϕ ∈ LAP ALint such that 6` ϕ, i.e., ϕ 6∈ AP ALint(Recall that AP ALint is the smallest theory). Then, by
Lemma 32, AP ALint + ¬ϕ is a consistent theory and, byLemma 31, ¬ϕ ∈ AP ALint + ¬ϕ. By Lemma 33, the consistent theory AP ALint + ¬ϕ can be extended to a maximally101

consistent theory y such that AP ALint + ¬ϕ ⊆ y. Since
y is maximally consistent and ¬ϕ ∈ y, we obtain ϕ 6∈ y(by Proposition 29). Then, by Lemma 36 (Truth Lemma),
X c , (y, θ∗ ) 6|= ϕ.

5.

COMPARISON TO OTHER WORK

Multi-agent epistemic systems with subset space-like semantics have been proposed in [11, 12, 4, 18], however, noneof these are concerned with arbitrary announcements. Our
goal in this paper is not to provide a multi-agent generalization of SSL per se, but to work with the effort-like modality 2 intended to capture the information change brought
about by any announcements (subject to some restrictions)in a multi-agent setting and modelling it by way of “openset shrinking” similar to the effort modality, rather than bydeleting states or neighbourhoods, so that the intuitive link
between the two becomes more transparent on a semantic
level. In [3], Balbiani et al. proposed subset space semantics for arbitrary announcements, however, their approach
does not go beyond the single-agent case and the semantics
provided is in terms of model restriction. An unorthodox
approach to multi-agent knowledge is proposed in [11, 12].
Roughly speaking, instead of having a knowledge modality
Ki for each agent in his syntax, Heinemann uses additional
operators to define Ki and his semantics only validate theS4-axioms for Ki . The necessitation rule for Ki does notpreserve validity under the proposed semantics [11, 12]. In[18] a multi-agent semantics for knowledge is provided, but
no announcements or further generalizations (unlike in theirother, single-agent, work [19]), and not in a topological setting. Their use of partitions for each agent instead of asingle neighbourhood is compatible with our requirementthat all neighbourhoods for a given agent be disjoint. A further difference from the existing literature is that we restrict
our attention to topological spaces and prove our results bymeans of topological tools.
We applied the new completeness proof for arbitrary public announcement logic of [2] to a topological setting. The
canonical modal construction is as in [6] with some multiagent modifications. The modality int in our system demands a different complexity measure in the Truth Lemmaof the completeness proof than in [2].

6.

CONCLUSIONS

We have proposed topological semantics for the multiagent extensions of the public announcement logic of [6], andfurther extended the logic with arbitrary announcements.
We showed topological completeness of these logics. Our
work can be seen as a step toward discovering the interplay
between dynamic epistemic logic and topological reasoning.
For further research, we envisage a finitary axiomatization for AP ALint wherein the infinitary derivation rule(DR5) is replaced by a finitary rule. The obvious derivation
rule would derive something after any announcement if it
can be derived after announcing a fresh variable [1]. Under
subset space semantics, it is unclear how to prove that thisrule is sound.
We are still investigating expressivity and (un)decidability.
If the logic AP ALint is undecidable, this would contrast
nicely with the undecidability of arbitrary public announcement logic. Otherwise, there may be interesting decidable
versions when restricting the class of models to particulartopologies.
The logic AP ALint is also axiomatizable on the class where
the K modalities have S4 properties, a result we have notreported in this paper for consistency of presentation. This
class is of topological interest.
In our setup all agents have the same observational powers. If agents can have different observational powers, we
can associate a topology with each agent and generalize thelogic to an arbitrary epistemic action logic.
Furthermore, we would like to explore the exact difference
between the effort modality and the arbitrary announcement
modality (in the single agent case, see [16]) by constructing a
topological model which distinguishes the two: a topologicalmodel might have more than epistemically definable openswith respect to the proposed semantics.

Acknowledgements
We thank Philippe Balbiani for various detailed suggestions
over the past year on how to improve our single-agent andmulti-agent results in subset space logic and topological logics. We have found him very supportive of our efforts. We
also thank the TARK reviewers for their valuable comments.
Hans van Ditmarsch is also affiliated to IMSc (Institute ofMathematical Sciences), Chennai, as research associate. We
acknowledge support from European Research Council grant
EPS 313360.

[13] Moss, L. S. and Parikh, R. (1992) Topologicalreasoning and the logic of knowledge. Proc. of the 4th
TARK , pp. 95–105, Morgan Kaufmann.
[14] Parikh, R., Moss, L., and Steinsvold, C. (2007)
Topology and epistemic logic. Handbook of Spatial
Logics, pp. 299–341.
[15] Plaza, J. A. (1989) Logics of public communications.
Proc. of the 4th ISMIS , pp. 201–216, Oak Ridge
National Laboratory.
[16] van Ditmarsch, H., Knight, S., and Özgün, A. (2014)
Arbitrary announcements on topological subset spaces.
Proc. of the 12th EUMAS , pp. 252–266, Springer.
[17] van Ditmarsch, H., van der Hoek, W., and Kooi, B.
(2007) Dynamic Epistemic Logic, vol. 337 of Synthese
Library. Springer.
[18] Wang, Y. N. and Ågotnes, T. (2013) Multi-agent
subset space logic. Proc. of the 23rd IJCAI , pp.
1155–1161, IJCAI/AAAI.
[19] Wáng, Y. N. and Ågotnes, T. (2013) Subset space
public announcement logic. Proc. of 5th ICLA, pp.
245–257, Springer.

7. REFERENCES[1] Balbiani, P., Baltag, A., van Ditmarsch, H., Herzig,
A., Hoshi, T., and de Lima, T. (2008) ‘Knowable’ as‘known after an announcement’. The Review ofSymbolic Logic, 1, 305–334.
[2] Balbiani, P. and van Ditmarsch, H. (2015) A simple
proof of the completeness of APAL. Studies in Logic,
8 (1), 65–78.
[3] Balbiani, P., van Ditmarsch, H., and Kudinov, A.
(2013) Subset space logic with arbitrary
announcements. Proc. of the 5th ICLA, pp. 233–244,
Springer.
[4] Baskent, C. (2007) Topics in Subset Space Logic.
Master’s thesis, University of Amsterdam.
[5] Baskent, C. (2012) Public announcement logic ingeometric frameworks. Fundam. Inform., 118,
207–223.
[6] Bjorndahl, A. (2013) Subset space publicannouncement logic revisited. CoRR, abs/1302.4009.
[7] Chellas, B. F. (1980) Modal logic. Cambridge
University Press, Cambridge.
[8] Dabrowski, A., Moss, L. S., and Parikh, R. (1996)
Topological reasoning and the logic of knowledge.
Ann. Pure Appl. Logic, 78, 73–110.
[9] Dugundji, J. (1965) Topology. Allyn and Bacon Series
in Advanced Mathematics, Prentice Hall.
[10] Goldblatt, R. (1982) Axiomatising the Logic of
Computer Programming. Springer-Verlag.
[11] Heinemann, B. (2008) Topology and knowledge ofmultiple agents. Proc. of the 11th IBERAMIA, pp.
1–10, Springer.
[12] Heinemann, B. (2010) Logics for multi-subset spaces.
Journal of Applied Non-Classical Logics, 20, 219–240.

102

Single-Peaked Consistency for Weak Orders Is Easy
Zack Fitzsimmons

College of Computing and Information Sciences
Rochester Institute of Technology
Rochester, NY 14623, USA

zmf6921@rit.edu

ABSTRACTIn economics and social choice single-peakedness is one of
the most important and commonly studied models for preferences. It is well known that single-peaked consistency
for total orders is in P. However in practice a preference
profile is not always comprised of total orders. Often voters have indifference between some of the candidates. In a
weak preference order indifference must be transitive. We
show that single-peaked consistency for weak orders is in
P for three different variants of single-peakedness for weakorders. Specifically, we consider Black’s original definitionof single-peakedness for weak orders, Black’s definition of
single-plateaued preferences, and the existential model recently introduced by Lackner. We accomplish our results bytransforming each of these single-peaked consistency problems to the problem of determining if a 0-1 matrix has theconsecutive ones property.

Categories and Subject Descriptors
F.2.2 [Theory of Computation]: Analysis of Algorithms
and Problem Complexity—Nonnumerical Algorithms andProblems; I.2.11 [Artificial Intelligence]: Distributed Artificial Intelligence—Multiagent systems

General Terms
Algorithms, Economics, Theory

Keywords
computational social choice, partial preferences, domain restrictions

1. INTRODUCTION
Single-peakedness is one of the most important and commonly examined domain restrictions on preferences in economics and social choice. The study of single-peaked preferences in computational social choice is often restricted tototal orders, but in practical settings voters often have somedegree of indifference in their preferences. This is seen in the
online repository PrefLib, which contains several datasetscomprised of voters with various degrees of partial preferences, many of which are weak orders [26]. Additionally,some election systems are defined for weak orders, e.g., theKemeny rule [22] and the Schulze rule [30], or can be easily
extended for weak orders.
Single-peaked preferences were introduced by Black [5]and they model the preferences of a collection of voters with103

respect to a one-dimensional axis, i.e., a total ordering of
the candidates. Each voter in a single-peaked electorate hasa single most preferred candidate (peak) on the axis and thefarther that a candidate is from the voter’s peak the less preferred they are by the voter. Black extended his model tosingle-plateaued preferences, which models the preferencesof a collection of voters in a similar way, but allows votersto have multiple most preferred candidates (an indifference
plateau) in their preferences [6, Chapter 5]. We mention
that the definition of single-peaked preferences from Fishburn [18, Chapter 9] for weak orders is the same as Black’s
definition of single-plateaued preferences.
Elections where the voters have single-peaked preferences
over the candidates have many desirable properties in economics and social choice, e.g., the majority relation is transitive [5] and there exist strategy-proof voting rules [28]. Additionally, computational problems often become easier whenpreferences are single-peaked. For example, when voters inan election have single-peaked (or even nearly single-peaked)
preferences the complexity of determining if a manipulative
action exists often becomes easy [17, 16] and determining thewinner for Dodgson and Kemeny elections becomes easy [8]when it is Θp2 -complete in general [20, 21].
The problem of single-peaked consistency is to determine
if an axis exists such that the preferences of a collection of
voters are single-peaked. The first paper to computationallystudy single-peaked consistency for partial preferences was
Lackner [24], where a partial order is said to be single-peakedwith respect to an axis if it can be extended to a total orderthat is single-peaked with respect to that axis. For clarity
we refer to this as existentially single-peaked, or ∃-singlepeaked, throughout this paper. Lackner presents algorithms
and complexity results for determining the ∃-single-peakedconsistency for preference profiles of varying degrees of partial preferences, including top orders, weak orders, local
weak orders, and partial orders. Lackner shows that if agiven preference profile contains an implicitly specified totalorder (which is not guaranteed to exist) then ∃-single-peakedconsistency for weak orders is in P [24]. Lackner also showsthat the general case of ∃-single-peaked consistency for top
orders (weak orders with all indifference between last-ranked
candidates) is in P [24]. The complexity of the general caseof ∃-single-peaked consistency for weak orders was explicitlyleft as the main open problem in Lackner [24] and we show
in this paper that it is in P.
We show that an algorithm to determine if a 0-1 matrix has the consecutive ones property can be used to determine the single-peaked, single-plateaued, and ∃-single-

peaked consistency for weak orders without requiring an implicitly specified total order. So given a preference profile of
weak orders not only can we determine if it is single-peaked,
single-plateaued, or ∃-single-peaked, we can find all consistent axes by using the PQ-tree algorithm for determining if
a 0-1 matrix has the consecutive ones property [7]. This algorithm was previously used to determine the single-peakedconsistency for total orders by Bartholdi and Trick [4] and to
determine the single-crossing consistency for total orders by
Bredereck et al. [9]. The model of single-crossing preferences
is another domain restriction [27] and its corresponding consistency problem for total orders was first shown to be in P
by Elkind et al. [12]. We also mention that after singlepeaked consistency for total orders was shown to be in P,
both Escoffier et al. [15] and Doignon and Falmagne [10]independently found faster direct algorithms.
This paper is organized as follows. In Section 2 we define
the types of partial preferences studied, the different variantsof single-peakedness, and the consecutive ones matrix problem. We present our results in Section 3, which is split into
three sections with each corresponding to a different variantof single-peakedness. Section 3.1 contains our results for∃-single-peaked preferences, Section 3.2 for single-plateaued
preferences, and Section 3.3 for single-peaked preferences.
In each of these sections we redefine the variant of singlepeakedness using forbidden substructures and describe thetransformation from its consistency problem to the problem
of determining if a 0-1 matrix has the consecutive ones property. We conclude in Section 4 by summarizing our results
and stating some possible directions for future work.

2. PRELIMINARIESA preference order, v, is an ordering of the elements of afinite candidate set, C. A multiset of preference orders, V ,is called a preference profile. (We sometimes refer to each v
as a voter with a corresponding preference order.) A partial
order is a transitive, reflexive, and antisymmetric binary
relation on a set. A weak order is a partial order where theindifference relation is transitive, a top order is a weak orderwhere all indifference is between candidates ranked last, anda total order is a partial order with no indifference between
candidates.
Example 1. Given the set of candidates {a, b, c, d}, an
example of a total order is (a > b > d > c), an example
of a weak order is (a ∼ c > d > b), and an example of a
top order is (a > c > b ∼ d), where “∼” is used to denoteindifference between candidates.
We focus on weak orders since they model easily understood incompleteness in preferences. Voters are often notable to discern between two candidates or view them as truly
equal. Allowing each voter to state a weak preference orderstill requires that they specify each candidate in their order,
but gives them the ability to have multiple candidates at
each position.
It is very natural for election systems to be defined forweak orders. The Kemeny rule and Schulze rule are defined for weak orders [22, 30] and clearly election systems
based on pairwise comparisons (e.g., Copeland) can be used
to evaluate a preference profile of partial votes. The Borda
count can be extended for top orders [13] and a recent paper has even explored the complexity of the manipulation
problem on such extensions to the Borda count and defined

additional extensions for election systems to be defined on
top orders [29].

2.1

Variants of Single-Peakedness

In our definitions of each variant of single-peakedness werefer to a total ordering of the set of candidates that eachpreference profile is consistent with as an axis A. Like
Bartholdi and Trick [4], who were the first to show singlepeaked consistency for total orders in P, we say that a preference order v is strictly increasing (decreasing) along a segment X of A if each candidate in X is strictly preferred toeach candidate to its left (right) in X. Similarly, we say thata preference order is increasing (decreasing) along a segment
X of A if each candidate in X is strictly preferred or ranked
indifferent to each candidate to its left (right) in X. When
we say that a preference order v is remaining constant along
a segment, then all candidates in that segment are ranked
indifferent to each other.
We begin our discussion of single-peaked preferences bystating the definition of single-peaked preferences for totalorders. We use the definition found in the work by Bartholdi
and Trick [4].
Definition 2. A preference profile V of total orders issingle-peaked with respect to an axis A if for every v ∈ V ,A can be split at the most preferred candidate (peak) of v
into two segments X and Y (one of which can be empty)
such that v has strictly increasing preferences along X andv has strictly decreasing preferences along Y .
We now define each of the three variants of single-peaked
preferences for weak orders that we study in this paper andpresent an example of each in Figure 1.

2.1.1

Single-Peaked Preferences

Single-peaked preferences for weak orders can be definedin the same way as single-peaked preferences for total orders.
Definition 3. A preference profile V of weak orders issingle-peaked with respect to an axis A if for every v ∈ V ,A can be split at the most preferred candidate (peak) of v
into two segments X and Y (one of which can be empty)
such that v has strictly increasing preferences along X andv has strictly decreasing preferences along Y .
Notice that for a weak preference order to be single-peaked
it must have a single most preferred candidate and can only
contain indifference between at most two candidates at eachposition. Otherwise the segments X and Y referred to in
Definition 3 would not be strictly increasing/decreasing. We
define the corresponding problem of single-peaked consistency for weak orders below.
Given: A preference profile V of weak orders and a set ofcandidates C.
Question: Does there exist an axis A such that V is singlepeaked with respect to A?

2.1.2 Single-Plateaued Preferences
A slightly weaker restriction than single-peakedness forweak orders is single-plateauedness [6, Chapter 5]. Singlepeaked and single-plateaued preferences are closely related
domain restrictions and Barberà [2] discusses how theamounts of indifference permitted in these restrictions impact their properties.

104

Building upon the definition for single-peaked preferences,
we state a definition for single-plateaued preferences.
Definition 4. A preference profile V of weak orders issingle-plateaued with respect to an axis A if for every v ∈ V ,A can be split into three segments X, Y , and Z (X and Z
can each be empty) where v’s most preferred candidates are
Y , v has strictly increasing preferences along X, and v has
strictly decreasing preferences along Z.
We define the corresponding problem of single-plateaued
consistency for weak orders below.
Given: A preference profile V of weak orders and a set ofcandidates C.
Question: Does there exist an axis A such that V is singleplateaued with respect to A?

2.1.3

Existentially Single-Peaked Preferences

So far we have considered the given preference orders as
the true preferences of the voters. One approach to dealing
with partial preferences is to assume that voters have an
underlying total preference order and consider extensionsof their preferences to total orders (see, e.g., [23]). This is
the approach taken by Lackner for the existential model ofsingle-peakedness [24].

a

d

b

e

c

Figure 1: The solid line represents the single-peaked
preference order (b > d ∼ e > c > a), the dottedline represents the single-plateaued preference order(a ∼ d > b > e > c), and the dashed line represents the∃-single-peaked preference order (c > b ∼ e > d ∼ a).
First we show that there exists an ∃-single-peaked consistent preference profile that does not have a transitive majority relation. We say that a majority relation is transitive if when x > y and y > z by majority, then x > z
by majority. Note that single-peaked and single-plateaued
preferences both have transitive majority relations [5, 6].
Consider the preference profile V comprised of the following five voters from Table 9.1 in Fishburn [18].
v1v2 , v3
v4 , v5(b > a > c)(c > b > a)(a > b ∼ c)

Definition 5. [24] A preference profile V of weak orders
is ∃-single-peaked with respect to an axis A if for every v ∈V , v can be extended to a total order v 0 such that the profile
V 0 of total orders is single-peaked with respect to A.

When we evaluate this preference profile under the simple
majority rule where x > y by simple majority if more votersstate x > y than y > x, then V has the majority cycle
a > c > b > a [18]. Clearly V is ∃-single-peaked consistent
with respect to the axis A = a < b < c, so we can make thefollowing observation.

We can restate Definition 5 without referring to extensions to better see how it relates to single-peaked and singleplateaued preferences.

Observation 7. There exists a preference profile of weakorders that is ∃-single-peaked and does not have a transitive
majority relation.

Observation 6. A preference profile V of weak orders is∃-single-peaked with respect to an axis A if and only if forevery v ∈ V , A can be split into three segments X, Y , andZ (X and Z can each be empty) where v’s most preferredcandidates are Y , v has increasing preferences along X, andv has decreasing preferences along Z.

The existential model for single-peakedness considers theexistence of a single extension of the preferences of all of thevoters to total orders. We briefly consider the case where all
extensions to total orders must be single-peaked and maketwo observations.

We define the corresponding problem of ∃-single-peakedconsistency for weak orders below.
Given: A preference profile V of weak orders and a set ofcandidates C.
Question: Does there exist an axis A such that V is ∃single-peaked with respect to A?
Figure 1 illustrates an example of each variant of singlepeakedness for weak orders described where each preference
order is consistent with respect to the axis A = a < d < b <e < c. In Figure 1 the preference order (b > d ∼ e > c > a)is single-peaked, single-plateaued, and ∃-single-peaked. The
preference order (a ∼ d > b > e > c) is single-plateauedand ∃-single-peaked, but not single-peaked since it has morethan one most preferred candidate. The preference order(c > b ∼ e > d ∼ a) is ∃-single-peaked and not singleplateaued or single-peaked since it is not strictly increasing
to its most preferred candidate(s).
We conclude our discussion of these variants of singlepeakedness for weak orders by stating several observations.

105

Observation 8. If a preference profile of weak orders issingle-peaked then all extensions of the preferences to totalorders are also single-peaked.
Observation 9. If a preference profile of weak orders issingle-plateaued and each preference order has at most two
most preferred candidates, then all extensions of the preferences to total orders are single-peaked.

2.2

Consecutive-Ones Matrices

All of our polynomial-time results are due to transformations to the following problem of determining if a 0-1 matrix
has the consecutive ones property.
Given: A 0-1 matrix M .
Question: Does there exist a permutation of the columns of
M such that in each row all of the 1’s are consecutive?
The above problem was shown to be in P by Fulkerson and
Gross [19]. Booth and Lueker [7] improved on this result byfinding a linear-time algorithm through the development anduse of the novel PQ-tree data structure, which contains all
possible permutations of the columns of a matrix such that
all of the 1’s are consecutive in each row.

3.

RESULTSThe following three sections consist of our results and they
are structured as follows. We examine each variant of singlepeakedness starting with the weakest restriction and ending with the strongest. When we examine each restriction
we present an alternative definition of the variant of singlepeakedness using forbidden substructures and the transformation to the problem of determining if a 0-1 matrix has
the consecutive ones property.

3.1

Existentially Single-Peaked Consistency

The most general of the three variants mentioned in Section 2.1 is the model of ∃-single-peaked preferences. The
construction and corresponding proof will be the basis forshowing that single-peaked and single-plateaued consistency
for weak orders are each also in P.
Given an axis A and a preference order v, if v is ∃-singlepeaked with respect to A then v cannot have strictly decreasing and then strictly increasing preferences with respect toA. Following the terminology used by Lackner [24], we referto this as a v-valley.
Definition 10. A preference order v over a candidate set
C contains a v-valley with respect to an axis A if there exist
candidates a, b, c ∈ C such that a < b < c in A and (a > b)and (c > b) in v.
Using the v-valley substructure we can state the following
lemma, which will simplify our argument used in the proofof Theorem 14.

iteratively in the following way. If there exists a v ∈ V suchthat the last ranked candidate in v is not ranked indifferently
with any other candidate, then that candidate is appended
to the top of the guiding order. This is then repeated on thepreference profile restricted to the candidates not yet added
to the guiding order until either the guiding order is a totalorder or there is no v ∈ V with a unique last ranked candidate, the case where no guiding order exists [24]. Observethat if a given preference profile is ∃-single-peaked then it
remains ∃-single-peaked if a guiding order is added as an additional preference order [24]. It is important to point out
that our results do not depend on the existence of a guiding
order in a preference profile. Below we show how Construction 12 is applied to a preference profile of weak orders thatis ∃-single-peaked.
Example 13. Consider the preference profile V that consists of the preference orders v and w. Let the preference order v be (a ∼ c > b > e ∼ d > f ) and the preference order wbe (a > b > c > e ∼ d > f ). Notice that V does not containa guiding order, which is required by the polynomial-timealgorithm for weak orders found in Lackner [24].
a11Xv =1
1
1

b c d e f

0 1 0 0 0
0 1 0 0 0
1 1 0 0 0
1 1 1 1 0
1 1 1 1 0a11
Xw =1
1
1

a
1
1

1

1

1
X =
1
1

1

1
1

b c d e f

0 1 0 0 0
0 1 0 0 0
1 1 0 0 0
1 1 1 1 0
1 1 1 1 0
0 0 0 0 0
1 0 0 0 0
1 1 0 0 0
1 1 1 1 0
1 1 1 1 0b a c d e f

0 1 1 0 0 0
0 1 1 0 0 0
1 1 1 0 0 0
1 1 1 1 1 0
1 1 1 1 1 0
X 0 =

0 1 0 0 0 0
1 1 0 0 0 0
1 1 1 0 0 0
1 1 1 1 1 0
1 1 1 1 1 0

To construct a matrix from a preference profile of weakorders, we apply essentially the same transformation as used
Bartholdi and Trick [4] for total orders (see Example 13).
We describe the construction below.

The main difference in our construction is that we have
one fewer row in each of the individual preference matrices.
In the construction used by Bartholdi and Trick [4], given apreference order v over a set of candidates C, for all a, b ∈C, (a > b) in v if and only if the number of 1’s in thecolumn corresponding to a is greater than the number of
1’s in the column corresponding to b in v’s corresponding
individual preference matrix. Notice that this still holds forour construction.
The polynomial-time results for ∃-single-peaked consistency for weak orders and local weak orders proved in Lackner [24] require that the given preference profile contains a
guiding order, i.e., an implicitly specified total order. Givena preference profile V , a guiding order can be constructedb c d e f

0 0 0 0 0
1 0 0 0 0
1 1 0 0 0
1 1 1 1 0
1 1 1 1 0

We then row-wise concatenate Xv and Xw to construct X.

Lemma 11. [24] Let V be a preference profile of weak orders. V is ∃-single-peaked with respect to an axis A if andonly if no preference order v ∈ V contains a v-valley withrespect to A.

Construction 12. Let V be a preference profile of weakorders over candidate set C. For each v ∈ V construct a(kCk − 1) × kCk matrix Xv . Each column of Xv corresponds
to a candidate in C. For each candidate c ∈ C let k be
the number of candidates that are strictly preferred to c inv and let the corresponding column in matrix Xv contain k
0’s starting at row one, with the remaining entries filled with1’s. All kV k of the matrices are row-wise concatenated toyield the (kV k · (kCk − 1)) × kCk matrix X.



Next, we permute the columns of X so that in each row allof the 1’s are consecutive to yield X 0 . Observe that V is∃-single-peaked with respect to b < a < c < d < e < f , theordering of the columns of X 0 as its axis.
We now show that ∃-single-peaked consistency for weakorders and the problem of determining if the constructed
0-1 matrix has the consecutive ones property are equivalentusing Lemma 11 and Construction 12.
Theorem 14. A preference profile V of weak orders is∃-single-peaked consistent if and only if the matrix X, constructed using Construction 12, has the consecutive ones
property.
Proof. Let V be a preference profile of weak orders.
Essentially the same argument as used by Bartholdi andTrick [4] holds.
If V is ∃-single-peaked with respect to an axis A then by
Lemma 11 we know that no preference order v ∈ V contains a v-valley with respect to A. When the columns of thematrix X are permuted to correspond to the axis A no row

106will contain the sequence · · · 1 · · · 0 · · · 1 · · · since this corresponds to a preference order that strictly decreases and thenstrictly increases along the axis A (a v-valley). Therefore X
has the consecutive ones property.
For the other direction suppose that V is not ∃-singlepeaked, then by Lemma 11 we know that for every possible axis there exists a preference order v ∈ V such that v
contains a v-valley with respect to that axis. So every permutation of the columns of X will correspond to an axis
where some preference order has a v-valley. As stated in theother direction, a v-valley corresponds to a row containing
the sequence · · · 1 · · · 0 · · · 1 · · · so clearly X does not havethe consecutive ones property.
The only difference from the argument used by Bartholdi
and Trick [4] for total orders is that in our case the preference
orders can remain constant at the peak and at points oneither side of the peak. The same argument still applies
since by Lemma 11 the absence of v-valleys with respect
to an axis is equivalent to a profile of weak orders being∃-single-peaked with respect to that axis.
Corollary 15. ∃-Single-peaked consistency for weak orders is in P.

3.2

Single-Plateaued Consistency

Single-plateaued preferences are a much more restrictivemodel than ∃-single-peaked preferences since they are essentially single-peaked except that each preference order canhave multiple most preferred candidates [6, Chapter 5].
Since a preference order must be strictly increasing andthen strictly decreasing with respect to an axis (excluding
its most preferred candidates) we can again use the v-valley
substructure. However we will need another substructure
to prevent two candidates that are ranked indifferent in a
voter’s preference order from appearing on the same side ofthat voter’s peak (plateau).
Definition 16. A preference order v over a candidate set
C contains a nonpeak plateau with respect to A if there exist
candidates a, b, c, ∈ C such that a < b < c in A and either(a > b ∼ c) or (c > b ∼ a) in v.
We use the v-valley and nonpeak plateau substructures tostate the following lemma.
Lemma 17. Let V be a preference profile of weak orders.
V is single-plateaued with respect to an axis A if and only if
no preference order v ∈ V contains a v-valley with respect to
A and no preference order v ∈ V contains a nonpeak plateau
with respect to A.
Proof. Let C be a candidate set, V be a preference profile of weak orders, and A be an axis.
If V is single-plateaued with respect to A then for every
preference order v ∈ V , A can be split into segments X, Y ,and Z such that v is strictly increasing along X, remaining
constant along Y , and strictly decreasing along Z. Since v is
only ever strictly decreasing along Z and Z is the rightmost
segment of A, v cannot contain a v-valley with respect toA. For a nonpeak plateau to exist with respect to A there
must exist candidates a, b, c ∈ C such that a < b < c in A
and either (a > b ∼ c) or (c > b ∼ a) in v.
We first consider the case of (a > b ∼ c) in v. Since a isstrictly preferred to b and c in v and both b and c are to theright of a on the axis we know that both b and c must be in107

segment Z. However, v is strictly decreasing along Z, so v
cannot have a nonpeak plateau of this form.
We now consider the case of (c > b ∼ a) in v. Since c isstrictly preferred to a and b in v and both a and b are to theleft of c on the axis we know that both a and b must be insegment X. However, v is strictly increasing along X, so v
cannot have a nonpeak plateau of this form.
For the other direction we consider the case when no preference order v ∈ V contains a v-valley with respect to A andno preference order v ∈ V contains a nonpeak plateau withrespect to A.
Since no preference order v ∈ V contains a v-valley withrespect to A, we know from Lemma 11 that V is ∃-singlepeaked with respect to A. Since we also know that no preference order v ∈ V contains a nonpeak plateau with respect
to A it is easy to see that V is single-plateaued with respect
to A.
Since the nonpeak plateau substructure is needed in addition to the v-valley substructure, we need to extend Construction 12 so that if a preference order contains a nonpeak
plateau with respect to an axis A, then when the columnsof its corresponding preference matrix are permuted according to A the matrix will contain a row with the sequence···1···0···1···.
Notice that if a preference order ranks three candidates
indifferent to each other below its peak (plateau) that it
will have a nonpeak plateau with respect to every possible
axis. To handle this case in the extension to Construction 12
we need to ensure that its corresponding preference matrix
will contain a row with the sequence · · · 1 · · · 0 · · · 1 · · · forevery permutation of its columns.
Construction 18. Let V be a preference profile of weakorders over candidate set C. For each v ∈ V construct a(kCk − 1) × kCk matrix Xv . Each column of Xv corresponds to a candidate in C. For each candidate c ∈ C let k
be the number of candidates that are strictly preferred to cin v and let the corresponding column in matrix Xv containk 0’s starting at row one, with the remaining entries filledwith 1’s (as in Construction 12). The following extensions
to Construction 12 ensure that if v has nonpeak plateau withrespect to an axis A then when the columns of Xv are permuted according to A it will not have consecutive ones inrows.
For each pair of candidates a, b ∈ C such that (a ∼ b) in v,
they are not the most preferred candidates in v, and there isno candidate c ∈ C − {a, b} such that v is indifferent amonga, b, and c, then append three additional rows to the ma0
trix Xv where the column corresponding to a is 0 1 1 ,0
the column corresponding to b is 1 1 0 , each column
to a candidate strictly preferred to a and b iscorresponding
0
1 1 1 , and each column corresponding to a remaining0candidate is 0 0 0 .
If there exist three candidates a, b, c ∈ C such that (a ∼b ∼ c) in v and they are not the most preferred candidates
in v, then output a matrix that has no solution.
After constructing an Xv matrix for each v ∈ V , all kV kof the matrices are row-wise concatenated to yield a matrix
X, except in the case where the input resulted in a matrix
with no solution.
We now show how Construction 18 is applied to a preference profile of weak orders that is single-plateaued.

Example 19. We consider the same preference profile asin Example 13 and we bold the additional rows in this example. Let the preference order v be (a ∼ c > b > e ∼ d > f )and the preference order w be (a > b > c > e ∼ d > f ).
a
1
1

1

1
Xv =
1
1

1
1

b
0
0
1
1
1
1
1
1

a
1
1

1

1

1

1

1

1
X =
1
1

1

1

1

1

1
1

b
0
0
1
1
1
1
1
1
0
1
1
1
1
1
1
1



c d e f
1 0 0 01 0 0 0

1 0 0 0

1 1 1 0

1 1 1 0
1 0 1 0

1 1 1 0
1 1 0 0

a
1
1

1

1
Xw =
1
1

1
1

b
0
1
1
1
1
1
1
1

c d e f
1 0 0 01 0 0 0

1 0 0 0

1 1 1 0

1 1 1 0

1 0 1 0

1 1 1 0

1 1 0 0

0 0 0 0
0 0 0 0

1 0 0 0

1 1 1 0

1 1 1 0

1 0 1 0

1 1 1 0
1 1 0 0

e
0
0

0

1

1

1

1

0
0
X =
0
0

0

1

1

1

1
0

b a c d f

0 1 1 0 0
0 1 1 0 0
1 1 1 0 0
1 1 1 1 0
1 1 1 1 0
1 1 1 0 0
1 1 1 1 0
1 1 1 1 0
0 1 0 0 0
1 1 0 0 0
1 1 1 0 0
1 1 1 1 0
1 1 1 1 0
1 1 1 0 0
1 1 1 1 0
1 1 1 1 0c d e f0 0 0 00 0 0 0

1 0 0 0

1 1 1 0

1 1 1 0
1 0 1 0

1 1 1 0
1 1 0 0We then row-wise concatenate Xv and Xw to construct X.


Next, we permute the columns of X such that in each row
all of the ones are consecutive to yield X 0 . Observe that V
is single-plateaued with respect to this new ordering e < b <a < c < d < f as its axis. Also notice that an axis containing
d and e adjacent to each other (as seen in Example 13) would
not correspond to an ordering of the columns of X withconsecutive ones in rows due to the additional rows fromthe extensions made to Construction 12 in Construction 18.
Construction 12 ensures that no preference order contains
a v-valley and the extensions made in Construction 18 ensurethat no preference order contains a nonpeak plateau. So theproof of the following theorem uses a similar argument to
the proof of Theorem 14. Now the presence of v-valleys or
nonpeak plateaus, not just v-valleys, is equivalent to a row
containing the sequence · · · 1 · · · 0 · · · 1 · · · .
Theorem 20. A preference profile V of weak orders issingle-plateaued consistent if and only if the matrix X, constructed using Construction 18, has the consecutive ones
property.
Proof. Let V be a preference profile of weak orders. We
extend the argument used by Bartholdi and Trick [4] and theproof of Theorem 14 except in this case we use Lemma 17
instead of Lemma 11.
If V is single-plateaued with respect to an axis A then by
Lemma 17 we know that no v ∈ V contains a v-valley withrespect to A and no v ∈ V contains a nonpeak plateau withrespect to A. When the columns of the matrix X are permuted to correspond to the axis A no row will contain thesequence · · · 1 · · · 0 · · · 1 · · · since this would correspond to apreference order that strictly decreases and then strictly increases along the axis A (a v-valley) or it would correspondto a preference order that has two candidates ranked indifferent appearing on the same side of its peak (a nonpeak
plateau). Therefore X has the consecutive ones property.
If V is not single-plateaued then we know from Lemma 17
that for every possible axis there exists a preference orderv ∈ V such that v contains a v-valley or v contains a nonpeak plateau with respect to that axis. So every permutation of the columns of X will correspond to an axis wherea preference order has a v-valley or a nonpeak plateau. As
stated in the other direction, the presence of a v-valley ora nonpeak plateau corresponds to a row containing the sequence · · · 1 · · · 0 · · · 1 · · · . Therefore X does not have theconsecutive ones property.
Corollary 21. Single-plateaued consistency for weakorders is in P.

3.3

Single-Peaked Consistency

We now present our results for the strongest domain restriction on weak orders that we examine, single-peaked
preferences. Recall that a preference order is single-peakedwith respect to an axis A if it is strictly increasing to asingle most preferred candidate (peak) and then strictly decreasing with respect to A. So we again use the v-valley
substructure, but like the previous case of single-plateaued
preferences we need an additional substructure. Even if nopreference order has a v-valley with respect to A it may not
be single-peaked because it is indifferent between two candidates on the same side of its peak or has more than one
most preferred candidate.
We can handle the first condition just mentioned with
the nonpeak plateau substructure used in Section 3.2, but
the second condition requires us to view any plateau as aforbidden substructure.
Definition 22. A preference order v over a candidate set
C contains a plateau with respect to an axis A if there exist
candidates a, b ∈ C such that a and b are adjacent in A and(a ∼ b) in v.
We can now use the plateau substructure and the v-valley
substructure to state the following lemma.
Lemma 23. Let V be a preference profile of weak orders.
V is single-peaked with respect to an axis A if and only if
no preference order v ∈ V contains a v-valley with respect
to A and no preference order v ∈ V contains a plateau withrespect to A.
Proof. Let C be a candidate set, V be a preference profile of weak orders, and A be an axis.
If V is single-peaked with respect to A then clearly V isalso single-plateaued with respect to A. So by Lemma 17
we know that no preference order v ∈ V contains a v-valley
with respect to A and no preference order v ∈ V contains anonpeak plateau with respect to A. Since V is single-peaked
we also know that no preference order v ∈ V has more than
one most preferred candidate so clearly no preference orderv ∈ V contains a plateau with respect to A.
For the other direction we consider the case when no preference order v ∈ V contains a v-valley with respect to A andno preference order v ∈ V contains a plateau with respect
to A.

108

Since no preference order v ∈ V contains a v-valley withrespect to A we know from Lemma 11 that V is ∃-singlepeaked with respect to A. Since we also know that no preference order v ∈ V contains a plateau with respect to A itis easy to see that V is single-peaked with respect to A.
We extend Construction 18 so that if a preference ordercontains a plateau with respect to an axis A, then when thecolumns of its preference matrix are permuted according to
A the matrix will contain the sequence · · · 1 · · · 0 · · · 1 · · · .
Since Construction 18 already ensures this for the case of
nonpeak plateaus, our extended construction below only
needs to add a condition for plateaus that contain the most
preferred candidates in a given preference order.
Construction 24. Follow Construction 18 except add
the following condition while constructing a preference matrix Xv for each preference order v ∈ V .
If there exist two candidates a, b ∈ C such that (a ∼ b)in v and they are the most preferred candidates in v, thenoutput a matrix that has no solution.
Clearly the extension to Construction 18 above ensuresthat if there are multiple most preferred candidates in a preference order then the preference matrix constructed fromthat order does not have the consecutive ones property.
When a preference order has a unique most preferredcandidate and is single-plateaued, it is clearly also singlepeaked. Construction 24 ensures that no preference ordercontains more than one most preferred candidate the same
way that Construction 18 ensures that no preference ordercontains three or more candidates that are all ranked indifferent to each other and that are not the most preferredcandidates, since this always results in a nonpeak plateau.
So the proof of the following theorem follows from the proofof Theorem 20, but using Lemma 23 instead of Lemma 17.
Theorem 25. A preference profile V of weak orders issingle-peaked consistent if and only if the matrix X, constructed using Construction 24 has the consecutive ones
property.
Corollary 26. Single-peaked consistency for weak orders is in P.

4.

CONCLUSIONS AND FUTURE WORK

We presented three different variants of single-peaked
preferences for weak orders and showed that each of theircorresponding consistency problems are in P. Since we accomplished this by using transformations to the problem ofdetermining if a 0-1 matrix has the consecutive ones property we are able to apply the PQ-tree algorithm from Boothand Lueker [7]. Using this algorithm we can actually go further than just determining the consistency problem for eachof these variants and find all consistent axes. An interesting
open direction is how the consecutive ones matrix problem
relates to other domain restrictions and what benefits there
are to having all consistent axes for a given preference profile.
The existential approach introduced by Lackner for singlepeaked preferences [24] has been recently applied to otherdomain restrictions. The model of single-crossing preferences [27] was studied in the existential model by Elkind etal. [11] and the model of top-monotonic preferences [3] was109

studied in the existential model by Aziz [1]. An interesting
direction for future work would be to apply the existential
model to other domain restrictions.
Single-peaked preferences are studied because they are asimply stated and important domain restriction that gives
insight into how the voters view the candidates and elections with single-peaked voters have nice properties. However, experimental study suggests that in real-world settings
voters are often not single-peaked [25], but in this study thesingle-peaked results only used Black’s definition for totalorders. It would be interesting to see if real-world datasetsof weak orders contain voters that are single-peaked, singleplateaued, or ∃-single-peaked.
In single-peaked and nearly single-peaked elections computational problems often become easier [17, 16]. As mentioned by Lackner [24] an important open problem is todetermine what computational benefits are gained when apreference profile is ∃-single-peaked or even nearly ∃-singlepeaked. There are several different types of nearly singlepeakedness and determining if a given preference profile isnearly single-peaked with respect to a certain distance measure is an interesting computational problem [14]. It would
be interesting to see how preference profiles of weak orders
impact the complexity of nearly single-peakedness or, as alsomentioned by Lackner [24], nearly single-peakedness in theexistential model.

5.

ACKNOWLEDGMENTS

The author thanks Edith Hemaspaandra, Martin Lackner,
David Narváez, Stanislaw Radziszowski, and the anonymousreferees for their helpful suggestions and feedback. This
work was supported in part by NSF grant no. CCF-1101452
and a National Science Foundation Graduate Research Fellowship under NSF grant no. DGE-1102937.

6.

REFERENCES[1] H. Aziz. Testing top monotonicity. Technical Report
arXiv:1403.7625 [cs.GT], arXiv.org, June 2014.
[2] S. Barberà. Indifferences and domain restrictions.
Analyse & Kritik, 29(2):146–162, 2007.
[3] S. Barberà and B. Moreno. Top monotonicity: A
common root for single peakedness, single crossing
and the median voter result. Games and Economic
Behavior, 73(2):345–359, 2011.
[4] J. Bartholdi, III and M. Trick. Stable matching withpreferences derived from a psychological model.
Operations Research Letters, 5(4):165–169, 1986.
[5] D. Black. On the rationale of group decision-making.
Journal of Political Economy, 56(1):23–34, 1948.
[6] D. Black. The Theory of Committees and Elections.
Cambridge University Press, 1958.
[7] K. Booth and G. Lueker. Testing for the consecutive
ones property, interval graphs, and graph planarity
using PQ-tree algorithms. Journal of Computer and
System Sciences, 13(3):335–379, 1976.
[8] F. Brandt, M. Brill, E. Hemaspaandra, and L. A.
Hemaspaandra. Bypassing combinatorial protections:
Polynomial-time algorithms for single-peaked
electorates. In Proceedings of the 24th AAAI
Conference on Artificial Intelligence, pages 715–722,
July 2010.

[9] R. Bredereck, J. Chen, and G. Woeginger. A
characterization of the single-crossing domain. Social
Choice and Welfare, 41(4):989–998, 2013.
[10] J.-P. Doignon and J.-C. Falmagne. A polynomial timealgorithm for unidimensional unfolding
representations. Journal of Algorithms, 16(2):218–233,
1994.
[11] E. Elkind, P. Faliszewski, M. Lackner, and
S. Obraztsova. The complexity of recognizing
incomplete single-crossing preferences. In Proceedingsof the 29th AAAI Conference on Artificial
Intelligence, pages 865–871, Jan. 2015.
[12] E. Elkind, P. Faliszewski, and A. Slinko. Clone
structures in voters’ preferences. In Proceedings of the
13th ACM Conference on Electronic Commerce, pages
496–513, June 2012.
[13] P. Emerson. The original Borda count and partial
voting. Social Choice and Welfare, 40(2):352–358,
2013.
[14] G. Erdélyi, M. Lackner, and A. Pfandler.
Computational aspects of nearly single-peaked
electorates. In Proceedings of the 27th AAAI
Conference on Artificial Intelligence, pages 283–289,
July 2013.
[15] B. Escoffier, J. Lang, and M. Öztürk. Single-peakedconsistency and its complexity. In Proceedings of the
18th European Conference on Artificial Intelligence,
pages 366–370, July 2008.
[16] P. Faliszewski, E. Hemaspaandra, and L. A.
Hemaspaandra. The complexity of manipulative
attacks in nearly single-peaked electorates. Journal of
Artificial Intelligence Research, 207:69–99, 2014.
[17] P. Faliszewski, E. Hemaspaandra, L. A.
Hemaspaandra, and J. Rothe. The shield that never
was: Societies with single-peaked preferences are more
open to manipulation and control. Information andComputation, 209:89–107, 2011.
[18] P. Fishburn. The Theory of Social Choice, volume 264.
Princeton University Press, 1973.
[19] D. Fulkerson and O. Gross. Incidence matrices andinterval graphs. Pacific Journal of Math,15(3):835–855, 1965.
[20] E. Hemaspaandra, L. A. Hemaspaandra, and J. Rothe.
Exact analysis of Dodgson elections: Lewis Carroll’s
1876 voting system is complete for parallel access to
NP. Journal of the ACM, 44(6):806–825, 1997.
[21] E. Hemaspaandra, H. Spakowski, and J. Vogel. The
complexity of Kemeny elections. Theoretical Computer
Science, 349(3):382–391, 2005.
[22] J. Kemeny. Mathematics without numbers. Daedalus,
88:577–591, 1959.
[23] K. Konczak and J. Lang. Voting procedures withincomplete preferences. In Proceedings of the 1stMultidisciplinary Workshop on Advances in Preference
Handling, pages 124–129, Aug. 2005.
[24] M. Lackner. Incomplete preferences in single-peaked
electorates. In Proceedings of the 28th AAAI
Conference on Artificial Intelligence, pages 742–748,
July 2014.
[25] N. Mattei, J. Forshee, and J. Goldsmith. An empirical
study of voting rules and manipulation with large

datasets. In Proceedings (Workshop Notes) of the 4thInternational Workshop on Computational Social
Choice, Sept. 2012.
[26] N. Mattei and T. Walsh. PrefLib: A library for
preferences. In Proceedings of the 3rd International
Conference on Algorithmic Decision Theory, pages
259–270, Nov. 2013.
[27] J. A. Mirrlees. An exploration in the theory of
optimum income taxation. The Review of Economic
Studies, 38(2):175–208, 1971.
[28] H. Moulin. On strategy-proofness and single
peakedness. Public Choice, 35(4):437–455, 1980.
[29] N. Narodytska and T. Walsh. The computational
impact of partial votes on strategic voting. In
Proceedings of the 21st European Conference on
Artificial Intelligence, pages 657–662, Aug. 2014.
[30] M. Schulze. A new monotonic and clone-independent,
reversal symmetric, and Condorcet-consistent
single-winner election method. Social Choice andWelfare, 36(2):267–303, 2011.

110

Theory Choice, Theory Change, and Inductive
Truth-Conduciveness[Extended Abstract]
Konstantin Genin

Kevin T. Kelly

Carnegie Mellon University

Carnegie Mellon University

konstantin.genin@gmail.com

kk3n@andrew.cmu.edu

ABSTRACT

revision, reliability, formal epistemology

Synchronic norms of theory choice—traditional to the philosophy of science—restrict the theories one can choose inlight of given information. Ockham’s razor is a famous
example. How can one argue that these biases are truthconducive, without begging the question with material assumptions?
Diachronic norms of theory change—as studied in belief
revision—restrict how one should change one’s current beliefs in light of new information. How do the diachronic
norms relate to the synchronic norms? Furthermore, is there
some sense in which the diachronic norms are truth-conducive?
If one insists upon an overly strict standard of truthconduciveness in inductive contexts, the epistemic justification of inductive norms becomes intractable. Theoretical virtues are not guaranteed to indicate truth with a low
chance of error, the way litmus paper indicates pH. But
there is a spectrum of truth-conduciveness concepts, ranging
from the strict standard of truth-indicativeness to the weakstandard of mere convergence in the limit. Neither extreme
suffices for epistemic justification. The former is too strict
to apply and the latter, notoriously, mandates no short-run
norms at all. There are more nuanced concepts of optimally direct convergence to the truth, lying between these
extremes. We consider two such concepts: convergence withminimal reversals of opinion and convergence with minimal
cycles of opinion.
We show how the rationality principles of belief revision
can be thought of as truth-conducive norms of direct convergence. Furthermore, we prove that preferring simple, falsifiable theories is a necessary condition for satisfying thenorms of rational theory change. The results forge deep
and, perhaps, surprising connections between synchronic rationality norms, diachronic rationality norms, and the truthconduciveness of both.

Keywords
theory choice, simplicity, Ockham’s razor, learning, belief

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.
Copyright 20XX ACM X-XXXXX-XX-X/XX/XX ...$15.00.

111

1.

INTRODUCTION

This work is concerned with three things: the synchronic
norms of theory choice, the diachronic norms of theory change,
and the justification of these norms by their reliability, or
truth-conduciveness.
Synchronic norms of theory choice restrict the theories one
can choose in light of given, empirical information. They are
the traditional purview of the philosophy of science. But
how do they facilitate arrival at true theories? Are they
better than other means toward that end and, if so, in what
sense? For example, it is widely agreed that scientific theory
choice proceeds in accordance with a bias toward simpler ormore sharply testable theories. However, the truth might
not be simple, in which case that bias would probably lead toerror. So how can one argue that the characteristic scientificbiases toward simplicity or testability are truth-conducive,
without begging the question with material assumptions?
Diachronic norms of theory change restrict how one shouldchange one’s current beliefs in light of new information. The
crucial difference from synchronic norms is dependency upon
one’s prior beliefs. Such norms are studied propositionallyin belief revision theory and non-monotonic logic and quantitatively in Bayesian epistemology. The question of truthconduciveness arises for such norms, just as it does for normsof theory choice. Furthermore, there is the additional question how diachronic norms relate to the more traditional,
synchronic ones.
If one insists upon an overly strict standard of truthconduciveness in inductive contexts, the crucial question oftruth-conduciveness becomes intractable. Theoretical virtues
are not guaranteed to indicate truth with a low chance of error, the way litmus paper indicates pH—inductive inferencesin accordance with the rationality principles are subject toarbitrarily high chances of error, because the available information can probably be arbitrarily similar, regardless ofwhich conclusion is true.
Maturity is a matter of ceasing to demand the impossible.
In that spirit, it makes more sense to adjust the standards oftruth-conduciveness to the intrinsic difficulty of the inference
problem one faces—a view we call feasibility-contextualism.
Feasibility contextualism presupposes a spectrum of alternative concepts of truth-conduciveness. Just such a spectrum
is routinely studied in the subject known as formal learningtheory, which studies concepts of truth-conduciveness ranging from the very strict standard of truth-indicativeness tothe very weak standard of mere convergence to the truth

in the limit. Neither extreme suffices for epistemic justification. The former is too strict to apply without questionbegging assumptions and the latter, notoriously, mandates
no short-run norms at all, since convergence in the limit
is compatible with any inductive behavior whatever in the
short run. In between these extremes, however, are more nuanced concepts of optimally direct convergence to the truth.
We consider two such concepts in this paper: convergencewith minimal reversals of opinion and convergence with minimal cycles of opinion. Since a strategy is conducive to a goal
insofar as it leads as directly as possible to the goal, we view
directness of approach to the truth as constitutive of truthconduciveness and, hence, of epistemic justification, rather
than as an auxiliary, “pragmatic” consideration.
In this work, we show how the rationality principles ofbelief revision can be thought of as truth-conducive normsof maximally direct convergence in the sense just described.
Furthermore, we prove that preferring simple, falsifiable theories (Ockham’s razor) is a necessary condition for achieving
optimally truth-conducive performance. The results forge
deep and, perhaps, surprising connections between synchronic
rationality norms, diachronic rationality norms, and the truthconduciveness of both.

1.1

Reliability and the Norms of Theory Choice

It is commonplace to observe that science seeks true theories about the world. But that banal observation raises
a profound question about scientific method: how, and inwhat sense, do such hallmark scientific values as simplicity,
precision, scope, and novelty help one find true theories? To
demand an answer to that question is to demand an epistemic justification of scientific values [16, 24, 12]. One goalof this work is to provide such a justification for Ockham’srazor, the pervasive scientific bias in favor of simple theories.
An epistemic justification of Ockham’s razor is traditionally understood to be a demonstration that simpler theories are more likely to be true.1 That narrow, synchronic
concept of justification makes a hopeless conundrum out of
Ockham’s razor. It has led theorists into metaphysical speculations less plausible than the scientific conclusions they aremeant to justify. Both Kepler and Dirac expressed the conviction that Nature loves mathematical elegance, and thatphysicists ought to adopt the same passion to more surely
uncover her secrets [29, 11]. It has led others to give up on
epistemic justification entirely: “no one has shown that anyof these rules is more likely to pick out true theories thanfalse ones. It follows that none of these rules is epistemic incharacter” [25].
There is no shortage of non-epistemic justifications. Predictive accuracy, not truth, is the target of frequentist justifications: a bias toward simple theories prevents over-fitting
and improves prediction when extrapolating from small samples [1, 13, 38]. But while they may be more predictively
accurate at small sample sizes, simple theories are not more
likely to be true in any objective sense of likelihood. Akaike’s
method does not even converge to the true theory in thelimit of infinite data. Some frequentists take that to be a1

Baker [3] is representative: “justifying an epistemic principle requires answering an epistemic question: why are parsimonious theories more likely to be true?” The demandis trivial if “likely” is understood subjectively, so we understand it objectively, in the sense of a guaranteed low chanceof error.

design feature, rather than a flaw, and warn against methods
that both impose penalties on complexity and converge tothe truth [26]. Bayesians, on the other hand, explicate Ockham’s razor as the result of conditioning over a wide classof plausible, prior probabilities that impose flattish distributions over theoretical parameters [17, 5, 39, 30]. But that
does not begin to explain how such prior probabilities lead
one to true theories better than alternative biases would—
unless one begs the question by appealing to the prior probabilities themselves. The question of epistemic justification,
if not begged, is dodged.
The point can be sharpened with a bit of terminology.
Say that a method is truth-indicative if at every stage ofinquiry the theory it selects is probably true. But truthindicative performance is impossible in inductive inference
problems—insisting on achieving that impossible synchronic
standard leads to inductive skepticism. More plausibly, one
can entertain a range of weaker concepts of diachronic truthconduciveness, and understand epistemic justification as achievement of the strongest performance possible for the problem
one faces.2 Weaker demands do not fall short of epistemic
demands. They are, rather, the appropriate epistemic demands, in light of the intrinsic difficulty of the task at hand.
Over half a century ago, Carnap already sketched the idea
in On Inductive Logic [8]. He recognized, that for inductive
methods, synchronic truth-indicativeness is too high a standard: “the fact that the truth of the predictions reached byinduction cannot be guaranteed does not preclude a justification in a weaker sense”. On the other hand, Reichenbach’s
diachronic norm of limiting convergence is too low a standard:
Reichenbach is right in the assertion that any
procedure which does not [converge to the truthin the limit] is inferior to his rule of induction.
However, his rule, which he calls “the” rule of induction, is far from being the only one possessing
the characteristic. The same holds for an infinite
number of other rules of induction. . . . Therefore
we need a more general and stronger method for
examining and comparing any two given rules of
induction ... [8, p. ]The relevant notions of truth-conduciveness have to lie somewhere between those two extremes. If they are to be feasible, they must relax truth-indicativeness. If they are to
mandate interesting short-run methodological principles like
Ockham’s razor, they must demand more than mere limiting convergence. We propose that such notions can be developed by adapting and refining existing concepts from formal
learning theory.

1.2

Learning Theory and Truth-Conduciveness2
In statistics, the distinction between truth-indicative andtruth-conducive methods is closely tracked by the distinction between uniformly and point-wise consistent methods.
Both types of consistency entail convergence to the truth in
the limit, but for uniformly consistent methods, the probability and severity of error can be quantified and boundedat each sample size. For point-wise convergent methods nosuch guarantees can be given. Of course, we prefer uniformly convergent methods, but these do not always exist.
No statistician claims that using a point-wise convergent
method is not epistemically justified when there is no better
alternative.

112Formal learning theory is a mathematical framework for
those postulates of rationality have anything to do withstudying inductive problems and the methods that solve
truth-conduciveness. In §3.1, we demonstrate that such athem [33, 15, 31, 18]. As in computational complexity theconnection does, in fact, exist: we show that a weakened
ory, inductive problems are classified by their intrinsic diffiversion of the rationality postulates is equivalent to a truthculty. Inductive methods are justified so long as they solve a
conduciveness norm from formal learning theory, once theproblem as efficiently as problems of comparable complexity
requirement of limiting convergence has been imposed.
can be solved. From the perspective of formal learning theIt is also not obvious that there should be any connecory, it makes no more sense to demand truth-indicative pertion between the rationality postulates that belief revision
formance in inductive problems than it does to demand gentheorists take to govern theory change, and the synchroniceral polynomial time solutions to NP-hard problems. The
theoretical virtues investigated by philosophers of science.
demands of epistemic justification are kept proportionate toRott expresses a hope that such connections exist:
epistemic complexity.
In his joint book with J.S. Ullian, The Web of
The baseline notion of truth-conduciveness in formal learnBelief (1978), Quine has added more virtues thating theory is limiting convergence: methods eventually settle
good theories should have: modesty, generality,
on the truth as information accumulates, without ever berefutability, and precision. Again, belief revicoming certain that the future holds no surprises in store. As
sion as studied so far has little to offer to reCarnap observed, limiting convergence is compatible with
flect the quest for these intuitive desiderata. Exany arbitrary behavior in the short-run. To narrow the field
cept for the issue of conservatism, Quine’s list isof admissible methods, learning theorists have developed
one of theory choice rather than theory change
several refinements of limiting convergence that fall short ofin that it lists properties that a good posterior
short-run guarantees. One possible refinement is to require
theory should have, independently of the propmethods to minimize the number of mind changes on theerties of the prior theory. It is a strange coinciway to convergence [33, 9, 36, 28, 19, 21, 22]. Another refinedence that the philosophy of science has focussed
ment guards against “U-shaped learning,” wherein learners
on the monadic (nonrelational) features of theory
conjecture a theory, reject it, and then return to it again [7,choice, while philosophical logic has emphasized
6]. We propose that these convergence criteria—lying midthe dyadic (relational) features of theory change.
way between truth-indicativeness and mere limiting convergence—
I believe that it is time for researchers in both
can answers Carnap’s challenge. If scientists are in pursuit
fields to overcome this separation and work toof truth, then virtuous inquiry ought to exhibit the virtues
gether on a more comprehensive picture [34, p.
of pursuit. If the target of pursuit is evasive, then a certain
15].
amount of swerving may be expected. But false starts andU-turns ought to be avoided if possible—virtuous pursuit isas direct as the problem situation allows. We show how optimal truth-conduciveness mandates a preference for simple
theories. That solves the traditional puzzle of justification
for Ockham’s Razor, a hallmark virtue of theory choice.

1.3

Belief Revision and Scientific Inquiry

Belief revision is an alternative, formal framework in which
to analyze belief change driven by new information. Reliability and truth-conduciveness are not central to the belief
revision framework. Gärdenfors seems indifferent—if not
hostile—to concerns about truth:[T]he concepts of truth and falsity are irrelevant
for the analysis of belief systems. These concepts
deal with the relation between belief systems andthe external world, which I claim is not essential
for an analysis of epistemic dynamics. . . . My
negligence of truth may strike traditional epistemologists as heretical. However, one of myaims is to show that many epistemological problems can be attacked without using the notionsof truth and falsity [14, p. 20].
Instead, belief revision theorists derive epistemic justification from conformity with a set of idealized postulates thatgovern rational belief change. The postulates are usually
motivated by considerations of preservation, or minimal change:
the injunction to (1) add only those new beliefs and (2) remove only those old beliefs, that are absolutely compelled
by incorporation of new information.3 It is not obvious that
3

Rott [34] questions whether the rationality postulates of113

In §5, we demonstrate that theoretical refutability is a necessary condition for the rationality postulates of belief revision, in light of the requirement of limiting convergence.
What’s more, we show that theoretical refutability is equivalent to a version of theoretical simplicity.

2.

PROBLEMS AND SOLUTIONS

We first give a minimal characterization of the context
of empirical inquiry. Let W be a set of possible worlds.
A proposition is a set P ⊆ W . The set of all propositions is denoted P(W ). The contradictory proposition is ∅and the necessary proposition is W . We assume the usual
correspondence between logical and set-theoretic operations:P ∧ Q = P ∩ Q, P ∨ Q = P ∪ Q, P c = W \ P and P entails
Q iff P ⊆ Q.
Some propositions correspond to possible information
states. Propositional information is understood to be true.
Examples include propositions concerning discrete experimental outcomes and inexact measurements of continuous
quantities. Let I ⊆ P(W ) be the set of all possible information states one might be in. We denote the set of all
information states in world wSas: I(w) = {E ∈ I : w ∈ E}.
For P ⊆ W , we let I(P ) = w∈P I(w). The set of information states I is an informationbasis iff the followingS
postulates are satisfied. I1. I = W ; I2. If A, B ∈ I(w),belief revision are correctly thought of as principles of minimal change. Alternatively, one can think of them as monotonicity principles. That also suggests a rapprochement withthe learning-theoretic viewpoint: adherence to the rationality principles throughout the course of inquiry guarantees acertain degree of monotonicity—or “directness” – of convergence to the truth.

then A ∩ B ∈ I(w).4 I3. |I| ≤ ω. The second thesis is themost interesting: it ensures that information accumulates
through time. The third thesis is partly for mathematicaltractability, but it is also well-motivated by Turing’s [37]argument that infinite gradations of input information are
indistinguishable.
The closure of the information basis under arbitrary union,I ∗ , determines a topological space. An open set of I ∗ is averifiable proposition — if true, there is information available that entails it. A closed set of I ∗ is refutable — its
complement is verifiable. A set is locally closed in I ∗ iff
it is closed in an open subspace of I ∗ . The locally closed
sets are verifutable propositions — if true, it is eventually
verified that they are refutable. A set is constructible in
I ∗ iff it is a finite union of locally closed sets. A set isΣ02 in I ∗ iff it is a countable union of locally closed sets.
The topological closure of P ⊆ W , written as P , is the set{w : I(w) ⊆ I(P )} — the set of worlds where P is never∨refuted. The topological frontier, written as P , is definedas P \ P — the set of worlds where P is false, but neverrefuted. The following characterization of locally closed sets
will prove useful in what follows.
∨

Theorem 1. P is locally closed iff P is closed.
A question Q on W is a countable partition of W into
mutually exclusive and exhaustive answers. We denote theunique answer
true in w ∈ W as Q(w). For P ⊆ W , we letS
Q(P ) = w∈P Q(w). A question is locally closed iff each of
its answers is. One question refines another, if each answerof the one entails some answer of the other. An empiricalproblem is a triple P = (W, I, Q), specifying the set of
possibilities entertained (W ), the information provided (I)and the information demanded (Q). A learner is a function
from information states to conjectures λ : I → Q∗ , whereQ∗ is the closure of Q under arbitrary disjunction. A learner
λ solves P in the limit iff for all w ∈ W , there is E ∈ I(w)such that for all F ∈ I(w), λ(E ∩ F ) = Q(w). Call sucha learner a solution to P. No matter which world is thetrue one, a solution eventually converges on the answer truein that world. For learner λ and w ∈ W , define Lock(λ, w)
as the set of all E ∈ I(w) such that if F ∈ I(w), thenλ(E ∩ F ) ⊆ Q(w). We call these the locking propositions,following the usage in [31]. An empirical problem is solvable
iff there is a learner that solves it in the limit. The following
theorem characterizes solvable problems.5
Theorem 2. The following propositions are equivalent:
1. Problem P = (W, I, Q) is solvable in the limit;
2. Each answer Q ∈ Q is a Σ02 proposition;
3. Question Q is refined by a locally closed question Q0 .
4This can be weakened to I’2. If A, B ∈ I(w), then there is
C ∈ I(w) such that C ⊆ A ∩ B. Nothing essential would bechanged, though the statements of the theorems and their
proofs would be more cumbersome.
5
A version of this result is given by de Brecht and Yamamoto[10, Theorem 5], where it is couched in the terms of computable analysis. A similar theorem was proven independently by Kelly [20, Corollary 1] in a first-countable setting.
It was arrived at independently by Baltag, Gierasimczuk,and Smets [4, Theorem 8]. It is proven for the Baire spaceby Kelly [18, Proposition 4.10].

We have two easy corollaries. Say that a learner is consistentiff λ(E) ∩ E is non-empty, for all non-empty E ∈ I. Then:
Theorem 3. Every solvable problem has a consistent solution.
New information E may be thought of as shifting the given
problem P to the restricted problem
P|E = (E, I|E , Q|E ),where I|E is the set of all information states F ∩E such that
F is in I, and Q|E is the set of all H ∩ E such that H is ananswer to Q. Then:
Theorem 4. If P is a solvable problem and C ⊆ W , then
P|C is a solvable problem.

3.

REFINING LIMITING SOLVABILITY

Solution in the limit furnishes a minimal notion of inductive truth-conduciveness. But as many have observed,
it enforces no interesting methodological norms, since it isconsistent with any short-run behavior. We introduce somenotions of truth-conduciveness that refine solution in thelimit. As we will show, these notions of truth-conducivenessare weak enough to be feasible in a broad class of problems,and strong enough to mandate interesting norms of theory
choice. We also demonstrate how these norms relate to thediachronic norms of theory change advocated as principles
of rationality in belief revision and non-monotonic logic.
We first define three different forms of non-monotonicity,
in decreasing generality. A reversal sequence is a sequence∗c(Ai )ni=0 , where each Ai ∈ Q \ {∅}, and Ai+1 ⊆ Ai . A
cycle sequence is a reversal sequence such that An ⊆ A0 .
For reversal sequences a, b of length n, define the severity
pre-order b ≤ a to hold iff Bi ⊆ Ai , for each i ≤ n, wherea ≤ b means that a reverses as severely as b. The strict
relation < holds iff ≤ holds one way and not the other. For
example, (A, B, C) < (A ∪ D, B, C). A reversal sequencea = (Ai )ni=0 is forcible in P iff for every λ that solves P,
there is a nested set of information states e = (Ei )ni=0 , suchthat λ(e) = (λ(Ei ))ni=0 ≤ a. We can characterize the forcible
sequences as follows.
Theorem 5. If P is solvable, then reversal sequence a =(A0 , . . . , An ) is forcible in P iff
A0 ∩ A1 ∩ . . . ∩ An−1 ∩ An 6= ∅.
We define the forcible paths in P by a recursion on thelength of paths.
Π1 = {A : A ∈ Q};

Πn = {A ∩ B : A ∈ Q and B ∈ Πn−1 },S
and Path(P) = i∈N Πi .

3.1

Avoiding Cycles

One refined notion of truth-conduciveness is to avoid cycles altogether. A learner λ is cycle free iff there exist
no nested set of information states e, such that λ(e) is acycle sequence. That truth-conduciveness notion is closely
related to several norms of rational theory change. Say thata learner λ satisfies conditionalization iff λ(E) ∩ Q(E ∩F ) ⊆ λ(E ∩ F ). A learner λ is rationally monotone iff
114λ(E∩F ) ⊆ λ(E)∩Q(E∩F ) whenever λ(E)∩Q(E∩F ) 6= ∅.6
Many authors have suggested that these principles may be
too strong to govern inductive inference [14, 35, 27]. However, both these principles are weakened by the following.
A learner λ is reversal monotone iff λ(E ∩ F ) meets λ(E)whenever λ(E) ∩ Q(E ∩ F ) 6= ∅. We show that for learners,
reversal monotonicity is equivalent to avoiding cycles. This
demonstrates a tight connection between a norm of truthconduciveness, and two norms of theory change.

Theorem 9. If question Q is locally closed, then ≺ istransitive on Q.7
It follows immediately that ≺ is a strict order on Q, andthat  is a partial order on Q. Our notion of simplicity isclosely related to Popper’s. Popper [32] proposes to define
simplicity in terms of the falsifiability relation:
A statement x is said to ‘falsifiable in a higher
degree’ or ‘better testable’ than a statement y ...
if and only if the class of potential falsifiers of x
includes the class of the potential falsifiers of y as
a proper subclass’ ... The epistemological questions which arise in connection with the conceptof simplicity can all be answered if we equate thisconcept with degree of falsifiability.

Theorem 6. If consistent λ solves P, then λ is cycle freeiff λ is reversal monotone.
Although not all solvable problems have cycle-free solutions,
every solvable problems is refined by one that does. This
universality principle is a direct consequence of Theorem 3of Baltag, Gierasimczuk, and Smets [4].
Theorem 7. If P = (W, I, Q) is a solvable problem, thenthere is P0 = (W, I, Q0 ) such that Q0 is locally closed, Q0
refines Q, and P0 is solved by a cycle-free learner.

3.2

Minimizing Reversals

Although in many cases cycles can be avoided altogether,reversals cannot be avoided in inductive problems. But thisdoes not mean that they cannot be minimized. Say that λ isreversal optimal for P iff λ solves P and every λ-reversal
sequence is forcible. Not all problems have reversal optimal
solutions, but we can characterize the ones that do. For
A ∈ Q, define A⊥ = {B ∈ Path(P) : A ∩ B = ∅}.
Theorem 8. Locally closed P has a reversal optimal solution iff for every A ∈ Q, A ∩ ∪A⊥ = ∅.
The characterization is straightforward: a problem has areversal-optimal solution iff in each answer A, the set of allpaths that are not forcible from A, A⊥ , are topologicallyseparable from A.

4. SIMPLICITY AND FALSIFIABILITY
If we live in a world where bread will always nourish, allinformation we ever receive will be consistent with breadceasing to nourish sometime in the future. But the situation is asymmetrical: if we lived in a world where breadceases to nourish eventually, we would find out sooner orlater. There is a natural order that captures the structureof inductive underdetermination between possibilities. For∨

X, Y ∈ P(W ) set X ≺ Y iff X ⊆ Y , and X  Y iff X ≺ Y
or X = Y . If X ≺ Y , we say that X faces the problem
of induction with respect to Y , i.e. X is inconsistent withY , but any information consistent with X is consistent with
Y . For all X, Y ∈ P(W ), we say that X is simpler thanY if and only if X ≺ Y. That straightforward relation captures many of our simplicity intuitions. Exactly that relation holds between sets of polynomials of lower and higher
degree; between nested statistical models; and between universal generalizations like “all ravens are black” and their
negations. The ≺ relation is not in general a strict order.
However, if Q is locally closed then ≺ determines a strict
order over the elements of Q.

6
Conditionalization and rational monotonicity are intended
as analogues to principles (K ∗ 7) and (K ∗ 8) of AGM revisionrespectively and the defeasible inference principles of thesame name.

115

But Popper’s notion and ours are not equivalent. To see
that, define the class of potential falsifiers of X ⊆ W as theset of all information states inconsistent with X: F(X) =I \ I(X). Then X is more falsifiable than Y , if F(Y ) ⊆F(X), or, equivalently, X ⊆ Y . That notion of simplicity
has the bizarre defect that every proposition is simpler thanits consequences. Our notion of simplicity does not face thisdifficulty, since X ≺ Y iff X is more falsifiable than Y , and
X and Y are incompatible.
We can make the make the relation between simplicity
and falsifiability even more explicit. For any propositionA, define the S
set of propositions strictly simpler than A as
follows: A≺ : {B ≺ A : B ⊆ W }. Say that A is simplest
iff A is minimal in ≺, i.e. A≺ = ∅. Then it is easy to show∨that ∪A≺ = A, and that A≺ = ∅ iff A is closed (falsifiable).85.

THE NORMS OF CHOICE

In this section we give two different methodological principles that are necessary for avoiding cycles, and minimizing
reversals, respectively. That furnishes the necessary connection between truth-conduciveness and the norms of theory
choice. Say that a learner λ is Ockham iff for all E ∈ I,
λ(E)≺ ∩ E = ∅, i.e. λ(E) is always simplest in E. On
this conception, Ockham’s razor is content-neutral: it does
not say whether the conjecture has to be weak or strong,requiring only that there be nothing strictly simpler compatible with current information. That is equivalent to therequirement that the learner’s conjecture is falsifiable at every stage of inquiry, which is precisely Popper’s requirementof severe testability—an enjoiner for bold conjectures, vulnerable before the tribunal of experience. We show thatobeying Ockham’s razor is necessary for avoiding cycles.
Theorem 10. If λ solves P and λ is cycle free, then λ is
Ockham.
As an immediate consequence of Theorems 7, and 10, we
have the following Theorem.
Theorem 11. If P = (W, I, Q) is a solvable problem,
then there is P0 = (W, I, Q0 ) such that Q0 is locally closed,
Q0 refines Q, and P0 is solved by an Ockham learner.
We state our second methodological principle as follows.
A learner λ is patient iff for all E ∈ I and Q ⊆ Q(E), there

7
This can be weakened to the requirement that every Q ∈ Q
is constructible.
8
Recall that a set is closed iff its frontier is empty.

is Q0 ⊆ λ(E) such that Q0 ∩ Q 6= ∅.9 Although Ockham
methods can favor some simplest possibilities over others, apatient method must disjoin all simplest possibilities compatible with current information. Furthermore, if a patient
conjecture entails the negation of a particular answer, it isbecause it concedes a simpler possibility. In particular, if
λ(E) ⊆ Qc , then λ(E) ∩ Q≺ 6= ∅. A patient method always
has a simplicity-based reason for disbelieving an answer.

8.

Theorem 12. If λ is reversal optimal for P = (W, I, Q),
then λ is patient.

which is an intersection of two closed sets and therefore
closed. The final equality follows from the fact that O ∩ C ∩As a consequence of theorem 8, we have the following
theorem.
Theorem 13. Locally closed P has a patient, Ockham solution iff for every A ∈ Q, A ∩ ∪A⊥ = ∅.

6. CONCLUSION
In this work we have proposed two norms of inductive
truth-conducivness: the avoidance of theoretical cycles, andthe minimization of reversals. We have shown that once
the requirement of limiting convergence is imposed, avoiding
cycles is equivalent to a weakening of two norms of theorychange from belief revision and non-monotonic logic. We
have also given a topological characterization of theoretical simplicity, and formulated three related norms of theory choice: falsifiability, Ockham’s razor, and patience. We
have shown that minimizing reversals requires learners to bepatient. We have also shown that avoiding cycles requires
learners to be Ockham, and always make falsifiable conjectures. This means that the preservation principles from belief revision and non-monotonic logic, combined with therequirement of limiting convergence, all necessitate a preference for simpler, more falsifiable theories. We take thisto demonstrate a surprising connection between the Popperian preference for falsifiable theories, and the principles ofrational belief change.

7. ACKNOWLEDGEMENTS
We are indebted to Thomas Icard for suggesting connections between our topological conception of simplicity andrelated work in the semantics of provability logic, which
proved to be very fruitful. We are also indebted to Alexandru Baltag, Nina Gierasimczuk, and Sonja Smets, for comments and discussions, for informing us of related work by
Debrecht and Yamamoto [10], and for sharing their related
results with us, particularly their sharpening of our Theorem 7. We are indebted for discussions following invited
presentations of earlier versions of the preceding results at
a number of venues: The University of Amsterdam (the2011 Beth/Vienna Circle Lecture), the ILLC in Amsterdam(2012), the Sorbonne (2012, 2013), Ludwig-Maximiliens Univerity in Munich (2012), the University of Copenhagen (2012),
The Perimeter Institute for Theoretical Physics (2010), the
CSLI at Stanford (2012), and the Argentinian Society for
Philosophical Analysis (2012).
We say that a problem is stratified if for all Q, Q0 ∈ Q,
Q ∩ Q0 6= ∅ entails Q  Q0 . Most natural problems satisfy
this condition. In stratified problems, patience is equivalent
to the requirement that every conjecture is co-initial in thesimplicity order over the answers. For a discussion of naturalproblems see [23].
9

PROOFS AND LEMMAS

Proof of theorem 1. ⇒: Suppose that A = O ∩ C for
O, C c open. Then:∨

A = O ∩ C ∩ (O ∩ C)c = O ∩ C ∩ Oc ∪ O ∩ C ∩ C c= O ∩ C ∩ Oc ,∨C c ⊆ O ∩ C ∩ C c = O ∩ C ∩ C c = ∅. ⇐: Suppose that A∨

is closed. Since for every A, A = A \ A, we have that A is adifference of closed sets, and is, therefore, locally closed.
Proof of theorem 2. To see that (1) implies (2), suppose that λ solves P in the limit. For each w, choose
Ew ∈ Lock(λ, w). Then {Ew : w ∈ W } is a (countable)S
cover of W by locking information. Let Fw = {E ∈ I :E ⊂ Ew and λ(E) 6= Q(w)}.
Then Ew \ Fw is locally closed.
S
We claim that A = w∈A Ew \ Fw for each A ∈ Q. Letw ∈ A. Then w ∈ Ew , andSw ∈/ Fw , since Ew is locking
for w.
S So w ∈ Ew \ Fw ⊆ w∈A Ew \ Fw . Suppose thatv ∈ w∈A Ew \ Fw . Then, for some w ∈ A, v ∈ Ew \ Fw .
Suppose that Q(v) = B 6= A. Then there is Ev ∈ Lock(λ, v),
and λ(Ev ∩ Ew ) = B 6= A. But then v ∈ Fw . Contradiction.
We have shown that each A ∈ Q is a countable union of
locally closed propositions.
S
To see that (2) implies (3), suppose that A = i∈N Li , where
Li is locally closed, for each A ∈ Q. It is a standard factthat the constructible propositions are closed underfiniteS
union and complementation. Letting Ci = Li \ j<i Lj wehave that each A is a disjoint union of constructible propositions Ci . Furthermore, by a result from [2], proposition
P is constructible iff there is a least integer n such that P
admits a decomposition into n disjoint, locally closed sets.
Therefore, each Ci is a disjoint union of finitely many locallyclosed propositions. So A is a countable, disjoint union of
locally closed sets.10
To see that (3) implies (1), suppose that Q is locallyclosed. Enumerate the elements of Q as A1 , A2 , . . . . By Theorem 1, each Ai can be written canonically as a difference∨cof open sets (Ai )c \ Ai . Define: min E ⊆ (A∨ )c and E * A ciiλ(E) = Ai ∈Q
Q(E)∨if defined;
otherwise.
c

Let w ∈ Ai . For j < i, either w ∈/ (Aj )c , or w ∈ Aj . Let:E =\j<i and w∈Aj

c

c

∨

Aj ∩ (Ai )c .
∨Let F ∈ I(w) such that F ⊆ E. Then F ⊆ E ⊆ (Ai )c and,csince F ∈ I(w), we have that F * Ai . Furthermore, since∨c

F ∈ I(w), it follows that F * (Aj )c , or F ⊆ Aj , for all
j < i. So λ(F ) = Q(Ai ) = Q(w). So E ∈ Lock(λ, w), as
required.
10

The proof strategy for this step was suggested by Alexandru Baltag in personal communication.

116

Proof of theorem 3. Recall the solution given in the
proof of theorem 2. Let E ∈ I be non-empty. Suppose that
λ(E) = Q(E). Then E ∩ λ(E) = E ∩ Q(E) = E 6= ∅.
Suppose that λ(E) = Q(Eσ(E) \ Fσ(E) ). Then E ⊆ Eσ(E)and E 6⊆ Fσ(E) , so E ∩λ(E) = E ∩Q(Eσ(E) \Fσ(E) ) 6= ∅.
Proof of theorem 4. The result follows immediatelyfrom theorem 2 and the fact that a set is locally closed in
the subspace topology I|C iff it is the intersection of C witha locally closed set.
Lemma 1. For arbitrary B ⊆ W,
An |B ∩ An−1 |B ∩ . . . ∩ A1 |B ∩ A0 |B⊆ An |B ∩ An−1 ∩ . . . ∩ A1 ∩ A0 ,where, for convenience, we have taken A|B as shorthand for
A ∩ B.
Proof of Lemma 1. In the base case n = 1. We have
that: A1 |B ⊆ A1 |B . By the induction hypothesis, we havethat:

An+1 |B ∩ An |B ∩ An−1 |B ∩ . . . ∩ A1 |B ∩ A0 |B⊆ An+1 |B ∩ An |B ∩ An−1 ∩ . . . ∩ A1 ∩ A0
where in each step we have used the fact that if B ⊆ C, thenA ∩ B ⊆ A ∩ C.
Proof of theorem 5. ⇐: Suppose that λ is a solution
to P, and A0 ∩ A1 ∩ . . . ∩ An−1 ∩ An 6= ∅. Let:

wi+1

∈ A0 ∩ A1 ∩ . . . ∩ An−1 ∩ An ;\∈Ej ∩ Ai+1 ∩ Ai+2 ∩ . . . ∩ An−1 ∩ An ;j≤iwhere Ej ∈ Lock(λ, wj ). Letting e = (that λ(e) ≤ a.

Proof of theorem 6. ⇐: Suppose that λ performs acycle. Then there is a nested information sequence e =(Ei )ni=0 such that λ(En ) ⊆ λ(E0 ) and m such that 1 < m <n, and λ(Em ) ⊆ λ(E0 )c . By consistency ∅ 6= λ(En ) ∩ En ⊆λ(En ) ∩ Em ⊆ λ(E0 ) ∩ Q(Em ) = λ(E0 ) ∩ Q(E0 ∩ Em ). But
λ(E0 ) ∩ λ(E0 ∩ Em ) = ∅.
⇒: Suppose that λ is not reversal monotone. Then there
are E, F ∈ I such that λ(E ∩ F ) is disjoint from λ(E), although λ(E) ∩ Q(E ∩ F ) 6= ∅. Let w ∈ E ∩ F such that
Q(w) ⊆ λ(E) ∩ Q(E ∩ F ) and G ∈ Lock(λ, w). Thenλ(E ∩ F ∩ G) = Q(w) ⊆ λ(E) and λ performs a cycle.
Proof of theorem 7. This is a direct consequence of
Proposition 2 and Theorem 17 in [4].
Proof of theorem 8. ⇒: Suppose that λ is a solution
to P, and for some A ∈ Q, there is w ∈ A ∩ ∪A⊥ . Let
E ∈ Lock(λ, w). Then there is v ∈ E, and P = A1 ∩. . . ∩ An−1 ∩ An , such that v ∈ P and P ∈ A⊥ . Let F ∈Lock(λ, v). Then since P is forcible in P|E∩F by theorem 5,
λ performs the reversal (A, A1 , . . . , An ), but by assumption,⊆ An+1 |B ∩ An ∩ An−1 ∩ . . . ∩ A1 ∩ A0 ,w0Since C is refutable, it is possible to focus on the P|C subproblem, until C is refuted. Since, by assumption, A0 ∩ C =∅, the solution λ1 never conjectures A0 . Therefore, λ1
cannot perform the reversal sequence (A0 , A1 , . . . , An+1 ).
Furthermore, since λ2 never performs the reversal sequence(A1 , ..., An+1 ), we have that λ∗ solves P, and never performs
the reversal sequence (A0 , A1 , . . . , An+1 ).

Tnj≤i Ej )i=0 , we have⇒: Suppose that P is solvable. We proceed by induction
on n. Base case: n = 0. Suppose that A0 = ∅. By theorem 3, there is a solution λ such that λ(E) 6= ∅ for all
nonempty E ∈ I. So a = (A0 ) is not forcible. For the inductive case, suppose that A0 ∩ A1 ∩ . . . ∩ An ∩ An+1 = ∅.

Let C = A1 ∩ . . . ∩ An ∩ An+1 . To make the subsequent
expressions manageable, write A|B for A ∩ B. By Lemma 1,
A1 |C c ∩ . . . ∩ An |C c ∩ An+1 |C c ⊆ A1 |C c ∩ . . . ∩ An ∩ An+1c

= C ∩ C = ∅.

So, by the induction hypothesis, (A1 , ..., An+1 ) is not forcible
in the P|C c subproblem. Let λ2 be a solution that neverperforms that reversal sequence in P|C c . Furthermore, let
λ1 be a solution to the P|C subproblem. The solutions toboth subproblems exist by theorem 4. Furthermore, we canassume that solution λ1 is consistent, by proposition 3. Now,
define the method:(Q(λ1 (E ∩ C)) if C ∩ E 6= ∅;∗λ (E) =Q(λ2 (E))
otherwise.
117

A ∩ A1 ∩ . . . ∩ An−1 ∩ An = ∅, so by theorem 5, that reversal sequence is not forcible.
⇐: Suppose that P is locally closed, and that for everyA ∈ Q, A ∩ ∪A⊥ = ∅. Enumerate the elements of Q:
A1 , A2 , . . . . Define Root(E) to be the first element in the∨cenumeration such that E ⊆ (A)c ∩ ∪A⊥ , if it exists, and ωotherwise.
(
Root(E) if Root(E) 6= ω,λ(E) =
Q(E)otherwise.
First we show that λ is a solution to P. Let w ∈ Ai , forsome Ai ∈ Q. As in the proof of theorem 2, there is E ∈c
I(w) that refutes all Aj such that j < i and w ∈ Aj .
∨Furthermore, for j < i such that w ∈ Aj , clearly E * (Aj )c ,∨csince E ∈ I(w). So E ∩ (Ai )c ∩ ∪Ai⊥ is locking for λin w. We proceed to show that λ is reversal optimal, by
induction on the length of reversal sequences. Base case:n = 1. Since λ is a solution, any singleton reversal sequence(A0 ) is forcible. For the inductive step, suppose that λ(e) =(λ(Ei ))ni=0 is a reversal sequence. WLOG, each λ(Ei ) ∈ Q.
By hypothesis (λ(Ei ))ni=1 is forcible. But by construction
λ(E0 ) ∩ λ(E1 ) ∩ . . . ∩ λ(En ) 6= ∅. So by Theorem 5, λ(e) is
forcible.
Proof of theorem 9. Let A, B, C ∈ Q, A ≺ B, andB ≺ C. Thus, A ⊆ B ⊆ C. So it remains only to show that
A 6= C. Each proposition is disjoint from its frontier, so∨∨

B ⊆ (B)c . Furthermore, if B locally closed (B)c is open, by∨∨

theorem 1. So, since A ⊆ B, (B)c is an open set containing
B and disjoint from A. Therefore B ⊀ A, and A 6= C as
required.

Proof of theorem 10. Suppose that λ is not Ockham.
Then for some E ∈ I there is w ∈ λ(E)≺ ∩ E. Let F ∈Lock(λ, w), then λ(E ∩ F ) = Q(w) is disjoint from λ(E),although λ(E) ⊆ Q(E ∩F ), since w is in the frontier of λ(E).
Therefore λ is not reversal monotone. So by Theorem 6, λis not cycle free.

[15] E. M. Gold. Language identification in the limit.
Information and control, 10(5):447–474, 1967.
[16] C. Hempel. Valuation and objectivity in science(1983). reprinted in Fetzer, j. (ed.): The philosophy ofCarl G, 2001, 1983.
[17] H. Jeffreys. Theory of probability, 1961.
[18] K. T. Kelly. The Logic of Reliable Inquiry. Oxford
Proof of theorem 12. Suppose that λ is not patient.
University Press, 1996.
Then for some E ∈ I there is Q ⊆ Q(E) such that for all[19] K. T. Kelly. Justification as truth-finding efficiency:
Q0 ⊆ λ(E), Q0 ∩ Q = ∅. Therefore λ(E) ∩ Q = ∅. Let
How Ockham’s razor works. Minds and Machines,w ∈ E ∩ Q and F ∈ Lock(λ, w). Then λ(E ∩ F ) = Q
14(4):485–505, 2004.
is disjoint from λ(E). So (λ(E), λ(E ∩ F )) is a λ-reversal[20] K. T. Kelly. A topological theory of learning andsequence, but λ(E) ∩ λ(E ∩ F ) = ∅. So by Theorem 5, that
simplicity. Manuscript, 2005.
sequence is not forcible, and λ is not reversal optimal.
[21]
K.
T. Kelly. Ockham’s razor, empirical complexity,
Proof of theorem 13. It suffices to notice that the method
and truth-finding efficiency. Theoretical Computer
constructed in the proof of theorem 8 always conjectures an
Science, 383(2):270–289, 2007.
answer closed in the subspace of current information.
[22] K. T. Kelly. Simplicity, truth and probability. In P. S.
Bandyopadhyay and M. Forster, editors, Handbook of
9. REFERENCES
the Philosophy of Science Volume 7: Philosophy of[1] H. Akaike. A new look at the statistical model
Statistics. North Holland, 2011.
identification. Automatic Control, IEEE Transactions[23] K. T. Kelly, K. Genin, and H. Lin. Realism, rhetoric,on, 19(6):716–723, 1974.
and reliability. Synthese, (forthcoming) 2014.
[2] J. Allouche. Note on the constructible sets of a[24] L. Laudan. Science and Values, volume 87. Cambridge
topological space. Annals of the New York Academy of
Univ Press, 1984.
Sciences, 806(1):1–10, 1996.
[25] L. Laudan. The Epistemic, the Cognitive, and the[3] A. Baker. Simplicity. In E. N. Zalta., editor, The
Social, pages 14–23. 2004.
Stanford Encyclopedia of Philosophy. Fall 2013 edition,[26]
H.
Leeb and B. M. Pötscher. Sparse estimators and2013.
the oracle property, or the return of hodges?
[4] A. Baltag, N. Gierasimczuk, and S. Smets. On theestimator. Journal of Econometrics, 142(1):201–211,
solvability of inductive problems: a study in epistemic
2008.
topology (forthcoming). In Proceedings of the fifteenth[27]
H.
Lin and K. T. Kelly. Propositional reasoning that
conference on Theoretical Aspects of Rationality andtracks probabilistic peasoning. Journal of philosophicalKnowledge, 2015.
logic, 41(6):957–981, 2012.
[5] P. S. Bandyopadhayay, R. J. Boik, and P. Basu. The[28]
W.
Luo and O. Schulte. Mind change efficient learning.
curve fitting problem: A Bayesian approach.
Information and Computation, 204(6):989–1011, 2006.
Philosophy of Science, pages S264–S272, 1996.
[29] M. Morrison. Unifying Scientific Theories: Physical[6] L. Carlucci and J. Case. On the necessity of U-Shaped
Concepts and Mathematical Structures. Cambridgelearning. Topics in Cognitive Science, 5(1):56–88,
University Press, 2007.
2013.
[30]
W.
C. Myrvold. A bayesian account of the virtue of[7] L. Carlucci, J. Case, S. Jain, and F. Stephan. Nonunification.
Philosophy of Science, 70(2):399–423,
U-shaped vacillatory and team learning. In
2003.
Algorithmic Learning Theory, pages 241–255. Springer[31] D. N. Osherson, M. Stob, and S. Weinstein. Systems
Berlin Heidelberg, 2005.
that Learn: An Introduction to Learning Theory for[8] R. Carnap. On inductive logic. Philosophy of Science,
Cognitive and Computer Scientists. The MIT Press,
12(2):72, 1945.
1986.
[9] J. Case and C. Smith. Comparison of identification[32]
K.
R. Popper. The Logic of Scientific Discovery.
criteria for machine inductive inference. TheoreticalLondon:Hutchinson, 1959.
Computer Science, 25(2):193–220, 1983.
[33]
H.
Putnam.
Trial and error predicates and the[10] M. de Brecht and A. Yamamoto. Interpreting learners
solution to a problem of mostowski. Journal ofas realizers for Σ02 -measurable functions. (Manuscript),
Symbolic logic, pages 49–57, 1965.
2009.
[34]
H.
Rott. Two dogmas of belief revision. The Journal[11] P. Dirac. The relation between mathematics andof Philosophy, pages 503–522, 2000.
physics. In Proc. Roy. Soc. Edinburgh, volume 59,[35] G. Schurz. Abductive belief revision in science. In
page 122, 1939.
Belief Revision Meets Philosophy of Science, pages[12] H. Douglas. Inductive risk and values in science.
77–104. Springer, 2011.
Philosophy of Science, pages 559–579, 2000.
[36]
A.
Sharma, F. Stephan, and Y. Ventsov. Generalized[13] M. Forster and E. Sober. How to tell when simpler,
notionsof mind change complexity. In Proceedings ofmore unified, or less ad hoc theories will provide more
the tenth annual conference on Computational
accurate predictions. The British Journal for thelearning theory, pages 96–108. ACM, 1997.
Philosophy of Science, 45(1):1–35, 1994.
[37]
A.
M. Turing. On computable numbers, with an[14] P. Gärdenfors. Knowledge in Flux. MIT Press,
application
to the entscheidungsproblem. J. of Math,
Cambridge, Mass, 1988.

118

58:345–363, 1936.
[38] V. Vapnik. Statistical Learning Theory, volume 2.
Wiley New York, 1998.
[39] L. Wasserman. Bayesian model selection and model
averaging. Journal of Mathematical Psychology,
44(1):92–107, 2000.

119

Do players reason by forward induction
in dynamic perfect information games?
[Extended Abstract]
Sujata Ghosh

Indian Statistical InstituteChennai, India

sujata@isichennai.res.in

Aviad Heifetz

The Open University of Israel
Raanana, Israel

aviadhe@openu.ac.il

ABSTRACT
We conducted an experiment where participants played aperfect-information game against a computer, which wasprogrammed to deviate often from its backward induction
strategy right at the beginning of the game. Participants
knew that in each game, the computer was nevertheless optimizing against some belief about the participant’s future
strategy.
It turned out that in the aggregate, participants werelikely to respond in a way which is optimal with respect
to their best-rationalization extensive form rationalizability
conjecture - namely the conjecture that the computer is aftera larger prize than the one it has foregone, even when thisnecessarily meant that the computer has attributed future
irrationality to the participant when the computer made thefirst move in the game. Thus, it appeared that participants
applied forward induction. However, there exist alternative
explanations for the choices of most participants; for example, choices could be based on the extent of risk aversion thatparticipants attributed to the computer in the remainder ofthe game, rather than to the sunk outside option that thecomputer has already foregone at the beginning of the game.
For this reason, the results of the experiment do not yet provide conclusive evidence for Forward Induction reasoning on
the part of the participants.

Categories and Subject Descriptors
Applied computing [Law, Social and Behavioural Sciences]: Economics

Keywords
game theory, experiment, forward induction

1. INTRODUCTION
Backward Induction (BI) is a canonical approach for solving extensive-form games with perfect information. In genericgames with no payoff ties, BI yields the unique subgame
perfect equilibrium [13, 22]. Nevertheless, BI embodies aconceptual difficulty: in subgames following a deviation ofsome player (or players) from their BI strategy, it is not
obvious why players should necessarily believe that the deviators will ‘return to their senses’ and realign their behavior
in the subgame with the BI dictum. Because such certainty
is absent, BI might itself be suboptimal for players who areskeptical about such re-adherence to rationality. Thus, the121

Rineke Verbrugge

University of Groningen
Groningen, The Netherlands

rineke@ai.rug.nl

epistemic assumptions underpinning BI are those of relentlessly reborn optimism (see e.g. the surveys in [21] and [23]),
with all players, under all contingencies, commonly believingin everybody’s future rationality, no matter how irrational
players’ past behavior has already proven: “after all, tomorrow is another day!”An alternative, more sober approach on the part of aplayer may be to employ Forward Induction (FI) reasoning, and to try to rationalize her opponent’s past behavior
in order to assess his future moves. For example, even ina subgame where there exists no strategy of the opponent
which is consistent with common knowledge of rationality
and his past behavior, she may still be able to rationalize
his past behavior by attributing to him a strategy which is
optimal as against a presumed suboptimal strategy of hers.
Or, even better, it may sometimes be possible for her to attribute to him a strategy which is optimal with respect to arational strategy of hers, which is, though, in return only optimal as against a suboptimal strategy of his. If the player
pursues this rationalizing reasoning to the highest extent
possible [3] and reacts accordingly, she will end up choosing
an Extensive-Form Rationalizable (EFR) strategy [4, 20].
EFR strategies may thus be distinct from BI strategies,
as an example by Reny [24] shows (see game 1 in Figure 1).
Given this difference, it is therefore a completely non-trivial
result that in perfect information games with no relevantties,1 there is nevertheless a unique EFR outcome, which coincides with the unique BI outcome [4,7,8,14,22]. Only whenrelevant payoff ties are allowed, an outcome-discrepancy between the two solution concepts may appear. In such cases,
the EFR outcomes constitute a subset of the BI outcomes[7, 8, 22], and the inclusion may be strict, as demonstrated
by Chen and Micali [8] (see game 3 in Figure 1).
We note here that experimental studies in behavioral economics have shown that the backward induction outcome
is often not reached in large centipede games. Instead ofimmediately taking the ‘down’ option, people often show
partial cooperation, moving right for several moves before
eventually choosing ‘down’ [6, 16, 19]. Nagel and Tang [19]suggest that people sometimes have reason to believe that
their opponent could be an altruist who usually cooperates
by moving to the right and McKelvey and Palfrey [16] suggest that players may believe that there is some possibility
that their opponent has payoffs different from the ones theexperimenter tries to induce by the design of the game.
1

That is, where each player has a strict ranking over all thegame-tree leaves following each of her decision nodes.

C

b

P

d

C

f

P

h

C

b

P

d

C

f

P

h

a

c

a
(3,1)

c
(1,2)

(3,1)

(1,2)

e

g

(4,1)

C

b

(4,1)

a

C

b

P

d

P

d

c

e
(2,0)

g
(0,3)

a
(2,1)

c
(1,2)

(2,0)

(0,3)

(2,1)

(1,2)

Game 1

b

C

b

a

P

d

P

d

c

a
(3,1)

c
(0,3)

(3,1)

(0,3)

C

f

C

f

e

C

f

e

P

h

P

h

g

e
(3,0)

g
(0,3)

(3,0)

(0,3)

(4,1)

(4,1)

Game 2

P

h

P

h

g

(4,4)

(4,4)

C

b

C

b

a

P

d

P

d

c

e
(2,2)

g
(1,4)

a
(2,1)

c
(0,3)

(2,2)

(1,4)

(2,1)

(0,3)

Game 3f

Game 2

Game 1

C

C

C

f

C

f

e

P

h

P

h

g

e
(3,2)

g
(1,4)

(3,2)

(1,4)

Game 4

(4,4)

(4,4)

Figure 1: Collection of the main games used in the experiment. The ordered pairs at the leaves represent
Game 4Game
pay-offs for the computer(C)3and the participant (P ), respectively.
We could also ask the following question: Are people inclined to use forward induction when they play a game, andin particular in games like those of Reny or Chen-Micali
mentioned above? This question was the motivation for theexperiment on which we report here. Our pivotal interest
was to examine participants’ behavior following a deviationfrom BI behavior by their opponent right at the beginningof the game.

1.1

Designing an experiment about forward
induction behavior

When designing an experiment to tackle the question whether people are inclined to use FI when they play dynamic
perfect-information games, such as the Reny or Chen-Micali
games mentioned above, a first challenge was to neutralize
repeated-game effects across repetitions of the same game.
Such visible repetitions could enable a folk-theorem styleaugmented cooperation level among participants playing one
against the other, bypassing their cooperation opportunities
in a one-shot play of the same game (cf. [9]). We chose toaddress this challenge by letting participants, adult university students with little or no knowledge of game theory,knowingly play against a computer.2
2
Another important advantage of using computer opponents
in experiments with dynamic games is that the experimentercan control the strategies used by the computer opponent,
which allows better interpretation of the participants’ decisions. Using a computer opponent also has disadvantages,
for example, players might reason quite differently about
their opponent if they know they are playing against ahuman player. Interestingly, Hedden and Zhang [12] misinformed a part of their subjects that they were playing
against a human opponent while in fact everyone was playing against a computer. They found little difference between
the decisions of these groups, and only around 10 % of par-We programmed the computer so as to follow, in each
repetition of each game, a strategy which is optimal withrespect to some strategy of the human participant. This
strategy for the computer was decided in advance for eachround, so that the computer did not learn from experience inprevious games. This information was honestly and simply
conveyed to the participants at the beginning of the experiment, by the following item on the instruction sheet (seeAppendix A):
How does the computer reason in each particular
game of the experiment?
- The computer thinks that you already have aplan for that game, and it plays the best responseto the plan it thinks that you have for that game.
- However, the computer does not learn from previous games and does not take into account yourchoices during the previous games.
Given that the participants were playing against a computer, a second challenge was to create variability in theappearance of repetitions of the same game, so that each
repetition looks different and forces the participant to thinkanew about her or his strategy in the current repetition.
This was achieved by two measures:- repeating in each round a set of 6 games, distinct interms of pay-off structures (see more on the games in
Section 2); and- presenting the game to the participant in a different
graphical fashion in each round in which the gameticipants expressed a suspicion on an exit questionnaire thatthey were playing against a computer rather than a person.

122

P

d

C

f

P

h

(4,1)

P

d

C

f

P

c

e

g

c

e

g

(1,2)

(2,0)

(0,3)

(0,3)

(2,2)

(1,4)

Game 1’h(4,4)

Game 3’Figure 2: Truncated versions of Game 1 and Game 3. The ordered pairs at the leaves represent pay-offs for
C and P , respectively.
was repeated. The game play was animated, and proceeded by consecutive dropping of a marble throughtrapdoors controlled alternately by the players, leading ultimately to one of several possible bins with orange marbles for the participant and blue marbles forthe computer.3 In repetitions of the same extensiveform game, changes were made in right/left directionsof trapdoors in junctions on paths leading to each bin(see, for example, screenshots of different representations of the same game in Appendix C, Figure 6).
In the earlier experiments that investigated FI reasoning in human participants, the experimental games mostly
considered an outside option together with some form ofimperfect information games, see, e.g., [2, 5, 15, 26]. Suchgames are more complicated in nature than the dynamical
games of perfect information we consider here. The novelty of the current experiment lies in its simplicity, using
perfect-information games only.

2. EXPERIMENTAL GAMESThe list of games that were used in the experiment is given
in Figure 1 and Figure 2. In these two-player games, theplayers play alternately. Let C denote the computer, and
let P denote the participant. In the first four games (Figure1), the computer plays first, followed by the participant, andeach of the players can play at two decision nodes. In the lasttwo games (Figure 2), which are truncated versions of twoof the games in Figure 1, the participant gets first chance tomove. We will now discuss the BI and EFR (FI) strategiesof all the 6 games; these are summarized in Table 1.

2.1

BI and EFR strategies in four main games

Game 1 has been introduced by Reny [24]. Here, theunique Backward Induction (BI) strategies for player C andplayer P are a; e and c; g, respectively. In case the last decision node of the game is reached, player P will play g(which will give P better payoff at that node) yielding 0 for
C. Thus, in the previous node, if reached, C will play e
to be better off. Continuing like this from the end to thestart of the game (by BI reasoning) it can be inferred that
whoever is the current player will play so as to end the gameimmediately.
3

The game presentation was inspired by Ben Meijering’s‘marble drop’ games, also used in [10, 17, 18].

123

Forward induction, in contrast, would proceed as follows.
Among the two strategies of player C which are compatible
with reaching the first decision node of player P , namelyb; e and b; f , only the latter is rational for player C. Thisis because of the fact that b; e is dominated by a; e, whileb; f is optimal for player C if she believes that player P
will play d; h with a high enough probability. Attributing
to player C the strategy b; f is thus player P ’s best way torationalize player C’s choice of b, and in reply, d; g is player
P ’s best response to b; f . Thus, the unique Extensive-Form
Rationalizable (EFR) strategy of player P is d; g, which isdistinct from her BI strategy c; g. Nevertheless, player C’s
best response to d; g is a; e, which is therefore player C’s EFR
strategy. Hence the EFR outcome of the game (with the
EFR strategies a; e, d; g) is identical to the BI outcome. Thisis an instance of the general theorem [1, 4, 8, 14] mentioned
in the Introduction, by which in perfect-information gameswith no relevant payoff ties, the unique BI outcome coincides
with the unique EFR outcome (even when, as in this game,for some player the EFR strategy is different from the BI
strategy).
Game 2 is popularly known as the Centipede game [25].
Here, the structure of the tree is as in the Reny game (cf.
Figure 1, game 1), but the payoffs of player C following aand e are interchanged. As in game 1, the unique Backward
Induction (BI) strategies of player C and player P are alsoa; e and c; g, respectively (and the BI outcome is the leaf
following a). However, when considering FI reasoning forgame 2, unlike in game 1, there does exist a belief of player
C with respect to which b; e is optimal. This is the belief
that player P is playing with high probability the strategyd; g, a strategy that is actually optimal for player P if P
believes that C is playing with high probability b; f , which
in turn is optimal for player C if C believes that player P is
playing with high probability d; h and is thus irrational onlyat the last decision node. For this reason, in game 2 it turns
out that a; e and c; g are the unique EFR strategies of thecorresponding players, and hence coincide with their unique
BI strategies.
Game 3 has been introduced by Chen and Micali [8]. Notethat player P has identical payoffs at both leaves following
her second and final decision node. As a result, there are
two ways to fold the game backwards, and every action of
every player at each decision node is a Backward Induction(BI) choice. Consequently, all possible outcomes of the gameare BI outcomes. However, the strategy b; e of player C
is dominated by its strategy a; e; and thus b; e is not an
Extensive-Form Rationalizable (EFR) strategy for player C.
In contrast, b; f is optimal for player C under the belief that
player P will pursue d; h with a high enough probability.
Hence, if player P finds herself in her first decision node,
her best way to rationalize player C’s first move of b is toattribute to it the strategy b; f , to which only d; g and d; hare best replies. Therefore, the path b; c with the eventual
payoffs (0, 3) for players C and P , respectively, is not an
EFR outcome of the game. This is an instance of the general
result by [7, 8] mentioned in the Introduction, by which theset of EFR outcomes is a (possibly proper) subset of the setof BI outcomes.
Finally, in game 4, the structure of the tree is as in theChen and Micali game (cf. Figure 1, game 3), but the payoffs
of player C following a and e are interchanged with respect
to game 3. Here too, every action of every player at eachdecision node is a BI choice, and hence all possible outcomesof the game are BI outcomes. However, in this case, eachof the three strategies that player C has – namely strategy
a, strategy b; e and strategy b; f – is a best reply to someconjecture about player P ’s strategy. Similarly to the game2 scenario, b; e is a best reply to the conjecture that player P
is likely playing d; g, and b; f is a best reply to the conjecturethat player P is likely playing d; h. Thus, for player P , incase her first decision node is reached, both her choices cand d constitute rationalizable (EFR) choices. Hence, inthis case, all possible outcomes are EFR outcomes as well,identical to the BI outcomes.

2.2

BI and EFR strategies in truncated games

Game 10 is a truncated version of game 1, with player P
being the starting player. The BI strategy for player P in
this game is to play c. In case player P plays d and thefirst decision node of player C is reached, both BI and EFR
strategies for player C are the same − to play e. Thus the
EFR strategy for player P in this game is to play c, andthe BI and EFR outcomes coincide, as they should in finite
perfect-information games without relevant ties.
Game 30 is a truncated version of game 3, with player
P starting the game. Player P has identical payoffs at theleaves following her second decision node. Here again, every
action of every player at each decision node is a BI choice,and hence all possible outcomes of the game are BI outcomes. In case the first decision node of player C is reached,each of the two strategies that player C has – namely strategy e and strategy f – is a best reply to some conjecture
about player P ’s strategy. Strategy e is a best reply to theconjecture that player P is likely playing d; g, and f is abest reply to the conjecture that player P is likely playingd; h. Thus, for player C, in case its second decision node isreached, both its choices e and f constitute rationalizable(EFR) choices. Hence, in this case also, all possible outcomes are EFR outcomes as well, identical to the BI outcomes, in contrast to what happens in game 3.

3. EXPERIMENTAL PROCEDURE
The experiment was conducted at the Institute of Artificial Intelligence (ALICE) at the University of Groningen,
The Netherlands. A group of 50 Bachelor’s and Master’s students from different disciplines at the university took part inthis experiment. The participants had little or no knowledge

Games | Strategies

BI strategy

EFR strategy

Game 1

C: a; e
P: c; g

C: a; eP: d; gGame 2

C: a; e

C: a; e

P: c; g

P: c; g

C: a; e, b; e, a; f, b; f

C: a; e, a; f, b; f

P: c; g, d; g, c; h, d; h

P: d; g, d; hGame 4

C: a; e, b; e, a; f, b; f
P: c; g, d; g, c; h, d; h

C: a; e, b; e, a; f, b; f
P: c; g, d; g, c; h, d; hGame 10

C: e

C: e

P: c; g

P: c; g

C: e, f

C: e, f

P: c; g, d; g, c; h, d; h

P: c; g, d; g, c; h, d; hGame 3

Game 30

Table 1: BI and EFR (FI) strategies for the 6 experimental games in Figures 1 and 2of game theory, so as to ensure that neither backward induction nor forward induction reasoning was already known tothem. The participants played the finite perfect-information
games in a graphical interface on the computer screen (cf.
Figure 3). In each case, the opponent was the computer,which had been programmed to play according to plans thatwere best responses to some plan of the participant. The
participants were instructed accordingly. In each game, amarble was about to drop, and both the participant and thecomputer determined its path by controlling the orange andthe blue trapdoors: The participant controlled the orange
trapdoors, and the computer controlled the blue trapdoors.
The participant’s goal was that the marble should drop into
the bin with as many orange marbles as possible. The computer’s goal was that the marble should drop into the binwith as many blue marbles as possible.
At first, 14 practice games were played (see Figure 5, Appendix C), which were simpler than the 6 games outlined in
Section 2. At the end of each practice game, the participant
could see how many marbles he or she had gained in thatgame, and also the total number of marbles gained so far.
These games were presented in increasing levels of difficultyin terms of the reasoning the participants needed to perform with respect to their and the opponent’s (computer’s)
choices, to maximize their gains.
The 14 practice games were followed by 48 experimental
games and the participants got access to similar information regarding the number of marbles gained. There were 8
rounds, each comprised of the 6 games that were describedin Section 2. Different graphical representations of the same
game were used in different rounds (cf. Figure 6, Appendix
C). A break of 5 minutes was given after the participants finished playing 4 rounds of the experimental games. The participants earned between 10 and 15 euros for participatingin the experiment. The amount depended on the number ofmarbles won during the experimental phase, and they weretold about this before the start of the experiment. They
earned 10 euros for participation, and each marble a participant won added 4 cents to the amount. The final amount
was rounded off to the nearest 5 cents mark.
At some points during the experimental phase, the participants were asked a multiple-choice question as follows:“When you made your initial choice, what did you think thecomputer was about to do next?” (cf. Figure 4). Three
options were given regarding the likely choice of the com-124

THE GAMEThe computer decides here.

You decide here.

The computer decides here.

You decide here.

Figure 3: Graphical interface for the participants.
The computer controls the blue trapdoors and acquires pay-offs in the form of blue marbles (represented as dark grey in a black and white print),
while the participant controls the orange trapdoors
and acquires pay-offs in the form of orange marbles(light grey in a black and white print).

Figure 4: Question on computer’s behavior
Step 1Introduction and instructions.

Step 2

Practice Phase: 14 games.

Step 3- Experimental Phase: 48 game items, divided into
8 rounds of 6 different games each, in terms of
isomorphism class of pay-off structures;

puter: “I thought the computer would most likely play left”
or “I thought the computer would most likely play right” or“neither of the above”. The first two answers translated tothe moves e or f of the computer, respectively. In case of
the third answer, we assumed that the participant was undecided regarding the computer’s next choice. The participants had been randomly divided into two groups: Group A
and Group B, each consisting of 25 persons. The membersof group A were asked the question about the computer’s
next possible move once they had played at their first decision node in each game in rounds 3, 4, 7, and 8, whereas themembers of group B were asked the same question but lessoften, namely only in each game in rounds 7 and 8.
At the end of the experiment, each participant was asked
the following question: “When you made your choices inthese games, what did you think about the ways the computer would move when it was about to play next?” The participant needed to describe the plan he or she thought was
followed by the computer on its next move after the participant’s initial choice, in his or her own words. In summary,during the experiment, the participants had to perform thetasks specified in Table 2 in the order given there.
During the 48 games of the experimental phase, playedby each participant, a varied amount of data were collected.
In particular, for each participant, for each game, for eachround of the game, we collected the following data:- Participant’s decision at his/her first decision node, if
the node was reached. In particular, whether move c

125- Each of the 6 games occurs once in each round;these games occur in the same order in each round;- Question on computer’s behavior (cf. Figure 4) inseveral rounds: Group A in rounds 3, 4, 7, 8; Group B
in rounds 7, 8.
Step 4

Final Question.

Table 2: Steps of the experimentor d had been played.4

4.

RESULTS AND ANALYSISAs mentioned above, we report and analyze only the behavior of the participants in their first decision node, that istheir choice between actions c or d whenever that decisionnode was reached. We found no significant variation (Proportion test, p = 0.21) between the behavior of the 25 participants of Group A, who were asked questions (cf. Figure4) after each game in rounds 3, 4, 7, and 8, and the 25 participants of Group B, who were asked those questions only4
In addition, we also took note of other aspects, such as
the participant’s behavior at the second decision node andtime taken by the participant at various stages. We leaveout the details, because these are not relevant for our main
research question, whether participants are applying forward
induction. See Appendix B for recorded data types, andsee [11] for a typology of players’s reasoning strategies basedon these richer data.

after the games in rounds 7 and 8. Therefore henceforth, wewill analyze the data of all 50 participants together.
The 6 graphs on the next page give the sequence of choices(across the repetitions of each game) at the first decisionnode, per participant (named A1 . . . A25, B1 . . . B25). The
dark grey color corresponds to the rounds the participant
played the move c, and the light grey color corresponds to
the rounds the participant played move d, whenever the participant’s first decision node was reached. They clearly showthat d was played more often in game 1 than in game 2(which has the same payoffs as game 1 except for C’s payoffs
interchanged at two leaves). Moreover, d was played more
often in game 3 than in game 4 (which similarly has thesame payoffs as game 3 except for C’s payoffs interchanged
at two leaves). These observations may initially suggest corroboration of FI reasoning because (as the reader can checkin Table 1), d is P ’s only EFR move in game 1 while c isthe only EFR move in game 2, and d is the only EFR movein game 3 while both c and d are EFR moves in game 4.
This would provide a positive answer to our research question whether players apply forward induction when playing
against a computer which sometimes deviates from rational
behavior.

Choices of c and d of in game 3
6

5

4

3

d
c

2

1

A1
A2
A3
A4
A5
A6
A7
A8
A9
A10
A11
A12
A13
A14
A15
A16
A17
A18
A19
A20
A21
A22
A23
A24
A25
B1
B2
B3
B4
B5
B6
B7
B8
B9
B10
B11
B12
B13
B14
B15
B16
B17
B18
B19
B20
B21
B22
B23
B24
B25

0

Choices of c and d of in game 4
6

5

4

3

d
c

2

1

A1
A2
A3
A4
A5
A6
A7
A8
A9
A10
A11
A12
A13
A14
A15
A16
A17
A18
A19
A20
A21
A22
A23
A24
A25
B1
B2
B3
B4
B5
B6
B7
B8
B9
B10
B11
B12
B13
B14
B15
B16
B17
B18
B19
B20
B21
B22
B23
B24
B25

0

Choices of c and d of in game 1'
8

7

Choices of c and d of in game 1

6

6
5

5

4

d
c

3
4
2
3

d
1

c
2

A1
A2
A3
A4
A5
A6
A7
A8
A9
A10
A11
A12
A13
A14
A15
A16
A17
A18
A19
A20
A21
A22
A23
A24
A25
B1
B2
B3
B4
B5
B6
B7
B8
B9
B10
B11
B12
B13
B14
B15
B16
B17
B18
B19
B20
B21
B22
B23
B24
B25

0

1

Choices of c and d of in game 3'
0
A1
A2
A3
A4
A5
A6
A7
A8
A9
A10
A11
A12
A13
A14
A15
A16
A17
A18
A19
A20
A21
A22
A23
A24
A25
B1
B2
B3
B4
B5
B6
B7
B8
B9
B10
B11
B12
B13
B14
B15
B16
B17
B18
B19
B20
B21
B22
B23
B24
B25

8

7

Choices of c and d of in game 2

6

6
5
5

4

d
c

3

4

2
3

d
1

c
2

A1
A2
A3
A4
A5
A6
A7
A8
A9
A10
A11
A12
A13
A14
A15
A16
A17
A18
A19
A20
A21
A22
A23
A24
A25
B1
B2
B3
B4
B5
B6
B7
B8
B9
B10
B11
B12
B13
B14
B15
B16
B17
B18
B19
B20
B21
B22
B23
B24
B25

0

1

A1
A2
A3
A4
A5
A6
A7
A8
A9
A10
A11
A12
A13
A14
A15
A16
A17
A18
A19
A20
A21
A22
A23
A24
A25
B1
B2
B3
B4
B5
B6
B7
B8
B9
B10
B11
B12
B13
B14
B15
B16
B17
B18
B19
B20
B21
B22
B23
B24
B25

0

However, a closer look at individual choices while also taking the truncated games 10 and 30 of Figure 2 into account,casts doubt that these findings can be attributed to any substantial FI reasoning. When comparing game 1 to game 10 ,EFR prescribes d in game 1 and c in game 10 (see Table1). However, only two participants out of 50 (4%) playedd much more often in game 1 than in game 10 ;5 four additional participants (8%) played d in game 1 only slightly
5
The verbal elaboration of one of the two participants at theend of the experiment is indeed compatible with EFR, see
Appendix D.

126

more often than in game 10 ; but 24 other participants (48%)
actually played d more often in 10 than in 1!
Similarly, when comparing game 3 to game 30 , EFR prescribes d in game 3, while in game 30 , both c and d arecompatible with EFR. However, only two participants outof 50 (4%) played d much more often in game 3 than ingame 30 ; ten additional participants (20%) played d in game
3 only slightly more often than in game 30 ; but 17 other participants (34%) actually played d more often in game 30 thanin game 3. In summary, comparing games 1 and 3 to theirtruncated versions does not lend support for FI reasoning.
Now, comparing game 3 to game 4, we find that 47 participants (94%) played d at least as often in game 3 as ingame 4, and the remaining three players (6%) played d only
slightly less often in game 3 than in game 4. As mentioned
at the beginning of this section, at first glance this may thensuggest support for EFR behavior (since EFR prescribes d
in game 3 and allows for both c and d in game 4). However,
because we did not see support for EFR when comparing
game 3 to game 30 , it could very well be that a cardinal
effect rather than an ordinal effect has played a role here:- In game 4, a participant’s playing d implied that thecomputer would have to choose between a payoff 3 thatit could reach for certain by going down, and a ‘lottery’ between the payoffs 1 and 4 that it would meet if
it would continue to the right to the next P -node, due
to the fact that at P ’s last decision node, the participant P gains the same payoff of 4 points after either
choice. Consequently, most participants might havefeared that the computer would go for the certain payoff 3, so preempted that by choosing c.
- In game 3, in contrast, a participant’s playing d implied that the computer would have to choose between
the relatively low payoff 2 that it could achieve forsure by going down, and again a ‘lottery’ between the
payoffs 1 and 4. Consequently, most participants may
have been betting that the computer would go for the‘lottery’, and hence chose d.6
Similarly, comparing game 1 to game 2, we find that 42 out
of 50 participants (84%) played d at least as often in game1 as in game 2. Here again, at first glance this may seemto lend support for EFR behavior, since EFR prescribes d
in game 1 and c in game 2. However, here too, a cardinal
effect may have played a role, as follows:- In game 2, a payoff of 3 may have seemed (in the eyes ofmost participants) to tempt the computer to go down
at its second decision point and settle for it for sure,rather than hoping that the participant would err at
the end by choosing h – an error which would yield onlya slightly better payoff of 4 to the computer, while C’s
pay-off would be only 0 if the participant did not err
at the end.
- In game 1, in contrast, participants may have attributed
a greater temptation to the computer to gamble for the
payoff 4 (which is what the computer would get if the6Some verbal comments at the end attributed to the computer a 50%-50% belief in this lottery and expected payoff
maximization, which is indeed consistent with choosing c ingame 4 and d in game 3.

127

participant were to err by choosing h) versus 1 (if theparticipant did not err); the participant would compare this ‘lottery’ with what C could settle for withcertainty by going down at its second decision point,
which is only 2.
These considerations may have led most players to choosed more often in game 1 than in game 2, irrespective of any
FI considerations.

5.

CONCLUSION

To the best of our knowledge, the experiment carried out
and reported here is the first experiment that has been
designed to test Forward Induction (FI) behavior (particularly, Extensive-Form Rationalizable (EFR) behavior) inextensive-form games with perfect information.
In the experiment, 50 participants played against a computer, which they knew to have been programmed so as not
to make deductions or learn from previous game rounds, butrather to optimize, in each round, against some belief about
the participant’s strategy. Moreover, different rounds of thesame game were interspersed in between different roundsof other games, and in different rounds of the same gamethe game tree was presented to the participants in distinct
interactive “marble-drop” forms. Thus, unlike in other experiments where each pair of participants plays repeatedly
many rounds of the same game, our design was structured
so as to neutralize, as much as possible, repeated-game cooperation considerations on the part of each participant.
In the aggregate, the participants were more likely to respond in a way which is optimal with respect to their bestrationalization EFR conjecture - namely the conjecture thatthe computer is after a larger prize than the one it has foregone, even when this necessarily meant that the computer
has attributed future irrationality to the participant when
the computer made the first move in the game. Thus, itappeared that participants did apply forward induction.
However, there exist alternative explanations for the choices of most participants, and such alternative explanations
also emerge from several of the participants’ free-text verbal descriptions of their considerations (cf. Appendix D),
as solicited from them at the end of the experiment. These
alternative considerations have to do with the extent of risk
aversion that participants attributed to the computer in theremainder of the game, rather than to the sunk outside option that the computer has already foregone at the beginningof the game. For this reason, the results of the experiment
do not yet provide conclusive evidence for Forward Induction reasoning on the part of the participants.
In current ongoing work, we are using data from this experiment, such as response times and answers to questions,in order to investigate how participants can be divided intomeaningful classes according to other cognitive considerations, for example, whether they are applying quick, instinctive thinking or contemplative, slower deliberation, whether
they are applying higher orders of theory of mind, and so on,see [11]. In future work, we aim to investigate which strategies participants actually apply in dynamic games with perfect information in which the opponent occasionally deviatesfrom backward induction. We plan to use new games withdifferent pay-off structures and will perform an eye-tracking
study to check the points in the games to which participants
attend while reasoning.

Acknowledgements
We would like to acknowledge research assistants Sourit Manna, Damian Podareanu, and Michiel van de Steeg, without
whose help this work would not have been possible. We
would also like to thank Harmen de Weerd for his contribution to the graphs in this article. Furthermore, we are
grateful to Shaul Tsionit for his statistical advice. Sujata
Ghosh and Rineke Verbrugge acknowledge the Netherlands
Organisation of Scientific Research for Vici grant NWO 22780-001 awarded to the latter.

Appendix A: Instruction sheet- In this task, you will be playing two-player games. The
computer is the other player.
- In each game, a marble is about to drop, and both you
and the computer determine its path by controlling theorange and the blue trapdoors.
- You control the orange trapdoors, and the computer
controls the blue trapdoors.
- Your goal is that the marble drops into the bin withas many orange marbles as possible. The computer’s
goal is that the marble drops into the bin with as many
blue marbles as possible.
- Click on the left trapdoor if you want the marble to go
left, and on the right trapdoor if you want the marble
to go right.

- The money you will earn is between 10 and 15 euros and depends on how many marbles you have won
during the experiment phase. You will get 10 euros
for participation, and each marble you win will add 4cents to your amount. The final amount will be given
to you rounded off to the nearest 5 cents mark.7

Appendix B: Recorded data types
As mentioned earlier, 50 students participated in this experiment. The participants were first requested to provide thefollowing information:
Name; Age; Gender; Field of study.
Then they were given instruction sheets mentioning what
they were supposed to do (see Appendix A) together witha representative figure (cf. Figure 3) of the graphical interface of the games they were supposed to play. Once they
got accustomed with what they were expected to do, theparticipants played the first 14 practice games. As mentioned in Section 3, at the end of each game, a participant
could see how many orange marbles he or she had won tillthat moment - this was to show how his/her winnings were
getting calculated. At the end of the practice phase, theexperimental phase began.
Here, each participant played 48 experimental games, playing each of the six games depicted in Figures 1 and 2, eighttimes, in different representations. During these 48 games,
played by each participant, a varied amount of data were collected. For each participant, for each game, for each roundof the game, we collected the following data:- How does the computer reason in each particular game?

- participant’s decision at his/her first decision node, if
the node was reached. In particular, whether move cor d had been played;

The computer thinks that you already havea plan for that game, and it plays the best
response to the plan it thinks that you havefor that game.

- participant’s decision at his/her second decision node,
if the node was reached. In particular, whether moveg or h had been played;

However, the computer does not learn fromprevious games and does not take into account your choices during the previous games.

- time taken by the participant in starting the game, i.e.
the time between the moment the game was shown tothe participant, and the moment he/she clicked on the“start” button;- The first 14 games are practice games. At the end ofeach practice game, you will see how many marbles
you gained in that game, and also the total number ofmarbles you have gained so far.

- time taken by the participant in making his/her decision at the first decision node, if the node was reached,i.e. the time between the moment the computer passed
the playing marble to the participant on its first decision node, and the moment he/she clicked on the next
trapdoor for the marble to be dropped;- The practice games are followed by 48 experiment games.
At the beginning of the experiment games, the total
number of marbles won will be set at 0 again. At theend of each experiment game, you will see how many
marbles you gained in that game, and also the total
number of marbles you have gained so far.

- time taken by the participant in making his/her decision at the second decision node, if the node wasreached, i.e. the time between the moment the computer passed the playing marble to the participant
on its second decision node, and the moment he/sheclicked on the next trapdoor for the marble to be dropped.

- You will be able to start each game by clicking on the“START GAME” button, and move to the next gameby clicking on the “NEXT” button.
- At some points during the experiment phase, you will
be asked a few questions regarding what guided yourchoices.
- There will be a break of 5 minutes once you finish 24of the 48 experiment games.

7

We chose the relatively large ‘show-up fee’ because Dutch
student participants tend to complain in case of large differences in pay between participants. However, most participants attained a fairly large award, so in future we aim toincentivise participants more by offering a lower show-up fee
and a higher fee per marble.

128

Level 1

Level 3

Level 2

Level 4

Figure 5: Levels of practice games
The first two items correspond to categorical or qualitative data, whereas the next three, which are response times
recorded in milliseconds, correspond to numerical or quantitative data. As mentioned in Section 3, the participants
were randomly divided into two groups. namely Group A
and Group B, where members of group A were asked to answer a question (cf. Section 3) in rounds 3, 4, 7, 8, and members of group B were asked to answer the same questions but
only in rounds 7 and 8. For each participant, depending on
the group (Group A or Group B), we collected the following
data:- participant’s answer to the given question (cf. Figure4) at the ends of the rounds in which it was asked.
In particular, whether the answer was e or f or undecided;- time taken by the participant in giving the answer,i.e. the time between the moment the question appeared on the screen and the moment he/she clickedon his/her choice of answer.
The first data item is categorical, whereas the second one,recorded in milliseconds, is numerical. Finally, at the end of
the experiment each participant was asked a final question(cf. Section 3), the answers to which were recorded in a
separate sheet. A limited amount of space was given inwhich the answer was to be formulated.

Appendix C: Experimental interface
During the training phase, the participants were given 14
training games of increasingly difficult levels in terms ofnumber of decision points, as explained in Section 3. Figure 5 shows example games for each of the four levels.
In each of the 8 rounds of the experimental phase, participants were confronted with all 6 games described in Section
2. Different graphical representations of the same game wereused in different rounds. As an example, Figure 6 shows six
visually different variations of game 1.

129

Figure 6: Experimental games, various representations of game 1 of Figure 1

Appendix D: Answers to the final question
As mentioned in Section 3, at the end of the experiment,each participant was asked the following question: “Whenyou made your choices in these games, what did you think
about the ways the computer would move when it was about
to play next?” The participant needed to describe the plan
he or she thought was followed by the computer on its next
move after the participant’s initial choice, in his or her own
words.
We found that one student who had made choices in thegame that were consistent with FI reasoning, also provided
an answer that suggested FI reasoning:- “I first thought it would try to maximize the outcomes,
taking into account that I would do the same. But I
noticed that it did not always do that. Sometimes it
did and sometimes it didn’t. So after the break, I tried
to maximize my outcomes, assuming the computer did
the same, but if I noticed that the computer was notassuming that I would maximize my outcomes, I took
a risk and I won a lot more.”
Here follows a selection of answers provided by the otherparticipants, which shows that participants might have givenmore importance to risk aversion and/or expected gains,rather than considering the outside option which the computer has already foregone.
- “I thought the computer took the option with the highest expected value. So if on one side you had a 4 blue
+ 1 blue marble and on the other side 2 blue marbles
he would take the option 4+1= 2.5.”- “It was going to take the turn with the highest reward,
considering the risk. For example, when the computer
can take a reward of 2 marbles instantly or choose to let
the ball roll to an orange gate which has the potentialof rewarding 4 marbles, the computer would go for theorange gate. With a difference of 1 marble between
choices the computer is most likely to take the easiest
way.”- “It would choose for the safe 2/3 marble option insteadof the dangerous 0/1 or 4 marble option.”- “I made my choices based on how many marbles I could
miss if the computer would turn left or right. In mostcases I made the safe choice.”- “My thoughts were about which most profitable routethe computer would take, by looking at how many marbles the computer would get in comparison to me. Ifthey were even or less then I think the computer would
play safely and take the best and safest option available at that point.”- “Look at the potential payoffs for blue in relation to thepotential payoff for orange and check for probabilities.”

6. REFERENCES[1] I. Arieli and R. Aumann. The logic of backward
induction. Available at SSRN:
http://ssrn.com/abstract=2133302 or
http://dx.doi.org/10.2139/ssrn.2133302, 2012.
[2] D. Balkenborg and R. Nagel. An experiment on
forward versus backward induction: How fairness andlevels of reasoning matter. Working paper, 2008.
[3] P. Battigalli. Strategic rationality orderings and thebest rationalizability principle. Games and Economic
Behavior, 13:178–200, 1996.
[4] P. Battigalli. On rationalizability in extensive games.
Journal of Economic Theory, 74:40–61, 1997.
[5] G. Cachon and C. Camerer. Loss-avoidance andforward induction in experimental coordination games.
The Quarterly Journal of Economics, 111(1):165–94,
1996.
[6] C. Camerer. Behavioral Game Theory. Princeton
University Press, 2003.
[7] J. Chen and S. Micali. The robustness ofextensive-form rationalizability. Working paper, 2011.
[8] J. Chen and S. Micali. The order independence of
iterated dominance in extensive games. Theoretical
Economics, 8:125–163, 2013.
[9] D. Fudenberg and E. Maskin. The folk theorem in
repeated games with discounting or with incomplete
information. Econometrica: Journal of the
Econometric Society, 54:533–554, 1986.

[10] S. Ghosh, B. Meijering, and R. Verbrugge. Strategic
reasoning: Building cognitive models from logicalformulas. Journal of Logic, Language and Information,23(1):1–29, 2014.
[11] T. Halder, K. Sharma, S. Ghosh, and R. Verbrugge.
How do adults reason about their opponent?
Typologies of players in a turn-taking game. In
Proceedings of the 37th Annual Conference of the
Cognitive Science Society, 2015.
[12] T. Hedden and J. Zhang. What do you think i think
you think?: Strategic reasoning in matrix games.
Cognition, 85(1):1–36, 2002.
[13] A. Heifetz. Game Theory: Interactive strategies in
Economics and Management. Cambridge University
Press, 2012.
[14] A. Heifetz and A. Perea. On the outcome equivalenceof backward induction and extensive form
rationalizability. International Journal of Game
Theory, 44:37–59, 2015.
[15] S. Huck and W. Müller. Burning money and (pseudo)
first-mover advantages: An experimental study on
forward induction. Games and Economic Behavior,
51(1):109–127, 2005.
[16] R. McKelvey and T. Palfrey. An experimental studyof the centipede game. Econometrica: Journal of the
Econometric Society, pages 803–836, 1992.
[17] B. Meijering, N. A. Taatgen, H. Van Rijn, and
R. Verbrugge. Modeling inference of mental states: As
simple as possible, as complex as necessary.
Interaction Studies, 15:455–477, 2014.
[18] B. Meijering, H. Van Rijn, N. Taatgen, and
R. Verbrugge. I do know what you think I think:Second-order theory of mind in strategic games is notthat difficult. In Proceedings of the 33rd Annual
Conference of the Cognitive Science Society, pages
2486–2491, 2011.
[19] R. Nagel and F. Tang. Experimental results on thecentipede game in normal form: An investigation onlearning. Journal of Mathematical Psychology,
42(2):356–384, 1998.
[20] D. Pearce. Rationalizable strategic behaviour and the
problem of perfection. Econometrica, 52:1029–1050,
1984.
[21] A. Perea. Epistemic foundations for backward
induction: An overview. In J. van Benthem,
D. Gabbay, and B. Löwe, editors, Interactive Logic,
volume 1 of Texts in Logic and Games, pages 159–193.
Amsterdam University Press, 2007.
[22] A. Perea. Epistemic Game Theory: Reasoning and
Choice. Cambridge University Press, 2012.
[23] A. Perea. Belief in the opponents’ future rationality.
Games and Economic Behavior, 83:231–254, 2014.
[24] P. Reny. Backward induction, normal form perfectionand explicable equilibria. Econometrica, 60:627–649,
1992.
[25] R. Rosenthal. Games of perfect information, predatory
pricing, and the chain store. Journal of Economic
Theory, 25(1):92–100, 1981.
[26] Q. Shahriar. An experimental test of the robustness
and the power of forward induction. Managerial andDecision Economics, 35(4):264–277, 2014.

130

Ceteris paribus logic in counterfactual reasoning
Patrick Girard

Marcus Anthony Triplett

The University of Auckland
Auckland, New Zealand

The University of Auckland
Auckland, New Zealand

ABSTRACTThe semantics for counterfactuals due to David Lewis has
been challenged on the basis of unlikely, or impossible, events.
Such events may skew a given similarity order in favour of
those possible worlds which exhibit them. By updating therelational structure of a model according to a ceteris paribus
clause one forces out, in a natural manner, those possible
worlds which do not satisfy the requirements of the clause.
We develop a ceteris paribus logic for counterfactual reasoning capable of performing such actions, and offer several
alternative (relaxed) interpretations of ceteris paribus. We
apply this framework in a way which allows us to reason
counterfactually without having our similarity order skewed
by unlikely events. This continues the investigation of formal
ceteris paribus reasoning, which has previously been applied
to preferences [22], logics of game forms [9], and questions
in decision-making [25], among other areas [16].

1. INTRODUCTION
The principal task of this paper is to work towards integrating ceteris paribus modalities into conditional logics sothat some dissonant analyses of counterfactuals may be reconciled. We also suggest that ceteris paribus clauses may beunderstood dynamically, in the sense of dynamic epistemic
logic [23], and we interpret our resulting ceteris paribus logic
accordingly. Ceteris paribus clauses implicitly qualify many
conditional statements that formulate laws of science andeconomics. A ceteris paribus clause adds to a statementa proviso requiring that other variables or states of affairs
not explicitly mentioned in the statement are kept constant,thus ruling out benign defeaters. For instance, Avogadro’s
law says that if the volume of some ideal gas increases then,everything else held equal, the number of moles of that gas
increases proportionally. Varying the temperature or pressure could provide situations that violate the plain statementof the law, but the ceteris paribus clause accounts for those.
It specifically isolates the interaction between volume andnumber of moles by keeping everything else equal. In thesame spirit, the Nash equilibrium in game theory is a solution concept that picks strategy profiles in which none of theagents could unilaterally (i.e., keeping the actions of others
constant, or equal) deviate to their own advantage.
We may understand a ceteris paribus clause as a linguistic
device intended to shrink the scope of the sentence qualified
by the clause. For instance, when I make the utterance “I
prefer fish to beef, ceteris paribus” I may mean something
different from if I simply uttered “I prefer fish to beef.” By
enforcing the ceteris paribus condition I rule out some situa-

131

tions which affect my preference. For example if, whenever I
eat fish I’m beaten with a mallet, while whenever I eat beef
I’m left in peace, I might retract the second utterance andmaintain the first. The ceteris paribus clause reduces thenumber of states of affairs under consideration. For modal
logicians, ‘ruling out’ states of affairs amounts to strengthening an accessibility relation, consequently changing the relational structure of a model. This bears similarity to the epistemological forcing of Vincent Hendricks [10], which seeks
to rule out ‘irrelevant alternatives’ in a way which allows
knowledge in spite of the possibility of error. Wesley Holliday [11] develops several interpretations of the epistemic
operator K based on the relevant alternatives epistemology;
namely, that in order for an agent to have knowledge of a
proposition, that agent must eliminate each relevant alternative. Holliday’s semantics are based on the semantics forcounterfactuals due to David Lewis [13], which we will recall
in the next section. One could see relevant worlds as thosewhich keep things equal. When reasoning using Avogadro’slaw, the relevant possible worlds are those where the temperature and pressure have not changed. Thus, in order foran agent to have knowledge, that agent must eliminate thealternatives among the worlds which ‘keep things equal.’Previously, ceteris paribus formalisms have been given for
logics of preference [22] and logics of game forms [9]. Here we
extend the analysis to counterfactual reasoning. The importance of counterfactuals in game theory is well known (see,for instance, [17]). For example, Bassel Tarbush [21] argues
that the Sure-Thing Principle 1 ought to be understood as aninherently counterfactual notion. We will motivate our discussion by thinking through Kit Fine’s well-known ‘minormiracles’ argument [8], a putative counterexample to Lewis’semantics. We will argue that ceteris paribus logic, suitably adapted to conditionals, provides a natural responseto this kind of argument. Moreover, we will see that ceteris paribus logic reveals a useful feature missing from thestandard formalisation of counterfactuals; namely, the explicit requirement that certain propositions must have theirtruth remain fixed during the evaluation of the counterfactual. This is implicitly thought to hold, to some degree,when one works with models which have similarity orders or
systems of spheres. The conditional logic of Graham Priest[15] makes just that assumption, but with no syntactic assurance. Ceteris paribus logic provides, in addition to the1
An outcome o of an action A is a sure-thing if, were anyother action A0 to be chosen, o would remain an outcome.
The Sure-Thing Principle [18] states that sure-things should
not affect an agent’s preferences.

underlying similarity order over possible worlds, a syntactic apparatus to reason with such ceteris paribus clauses
directly in the object language.

2.

COUNTERFACTUALS

Here we shall formalise counterfactuals in the style ofLewis. Let Prop be a set of propositional variables. We
are concerned with models of the form M = (W, , V ) suchthat the following obtain.
1. W is a non-empty set of possible worlds.
2.  is a family {w }w∈W of similarity orders, i.e., relations on Ww × Ww (with Ww ⊆ W ) such that:• w ∈ Ww ,• w is reflexive, transitive and total, and• w ≺w v for all v ∈ Ww \ {w}.

3. V is a valuation function assigning a subset V (p) ⊆ W
to each propositional variable p ∈ Prop.
Intuitively, Ww is the set of worlds which are entertainablefrom w. Worlds which are not entertainable from w are
deemed simply too dissimilar from w to be considered. Say
that u is at least as similar to w as v is when u w v, andthat it is strictly more similar when u ≺w v.
If M satisfies each of the three requirements we call M a
conditional model. A relation ≤ is said to be well-founded if
for every non-empty S ⊆ W the set
MinM≤ (S) = {v ∈ S ∩ W : there is no u with u < v}

(1)

is non-empty.2 We will suppress the superscript M if
it is clear from the context which model we’re discussing.
If a model M = (W, , V ) has only well-founded similarity
orders we say that M satisfies the limit assumption. For easeof exposition, we will assume that our conditional models
satisfy the limit assumption. Of course, we may generalise
the semantics for counterfactuals in the usual way [13], sothat our results work for models which do not satisfy thelimit assumption as well.

3.

p
s
m
h

Definition 1 (Language L ). The language L ofcounterfactuals is given by the following grammarF

We define ϕ ∧ ψ := ¬(¬ϕ ∨ ¬ψ), ϕ → ψ := ¬ϕ ∨ ψ, ϕ ψ := ¬(ϕ  ¬ψ).

Let w ∈ W . If w ∈ JϕK we write M, w |= ϕ, and if w 6∈ JϕK
we write M, w 6|= ϕ.
2

As usual, u < v is defined as u ≤ v and not v ≤ u

=
=
=
=

“Nixon pushes the button,”“the missile successfully launches,”“a miracle prevents the missile being launched,”“a nuclear holocaust occurs,”

and consider the following model, the Fine model :ϕ ::= p | ¬ϕ | ϕ ∨ ϕ | ϕ  ψ.

Definition 2 (Semantics). Let M = (W, , V ) be awell-founded conditional model. ThenJpKM = V (p)J¬ϕKM = W \ JϕKM
Jϕ ∨ ψKM = JϕKM ∪ JψKM
Jϕ  ψKM = {w ∈ W : Minw (JϕKM ) ⊆ JψKM }.

THE NIXON ARGUMENT

There is a problem dating back to the 1970s [1, 3, 8] surrounding the semantics for counterfactuals proposed by Lewis.
We have found that our ‘ceteris paribus counterfactuals’ (defined below) provide a unique perspective on the problem(a putative counterexample). The argument goes as follows.
Assume, during the Cold War, that President Richard Nixon
had access to a device which launches a nuclear missile atthe Soviets. All Nixon is required to do is press a button on
the device. Consider the counterfactual if Nixon had pushed
the button, there would have been a nuclear holocaust. Callit the Nixon couterfactual. It is not so difficult to see that
the Nixon counterfactual could be true, or could be imagined to be true. Indeed, one could argue that the Nixon
counterfactual ought to be true in any successful theory ofcounterfactuals. Fine and Lewis both agree (and so do we)that the counterfactual is true ([8, p. 452], [14, p. 468]), but
Fine used the Nixon counterfactual to argue that the Lewis’semantics yields the wrong verdict. This is because “a world
with a single miracle but no holocaust is closer to reality
than one with a holocaust but no miracle.” [8, p. 452] In
response, Lewis argues that, provided the Nixon situation ismodelled using a similarity relation which respects a plausible system of priorities (see below), the counterfactual will
emerge true. We will provide a different response using ceteris paribus counterfactuals, but first let us see how Fine
and Lewis model the situation.
Consider two classes of possible worlds. One class, u, consists of those worlds in which Nixon pushes the button, andthe button successfully launches the missile. The second, v,
consists of those worlds in which Nixon pushes the button,
but some small occurrence – such as a minor miracle – prevents the button’s correct operation. Certainly those worlds
where the button does not launch the missile bear more similarity to the present world than those where it does. This
is Fine’s interpretation of Lewis’ semantics. Any world in u
has been devastated by nuclear warfare, countless lives havebeen lost, there is nuclear winter, etc., whereas worlds in v
continue on as they would have done.
To illustrate Fine’s interpretation, let p, s, m, h be thepropositions:

u1
•

u2
•

un
•

v1
•

v2
•

vk
•

w
•

u
p, s, hv
p, m

An arrow from x to y indicates relative similarity to w, so
u1 is more similar to w than u2 is. Arrows are transitive, andthe ‘snake’ arrow between u and v indicates that vi w uj
for every i, j. For each ui ∈ u, F, ui |= p∧s∧h; and for eachvi ∈ v, F, vi |= p ∧ m. World w is intended to represent the132real world: Nixon did not push any catastrophic anti-Soviet
buttons,3 no nuclear missile was successfully launched at
the Soviets, no miracle prevented any such missile, and no
nuclear holocaust occurred. World v1 is more similar to w
than any world in u is, since in any u-world Nixon pushes
the button and begins a nuclear holocaust. By (1), v1 istherefore the minimal p-world. At v1 the proposition h isfalse, and so F, w 6|= p  h. Therefore, Fine concludes, theNixon counterfactual is false in Lewis’ semantics.
In response, Lewis argues that the proper similarity relation to model the Nixon counterfactual should respect thefollowing system of priorities:

our explicit judgments. It is not; else the objection we areconsidering never would have been put forward.”[14, p. 470]
Instead, we will treat the Nixon counterfactual with an explicit ceteris paribus clause, dispatching with the unintuitive
pre-eminence of perfect match in constructing the similarity
relation.
Our interpretation of the Nixon counterfactual is much
like in preference logic, where formal ceteris paribus reasoning was first applied [7, 22, 24]. Consider the following diagram, which shows a preference of a raincoat to an umbrella,provided wearing boots is kept constant:

1. It is of the first importance to avoid big, widespread,diverse violations of law.
2. It is of the second importance to maximize the spatiotemporal region throughout which perfect match ofparticular fact prevails.
3. It is of the third importance to avoid even small, localized, simple violations of law.
4. It is of little or no importance to secure approximate
similarity of particular fact, even in matters that concern us greatly. ([14, p. 472])
Based on this system of priorities world u1 is more similar to w than v1 is because “perfect match of particular
fact counts for much more than imperfect match, even if
the imperfect match is good enough to give us similarity inrespects that matter very much to us.” [14, p. 470] Thatis, worlds in v in which a small miracle prevents the missile being launched may look quite similar to our world, but
only approximately so. And in Lewis’ system of priorities,
perfect match outweighs approximate similarity. The Lewis
model, then, looks like this:
L

u1
•

u2
•

un
•

v1
•

v2
•

vk
•

w
•

v
p, m

In the Lewis model, u1 is the world most similar to w, andin u1 the missile successfully launches, there is a nuclear
holocaust, and so the Nixon counterfactual is true. Lewis
thus responds to Fine by defending a similarity order thatfavours u1 over v1 . He is justified by prioritising perfect overapproximate match in a similarity relation according to theaforementioned system.
The interpretation of the Nixon counterfactual we will offer is in line with Lewis’, though we do not rely on his systemof priorities. We will achieve a resolution similar to his without having to defend a model different from Fine’s. Afterall, as Lewis says: “I do not claim that this pre-eminence ofperfect match is intuitively obvious. I do not claim that it
is a feature of the similarity relations most likely to guide3

Although there is no way for us to know this, for the sakeof the argument we assume that it is so.

133

•

•

umbrellano boots

raincoat
boots

•

•

umbrellaboots

Arrows point to more preferred alternatives, and are transitive. Evidently, having an umbrella and boots is preferredto having a raincoat and no boots. The variation of having boots skews the preference. If a ceteris paribus clauseis enforced, guaranteeing that in either case boots will beworn or boots will not be worn, then the correct preference
is recovered. A similar situation occurs in the logic of counterfactuals. The variation of certain propositions can skew
the similarity order. In Fine’s argument, this is done by thevariation of physical law, a miracle. If we were to restrict theworlds considered during the evaluation of the counterfactual to those that agree with w on the proposition m, then in
F the world v1 would no longer assume the role of minimal
p-world. Rather, u1 would. In world u1 a nuclear holocaust
does occur, whence the counterfactual becomes true, as desired. This is our resolution of the Nixon argument, which
we next formalise.

4.

up, s, hraincoatno boots

CETERIS PARIBUS SEMANTICS

We introduce into our language a new conditional operator which generalises the usual one. In particular, it accommodates explicit ceteris paribus clauses. The authorsin [22] were the first to define object languages in this way.
They developed a modal logic of ceteris paribus preferencesin the sense of von Wright [24]. For now we will take theordinary conditional operator and embed within it a finite
set of formulas Γ understood as containing the other things
to be kept equal.4
Definition 3 (Language LCP ). Let Γ be a finite setof formulas. Then the language LCP is given by the grammar5
ϕ ::= p | ¬ϕ | ϕ ∨ ϕ | [ϕ, Γ]ψ.
We understand the modality [ϕ, Γ]ψ as the counterfactualϕ  ψ subject to the requirement that the truth of the4
The choice of Γ finite is largely technical. We will mention
some possibilities and difficulties regarding the case where
the ceteris paribus set Γ may be infinite in our concluding
remarks.
5
We redefine the language more precisely as Definition 8
in the appendix. For simplicity we work with the one now
stated.

formulas in Γ does not change. We define ϕ ∧ ψ := ¬(¬ϕ ∨¬ψ), ϕ → ψ := ¬ϕ ∨ ψ, hϕ, Γiψ := ¬[ϕ, Γ]¬ψ. We callthe conditional [ϕ, Γ]ψ a ceteris paribus conditional, or, if
the antecedent is false, a ceteris paribus counterfactual. LCP
is interpreted over standard conditional models, and thus
requires no additional semantic information.
Some additional notation is required, however. Let M =(W, , V ) be a conditional model and let w, u, v ∈ M. LetΓ ⊆ LCP be finite.
• Define the relation ≡Γ over W by u ≡Γ v if for all γ ∈Γ, M, u |= γ iff M, v |= γ. Then ≡Γ is an equivalence
relation.6

Note that the set Ww on which w is defined on may
change after the update. By updating the model M witha ceteris paribus clause Γ, worlds which disagree on Γ arerelegated to the class W \ Ww of infinitely dissimilar (indeed, irrelevant) worlds. Figure 1 shows how the Fine modelchanges after being updated by a ceteris paribus clause forcing agreement on m. This forces out the v-worlds from consideration during the evaluation of the counterfactual; insome sense syntactically ‘correcting’ the provided similarity
order. Of course, if each world already agreed with w on{m} the ceteris paribus clause would have no effect.

• Set [w]Γ = {u ∈ Ww : w ≡Γ u}, the collection of
w-entertainable worlds which agree with w on Γ.

Before F• Define Γw := w ∩ ([w]Γ × [w]Γ ), the restriction of
w to the above worlds.

ΓThus if u, v ∈ [w]Γ then either u Γw v or v w u.

J[ϕ, Γ]ψK=M{w ∈ W : MinΓw (JϕK

M

) ⊆ JψK}.

The semantics for the regular connectives are the same as
those in Definition 2. Notice that we recover the ordinary
counterfactual ϕ  ψ with [ϕ, ∅]ψ.
Consider again the Fine model F. As before we haveF, w 6|= p  h, but now
F, w |= [p, {m}]h.

5. CETERIS PARIBUS AS A DYNAMIC ACTION
The modality [ϕ, Γ]ψ behaves like a dynamic operator,
in the sense of dynamic epistemic logic. For modality-freeformulas ϕ and ψ, evaluating [ϕ, Γ]ψ at w ∈ W amounts totransforming
M = (W, {w }w∈W , V )into[Γ]M = (W, {Γw }w∈W , V )and evaluating ϕ  ψ at [Γ]M, w. This dynamic action ispossible since we are altering the relational structure of M
with only a finite amount of information from Γ.
6Technically, the relation ≡Γ should be defined together
with the semantics in Definition 4 by mutual recursion.
Again, we favour the simpler presentation.
7
See Schurz [19] on comparative ceteris paribus laws.

un
•

v1
•

v2
•

vk
•

u
p, s, hvp, mAfter [{m}]F(2)

We thus think about the Nixon counterfactual by way ofceteris paribus reasoning. Allowing the truth of arbitrary
formulas to vary during the evaluation of a counterfactual
can distort the given similarity order, thereby attributing
falsity to a sentence which may be intuitively true. By forcing certain formulas to keep their truth status fixed one canrule out these cases, which has just been demonstrated with(2). This ceteris paribus qualification is done in preference
logic, and indeed in more general scientific and economic
practice.7 The Nixon counterfactual is simply a situation involving a defeater, or an irrelevant alternative, which ought
to be forced out.

u2
•

w
•

Definition 4 (Semantics). Let M = (W, , V ) be a
conditional model. Then
M

u1
•

u1
•

u2
•

un
•

v1
•

v2
•

vk
•

w
•

u
p, s, hv
p, mFigure 1: The Fine model before and after w is{m}upgraded to w .
The modality-free condition on ϕ and ψ cannot be removed. In particular, one cannot iterate the dynamic ceterisparibus action and retain agreement with the static ceterisparibus counterfactual operator. To see this, consider theexample in Figure 2. Taking Γ = {s} and ∆ = ∅, one has
M, w |= [p, Γ][q, ∆]r, but [Γ]M, w 6|= p  [q, ∆]r.

6.

UNIFORMLY SELECTING CETERISPARIBUS CLAUSES

Having created a formalism which accommodates explicit
ceteris paribus clauses, one might desire a method for uniformly selecting the ceteris paribus set Γ. For von Wright[24], ceteris paribus means fixing every propositional variable which does not occur in the universe of discourse of
the ceteris paribus expression under consideration. Moreprecisely, let UD(ϕ) be the set of all propositional variables
occurring in the formula ϕ, defined inductively as follows.

134

M
v1
• q, s
w

u

w
•
s

u
•
p, s

w
•
s

sand are as they actually are? Then we shallhave to suppose that these tracks are produced
in a way quite different from the actual way. [...]
Are we to suppose that kangaroos have no tails
but that their genetic makeup is as it actually
is? Then we shall have to suppose that genes
control growth in a way quite different from theactual way (or else that there is something, unlike anything there actually is, that removes thetails). And so it goes; respects of similarity anddifference trade off. If we try too hard for exact similarity to the actual world in one respect,
we will get excessive differences in some otherrespect. ([13, p. 9])[∆][Γ]Mv1• q, s
w
•
s

u
•
p, s

• q, rv2• q, rv2

v1• q, sv1• q, s

u
•
p, s

w
•
s

u
•
p, s

• q, rv2• q, rv2

Figure 2: The horizontal panels labelled w and u
define the similarity orders w and u respectively.
UD(p)UD(¬ϕ)UD(ϕ ∨ ψ)UD([ϕ, Γ]ψ)UD({γ1 , . . . , γn })

=
=
=
=
=

{p}
UD(ϕ)
UD(ϕ) ∪ UD(ψ)UD(ϕ) ∪ UD(Γ) ∪ UD(ψ)
UD(γ1 ) ∪ · · · ∪ UD(γn ).

Then the ceteris paribus counterfactual if ϕ were the casethen, ceteris paribus, ψ would be the case amounts to theexpression[ϕ, Prop \ (UD(ϕ) ∪ UD(ψ))]ψ.

(3)

Now all propositional variables not occurring in the universeof discourse of the counterfactual antecedent or consequent
are fixed.
Updating the Fine model with respect to von Wright’s
ceteris paribus set yields the following model:[{s, m}]F

u1
•

u2
•

un
•

v1
•

v2
•

vk
•

w
•

u
p, s, hv
p, m

In fact, for the logic of ceteris paribus counterfactuals tofunction in a meaningful fashion, every formula occurring inΓ must be independent from the counterfactual antecedent.
In the Fine model, we insist that the truth values of s andm are kept fixed. These propositions, however, are nomologically related to p, so we can’t change the truth value ofp without affecting the truth values of s and m. This is why
the counterfactual [p, {m, s}]h is vacuously true, but thenso is the counterfactual [p, {m, s}]¬h. To accommodate a
uniform method for selecting ceteris paribus clauses, more
flexibility is required. What ought to be kept equal when
we can’t keep everything else equal? In the next section wewill consider two strategies for relaxing the interpretation ofceteris paribus to address this question.

7.

RELAXING THE CETERIS PARIBUS
CLAUSE

7.1

Naïve counting

We will now introduce another interpretation for the modalM
ity [ϕ, Γ]ψ. Let us write J[ϕ, Γ]ψKM
CP for the set J[ϕ, Γ]ψK
from Definition 4, and let |=CP act as the ordinary satisfaction relation for Boolean formulas, but withM, w |=CP [ϕ, Γ]ψ iff w ∈ J[ϕ, Γ]ψKM
CP .

Whereas in Definition 4 we required strict agreement on
the set Γ, in order to develop a logic for ceteris paribus counterfactuals with a weaker semantics we will instead relax therequirement to maximal agreement. The best we can do ispreserve the set Γ as much as possible for any given model.
Let Γ ⊆ LCP be finite, and let M = (W, , V ) be a condiΓ
tional model. Define AMΓ : W × W → 2 byAM
Γ (u, v) = {γ ∈ Γ : M, u |= γ iff M, v |= γ}.

We have F, w |= [p, {m, s}]h, but vacuously! It appears
that the relation Γ is too strong to interact with von Wright’s
definition. We are requiring that everything else is kept
equal. This is questionable metaphysics, to say the least.
Lewis made a similar observation in [13], about the counterfactual ‘if kangaroos had no tails, they would topple over’ :We might think it best to confine our attention
to worlds where kangaroos have no tails and everything else is as it actually is; but there areno such worlds. Are we to suppose that kangaroos have no tails but that their tracks in the(4)ΓDefine the relation Γw on Ww by u w v iffMeither |AMΓ (u, w)| > |AΓ (v, w)|,

M
or |AMΓ (u, w)| = |AΓ (v, w)| and u w v.

The relation Γw can be seen as a transformed w , reordering the similarity order so that worlds closer to w preserveat least as much of Γ as worlds further away, and if any
two worlds agree on Γ to the same quantity, then the nearer
world is more similar to w with respect to .
Definition 5 (Semantics). Let M = (W, , V ) be a
conditional model satisfying the limit assumption. Let Γ ⊆

135

8.

LCP be finite. ThenJ[ϕ, Γ]ψKM
NC={w ∈ W : MinΓw (JϕKM ) ⊆ JψKM }.

We write M, w |=NC [ϕ, Γ]ψ iff w ∈ J[ϕ, Γ]ψKM
NC .

Fact 1. Let M = (W, , V ) be a conditional model. Letw ∈ W , and let X ∈ {CP, NC}. Then the following are true,
where ±α is shorthand which uniformly stands for either α
or ¬α:
1. M, w |= ϕ  ψ iff M, w |=X [ϕ, ∅]ψGiven a ceteris paribus interpretation X ∈ {CP, NC, MS},let us write [Γ]X M for the model M updated with a ceterisparibus clause Γ according to interpretation X. Specifically,
we have the following definition.
Definition 7. Let M = (W, , V ) be a conditional model,
and let Γ ⊆ LCP be a finite set of formulas. We define theupdated models [Γ]X M, for X ∈ {CP, NC, MS}, by[Γ]CP M[Γ]NC M
[Γ]MS M

2. M, w |=X (±α ∧ hϕ, Γi(±α ∧ ψ)) → hϕ, Γ ∪ {α}iψ
3. M, w |=CP hϕ, Γiψ ⇒ M, w |=NC hϕ, Γiψ
4. M, w |=NC [ϕ, Γ]ψ ⇒ M, w |=CP [ϕ, Γ]ψ
The original ceteris paribus preference logic [22] could beaxiomatised using standard axioms together with Fact 1.2
and its converse. A crucial difference with NC semantics isthat the converse of Fact 1.2 does not hold. The existenceof a ϕ ∧ ψ-world which maximally agrees on Γ ∪ {α} does
not ensure that α actually holds at that world. In fact, it is
not guaranteed that any formula from Γ ∪ {α} is obtained.

7.2

Maximal supersets

An approach to counterfactuals familiar to the AI community [4–6,12] makes use of a selection function which chooses
the ‘closest’ world according to maximal sets of propositionalvariables. More specifically, each world w satisfies some set
Pw ⊆ Prop of propositional variables, and a world u is aworld closest to w if there is no v with Pu ⊂ Pv ⊆ Pw .
Taking this as a kind of ceteris paribus formalism we obtainthe following variant of our ceteris paribus counterfactuals.
ΓFirst let us define the relation vΓ
w on Ww by u vw v iff

DYNAMICS AND THE NIXONCOUNTERFACTUAL

:=
:=
:=

This provides us with three dynamic ceteris paribus updates. Let us see how they treat the Nixon counterfactual. We have already witnessed the CP update with ceteris paribus sets {m} and {m, s}, and concluded that bothmake the counterfactual true (vacuous truth with {m, s}).
NC and MS updates agree on the truth of the Nixon counterfactual with the CP update on {m}, but disagree on {m, s}.
Updating the Fine model with von Wright’s ceteris paribus
clause {m, s} according to the NC interpretation yields F
again. Thus F, w 6|=NC [p, {m, s}]h. Updating Fine’s modelwith {m, s} according to the MS interpretation gives thefollowing model:[{m, s}]MS F

Definition 6 (Semantics). Let M = (W, , V ) be a
conditional model satisfying the limit assumption. Let Γ ⊂LCP be finite. ThenJ[ϕ, Γ]ψKM
MS={w ∈ W : MinvΓw (JϕKM ) ⊆ JψKM }.

We write M, w |=MS [ϕ, Γ]ψ iff w ∈ J[ϕ, Γ]ψKM
MS . Now Γ is
maximally preserved in the sense that worlds which preserve the same propositions as another, and furthermore
preserve additional propositions from Γ, are deemed to approximate Γ more closely; while worlds u, v with neitherMMMAMΓ (u, w) ⊆ AΓ (v, w) nor AΓ (v, w) ⊆ AΓ (u, w) are considered incomparable.
Fact 2 (Extends Fact 1). Let M = (W, , V ) be a
conditional model. Let w ∈ W . Then the following are true.
1. M, w |= ϕ  ψ iff M, w |=MS [ϕ, ∅]ψ
2. M, w |=MS (±α ∧ hϕ, Γi(±α ∧ ψ)) → hϕ, Γ ∪ {α}iψ
3. M, w |=CP hϕ, Γiψ ⇒ M, w |=MS hϕ, Γiψ
4. M, w |=MS [ϕ, Γ]ψ ⇒ M, w |=CP [ϕ, Γ]ψ

u1
•

u2
•

un
•

v1
•

v2
•

vk
•

w
•

M
either AM
Γ (v, w) ⊂ AΓ (u, w),

Mor AMΓ (v, w) = AΓ (u, w) and u w v.

(W, Γ , V );(W, Γ , V );(W, vΓ , V ).

up, s, hv
p, mIn [{m, s}]MS F the Nixon counterfactual is not true, andneither is p  ¬h.
We summarise the truth of the Nixon counterfactuals p 
h and p  ¬h in the various updated Fine models in thefollowing table.

Counterfactualphp  ¬hClause{m}{m, s}{m}{m, s}

CP
true
true
false
true

Interpretation
NC
MS
true
true
false
false
false
false
true
false

The rows labelled with p  h and p  ¬h indicate thetruth value of those counterfactuals in the updated models[Γ]X F, where Γ is given by the cell in the Clause column
and X is given by the Interpretation column.
Formally, the table illustrates how different truth values
for the Nixon counterfactual may be obtained by combining
the various interpretations of ceteris paribus (CP, NC, MS)with the different ceteris paribus sets (the selected set {m}or von Wright’s set {m, s}). But this doesn’t mean that all
combinations are legitimate formalisations of Fine’s argument. Fine’s story is about small miracles that can interfere136

with Nixon’s ploy, not about whether the missile would successfully launch should Nixon press the button. That the
proposition s must be able to vary is crucial to the story,
so one shouldn’t attempt to keep it equal, on a par withm. We adhere to our favoured formalisation of the Nixon
argument in which the proposition m is the only one thatneeds to be kept equal. We have given principled reasons forthis choice, and our selection makes the counterfactual true
– all interpretations agree on that. The point of the table isa formal one, namely that the truth-values of counterfactuals vary with different ceteris paribus updates according totheir interpretation.

9. THEOREMSL

In the appendix (Corollary 1) we prove that the logic ΛC CP
of ceteris paribus counterfactuals over the class of conditional frames C is complete for CP/NC/MS semantics. The
proof works by translating formulas of LCP into formulas ofa comparative possibility language, in the style of Lewis, andaxiomatising the equivalent logic. This permits a clearer reduction of ceteris paribus modalities to basic comparative
possibility operators, albeit with a translation exponential
in the size of Γ.

10.

CONCLUDING REMARKSThis paper has introduced a ceteris paribus logic for counterfactual reasoning by adapting the formalism in [22]. We
have introduced some variants on ceteris paribus logic inlight of philosophical difficulties arising in the application of
conditionals. We apply our framework to the Nixon counterfactual, and with this bring a new perspective to the
problem. We have suggested and explored the dynamic
perspective of our various syntactic interpretations of ceteris paribus, which has resulted in a richer understandingof so-called comparative ceteris paribus reasoning in formal
settings. We have provided completeness theorems which
demonstrate that the ceteris paribus logics so obtained ultimately reduce to the underlying counterfactual logic; in ourcase Lewis’ VC. With our framework we defend Lewisiansemantics by appealing to examples from preference logic,
where ceteris paribus reasoning is more widely discussed.
Finally, we outline some limitations of our framework anddirections for future research.
Iterated ceteris paribus actions. We saw in Section 5 that
iterated ceteris paribus counterfactuals deviate in truth fromthe corresponding update-then-counterfactual sequence. While
this is undesirable, it is not so uncommon to face such technical difficulties with iterated counterfactuals. It remains tofurther understand the interaction between the two.
Cardinality restrictions on Γ. In general, ceteris paribus
reasoning requires keeping equal as much information as
possible, and sometimes unknown information (for example, unanticipated defeaters of laws). Keeping everythingelse equal may indeed mean keeping equal an indefinite, andpossibly infinite, set of things. Exploring ceteris paribus
logic without cardinality restrictions to Γ is thus more thana mere technical exercise. But it is not so straightforward toextend the present framework to accommodate the presenceof infinite Γ. The translations presented in the appendix
only carry over to the infinite case for infinitary languages,
which is not much of a solution. For the strict ceteris paribus
semantics, we instead suggest following the δ-flexibility ap-

137

proach of [20]. For the relaxed ceteris paribus semantics,
there are conceptual difficulties which arise with the comparison of infinite sets: when should we say of two infinite setsthat one keeps more things equal than the other? Clearlynaı̈ve counting will not suffice. Minimising distance with respect to vΓ is more promising, but has it’s own problems.
We leave this challenging technical enterprise for future research.

11.

ACKNOWLEDGMENTS

We wish to thank the participants at the Australasian Association of Logic and the Analysis, Randomness and Applications meetings held in New Zealand in 2014, and inparticular Sam Baron for valuable feedback on a previous
version of this paper. We also wish to thank anonymousreferees for valuable comments.

12.

REFERENCES[1] Jonathan Bennett. Review of Lewis ([13]). The
Canadian Journal of Philosophy, 4:381–402, 1974.
[2] Patrick Blackburn, Maarten de Rijke, and Yde
Venema. Modal Logic. Cambridge University Press,
New York, NY, USA, 2001.
[3] G. Lee Bowie. The similarity approach tocounterfactuals: Some problems. Noûs, pages 477–498,
1979.
[4] Mukesh Dalal. Investigations into a theory ofknowledge base revision: preliminary report. In
Proceedings of the Seventh National Conference on
Artificial Intelligence, volume 2, pages 475–479, 1988.
[5] Luis Fariñas del Cerro and Andreas Herzig.
Interference logic = conditional logic + frame axiom.
International Journal of Intelligent Systems,
9(1):119–130, 1994.
[6] Luis Fariñas del Cerro and Andreas Herzig. Beliefchange and dependence. In Proceedings of the 6th
conference on Theoretical aspects of rationality andknowledge, pages 147–161. Morgan Kaufmann
Publishers Inc., 1996.
[7] Jon Doyle and Michael P. Wellman. Representing
preferences as ceteris paribus comparatives. Ann
Arbor, 1001:48109–2110, 1994.
[8] Kit Fine. Review of Lewis’ counterfactuals. Mind,
84:451–458, 1975.
[9] Davide Grossi, Emiliano Lorini, and Francois
Schwarzentruber. Ceteris paribus structure in logics ofgame forms. Proceedings of the 14th conference on
theoretical aspects of rationality and knowledge. ACM.,
2013.
[10] Vincent F. Hendricks. Mainstream and formal
epistemology. Cambridge University Press, 2006.
[11] Wesley H. Holliday. Epistemic closure and epistemic
logic i: Relevant alternatives and subjunctivism.
Journal of Philosophical Logic, pages 1–62, 2014.
[12] Hirofumi Katsuno and Alberto O. Mendelzon.
Propositional knowledge base revision and minimalchange. Artificial Intelligence, 52(3):263–294, 1991.
[13] David Lewis. Counterfactuals. Harvard University
Press, 1973.
[14] David Lewis. Counterfactual dependence and time’s
arrow. Noûs, pages 455–476, 1979.

[15] Graham Priest. An introduction to non-classical logic:
From if to is. Cambridge University Press, 2008.
[16] Carlo Proietti and Gabriel Sandu. Fitch’s paradox andceteris paribus modalities. Synthese, 173(1):75–87,
2010.
[17] Dov Samet. Hypothetical knowledge and games withperfect information. Games and economic behavior,
17(2):230–251, 1996.
[18] Leonard J. Savage. The foundations of statistics.
Courier Corporation, 1972.
[19] Gerhard Schurz. Ceteris paribus laws: Classificationand deconstruction. Erkenntnis (1975-), 57(3):pp.
351–372, 2002.
[20] Jeremy Seligman and Patrick Girard. Flexibility inceteris paribus reasoning. The Australasian Journal of
Logic, 10(0), 2011.
[21] Bassel Tarbush. Agreeing on decisions: an analysis
with counterfactuals. Proceedings of the 14th
conference on theoretical aspects of rationality andknowledge. ACM., 2013.
[22] Johan van Benthem, Patrick Girard, and Olivier Roy.
Everything else being equal: A modal logic for ceterisparibus preferences. Journal of Philosophical Logic,
38(1):83–125, 2009.
[23] Hans van Ditmarsch, Wiebe van der Hoek, andBarteld Pieter Kooi. Dynamic epistemic logic, volume337. Springer, 2007.
[24] Georg H. von Wright. The Logic of Preference.
Edinburgh University Press, 1963.
[25] Zuojun Xiong and Jeremy Seligman. Open and closed
questions in decision-making. Electronic Notes in
Theoretical Computer Science, 278:261–274, 2011.

APPENDIX
We first recast Definition 3 in a more formally precise manner.
Definition 8. For each ordinal α let Lα be given by
ϕ ::= p | ⊥ | ¬ϕ | ϕ ∨ ψ | [ϕ, Γ]ψ
where Γ ⊆ Lβ is finite and β < α. LCP is then defined to beS
α Lα .
This ensures the sets Γ are well-defined. One can definea language L of comparative possibility in a similar style,though we will only give the following grammar
ϕ ::= p | ¬ϕ | ϕ ∨ ϕ | ϕ  ψ | ϕ Γ ψ | ϕ Γ ψ

JpKM
J¬ϕKM
Jϕ ∨ ψKM
Jϕ  ψKM

Jϕ Γ ψKM
Jϕ Γ ψKM

Jϕ vΓ ψKM

Definition 9model. ThenΓϕ≺ ψϕ <Γ ψ
ϕ

:=
:=
:=

=
=
=

V (p);W \ JϕKM ;JϕKM ∪ JψKM ;{w ∈ W : ∀u ∈ Ww ∃v ∈ Ww such that
if u ∈ JψKM then v ∈ JϕKM and v w u};{w ∈ W : ∀u ∈ Ww ∃v ∈ Ww such that
if u ∈ JψKM then v ∈ JϕKM and v Γw u};{w ∈ W : ∀u ∈ [w]Γ ∃v ∈ [w]Γ such that
if u ∈ JψKM then v ∈ JϕKM and v Γw u};{w ∈ W : ∀u ∈ Ww ∃v ∈ Ww such that
if u ∈ JψKM then v ∈ JϕKM and v vΓw u}.

Lemma 1. The modal operator [ϕ, Γ]ψ under NC semantics is definable in L.
Proof. We show thatM, w |=NC [ϕ, Γ]ψ iff M, w |= 3ϕ → (ϕ ∧ ψ) ≺Γ (ϕ ∧ ¬ψ).

⇒: Assume M, w |= 3ϕ. Then there is a world x ∈ Ww
such that M, x |= ϕ. By assumption there exists y ∈ WwΓsuch that y Γw x, M, y |= ϕ and for all z w y, we haveM, z |= ϕ → ψ. Since y Γy,we
obtainM,y|= ϕ ∧ ψ. Sow
there is u ∈ Ww such that for all v ∈ Ww , if M, u |= ϕ ∧ ψand v Γw u then M, v 6|= ϕ ∧ ¬ψ. This is exactly M, w |=(ϕ ∧ ψ) ≺Γ (ϕ ∧ ¬ψ).
⇐: By contrapositive. Assume M, w 6|= [ϕ, Γ]ψ. By thesemantic definition there exists x ∈ Ww such that M, x |= ϕand for all y ∈ Ww with y Γw x and M, y |= ϕ, thereis z ∈ Ww such that z Γw y and M, z |= ϕ ∧ ¬ψ. Since
M, x |= ϕ we have M, w |= 3ϕ. Now, take an arbitrary
world u ∈ Ww such that M, u |= ϕ ∧ ψ. Either (i) u Γw x orΓ(ii) x Γw u. If (i), then the fact that M, u |= ϕ and u w xΓimplies the existence of u0 ∈ Ww such that u0 Γ
u

w
w x
and M, u0 |= ϕ ∧ ¬ψ. If (ii), then M, x |= ϕ and x Γw x
Γtogether imply that there exists x0 ∈ Ww with x0 Γ
w x w u0
and M, x |= ϕ∧ ¬ψ. Either way we have that for all u ∈ Ww
there exists v ∈ Ww such that if M, u |= ϕ ∧ ψ then v Γw uand M, v |= ϕ∧ ¬ψ, whence M, w 6|= (ϕ∧ψ) ≺Γ (ϕ∧ ¬ψ).
Lemma 2. The modal operator [ϕ, Γ]ψ under CP semantics is definable in L.

ΓProof. Replace Γw with w in the above proof to showthat the following equivalence

M, w |=CP [ϕ, Γ]ψ iff M, w |= 3ϕ → (ϕ ∧ ψ) Γ (ϕ ∧ ¬ψ).

holds.

Lemma 3. The modal operator [ϕ, Γ]ψ under MS semantics is definable in L.

ΓProof. Replace Γw with vw in the above proof to showthat the following equivalence holds

M, w |=MS [ϕ, Γ]ψ iff M, w |= 3ϕ → (ϕ ∧ ψ) <Γ (ϕ ∧ ¬ψ).

| ϕ vΓ ψ.

We further set
ϕ≺ψ:= ¬(ψ  ϕ);ϕ Γ ψ := ¬(ψ Γ ϕ);
3ϕ:= ϕ ≺ ⊥;====Denote by L− the L-fragment given byΓ¬(ψ  ϕ);¬(ψ vΓ ϕ);¬3¬ϕ.

(Semantics). Let M, w be a conditional

ϕ ::= p ∈ Prop | ¬ϕ | ϕ ∨ ϕ | ϕ  ψ.
Given a set Γ ⊆ L or Γ ⊆ L− , let Γ be the set of all
possible conjunctions of formulas and negated formulasfromVΓ; that is, the set of all ψ such that ψ =±γ, whereγ∈Γ+γ = γ and −γ = ¬γ . So if Γ = {p, ¬q} thenΓ = {p ∧ q, ¬p ∧ q, p ∧ ¬q, ¬p ∧ ¬q}.

138

We will often identity a conjunction ϕ1 ∧ · · · ∧ ϕn with theset {ϕ1 , . . . , ϕn }.
Lemma 4. The modal operator Γ of L is expressible in
L .
−Proof. We show that^ 
Γ → (ϕ ∧ Γ)  (ψ ∧ Γ) .
ϕ Γ ψ ↔

(5)Γ∈Γ⇒: Without loss of generality write M, w |= Γ. Let u ∈ Ww
and suppose M, u |= ψ ∧ Γ. By hypothesis there existsv ∈ [w]Γ such that M, v |= ϕ and v Γ
w u. Now v ≡Γ w, so
M, v |= Γ, and v w u as required.
⇐: Write M, w |= Γ. Then M, w |= (ϕ∧Γ)  (ψ ∧Γ). Letu ∈ [w]Γ and suppose that M, u |= ψ. Then M, u |= ψ ∧ Γ,
so there exists v ∈ Ww with M, v |= ϕ ∧ Γ and v w u.
Then v ≡Γ w, and so v Γw u.
Lemma 5. The modal operator vΓ of L is expressible in
L .
−Proof. We show that
^
_
^
(Γ →
([(ϕ ∧ Λ) ∨ (ϕ vΓ ψ ↔
Γ∈Γ

Λ⊆Γ

Λ⊂Σ⊆Γ

Σ ∧ ϕ)]  ψ ∧ Λ)).
(6)

Γ⇒: Suppose M, w |= ϕ v ψ with M, w |= Γ, for someΓ ∈ Γ. Let u ∈ Ww be arbitrary and Λ ⊆ Γ such thatM, u |= ψ ∧ Λ. Then by hypothesis there is v ∈ Ww suchthat v vΓ
w u and M, v |= ϕ. We have the following twocases.
M
Case 1: AMΓ (w, u) ⊂ AΓ (w, v).
M
Then since Λ ⊆ AΓ (w, u) and M, v |= AMΓ (w, v) ∧ ϕ we
have_
M, v |=
Σ∧ϕΛ⊂Σ⊆Γ

which shows the implication.
M
Case 2: AMΓ (w, u) = AΓ (w, v) and v w u.
Then M, v |= Λ ∧ ϕ, which shows the implication.
⇐: Let u ∈ Ww be arbitrary such that M, u |= ψ. Then
M, u |= AMΓ (w, u) ∧ ψ. Let Γ be the unique element of Γsuch that M, u |= Γ. By hypothesis there exists v ∈ Ww
such that v w u and_Σ ∧ ϕ).
(7)
M, v |= (ϕ ∧ AM
Γ (w, u)) ∨ (AM
Γ (w,u)⊂Σ⊆Γ

If the second disjunct from (7) holds then there is Σ withAMHence
Γ (w, u) ⊂ Σ ⊆ Γ such that M, v |= Σ ∧ ϕ.
MΓAM
Γ (w, u) ⊂ AΓ (w, v). Thus u vw v, as required.
If the second disjunct from (7) fails then M, v |= ϕ ∧MMAMΓ (w, u). In particular AΓ (w, u) = AΓ (w, v), otherwise
the second disjunct would be true. Combining this withv w u we have v vΓw u, as required.
Lemma 6. The modal operator Γ of L is expressible in
L .
−Proof. Replace the subset condition
_
Σ∧ϕ
Λ⊂Σ⊆Γ

139

in (6) with the cardinality condition_Σ∧ϕ|Λ|<|Σ|≤|Γ|and repeat the above process.
Notice that, if Γ ∪ {ϕ, ψ} ⊆ L− , then the right hand sidesof the equivalences established above are in L− . This allows
us to apply the translation to a formula from the inside-out,
the resulting formula belonging to L− .
By a conditional frame we mean a pair F = (W, ), suchthat (F, V ) is a conditional model for any valuation function
V . Let C be the class of conditional frames. Using thenotation from [2], we write ΛL
C for the set of L-formulas
valid over C.
Theorem 1. The logic ΛL
C is complete.
Proof. We take as our axiomatisation the axioms for VC[13], plus the translations from Lemmas 4, 5, and 6.
L

Corollary 1. The logic ΛC CP is complete for CP/NC/MSsemantics.

An Information-Gap Framework for Capturing Preferences
About Uncertainty
Russell Golman

Department of Social and Decision Sciences
Carnegie Mellon University

rgolman@andrew.cmu.edu

ABSTRACT
We propose an integrated theoretical framework that captures preferences for acquiring or avoiding information as
well as preferences for exposure to uncertainty (i.e., risk or
ambiguity) by allowing utility to depend not just on material payoffs but also on beliefs and the attention devoted tothem. We use this framework to introduce the concept of an
information gap – a specific uncertainty that one recognizes
and is aware of. We characterize a specific utility functionthat describes feelings about information gaps. We suggestthat feelings about information gaps are the source of curiosity as well as a second motive to manage one’s thoughts
through information acquisition or avoidance. In addition,
we suggest that feelings about information gaps also contribute to risk- and ambiguity preferences.

Keywords
Ambiguity, curiosity, information gap, motivated attention,
ostrich effect, risk

1. INTRODUCTION
In a seminal paper titled “The Mind as a Consuming Organ,” Thomas Schelling (1987) pointed out that much consumption is not of the material sort, but takes place largely“in the mind.” Research in psychology, decision theory, andeconomics has identified a number of motives underlying informational consumption, from the powerful force of curiosity (Loewenstein, 1994) to the pleasures of knowledge andinsight (Karlsson et al., 2004). Moreover, even when missing information is not available to an individual, demand forthis information plays a role in decision making under uncertainty. Here we propose a unified theoretical frameworkthat allows us to model feelings about information and about
information gaps – specific uncertainties that an individual
recognizes and is aware of. We present a specific utility function that takes as input beliefs and the attention devoted tothem (as well as material payoffs). This utility model canbe applied to decision making about information acquisition
or avoidance as well as to decision making under risk andambiguity (as described in Table 1).
In one branch of the economics literature, preferences about
information have been viewed as derivative from risk preferences (e.g., Kreps and Porteus, 1978; Wakker, 1988; Grant
et al., 1998; Dillenberger, 2010). We take a complementary perspective, considering preferences about information
as primitive and viewing preferences about risk and ambiguity as derivative of them.

141

George Loewenstein

Department of Social and Decision Sciences
Carnegie Mellon University

gl20@andrew.cmu.edu

Decision about:
Whether to address anuncertainty
Whether to expose oneself to an uncertainty

Domain of:Information acquisition
or avoidanceRisky or ambiguous
choiceTable 1: Two domains of decision making affectedby feelings about uncertainty.

The standard account of preferences about information
holds that information is valuable because, and only to theextent that, it enables people to make superior decisions thatraise their expected utility (Hirshleifer and Riley, 1979). Often, however, individuals seek information purely to satisfy
curiosity, which refers to the desire for information for its
own sake – i.e., specifically not for its ability to improve decision making. Curiosity correlates with brain activity inregions thought to relate to anticipated reward (Kang etal., 2009), suggesting that information is a reward in andof itself. Loewenstein (1994) proposed an information-gap
account of curiosity, and our framework allows us to capture
this motive for information acquisition within an expanded
utility model. While curiosity is a powerful motive for information acquisition, there nevertheless are many situations inwhich people actively choose to avoid information, e.g., notobtaining a costless medical test. We hypothesize that information avoidance derives from a desire to avoid increasing
attention on a negative anticipated outcome. More generally, we suggest that individuals have an inclination to seek
(or avoid) information whenever they anticipate that what
they discover will be pleasurable (or painful). Of course,
ex-ante beliefs about such events are already good or bad
respectively (Eliaz and Spiegler, 2006), but there can be a
big difference between discovering something for sure andsimply considering it a likely possibility. Our additional assumption is that obtaining news tends to increase attention
to it (as in Gabaix et al., 2006; Tasoff and Madarász, 2009),
which leads to the implication that people will seek information about questions they like thinking about and will
avoid information about questions they do not like thinking
about. We explore the implications of our proposed utility
model for information acquisition or avoidance in a companion paper (Golman and Loewenstein, 2015a), and we outline
this analysis in Section 5.
The standard account of preferences about risk and ambiguity considers these preferences to be primitives in a model(e.g., Anscombe and Aumann, 1963; Klibanoff et al., 2005).
However, research has shown that missing information has

a profound impact on decision making under risk and ambiguity. For example, Ritov and Baron (1990) studied hypothetical decisions concerning whether to vaccinate a child,when the vaccine reduces the risk of the child dying from adisease but might itself be harmful. When the uncertainty
was caused by salient missing information about the risks
from vaccination – a child had a high risk of being harmed
by the vaccine or no risk at all but it was impossible to findout which – subjects were more reluctant to vaccinate thanin a situation in which all children faced a similar risk andthere was no salient missing information. In a second companion paper (Golman and Loewenstein, 2015b) we argue
that the information-gap concept developed here underlies
an alternative account of risk and ambiguity aversion (andseeking) that is conceptually different from, and has different testable implications from, the usual account of risk
aversion involving loss aversion and the usual account of ambiguity aversion involving vague probabilities.1 In Section 6
we outline our argument that salient information gaps caneither increase or decrease preference for uncertain gambles
depending on whether it is painful or pleasurable to think
about the information one is missing.
Our expanded utility model builds on the insights of Caplin
and Leahy (2001).2 Caplin and Leahy recognize that anticipatory feelings about prizes that might be received in
the future can affect utility. We follow them (and Köszegi(2010) as well) in applying expected utility theory to psychological states rather than to physical prizes, but we expand the domain of psychological states that people canhave feelings about. In doing so, we incorporate Tasoff andMadarász’s (2009) insight that information stimulates attention and thus complements anticipatory feelings. Kreps
and Porteus (1978) present a model capturing preferences
for early or late resolution of uncertainty, and Dillenberger(2010) captures preferences for one-shot or sequential resolution of uncertainty; this line of research thus deals withwhen, but not whether, an individual prefers to acquire information. Our model focuses just on the latter issue, butwith it one could address the timing of uncertainty resolution
by making additional assumptions about time preference.
We rely on a reduced form model of knowledge and awareness to describe information gaps – and the desire to fillthem or ignore them – in order to avoid the complications ofworking with information partitions in a state-space modelof knowledge (as in Aumann, 1976). The standard partitional state-space framework permits a distinction between
two states of affairs – knowing and not knowing – but makes
it difficult to capture unawareness (Modica and Rustichini,
1994; Dekel et al., 1998). We introduce a question-answer
knowledge structure that allows us easily to draw an important distinction between three different states: knowing(represented by a question and a particular answer); notknowing, but knowing that one doesn’t know (represented
by a question and a set of possible answers); and not knowing and not knowing what one doesn’t know (represented by
the absence of an activated question). This third state corresponds to pure unawareness (Li, 2008), in the sense that1
For example, we show that low-stakes risk aversion (Rabin, 2000) could be attributed to the discomfort of thinking
about uncertainties.
2
Many have considered the notion that people derive utility
from their beliefs (Abelson, 1986; Geanakoplos et al., 1989;
Asch et al., 1990; Yariv, 2001; Kadane et al., 2008).

an individual is unaware of the question itself and does noteven distinguish different possible answers. (In contrast, ourquestion-answer structure does not capture partial unawareness, in the sense of an individual being aware of a questionand proper subset of possible answers, but unaware of someother remaining possible answers.) The question-answer
structure is consistent with, and could be cast in terms of, ageneralized state-space model (e.g., Modica and Rustichini,
1999; Heifetz et al., 2006), but we find the question-answer
structure convenient to use.
The question-answer knowledge structure is intended toreflect human information-processing capabilities. Our cognitive maps of the world are not sets of possible states, each
described in exquisite detail to account for all possible consequences of all possible decisions. Instead, people attend
to a few relevant aspects of a situation and use limited information to make a broad judgment that can be refinedlater, if necessary. People tend to set goals and monitor
their progress toward them in order to navigate a complex
world (Miller et al., 1960; Locke and Latham, 1990; Loewenstein, 1999). We advance the idea that the acquisition ofknowledge is also goal-oriented. We don’t simply seek out
information to maximize the data available to us or evento optimize future decisions, but instead tend to seek answers to questions that are either posed to us or that wepose to ourselves. Questions are, therefore, very much like
informational goals or reference points. Indeed, focusing on
a question that one cannot answer – e.g., a puzzle one cannot figure out – can torment a person and at the same time
motivate the search for an answer, much as a high reference
point can simultaneously detract from utility and motivate
one to strive to reach it.

2.

THEORETICAL FRAMEWORK

2.1

Cognitive States

Traditional economic theory assumes that utility is a function of consumption bundles or material outcomes, or (perhaps subjective) distributions thereof. Our basic premiseis that utility depends not only on such material outcomesbut also on one’s cognitive state, encompassing the attention
paid to each of the issues or questions that one is aware ofas well as subjective judgments about the possible answers
to these questions. While people have preferences about
their beliefs (and the attention paid to them), we do nottreat beliefs (or attention) as choice variables. People canchoose whether or not to acquire information that will influence beliefs, but we assume that one’s beliefs, given one’s
information, are constrained by Bayesian inference.
While there surely is an infinite set of possible states of theworld, we assume, realistically we believe, that a person can
only conceive of a finite number of questions at any one time.
We represent awareness with an array of ‘activated’ questions
and a remaining set of ‘latent’ questions. Activated questions are those that the individual is aware of. Latent questions are those that the individual could become, but is not
currently, aware of. The finite subset of questions a person isaware of (i.e., paying at least some attention to) is denoted
Q. We label these activated questions as Q1 , . . . , Qm . A vector of attention weights w = (w1 , . . . , wm ) ∈ Rm
+ indicates
how much attention each activated question gets.3 These3

We can think of the (presumably infinite) set of latent ques-142attention weights depend on three factors that we designate“importance,”“salience,” and “surprise.” We return to define
and discuss these concepts in Section 3.
A question Qi has a countable set4 of possible (mutually exclusive) answers Ai = {A1i , A2i , . . .}.5 A person may
not know the correct answer to a given question, but reasonably has a subjective belief about the probability thateach answer is correct.6 (The subjective probabilities acrossdifferent questions may well be mutually dependent.) This
framework allows us to capture information gaps, which are
represented as activated questions lacking known correct answers, as depicted in Table 2.
Question
LatentAnswerBelief–Unawareness
Unknown Uncertainty
Activated
Known
Certainty l information gap
Table 2: The question-answer knowledge structure.
Anticipated material outcomes, or prizes, can also be incorporated into this framework. We let X denote a countable set of prizes – i.e., material outcomes. The subjective
probability over these prizes is in general mutually dependent with the subjective probability over answers to activated questions; that is, the receipt of new information often leads to revised beliefs about the likelihood of answers
to many different questions as well as about the likelihoodof different material outcomes. Denote the space of answer
sets together with prizes as α = A1 × A2 × · · · × Am × X.
Thus, given a state of awareness defined by the set of activated questions Q,7 we represent a person’s cognitive state
C with a subjective probability measure π defined over α(i.e., over possible answers to activated questions as well as
eventual prizes) and a vector of attention weights w. We denote the set of all possible cognitive states as C = ∆(α)×Rm+(with the notation ∆(α) referring to the space of probability
distributions over α with finite entropy. The restriction to
distributions with finite entropy serves a technical purpose,but it should not trouble us – intuitively, it means that aperson cannot be aware of an infinite amount of information, which is also the basis for our assumption that the setof activated questions is finite.). Each marginal distribution
πi specifies the subjective probability of possible answers toquestion Qi , and similarly πX specifies the subjective probability over prizes.8
The formal representation of a cognitive state is depicted
in Table 3. Consider, for example, a college professor deciding whether or not to look at her teaching ratings. The set
tions as having attention weights of zero.
4
We use the term countable here to mean at most countable.
The restriction of a countable set of answers to a countable
set of possible questions does still allow an uncountable setof possible states of the world, but as awareness is finite, theprecise state of the world would be unknowable.
5
We assume that there is no such thing as an answer thatis disconnected from a question.
6
By subjective probability, we mean personal probability,
but we take it to be observable by direct elicitation.
7
In most cases, we will assume that activation of questions
is determined exogenously – i.e., by the environment. We
don’t model growing awareness (see Karni and Vierø, 2013).
8
For any Ã ⊆ Ai , we have πi (Ã) = π(A1 × · · · × Ai−1 ×
Ã × Ai+1 × · · · × Am × X).
143

of activated questions (and possible answers) might include:“How many of my students liked my teaching?” (0, 1, 2, . . . );“Did they applaud on the last day of class?” (yes/no); “Howgood a teacher am I?” (great, good, so-so, bad, awful); “Will
I get tenure?” (yes/no). Prior belief about the first question might be quite uncertain. The answer to the second
question, on the other hand, might already be known withcertainty. There may or may not be much uncertainty about
the third and fourth questions. All of these beliefs (to theextent they are uncertain) are jointly dependent. The material outcome might be next year’s salary, which would also
depend on (but not be completely determined by) whether
or not she gets tenure. Looking at the ratings will definitively answer the first question and may resolve some, but
not all, of the uncertainty surrounding the other issues.

2.2

Actions

A decision maker has the possibility of taking actions with
two kinds of effects: informational actions contribute to subjective judgments about the world by answering a question;and instrumental actions affect the chances of receiving various prizes (outcomes). For example, wagering on the colorof a ball drawn from an urn is an instrumental action. Examining the contents of the urn is an informational action.
Informational actions affect the subjective probability measure through the conditioning of beliefs on the discovered answer. Instrumental actions affect beliefs directly by changing
the distribution over prizes conditional on subjective judgments. Both instrumental and informational actions also
impact attention weights through their respective effects onimportance and surprise. Note that some actions will haveboth instrumental and informational effects. Examples include paying a fee for a property value appraisal or hiring a
private eye.
At any point in time an individual can be characterized
by a prior cognitive state consisting of subjective probability measure π 0 and attention weight vector w0 . Actions, ingeneral, are operators on cognitive states that map to new
cognitive states or to distributions over cognitive states. A
purely instrumental action acting on the prior cognitive state
determines a particular new cognitive state. Typically, it
preserves the prior subjective judgment about the probability of each answer set and then specifies a new distribution
over prizes conditional on each possible answer set. An instrumental action may also affect the importance of various
questions (as formalized in the next section) and thereby influence the attention weights. For example, the decision toparticipate in a karaoke session will likely raise the attention
weight on the question “Am I a good singer?”
Acquiring information also changes one’s cognitive state.
Ex ante, as one does not know which answer will be discovered, the prospect of acquiring information offers the decision maker a lottery over cognitive states. Upon learning answer Ai to question Qi , one’s subjective probability
measure over ∆(α) changes from π 0 to π Ai = π 0 (·|Ai ).9
We assume Bayesian updating here, which means that ex
ante, before one knows what one will discover, an informational action determines a distribution over subjective judgments such that the expectation of this distribution equals
the prior judgment. That is, by the law of total probability,
9

We thus denote a belief with complete certainty in A×x as
π A×x .

Activated Questions
Q1
..
.
Qm
N/A
∗

Possible Answers
A1 = {A11 , A21 , . . .}..
.
Am = {A1m , A2m , . . .}Possible Prizes
X = {x, x0 , x00 , . . .}Subjective Probabilities∗[π1 (A11 ), π1 (A21 ), . . .]..
.
[πm (A1m ), πm (A2m ), . . .]Attention Weights
w1
..
.
wm

[πX (x), πX (x0 ), . . .]N/A

Answers to different questions are not generally independent. Typically, the joint probability measure π 6= π1 · · · πm · πX .
Table 3: Representation of a cognitive state.

P0Ai= π 0 . An informational action would
Ai ∈Ai πi (Ai ) π
decrease expected entropy because conditioning reduces entropy (see, e.g., Cover and Thomas, 1991, pg. 27). New
information generates surprise (as formalized in the next
section), which changes the attention weights too. Given
the prior attention weight vector w0 based on salience andimportance, we let wAi denote the new attention weight vector immediately after learning Ai , resulting from surprise at
this discovery.

2.3

Preferences over (Distributions of) Cognitive States

The conventional theory of choice under risk assumes thata lottery over outcomes is evaluated according to its expected utility. Given that we may think of an informational
action as creating a lottery over cognitive states, we makethe natural assumptions leading to an expected utility representation in this new domain.

Independence Across Cognitive States
We assume that there is a complete and transitive preference relation  on ∆ (C) that is continuous (with respect toan appropriate topology)10 and that satisfies independence,
so there exists a continuous expected utility representation
u of  (von Neumann and Morgenstern, 1944).
The assumption here is that when information could put
a person into one of many possible cognitive states, preference is consistent with valuing each possible cognitive state
independently of any other cognitive states the person might
have found herself in.
This might seem to imply that the utility of a state ofuncertain knowledge is equal to the expected utility of eachof the possible beliefs – e.g., that being uncertain of whether
the object of my desire reciprocates my affections provides
the same utility as the sum of probabilities times the utilities
associated with the possible outcome belief states. It neednot, because (as we discuss in detail below) obtaining theinformation, and indeed the specific information one obtains,
is likely to affect one’s attention weights. Such a change inattention can encourage or discourage a decision maker fromresolving uncertainty, depending on whether the news thatwill be revealed is expected to be good or bad.

2.4

Choosing Between Sequences of Actions

The discovery of information following an initial action
can change the availability or desirability of subsequent actions. For example, the information in a college professor’s
10

The induced topology on C (derived from the order topology on ∆ (C)) should be a refinement of the order topology
on C (see Nielsen, 1984).

teaching ratings could help her decide whether to enroll ina teacher improvement class. A sequence of actions can beanalyzed with the convention that an action operator passes
through a distribution over cognitive states.11 Thus, we
represent a sequence of actions s acting on a cognitive state(π, w) as s·(π, w) ∈ ∆(C).
Choice from among a set of sequences of actions S, where
early actions may reveal information that will inform later
actions, is represented as utility maximization: a sequences∗ ∈ S may be chosen by a decision maker in the cognitive state (π, w) if s∗ ∈ arg maxs∈S u (s·(π, w)). We findit useful to define a utility function over cognitive states,
contingent on the set of sequences of actions that may subsequently be chosen:
U (π, w | S) = max u (s·(π, w)) .
s∈S

(1)

In the example of the professor’s teaching ratings, the setof available subsequent actions is to enroll in the teacher
improvement class or not to enroll in the class. Looking
at the ratings resolves a lottery over cognitive states, eachof which having utility that is conditional on making the
optimal choice of one of these subsequent actions.
We define the desirability of a sequence of actions s in cognitive state (π, w) as D(s | π, w) = u (s·(π, w)) − u (π, w).12
Desirability is simply marginal utility relative to the trivial‘action’ of doing nothing.

3.

PSYCHOLOGICAL INSIGHTS

In this section we introduce a number of specific psychological insights that lead us to specify a utility function thatgenerates a wide range of testable predictions concerning
informational phenomena. These insights help us characterize the factors that influence the level of attention paid
to a question as well as to identify distinctly the valence ofbeliefs and the desire for clarity.

3.1

Attention

Neuroeconomic research indicates that attention shapes
preference (Fehr and Rangel, 2011). Attention weights in
our model specify how much a person is thinking about particular beliefs and, in turn, how much those beliefs directly
impact utility. We may think of beliefs as having intrinsic
value, which is then amplified by these attention weights.
11

Analogous to the standard assumption in decision underrisk, the model assumes reduction of compound distributions
over cognitive states. This does not imply the traditional
reduction of compound lotteries.
12
The degenerate distributions in ∆ (C) correspond to individual states of knowledge. With the standard abuse of notation, we refer to the utility of the degenerate distributionon (π, w) ∈ C as u(π, w).

144

Our model (assuming monotonicity with respect to attention weights, as described in Section 4) provides a natural
distinction between beliefs that have positive or negative
intrinsic value: beliefs are positive specifically when more
attention enhances utility and are negative in the opposite
case. That is, a person likes thinking about (i.e., putting
more attention weight on) positive beliefs and does not likethinking about negative beliefs.
Here we formalize the concepts of importance, salience,and surprise, all of which, we assume, contribute to attention weight. The importance γi of a question Qi reflects thedegree to which one’s utility depends on the answer. Thus,
for example, for an egocentric, but insecure, individual, thequestion, “Do other people like me?” is likely to be of great
importance because the answer matters to the individual.
Salience, distinctly, reflects the degree to which a particular
context highlights the question. If, for example, an individual hears that another person was talking about her (withno further details), the question of whether the comments
were favorable or not will become highly salient. We denotethe salience of question Qi as σi ∈ R+ . Finally, surpriseis a factor that reflects the dependence of attention on the
dynamics of information revelation, and specifically on the
degree to which receiving new information changes one’s beliefs. If, having believed that she was generally well-liked,
our individual were to discover that the comments about
her were actually unfavorable, the discovery, necessitating aradical change in her belief, would be quite surprising (and,as we presently assume, would increase her attention to thequestion). We denote the surprise associated with a revised
belief about question Qi as δi . We assume that the attention
wi on an activated question Qi is a strictly increasing function of this question’s importance γi , its salience σi , and thesurprise δi associated with it.

Importance
The importance of a question depends on the spread of theutilities associated with the different answers to that question. The degree to which an individual’s utility varies withthe answers to a question depends both on the magnitude of
the utility function and on the perceived likelihood of different answers. Continuing with the example of the questionof how well-liked an individual is, one could distinguish two
relevant traits: egocentrism – the degree to which the individual cares about being well-liked; and insecurity – thedispersion of the individual’s subjective probability distribution across possible answers. By our definition of the concept, importance should be positively related to both traits.
Given a particular prior subjective probability measureπ 0 and a set S of sequences of actions available to the decision maker, the importance γi of question Qi is a function(only) of the likelihood of possible answers and the utilities
associated with these answers, captured asDEγi = φπi0 (Ai ), U π Ai , wAi | S
Ai ∈ supp(πi0 )where U is the utility function defined in Equation (1).
Without specifying the precise form of this function φ, weassume only that it (i.e., importance) increases with meanpreserving spreads of the (subjective) distribution of utilities
that would result from different answers to the question, andthat it is invariant with respect to constant shifts of utility. Thus, a question is important to the extent that one’s

145

utility depends on the answer. Raising the stakes increases
importance. On the other hand, if an answer is known withcertainty, then by this definition nothing is at stake, so theunderlying question is no longer important. While acquiring
information will affect the importance of the questions being
addressed, it takes time to adapt to news, so there should besome delay. We assume that the importance of a question is
updated only when the new information is incorporated into
a new default subjective probability measure.
Our definition of importance is, admittedly, circular. Importance depends on utility, which in turn depends on theattention weight, but importance also contributes to attention weight. There is, likely, some psychological realism tothis circularity which captures the dynamic processes giving
rise to obsession: attention to a question raises its importance, and the elevated importance gives rise to intensified
attention. If we assume that these processes unfold instantaneously, then importance (and, in turn, attention weight andutility) will be a fixed point of this composition of functions.
We can make simple comparisons of importance without going to the trouble of specifying precise values.

Salience
The salience of a question depends on a variety of exogenous
contextual factors. For example, a question could be salient
if it has recently come up in conversation (i.e., it has beenprimed) or if other aspects of the environment remind an
individual about it. Alternatively, a question could be moresalient to an individual if the answer is, in principle, knowable, and even more so if other people around her know theanswer but she does not.
Often a question may be salient despite being unimportant. Continuing the prior example, even if an individual
deems others’ perceptions of her as unimportant, the question of her popularity might nonetheless be highly salient
if the individual was asked, “Do you know what x thinksof you?” Conversely, there are myriad questions that areimportant by the definition just provided, but which lacksalience. There might be numerous people whose opinion ofus we would care about and be unsure of, but unless something raises the issue in our mind, we are unlikely to focus
on it. It seems natural to think that some degree of salienceis a necessary, and sufficient, condition for attention (whilesome degree of importance is not). Thus, we assume thata question Qi is activated (i.e., has strictly positive attention weight wi > 0) if and only if it has positive salience
σi > 0. Further, we assume that attention weight wi has
strictly increasing differences (i.e., a positive cross-partial
derivative, if we assume differentiability) in (γi , σi ). Thatis, an increase in importance produces a greater increase inattention for a more salient question.

Surprise
The third factor that we posit influences attention is the surprise one experiences upon acquiring new information. Surprise reflects the degree to which new information changes
existing beliefs. A natural measure of surprise was proposedin a theoretical paper by Baldi (2002) and, in an empirical follow-up investigation (Itti and Baldi, 2009), shown topredict the level of attention paid to information. Incorporating the insights from this line of research, we assume
that when the answer to a particular question Qj is learned,
thereby contributing information about the answers to asso-

ciated questions and causing their subjective probabilities tobe updated, the degree of surprise associated with a new belief about question Qi can be defined as the Kullback-LeiblerA
divergence of πi j against the prior πi0 :A

δi (πi j ||πi0 ) =XAπi j (Ai ) log

Ai ∈AiA

πi j (Ai ).
πi0 (Ai )Surprise is positive with any new information, and is greatest
when one learns the most unexpected answer with certainty.
However, the feeling of surprise is not permanent. We assume that when the decision maker adapts and gets used
to this new knowledge (formally, when the default subjective
probability measure is reset), it is no longer surprising.

The Belief Resolution Effect
The impact of new information on attention is greatest when
uncertainty about a question is resolved completely. Surprise immediately spikes, but in the long run fades, and theunderlying question becomes unimportant because, with theanswer known, there is no longer a range of possible answers.
Taken together, these factors create a pattern of change inattention weight following the discovery of a definitive answer, what we call the belief resolution effect – when an answer is learned with certainty, there is an immediate boost
in attention weight on it, but over time this attention weight
falls to a lower level. Specifically, when the decision maker
adapts and the certain belief is incorporated into the default subjective probability measure, the question then receives less attention. It is as if the brain recognizes that
because a question has been answered, it can move on toother questions that have yet to be addressed. Janis (1958)
recognized the belief resolution effect when he observed thatsurgical patients getting information about their upcoming
procedures initially worry more about the surgery but subsequently experience less anxiety.

3.2

Valence and Clarity

It is useful to distinguish two sources of a belief’s intrinsic value: valence and clarity. Valence refers to the value
attached to answers to questions. To illustrate the conceptof valence, we return to the example of a professor’s belief
about her teaching ability. Being a good (or bad) teacher
carries intrinsically positive (or, respectively, negative) valence. Clarity refers to preferences between degrees of certainty, independent of the answers one is certain of. We
assume that, ceteris paribus, people prefer to have greater
clarity (i.e., less uncertainty or more definitive subjective
beliefs). The aversion that people feel towards uncertainty
is reflected in neural responses in the anterior cingulate cortex, the insula and the amygdala (Hirsh and Inzlicht, 2008;
Sarinopoulos et al., 2010). It manifests in physiological responses as well. Subjects who know to expect an electric
shock, but who are uncertain whether it will be mild or intense, show more fear – they sweat more profusely, and theirhearts beat faster – than subjects who know for sure thatan intense shock awaits (Arntz et al., 1992).
When valence and clarity pull in opposite directions, it
may be the case that people prefer a certain answer to a
subjective belief that dominates it on valence or that people
prefer uncertainty when it leaves space for better answers.
While the preference for clarity violates Savage’s (1954) surething principle, we do assume a weaker version of it:

One-Sided Sure-Thing Principle
For any π ∈ ∆(α), let supp(π) ⊆ α denote the support ofπ. If for all A×x ∈ supp(π) we have u(π 0 , w) ≥ u(π A×x , w),
then u(π 0 , w) ≥ u(π, w), with the latter inequality strict
whenever there exist A0 × x0 and A00 × x00 ∈ supp(π) suchthat A0 6= A00 .
The one-sided sure-thing principle asserts that people always prefer a certain answer to uncertainty amongst answers
that all have valences no better than the certain answer(holding attention weight constant).

A Measure of Uncertainty
The assumption of a preference for clarity means that there
is a preference for less uncertain subjective beliefs. A usefulmeasure of the uncertainty about a particular question is theentropy of the subjective probability distribution over answers (Shannon, 1948). The P
entropy of a subjective (marginal)
probability πi is H(πi ) = − Ai ∈Ai πi (Ai ) log πi (Ai ) (withthe convention that 0 log 0 = 0).13 At one extreme, entropy
is high when there are many equally likely possible answers;
at the other extreme, there is minimal entropy of 0 when asingle answer is known for sure.

3.3

A Specific Utility Function

To make precise predictions about preferences for (or to
avoid) information, we consider a specific utility function
incorporating the preference for clarity and the role of attention weights:Xu(π, w) =πX (x)vX (x) +
x∈X

m
X
i=1



wi 

X

Ai ∈Ai

πi (Ai )vi (Ai ) − H(πi ) .

(2)

We represent the value of prize x as vX (x) and the valenceof answer Ai as vi (Ai ). We now describe properties (somequite strong and almost certainly not always satisfied) thatcharacterize (and necessarily imply) this utility function (seeTheorem 1 below).

4.

CHARACTERIZATION OF THE UTILITYFUNCTION

4.1

Properties

The utility function in Equation (2) satisfies the following
seven properties.

Independence Across Prizes
In Section 2 we assumed independence across cognitive states.
Independence might extend, as in traditional models, to material outcomes, holding beliefs constant.
P1. Holding the rest of the cognitive state constant, thepreference relation
satisfies independence across prizes if
PAu(π A , w) = x∈X πX(x) u(π A×x , w).

Property (P1) implies belief-dependent expected utility overlotteries that are independent of beliefs about the world. If
13

The base of the logarithm in the entropy formula is arbitrary and amounts to a normalization parameter.

146we also were to assume belief-independent utility for prizes,
then we would gain the ability to reduce compound lotteries consisting of horse races as well as roulette lotteries (Anscombe and Aumann, 1963) to single-stage lotteries.
However, we believe it is often the case that utility is beliefdependent. We might say that a decision maker often has ahorse in the race.

Separability Between Questions
Additive separability of utility between questions means thata person can place a value on a belief about a given question
without needing to consider beliefs about other questions.
P2. A utility function satisfies additive
beP separability 14
tween questions if u(π, w) = uX (πX ) + m
i=1 ui (πi , wi ).

Property (P2) may seem quite strong because we can imagine representations of sensible preferences that are not additively separable. For example, the value of a belief about
whether a car on sale has a warranty intuitively could depend on the cost of the car in the first place (not to mention
one’s desire for a new car, one’s estimation of the costs of car
repairs, etc.). However, we may be able to represent these
preferences as separable after all. We might suppose thatthese beliefs do have separable values but that they correlate with some other highly valued belief, perhaps about how
good a deal one can get on the car. That is, while intuition
tells us that the value of beliefs about different questions(e.g., “does she like me?” and “does she have a boyfriend?”)is often interdependent, this dependence may be mediatedby the existence of additional questions (e.g., “will she go
out with me?”), beliefs about which may be mutually dependent, but independently valued.

Monotonicity with respect to Attention Weights
Preferences satisfy the property of monotonicity with respect to attention weights if whenever increasing attention
on a given belief enhances (or diminishes) utility, it will doso regardless of the absolute level of attention weight. At a
psychological level, the interpretation of this monotonicity
property is that when a belief is positive, more attention
to it is always better, and when a belief is negative, more
attention is always worse. In fact, the property provides anatural definition of whether a belief is positive or negative.
P3. Preferences satisfy monotonicity with respect to atˆ ∈ Rm
tention weights if for any w, ŵ, and ŵ
+ such thatˆˆwi = ŵi = ŵi for all i 6= j and ŵj > ŵj > wj , we haveˆ ) ≥ u(π, ŵ), withu(π, ŵ) ≥ u(π, w) if and only if u(π, ŵ
equality on one side implying equality on the other, for allπ ∈ ∆(α).
In the case that these inequalities hold strictly, we say that
πj , the belief about question Qj , is a positive belief. If theyhold as equalities, we say that πj is a neutral belief. And, inthe case that the inequalities hold in the reverse direction,then πj is a negative belief.
14

A subset of questions
P Q̃ ⊂ Q can also be separable, inwhich case u(π, w) = i:Qi ∈Q̃ ui (πi , wi ) + u−Q̃ (π−Q̃ , w−Q̃ )where π−Q̃ is the marginal distribution over answers to theremaining questions and prizes and the vector w−Q̃ contains
the remaining components of w.

147

Linearity with respect to Attention Weights
The next property describes how changing the attention ona belief impacts utility. For any given attention weight, themarginal utility of a change in belief depends on what thosebeliefs are and how much the individual values them. The
property of linearity with respect to attention weights means
that, in general, the marginal utility associated with such achange in belief (assuming the utility of this belief is separable) is proportional to the attention on that belief.
P4. When the utility of question Qi is separable, linearity
with respect to attention weights is satisfied if for any wi andŵi ∈ R+ and πi0 and πi00 ∈ ∆(Ai ), we haveui (πi0 , ŵi ) − ui (πi00 , ŵi ) =ŵiui (πi0 , wi ) − ui (πi00 , wi ) .
wiProperty (P4) allows us, in the case of separable utility, toassign an intrinsic value v to beliefs such that ui (πi0 , wi ) −ui (πi00 , wi ) = wi (vi (πi0 ) − vi (πi00 )). We abuse notation by referring to the valence of answer Ai as vi (Ai ), with it being
defined here as the intrinsic value vi of belief with certainty
in Ai . We have taken the liberty of specifying a precise
relationship between attention weights and utility as a convenient simplification; it should be noncontroversial because
we do not claim to have a cardinal measure of attention
weight.

Label Independence
Intuitively, the value of a belief should depend on how anindividual values the possible answers and on how probableeach of these answers is, and these factors (controlling forattention weight of course) should be sufficient to determine
the utility of any (uncertain) belief. In particular, the valueof a belief should not depend on how the question or theanswers are labeled.
P5. Label independence is satisfied if, when the utility ofquestions Qi and Qj are separable, a bijection τ : Ai →Aj , such that vi (Ai ) = vj (τ (Ai )) and πi (Ai ) = πj (τ (Ai )),implies that vi (πi ) = vj (πj ).

Reduction of Compound Questions
The intuition behind the assumption of label independence
also seems to suggest that the utility of a belief perhaps
should not depend on the way the question giving rise to
the belief is asked, i.e., on whether a complicated questionis broken up into pieces. We should recall, however, that theactivation of a particular question directs attention to thebelief about this question. Thus, in general, the utility of abelief will not be invariant to the question being asked. Still,
it may be the case that utility remains invariant when a compound question is broken into parts as long as the attention
on each part is weighted properly. If utility remains invariant upon setting attention weights on conditional questions
to be proportional to the subjective probabilities of the hypothetical conditions, then we say that the utility function
satisfies the reduction of compound questions property. Figure 1 demonstrates the reduction of a compound question
with appropriate attention weights on each subquestion.
P6. A separable utility function satisfies the reduction ofcompound questions property if whenever there is a partition
ζ of the answers Ai (to question Qi ) into ζ = {Ai1 , . . . , Ain }Answers
Question Qi
60%

67%

A1

33%

A2

Answers
40%

1

Attentionweight .6 w

A1
A2

Question Q j

Question Qi

20%

Attentionweight w

Attentionweight w

30%

Question Qi

40%

3
75% A

25%

A4

10%

A4

Figure 1: Decomposition of a compound question.
and a bijection τ : ζ → Aj into the answers to some question
Qj such that for any h ∈ [1, n] and any Ai ∈ Aih ,vi (Ai ) = vj (τ (Aih )) + vih (Ai )and
πi (Ai ) = πj (τ (Aih )) · πih (Ai ),it follows thatui (πi , ω) = uj (πj , ω) +

n
X

h=1

uih (πih , πj (τ (Aih )) · ω).

Ruling Out Unlikely Answers Increases Clarity
A final property operationalizes the preference for clarity.
Controlling for the valence of one’s beliefs, by consideringsituations in which one is indifferent between different possible answers to a question, there should be a universal aversion to being uncertain about the answer to an activatedquestion. As a building block toward quantifying the uncertainty in a subjective belief, we assert here that when an
unlikely (and equally attractive) answer is ruled out, uncertainty decreases (and thus the utility of that uncertain belief
increases).
P7. Ruling out unlikely answers increases clarity if, when
the utility of question Qi is separable and all answers tothis question have the same valence, i.e. vi (Ai ) = vi (A0i ) for
all Ai and A0i ∈ Ai , then for any π where without loss ofgenerality πi (Ahi ) is weakly decreasing in h and for any π 0such that πi0 (Ahi ) ≥ πi (Ahi ) for all h ∈ [1, h̄] (with at least
one inequality strict) and πi0 (Ahi ) = 0 for all h > h̄, for someh̄, we consequently have vi (πi0 ) > vi (πi ).

4.2

Utility Representation Theorem

Theorem 1. If the properties P1-P7 are satisfied, thenXu(π, w) =πX (x)vX (x) +
x∈X

m
X
i=1



wi 

X

Ai ∈Ai5.

A3

2

Attentionweight .4 w

Ai ∈ Ai . Moreover, −F (·) satisfies Shannon’s (1948) axioms(continuity, increasing in the number of equiprobable answers, and reduction of compound
P questions) characterizing
the entropy function H(πi ) = − Ai ∈Ai πi (Ai ) log πi (Ai ).



πi (Ai )vi (Ai ) − H(πi ) .

Proof. Linearity with respect to attention weights allows us to pull an attention weight on question Qi outsideof the utility ui (πi , wi ) = wi vi (πi ) (using a neutral belief tocalibrate vi ). A partition of Ai into singletons Aih such thatvi (Ai ) = vih (Ai ) allows us, by reduction of the compound
question, to determine that the function F (πi ) = vi (πi ) −P
Ai ∈Ai πi (Ai )vi (Ai ) does not depend on vi (Ai ) for any

INFORMATION ACQUISITION AND AVOIDANCE

We can apply our utility function to decisions about information acquisition or avoidance. We develop our analysis in
a companion paper (Golman and Loewenstein, 2015a), andwe provide a broad outline here of its implications. The desire for information, in our model, can be decomposed into
three distinct motives: recognition of the instrumental valueof the information; curiosity to fill the information gap(s);and motivated attention to think more or less about what
could be discovered. The instrumental value of information arises from its impact on subsequent actions. As in thestandard account of informational preferences, it is definedas the difference between the expected utility of subsequentactions conditional on having the information and the utility expected in the absence of the information. Curiosity
arises from the expected reduction in uncertainty upon acquiring information. It is defined as the expected utility ofrevised beliefs, given prior levels of attention. The magnitude of curiosity depends on the attention devoted to each
information gap that stands to be addressed. Motivated attention arises from the surprise upon acquiring information.
It is defined as the expected utility from increased attention
on whatever happens to be discovered, conditioning on all
possible outcomes. Motivated attention is a motive to acquire information that’s expected to be good and to avoid
information that’s expected to be bad.
Putting the three motives together, our model makes many
predictions about when, and the degree to which, information will be sought or avoided. When anticipated answers
are neutral or even potentially positive, information should
be sought. The strength of the desire for this information
should increase with the number of attention gaps that canbe addressed, the attention paid to them, and the valence ofthe possible outcomes. However, when anticipated outcomesare sufficiently negative, information would be avoided. This
“ostrich effect” when anticipating bad outcomes is consistent with a growing body of empirical evidence (see, e.g.,Karlsson et al., 2009; Eil and Rao, 2011). In addition, thebelief-resolution effect in our model leads to a novel prediction: individuals who discount the future less should be lesslikely to exhibit the ostrich effect and more likely to acquire
information despite anticipated bad news.

6.

RISK AND AMBIGUITY PREFERENCE

Section 5 outlines how the model we have developed allows
us to describe a desire to acquire or to avoid information.
We can apply this same model to an entirely new domain:
preferences about wagers that depend on missing information. Risk and ambiguity aversion are complex topics, andwe develop these applications in depth in a companion paper (Golman and Loewenstein, 2015b). Here, we provide a
broad outline of the model’s implications in this domain.
Decision making under risk and under ambiguity both expose decision makers to information gaps. Imagine a choice
between a gamble and a sure thing. Deciding to play thegamble naturally focuses attention on the question: what148

will be the outcome of the gamble? Of course, deciding to
not play the gamble does not stop an individual from paying some attention to the same question (or, if not choosing the gamble means it will not be played out, the related
question: what would have been the outcome of the gamble?) but playing the gamble makes the question more important, and that brings about an increase in the attention
weight on the question. If the individual is aware of this effect, which is natural to assume, then whether it encourages
risk taking or risk aversion will depend on a second factor:
whether thinking about the information gap is pleasurable
or aversive. When thinking about the missing informationis pleasurable, then the individual will be motivated to increase attention on the question, which entails betting on it.
Conversely, when thinking about the missing information isaversive, the individual will prefer to not bet on it. Thismay help to explain why, for example, people generally prefer to bet on their home teams rather than on other teams,
especially in comparison to the home team’s opponent.
Decision making involving uncertainties that are ambiguous is similar to the case with known risks, but with anadditional wrinkle: with ambiguity, there are additional information gaps. In a choice between a sure thing and an
ambiguous gamble, for example, a second relevant question (in addition to the one above about the outcome of
the gamble) is: what is the probability of winning with the
ambiguous gamble? (And there may be additional relevant
questions that could inform someone about this probability,
so even a Bayesian capable of making subjective probability judgments would be exposed to these information gaps.)
Again, betting on the ambiguous gamble makes these questions more important and thus will increase the attention
weight on them. So, desire to play the gamble will be increasing with the degree to which thinking about the gambleis pleasurable. To the extent that abstract uncertainties are
not pleasurable to think about, this model provides a novel
account of standard demonstrations of ambiguity aversion,
including those first generated by Ellsberg (1961) in his seminal paper on the topic.

7. DESIRE FOR WISDOMOur utility model can be used to describe preferences between knowing and not knowing. But another comparison
is also of interest, albeit harder to investigate empirically –the difference between awareness and unawareness. While
we cannot easily give a person the choice whether or not
to become aware of a question, we can at least introspect.
We might posit that awareness of meaningful questions is
a source of utility. Equation (2), the utility function which
represents preferences between cognitive states given a fixed
set of activated questions Q, might be augmented with aterm vQ (Q) capturing the intrinsic value of awareness ofparticular issues.
Wisdom, the combination of awareness and clarity,15 is,
or at least tends to be, preferable to ignorance. We of course
must allow exceptions if we are serious that beliefs have va-

Question Answer
BeliefLatent–Unawareness↓ Awareness
Unknown Uncertainty
WisdomActivated↓ Clarity
Known Certainty↓

Table 4: Wisdom, the combination of awareness andclarity.
lence that may be negative. The popular adage that “ignorance is bliss” expresses concern for the negative beliefs
that awareness may entail. However, in many natural situations, a person may reasonably anticipate that newfoundawareness will bring about neutral or even positive beliefs.
In such contexts, information and awareness may be simultaneously acquired. For example, a bird-watcher typicallywould strictly prefer to learn the name of a previously unnoticed songbird rather than to remain unaware of its existence. Curiosity is behind the desire to catch the name upon
becoming aware of the bird’s existence, even though the particular name does not really matter, but utility from awareness implies that opening, and then immediately closing, anaversive information gap need not be zero sum. Rather,discovering the new bird’s name, acquiring both the question and the definitive answer, produces a net positive utility gain, which is what we designate, in the context of ourmodel, the utility of wisdom. We find the desire for wisdom in individuals’ varied pursuits of insight and expertise,from a naturalist’s passion for identifying flora and fauna toa fan’s thirst for new baseball statistics or a connoisseur’s
discriminating taste for wine.16
Aristotle in 350 B.C. asserted, “All men by nature desire toknow.” John Stuart Mill agreed, in his classic Utilitarianism,
arguing that, “It is better to be a human being dissatisfied
than a pig satisfied; better to be Socrates dissatisfied than afool satisfied.” We too assert that knowledge can be a very
real source of utility. A perspective that information derives
value solely from its ability to yield material consumption
fails to appreciate the most profound benefits provided byinformation, the knowledge and wisdom it confers.

8.

[1] Abelson, R. (1986). Beliefs are Like Possessions.
Journal for the Theory of Social Behavior 16 (3),223-250.
[2] Anscombe, F., Aumann, R. (1963). A Definition of
Subjective Probability, Annals of Mathematical
Statistics 34, 199-205.
[3] Arntz, A., Van Eck, M., de Jong, P. (1992).
Unpredictable Sudden Increases in Intensity of Pain
and Acquired Fear. Journal of Psychophysiology 6(1), 54-64.
[4] Asch, D., Patton, J., Hershey, J. (1990). Knowing for
the Sake of Knowing: The Value of Prognostic
Information. Medical Decision Making 10 (1), 47-57.
[5] Aumann, R. (1976). Agreeing to Disagree. Annals of
Statistics 4, 1236-1239.
[6] Baldi, P. (2002). A Computational Theory ofSurprise. In Information, Coding, and Mathematics,

15

We are aware that this may not be the most common usageof the word wisdom, but the distinction between knowledge
acquired from a state of uncertainty and knowledge acquired
from a state of unawareness is rarely made explicit. The
term, “wisdom” seems to adequately capture this distinction
if we think of a wise man or woman as not only having theright answers, but also asking the right questions.

149

REFERENCES

16

Lab studies also find that people prefer environments which
seem to stimulate new questions and promise to provide relevant information (Kaplan, 1992).

M. Blaum, P. Farrell, H. van Tilborg (Eds.) Norwell,
MA: Kluwer.
[7] Caplin, A., Leahy, J. (2001). Psychological Expected
Utility Theory And Anticipatory Feelings. Quarterly
Journal of Economics 116 (1) 55-79.
[8] Cover, T., Thomas, J. (1991). Elements ofInformation Theory. New York: Wiley.
[9] Dekel, E., Lipman, B., Rustichini, A. (1998).
Standard State-Space Models Preclude Unawareness.
Econometrica 66 (1), 159-173.
[10] Dillenberger, D. (2010). Preferences for One-Shot
Resolution of Uncertainty and Allais-Type Behavior.
Econometrica 78 (6), 1973-2004.
[11] Eil, D., Rao, J. (2011). The Good News-Bad News
Effect: Asymmetric Processing of Objective
Information about Yourself. American Economic
Journal: Microeconomics 3, 114-138.
[12] Eliaz, K., Spiegler, R. (2006). Can Anticipatory
Feelings Explain Anomalous Choices of Information
Sources? Games and Economic Behavior 56 (1),87-104.
[13] Ellsberg, D. (1961). Risk, ambiguity, and the Savage
axioms. Quarterly Journal of Economics, 75, 643-699.
[14] Fehr, E., Rangel, A. (2011). Neuroeconomic
Foundations of Economic Choice – Recent Advances.
Journal of Economic Perspectives 25 (4), 3-30.
[15] Gabaix, X., Laibson, D., Moloche, G., Weinberg, S.
(2006). Costly Information Acquisition:
Experimental Analysis of a Boundedly Rational
Model. American Economic Review 96 (4),1043-1068.
[16] Geanakoplos, J., Pearce, D., Stacchetti, E. (1989).
Psychological games and sequential rationality.
Games and Economic Behavior 1, 60-79.
[17] Golman, R., Loewenstein, G. (2015a). Curiosity,
Information Gaps, and the Utility of Knowledge.
Working Paper.
[18] Golman, R., Loewenstein, G. (2015b). Information
Gaps for Risk and Ambiguity. Working Paper.
[19] Grant, S., Kajii, A., Polak, B. (1998). Intrinsic
Preference for Information. Journal of Economic
Theory 83, 233-259.
[20] Heifetz, A., Meier, M., Schipper, B. (2006).
Interactive Unawareness. Journal of Economic
Theory 130 (1), 78-94.
[21] Hirsh, J., Inzlicht, M. (2008). The Devil You Know:
Neuroticism Predicts Neural Response to
Uncertainty. Psychological Science 19 (10), 962-967.
[22] Hirshleifer, J., Riley, J. (1979). The Analytics ofUncertainty and Information – An Expository
Survey. Journal of Economic Literature 17 (4),1375-1421.
[23] Itti, L., Baldi, P. (2009). Bayesian Surprise Attracts
Human Attention. Vision Research 49 (10),1295-1306.
[24] Janis, I. (1958). Psychological Stress: Psychoanalytic
and Behavioral Studies of Surgical Patients. New
York: Wiley.
[25] Kadane, J., Schervish, M., Seidenfeld, T. (2008). Is
Ignorance Bliss? Journal of Philosophy 105 (1), 5-36.
[26] Kang, M.J., Hsu, M., Krajbich, I., Loewenstein, G.,

McClure, S., Wang, J., Camerer, C. (2009). The
Wick in the Candle of Learning: Epistemic Curiosity
Activates Reward Circuitry and Enhances Memory.
Psychological Science 20 (8), 963-973.
[27] Kaplan, S. (1992). Environmental preference in aknowledge-seeking, knowledge-using organism. In
The Adapted Mind, J. H. Barkow, L. Cosmides, andJ. Tooby (Eds.), New York: Oxford UP.
[28] Karlsson, N., Loewenstein, G., McCafferty, J. (2004).
The Economics of Meaning. Nordic Journal of
Political Economy 30 (1), 61-75.
[29] Karlsson, N., Loewenstein, G., Seppi, D. (2009). The
Ostrich Effect: Selective Attention to Information.
Journal of Risk and Uncertainty 38 (2), 95-115.
[30] Karni, Edi, and Marie-Louise Vierø. (2013). “Reverse
Bayesianism”: A Choice-Based Theory of Growing
Awareness. American Economic Review 103 (7),2790-2810.
[31] Klibanoff, P., Marinacci, M., Mukerji, S. (2005). A
Smooth Model of Decision Making under Ambiguity.
Econometrica 73, 1849-1892.
[32] Köszegi, B. (2010). Utility from Anticipation and
Personal Equilibrium. Economic Theory 44 (3),415-444.
[33] Kreps, D., Porteus, E. (1978). Temporal Resolutionof Uncertainty and Dynamic Choice Theory.
Econometrica 46 (1), 185-200.
[34] Li, J. (2008). A Note on Unawareness and Zero
Probability. PIER Working Paper No. 08-022.
[35] Locke, E., Latham, G. (1990). A Theory of Goal
Setting and Task Performance. Englewood Cliffs, NJ:
Prentice Hall.
[36] Loewenstein, G. (1994). The Psychology of curiosity:
A review and reinterpretation. Psychological Bulletin
116 (1), 75-98.
[37] Loewenstein, G. (1999). Because it is there: The
challenge of mountaineering... for utility theory.
Kyklos 52, 315-344.
[38] Modica, S., Rustichini, A. (1994). Awareness andPartitional Information Structures. Theory andDecision 37 (1), 107-124.
[39] Modica, S., Rustichini, A. (1999). Unawareness andPartitional Information Structures. Games andEconomic Behavior 27 (2), 265-298.
[40] Miller, G., Galanter, E., Pribram, K.H. (1960). Plans
and the Structure of Behavior. New York: Holt.
[41] Nielsen, L. (1984). Unbounded Expected Utility andContinuity. Mathematical Social Sciences 8, 201-216.
[42] Rabin, M. (2000). Risk Aversion andExpected-utility Theory: A Calibration Theorem.
Econometrica 68, 1281-92.
[43] Ritov, I., Baron, J. (1990). Reluctance to vaccinate:
Omission bias and ambiguity. Journal of Behavioral
Decision Making 3 (4), 263-277.
[44] Sarinopoulos, I., Grupe, D., Mackiewicz, K.,
Herrington, J., Lor, M., Steege, E., Nitschke, J.
(2010). Uncertainty During Anticipation Modulates
Neural Responses to Aversion in Human Insula andAmygdala. Cerebral Cortex 20 (4), 929-940.
[45] Savage, L. (1954). The Foundations of Statistics.
New York: Wiley.

150

[46] Schelling, T. (1987). The Mind as a Consuming
Organ. In The Multiple Self, J. Elster (Ed.),
Cambridge: Cambridge UP, 177-196.
[47] Shannon, C. (1948). A Mathematical Theory ofCommunication. Bell System Technical Journal 27,
379-423.
[48] Tasoff, J., Madarász, K. (2009). A Model ofAttention and Anticipation. Working Paper.
[49] von Neumann, J., Morgenstern, O. (1944). Theory ofGames and Economic Behavior. Princeton:
Princeton UP.
[50] Wakker, P. (1988). Nonexpected Utility as Aversion
of Information. Journal of Behavioral Decision
Making 1 (3), 169-175.
[51] Yariv, L. (2001). Believe and Let Believe: AxiomaticFoundations for Belief Dependent Utility Functionals.
Cowles Foundation Discussion Paper No. 1344.

151

Deliberating between Backward and Forward Induction
Reasoning: First Steps
Aleks Knoks

Eric Pacuit

University of Maryland

University of Maryland

aknoks@umd.edu

epacuit@umd.edu

ABSTRACT
Backward and forward induction can be viewed as two stylesof reasoning in dynamic games. Since each prescribes taking a different attitude towards the past moves of the otherplayer(s), the strategies they identify as rational are sometimes incompatible. Our goal is to study players who are able
to deliberate between backward and forward induction, as
well as conditions under which one is superior to the other.
This extended abstract is our first step towards this goal. We
present an extension of Stalnaker’s game models [34, 35], inwhich the players can make “trembling hand” mistakes. Thismeans that when a player observes an unexpected move, she
has to figure out whether it is a result of a deliberate choiceor a mistake, thereby committing herself to one of the twostyles of reasoning.

1. INTRODUCTION AND MOTIVATION
We begin with a motivating example. Consider the gameG1 depicted in Figure 1. There are two players: Ann (A) andBob (B). Ann moves first (node h0 ) and can either choose
to go out (O), immediately ending the game, or stay in thegame (I). If she chooses to stay in, node h1 is reached. At h1 ,Ann and Bob move simultaneously (Ann’s available actions
are u and v while Bob’s are a, b and c). The structure of thisgame is similar to the extensively studied Battle of the Sexes
with an Outside Option (see, for instance, [7, 14, 37]).
h1

h0
A

u, a

O

3, 3

A, B

I

2, 2

2, 0

u, b

u, c

2, 1

d, a

d, b

1, 1

d, c

1, 5

4, 0

Figure 1: The game G1
Figure 1: Motivating Example

would result in a payoff of 5. Alternatively, Bob can avoid
speculating about the reasons behind Ann’s move and focus
on trying to figure out what is the rational thing for her to
do at h1 . Although Ann might hope initially that the game
will end in the rightmost node, she must realize that Bob
will never choose c since it is strictly dominated by a. So,
guaranteeing a payoff of 2, u is Ann’s rational choice at h1 .
Clearly, if Bob convinces himself that Ann is choosing u,
playing a is his best response. The two lines of reasoning,
thus, lead to different recommendations for Bob. The firstis an example of the so-called “forward induction reasoning”
requiring that the players think critically about the observed
past choices of their opponent(s) and find plausible explanations for these choices [7, 24, 28, 35, 37]. The second can becalled “backward induction reasoning” requiring the players
to only reason about their opponents’ future behavior andnot about their past moves [2, 12, 28, 30, 35].
There are many characterizations of both forward andbackward induction reasoning in the game theory literature(cf. [7, 28, 35]). These formal renderings match the informal explanation given above, recommending that Bob playsa and b, respectively. However, the formal models do notsolve what we take to be Bob’s real challenge, namely, deciding which of these two lines of reasoning is more plausible in the present case.1 Notice that a wrong choice leads
to an unwelcome consequence. Suppose that Bob interprets
Ann’s choice of I as an attempt to get a higher payoff, but
it turns out that she did it for some other reason—e.g. she
was careless—and that, at h1 , she decides to play u. In thiscase, Bob ends up with 0. Now, suppose that Bob disregards Ann’s previous move, as backward induction suggests
he should, but it turns out that Ann is hoping to get 4. In
this case, Bob’s payoff is 1 instead of 5.
These considerations bring us to the following generalquestion: How can a player deliberate between backward andforward induction in cases in which both seem plausible (atleast prima facie) while dictating incompatible choices? Admittedly, in the above situation, Bob seems to be faced witha particularly difficult choice, since his information does notseem to sway the scales in favor of one or the other style ofreasoning. But suppose that he is in a situation in which theplayers are prone to making mistakes relatively frequently—
we elaborate on this notion below; for now simply think ofSuppose that Bob initially believes that Ann is going tochoose O. To his amazement, however, Ann stays in thegame.
Now Bob has to figure out why Ann decided to play
hand
mistakesI, what she will choose at node h1 , and, most importantly,
what is his own best response. There are two plausible “lines
1
We do not mean to suggest that Bob is explicitly applyingof reasoning”
forMotivation
Bob. The first goes as follows: Ann chose
1 Introduction
andbackwardor forward induction himself. Rather, a theorist
I at h0 because she is hoping for a payoff greater than 3.
can identify his reasoning as an instance of one or the other.
Thus, Ann must be hoping that the game will terminate in
In our models, the players’ “choice” of reasoning style will
We begin with a motivating example. Consider the game depicted in Figure 1. There are
the rightmost node. Hence, the rational choice for Bob is
be traced back to their prior beliefs about how likely it isb, which—assuming
about
Ann
correct—that their opponents may make a mistake.
two players:
Ann (A) and Bob (B).his
Annconjecturegoes first (nodev0 ) andcan iseitherchoose to go out
Keywords: Backward induction, forward induction, epistemic game theory, trembling(O), immediately ending the game, or stay in the game (I). If she chooses to stay in, node

153are u and v
v1 is reached. At v1 , Ann and Bob move simultaneously (Ann’s available actions
while Bob’s are a, b and c). The structure of this game is similar to the extensively studied
Battle of the Sexes with an Outside Option (see, for instance, [6, 12, 33]).
Suppose that Bob initially believes that Ann is going to choose O. To his amazement,

h0
A

h2

h1
I1

O1

O2

3, 3

3, 3

A, B

I2

A

u, a

2, 2

u, b

2, 0

u, c

2, 1

d, a

d, b

1, 1

d, c

1, 5

4, 0

Figure 2: The game G2

Figure 2: Motivating Example #2

the so-called “trembling hand mistakes” [32]: Ann chooses O,• N is a finite set of players.
but plays I instead. In this case, backward induction reason• Act is the set of actions available to the players. To
ing should be preferred. Or let’s say that Ann and Bob areof 5.
simplify notation, we assume that Act is partitioned
playing a different game, G2 , in which mistakes are still posinto sets of actions for each player. For player i ∈ N ,sible, but Ann has two opportunities to go out before reachlet Acti ⊆Act denoteHow
playeri’s actions.
considerations
us to the following generalquestion:cana player deing the nodeTheseat whichthe players movebringsimultaneously—see
Figure 2. Intuitively, if Bob observes Ann play I1 , it is rea• H is a set of finite sequences of finite sequences of
sonable liberate
for him tobetween
interpret her
move as aandmistake.
Suppose,backward
forwardinduction incases in which both seem plausible (atelements of Act. Elements h ∈ H are called histories.
however, that Bob subsequently observes that Ann plays I2 .
We assume H satisfies the following constraints:
Now, the interpretation of her previous choices that is sugleast
prima
facie)seems
whilemoredictatingchoices? Admittedly, in the above situation,gested byforwardinduction
plausibleincompatible(we could–  ∈ H, where  denotes the empty history.
of course modify the game further by adding more opportunities forAnnseems
to exit tothebegame).
suggests is difficult choice,– If h ∈sinceH and h, then h0 ∈ does
H, whereBob
facedWhat
withthisa allparticularly
hish0 information
not h0  h
0that additional information about the context of the game—means that h is a initial segment of h. Formally,e.g., how probable it is that players make mistakes—can help
we write h0  h provided h = h0 u where u is a
seem
to
sway
the
scales
in
favor
of
one
or
the
other
style
of reasoning. But suppose that hethe player settle on a style of reasoning.
sequence of sequences from Act, and h0 u denotes
Our ultimate goal is to study players that are able to dethe concatenation of h0 with u.
liberate isbetween
forward andbackward
as wellin a situationin whichtheinduction,
players areprone to making mistakes relatively frequently—we– Each h ∈ H is finite. That is, we restrict attention
as conditions which make each style of reasoning superior toto finite horizon games.3 We write len(h) for thethe other. The above considerations suggest that to do soelaborate on this notion below; for now simply think of length
the so-called
mis-in h).
of h (i.e.,“trembling
the number handof elements
we need to endow standard game theoretic agents with (relatively) rich beliefs. We use a model introduced by Stalnaker
A history∈ H is calleda terminal
history if[28]:
Ann thechoosesO,beliefs.
but plays
I instead. In thiscase, hbackward
induction
reasoning[33, 34, takes”
35, 36] todescribeplayers’
In addition
there is no h0 ∈ H such that h0 6=  and hh0 ∈ H.
to having beliefs about their opponents’ strategies and the
Let Z ⊆ H denote the set of terminal histories. Let
game—standard
for preferred.
Epistemic Game
Theory27, 29]—should be
Or let’ssay[16,that
Ann and BobVarea di↵erent
game in histories.
which Each
= Hplaying− Z be theset of non-terminal
our players can also interpret observed behavior as either
non-terminal history is associated with a simultaneous
the result of deliberate action or as a mistake.
decision
for a setreachingof players.theFornodethis reason,
are still as
possible,Ann2 has
two opportunities
to problem
go out before
This mistakes
paper is structured
follows. but
Sectiondescribes
we sometimes call elements of V decision nodes. For
the formal framework that allows us to represent the twoh ∈ V , let A(h) be the possible extensions of h:lines of atreasoningabove:move
extensive
games with siwhichdiscussed
the players
simultaneously—see
Figure 2. Intuitively, if Bob observes Ann
multaneous moves (Section 2.1), our extension of Stalnaker’s
A(h) = {~a | h~a ∈ H and ~a is a sequence of actions}.
model (Section 2.2), two notions of rationality (Section 2.3),
play I1 , it is reasonable
for him to interpret her move as a mistake. Suppose, however, that
as well as an illustrative
example (Section 2.4). In Section 3,• τ is a turn function τ : V → ℘(N ) assigning a set of
we argue that our model offers an illuminating perspective
players4 to each non-terminal history h ∈ V . For each
Bob characterizations
subsequently observes
that
Ann plays
on epistemicof backward
inductionand isI2 . Now, thei ∈interpretation
N , let Vi = {hof∈her
V |previousi ∈ τ (v)}choices
be the set ofa conservative extension of Stalnaker’s model. Finally, Secnon-terminal histories where player i moves. Similarly,
tions 4 anddiscuss
related bywork
and outline
a numberof more plausible
we define theof actions
available
to i atthea decision
that5 issuggested
forwardinduction
seems(wesetcould
of course
modifydirections for future research.
node h ∈ Vi . For each h ∈ V and i ∈ τ (h), let Ai (h)be the set of actions available to i at h:

2.

game further by adding more opportunities for Ann to exit the game). What this all suggests
FRAMEWORK

2.1

is that additional
information
aboutmoves
the context of thehowprobableis thatExtensive
games withsimultaneous
If i game—e.g.,6∈ τ (h), then let
Ai (h)= ∅. Weitimpose
an addi-The examples from the introduction are extensive gameswith simultaneous moves.2 Following [26, Section 6.3.2],
we describe them as structures hN, Act, H, τ, {ui }i∈N i, where:2
We assume that the reader is familiar with the basics ofgame theory. The formal definitions are included here to fix
notation.

Ai (h) = {a ∈ Acti | there is an ~a ∈ A(h) containing a}tional constraint to ensure that each decision node is34This is a standard restriction in the literature on epistemic
characterization of backward induction.
4
Throughout this article, we assume that for all h ∈ V ,τ (h) 6= ∅. If we drop this assumption, then histories inwhich τ (h) = ∅ should be interpreted as a move by nature.

154

associated with a strategic game:5– For each h ∈ V , A(h) = Πi∈τ (h) Ai (v).
• For each i ∈ N , ui : Z → R is a utility function.
A strategy for player i assigns an action to each of i’s
decision nodes. Formally, a strategy for player i is a function
si : Vi → Act where for all h ∈ Vi , si (h) ∈ Ai (h). Let Si
be the set of all strategies for player i. As usual, a strategy
profile is a sequence of strategies, one for each player (i.e.,
an element of Πi∈N Si ). Given a strategy profile s, let si beplayer i’s component of s and s−i the sequence of strategies
from s for all players except i (i.e., s−i ∈ Πj6=i Sj ). Eachprofile of strategies s generates a terminal history ρs ∈ Z.
We say that a non-terminal history is reached by a strategy
profile provided h is an initial segment of ρs .
A strategy for player i represents her conditional plan for
the game. It prescribes a choice for player i at all of i’s
decision nodes, including those that are ruled-out by thestrategy itself. Suppose that h ∈ Vi . An action a ∈ Acti (h)rules out a decision node h0 ∈ Vi provided h  h0 , but
h~a 6 h0 for any ~a ∈ A(h) containing a. In addition, we saythat a rules out action a0 provided a0 ∈ Ai (h0 ) for somedecision node h0 ∈ Vi that is ruled-out by a.6 For example,
in Figure 1, A’s action O at h0 rules out the actions u andd (because O rules out h1 ).

2.2

Game models

A game model describes the players’ beliefs during aplay of the game. As discussed in the introduction, we areinterested in representing players that allow for the possibility that one or more of their opponents made a mistake.
This means that we must include states in which the movesof player i (i.e., the observed behavior of player i) does notmatch i’s choices. To make this precise, each state in thegame model will be associated with both strategies for theplayers and sequences of actions representing the observed
behavior of the players.
The players’ behavior in a game is represented by a sequence of actions. Recall that histories h ∈ H are sequencesof sequences of actions (one action for each player whose turnit is to move). For each h ∈ H, let behi (h) be the sequenceof i’s actions in h. Formally, behi is defined by induction on
the length of histories: behi () =  (at the initial node, noneof the players have made a choice), and(behi (h)a if i ∈ τ (h) and ~a contains abehi (h~a) =behi (h)i 6∈ τ (h)5There is a hidden notational difficulty here. Since different players move at different decision nodes, the indices ofthe sequences of actions change from decision node to decision node. Formally, we represent a sequence ~a at decisionnode h as a function ~a : τ (h) → ∪i∈τ (h) Ai (h) where foreach i ∈ τ (h), ~a(h) ∈ Ai (h). We write ~ai to denote the action a ∈ Ai (h) such that ~a(i) = a and say ~a contains a.
This implicitly assumes that i ∈ τ (h) (otherwise ~ai is not
well-defined). Alternatively, we could assume that all players
move at every decision node and introduce notation to distinguish “active” players from “passive” players. The passive
players at a decision node would only have a single action
available for their choice. We follow the first approach in thisarticle.
6
We are implicitly assuming that all the action labels are
unique. This assumption can be dropped, although it does
simplify the notation.

155

If X is a set, then X ∗ is the set of all finite strings of X.
An i-history is a sequence of actions such that α = behi (h)for some h ∈ H. Given an i-history α and a decision nodeh ∈ Vi , let αh be the component of α describing the action
chosen at h. If α does not specify a move at h (either because
the previous moves in α rule out h or α is not a maximal
history), then αh is undefined. For instance, in Figure 1,
there are four A-histories (, O, Iu, and Id) and four Bhistories (, a, b, and c). We use Ou to denote the strategysA in which sA (h0 ) = O and sA (h1 ) = u (similarly, for
Od). Furthermore, we have Iuh0 = I, Iuh1 = u, and Oh1 isundefined.
A player history may be a partial description of what that
player does in the game. This happens when the i-history α
does not specify a choice for i at a decision node h not ruledout by α. Of course, if an i-history α specifies an action forplayer i at a decision node h ∈ Vi , then α specifies an action for i at each h0 such that h0  h and h0 ∈ Vi . We areinterested in sets of player histories that represent possible
plays of the game. A set of player histories {αi }i∈N is coherent if there is a history h ∈ H such that for all i ∈ N ,behi (h) = αi . Note that a set of i-histories may be coherent,
yet not completely describe a path trough the game. For instance, {I, c} is a coherent set of player histories in the gamepictured in Figure 1: There are two histories h = (I)(u, c)and h0 = (I)(d, c) such that behB (h) = behB (h0 ) = c andbehA (h) = behA (h0 ) = I. However, there is a unique history
representing the play of the game associated with a coherent set of player strategies. The play of the game generated
by a coherent set of i-histories {αi }i∈N is the longest history h such that h  h0 for each h0 such that for all i ∈ N ,behi (h0 ) = αi . The play of the game associated with the coherent set {I, c} in the game in Figure 1 is h = (I). The playof the game associated with a coherent set of player histories may be empty and need not be maximal. For example,
the following table lists the coherent sets of strategies andthe corresponding play of the game for the game pictured in
Figure 1.
Coherent sets player strategies{, }{O, }

Play of the game(O)

{Iu, }, {Id, }{Iu, a}(I)(I)(u, a){Id, a}{Id, b}(I)(d, a)(I)(d, b){I, a}, {I, b}, {I, c}, {I, }{Iu, b}{Iu, c}{Id, c}(I)(I)(u, b)(I)(u, c)(I)(d, c)

The sets {O, a}, {O, b} and {O, c} are not coherent.
Suppose that W is a nonempty set, elements of which
are called states. Each player i will be associated with twofunctions βi and σi subject to the following constraints:
1. For each i ∈ N , βi (w) is a (possibly empty) i-historyand σi (w) is a strategy for player i.
2. The i-histories {βi (w)}i∈N are coherent.
We say that a player made a mistake at a history h ∈ Vi
in the world w provided her behavior is different than what

is prescribed by her chosen strategy σi (w) at h. Formally, imade a mistake at h ∈ Vi provided βi (w)h 6= σi (w)(h) (ifβi (w)h is defined).
Example. Recall the game in Figure 1 and consider threestates w1 , w2 and w3 . Suppose that σA (w1 ) = σA (w2 ) =σA (w3 ) = Ou (recall that Ou is the strategy in which A
chooses O at h0 and u at h1 , but it is not an A-history)and βA (w1 ) = βA (w2 ) = βA (w3 ) = I. Thus, A made a
mistake at h0 . The strategies for player B are σB (w1 ) = a,σB (w2 ) = b, σB (w3 ) = c (again, these are the strategies
in which B chooses, respectively, a, b, and c at h1 ), andβB (w1 ) = βB (w2 ) = βB (w3 ) = c. These states are pictured
as follows:

w1

A : Ou; B : a

A : Ou; B : b

A : Ou; B : c

A : I; B : c

A : I; B : c

A : I; B : c

w2

w3

The strategies σA (w) and σB (w) are displayed in the top
half of the circles and the histories βA (w) and βB (w) in
the bottom half (where w ∈ {w1 , w2 , w3 }). If these states
describe A’s beliefs (i.e., they are the set of doxastic possibilities for A), then A is certain that B will play c, but isuncertain about exactly why B is playing c. It might be because B made a mistake (as in states w1 and w2 ) or because
B simply followed through on his plan to play c. Furthermore, A arrived at these beliefs under the supposition thatshe (contrary to her chosen strategy) selected I at h0 .
The players’ beliefs and belief revision policies are represented in the standard way (cf. [4, 10, 36]). Each player i ∈ N
is associated with a prior probability on the set of states,
Pi ∈ ∆(W ),7 and a plausibility ordering i ⊆ W × W
satisfying the following constraints: for each i ∈ N and foreach w ∈ W , Pi (w) > 0 (i.e., Pi is a full support probability measure); i is a locally connected (for all w, v, x,
if w i x and w i v, then v i x or x i v) partial
order (reflexive and transitive relation) on W . The plausibility ordering i represents player i’s belief revision policy. For each i ∈ N and states w, v ∈ W , let w ≈i v iff
w i v or v i w. Since, i is a locally complete partial order, ≈i is an equivalence relation. For w ∈ W , let[w]i = {v | w ≈i v} denote the equivalence class of w for≈i , called i’s information cell. The intended interpretationis that w ≈i v means that w and v are subjectively indistinguishable to player i (i’s beliefs, knowledge, and conditional
beliefs are the same in both states).8 The players’ full beliefs at a state w are defined as usual: For each w ∈ W , letmaxi ([w]i ) = {x | there is no y ∈ [w]i such that y i x},
where y i x means that y i x but x 6i y.
Definition 1. For each w ∈ W and i ∈ N , player i’s (partial) beliefs at state w are given by the probability measure
Pi,w ∈ ∆(W ) defined as follows: For each E ⊆ W ,Pi,w (E) = Pi (E | max([w]i ))i7

For a set X, let ∆(X) be the set of all probability measures
on X. In this paper, we assume that the set of states is finite,
so we can assume that P is defined on all subsets of W .
8That is, the equivalence classes of ≈i are the different“types” for player i.

The players’ partial beliefs Pi,w represent their beliefs
about the possible choices, behaviors and beliefs of their opponents at state w.9 The belief revision policy describes how
the players revise their beliefs given any evidence F ⊆ W :Pi,w (E | F ) = Pi (E | max(F ∩ [w]i )).
iNote that this conditional probability is well-defined for any
set F such that F ∩ [w]i 6= ∅. In particular, there may be a
set F such that Pi,w (F ) = 0, yet Pi,w (· | F ) is well-defined.
This is a very general model of belief revision for the players,
since it describes how each player revises her beliefs given
any evidence consistent with her current information (i.e.,any F such that [w]i ∩ F 6= ∅). However, we are primarily
interested in how the players revise their beliefs given theactions that they observe in the game.10 Each state w ∈ W
is associated with a history h ∈ H as follows. Let hw be
the history corresponding to the play of game associatedwith {βi (w)}i∈N (see the discussion above). Note that hw
need not be a maximal history, so hw is the behavior that isobserved at state w. For any h ∈ H, let [h] = {w | βi (w) =behi (h) for all i ∈ N } be the event that the players behaved
according to history h. Then, Pi,w (E | [hw ]) is i’s probabilityof E given her most plausible explanation of the actions she
observed at state w. Thus, the belief revision policy describes
how the players’ beliefs change during a play of the game.11
Putting everything together, a game model for a game
G is a tuple MG = hW, {(βi , σi )}i∈N , {i }i∈N , {Pi }i∈N i. In
addition, we impose the following two constraints:• For all w ∈ W and i ∈ N , if v ∈ [w]i , then σi (w) =σi (v). That is, players know their own strategy.12• For all w ∈ W and i ∈ N , for each initial segment h0 ⊆hw (including the empty history), there is a w0 ∈ [w]isuch that hw = h0 .
The last constraint ensures that if a sequence of choices in
the game is consistent with a player’s information, then all of
its initial segments must be consistent with the player’s information. This is a consequence of assuming that the structure of the game is (commonly) known to all the players andthat players cannot think it is possible to observe a history
without observing the sequence of choices that generated
the history. Compare the above constraint with the stronger
assumption that for all w ∈ W , for all h ∈ H, there is a
w0 ∈ [w]i such that hw0 = h. This ensures that it is consistent with the players’ information that every possible history
in the game could be realized. Of course, ex ante, the players
do not rule out any histories.13 However, our models represent the players’ ex iterim beliefs. In such models, it may be
9That is, beliefs about the possible types of their opponents.
Our models of games are closely related to Bayesian extensive games with observable actions [26, Section 12.3]. However, there are important methodological and conceptual
differences between Bayesian games and epistemic models
of games (see [27, Section 1.4]). For this reason, we postpone a complete comparison between our game models andBayesian extensive games with observable actions to the fullversion of the paper.
11
Thus, our models are related to the type spaces based on
conditional probability systems from [7, Section 2.2].
12Each player can be associated with a standard knowledge
operator where for all E, Ki (E) = {w | [w]i ⊆ E}.
13
Assuming that all the players are aware (in the sense of[22, 23]) of the structure of the game.
10

156

consistent with a player’s information (which includes her
chosen strategy) that some history of the game will not beplayed (cf. the discussion of richness conditions on the modelin Section 5).

2.3

Rationality

A player chooses rationally provided her strategy choice
at a state maximizes the players subjective expected utility with respect to her beliefs about the past and expectedmoves of her opponents. We do not assess the rationality ofthe players’ moves themselves. Thus, a player may choose
rationally at a state, though she may not carry out her plan
because she made a mistake.
Suppose that G is an extensive game with simultaneous
moves and MG = hW, {(βi , σi )}i∈N , {i }i∈N , {Pi }i∈N i is agame model for G. For each w ∈ W , the strategy realized at
w by player i is si (w) : Vi → Acti defined as follows:(
βi (w)hif βi (w)h is definedsi (w)(h) =σi (w)(h) otherwiseThen, s(w) = (s1 (w), . . . , sn (w)) is a profile of strategies,
and let Out(s) be the (unique) terminal history generated
by s.
Definition 2. For any strategy si ∈ Si for player i, theexpected utility of si at state w is:X
EUi,w (si ) =Pi,w ({w0 } | [hw ])ui (Out(si , s−i (w))).
w0 ∈W

A player chooses optimally at state w provided her current strategy maximizes her subjective expected utility at
w, given the actions that she observed. Let Si (w) ⊆ Si be
the set of strategies for player i that conform to player i’s
moves in state w. That is, si ∈ Si (w) implies that for allh ∈ Vi , if βi (w)h is defined, then si (h) = βi (w)h . Then,Opti={w | σi (w) maximizes expected utility
with respect to Pi,w and Si (w)}.

If w ∈ Opti , then player i is adopting the best possible strategy given i’s observations at w. Rationality is more demanding. There are two versions of rationality. The first requiresthat a player is rational at a state w provided her strategy
at w is optimal given her beliefs at w and was optimal at all
previous decision nodes given her beliefs at the moment ofdecision. We say that a state w0 ∈ [w]i is an earlier choicestate provided βi (w0 ) is an initial segment of βi (w).
Definition 3. Player i is rational-1 at state w provided
w0 ∈ Opti for all earlier choice states w0 . Let Rat1i be theset of all states w such that i is rational-1 in w.
A player may be rational-1 even if she does not correctly
implement her strategy. The second version of rationality
requires that a player’s strategy is optimal even when theplayer learns that her beliefs are mistaken. That is, the strategy is optimal and remains optimal after any belief revision.
Definition 4. Player i is rational-2 at state w provided
w0 ∈ Opti for all states w0 ∈ [w]i . I.e., [w]i ⊆ Opti . Let Rat2i
be the set of all states w such that i is rational-2 in w.
Of course, Rat1i ⊆ Rat2i (if a player is rational-1, then theplayer is rational-2). However, in general, the converse is nottrue (this is illustrated by an example in the next section).

157

2.4

ExampleFigure 3 depicts models of the games from Figures 1 and2. These models represent the players’ initial beliefs and dispositions to change their beliefs that we discussed in the introduction. The model on the left, M1 , represents one playof the game in Figure 1, and the model on the right, M2 ,represents a play of the game in Figure 2. We draw an arrow from state v to state w when w i v. The solid arrows
represent Bob’s plausibility ordering B and the dashedarrows represent Ann’s plausibility ordering A (we only
represents Bob’s beliefs in M2 ). To keep down the clutter in the pictures, we assume that the remaining arrows
can be inferred by transitivity and reflexivity. The strategies σA (w) and σB (w) are displayed in the top half of thecircles and the histories βA (w) and βB (w) in the bottom
half (empty histories are left blank). We think of the players strategy choices and moves as discrete random variables.
Thus, [Choosehi = a] = {w | σi (w)(h) = a} is the eventthat player i chooses action a at decision node h. Similarly,[M ovehi = a] = {w | βi (w)h = a} is the event that player iplayed a at history h. The (common) prior probabilities are
displayed next to the states.
Suppose that w4 is the actual world in model M1 . Thus,
Ann chose the strategy Ou, but made a mistake and played
I followed by u (as originally planned). Bob chose strategy
a which he correctly implemented when given the chance tomove. His (overall) most plausible worlds are w1 and w2 .
This means that he is certain that Ann plays O at h0 (i.e.,
PB,w4 ([ChoosehA0 = O]) = 1). Moreover, he (initially) thinksthat Ann’s strategies Ou and Od are equally likely (i.e.,
PB,w4 ([ChoosehA1 = u]) = PB,w4 ([ChoosehA1 = d]) = 0.5).
If Ann surprises Bob by playing I, he is disposed to interpret this as a mistake on her part, rather than as revealingthat she is following a different strategy (i.e., maxB ([w4 ]B ∩[M ovehA0 = I]) = {w3 }, βA (w3 ) = I while σA (w3 ) = Ou).
Furthermore, after observing Ann play I, Bob is certain that
her next move will be u: PB,w4 ([ChoosehA1 = u] | [M ovehA0 =I]) = 1. This model also illustrates what it means for aplayer to be rational-1. Note that Ann made a mistake in
w4 , yet she is still rational-1 (w4 ∈ Rat1A ). Both w1 and w3are earlier choice states for Ann (as is w4 ), and she chooses
optimally in all these states: OptA = {w1 , w2 , w3 }.
The model M2 in Figure 3 represents Bob’s beliefs in thegame from Figure 2 in which Ann has two opportunities
to exit the game. Suppose that w6 is the actual world. No
mistakes are made with Ann playing I1 I2 c and Bob playing u. Initially, Bob believes that Ann is going to choose O1(maxB ([w6 ]B ) = {w1 , w2 } with σA (w1 )(h0 ) = σA (w2 )(h0 ) =O1 ). On the condition that Ann actually plays I1 , he is disposed to interpret her move as a mistake, predicting that she
is going to go out at the next opportunity (maxB ([w6 ]B ∩[M ovehA1 = I1 ]) = {w3 , w4 }, σA (w3 )(h1 ) = σA (w4 )(h1 ) =O2 . If Ann surprises Bob the second time by playing I2 , heis disposed to conclude that it is very likely that Ann ach0
tually chose to play I1 and I2 (PB,w6 ([ChooseA= I1 ] ∩h1h1h0[ChooseA = I2 ] | [M oveA = I1 ] ∩ [M oveA = I2 ]) = .9).
Intuitively, if Ann surprises Bob, he is disposed to reason
in the backward induction style (ignoring her mistake), but
if she surprises him a second time, Bob switches to forward
induction and conjectures that (it is highly probable that)
Ann is going to play d. The model M2 also illustrates the difference between rationality-1 and rationality-2. In w2 , Bob

.04

.25
w4

A:Ou; B:aw5

A:Iu ; B:a

A:Ou; B:aw3

A:I ; B:

.25
w1

.25
A:Ou; B:a

A:Od; B:a

A: ; B:

A: ; B:

A:O1 O2 u; B:a

A:I1 I2 d; B:a

A:I1 I2 ; B:

A:I1 I2 ; B:

.15

.25
w3

.36

w2

.15

A:O1 O2 u; B:a

A:O1 O2 d; B:a

A:I1 ; B:

A:I1 ; B:

.15
w1

w6

w4

.15

A:O1 O2 u; B:a

A:O1 O2 d; B:a

A: ; B:

A: ; B:w2

FigureThemodelsmodelsandM2
Figure3:
3: TheM1 M
and1 M2
is rational-1, since {w1 , w2 } ⊆ OptB , but he is not rational2, since w6 ∈ [w2 ]B , but w6 ∈/ OptB . That is, the strategy
that Bob chooses at w2 is not optimal with respect to thebeliefs he would have after revising his initial beliefs with
the information that Ann plays I1 and I2 .

3.

STALNAKER AND AUMANN

It is easy to see that our game model is a conservativeextension of Stalnaker’s [34, 35, 36]. Since we extend his
model by allowing states in which the players’ moves differ 12
from what is prescribed by their chosen strategy, the players
can know each other’s strategies and still be uncertain about
the way the game is going to end. In spite of this, however,
our models can accommodate the standard epistemic characterizations of backward induction found in the literature,and, in particular, Aumann’s classic characterization.
Aumann proved that if there is common knowledge that
all of the players are rational, then the backward induction path will be realized [2]. Our models are richer than
Aumann’s: We describe the players’ beliefs and belief revision policies in addition to the players’ knowledge. Recently,
Samet extended Aumann’s result to doxastic models, which
are much closer to the models we use. He proved that if there
is common belief 14 that all the players’ strategies are doxastically substantively rational, then the backward induction
path is realized [31]. Since we allow for mistakes, we will
have models in which there is common belief that the players choose optimally, but the backward induction path does
not obtain. There is another difference between our models
and Aumann’s and Samet’s. The behavior functions can beviewed as a temporal parameter. That is, our model includes
states that describe the players’ beliefs at different moments
during the play of the game (cf. [5, 12]). In general, the players’ beliefs may change even if the game unfolds according totheir chosen strategies. We can recover Samet’s characteri14

We assume that the reader is familiar with the formal definition of common belief. See Appendix A for the formal
definition in our framework.

zation of backward induction with an additional constraint:
For all i ∈ N and w, w0 ∈ W , if w0 ∈ [w]i , then for eachw00 ∈ maxi ([w]i ∩ [hw0 ]) there is a w000 ∈ maxi ([w]i ) suchthat σi (w00 ) = σi (w000 ) for all i ∈ N .
This constraint says that the players cannot learn anything about their opponents’ strategies that they did notalready know at the beginning of the game.
Proposition 1. Suppose that G is an extensive game(without simultaneous moves) in “general position” (see Appendix A) and MG is a model for G satisfying the above
constraint and that every possible mistake is considered: forall w ∈ W , every possible mistake that i can make given
i’s strategy at w is realized by the behavior at some state
w0 ∈ [w]i . Suppose that w ∈ W is a state in which the histories (β1 (w), . . . , βn (w)) generate a maximal path throughthe game. If there are no mistakes in w and common belief
at w that all the players are rational-1, then the path that is
generated by the histories is the backward induction path.
There are many other epistemic characterizations of backward induction.15 What is more relevant for our purposes is
Stalnaker’s criticism of Aumann’s epistemic characterizationof backward induction [35, Section 5]. The problem lies with
Aumann’s notion of rationality which is captured and refined by our rationality-1. A player is rational-1 provided
her strategy is optimal (given a sequence of moves) and was
optimal at all previous choices with respect to her beliefs at
the moment of choice. Stalnaker argues that this notion ofrationality is much too strong.16 His idea is that a strategy
15

It is beyond the scope of this article to survey all of thedifferent approaches. See [29, Section 8.11] and [27] for a
discussion and pointers to the literature.
16
We will not repeat Stalnaker’s argument here. The gist of
it is that it is important not to conflate “action a would be
optimal if node v were reached” and “if node v is reached,then action a is optimal”.

158for player i is optimal provided i would choose optimallyat node v (according to his strategy) given i’s beliefs under
the hypothesis that node v is reached. To formalize this idea,
Stalnaker introduces the notion of perfect rationality: “Incases where two or more [strategies] are [optimal], the agent
should consider, in choosing between them, how he should
act if he learned he was in error about something.” [34, pg.
148]. It is not hard to see that our definition of rationality-2
is equivalent to Stalnaker’s definition of perfect rationality.
Thus, our models can accommodate both Stalnaker’s andAumann’s analysis of backward induction. Of course, this,by itself, is not new (cf. [5] and [19]). However, our analysis
also opens the door for further refinements of the notions ofrationality they use.
For instance, perfect rationality (or rationality-2) requiresthat a player’s strategy is robustly optimal. That is, it is optimal even after the player learns that her beliefs are mistaken.
Variants of rationality-2 can be defined by fixing the set ofevidence that may induce a change of belief for a player. For
instance, we can require that a player’s strategy must be
robustly optimal with respect to evidence about her opponents’ moves (cf. [7, Section 2.2]), strategy choices, beliefs,
or even evidence about the player’s own moves. A complete
analysis of the different options will be left for the full version of the paper.

4. RELATED WORK
Our model allows for states in which the players’ moves
differ from what is prescribed by their chosen strategy. This
general idea (i.e., trembling hand mistakes) was used by
Selten and others to characterize refinements of the Nashequilibrium (cf. [18, 32]). Within the equilibrium refinement
program, Bicchieri’s work on forward and backward induction [9] comes closest to ours. In [9], the players respond to(hypothetical) surprising moves in an extensive game (thatmay be the result of a trembling-hand mistake) by revising
their beliefs à la AGM [1]. Our models differ in both important technical details and the underlying motivations. Most
importantly, we downplay the role that the Nash equilibrium (and its refinements) plays in the analysis of rational
behavior in game situations (this is in line with much of theepistemic game theory literature, cf. [13]).
More recently, Cubitt and Sugden develop a model inwhich a player’s behavior may, in principle, differ from her(rational) choice [15]. They include a postulate stating thatthe players’ behavior must all conform to the same principles
of rational choice. Among other things, they are interested
in highlighting the role that this assumption plays in theplayers’ reasoning about what to do in a game situation (cf.
also Bacharach’s discussion of the transparency of reason in[3, Section 4.2]). There are some intriguing connections between our work and theirs, but a complete discussion willbe left for the full version of this paper.

discussed in relation to forward induction reasoning is theso-called rationalizability principle [8]: “A player should
always try to interpret her information about the behaviorof her opponents assuming that they are not implementing‘irrational’ strategies.” (cf. [6]). In order to represent thisbelief revision policy, Stalnaker includes a “richness” condition on his models [35, pg. 35, footnote 5] ensuring that theplayers have the conditional beliefs needed to rationalize any
observed behavior.17 With such a richness condition, we canformally prove Stalnaker’s characterization of the belief revision policy in which the players apply the rationalizability
principle at most once.18
Another direction for future research is to compare ourapproach to belief revision with non-standard probabilities,
lexicographic probability systems, and conditional probability systems [21, 25]. Once the relationship between thesedifferent models is understood, we can connect our workwith Battgalli and Siniscalchi’s characterizations of common strong belief of rationality [7] and Halpern’s recent epistemic characterizations of trembling-hand equilibria using
non-standard probabilities [20].
Finally, note that the games in Figures 1 and 2 have thesame reduced normal form. However, our analysis in this paper suggests that there are strategically relevant differences
between the two games (cf. [24]). In particular, the players
may be able to learn about their opponents’ strategies during a play of the game. This suggests possible connections
with models of learning in extensive games [17].

6.

5. CONCLUDING REMARKS
We have imposed only two minimal constraints on ourmodels: every information cell must include the player’s beliefs at all previous choice points, and the players “know”
their own strategy choice. The literature on forward induction, and, more generally, belief revision in games [5, 11,35], contains other natural constraints that we may want toimpose. One belief revision policy that has been extensively

159

REFERENCES[1] C. E. Alchourrón, P. Gärdenfors, and D. Makinson.
On the logic of theory change: Partial meet
contraction and revision functions. Journal of
Symbolic Logic, 50:510 – 530, 1985.
[2] R. Aumann. Backward induction and common
knowledge of rationality. Games and Economic
Behavior, 8(1):6 – 19, 1995.
[3] M. Bacharach. A theory of rational decision in games.
Erkenntnis, 27(1):17 – 55, 1987.
[4] A. Baltag and S. Smets. Conditioanl doxastic models:
A qualitative approach to dynamic belief revision. In
Electornic notes in theoretical computer science,
volume 165, pages 5 – 21. Springer, 2006.
[5] A. Baltag, S. Smets, and J. Zvesper. Keep ‘hoping’ forrationality: a solution to the backwards induction
paradox. Synthese, 169:301–333, 2009.
[6] P. Battigalli. On rationalizability in extensive games.
Journal of Economic Theory, 74:40 – 61, 1997.
[7] P. Battigalli and M. Siniscalchi. Strong belief andforward induction reasoning. Journal of Economic
Theory, 106(2):356 – 391, 2002.
[8] D. Bernheim. Rationalizable strategic behavior.
Econometrica, 52:1007 – 1028, 1984.
[9] C. Bicchieri. Rationality and Coordination (Cambridge
Studies in Probability, Induction, and Decision
17

This is related to Battigalli and Siniscalchi’s use of complete (conditional) type spaces in their characterization ofextensive-form rationalizability and related concepts. See [7,Section 6.2.3] for a discussion.
18
Stalnaker states and hints at a proof of such a theorem in[35, pg. 51, footnote 22].

Theory). Cambridge: Cambridge University Press,
1993.
[10] O. Board. Dynamic interactive epistemology. Gamesand Economic Behavior, 49(1):49–80, 2004.
[11] G. Bonanno. AGM belief revision in dynamic games.
In K. R. Apt, editor, Proceedings of the 13th
Conference on Theoretical Aspects of Rationality andKnowledge (TARK XIII), pages 37 – 45, 2011.
[12] G. Bonanno. A doxastic behavioral characterization ofgeneralized backward induction. Games and Economic
Behavior, 88:221 – 241, 2014.
[13] A. Brandenburger. Introduction. In The Language ofGame Theory: Putting Epistemics into the
Mathematics of Games. World Scientific Series in
Economic Theory, 2014.
[14] R. Cooper, D. DeJong, R. Forsythe, and T. Ross.
Forward induction in the battle-of-the-sexes games.
The American Economic Review, 83(5):1303 – 1316,
1993.
[15] R. P. Cubitt and R. Sugden. Common reasoning ingames: A Lewisian analysis of common knowledge ofrationality. Economics and Philosophy, 30:285 – 329,
2014.
[16] E. Dekel and M. Siniscalchi. Epistemic game theory.
In H. P. Young and S. Zamir, editors, Handbook ofGame Theory, Volume 4. Elsevier, 2014.
[17] D. Fudenberg and D. M. Kreps. Learning in
extensive-form games I. self-confirming equilibria.
Games and Economic Behavior, 8:20 – 55, 1995.
[18] S. Govindan and R. Wilson. Nash equilibrium,
refinements of. In S. N. Durlauf and L. E. Blume,
editors, The New Palgrave Dictionary of Economics.
Palgrave Macmillan, Basingstoke, 2008.
[19] J. Halpern. Substantive rationality and backward
induction. Games and Economic Behavior, 37(2):425 –
435, 2001.
[20] J. Halpern. A nonstandard characterization ofsequential equilibrium, perfect equilibrium, and proper
equilibrium. International Journal of Game Theory,3838(1):37–50, 2009.
[21] J. Halpern. Lexicographic probability, conditional
probability, and nonstandard probability. Games andEconomic Behavior, 68(1):155 – 179, 2010.
[22] J. Halpern and L. Rêgo. Extensive games withpossibly unaware players. Mathematical Social
Sciences, 70:42 – 58, 2014.
[23] A. Heifetz, M. Meier, and B. Schipper. Interactive
unawareness. Journal of Economic Theory, 130(1):78 –94, 2006.
[24] E. Kohllberg and J. Francois Mertens. On thestrategic stability of equilibria. Econometrica,
54(5):1003 – 1037, 1986.
[25] D. Lehman and M. Magidor. What does a conditional
knowledge base entail? Artificial Intelligence, 55(1):1– 60, 1992.
[26] M. J. Osborne and A. Rubinstein. A Course in Game
Theory. MIT Press, 1994.
[27] E. Pacuit and O. Roy. Epistemic foundations ofgames. In E. N. Zalta, editor, Stanford Encyclopedia of
Philosophy, 2015.
[28] A. Perea. Backward induction versus forward

induction reasoning. Games, 1(3):168–188, 2010.
[29] A. Perea. Epistemic Game Theory: Reasoning and
Choice. Cambridge UP, 2012.
[30] A. Perea. Belief in the opponents’ future rationality.
Games and Economic Behavior, 83:231 – 254, 2014.
[31] D. Samet. Common belief of rationality in games ofperfect information. Games and Economic Behavior,
79:192 – 200, 2013.
[32] R. Selten. A reexamination of the perfectness concept
for equilibrium points in extensive games.
International Journal of Game Theory, 4:25 – 55,
1975.
[33] R. Stalnaker. On the evaluation of solution concepts.
Theory and Decision, 37(1):49 – 73, 1994.
[34] R. Stalnaker. Knowledge, belief and counterfactual
reasoning in games. Economics and Philosophy,
12(02):133 – 163, 1996.
[35] R. Stalnaker. Belief revision in games: forward andbackward induction. Mathematical Social Sciences,36(1):31 – 56, 1998.
[36] R. Stalnaker. Extensive and strategic forms: Gamesand models for games. Research in Economics,
53(3):293 – 319, 1999.
[37] E. van Damme. Stable equilibria and forward
induction. Journal of Economic Theory, 48:476 – 496,
1989.

APPENDIXA.

PROOF OF PROPOSITION 1

In this appendix, we restrict attention to extensive games
G = hN, Act, H, τ, {ui }i∈N i without simultaneous moves.
So, for all decision nodes v ∈ V , |τ (v)| = 1. In this case,
we can view histories as sequences of actions rather thansequences of sequences of actions. Furthermore, following[2], we assume that the payoff for each of the players isdifferent at different terminal nodes (the game is in “general
position”). This implies that the result of applying the backward induction algorithm19 is uniquely defined.
Belief operators: Suppose that
MG = hW, {(βi , σi )}i∈N , {i }i∈N , {Pi }i∈N i
is a game model. For each event E ⊆ W , we say that player
i believes E, Bi (E), provided E is implied by i’s full beliefs.
That is, Bi (E) = {w | maxi ([w]i ) ⊆ E}.
Samet’s game model: Samet’s game model is a tuplehW, {Πi , ti }i∈N , si, where W is a non-empty set of states,for each i ∈ N , Πi is a partition on W , ti : W → ∆(W ) is a
type function assigning a probability measure to each state,and s : W → S (where S = Πi Si ) assigns a strategy toeach state. Let [si (w) = si ] be the set of states w such thatsi (w) = si . The knowledge and belief operators are definedas usual: for all E ⊆ W , Ki (E) = {w | Πi (w) ⊆ E} andBi (E) = {w | ti (w)(E) = 1}. Samet includes the following
constraints:
19

The terminal nodes are labeled with the payoffs for eachplayer. For each non-terminal history h with τ (h) = {i},label h with the maximum of all the labels of the successors of h. This labeling is then used to identify the so-calledbackward induction path.

160

• For all w ∈ W , if v ∈ Πi (w), then ti (w) = ti (v)• For all w ∈ W , ti (w)(Πi (w)) = 1• For all w ∈ W , if v ∈ Πi (w), then [si (w) = si ] ⊆Bi ([si (w) = si ]).

Rationality in Samet’s model: Building on the notation introduced in Section 2, for a strategy profile s, letOuth (s) be the (unique) terminal history that is reached if
the players follow their strategies in s starting at h. Then,for a state w ∈ W and strategy si ∈ Si , Outv (si , s−i (w))
is the terminal node that is reached, starting at h, if player
i follows the strategy si and the other players follow thestrategies associated with state w. Then, let[Outh (si , s−i ) >i Outh (s)]= {w | ui (Outh (si , s−i (w))) >ui (Outh (s(w)))}.

Player i is said to be doxastically substantively rational
at all states when:
Rids =\\h∈Vi si ∈Si

Let Rds =T¬Bi ([Outh (si , s−i ) >i Outh (s)])

dsi∈N Ri .

Common belief: Given belief operators Bi : ℘(W ) →℘(W ) for each player i ∈ N (defined in Samet’s game modelor our game model), we define a common belief operator
CB : ℘(W ) → ℘(W ) in the usual way.
T First, define everyone
believes: For all E ⊆ W , B(E) = i∈N Bi (E). Then define
the nth power of B, B n , as follows: for all E ⊆ W , B 1 (E) =B(E) and for n > 1, B n (E) = B(B n−1 (E)).
Finally, comT
mon belief of an event E is CB(E) = n≥1 B n (E)Samet’s Theorem 3 states that, in any of his models,CB(Rds ) ⊆ I, where I is the set of states in which the
backward induction path is played.
Suppose that MG = hW, {(βi , σi )}i∈N , {i }i∈N , {Pi }i∈N i
is a game model for our game G. The forgetful projection of MG , denoted M◦G , is the tuple hW, {Πi , ti }i∈N , si,
where for each w ∈ W , let Πi (w) = [w]i , ti (w) = Pi,w , ands(w) = (σ1 (w), . . . , σn (w)). It is not hard to see that M◦Gsatisfies the constraints imposed by Samet. For instance, we
have, for all w0 ∈ Πi (w), ti (w) = ti (w0 ), since if w0 ≈i w,
then maxi ([w]i ) = maxi ([w0 ]i ).
We first state and prove a simple Lemma that will be used
to relate Samet’s notion of doxastic substantive rationality
with our rationality-1.
Lemma 1. Suppose that the game G and game model MG
and state in w satisfy the assumption of Proposition 1. Then,for all players i ∈ N , for all w0 ∈ W , if w ∈ Rat1i , thenfor all v ∈ Vi , there is some w0 ∈ maxi ([w]i ) such thatui (Outh (s(w0 )) > ui (Outh (si ; s−i (w0 ))).
Proof. First of all, it is easy to see that if a strategy si is
optimal for player i at state w, then for all strategies ti 6= si ,there must be at least one state w0 ∈ maxi ([w]i ∩[M ovewi =
βi (w)) such that ui (Out(si ; s−i (w))) > ui (Out(ti ; s−i (w))).
Suppose that w ∈ Rat1i . Then, for all h ∈ Vi , if βi (w)his defined (i.e., h on the path generated by the behaviorof the players in state w), then for all si ∈ Si (w), there
is at least one state w0 ∈ maxi ([w]i ∩ [hw ]) such thatui (Outh (s(w))) > ui (Outh (si ; s−i (w))). Note that we can161move from Out(·) to Outh (·) since we restrict attention to
strategy profiles that conform to the behavior of the players at w. By the constraint stated before Proposition 1,
this implies that there is a w00 ∈ maxi ([w]i ) such thatui (Outh (s(w00 ))) > ui (Outh (si ; s−i (w00 ))). This, together
with the assumption that all mistakes are realized by somestate in i’s information cell, ensures that, for every decisionnode h ∈ Vi , there is some w0 ∈ maxi ([w]i ) such thatui (Outh (s(w0 ))) > ui (Outh (si ; s−i (w0 ))).

This completes the proof of the Lemma.
The proof of the proposition follows immediately:
Proof of Proposition 1. Suppose that w ∈ W and(β1 (w), . . . , βn (w))Tgenerate a maximal path through thegame. If w ∈ CB( j Rat1j ) in MG , then Lemma 1 implies
that w ∈ CB(Rds ) in the forgetful projection M◦G . Since
M◦G is a Samet model of a game, Samet’s Theorem 3 implies that w ∈ I. Since no mistakes are made in w, this
implies that (β1 (w), . . . , βn (w)) is the backward inductionpath.

Standard State Space Models of Unawareness∗[Extended Abstract]
Peter Fritz

Faculty of Philosophy, University of Oxford
Jesus College, Turl Street
Oxford OX1 3DW, UK

peter.fritz@philosophy.ox.ac.uk

ABSTRACTThe impossibility theorem of Dekel, Lipman and Rustichini[8] has been thought to demonstrate that standard statespace models cannot be used to represent unawareness. We
first show that Dekel, Lipman and Rustichini do not establish this claim. We then distinguish three notions of awareness, and argue that although one of them may not be adequately modeled using standard state spaces, there is no reason to think that standard state spaces cannot provide models of the other two notions. In fact, standard space modelsof these forms of awareness are attractively simple. They allow us to prove completeness and decidability results withease, to carry over standard techniques from decision theory,and to add propositional quantifiers straightforwardly.

Keywords
awareness, standard state space models, epistemic logic

1. INTRODUCTION
Dekel, Lipman and Rustichini [8], hereafter, “DLR”, claim
to show that “standard state space models preclude unawareness”. Their claim has achieved the status of orthodoxy.1 The
first task of this paper is to clear the way for standard state
space models of unawareness by showing that the formal
result DLR present does not establish their headline conclusion. DLR informally motivate certain axioms concerning
unawareness, but in their formal impossibility result, they
rely on the claim that these axioms hold at all states in
the model. As section 2 argues, the assumption that axioms
hold at all states of the model is unwarranted; in fact, DLR
themselves reject it. While DLR’s formal results are valid,
they are not sufficiently general to rule out standard state
space models of unawareness. As we show, the impossibility
results do not hold if one assumes only that DLR’s explicit
assumptions about unawareness hold at some “real” states,
as opposed to at all states. Even strengthening those explicit
assumptions considerably does not reinstate the results.
But this does not yet vindicate standard state space models of unawareness. Section 3 presents a novel impossibility
result which uses widely shared assumptions about awareness. The new impossibility theorem relies on the assumption that an agent who is aware of a conjunction is aware∗A draft of the full version of this paper is available at
http://users.ox.ac.uk/~hert2388/
1
See, e.g., [53, p. 2], [54], [24, p. 78], [25, p. 305], [26, p. 101],[40, p. 220], [31, p. 2790], [39, pp. 977–978], [27, p. 2454], [28,p. 257], [15, p. 42], [58, p. 516], and [57].

163

Harvey Lederman

Department of Philosophy, New York University
5 Washington PlaceNew York, NY 10003, USA

hsl306@nyu.edu

of its conjuncts. If awareness satisfies this assumption, thenstandard state space models do in fact preclude unawareness. We then distinguish three notions of awareness, andsuggest that two important ones do not satisfy this assumption, leaving open the possibility that they could be adequately modeled by standard state space models.
The remainder of the paper continues in a more positivevein. We describe a simple class of standard state space models which represent key features of awareness. In section 4, weestablish completeness and decidability for the logic of thesemodels. We also show that adding propositional quantifiers,
a topic which has presented major difficulties for existing
approaches to awareness, is straightforward in our standard
state space models. In section 5, we present one way of implementing a choice-based approach to decision theory withinthese models, and show how non-trivial unawareness is consistent with speculative trade. Section 6 concludes.

2.

DLR’S TRIVIALITY RESULT

2.1

Standard State Space Models

Standard state space models for the knowledge and awareness of a single agent can be understood as certain tupleshΩ, k, ai. Ω is required to be a set, called the set of states,from which a set of events is derived by taking an event tobe a set of states. k and a are functions on events, which represent the agent’s knowledge and awareness, respectively: kmaps each event E to the event k(E) of the agent knowing
E; a similarly takes each E to the event a(E) of the agent
being aware of E.
Such models are straightforwardly used to interpret a formal language in which one can talk about knowledge andawareness. Let L be such a language built up from proposition letters p, q, . . . , using a unary negation operator ¬, abinary conjunction connective ∧ and two unary operators
K and A, respectively ascribing knowledge and awarenessto the agent. Formulas of this language are interpreted relative to a model M = hΩ, k, ai and a valuation function v
which maps each proposition letter p to the event v(p). Theinterpretation uses a function J·KM,v which maps each formula ϕ of L to the event expressed by ϕ in M , which canbe understood as the set of states in which ϕ is true in M .
To state the constraints on such a function let −E = Ω\E.
JpKM,v = v(p)J¬ϕKM,v = −JϕKM,vJϕ ∧ ψKM,v = JϕKM,v ∩ JψKM,v

JKϕKM,v = k(JϕKM,v )JAϕKM,v = a(JϕKM,v )The agent being unaware of something can of course be understood as it not being the case that she is aware of it.
We therefore syntactically use U ϕ as an abbreviation for¬Aϕ. Similarly, we introduce the other connectives of classical propositional logic as abbreviations, using ∨ for disjunction, → for material implication, ↔ for bi-implication,and > and ⊥ for an arbitrary tautology and contradiction,
respectively. On the semantic side, we adopt the conventionof writing f g for the composition of functions f and g, whichallows us two write, e.g., k − a(E) instead of k(−(a(E))).
In order to express general constraints on these models,
we say that a formula ϕ is valid on M if JϕKM,v = Ω for eachvaluation function v; this can be understood as requiring ϕ
to be true in every state of M according to every valuation
function. In order to limit this constraint to a particularstate ω ∈ Ω, we say that ϕ is valid in ω if ω ∈ JϕKM,v foreach valuation function v.
These models count as “standard” in the sense of DLR.
First, the events expressed by Aϕ and Kϕ are each a function of the event expressed by ϕ. (DLR call this “eventsufficiency”.) Second, negation is interpreted as set-complement and conjunction as intersection, so that all tautologies
of classical propositional logic, such as p ∨ ¬p, are interpreted as the set of all states in every model. (DLR call thisassumption “real states”.)

2.2

DLR on Standard State Space Models

DLR introduce three constraints on awareness, which canbe stated using the following three axioms:
Plausibility: U p → (¬Kp ∧ ¬K¬Kp)
KU -Introspection: ¬KU p
AU -Introspection: U p → U U p
Their constraints on knowledge can be stated using the following three axioms:
Necessitation: K>
Monotonicity: K(p ∧ q) → (Kp ∧ Kq)
Weak Necessitation: Kp → K>
Their main results are then:Theorem 1 (DLR). Let M = hΩ, k, ai be a model on
which Plausibility, KU -Introspection and AU -Introspection
are valid.
1(i) If Necessitation is valid on M , then Ap is valid on M .
1(ii) If Monotonicity is valid on M , then Kp → Aq is valid
on M .
2 If Weak Necessitation is valid on M , then Kp → Aq andAp ↔ Aq are valid on M .
Our presentation of DLR’s result differs in superficial respects from their original presentation. DLR do not present
their constraints in terms of the validity of certain axioms.
Thus, for example, instead of requiring KU -Introspection to
be valid on M , they require k − a(E) = ∅ for all events E.

However, it is a routine exercise to show that this condition is equivalent to the validity of our corresponding axiom. The same point holds for the other axioms. In short,
our later models will not be escaping their triviality result
by a sleight of hand which depends on this presentation.
One reason for the variant presentation is that it will facilitate the later exposition. It also serves to demonstrate thatstandard state space models as discussed here are equivalent to what are now commonly known as neighborhood or
Scott-Montague frames (see [55] and [44]). It is well known
that given certain restrictions on the function interpretingknowledge, this function can be turned into a binary relation
among states along the lines of those used by [37] and [29].
This representation as a binary relation is, in turn, formallyinterchangeable with the “possibility correspondences” introduced by [2] (see also [3]) and used throughout economic
theory (see, e.g. [11]).

2.3

Two Kinds of States

In response to their triviality results, DLR suggest distinguishing informally between “real” states and “subjective”
states. As we understand it, this distinction can be explained
as follows. An epistemic model makes predictions about how
an agent or group of agents will or would behave in particular situations. The model makes predictions about thesesituations by including states which represent them. The
real states in a model are the states which represent situations the model is intended to describe. The model predicts
an agent will behave a certain way in a particular situation
just in case the agent behaves that way in the real state
which represents that situation. The predictions of a single
model are given by what holds at all its real states; the behavioral theory of a class of models is given by what holds
at all real states in all its models.
A state in an epistemic model is subjective if it figuresin the specification of what the agent knows or is aware ofat some real state. According to this way of understanding
real and subjective states, states may be both real and subjective. Suppose we wish to represent an agent who knowsthat a particular coin will be flipped, but who will not learn
the outcome of this coin flip. If our model is intended tomake predictions no matter how the coin lands, the subjective states needed to specify the agent’s knowledge (heads
and ignorant, tails and ignorant) will be exactly the realstates; every state will be both real and subjective. But as
DLR recognize, there is no reason to require all states toplay both roles. In the earlier example, if we only wanted
to make predictions about the situation in which the coincomes up heads, we would not count one of the states (tails
and ignorant) as a real state; even so, to represent the agent’s
ignorance given heads, a state where the coin comes up tails
would still have to be included as a subjective state. The
point can also be illustrated with a less artificial example.
Consider an analyst who wishes to model the interactions ofagents who are rational, but who do not believe each otherto be rational. To represent the beliefs of these agents, theanalyst must include subjective states in which the agentsare irrational. But although she includes these subjective
states, the analyst has no intention of eliminating the claimthat the agents are rational from the predictions of her theory. Rather, it is understood informally that these subjective
states are not real; they do not represent situations the analyst aims to describe.

164

Put in the terms of our presentation, DLR’s proposal is toallow subjective states in which the law of excluded middle,p ∨ ¬p, may not be true. DLR have no intention of eliminating the law of excluded middle from the predictions of
their theory. Rather, they introduce these subjective statesto specify the agent’s knowledge and unawareness at realstates where classical logic holds. The theory of DLR’s models is given by what holds at these classical real states, notby what holds at all states whatsoever. Still, since classicaltautologies may fail in DLR’s subjective states, their models
violate the “real states” assumption, and so are not standard
state space models.
But once we acknowledge that some states may not be intended to represent situations the analyst wishes to describe,a question arises: Why should one require DLR’s three axioms on unawareness to be valid in all states? DLR do notargue for this assumption. At best, their arguments in favorof their axioms only motivate imposing these axioms at realstates. These arguments provide no motivation for imposing the axioms on subjective states that are not real, sincethese states are merely included in the model to specify theagent’s knowledge and unawareness at real states.
There is a general methodological principle in epistemic
modeling that axioms are to be imposed at all states. But
in the literature on awareness, following DLR, this methodological principle has long been abandoned. DLR’s own nonstandard models violate this requirement, as do the current
leading proposals for representing awareness, for example
that of [24]. The subjective states in DLR’s models include
states in which logical axioms, including the law of excluded
middle, do not hold. In DLR’s models, as in the ones wewill propose, even logical axioms are allowed to fail at somestates. The only difference between our proposal and theirs
concerns which axioms are allowed to fail. DLR preserve
their own axioms at all states, and move to non-standard
models in which classical propositional logic may fail at subjective states. We will preserve classical propositional logic
at all states, and work with standard models in which DLR’s
axioms may fail at subjective states.
Still, one might ask: How should we understand a state
where DLR’s axioms are false? DLR interpret their own subjective states as “descriptions of possibilities as perceived by
the agent” (p. 171). This interpretation does not seem appropriate for our models, in which DLR’s axioms may fail at
subjective states. But such metaphorical interpretations ofthese states are unnecessary. Subjective states where the axioms of awareness are invalid are simply to be understood interms of the agent’s knowledge and awareness at real states
where the axioms are valid.
In fact, we can give a direct argument for not imposing
DLR’s axioms at all states, and in particular, for including states where KU -Introspection is invalid. First, by AU Introspection, if an agent is unaware of p, then she must be
unaware of being unaware of p. But then, by Plausibility,
the agent does not know that she does not know that she isunaware of p. (Essentially, DLR already give this argument
on p. 169.) In epistemic models, we generally represent anagent’s not knowing q by including a state in which q is false.
So, to allow for real states in which the agent does not knowthat she does not know that she is unaware of p, we must
include subjective states in which the agent knows that she

165

is unaware of p, and so violates KU -Introspection.2
To sum up: after rejecting standard state space models,DLR propose that we should use models in which the laws oflogic fail at subjective states. They implement this proposal
by countenancing states where propositional logic fails, sothat their models are non-standard. But if we allow models
in which the law of excluded middle may fail at subjective
states, we must also consider models in which other axioms,
including DLR’s, may fail at subjective states. DLR’s formal
results only apply to standard state space models in which
their axioms are imposed at all states; the results do notconcern standard state space models in which the axioms are
imposed only at real states. As a consequence, these formal
results cannot provide a basis for the conclusion DLR draw:that standard state space models preclude unawareness.

2.4

Non-Triviality

We have not argued against the validity of DLR’s threeaxioms in real states – states representing the situations to
be modeled. In our setting, we can formalize the distinction
between real states and subjective states which are not real,by only assuming the validity of the axioms in some subsetof the states in a model. We can then ask: is assuming thevalidity of the three axioms in such distinguished real states
enough to lead to triviality? The following example showsthat it is not:
Theorem 2. There is a model M = hΩ, k, ai, state α ∈Ω and event E ⊆ Ω such that Necessitation is valid on
M , Plausibility, AU -Introspection and KU -Introspection are
valid in α, and α ∈/ a(E).
Proof. Let Ω = {α, ω1 , ω2 }. Define a binary (accessibility) relation R on Ω as follows:
α

$ 

ω1


 z

ω2

R induces a possibility correspondence P such that P (σ) ={τ : Rστ }. With P , define k and a such that for all F ⊆ Ω:k(F ) = {σ ∈ Ω : P (σ) ⊆ F }({ω2 } if ω1 ∈ F and ω2 ∈/F
a(F ) =Ω
otherwise
It is routine to check that M = hΩ, k, ai, α and E = {α, ω1 }witness the claim to be proven.3
This shows that DLR’s Theorem 1(i) cannot be extended
to standard state space models in which DLR’s three axioms
are only required to be valid in real states. In fact, the model
used in the above proof of Theorem 2 can also be used to2This argument differs from DLR’s main proof of triviality,
since it only assumes that the axioms hold at the real state.
3
As suggested in [43], one might consider an extension ofPlausibility along the following lines: For each natural number n, let n-Plausibility be U p → (¬K)n p, where (¬K)n ϕ
is defined inductively by the two clauses (¬K)1 ϕ = ¬Kϕand (¬K)n+1 ϕ = ¬K(¬K)n ϕ. For each natural number n,n-Plausibility is valid in α.

show that DLR’s other two results cannot be extended either. For 1(ii), note that Monotonicity is valid on M , andthat α ∈ k(Ω) although α ∈/ a(E). For 2, note first that Weak
Necessitation is valid on M , and that α ∈ a(Ω) but as beforeα ∈ k(Ω) and α ∈/ a(E). More generally, any state in any
model which satisfies both Necessitation and Monotonicity,in addition to DLR’s three axioms, will be a counterexample
not just to extensions of DLR’s Theorem 1(i), but also toextensions of their Theorems 1(ii) and 2.
We conclude that none of DLR’s three triviality results
show that standard state space models preclude unawareness. One might wonder whether plausible strengtheningsof the axioms on knowledge and unawareness allow us toreinstate the triviality results. In the full paper, we argue
first that this cannot be achieved by strengthening their axioms governing knowledge, and, second, that it cannot be
achieved by a particular strengthening of the axioms governing unawareness. The theorems and proofs may be foundin Appendix A.

discussions in [10]; another detailed discussion can be foundin [54]. We will argue that that at least some important notions of awareness do not satisfy AC (for others, as we willsee, the situation is more complex).. In order to do so, weroughly distinguish the following three ways of understanding a claim of the form “The agent is aware of . . . ”:(i) The agent is attending to . . .
(ii) The agent has the conceptual resources required toconceive of . . .
(iii) The agent is able to process . . .
We will introduce these notions – and distinguish between
them – using various examples found in the literature.
Consider first attention. An influential example, which
first appeared in [16], and which is discussed at length by
DLR and numerous places in the subsequent literature, isbased on the following quote from one of Arthur Conan
Doyle’s Sherlock Holmes stories [9]:3. THREE KINDS OF AWARENESS
3.1“‘Is there any other point to which you would
wish to draw my attention?’‘To the incident of the dog in the night-time.’‘The dog did nothing in the night-time.’‘That was the curious incident’ remarked Sherlock Holmes.”

A New Triviality Result

DLR’s result had limited implications for state space models because it depended on the validity of their axioms at allstates. Is there a triviality result which only uses the validityof axioms on awareness in real states, rather than their validity in all states? In fact, as we now show, widely accepted
axioms on awareness do lead to triviality even if they are
imposed only at real states. The result uses the following
two axioms:
AS: A¬p → Ap
AC: A(p ∧ q) → (Ap ∧ Aq)
Awareness is widely assumed to satisfy both of these axioms;
see, e.g., [43, pp. 274–275], [18, p. 331] (axioms A1 and A2)
and [25, p. 309] (axioms 1 and 2).
As the next theorem shows, these axioms lead straightforwardly to triviality.
Theorem 3. Let M = hΩ, k, ai be a model and α ∈ Ωsuch that AS and AC are valid in α. Then Ap → Aq is valid
in α.

Holmes’s interlocutor is Inspector Gregory, a Scotland Yard
detective. Before Holmes pointed out to Gregory that thedog did nothing in the night-time, Gregory was unaware of
the dog doing nothing in the night-time. Gregory’s state ofunawareness is naturally understood as one of inattention– Holmes makes Gregory aware of the dog doing nothing
in the night time in the sense of bringing this fact to his
attention.
Gregory’s failing to attend to the dog doing nothing in thenight-time must be sharply distinguished from Gregory’s not
being able to conceive of the the dog doing nothing in thenight-time. Before Holmes alerted Gregory to the dog doing
nothing in the night-time, Gregory possessed the concepts
required to entertain thoughts about the dog doing nothing
in the night time. Contrast this with the following example
for unawareness from [10, p. 40]:
“How can someone say that he knows or doesn’tknow about p if p is a concept he is completely
unaware of? One can imagine the puzzled frownof a Bantu tribesman’s face when asked if heknows that personal computer prices are going
down!”

Proof. Consider any events E and F , and assume α ∈a(E). Since E = E ∩ Ω, α ∈ a(E ∩ Ω), and so by AC,α ∈ a(Ω). By AS, α ∈ a(∅). Since ∅ = ∅ ∩ F , α ∈ a(∅ ∩ F ),and so by AC, α ∈ a(F ).
The crucial difference between this and DLR’s triviality
result is that AS and AC are only assumed to be valid ina distinguished state, for which it is shown that non-trivial
unawareness in it is ruled out.
But does awareness really satisfy both AS and AC? In thefollowing, we will focus in particular on AC, arguing thatfor some important notions of unawareness, being aware ofa conjunction does not entail being aware of its conjuncts.

3.2

Attending vs. Conceiving vs. Processing

In the literature on awareness, it is uncontentious that
there is no single attitude of awareness; what is expressedby “aware” is a loose cluster of notions. This was noted atthe very start of the literature, as witnessed by the lengthy

The relevant state of unawareness in this example is not
merely a matter of the agent failing to attend to the relevant
event or subject matter. For example, if one is unaware in thesense of being unable to conceive of an event, it must be that
one does not understand the words for those notions in any
language. Contrast this with the case of Inspector Gregory.
Gregory understands what Holmes says: he can conceive ofthe dog’s doing nothing. But the purported example of inconceivability does not have this structure: the tribesman is
supposed to be unable to think about computers using anyof his conceptual resources, no matter what he attends to.
The two notions of awareness – attending to versus being
able to conceive of – are therefore clearly distinct.

166

The third notion of unawareness we want to single out is
one which [10] (see also [19] and [20]) focus on; it can beunderstood as an attempt to deal with what is known as
the “problem of logical omniscience” in epistemic logic. In
standard state spaces, if two sentences ϕ and ψ are equivalent in classical propositional logic, then Kϕ and Kψ willbe true in the same states. In particular, if K(p ∨ ¬p) is truein a given state, then so is Kτ for any propositional tautology τ . One of Fagin and Halpern’s reasons for developing a
logic of awareness is to obtain logics which do not have thisproperty. They write:“The notion of awareness we use in this approachis open to a number of interpretations. One ofthem is that an agent is aware of a formula if hecan compute whether or not it is true in a given
situation within a certain time or space bound.
This interpretation of awareness gives us a wayof capturing resource-bounded reasoning in ourmodel.”
Being unaware of ϕ in the sense of not being able to process ϕ is clearly distinct from failing to attend to ϕ: although Gregory did not attend to the dog doing nothing in
the night-time, he had no difficulties processing the claimthat the dog did nothing in the night-time. Not being able
to process ϕ is also clearly distinct from not being able toconceive of ϕ: Gregory might not have been able to process
an extremely complicated propositional tautology using only
negation, conjunction and the sentence “the dog did nothing
in the night-time”, but he clearly possessed all the concepts
required to entertain it.

3.3

Awareness of Conjunctions

Let’s return to the new triviality result introduced at thestart of this section. As already advertised, we believe thatthe principle AC, which says that an agent who is aware ofa conjunction is aware of its conjuncts, may be plausible for
one notion of awareness, but it is not for the other two.
Consider first awareness as the ability to process. This isplausibly a relation of agents to sentences, as part of what
it takes to process . . . is to be able to find out what thesentence “. . . ” means. AC may hold if awareness is understood as the ability to process: It is natural to assume thatan agent who is able to process a conjunction “... and —” isalso able to process “...” and “—”. As noted already in [10,p. 54], even this may fail: an agent might be able to recognize that a very long sentence has the form ϕ∧¬ϕ, and so beable to process it, although she is unable to process the complex ϕ on its own. Resolving this controversy may requiredistinguishing further among different notions of processing,and the appropriate resolution may depend on the intended
application.
The relations of attention and conceivability are different
from the ability to process. In particular, they are plausibly
relations agents have not to sentences but to what sentences
express. We might call these entities contents or propositions, but in keeping with the terminology employed above,
we call them events. In the full paper, we discuss the difference between sentences and events in more detail, and motivate the assumption of a coarse-grained theory of events,
according to which events form a complete atomic Boolean
algebra. A consequence of this assumption – which is implicit in all standard state-space based modeling techniques167– is that sentences which are equivalent in propositional logic
have identical contents.
With this understanding of attention and conceivability
as relations of agents to coarse-grained events, consider firstattention. Assume that after the conversation with Holmes
quoted above, Gregory is alone thinking about the case, andattending to the event of the dog barking in the night (p).
He is not, however, attending to event of Holmes at thatmoment smoking a pipe (q). It is then natural to say that
Gregory is also not attending to the conditional event that if
Holmes is currently smoking a pipe then the dog barked in
the night (q → p). But notice that according to the coarsegrained Boolean theory of events, the event that the dog
barked in the night (p) is identical to the event that if Holmes
is smoking a pipe then the dog barked in the night, andthe dog barked in the night ((q → p) ∧ p). So if AC werevalid, then since Gregory is attending to the event of thedog barking in the night, he would be attending to the eventthat if Holmes is smoking a pipe then the dog barked in thenight. But by assumption Gregory is not attending to thislast event. Thus it follows from the coarse-grained theory of
events that AC must be rejected.
A similar example can be given if we understand awareness as conceivability. Assume our agent does not have theconceptual resources to entertain the event of there being a
black hole. According to the assumed coarse-grained theoryof content, the event of there being a black hole and there
being no black hole is identical to any event expressed bya propositional contradiction, such as the event of there being a sheep and there being no sheep. The agent might wellhave the conceptual resources to entertain the event of there
being a sheep and there being no sheep, without having theconceptual resources to entertain the event of there being a
black hole.
If we adopt, as usual, a theory of events which identifies
the event expressed by sentences which are equivalent inpropositional logic, AC appears to be inappropriate. Thus
the new triviality result with which we started this section
also does not establish that standard state space models
preclude unawareness understood as inattention or inability
to conceive.

4.

PARTITIONAL MODELS

So far, we have shown that standard state-space models
escape certain putative impossibility results for models of attention and conceivability. But this does not establish thatstandard space models can provide fertile models of thesenotions. In the remainder of the paper, we define, motivate,
and examine a class of standard state space models for representing attention and the ability to conceive.
To show that our models generalize smoothly to the multiagent case, from now on we use a language LI parametrized
to an arbitrary set of agent-indices I which is defined as thelanguage L above, except that the operators Ai and Ki areindexed to i ∈ I. Models are consequently tuples of the formhΩ, ki , ai ii∈I .
The models we will be working with are defined as follows:
Definition 1. hΩ, Ri , ≈i ii∈I is a partitional model if Ω
is a set and for each i ∈ I, Ri is a binary relation on Ω
which is reflexive and transitive, and ≈i a function whichmaps each ω ∈ Ω to an equivalence relation ≈iω on Ω.
Here and in what follows, we make use of the fact that eachequivalence relation corresponds to a unique partition, andvice versa; accordingly, we treat them as interchangeable.
Partitional models can be used to generate standard models in the following way: Ri specifies states of knowledge justas in Theorem 2. The idea behind ≈i is that the events theagent is aware of at ω are the events which are unions of setsof equivalence classes of ≈iω (equivalently: unions of sets ofcells of the induced partition). So for each i ∈ I, let Ri and≈i determine functions ki and ai on events on Ω as follows:ki (E) = {σ ∈ Ω : P i (σ) ⊆ E}, where P i (σ) = {τ : Ri στ }≈4 : {1}, {4}, {2, 3}Thus, at 1, the strongest event known by the agent is Ω, andat each other state n, it is {n}. At each state n, the events
the agent is aware of are the events which don’t distinguishbetween any states in Ω\{1, n}.
Drawing the four states in a circle, starting with 1 at the
top and going clockwise, we can draw each equivalence relation in a similar smaller circle, connecting two states by asequence of lines if they are related by the relevant equivalence relation:ai (E) = {σ ∈ Ω : for all ρ and τ such that ρ ≈iσ τ , ρ ∈ E iffτ ∈ E}
Let the standard model determined by a partitional modelhΩ, Ri , ≈i ii∈I be hΩ, ki , ai ii∈I , with ki and ai as just defined.
On such a standard model, LI can be interpreted as above;
obviously, this induces a way of interpreting LI directly on
partitional models.

4.1

The Attitude of Attention

In order to motivate partitional models as models of limited attention, we suggest that attention in the sense wehave been using the term should primarily be understoodas an attitude towards questions. There are many available
formal approaches to modeling questions (for an overview,see [35]). For concreteness, we’ll adopt a standard approach,representing questions as partitions of the state space (see[17], building on [23] and [32]). Although we think the attitude to questions is primary, we will follow the literature
on awareness, in axiomatizing a notion of attention which
has events as its objects. The relationship between this attitude to events and the attitude toward questions will be
as follows: an agent attends to the question Q if and only if
the agent attends to every partial answer to Q. Using partitions to model questions, partial answers are unions of setsof cells, corresponding to how standard models are derived
from partitional models.4

4.2

This is a partitional model in which there is non-trivial unawareness at each state. We will appeal to it below in orderto show the consistency of various constraints.

4.4

Theorem 4. A formula is valid on all partitional models
if and only if it is derivable in the calculus with the following
axiom schemas and rules:
PL: Any substitution instance of a theorem of propositional logic.

Partitions for Conceivability

K-K: Ki (ϕ → ψ) → (Ki ϕ → Ki ψ)

To motivate the use of partitional models of conceivability, assume that the agents to be modeled have the conceptof negation and (infinitary) conjunction, so that the set of
events they can conceive of are closed under complement
and arbitrary intersection. This is mathematically equivalent to requiring that this set is derived from an equivalence
relation as above.

4.3

K-T: Ki ϕ → ϕ
K-4: Ki ϕ → Ki Ki ϕ
A-Neg: Ai ϕ → Ai ¬ϕ
A-M: (Ai ϕ ∧ Ai ψ) → Ai (ϕ ∧ ψ)

An Example

It will be useful to have a concrete partitional model before us, as a running example. The following model showsthat there are non-trivial partitional models; for simplicity, asingle-agent case is specified. Let M = hΩ, R, ≈i, with ωRνiff ω = 1 or ω = ν, and ≈ given by the following equivalenceclasses:AxiomsGiven a class of models C, a set of sentences Σ ⊆ LI is
the logic of C if and only if Σ contains exactly those sentences which are valid on C. Characterizing the logic of aclass of models gives us a formal perspective from which toassess what assumptions our models encode about agentsknowledge and awareness.
Thus we may ask: What is the logic of partitional models?
Standard techniques on completeness results in modal logic
are easily adapted to obtain the following result.

A-N: Ai >K-RN: From ` ϕ infer ` Ki ϕ
A-RE: From ` ϕ ↔ ψ infer ` Ai ϕ ↔ Ai ψ
Moreover, the logic is decidable.
A proof is given in Appendix B.

≈1 : {1}, {2, 3, 4}≈2 : {1}, {2}, {3, 4}≈3 : {1}, {3}, {2, 4}

4

Section 5 discusses related models developed in [33, 34].

4.5

DLR Once More

Consider again DLR’s three axioms. Given our discussion
above, it is natural to consider partitional models where
DLR’s axioms are required to be valid in some distinguishedstate:

168

Definition 2. hΩ, α, Ri , ≈i ii∈I is a partitional DLR model if hΩ, Ri , ≈i ii∈I is a partitional model and Plausibility,
KU -Introspection and AU -Introspection (for each i ∈ I) are
valid in α.
We now show that DLR’s triviality result cannot be revived in partitional models:Theorem 5. There is a partitional DLR model hΩ, α, Ri ,≈ ii∈I and an event E ⊆ Ω such that α ∈ U (E).
i

Proof. Simply distinguish state 1 in the model presented
in section 4.3.

We conjecture that the logic of partitional DLR models
can be axiomatized as follows:
Conjecture 1. Add the following axioms to the theorems of the axiom system in Theorem 4 and close undermodus ponens:
P: Ui ϕ → (¬Ki ϕ ∧ ¬Ki ¬Ki ϕ)
AU: Ui ϕ → Ui Ui ϕ
A formula is derivable in this calculus if and only if it isvalid in every distinguished state of every partitional DLR
model.
Note that ¬KU ϕ can be derived using P, AU and K-T.
The present result shows that we can impose the DLR axioms without trivializing partitional models. But we confessto doubts about whether these axioms are appropriate. Just
as with AC, once we understand more clearly the character of attention and conceivability, as well as the distinction
between sentences and what they express, DLR’s axioms become much less compelling. The clearest case concerns Plausibility and attention. A consequence of the contrapositiveof Plausibility is Kϕ → Aϕ. But this principle is false forattention. You know that there are more than four stars in
the universe, but we doubt that you were attending to thequestion of how many stars there are prior to reading theprevious clause. As we discuss in more detail in the full paper, the coarse-grained conception of content together withclarity about the notion of awareness to be modeled cast
doubt on DLR’s axioms.

4.6

Propositional Quantification

A challenge to some approaches to unawareness is to represent propositionally quantified statements. E.g., earlier models by Halpern made the claim that the agent knew she wasunaware of something unsatisfiable (cf. [21] and [22]). In
standard state space models such as ours, it is trivial to add
propositional quantifiers without any such consequences. To
do so, we write v[E/p] for the valuation function which maps
p to E and every other proposition letter q to v(q):5T
J∀pϕKM,v = E⊆Ω JϕKM,v[E/p]
To illustrate that these quantifiers behave just as one would
expect, note that in state 1 of the example described in
section 4.3, the agent knows that she is unaware of something
without there being something that she knows to be unawareof: K∃pU p ∧ ¬∃pKU p is true in this state.

5
See already [36], and [12] for a more systematic development. See [13] for results on the complexity of propositional
quantifiers in the related setting of [14].

169

4.7

Closure and Automorphisms

In partitional models, what agents are aware of (attend
to/can conceive) is closed under negation and conjunction.
One might wonder whether we can also impose the constraints that what agents are aware of must be closed underawareness and knowledge. In other words, whether there aremodels on which the following axioms are valid:A-4ij Ai p → Ai Aj p
AK-4 Ai p → Ai Kj p
To provide models which validate these principles we adapt
the coherence constraint of [14].6 The idea behind it is mosteasily described for awareness as conceivability, taking theequivalence relations of partitional models to represent a
relation of indistinguishability using conceptual resources
available to the relevant agent at the relevant state. Coherence requires that if two states are indistinguishable inthis way, then there must be a way of permuting the state
space in a way which preserves all structural facts aboutknowledge and awareness, as well as all the events which
the relevant agent is aware of at the relevant state.
Let M = hΩ, Ri , ≈i ii∈I be a partitional model. A permutation f of Ω is an automorphism of M if for all i ∈ I,(i) for all x, y, z ∈ Ω, y ≈ix z iff f (y) ≈if (x) f (z), and(ii) for all x, y ∈ Ω, Ri xy iff Ri f (x)f (y).
A state x ∈ Ω coheres if for all i ∈ I and y, z ∈ Ω such thaty ≈ix z there is an automorphism f of M such that f (y) = z
and f ⊆ ≈ix (i.e., ω ≈ix ω for all ω ∈ Ω). It’s routine toverify that A-4ij and AK-4 are valid in any coherent stateof a partitional model.
Once again, the model presented in section 4.3 demonstrates the satisfiability of this constraint: every state in thismodel is coherent. Since the model also satisfies the DLR axioms at state 1, it shows that even if we were to uphold the
DLR axioms, imposing them together with coherence would
not trivialize state space models of awareness.

5.

DECISION THEORYIn section 4.5 the example of the number of stars illustrated how one may believe and know things to which one is
not attending; clearly this kind of inattention may also affectchoice-behavior. One advantage of standard state spaces isthat we can use the usual decision-theoretic framework to
represent the effects of inattention on choice-behavior.7
The usual decision theoretic representation of an agent’s
beliefs is given by a measure-space hS, B, µi.8 To generatea partitional model, we enrich this description of the agent
by selecting BC , a complete atomic subalgebra of B, to generate a representation of what the agent attends to in thecontext of choice: hS, B, µ, BC i. The atoms of BC are a partition of S, so this structure gives rise to a partitional model of
unawareness. The distribution the agent “uses” in a choice6

The following notion of coherence differs importantly fromthat of [14] in that ≈ix here need not relate x only to x.
7
We do not here attempt to back-form what the agent isaware of from her choice-dispositions, as [46, 47] do for belief,and [51, 52] do for awareness.
8
This can be derived in any of the standard ways: e.g. [50],[59], [1], [5], [30], [6].

context is given by letting µC (E) = µ(E) for all E ∈ BC
and undefined otherwise. The events the agent “explicitly
believes” in the context can then be defined as the events ofwhich the agent is certain in µC . The algebra BC can also be
used to parametrize “expanding” and more generally “changing” awareness, represented as transitions between different
complete atomic sub-algebras of B.9 Since different algebras
will determine different explicit beliefs in different contexts,
this changing awareness can also represent effects of limited
attention such as framing effects or failures of recall.
An approach along these lines has already proven fruitfulin epistemic game theory. [33] and [34] develop Harsanyi typespaces in which players’ beliefs may be defined on different
σ-algebras. If the algebras are taken as the events the agent
is attending to, one may interpret these models as examplesof agents who fail to attend to questions about the higherorder beliefs of others, and thus do not have explicit beliefs
over events which can be defined only by the level-n beliefs
of others for large enough n.

5.1

Speculative Trade

An important test of approaches to unawareness has been
how they fare with speculative trade ([26]). Building on thework of [2], [41] proved their famous “no-trade” theorem,
illustrating the extreme strength of S5 knowledge together
with a common prior. One aim of representing “bounded”
agents such as those with limited attention is to escape suchparadoxes (for this perspective, see [45], [38]). Accordingly,
we now provide a partitional DLR model with a common
prior in which speculative trade is possible.
As is well known (see [16], [49], [48]), the “no-trade” theorem does not hold in general if agents’ accessibility relations
Ri are merely transitive and reflexive, but are not required
to form an equivalence relation. Plausibility is incompatible with the Ri forming an equivalence relation ([42]). Still,
the DLR axioms together with partitional awareness models impose substantial further constraints, which might bethought to rule out speculative trade. We now construct apartitional DLR model to show that speculative trade canstill occur in the presence of DLR’s axioms and nontrivial
unawareness.
Let the states be W = {1, 2, 3, 4, 5} and the agents be
Alice, A, and Bob B. The accessibility relations are definedso that: 1RA x iff x 6 3; 5RA x iff x > 3 and otherwise
wRA x iff w = x, while RB = W × W . The partitions ofA
the agents are induced by ≈A
1 =≈2 = {{1}, {2, 3}, {4}, {5}};AAA≈4 =≈5 = {{1}, {2}, {3, 4}, {5}}; and for all w, ≈B
w =≈3 ={{1}, {2}, {3}, {4}, {5}}. The agents’ common prior is theuniform one, and two acts f and g have utility as follows:f (1) = f (5) = 1, f (2) = f (4) = 5, f (3) = 7; g(w) = 4
for all w ∈ W . If the agents update by conditionalization on
their implicit knowledge, then Alice will invariably maximize
utility by choosing f (since in states 2, 3, 4 she is certain it
does better, and in states 1 and 5 she expects to gain 1/3·1+
1/3 · 5 + 1/3 · 7 > 4). Bob meanwhile does not update at all,so that he strictly prefers g (since 4 > 2/5·1+2/5·5+1/5·7).

6. CONCLUSION
Standard state space models of attention and conceivability are at least as successful as current non-standard state
space models. The non-standard models are, however, more
9

See the full paper for an alternative related to that of [31].

complicated, and it is unclear that this complexity affords
any advantages in predictive strength or accuracy. Standard
state space models of these phenomena promise to lead toa rich and rewarding theory, posing technical and conceptual challenges, and offering connections to related work by
linguists, philosophers and logicians – as well as work on
bounded reasoning elsewhere in economic theory.

7.

ACKNOWLEDGMENTS

Thanks to Eddie Dekel, Bart Lipman and Tim Williamson
for very helpful discussions of drafts of this paper. Peter Fritz
was supported by an AHRC/Scatcherd European Scholarship and the German Academic Exchange Service; Harvey
Lederman acknowledges the support of the Clarendon Fund.

8.

REFERENCES[1] F. J. Anscombe and R. J. Aumann. A definition ofsubjective probability. Annals of mathematicalstatistics, pages 199–205, 1963.
[2] R. J. Aumann. Agreeing to disagree. The Annals of
Statistics, 4(6):1236–1239, November 1976.
[3] R. J. Aumann. Interactive epistemology I: Knowledge.
International Journal of Game Theory, 28:263–300,
1999.
[4] P. Blackburn, M. de Rijke, and Y. Venema. Modal
Logic. Cambridge University Press, 2001.
[5] E. D. Bolker. A simultaneous axiomatization of utility
and subjective probability. Philosophy of Science,
pages 333–340, 1967.
[6] J. Broome. Bolker-Jeffrey expected utility theory andaxiomatic utilitarianism. The Review of Economic
Studies, 57(3):477–502, 1990.
[7] B. F. Chellas. Modal Logic: An Introduction.
Cambridge: Cambridge University Press, 1980.
[8] E. Dekel, B. L. Lipman, and A. Rustichini. Standard
state-space models preclude unawareness.
Econometrica, 66(1):159–73, 1998.
[9] A. C. Doyle. The adventure of Silver Blaze. The
Strand Magazine, 1901.
[10] R. Fagin and J. Y. Halpern. Belief, awareness, andlimited reasoning. Artificial Intelligence, 34:39–76,
1988.
[11] R. Fagin, J. Y. Halpern, Y. Moses, and M. Y. Vardi.
Reasoning about Knowledge. MIT Press, 1995.
[12] K. Fine. Propositional quantifiers in modal logic.
Theoria, 36:336–346, 1970.
[13] P. Fritz. Logics for propositional contingentism.
unpublished.
[14] P. Fritz. Propositional contingentism. unpublished.
[15] S. Galanis. Unawareness of theorems. Economic
Theory, 52:41–73, 2013.
[16] J. Geanakoplos. Game theory without partitions, andapplications to speculation and consensus. Technicalreport, Cowles Foundation Discussion Paper, 1989.
[17] J. Groenendijk and M. Stokhof. Studies on the
Semantics of Questions and the Pragmatics ofAnswers. PhD thesis, University of Amsterdam, 1984.
[18] J. Y. Halpern. Alternative semantics for unawareness.
Games and Economic Behavior, 37(2):321–339, 2001.
[19] J. Y. Halpern, Y. Moses, and M. Y. Vardi.
Algorithmic knowledge. In Proceedings of the 5th170conference on Theoretical aspects of reasoning aboutknowledge, pages 255–266. Morgan Kaufmann
Publishers Inc., 1994.
[20] J. Y. Halpern and R. Pucella. Dealing with logicalomniscience: Expressiveness and pragmatics. Artificial
Intelligence, 175(1):220–235, 2011.
[21] J. Y. Halpern and L. C. Rêgo. Reasoning aboutknowledge of unawareness. Games and Economic
Behavior, 67(2):503–525, 2009.
[22] J. Y. Halpern and L. C. Rêgo. Reasoning aboutknowledge of unawareness revisited. Mathematical
Social Sciences, 65(2):73–84, 2013.
[23] C. L. Hamblin. Questions in Montague English.
Foundations of Language, 10:41–53, 1973.
[24] A. Heifetz, M. Meier, and B. C. Schipper. Interactive
unawareness. Journal of Economic Theory,
130(1):78–94, 2006.
[25] A. Heifetz, M. Meier, and B. C. Schipper. A canonicalmodel for interactive unawareness. Games andEconomic Behavior, 62:304–324, 2008.
[26] A. Heifetz, M. Meier, and B. C. Schipper.
Unawareness, beliefs, and speculative trade. Gamesand Economic Behavior, 77:100–121, 2013.
[27] S. Heinsalu. Equivalence of the information structurewith unawareness to the logic of awareness. Journal of
Economic Theory, 147(6):2453–2468, 2012.
[28] S. Heinsalu. Universal type structures withunawareness. Games and Economic Behavior,
83:255–266, 2014.
[29] J. Hintikka. Knowledge and Belief. Cornell University
Press, 1962.
[30] R. C. Jeffrey. The logic of decision. Chicago:
University of Chicago Press, second edition, 1983.
[31] E. Karni and M.-L. Vierø. “Reverse Bayesianism”: A
choice-based theory of growing awareness. The
American Economic Review, 103(7):2790–2810, 2013.
[32] L. Karttunen. Syntax and semantics of questions.
Linguistics and Philosophy, 1:3–44, 1977.
[33] W. Kets. Bounded reasoning and higher-orderuncertainty. Submitted MS, September 2014.
[34] W. Kets. Finite depth of reasoning and equilibrium
play in games with incomplete information. Submitted
MS, February 2014.
[35] M. Krifka. Questions. In K. von Heusinger,
C. Maienborn, and P. Portner, editors, Semantics. An
International Handbook of Natural Language Meaning,volume 2, pages 1742–1785. Berlin: Mouton de
Gruyter, 2011.
[36] S. A. Kripke. A completeness theorem in modal logic.
Journal of Symbolic Logic, 24:1–14, 1959.
[37] S. A. Kripke. Semantical analysis of modal logic I:
Normal modal propositional calculi. Zeitschrift fürmathematische Logik und Grundlagen der Mathematik,
9:67–96, 1963.
[38] H. Lederman. People with common priors can agree todisagree. The Review of Symbolic Logic, pages 1–35,
2014.
[39] J. Li. Information structures with unawareness.
Journal of Economic Theory, 144(3):977–993, 2009.
[40] M. Meier and B. C. Schipper. Bayesian games withunawareness and unawareness perfection. Economic

171

Theory, 56:219–249, 2014.
[41] P. Milgrom and N. Stokey. Information, trade andcommon knowledge. Journal of Economic Theory,26(1):17–27, 1982.
[42] S. Modica and A. Rustichini. Awareness andpartitional information structures. Theory andDecision, 37(1):107–124, 1994.
[43] S. Modica and A. Rustichini. Unawareness andpartitional information structures. Games andEconomic Behavior, 27(2):265–298, 1999.
[44] R. Montague. Universal grammar. Theoria,
36:373–398, 1970.
[45] S. Morris. The common prior assumption in economic
theory. Economics and Philosophy, 11(2):227–253,
1995.
[46] S. Morris. The logic of belief and belief change: A
decision theoretic approach. Journal of Economic
Theory, 69(1):1–23, 1996.
[47] S. Morris. Alternative definitions of knowledge. In
M. O. L. Bacharach, editor, Epistemic logic and thetheory of games and decisions, pages 217–233. Kluwer
Academic Publishers, 1997.
[48] A. Rubinstein and A. Wolinsky. On the logic of“agreeing to disagree” type results. Journal of
Economic Theory, 51(1):184–193, 1990.
[49] D. Samet. Ignoring ignorance and agreeing to disagree.
Journal of Economic Theory, 52(1):190–207, 1990.
[50] L. J. Savage. The Foundations of Statistics. John
Wiley and Sons, 1954.
[51] B. C. Schipper. Awareness-dependent subjective
expected utility. International Journal of Game
Theory, 42(3):725–753, 2013.
[52] B. C. Schipper. Preference-based unawareness.
Mathematical Social Sciences, 70:34–41, 2014.
[53] B. C. Schipper. Unawareness—a gentle introduction toboth the literature and the special issue. Mathematical
Social Sciences, 70:1–9, 2014.
[54] B. C. Schipper. Awareness. In H. van Ditmarsch, J. Y.
Halpern, W. van der Hoek, and B. Kooi, editors,
Handbook of Logics for Knowledge and Belief. London:
College Publications, forthcoming.
[55] D. Scott. Advice on modal logic. In K. Lambert,
editor, Philosophical Problems in Logic: Some Recent
Developments, pages 143–173. Dordrecht: D. Reidel,
1970.
[56] K. Segerberg. An Essay in Classical Modal Logic.
Number 13 in Filosofiska Studier. Uppsala: Uppsala
Universitet, 1971.
[57] G. Sillari. Models of awareness. In G. Bonanno,
W. van der Hoek, and M. Wooldridge, editors, Logic
and the Foundations of Game and Decision Theory( loft 7). Amsterdam: Amsterdam University Press,
2006.
[58] G. Sillari. Quantified logic of awareness and impossible
possible worlds. The Review of Symbolic Logic,
1:514–529, 2008.
[59] J. von Neumann and O. Morgenstern. Theory ofGames and Economic Behavior. Princeton: Princeton
University Press, 1944.

A.

MATERIAL SUPPORTING SECTION 2

A.1

Stronger Assumptions about Knowledge

The model used in Theorem 2 already validates a numberof attractive axioms on knowledge, suggesting that strengthening DLR’s constraints on knowledge is unlikely to yield aninteresting triviality result. In particular, the following axioms are valid on the model:
Distribution: (Kp ∧ Kq) → K(p ∧ q)
Anti-Necessitation: ¬K⊥
Reflexivity: Kp → p

Positive Introspection: Kp → KKp

We can show more systematically that any strengtheningof the axioms of knowledge which rules out unawarenessdoes so trivially in the same way as Negative Introspection
does. On the very mild assumption that the agent doesn’tknow the contradiction ⊥, we can characterize the conditions
under which a given model for the knowledge of an agent canbe extended to an unawareness model in which the agent isunaware of a given p at a given point α in which DLR’s threeaxioms are valid. Let a model M 0 extend a knowledge model
M = hΩ, ki just in case M 0 = hΩ, k, ai for some functiona : 2 Ω → 2Ω .
Theorem 6. Let M = hΩ, ki be a knowledge model, α ∈Ω and E ⊆ Ω such that Anti-Necessitation is valid in α. M
has an extension such that Plausibility, KU -Introspection
and AU -Introspection are valid in α and α ∈/ a(E) if andonly if(i) α ∈ −k(E) ∩ −k − k(E), and(ii) there is an event F such that a ∈ F ∩ −k(F ) ∩ −k −k(F ).10
Proof. Assume first that (i) and (ii). Let a : 2Ω → 2Ω
be defined so that a(E) = a(F ) = −F , and a(G) = Ω for allother events G, and consider the model M 0 = hΩ, k, ai. It
is routine to verify that Plausibility, KU -Introspection and
AU -Introspection are valid in α, and α ∈/ a(E), appealing
to Anti-Necessitation in the proof for KU -Introspection.
For the converse, note that (i) follows the validity of Plausibility in α. For (ii), let F = −a(E). Then α ∈ F , by
AU -Introspection, α ∈/ a(F ), and so by Plausibility, α ∈−k(F ) ∩ −k − k(F ).
In particular, as long as the constraints on knowledge allow for there to be an event E and a state α ∈ E such thatα ∈ −k(E) ∩ −k − k(E), standard state space models and
DLR’s three axioms will not preclude non-trivial unawareness.

A.2

Stronger Assumptions about Awareness

These results demonstrate that no plausible strengtheningof the axioms governing knowledge will re-instate triviality.
But what if we strengthen the axioms on awareness themselves?
To investigate this issue formally, extend the language L
by a unary operator CK for common knowledge. To define its interpretation on a model M = hΩ, k, ai, derive thefollowing functions onTevents: k1 (E) = k(E), kn+1 (E) =kkn (E), and ck (E) = 1≤n<ω kn (E).
10JCK ϕKM,v = ck (JϕKM,v )This result also holds if we replace Plausibility by nPlausibility, for all natural numbers n, and (i) and (ii) by
the correspondingly iterated conditions.

With this, we consider the following additional axioms onawareness:
CK -Plausibility: Ap → CK (U p → (¬Kp ∧ ¬K¬Kp))

CK -KU -Introspection: Ap → CK (¬KU p)

CK -AU -Introspection: Ap → CK (U p → U U p)

These additional axioms are also compatible with nontrivial unawareness. In fact, they are valid in state α of theexample in the proof of Theorem 2. More generally, Theorem 6 can be extended straightforwardly to these three additional axioms, given the weak assumption that Necessitation
and Anti-Necessitation are valid:
Theorem 7. Let M = hΩ, ki be a knowledge model inwhich Necessitation and Anti-Necessitation are valid, α ∈ Ωand E ⊆ Ω. M has an extension such that Plausibility, KU Introspection, AU -Introspection, CK -Plausibility, CK -KU Introspection and CK -AU -Introspection are valid in α andα∈/ a(E) if and only if(i) α ∈ −k(E) ∩ −k − k(E), and(ii) there is an event F such that α ∈ F ∩ −k(F ) ∩ −k −k(F ).11
Proof. We establish (i) and (ii) as in the proof of Theorem 6. Assuming (i) and (ii), we define a as in the proofof Theorem 6, where it is noted that Plausibility, KU -Introspection and AU -Introspection are valid in α, and α ∈/ a(E).
For the CK -conditions, consider any event G such that α ∈a(G). Then by construction of a, a(G) = Ω. Therefore a(G)∪H = Ω for any event H, and so α ∈ ck (a(G)∪H) by Necessitation, which establishes the validity of CK -Plausibility andCK -AU -Introspection in α. For CK -KU -Introspection, notethat by Anti-Necessitation, k − a(G) = ∅, so −k − a(G) = Ω,from which α ∈ ck (−k − a(G)) follows again by Necessitation.

B.

PROOF OF THEOREM 4

Proof. Since the formulas derivable in this calculus forma classical model logic in the sense of [56], we can apply thestandard canonical model construction technique; in particular, consider the smallest canonical model (see [7, chapter 9], especially p. 254). Consider any formula ϕ not provable in the above calculus, and let Γ be the set of subformulas of ϕ closed under Boolean combinations. A standard
filtration of the canonical model through Γ produces a finite
model in which ϕ is false. It is routine to prove that theneighborhood function for Ai associates with each state afield of sets; since the model is finite this field is generated
by an equivalence relation, as required. The above filtration
can be chosen in such a way as to preserve the transitivity of
the relation for Ki ; reflexivity is preserved by any filtration(see, e.g. [7, chapter 3], especially p. 106, or [4] p. 80).
The above argument also establishes that the logic thus
axiomatized has the finite model property and so is decidable.

11

Again, we can extend this result to n-Plausibility for all nanalogously to the extension in the previous footnote.

172

An Axiomatic Approach to Routing
Omer Lev

Hebrew University andMicrosoft Research, Israel

omerl@cs.huji.ac.il

Moshe Tennenholtz
Technionmoshet@ie.technion.ac.il

ABSTRACT

Aviv Zohar

Hebrew University andMicrosoft Research, Israel

avivz@cs.huji.ac.il

there are various properties we may desire in our system. We
might want the system to be able to handle small changes
Information delivery in a network of agents is a key issue forin connections without causing disruptions throughout thelarge, complex systems that need to do so in a predictable,
network; we may want it to be flexible when we change its
efficient manner. The delivery of information in such multiparameters so that various routing options are possible, andagent systems is typically implemented through routing promore. Indeed, the search for the right communication structocols that determine how information flows through theture has played a role in early work on the foundations of
network. Different routing protocols exist each with its own
the area of multi-agent systems [7, 15, 5], based on classicalbenefits, but it is generally unclear which properties canwork in organization theory [8, 16].
be successfully combined within a given algorithm. We apMore concretely, examining networking, one of the most
proach this problem from the axiomatic point of view, i.e.,
important aspects of the design of a communication netwe try to establish what are the properties we would seek
work is the way it routes information through its physito see in such a system, and examine the different propercal links. Routing protocols, such as those used in packet
ties which uniquely define common routing algorithms used
switching networks, circuit switching, or ad-hoc networks
today.
are designed with many goals in mind. They must adapt
We examine several desirable properties, such as robustto changing network conditions, withstand failures, and opness, which ensures adding nodes and edges does not change
erate in a distributed fashion while constructing a “good”
the routing in a radical, unpredictable ways; and properrouting scheme. Nodes in the network are, in fact, auties that depend on the operating environment, such as an
tonomous agents that can control the flow of information“economic model”, where nodes choose their paths based on
through them and can choose to forward it according to
the cost they are charged to pass information to the next
their own considerations. Agents may be controlled by difnode. We proceed to fully characterize minimal spanningferent economic entities (such as in the internet, where differtree, shortest path, and weakest link routing algorithms,
ent internet service providers control some of the routers),
showing a tight set of axioms for each.
and may route according to complex preferences that arederived from economic relations [9, 14]. Even in the cooperCategories and Subject Descriptors
ative local-network setting where all routers are controlled
by a single network operator, different considerations such as
C.2.1 [Computer Systems Organization]: Computerbandwidth utilization, latency, and the risks of link failures
Communication Networks—Network Architecture and Decome into play.
sign; C.2.2 [Computer Systems Organization]: ComputerCommunication Networks—Network Protocols; C.2.6 [Computer The multitude of previous treatments of the problem suggest a myriad of routing protocols, each with their own benSystems Organization]: Computer-Communication Netefits and shortcomings. In contrast, this work examines theworks—Internetworking; G.2.2 [Mathematics of Comrouting problem through the lens of the axiomatic approach,puting]: Discrete Mathematics—Graph Theory
which seeks to formulate different elementary properties thatare desirable in this context. One approach to an axiomatic
General Terms
treatment, which we take in this work, is that of characterDesign, Algorithms, Theory
ization: a set of elementary properties is shown to uniquelydetermine some routing algorithm, and hence the routing
Keywords
outcome on any specific graph. From the designer’s perspective, such a result implies a great deal – any additional
Routing, Axiomatic Approach, Routing algorithms
property that is not already achieved by the protocol cannot
be added to it without giving up on another basic property.
1. INTRODUCTION
The approach thus provably bounds the design space of algoThe proper way to distribute power, disseminate inforrithms and makes explicit the choices made when selecting
mation, or establish hierarchies in organizations is an issue
one over the other.
encountered whenever there is a large enough network of
As we are not aware of any previous axiomatic treatment
agents that needs to interact in an orderly manner. For exof routing, we focus our attention on a domain that mostample, when trying to establish efficient lines of communicaclosely resembles the internet as it is built today, and fotions between agents which all need to reach a central hub,173cus our efforts within this domain on what one may consider classic, or natural routing schemes. In particular, we
assume that routing choices are independent of the congestion on links (such is the case in the internet, where routing
protocols such as BGP first establish paths, and congestion
control protocols such as the one embedded into TCP manage the load on each flow’s path and ensures that rates arethrottled to match the bottleneck of the flow). Furthermore, as with internet routing where routers decide on thenext hop of each packet using a routing table that maps its
destination to the next hop, routing choices made to different destinations are done independently. Finally, packets
addressed to the same destination are not split between different paths, and are routed in the same manner regardlessof their source. These choices, which greatly restrict thepower of any routing algorithm may seem arbitrary, but arein fact derived from real-world design considerations. For
example, the need to quickly forward packets towards theirdestination at each router mandated that most routing bedone in specialized hardware. No complex computation is
performed (only a lookup into a routing table) and no deep
inspection of the packet is performed. Keeping routing simple has made it fast and robust.
More advanced routing schemes that have been proposedin the literature may split traffic, allow routing choices todepend on the source of the packet or its previous hops, or
may even change the routes in response to link congestion.
These are notoriously difficult to coordinate and to implement. We leave treatment of these more advanced schemes
to future work.
Our set of axioms or “desirable properties” are also motivated by similar considerations. For example, one of the fundamental features we desire in our algorithms is one of robustness, which is the ability of a system to endure changes
in the network without creating disruption in parts of the
network that have not undergone changes.
A different feature, which might be desirable only in certain cases, is “first hop”, which is particularly relevant for
diffuse networks with independent nodes. It means, broadly,that network nodes care only about their immediate surroundings, or the “next step” in the network data transfer.
Such a property might be relevant when nodes pursue an“economic model”, paying for transferring information, andhence only caring about the cost they need to pay to movetheir information to the next node, and following that, they
have no preference on the route the information should passen route to its destination. Other properties, desirable only
in some cases include an indifference between two parallelpaths, as long as they change their weights by the sameamount concurrently.
Ultimately, after devising our axioms we successfully fully
characterized 3 natural routing algorithms:• Minimum spanning tree: A tree with the smallest
overall weight is a result, among others, of the “firsthop” axiom (the “economic model”).
• Shortest path: A tree where each node has the shortest possible path to its destination is a result, among
other axioms, of viewing as immaterial to the routing
decision any parallel paths which change their weightby the same amount.
• Weakest link tree: A tree where each node takes thepath with the maximal “lightest” weight available toit. This results from considering higher edge-weights
as beneficial (e.g., representing bandwidth which one
wishes to increase in contrast to delay that one wishes
to decrease), and from considering designers that choose
between parallel paths in a slightly different manner.
We proceed to review relevant previous research and thencontinue to define our model and expand on the axioms,
which are motivated with a brief explanation and presented
formally. Following that we show (and prove) our characterization of the minimal spanning tree, the shortest path tree,
and the weakest link tree.

2.

RELATED WORKIn the past decade, as routers became more flexible, research on routing (particularly inter-domain) and its techniques has been rekindled and extended beyond the technical issues dealt with in the past. The harbinger for much ofthis research was [10], which was further expanded by several researchers (see updating report here: http://www.cl.
cam.ac.uk/~tgg22/metarouting/ ). However, this line ofresearch, while introducing many interesting mathematicaland theoretical concepts to the field of routing, has refrained
from phrasing its models as requirements by users, to be
filled by various routing algorithms.
The axiomatic approach, which does approach problems
with this outlook, has been first introduced in CS contexts
as extensions to the classical theory of choice [4], and has
been applied to ranking systems [1, 2] and trust systems [3],as well as to other multi-agent setups such as multi-levelmarketing [6].
In relation to networking, usage of the axiomatic approachhas generally been concentrated in two main areas: applying to general graph theory (e.g., [18]) or in more technicalapproaches to networks: papers such as [13] which deal withparticular wireless models and implementations, and, somewhat closer to our line of work, [12], whose basic axioms arebasic enough to be covered through our models, while therouting related axioms involve various assumptions on how
routers work (tables, etc.), which we refrain from approaching in our more abstract considerations.
Further work connecting networking and the axiomatic
approach has focused on particular instances of problems:[11] try to use the axiomatic approach to extract the costsof multicast routing and decide who is to pay them. Trust
networks and social networks (e.g., recommendation systems) have been analysed many times using the axiomatic
approach to understand their desirable features and better
understand desirable algorithms in these cases [17, 3]. However, none of these papers deal with the basic routing mechanism by which messages and information arrive at each
node.

3.

SETUP

Before introducing our axioms, we begin by setting up
our routing model. It is, naturally, only a simplification ofrouting as it is done in large, complex networks such as theinternet, but we believe it is robust enough to display many
networking characteristics.
Our world will be a weighted graph G(V, E, W ) and a
destination d, where V is a set of nodes, E is a set of edges,
and W is a function assigning weights to edges, and d ∈ V .
A routing solution is a tree T over that graph, as defined174below (we do not concern ourselves with non-tree routing,as passing through the same node several times does notserve any purpose).

aDefinition 1. A routing function fd : G → T is a function from connected weighted graph G(V, E, W ) ∈ G in whichd ∈ V , to a tree T (V, E, W ) ∈ T such that T ⊆ G.
We can look at the graph as one with directed edges if
we consider each edge’s direction to be the one pointing at
the vertex from which there is a path to d (without going
through the same edge again).
We discuss 3 different routing options:• Minumum spanning tree (MST): a tree connecting all
nodes in the graph with the minimal weight, i.e., forevery
tree T 0 ⊆ GP
that encompasses all of G’s nodes,
Pe∈fd (G) W (e) ≤e∈T 0 W (e).

• Shortest path: each node is connected to d using ashortest length path in the graph. For every node v ∈V , let (e1 , . . . , es ) be a path without cycles from v to dsuch that ei ∈ T , and let (e01 , . . . , e0k ) a different pathP
Pfrom v to d, then si=1 W (ei ) ≤ kj=1 W (e0j ).
• Weakest link : looking at each potential path from eachnode to d, we give each path the value of its smallest
valued edge. The routing tree will contain, for eachnode the path to d with the maximal value. So forevery node v ∈ V , let (e1 , . . . , es ) be a path without
cycles from v to d such that ei ∈ T , and let (e01 , . . . , e0k )a different path from v to d, then min1≤i≤s W (ei ) ≥min1≤j≤k W (e0j ).

Notice that while for the minimal spanning tree and shortest path routing options weights are interpreted as costs(e.g. payments, delays), so these algorithms seek to minimize them, the weakest link views weights as measure for capability such as bandwidth, so seeks to maximize the weight.

4. AXIOMSHaving introduced our framework, we introduce our axioms, which are, basically, desirable properties of the function fd (in the axioms below we use f , as these are properties
which do not depend on a specific d destination).
Robustness indicates the routing being quite unsusceptible
to changes – only if a path in the routing is destroyed, will
it require any change. As indicated in Figure 1, the pathfrom node a changes, but not from node b.
Axiom 1 (Robustness). f is robust if removing anedge e ∈ E from G(V, E, W ), yielding G0 , then for every
vertex v ∈ V : if the cycle-less path from v to d in fd (G) did
not contain e, then this is still the selected path according to
fd (G0 ) (see example Figure 1).
The following axioms deal with global changes to the graphweights, additive or multiplicative:
Axiom 2 (Scale Invariance). f is scale invariant if
for a graph G(V, E, W ), for any positive scalar α ∈ R+ ,defining G0 (V, E, αW ), for every d ∈ V , fd (G) = fd (G0 ).
Axiom 3 (Shift Invariance). f is shift-invariant if fora graph G(V, E, W ), for any α ∈ R, defining G0 (V, E, α +W ), for every d ∈ V , fd (G) = fd (G0 ).
175

b

a

d

d
a

b

d

b

a

b

d

Figure 1: An edge is removed, but only a, whose
path used that edge changes its path (the left side is
the graph, the right side is the routing algorithm’s
output)
The monotonicity axiom below seeks to establish that if anedge does not have to be in every tree, if its weight increases
enough, it will not be a part of the routing tree:
Axiom 4 (Monotonicity). f is monotone if for a graph
G(V, E, W ) and d ∈ V , for e0 ∈ E, if e0 ∈/ fd (G), thenfor every G0 (V, E, W 0 ), there is a value MW 0 such that for
W 00 such that W 00 (e) = W 0 (e) for all e ∈ E \ {e0 } andW 00 (e0 ) ≥ MW 0 , e0 ∈/ fd (G00 (V, E, W 00 )). Similarly, we candefine the opposite direction, an edge in fd (G) will not be in
the routing tree if it has a small enough value; we will refer
to it as inverse monotonicity.
While the phrasing of the following axiom is somewhat technical, the first hop axiom below simply means that if a vertex has several potential edges to connect to a path to d, therouting only depends on the weights of the edges connecting
it to these potential paths, and unrelated to weights of otheredges in the graph.
Axiom 5 (First Hop). Let G(V, E, W ) be a weighted
graph and let v, d ∈ V and d 6= v. Suppose C = {c1 , . . . , cs }are the vertices such that (v, ci ) ∈ E and there is a path fromci to d in fd (G) which does not pass through v. W.l.o.g., let(v, c1 ) be the first step in the path from v to d in fd (G).
We say that f satisfies first hop if for any W 0 such that
W 0 (v, ci ) = W (v, ci ) and if for all ci ∈ C fd (G0 (V, E, W 0 ))contains paths to d from ci that do not pass through v, andthere is no c0 ∈/ C such that (v, c0 ) ∈ E and there is a path0
from c to d in fd (G0 ), then the cycle-less path from v to d
in fd (G0 (V, E, W 0 )) starts with (v, c1 ).
The rational for the first hop axiom is to capture a common economic model, in which edge weights indicate thecost of passing information. In distributed networks, suchas the internet, each agent only minds the amount it needs
to pay to transfer its data to the next node, not caring about
the path the data will take from there.

d

d

4
2

d

( )
1

( )= ( )
1

fd

fd

d

1

2

5

fd

4

1

5

5

d

( )= ( )
1

fd

1

1

1

fd

2

5

5

d

d

(

3

d

1

5

3

d

Figure 2: Selected path does not change when eachpath from the top node is added 2.
Path cardinal/ordinal invariance intends to see the planner’s considerations when multiple paths exist. As there
might be many potential behaviours, we only limit ourselves
to examining the narrow case of what the planner considers
important when there is only one cycle in the graph (i.e., theaxiom does not strongly enforce a general behaviour on theplanner). Cardinal invariance deals with adding the same
weight to potential paths, and how it does not effect therouting. Ordinal invariance similarly does not change therouting if all that has changed are the weights of the competing paths, as long as edges in each path maintain theirrelative position.

Axiom 6 (Path Cardinal Invariance). Let G(V, E, W )be a graph which contains a single cycle, d ∈ V , and let
d 6= v ∈ V be a part of this cycle. Hence there are twoalternative paths from v to d – p1 ⊂ E and p2 ⊂ E (oneof them is actually a part of fd (G)). f is path cardinal invariant if it treats those paths as such: Choosing an edgee0 ∈ p1 and e00 ∈ p2 , for any α ∈ R, we define W 0 as
W (e) = W 0 (e) for e ∈ E \ {e0 ∪ e00 } and W 0 (e0 ) = W (e0 ) + α
and W 0 (e00 ) = W (e00 ) + α, the path from v to d will notchange in fd (G(V, E, W 0 )) (see example Figure 2).

Axiom 7 (Path Ordinal Invariance). Let G(V, E, W )be a graph which contains a single cycle, d ∈ V , and let
d 6= v ∈ V be a part of this cycle. Hence there are two alternative paths from v to d – p1 ⊂ E and p2 ⊂ E (one ofthem is actually a part of fd (G)). f is path ordinal invariant if it treats those paths as such: Taking an edge e0 ∈ pi(i ∈ {1, 2}) that is not maximal or minimal in p1 ∪p2 , we define W 0 as W (e) = W 0 (e) for e ∈ E \ {e0 } and allow W’(e’)
to be any value it chooses as long as for every e00 ∈ pi if
W (e0 ) ≥ W (e00 ) then W 0 (e0 ) ≥ W 0 (e00 ) = W (e00 ), and thepath from v to d will not change in fd (G(V, E, W 0 )) (see example Figure 3).

5.

MINIMAL SPANNING TREE

Theorem 1. A robust, scale invariant, shift invariant,monotone, first-hop (axioms 1-5) routing function f , for any
graph G(V, E, W ) and d ∈ V , fd (G) will always be a minimal
spanning tree of G.
Reminder 1. As our minimal spanning tree proof relieson the Kruskal algorithm, we will briefly describe it:
1. Order edges according to weights

Figure 3: Selected path does not change when thebottom right edge is slightly increased.
2. Define a set S, initialized to the empty set.
3. Going over edges from lightest to heaviest, if the set
S ∪ {e} has no cycles, S = S ∪ {e}.
Proof Proof of Theorem 1. We shall prove the theorem using complete induction on the number of non-cycle
lightest edges in the tree fd (G). Hence, we shall begin byproving that the lightest edge in the graph G is in the routing
tree T = fd (G). Assuming we are mistaken, let us consider
the lightest edge in G – e = (u, v) ∈ E – and assume e ∈/ T.
We create G0 (V, E 0 , W ) = fd (G) ∪ {e}, and thanks to therobustness axiom, we know fd (G) = fd (G0 ).
If v’s path to d in fd (G0 ) goes through u, we shall switch
the nodes’ names, so that v’s path to d does not pass throughu. As e is not in fd (G0 ), there is an edge e0 = (u, s) that is
the first step from u towards d. We now define x = W (e)and y = W (e0 ), and due to our minimality assumption, weknow x < y.
Using the monotonicity axiom, we change graph weights
to W 0 that is identical to W except that e0 weight is large
enough so that we create a tree T 0 in which there is a pathfrom s to d that does not pass through u (e.g., the same
path that is in fd (G0 )), and v passes through u towards d(i.e., e ∈ T 0 ). We define y 0 = W 00 (e0 ).
Using scale invariance we now multiply all edges by yy−x
0 −x ,0and using shift invariance, we add to all edges y − yy−x
0 −x y .
This means the weight of edge e is now
y−x
y−x 0
y−x
x 0
+y− 0
y = (x − y 0 ) 0
+y =x
y −x
y −x
y − x0

While the weight of edge e0 is now
y0

y−x
y−x 0
+y− 0
y =y
y0 − xy −xHowever, the routing tree contains e and not e0 , and apath from both v and s to d, contradicting the “first hop”
axiom, which should have caused e0 to be chosen over e, as
the edge weights for e and e0 have not changed.
We now turn to the induction step – we assume all bottom
weighted k−1 edges that do not create a cycle are included in
the tree T = fd (G), and we now seek to include the k-lightest
edge that does not create a cycle. We pursue a similar path
as we did as previously, and we shall mark the edge as e =(u, v), and assume it is not included in T = fd (G), andinstead e0 = (u, s) is included, and there is a path to d fromv and s. Again, we create G0 (V, E 0 , W ) = fd (G) ∪ {e}, andthanks to the robustness axiom, we know fd (G) = fd (G0 ).
Using monotonicity we create weights W 0 that just increase
e0 weight, so that G00 = (V, E 0 , W 0 ) has T 0 = fd (G00 ) which
include the same bottom k which do not create cycles (from

176

5

d

the induction hypothesis), and u reaches d via the edge e.
Recall that we know the bottom k − 1 edges will definitely
be in fd (G00 ), and we wish to ensure that there will still be apath from v to d and from s to d. The same arguments usedin the initial step of the induction ensure that, as well as
returning the weights of e and e0 to their values in G, whilerouting u through e and not e0 in the routing tree, reachinga contradiction with our initial assumption due to the “firsthop” axiom.
What is left is to show MST indeed follows our axioms:
Robustness (axiom 1) Trivial thanks to the Kruskal algorithm – if the removed edge (e0 ) was not in the routing tree, it means it was not selected in the first place,and hence the same routing tree will be chosen. If it
was, then any edge added after its removal (e00 ) closed
a cycle with it, and hence, if affecting the edges in any
path that did not include e0 , it means e00 closes a cyclewith them, hence e0 would have closed a cycle as well.

u to d to e and e0 respectively, with everything else being 0.
Now, using monotonicity, we increase the weight of e0 abovethat of e, with the weight of (u, v) being x (its path weights
have not changed) and (u, s) being y 0 .
Finally, we multiply all edges by yy−x
0 −x (using scale invariance), and using path cardinal invariance, we add to e and0
e0 the amount y − yy−x
0 −x y . The weight of e is now:
y−x 0
y−x
y−x
+y− 0
y = (x − y 0 ) 0
+y =x
x 0
y −x
y −x
y −x

While the weight of edge e0 is now
y0

y−x
y−x 0
+y− 0
y =y
y0 − xy −xAs all edges are the same weight as before, therefore wereached a contradiction regarding the inclusion of e0 insteadof e (whose weights are the same as well).
We will now show shortest path follows our axioms:

Scale invariance (axiom 2) Multiplying all edges by afixed amount does not change their order in relation
to others, hence Kruskal will choose the same routing
tree.

Robustness (axiom 1) Removing an edge, at most, eliminates a potential path from a node to the destination
d. If the path was not on the shortest path, the previous shortest path remains so.

Shift invariance (axiom 3) Adding a fixed amount to alledges does not change their order in relation to others,
hence Kruskal will choose the same routing tree.

Scale invariance (axiom 2) Multiplying by a fixed amountall edges means the value of each path is multiplied by
the same amount, maintaining their relative ordering,hence what was shortest remains so.

Monotonicity (axiom 4) Giving an edge the maximal possible edge value ensures it will only be selected if noother edge can replace it – and if there exists a tree
without some edge, we know it will be chosen before.
“First hop” (axiom 5) Kruskal ensures that if there are
the same possible options of connecting a node to thetree, only the lightest edge will be chosen.

6.

Monotonicity (axiom 4) Giving an edge the value of thesum of all other edges ensures it will only be selected
if no other path can replace it — and if there exists atree without some edge, we know there is such a path.
Path cardinal invariance (axiom 6) Having multiple paths
from a node, adding the same amount to each path
doesn’t change the ordering of the paths (i.e., whichpath is “shorter” than another), hence selection of shortest path will be identical.

SHORTEST PATH

Theorem 2. A robust, scale invariant, monotone, andpath cardinal invariant (axioms 1-2, 4, 6) routing function
f , for any graph G(V, E, W ) and d ∈ V , fd (G) will always
be a shortest path graph to d of G.
Proof. Suppose T = fd (G) is not a shortest path routing
tree. Let u be the closest node to d that is not connected tod with a shortest path. Hence, there is an edge e = (u, v)
which will make u’s path a shortest path one (v, being closer
to d, is already connected to d with a shortest path), bute ∈/ T , and instead e0 = (u, s) is included in T . Using
robustness, we create G0 (V, E 0 , W ) = T ∪ e. G0 contains twoalternate paths from u to d, and fd (G) = fd (G0 ).
Using path cardinal invariant, we “move” all the value ofthe edges on each path to it’s “source”, i.e., to (u, v) or (u, s)(we do thisP by adding to the weight of (u, v) and (u, s) thevalue ofe∈(p1 ∪p2 )\(p1 ∩p2 ) W (e) − W ((u, v)) − W ((u, s)),and reduce from W ((u, s)) the weight of all edges of thepath from u to d through (u, v) and vice versa). We shallrefer to W (e) = x and W (e0 ) = y. We now use monotonicity
to create a new tree, with e but without e0 , with the graph’s
weight now W 0 (identical to W except for increase in e0
weight). Once again, we transfer all value of the paths from

177

7.

WEAKEST LINK

Theorem 3. A robust, scale invariant, shift invariant,inverse monotone, and path ordinal invariant (axioms 1-4,
7) routing function f , for any graph G(V, E, W ) and d ∈ V ,fd (G) will always be a weakest link graph to d of G.
Proof. Suppose T = fd (G) is not a weakest link routing
tree. Let u be a node that requires just one edge missing
from T that is not connected to d with a weakest link1 , andwe mark this edge as e = (u, v). Since e ∈/ T , there is an edge
instead e0 = (u, s) that is included in T . Using robustness,
we create G0 (V, E 0 , W ) = T ∪ e. G0 contains two alternate
paths from u to d, and fd (G) = fd (G0 ).
Using path ordinal invariant, we change the value of alledges on each alternate path from u to d to its weakest link
value (we do this by taking the 2nd smallest edge in the pathand changing its value to that of the weakest link, which by1such a node exists as there is a node not connected byweakest link in T , hence adding the necessary path for that
node, taking the node just before the final edge that we add
to T (i.e., closest to d), answers our criterion.

the axiom does not change2the path chosen, and we proceed
doing so to all edges on the path). We shall refer to W (e) =1 assuming
x and W (e0 ) = y (from1d 1 u is not in a weakest
d link1
path we know x > y). Using inverse monotonicity, we create
W 0 identical to W except for e0 weight, that is low enough3
3
that it is not included in fd (G00 (V, E 0 , W 0 )). Once again, wechange the values of theu to d to their weakest
2 paths fromd 2for the path through ed0 , as2
link value (this is only relevant
the other path has not changed). We term the the new value
for e0 – y 0 , and we know x > y 0 .
Finally, we multiply all edges by yy−x
0 −x (using scale in2
variance), and using shift invariance, we add to all edges0y − yy−x
1 has the1 weight:
0 −x y . Edge e now
1
1

d

d

y0

y−x
y−x 0
+y− 0
y =y
y0 − x
y −x

d

1

1

1

4
2

5

2

d

d

1

5
1

d

y−x
y−x 0
y−x
x 0
+y− 0
y = (x − y 0 ) 0
+y =x
4
y −x
y − x4y −x0
While the weight of edge
2 e is now
2

1

1

1
2

dd

1

4

1

d

3

2

1

2

5

2
1

5

3

( )= ( )

f

f

2

5

As all edges are the same weight as before, hence wereached a contradiction regarding the inclusion of e0 instead
of e.
1
1
1
1
We shall now show weakest link also follows our axioms:( ) ( )

d

1

d

d

1

1

d

4

inates a potential5path from
2 a node to the destination
53
d. If the path was not a weakest link, the previousddweakest link remains so.

2

d

5
1

d

4
2

d

2

fd

(

Scale invariance (axiom 2) Multiplying by a fixed amountall edges means the value of each path (its smallestedge) is multiplied by the same amount, maintaining
their relative ordering, hence what was weakest linkremains so.

Figure 5: Eliminating scale invariance results in aminimal spanning tree/shortest path/weakest linkrouting (above) ending up in neither (below).

Shift invariance (axiom 3) Adding a fixed amount all edgesmeans the value of each path (its smallest edge) is
added the same amount, maintaining their relative ordering, hence what was weakest link remains so.

ddProof.
Going over all MST axioms,
we detail potential
algorithms 5which workthat one, and2 with all axioms except
5
3 axiom,
are not MST. We will refer below to each relaxedddand to the new/additional system which can obtained bythat relaxation:

Monotonicity (axiom 4) Giving an edge the value of theminimum of all other edges ensures it will only be selected if no other path can replace it — and if there
exists a tree without some edge, we know there is sucha path.

Robustness See example in Figure 4. Apply MST to any
other graph that is not a linear transformation of thebottom one.

( )= ( )
1

f

1

1

1

f

Scale invariance See example in Figure 5. On all graphs
except those which contain as a subgraph a linear transformations of the bottom one, apply MST.
Shift invariance See example in Figure 6. On all graphs
except those which contain as a subgraph a linear transformations of the bottom one, apply MST.
Monotonicity A maximal spanning tree implements all
axioms but monotonicity.

8.

TIGHTNESS OF AXIOMS

We will now show that the above characterizations aretight, and that without each axiom, other routing algorithms
become possible.
Theorem 4. All MST axioms (1-5) are necessary, andwithout even one of them, other routing algorithms are possible.

First Hop Weakest link implements all of the other axioms.

Theorem 5. All shortest path axioms (1-2, 4, 6) are necessary, and without even one of them, other routing algorithms are possible.

178

1

5

5

2

fd 1) Removing an=f
Robustness (axiom
edge, dat most, elim-

Path ordinal invariance (axiom 7) Having multiple paths
from a node, the weakest link edge (the one with smallest value) of the selected path can’t become lower than
the weakest link of the non-selected path, hence weakest link choice does not change.

dFigure 4:
robustness results in a1 minimal
1 Lack 1of
4 d 2 ending
spanning2tree/shortest
path routing (above)d 2
up in a routing tree that is weakest link but notdd or shortest path (below).
MST

1

5

d

2
1

d

See example
in 1Figure 6.
1Shift invariance
1
1
1

1

3
2

d

d

1
5

3
2

d

2
1

Figure 6: Eliminating
shift invariance results in a2
minimal spanning tree/weakest link routing (above)
ending up(below).
1 in neither
1
1
1

d

Monotonicity A strongest link tree implements all axioms
but2 monotonicity. 525dProof. Going
we detail4 over all shortest path axioms,
4
potential algorithms which work with all axioms except one,
2 shortest2 path. We will refer below 2to the eachand are notddrelaxed axiom, and to the new/additional system which canobtained by that relaxation:

fd

Taking thebottom example and for the group that includes all
4graphs for which it is a subgraph and those that canbe formed by scale invariance and only for them do not2
apply weakest link but rather2 the example (it won’t
d trample on the top example,
d as it can’t be reached byshift invariance, and as the edge weights are all minimal/maximal, they change change by ordinal invari1
1ance).

Pathd ordinal invariant Minimald spanning tree implements
all other axioms.

( )= ( )
1

1

9.

4

1

DISCUSSIONfd
In this paper we explore
the basic issue of routing – how2
5
5
Robustness See example in Figure 4. Apply shortest path5
should information flow through a network
and what propto any other graph that isn’t a scale of the structure
ertiesmight this process have. Ind the process of consideringdof the bottom one. Any edge in that structure that is
this issue we developed several properties we believe might
1
1 others is removed1 in the tree.
1
100 times
all thebe desirable by system planners. For example, robustness, orthe ability of a routing protocol to keep small changes from
Scale invariance See example in Figure 5. Taking the botd
ddisrupting the whole routing process, is a property especially
tom example and for the group that includes all graphs5
required in fast, changing networks.
for which
it is a2 subgraph and those5that can3be formed
Naturally, creating a structure from possible interactions
by path cardinalinvariance, and only fordd them do not
between agents defined by a connections’ graph is not limapply shortest path but rather the example (it won’t
ited just to information routing in networks such as the intrample on the top example, as if the upper example
ternet. Looking at organizations, where workers are conadds y to lower-right edge, and y to the rest, and thenected according to their ability to work with other workbottom example adds x, it would require 2 + x = 4 + x,
ers, and instead of routing messages between them we seek
reaching an impossibility).
to construct an organizational hierarchy, we face a similar
Inverse Monotonicity A longest path tree implements all
challenge. Again, robustness is a desirable property, as it
axioms but monotonicity.
means that if some workers have a worsening relationshipwith others, if they’re not very senior in the organization, it
Path cardinal invariant Minimal spanning tree implements
has little effect on many others. In this case, we may conall other axioms.
sider the “economic model” (“first hop” axiom) appropriate
as well – if workers only interact with their boss, we only
care about the edge from each worker to his/her boss, andTheorem 6. All weakest link axioms (1-4, 7) are neceseach worker does not care what happens further up in thesary, and without even one of them, other routing algorithms
hierarchy2 .
are possible.
Beyond setting up the axioms, we also examined common
routing algorithms – minimal spanning tree, shortest path
Proof. Going over all weakest link axioms, we detail poand weakest link, and fully characterized them. Obviously,
tential algorithms which work with all axioms except one,this is only the beginning of the road for this line of research
and are not weakest link. We will refer below to the each– further steps will entail developing more axioms and using
relaxed axiom, and to the new/additional system which canthem to characterize more algorithms, with the aim of giving
obtained by that relaxation:a set of tools for system designers, allowing them to choose
Robustness See example in Figure 4. Apply weakest linkdesirable properties which would dictate appropriate routing
to any other graph that isn’t of the structure of the
protocols.
bottom one, Any edge that in that structure that is
100 times less that all the others’ weight is removed.

f( )= ( )fScale invariance See example in Figure 5. Taking thebottom example and for the group that includes allgraphs for which it is a subgraph and those that canbe formed by shift invariance and only for them do not
apply weakest link but rather the example (it won’t
trample on the top example, as it can’t be reached byshift invariance, and as the edge weights are all minimal/maximal, they change change by ordinal invariance).

179

Acknowledgments

The authors thank Michael Schapira for his insightful discussions on this matter. Moshe Tennenholtz carried out thiswork while at Microsoft Research, Israel. Aviv Zohar is
supported in part by the Israel Science Foundation (Grants
616/13 and 1773/13), and by the Israel Smart Grid (ISG)
Consortium.
2

Similarly, in a highly centralized organization, a path cardinal invariance is probably a sensible axiom.

10.

REFERENCES[1] A. Altman and M. Tennenholtz. Ranking systems:
The pagerank axioms. In Proceedings of the 6th ACM
conference on Electronic commerce (EC), pages 1–8,
Vancouver, Canada, June 2005.
[2] A. Altman and M. Tennenholtz. An axiomatic
approach to personalized ranking systems. Journal of
the ACM, 57(4):1–35, 2010.
[3] R. Andersen, C. Borgs, J. Chayes, U. Feige,
A. Flaxman, A. Kalai, V. Mirrokni, and
M. Tennenholtz. Trust-based recommendation
systems: an axiomatic approach. In Proceedings of the
17th international conference on World Wide Web(WWW), pages 199–208, Beijing, China, April 2008.
[4] K. J. Arrow. Social Choice and Individual Values. Yale
University Press, 1951.
[5] E. Durfee, V. R. Lesser, and D. D. Corkill. Coherent
cooperation among communicating problem solvers.
IEEE Transactions on Computers, 36(2):1275–1291,
1987.
[6] Y. Emek, R. Karidi, M. Tennenholtz, and A. Zohar.
Mechanisms for multi-level marketing. In Proceedingsof the 12th ACM conference on Electronic commerce(EC), pages 209–218, San Jose, Califronia, June 2011.
[7] M. S. Fox. An organizational view of distributed
systems. IEEE Transactions on Systems, Man andCybernetics, 11(1):70–80, 1981.
[8] J. R. Galbraith. Designing Complex Organizations.
Addison-Wesley, 1973.
[9] L. Gao and J. Rexford. Stable internet routing
without global coordination. IEEE/ACM Transactions
on Networking, 9(6):681–692, December 2001.
[10] T. G. Griffin and J. L. Sobrinho. Metarouting. In
Proceedings of the 2005 conference on Applications,
technologies, architectures, and protocols for computercommunications (SIGCOMM), pages 1–12,
Philadelphia, Pennsylvania, August 2005.
[11] S. Herzog, S. Shenker, and D. Estrin. Sharing the”cost” of multicast trees: An axiomatic analysis.
IEEE/ACM Transactions on Networking,5(6):847–860, December 1997.
[12] M. Karsten, S. Keshav, S. Prasad, and M. Beg. An
axiomatic basis for communication. In Proceedings ofthe 2007 conference on Applications, technologies,
architectures, and protocols for computercommunications (SIGCOMM), pages 217–228, Kyoto,
Japan, August 2007.
[13] D. Kotz, C. Newport, R. S. Gray, J. Liu, Y. Yuan, and
C. Elliott. Experimental evaluation of wirelesssimulation assumptions. In Proceedings of the 7th
ACM international symposium on Modeling, analysis
and simulation of wireless and mobile systems(MSWiM), pages 78–82, Venice, Italy, October 2004.
[14] H. Levin, M. Schapira, and A. Zohar. Interdomain
routing and games. In Proceedings of the 40th annual
ACM symposium on Theory of computing (STOC),
pages 57–66, Victoria, Canada, May 2008.
[15] T. W. Malone. Informational efficiency in networks
and hierarchies. Sloan School of Management Working
Paper 1849, MIT, 1986.
[16] J. G. March and H. A. Simon. Organizations. John
Wiley and Sons, 1958.

[17] Y. L. Sun, W. Yu, Z. Han, and K. J. Liu. Information
theoretic framework of trust modeling and evaluation
for ad hoc networks. IEEE Journal on Selected Areasin Communications, 24(2):305–317, September 2006.
[18] T. S. Verma and J. Pearl. Causal networks: Semantics
and expressiveness. In Proceedings of the 4th
Conference on Uncertainty in Artificial Intelligence(UAI), pages 69–78, 1988.

180

Preference at First Sight
Chanjuan Liu

School of Electronics Engineering and Computer Science, Peking University
Institute for Logic, Language and Computation, University of Amsterdam

chanjuan.pkucs@gmail.com

ABSTRACT
We consider decision-making and game scenarios in which anagent is limited by his/her computational ability to foresee
all the available moves towards the future – that is, westudy scenarios with short sight. We focus on how shortsight affects the logical properties of decision making in
multi-agent settings. We start with single-agent sequential
decision making (SSDM) processes, modeling them by a new
structure of ‘preference-sight trees’. Using this model, we
first explore the relation between a new natural solution
concept of Sight-Compatible Backward Induction (SCBI)
and the histories produced by classical Backward Induction(BI). In particular, we find necessary and sufficient conditions for the two analyses to be equivalent. Next, westudy how computational complexity changes when shortsight is involved, and also, whether computationally costly
larger sight always contributes to better outcomes. Then wedevelop a simple logical special-purpose language to formally
express some key properties of our preference-sight models.
Lastly, we show how short-sight SSDM scenarios call for
substantial enrichments of existing fixed-point logics thathave been developed for the classical BI solution concept.
We also discuss changes in earlier modal logics expressing‘surface reasoning’ about best actions in the presence ofshort sight. Our analysis may point the way to logical andcomputational analysis of more realistic game models.

1. INTRODUCTION
There is a growing interest in the logical foundations,computational implementations, and practical applicationsof single-agent sequential decision-making (SSDM) problems[13; 18; 9; 16; 14] in such diverse areas as Artificial Intelligence, Control, Logic, Economics, Mathematics, Politics,
Psychology, Philosophy, and Medicine. Making decisions iscentral to agents’ routine and usually, they need to makemultiple decisions over time. Indeed, a current situation isa result of past sequentially linked decisions, each impacted
by the preceding choices.
It is quite natural in sequential decision-making scenarios,
particularly, in large systems, that agents may have someuncertainties and limitations on their precise view of theenvironment. The current literature [18] has studied uncertainty which an agent faces in recognizing possible outcomes
after taking an action and the probabilities associated withthese outcomes, as well as the partial observability of what
the actual state is like. In addition to these, a realistic aspectthat affects a SSDM process is the short-sightedness of theagent, which blocks a full view of all the available actions.

181

Short sight plays a critical role in such a situation, since,
while making a choice, the ability to foresee a variety ofalternatives and predict future decision sequences for eachof them, may make a significant difference. Nonetheless,such restrictions have not been discussed systematically yet
in decision theory or game theory.
In [6], a game-theoretic framework called games withshort sight was proposed. This framework explicitly models
players’s limited foresight in extensive games and calls
for a new solution termed as Sight-Compatible Backward
Induction (SCBI). However, many essential issues related
to sight remain unclear, such as: What is the exact
role of sight? Will the outcome be better when sight islarger? What is the relation between SCBI and classicalbackward induction(BI)? There are also unexplored issues
pertaining to logical aspects. Which minimal logic is needed
for formally characterizing a short-sight framework? Are
existing logics for BI still applicable, or can they be extended
to fit short-sight scenarios? How different are the logicalproperties of the game frames for SCBI and for BI? Without
such a logical analysis, the framework of [6] does not sufficefor disclosing the general features of short sight and thechanges it brings about in thinking about decisions andgames. Additionally, in multi-player games, short sight has
to interact with many other factors, such as agents’ mutual
knowledge and interactive decisions and moves.
Having said this, we still start by focusing on short sightin single-agent sequential decision-making process. For this,
we propose a model of ‘preference-sight trees’ (P-S trees). As
the term says, a P-S tree combines the agent’s preference
and its sight, as both are essential to decision problems [21].
We will study how the two are correlated, and cooperate toact on decision-making processes and their final outcomes.
As a preliminary illustration, consider the connection
between larger sight and better outcome. A first impression
might be that an agent will always perform better withlarger sight. Surprisingly, this is not always true. Sometimes, one can see much further into the future but receive asmall payoff, while having one’s vision restricted to a limited
set of future alternatives yields a better payoff.
Example 1.1. Alice has to make sequential decisions at
two stages (shown in Figure 1). For each stage, she canchoose either L or R. Assume that the preference order(from most preferable to least preferable) among the four
outcomes is RR, LL, RL, LR. Now consider two cases:
Case 1. At the start, Alice sees two paths, LR and RL.
She chooses R since it initiates RL which is preferable to
LR. At the second-stage, Alice then foresees RR and RL.

She happily makes the best decision RR.
Case 2. Alice sees more, e.g., LL, LR, and RL,
immediately at the first stage. Therefore she thinks that L isa better initial choice than R. Consequently, at the secondstage, she can only choose from LL and LR.
Conclusion: Even though Alice could see more in Case 2,
she ultimately obtains a less preferable outcome.

L

L

R

R

L

R

Figure 1: Two-stage decision-making
This example demonstrates some of the crucial featuresthat govern SSDM situations:
1) What an agent can foresee plays a crucial role in thedecision-making process, since her sight determines the setof available choices.
2) Sight also updates her preferences over the options, andthereby the outcomes obtained in rational play.
3) Although in Case 2, Alice does not get the best result,
we can say that, given her sight, she plays optimally in a
local sense. In other words, this is a rational plan for her,even though it is not equivalent to the rational outcome ofclassical decision theory or game theory [19].
In this paper, we address all three challenges, but first
we clarify our approach. To focus on sight, we ignore otherfactors such as the probability of moves by Nature. Also, we
model the outcome of a decision as completely determined,
or in other words, possible outcomes for each alternative
and the probability corresponding to each outcome areencapsulated as a black box.

2.

MODELING SINGLE-AGENT SEQUENTIAL DECISION-MAKING

We begin by defining a structure called preference-sighttree for modelling single-agent sequential decision-making(SSDM) processes. Using this model, we then clarify the rolethat sight plays by discussing a series of changes it produces
in agent’s preferences, decision-making procedures and theiroutcomes, as well as computational complexity.

2.1

ModelsThere are two kinds of models for decision-making scenarios corresponding to two perspectives. One is an explicit model from the perspective of Nature, or an outsider/designer; the other is the implicit model from theperspective of the agent involved, or an insider/decider.
The former is complete and perfect in the sense that theoutsider holds a full view of all the options together withthe objective quality of these options, and thus can explicitly
specify the reward of each situation for the decision-maker.
In contrast with this, the latter’s views are possibly limited
to a near future, especially in large-scale surroundings.
Moreover, owing to limited foresight, the agent may also
reason mistakenly about the quality of different choices,leading to what we call subjective preference.
Both the above perspectives are essential: the former
offers a whole picture of the environment, the latter shows

the actual play of the decider. In this section, we first
introduce an explicit model of preference trees. After this,by endowing such trees with the agent’s view of the process
and his/her subjective preference in this view, we formulate
an integrated model of preference-sight trees which allows us
to model both perspectives together.

2.1.1

Preference trees (P trees)

A preference tree is a decision tree with only two elements:
histories and preferences. Each history corresponds to asituation resulting from previous decision actions, and apreference represents the objective quality of each of thesesituations. To ensure the existence of backward induction
solutions, we confine ourselves to finite histories.
Definition 2.1. (Preference tree) A preference tree isa tuple T = (H, ) where H is a non-empty set of finite
sequences of actions, called histories;  is a total order
over H. The empty sequence ε is a member of H; If(ak )k=1,...,K ∈ H and L < K then (ak )k=1,...,L ∈ H.
Let A denote the set of all actions. Any history h can be
written as a sequence of actions: (ak )k=1,...,n , where eachak ∈ A. If there is no an+1 s.t. (ak )k=1,...,n+1 ∈ H, thenhistory (ak )k=1,...,n is a terminal one. The set of terminal
histories is denoted Z. The set of actions that are available
at h is denoted A(h) ⊆ A. For any histories h, h0 , if h is aprefix of h0 we write h  h0 . The strict part of  is , withh1  h2 if h1  h2 and not h2  h1 for any two histories h1
and h2 . Accordingly, h1 ∼ h2 iff h1  h2 and h2  h1 .

Several remarks need to be made on the role of preference
relations in the above definition:(1) Instead of defining preference merely over terminal
histories, we have defined it over all histories, an idea goingback to [11]. Here preference over intermediate histories
is necessary for our aim of modelling an agent’s decisionmaking under limited foresight, which usually consists ofintermediate histories.
(2) For convenience, we do not strictly differentiate thetwo main views of preference: qualitative and quantitative.
Although we use qualitative order generally, we sometimes
switch to numerical payoff when it is advantageous.1

2.1.2

Preference-sight Trees (P-S trees)

P tree is an explicit model for decision-making scenarios
which is independent of an agent. However, for an agent,
the tree may appear differently in his/her limited view. [6]proposes the idea of short sight, where the authors use asight function to denote the set of states that players canactually see at every position in an extensive game. Let us
start by adapting their technique to preference trees.
Definition 2.2. Let T = (H, ) be a preference tree.
A sight function for T is a function s : H → 2H \{∅}
satisfying s(h) ⊆ H|h and |s(h)| < ω, where H|h represents
the set of histories extending h. As a special case, h ∈ H|h .
In words, the function s assigns to each history h a finite
subset of all available histories extending h.
The first effect that sight produces is that given a P tree,
for any history h, it always gives us a restricted tree.
1
There is a debate on whether preference and utilities are thesame [9; 2]. Here we adopt the operational understandingof utility and do not distinguish it from preference.

182

Definition 2.3. Let T = (H, ) be a P tree. Given
any history h of T , a visible tree Th of T at h is
a tuple (Hh , h ), where Hh = s(h), i.e., Hh captures
the decider’s view of the decision tree; h represents the
subjective preference over Hh .
A visible tree is actually an implicit model in our earlier
terms. Hh also contains a set of terminal histories Zh , which
are those without successors in s(h). Note that typically, theZh are non-terminal for T .
Further, the preference order h is different from theobjective preference . In fact, the formation of h is an
update via a bottom-to-top process in terms of an agent’s
sight. This updating process involves leaving the payoffs of
Zh as the same as their objective payoffs, then updating thepayoffs of other histories in Hh backwards, starting from theleaf nodes and proceeding towards the root of the tree.
The reason why we employ such an updating process isthat, while the objective payoffs reflect the goodness of thesesituations, they are not the actual reward that an agent can
get if he/she chooses this option. At each decision point, thesubjective payoff of one available option is inherited from thebest reachable terminal histories of the current visible tree.
Therefore, the preference relation h in Th is not always
consistent with the preference relation  in T .
This updating process is described by Algorithm 1, which
essentially involves a backward computation and update of
the preference over intermediate nodes within the sight:*For convenience, here we use payoffs P to represent rewards.

Algorithm 1: Preference updating in visible trees
PU(T, h, s)
Input: A P tree T = (H, ) (or T = (H, P )), current
history h, and a sight function s
Output: A visible tree Th = (Hh , h ) or
(Th = (Hh , Ph ))2 begin3
H ∩ s(h) → Hh ;4
for any z ∈ Zh /* Keep the payoffs of terminal
histories unchanged */ do5
P (z) → Ph (z); 1 → flag[z];
1

6
7
8

9
10
11

while flag[h] == 0 do
for any h0 ∈ Hh do
if (for all (h0 a) ∈ Hh , flag[(h0 a)] == 1)/* If all of its children have been
visited, reset its payoff as thehighest one among them */thenmax{Ph (h0 a)} → Ph (h0 ); 1 → flag[h0 ];Return Th ;Fact 2.1. Let T = (H, ) be a P tree. Each visible tree
Th = (Hh , h ) is a P tree.
Correspondingly, we denote the prefix relation in Th byh , and the actions that are available at h by Ah (h).
Finally we proceed to define our model of preference-sighttrees. A preference-sight tree allows us not only to represent
the outsider’s view, i.e., (H, ), but also to derive a seriesof implicit models, i.e., (Hh , h ), one for each h.
183

Definition 2.4. (Preference-sight tree) A preferencesight tree (P-S tree) is a tuple (T, s), where T = (H, ) isa preference tree and s a sight function for T .
In P-S trees, an agent’s sight should satisfy the following
properties: First, if an agent can see a given future history,
then he/she can also see any intermediate history up tothat point. Second, if the agent can see a history two steps
forward, then after moving one step ahead, he/she can stillsee it. These features are formally stated as follows.
Fact 2.2. (Properties of sight function) Let (T, s) be a
P-S tree. For all h, h0 , h00 ∈ H, with h  h0  h00 , s satisfies :DC (Downward-Closed): if h00 ∈ s(h), then h0 ∈ s(h).
NF (Non-Forgetting): if h00 ∈ s(h), then h00 ∈ s(h0 ).

2.2

Solution concepts

Solution concepts are at the center of all choice problems.
In what follows, we define two solution concepts for PS trees, adapted from [20; 6]. After this, we investigate
the conditions for their equivalence, followed by providing
procedures for calculating the number of them.

2.2.1

BI history and SCBI history

Backward Induction (BI) is a well-known process runninglike this. First, one determines the optimal strategy ofthe player who makes the last move of the game. Using
this information, one can then determine the optimal actionof the next-to-last moving player. The process continues
backwards in this way until all players’ actions have been determined in the whole game. Its adaptation to single-agent
decision-making process becomes a maximality problem forthe agent involved.
In a P-S tree, we say that one history h is max in a setof histories Γ ⊆ H, if h ∈ Γ and for any other history h0 in
Γ, it holds that h  h0 , and we write this as h ∈ max Γ.
The strict part for max is max .
Definition 2.5. (BI history) Let (T, s) be a P-S tree. A
history h∗ ∈ Z is a BI history of T , iff h∗ ∈ max Z. Also,
we use BI to denote the set of BI histories in T .
A BI history of a P-S tree is a terminal history that ismost preferable or equivalently, that has a maximal payoff.
Backward induction precludes short-sight, while in practice it is impossible for an agent to foresee all final outcomes
all the time. In [6], a new solution concept was proposed
to capture optimal play of short-sighted players: sightcompatible subgame perfect equilibrium. The main idea isthat at each decision point, the current player chooses a
locally optimal move by a local BI analysis within the visible
part. Here, we adapt this notion to P-S trees, yielding thesight-compatible backward induction history.
Definition 2.6. (SCBI history) Let (T, s) be a P-S tree.
A history h∗ ∈ Z is a Sight-Compatible Backward
Induction history (SCBI history) of T , iff for each history
h with h  h∗ , and the action a following h, i.e., (ha)  h∗ ,we have that ∃z ∈ max Zh such that (ha)  z. Also, we use
SCBI to denote the set of SCBI histories in T .
The difference between SCBI and BI histories is obvious.
A BI history is one with highest payoff among the set ofterminal histories in the P-S tree, while for a SCBI history

every restriction of it should be a local BI history forthe visible tree. Thus, BI histories are the BI outcomes
for the objective model (H, ), while SCBI histories are
a combination of best responses to all subjective models(Hh , h ). Typically it is the case that SCBI 6= BI.
Example 2.1. Consider the P-S tree (T, s) in Figure 2,
where s(ε) = {L}, and s(L) = {LR}. It is easy to checkthat BI 6= SCBI, since BI = {LL}, while SCBI = {LR}.
R

L

1

2
R

L

2

R

L

1

1

0

Figure 2: BI 6= SCBI
However, sometimes the two notions can be equivalent.
Example 2.2. Consider a P-S tree, with T and s shownby Figure 3 (a), and Figure 3 (b) respectively. In (b) the
three dotted circles represent s(ε), s(L) and s(R). For
histories L and R, their objective payoffs in (a) are 1 and
2, respectively. However, in Tε , the subjective payoff of L is
updated to 3 and R to 2. Obviously, BI = SCBI = {LL}.
L

L

R

1

2

Proof. The situation in Figure 3 is a counterexample,
in which BI = SCBI = {LL}, but (T, s) is not preferencesight consistent, since R  L and L ε R.

What is the exact condition for BI = SCBI? From thefailure of preference-sight consistency, we can draw a lesson.
In Figure 2, the main reason for (T, s) being inconsistentis that at history L, the branch LL, which in fact forms
a BI history, is non-observable to the agent. This tells usthat the one with maximal payoff should always be visible.
Consider then the example in Figure 3. Here all the options
are within agent’s sight, but we notice that although thepath LL following L finally turns out to be better than that
following R, which makes subjectively L ε R, the objective
payoff of L itself is lower than R. Thus, it fails to imply theconsistency between preference and sight.
Based on the above analysis, we now isolate necessary
and sufficient conditions for BI = SCBI. First, we define
an auxiliary property of sight-reachability, which intuitively
reflects whether each restriction of a history is visible.
Definition 2.8. (Sight-reachability) A BI history h∗ issight-reachable if, for all (ha)  h∗ , we have (ha) ∈ Hh ,where h, h0 are histories, and a is an action following h.

R

3Theorem 2.5. (Equivalence Theorem) For any P-S tree
(T, s), SCBI= BI iff the following conditions are satisfied:

2

L

R

L

R

L

R

L

3

1

2

1

3

1

2

(a)

R

1

(b)

Figure 3: (BI = SCBI)

2.2.2

Next, does the other direction hold?
Fact 2.4. Preference-sight consistency does not follow
from BI = SCBI.

I). Any history h∗ ∈ BI is sight-reachable.

II). Any history h∗ ∈ BI is locally optimal: For any history(hh0 )  h∗ , if (hh0 ) ∈ Zh , then (hh0 ) ∈ max Zh andfor any other (hh00 ) ∈ Zh , (hh0 ) ∼ (hh00 ) iff ∃z ∈ BI
such that (hh00 )  z.
Proof. (⇒) I). We show that every h∗ ∈ BI is sightreachable. That is, for all (hh0 )  h∗ , it holds that(ha) ∈ Hh . By SCBI= BI, we know that any historyh∗ in BI, is also in SCBI. By Definition 2.6, for eachof its prefix h, h∗h is max in Zh . So h∗h is in Zh . In
addition, by non-emptiness of Zh , h∗h is not an empty
sequence. Thus, for all (ha)  h∗ , it holds that (ha) ∈Hh . So h∗ ∈ BI is sight-reachable.

Equivalence condition

Then an interesting question on BI and SCBI histories
arises: are there conditions under which the two will beequivalent? To get a feeling for this, a first attempt at an
answer looks for a condition related to consistency between
subjective and objective preferences.
Two histories are said to be ‘preference-sight consistent’
if the subjective preference in each sight-restricted tree isconsistent with the objective preference over them:

To show condition II), take any h∗ in BI, we have that
it is in SCBI. Thus, for all (hh0 )  h∗ , if (hh0 ) ∈ Zh ,then (hh0 ) is max in Zh . Moreover, for any (hu) ∈ Zhsuch that (hh0 ) ∼ (hu), we have (hu) is a prefix of a
BI history, i.e., (hu) ∈ BIh . For suppose not, then(hu) is not a prefix of SCBI history. Then it must be(hh0 )  (hu). Contradict.

Definition 2.7. (Preference-sight consistency) Let (T, s)
be a P-S tree, and Th be the visible tree at an arbitrary
history h. Then for any two histories h1 , h2 of Th , we say(h1 , h2 ) satisfies preference-sight consistency at h iff
h1  h2 iff h1 h h2
If for any history h ∈ T , the pair of arbitrary two histories(h1 , h2 ) in Th is preference-sight consistent (at h), then wesay (T, s) is preference-sight consistent.

(⇐) Suppose conditions I) and II) are satisfied. It suffices
to show (a)“every BI history is SCBI history of T ”, and(b) “ every SCBI history is BI history of T ”.

Fact 2.3. Preference-sight consistency does not guarantee that BI = SCBI.

For (a), take any BI history h∗ . By I), all BI histories
are sight reachable. Further by II), for all (hh0 )  h∗ ,if (hh0 ) ∈ Zh , then (hh0 ) is max in Zh . This is to
say that for each of its prefix h, h∗h is max in Zh . By
definition 2.6, h∗ is a SCBI history.

Proof. Consider Figure 2. Suppose that s(R) contains
only one successor. Then it is easy to see that (T, s) ispreference-sight consistent. However, BI 6= SCBI.

For (b), take any SCBI history h∗ . We can show it isa BI history, i.e., h∗ is max in Z. For suppose not,then there exists a BI history h0 such that h0  h∗ .

Is preference-sight consistency an appropriate condition
for BI = SCBI? We have the following observation:

184

Notice that there must be some history u which is thecommon prefix of h∗ and h0 . Since h0 is a BI history,
by condition I) and II), we know that h0u  h∗u . Thenh∗u is not a prefix of a SCBI history. Thus, h∗ is not a
SCBI history. Contradiction.

2.2.3

Proof. Case (a) has been shown in Example 1.1. Case(b): Obviously, Figure 2 offers an instance for this. Case(c): The scenario depicted in Figure 3 is an example.
In conclusion, full sight guarantees a maximal payoff.
However, with short sight, increase of sight does not always
improve the outcome. The added sight may bring misleading
information, e.g., a branch which is temporarily nicer but
actually unpromising, and finally gives rise to an even worse
outcome. Still, this does not mean that SCBI is deficient:rather, these observations seem realistic for real agents.
These issues will be discussed further in Section 4.

A LOGICAL ANALYSIS

After modelling decision-making with short sight by preference tree models, it is instructive to see what a logical language looks like for reasoning about these models, especiallythe role of sight in a SSDM process. So far, no such logic has
been proposed, though logics of game-theoretic structures
have been extensively studied – see [27; 10] – while there
are a few preliminary logic analyses of sight on its own, [4;17]. In this section, we design a minimal and natural logicalsystem that supports reasoning about sight in the context of
single-agent decision-making processes, characterizing basic
properties of preference-sight trees, and formally capturing
the results in the previous section.

3.1

ϕ ::= p |¬ϕ |ϕ ∧ ψ |[!ϕ]ψ | Aϕ.

More sight, better outcome?

We have seen earlier on that, SCBI may loss global
optimality. The BI history definitely has a maximal payoff,
while it might not be the case for SCBI, since each action ischosen with a limited sight. So BI  SCBI holds without
exception, in the sense that any BI history is no worse thanany SCBI history. One might conjecture that more sightalways contributes to better outcomes. Yet, the fact below
falsifies this.
Fact 2.6. Let T be a P tree. Also, let s1 and s2 be twosight functions for T satisfying s1 (h) ⊆ s2 (h) for any history
h in T . Take any two SCBI histories z1 and z2 of (T, s1 )and (T, s2 ) respectively. Then the following three cases are
all possible: a) z1  z2 ; b) z2  z1 ; c) z1 ∼ z2 .

3.

Definition 3.1. (Preference-sight language) Take any set
of atomic letters P (T,s) . The preference-sight language
L is given by the following BNF, where p ∈ P (T,s) :We write h!ϕiϕ to abbreviate ¬[!ϕ]¬ϕ.
Definition 3.2. (Preference-sight models) For a P-S tree
(T, s), a preference-sight model M (T,s) is a tuple (H, , V)
where the following holds:• H is the set of possible worlds, one for each history,•  is the reachability (prefix) relation among worlds,• V : PT → ρ(H) is an evaluation function satisfying:(1) ∀h ∈ H, V(h) = {h0 |h0  h}.
(
H, IF h1  h2 ,(2) V(h1 ≥ h2 ) =∅, Otherwise.
(3) ∀h ∈ H, V(s(h)) =To reason about the key ingredients (i.e., histories, preferences, and sights) of a P-S tree, we take P (T,s) as a set of
propositional letters, which at least contains the following 2 :• h for each history h.

• h1 ≥ h2 encoding the preference relation of the agent
over all histories, and the strict part of which is h1 > h2 .

There seems to be nothing striking in this syntax. However, given the special role of atoms, the natural model
update differs from the usual one in dynamic-epistemic logic.
Definition 3.3. (Model update) Given a preference-sightmodel M (T,s) = (H, , V) and a set X ⊂ H, the updated(T,s)
model M!X produced by the restriction of X is defined asa tuple (X,  ∩ X 2 , V!X ), where 3
V!X (p) =

185


X,









∅,

IF p is of the form h1 ≥ h2
Otherwise

IF V(z1 ≥ z2 ) = H, wherez1 ∈ max {z ∈ ZX |h1  z},
z2 ∈ max {z ∈ ZX |h2  z}
Otherwise

M!X is the update of the model M (T,s) restricting theset of states to X, and the valuation function accordingly.
But crucially, the valuation for preference atoms in the newmodel reflects the updating process in the visible tree of
Algorithm 1. In the following, we omit superscripts (T, s).
The semantics for this language is basically standard, [3],so we only mention the truth condition of [!ϕ]ψ:Let M be a preference-sight model. For any state h in M ,M, h |= [!ϕ]ψ iff M, h |= ϕ ⇒ M!JϕK , h |= ψ,Based on P (T,s) , we give a language L for reasoning about
P-S trees. In L, we have a key dynamic operator [!ϕ]for restricting to the worlds satisfying ϕ, and a universal
modality with Aϕ saying that ϕ is true in every world.
The idea of defining h is motivated by [1], where the authors
define an atomic sentence o for each leaf in a game tree.

(
V!X (h1 ≥ h2 ),V(p) ∩ X,

V!X (h1 ≥ h2 ) =• s(h) encoding the sight at each history h in T .

2

V(h0 ).

Intuitively, h is true at all the worlds leading to h. h1 ≥ h2
is true everywhere if h1  h2 , and nowhere otherwise.
Finally, V(s(h)) is a union of the worlds that make the given
atom true for at least one element of s(h).

(T,s)

Syntax and Semantics

S

h0 ∈s(h)where JϕK = {h0 ∈ H|M, h0 |= ϕ}.

Validity of formulas is defined as usual, cf. [3].
3

In this definition, ZX denotes the terminal histories in X,
i.e., the set of histories that have no successors in X.

3.2

Main characterization results

Despite its simplicity, L can express our results in previous
sections concerning properties and solutions of P-S trees.
We introduce some helpful syntactic abbreviations, and thenstate our main characterization results.
W• Zh = { z | z ∈ Zh }.
W• max≥ X= { h | h ∈ X, and h  h0 for ∀h0 ∈ X}.
W• BI = { z | z ∈ BI} (BI holds at T ’s BI histories).
W• SCBI = { z | z ∈ SCBI}, that is, the formula SCBI
holds at the SCBI histories of T .
Proposition 3.1. Let (T, s) be a P-S tree and M be a
L-model for it. Then (T, s) is preference-sight consistent iff
the following formula is valid in M :^ ^^((h1 ≥ h2 → [!s(h)]h1 ≥ h2 )∧h h1 ∈Hh h2 ∈Hh(h!s(h)ih1 ≥ h2 → h1 ≥ h2 )).

Lemma 3.2. For any P-S tree (T, s) and model M for it,a BI history h∗ is sight-reachable if and only if the following
formula holds in M :V V(A((ha) → h∗ ) → (A((ha) → s(h)))).
(SR) :
h a∈A(h)Proof. (⇒) Suppose that BI history h∗ is sight-reachable.
By Definition 2.8, we have that, for all (ha)  h∗ , it holds
that (ha) ∈ s(h), where h, h0 are histories, and a is anaction following h. More formally, (ha)  h∗ can be definedby the formula A((ha) → h∗ ) in the sense that, in T , forall h and a ∈ A(h), (ha)  h∗ iff M |= A((ha) → h∗ ).
And similarly (ha) ∈ s(h) is defined by A((ha) → s(h)).
Thus if a BI history h∗ is sight-reachable, then M |=V V∗
ha∈A(h) (A((ha) → h ) → (A((ha) → s(h)))). The otherdirection can be proved in a similar way.
Lemma 3.3. Let (T, s) be a P-S tree and M be a L-model
for it. A BI history h∗ is locally optimal iff the following
formula is valid in M :^ ^(LO) :((A((hh0 ) → h∗ ) →h (hh0 )∈Zh^

(hh00 )∈Zh

(A((hh0 ) → max Zh )∧_((hh0 ) ∼ (hh00 ) ↔
(A((hh00 ) → z))))).
z∈BI

Proof. (⇐) Suppose BI history h∗ is locally optimal.
Then for (hh0 )  h∗ , if (hh0 ) ∈ Zh , we have (hh0 ) is maxin Zh . And for any (hh00 ), (hh00 ) ∼ (hh0 ) iff ∃z ∈ BI s.t.
(hh00 )  z. Similar with the above proposition, A((hh0 ) →h∗ ) captures that (hh0 )  h∗ . And A((hh00 ) → z) shows that(hh00 )  z. Finally, ((hh0 ) → max Zh ) demonstrates that(hh0 ) is max in Zh . Direction (⇒) uses a similar check.
Proposition 3.4. (L-characterization of equivalence) Let(T, s) be a preference-sight tree and M a model for it. Thenthe following formula is valid in M :
^

h∗ ∈Z

|= (A(BI ↔ SCBI)) ↔

((A(h∗ → BI)) → (SR ∧ LO)).

Proof. Direction (⇒). We need to prove the following:V1) (A(BI ↔ SCBI)) → h∗ ∈Z (A(h∗ → BI) → SR).
V
2) (A(BI ↔ SCBI)) → h∗ ∈Z (A(h∗ → BI) → LO).
For 1). It is equivalent to prove that, for any h∗ ∈ Z,
(BI ↔ SCBI) ∧ (A(h∗ → BI)) → SR. Suppose ¬(SR).
Then ∃(ha)  h∗ , and (ha) ∈/ Th , and so, at h, the branchleading to h∗ is not visible in Th . Thus, the BI history in
Th could not be a branch leading to h∗ . By the definition
SCBI, it follows that h∗ ∈/ SCBI. However, by h∗ → BI weknow that h∗ is a BI history. This contradicts BI ↔ SCBI.
2) can be proved in a similar style.
Direction (⇐). Suppose that ¬(A(BI ↔ SCBI)). Then(a): ∃z ∗ ∈ BI and z ∗ ∈/ SCBI, or(b) : ∃z ∗ ∈ SCBI and z ∗ ∈/ BI.
If (a), then, by the antecedent, we have that: ∀(ha) z ∗ , (ha) ∈ Hh . Also, ∀(hh0 ) ∈ Zh and (hh0 )  h∗ , it holds
that (hh0 ) ∈ max Zh . Then it directly follows that z ∗ is a
SCBI history. Contradiction.
If (b), then take any z ∈ BI, which shares a prefix uwith z ∗ , i.e., u  z and u  z ∗ . By the antecedent, we havezu ∈ max Zh . Since z ∗ ∈/ BI, it follows that zu > zu∗ . Then∗z ∈/ SCBI. Once more, we have a contradiction.

3.3

Valid principles

The operator [!ϕ] makes L a PAL-like language. However,the special model-update makes it different from standard
PAL [28]. This suggests a close look at what is and what is
not valid in preference-sight models.
First, some axioms in standard PAL do not hold inpreference-sight models. For example, the !ATOM axiom,[!ϕ]p ↔ (ϕ → p), is not valid when it is of the form below.
Proposition 3.5. The following is not valid in preferencesight models, where h, h1 , h2 represent arbitrary histories.
!Sight-Preference :[!s(h)]h1 ≥ h2 ↔ (s(h) → h1 ≥ h2 ).
This proposition says that subjective preference in visible
trees is not necessarily consistent with objective preference.
Now let us see some interesting valid principles and theirintuitive interpretations.
Lemma 3.6. The formulas shown in Table 1 are valid,
where h, h1 , h2 , and h3 are arbitrary histories.
Proof. We only prove some cases, proofs for the others
are trivial or standard.
For Ts . Take any state u with M, u |= h. Then u ∈ V(h).
As the sight function is reflexive, i.e., h ∈ s(h), it holds that
V(h) ⊆ V(s(h)). So u ∈ V(s(h)). Thus, M, u |= s(h).
For T M . Take any state u, any history h and any z ∈ Z,
and suppose M, u |= A(z → h). Then for any u0 , u0 ∈ V(z)implies that u0 ∈ V(h). Thus, V(z) ⊆ V(h). It follows thatz ∈ V(h). Given that z is terminal, by the definition of V(h),it must be that h = z. Thus, M, u |= A(h → z).
For DC. Take any state u, suppose for some h1  h2  h3 ,M, u |= A(h3 → s(h1 )). Then we know V(h3 ) ⊆ V(s(h1 )).
It follows that h3 ∈ s(h1 ). As the sight function is downward
closed, we have h2 ∈ s(h1 ). Thus, M, u |= A(h2 → s(h1 )).
For !ATOM\SP. Take any state u, and let M, u |= [!ϕ]p where
ϕ is not of the form !s(h) and p is not of the form h1 ≥ h2 . It
holds that M, u |= ϕ implies that M!ϕ , u |= p. By Definition
3.3, M!ϕ , u |= p iff M, u |= p. Therefore, M, u |= ϕ implies
M, u |= p. Equivalently, then, M, u |= ϕ → p.
186

Taut

all propositional tautologies

T≥
4≥
to≥

h≥h
h1 ≥ h2 ∧ h2 ≥ h3 → h1 ≥ h3
h1 ≥ h2 ∨ h1 ≥ h2

TsTM

h → s(h)V V(A(z → h) → A(h → z))
z∈Z h
V V
V
(A(h3 → s(h1 )) → A(h2 → s(h1 )))h3 h2 h3 h1 h2
V
V V(A(h3 → s(h1 )) → A(h3 → s(h2 )))DCNF

h3 h2 h3 h1 h2!ATOM\SP
!NEG
!CON
!COM
Dual

[!ϕ]p ↔ (ϕ → p)
(excluding the schema !Sight-Preference)[!ϕ]¬ψ ↔ (ϕ → ¬[ϕ]ψ)[!ϕ](ψ ∧ χ) ↔ ([!ϕ]ψ ∧ [!ϕ]χ)[!ϕ][!ψ]χ ↔![ϕ ∧ [!ϕ]ψ]χ
[!ϕ]ψ ↔ ¬h!ϕi¬ψ
Table 1: Valid principles of L

Interpretation of valid principles. Each of theseaxioms has some intuitive appeal. T≥ , 4 and to≥ show thereflexivity, transitivity and totality of the preference relation,
respectively. Likewise, Ts says that sight is reflexive. DC
characterizes the (downward-closure) property of sight. N F
encodes the non-forgetting property of sight. TM guarantees
that terminal histories of the P-S tree are actually terminal.
One further interesting point is that there is no correspondence of TM for terminal histories of visible trees.
Fact 3.7. The following formula is not valid in preferenceVsight models: V V(A(z → h) → A(h → z)).
u

z∈Zu

h

Other validities in the table are axioms for standard PAL.
We postpone the study of a complete axiomatization of thelogic L until future work.
To conclude this section, in L, the ingredients including
histories, preferences and sights are encoded as primitive
propositions. Various earlier phenomena in P-S trees canthus be captured in a simple, direct and intuitive manner.
This special-purpose logic, as we will see soon, is modeldependent, but it can also be formulated generically.

4. BACKGROUND IN GAME LOGICS
In this section, we relate our logic L to existing logics forclassical game theory, showing how ideas can be combined
where useful. Since so far we have been working with BI
and SCBI histories, we first define strategies for P-S trees:
A strategy for a P-S tree (T, s) is a function σ : H → A
such that σ(h) ∈ A(h). That is, σ assigns each history han action that follows h. In particular, for a visible tree Th ,a ‘local strategy’ σh is a restriction of σ to Th , such thatσh (h0 ) = σ(h0 ) for any h0 ∈ Th .

4.1

Generic formulation of L

In applied logic for structure analysis, there exist two extremes, viz. model-dependent ‘local languages’ and ‘genericlanguages’ that work across models. For a generic logic,
a definition of a property π is a formula ϕ such that for
all models M , M has property π iff M |= ϕ. For a local
language, such a formula can depend on a given model M :187

there exists a formula ϕM which depends on M , such thatany model M has the property π iff M |= ϕM . However, in
this case, the defining formula can be trivial. For example,
one might define ϕM simply as follows.
(>, if M satisfies π
ϕM =⊥, Otherwise
In this subsection, using a well-known Rationality property as an example, we discuss how model-dependent ourearlier language L is, and then show how it can be formulated in a generic way. We first recall the results on classical
BI. Given that we have been dealing with single-agent cases
until now, in this Section, we will adapt the results from theliterature on multi-player games to the single-player case.
The BI strategy [22; 23] is the largest subrelation σ of thetotal move relation that has at least one successor at eachnode, while satisfying the rationality (RAT) property:
RAT
No alternative move for the player yields an
outcome via further play with σ that is strictly better than
all the outcomes resulting from starting at the current moveand then playing σ all the way down the tree.
As argued in [22; 23], this rationality assumption is a
confluence property for action and preference:CF∀x∀y(xσy → ∀z(x move z →∃u(end(u) ∧ yσ ∗ u ∧ ∀v((end(v) ∧ zσ ∗ v) → u ≥ v)))).
We can observe that there is also a corresponding rationality property for the local BI strategies that constitute
an SCBI, which should however now express a confluence
property for action, preference and sight. Specifically, for a
P-S tree, each local BI strategy for the visible tree Th at h is
the largest subrelation σh of the total move relation in Th ,satisfying 1) σh has at least one successor at each h0 ∈ Th ,and 2) the following rationality property holds:
RATS In the visible tree, there is one outcome obtainedby playing σh from the start to the end, that is no worse
than all the outcomes yielded from any alternative first movefollowed by further play with σh .
This confluence property involving sight is expressible asfollows in our language L:

Proposition 4.1. Let (T, s) be a P-S tree, and let M be
any model for it. M satisfies RATS iff M validates thefollowing L-formula, where σh is the BI strategy for visible
tree at h and where (h(σh )k ) stands for the history reachedfrom h after executing σh for k times.
^ __(A((h(σh )k ) ↔ z)
CFSM
^ h z∈Z^ h k=l(z)−l(h)^→((A((ha0 (σh )m ) ↔ z 0 )) →a0 ∈Ah (h) z 0 ∈Zh m=l(z 0 )−l(ha0 )z ≥ z 0 )).

Proof. We first claim that in any preference-sight model
M , and state h ∈ H, for any terminal history z ∈ Zh ,and h0 ∈ Hh , A(h0 ↔ z) implies that h0 = z. This isstraightforward since A(h0 ↔ z) demonstrates that prefixesof h0 are the same with those of z, which means that h0 = z.
Then M |= CFSM says that there is a terminal history
zh following h by playing a local BI strategy σh , such thatz  z 0 for any other z 0 ∈ Zh which follows an alternative
first move a0 ∈ Ah (h) via further play of σh . Therefore, weknow that M satisfies RATS.

However, compared with the generic logic in [24; 23; 22],
the given definition in our logic is local. It is obvious that
CF, the formula defining the property RAT, is insensitive
to models – while our CFSM relies on a given model for
its ranges of big disjunctions and conjunctions, and in its
model-dependent notations like s(h) and h1 ≥ h2 . Still, it
is also clearly true that our definition is not as trivial as theearlier local trick. Therefore, our logic L seems somewhere
between the two extremes of locality and genericity. Thisfeeling can be made precise by moving to a closely related
truly generic first-order logic of preference-sight trees.
The relevant modified formula involves some natural
auxiliary predicates. x  y says that x is a prefix of y; x ^ ymeans that x can see y. Corresponding to the BI relationσ, yσ(x)z says that from y, z is a local backward induction
move in the visible tree at x; σ k describes σ being composedfor k times with k ∈ N 4 ; move and ≥ are still the move
relation and preference relation, respectively, of the game.
Proposition 4.2. Any model M satisfies RATS iff itvalidates the following formula.
CFS(F O):∀x{(∃y(x  y)) →∀u[(xσ(x)u) → ∀t((x move t ∧ x^t) →∃z((x^z ∧ ¬∃z 0 (z  z 0 ∧ x^z 0 ) ∧ ∃k(u(σ(x))k z))∧∀v((x^v ∧ ¬∃v 0 (v  v 0 ∧ v^v 0 ) ∧ ∃l(t(σ(x))l v)) →∧z ≥ v)))]}.
Proof. It is easy to show that
M |= CFS(F O) iff M |= CFSM .
In summary, incorporating basic elements of P-S trees directly into first-order syntax makes L intuitive and natural.
Even so, other logics exist for dealing with further aspectsof game trees and solution procedures, and we will discussa few examples in what follows with a view to how they
behave in the presence of sight.

4.2∀x∀y∀z(bisight (x, y, z) → ∀t((see(x, t) ∧ move(y, t))→ (∃u(endsight (x, u) ∧ bi∗sight (x, z, u) ∧ ∀v((endsight (x, v)∧bi∗sight (x, t, v)) → u ≥ v))))).
Notice that all occurrences of bisight in the above formulas
are still syntactically positive. This allows us to define local
BI strategy bisight with LFP(FO).
Proposition 4.3. The strategy bisight can be defined as
the relation R in the following LFP(FO) formula.
νR, xyz • ∀x∀y∀z(R(x, y, z) → ∀t((see(x, t) ∧ move(y, t))→ (∃u(endsight (x, u) ∧ R∗ (x, z, u) ∧ ∀v((endsight (x, v)∧
R∗ (x, t, v)) → u ≥ v))))).

It can be proved formally that bisight is a greatest-fixedpoint of the above formula. Based on bisight , we now proceed
to show that the SCBI relation is LFP(FO) definable.
Corollary 4.4. The SCBI relation scbi for a P-S tree
can be represented in the following formula:∀x∀y(scbi(x, y) ↔ bisight (x, x, y)).
As in the original classical case, this LFP(FO) definabilityof scbi exposes an intersection between the logical foundation of computation and the recursive nature of sightcompatible backward induction solutions for P-S trees.

4.3

In contrast with detailed formalism of solutions with
LFP(FO), there is the modal surface logic of [25], whichenables direct and natural reasoning about best actions
without considering the underlying details of recursive computation. First of all, we list its modalities for classical BI.
[bi] and [BI] encode the BI move and BI paths respectively.
[best]ϕ says that ϕ is true in some successor of the current
node that can be reached in one step via the bi move.
M, h |= end iff h ∈ Z.
M, h |= [move]ϕ iff ∀ h0 = (ha) with a ∈ A(h), M, h0 |= ϕ.
M, h |= [best]ϕ iff for all h0 with h0 ∈ bi(h), M, h0 |= ϕ.
M, h |= [bi]ϕ iff for all h0 with h0 ∈ bi(h), M, h0 |= ϕ.
M, h |= [bi∗ ]ϕ iff M, u |= ϕ for all u with u ∈ (bi)∗ (h).
M, h |= [BI]ϕ iff for all z with z ∈ BI, M, z |= ϕ.

Solution procedures and fixed-point logics

Recursive solution procedures naturally correspond to
definitions in existing fixed-point logics, such as the widelyused system LFP(FO). An LFP(FO) formula mirroring therecursive nature of BI is constructed in [24; 26] to define
the classical BI relation, based on the above property RAT.
Now, we have shown that sight-restricted SCBI, too, is arecursive game solution procedure. Can LFP(FO) be used
to define SCBI as well – and if so, how?
The answer is yes, but we need an extension. Rather thana binary relation bi as in [24; 26], characterizing SCBI needs
a ternary relation. First, we define the local BI relation invisible trees, which will be denoted by bisight . For any states
x, y, z, bisight (x, y, z) means that in the visible tree at x, thelocal BI strategy is bisight , which chooses z when the current
state is y. It is then obvious that bisight should satisfy thefollowing simple first-order definable property, requiring therelevant states to be visible and reachable:

Modal surface logic of best action

The above logic is still applicable in our setting, but it
requires substantial extension for sight-related concepts. In
accordance with [bi] and [BI], we use [scbi] and [SCBI] as
operators for the SCBI strategy and SCBI path, respectively.
For the local BI strategy and path in visible trees, the modalities are [bisight ] and [BIsight ]. Moreover, recall that M!s(h) is
the updated model obtained in the way of Definition 3.3.
M, h |= [scbi]ϕ iff for all h0 with h0 ∈ scbi(h), M, h0 |= ϕ.
M, h |= [SCBI]ϕ iff for all h0 with z ∈ SCBI, M, z |= ϕ.
M, h |= [!sight]ϕ iff M!s(h) , h |= ϕ.
M!s(h) , u |= endsight iff u ∈ Zh .
M!s(h) , u |= [movesight ]ϕ iff for ∀u0 = (ua) with a ∈ Ah (u),M!s(h) , u0 |= ϕ.
M!s(h) , u |= [bestsight ]ϕ iff M, u0 |= ϕ for ∀u0 ∈ bih (u).
M!s(h) , u |= [bisight ]ϕ iff M, u0 |= ϕ for ∀u0 ∈ bih (u).
M!s(h) , u |= [(bisight )∗ ]ϕ iff M!s(h) , u0 |= ϕ for all u0 ,bisight (x, y, z) → see(x, y) ∧ see(x, z) ∧ move(y, z).
The intuition of bisight (x, y, z) is then captured as follows:
4

Here xσ k y is the abbreviation of ∃y1 ∃y2 · · · ∃yk (xσy1 ∧y1 σy2 ∧ · · · ∧ yk−1 σyk ∧ (yk = y)).
188

such that u0 ∈ (bih )∗ (u).
M, h |= [BIsight ]ϕ iff for all z with z ∈ BIh , M, z |= ϕ.
We give a few illustrations of new issues that arise now.
Capturing the SCBI strategy For a start, we are now
able to characterize the SCBI strategy, in a similar vein as
the frame correspondence for the classical BI strategy in [25].
Proposition 4.5. The BI strategy is the unique relationbi satisfying this modal axiom for all propositions p:(hbi∗ i(end ∧ p)) → ([move][σ ∗ ](end ∧ h≤ip)).

Along the same lines, we can express the SCBI strategy
in P-S trees based on the idea that each scbi move coincides
with a local BI move within the current visible tree.
Proposition 4.6. The SCBI strategy is the relation scbi
satisfying the following axioms for all propositions p:(1)(2)

hscbiip ↔ [!sight]hbisight ip.

Proposition 4.12. We have the following three facts:(a) The formula[BI]ϕ → [BI][BI]ϕ is valid.
(b) For SCBI, the following formula does not hold:[BIsight ]ϕ → [BIsight ][BIsight ]ϕ.
(c) The formula [SCBI]ϕ → [SCBI][SCBI]ϕ is valid.
Here (a) says that from a BI outcome only a terminal
history can be reached; (b) shows that the local BI history
may not be a terminal history of the whole tree, and (c) says
the SCBI history for the whole tree is always terminal.
Another phenomenon regarding these operators is the
local optimality of SCBI at the cost of being more realistic
than BI. We have mentioned this point already in Section
2.2.4: now we can present a precise formal version.

[!sight](h(bisight )∗ i(endsight ∧ p) →Proposition 4.13. Let σ be any strategy profile,∗[movesight ]h(bisight ) i(endsight ∧ h≤ip)).

(a). For BI, the following is valid:

Best action and preference-consistency Turning toproperties of frames for the extended modal logic of best
action with sight, there are interesting differences whencomparing SCBI and classical BI. To see this, we employ
operators hbesti, hbi∗ i, hscbi∗ i, hbestsight i and h(bisight )∗ i.
Now we can make some interesting comparisons.
Proposition 4.7. For classical backward induction, theaxiom hbestihbi∗ iϕ ↔ hbi∗ iϕ holds.
However, the new frames do not have the corresponding
axiom for the SCBI strategy, since the actions it recommends
are not necessarily the actual best actions according to BI.
Even in visible trees, this is also not true.
Proposition 4.8. The following formulas are not valid:∗∗(a) hbestihscbi iϕ ↔ hscbi iϕ.

(b) [!sight](hbestih(bisight )∗ iϕ ↔ h(bisight )∗ iϕ).
Nevertheless, there is a certain coherence between thelocal BI strategy and local best actions returned by it.
Proposition 4.9. The following formula is valid:[!sight](hbestsight ih(bisight )∗ iϕ ↔ h(bisight )∗ iϕ).
As for the preference relation, SCBI has a property thatclassical BI lacks: local BI moves never conflict with thepreferences in submodels. In other words, within a visible
tree, the initial move determined by the local BI strategy ismore preferable for the agent than any other first move.
Proposition 4.10. For SCBI, it holds that[!sight](hbestsight iϕ → [movesight ]h≤iϕ).
For BI, although it returns a final optimal path, there is
no guarantee that its intermediate histories be preferable.
Proposition 4.11. For BI, the following does not hold:hbestiϕ → [move]h≤iϕ.
Path terminality and optimality Using a similar styleof modal analysis, we can make the following observations
concerning the obvious operators [BI], [SCBI] and [BI]sight .

189

hBIiϕ → [σ]h≤iϕ.
(b). The following does not hold:
hSCBIiϕ → [σ]h≤iϕ.
(c). For SCBI, it holds that[!sight](hBIsight iϕ → [σsight ]h≤sight iϕ).
Here (a) shows the global optimality of the BI path. (b)and (c) together say the SCBI path is not globally optimal,
but each move on this path leads to a locally optimal path.
Altogether, this section has shown the broad logicalfoundations of our framework, embedding our local languagein existing broader generic formalisms, but also enrichingand extending these frameworks with aspects of short sight.

5.

TOWARD MULTI-PLAYER GAMESWhile our models and results are about single-agent
sequential decision-making processes, we believe they areapplicable well beyond that. They can be naturally extended
to multi-player extensive game-scenarios with short sight.
For such a game model, we can build on [6], which makes
an assumption that the current player only knows his own
sight, and that he believes other players can see as muchas he can see and will play according to this belief. Thatis, this model precludes more complex forms of interactive
knowledge and reasoning. But using this same assumption,
our model in this paper can be extended to multi-playercases directly. The only thing we have to do is add agentlabeling to SSDM: even though players can change withtime, everything including sight, preference, and actions canbe modeled from the current player’s perspective.
We will not state any results for the extended multi-player
model since they are quite similar to what we have shown
already. The case where we drop the above assumption andallow a more free modeling of players’ mutual knowledge andbeliefs about sight and preference would be more interesting.
We will leave this for future work.

6.

DISCUSSION AND CONCLUSION

Though motivated by single-agent decision-making process, we have gone towards a much more general goal. In
the process, our analysis significantly adds to current connections between logic, computation, and game solutions.
In many recent game-theoretic papers centering on bounded
rationality, a model has been used of games with awareness,[7; 12; 5; 8].
This approach generalizes the classicalrepresentation of extensive games by modeling players who
may not be aware of all the paths. While [6] shows thatgames with short sight are a well-behaved subclass of gameswith awareness, there exists a fundamental difference infocus. Players in the latter approach may be unawareof some branches but they can always see some terminal
histories, while in the former, players’ sight may only
include intermediate histories, ruling out all terminal ones.
Moreover, we have shown how short-sight games allow for anatural co-existence of two views of a game, that of insiders
and that of outsiders. Having said this, it is clearly aninteresting issue to see if our approach in this paper canbe extended to cover awareness.
Another obvious interface for our logics are heuristic
evaluation approaches for intermediate nodes used by the AI
community for computational game-solving, [15; 21]. This,
too, is a connection that deserves further exploration.
There are many additional topics to pursue. For instance,
we already mentioned multi-player scenarios with non-trivial
interactive reasoning about other agents’ preferences, sights,
and strategies. This has also been identified as a key taskfor epistemic game theory.

Acknowledgments
I thank Fenrong Liu for our fruitful collaboration on earlier
versions of this paper. Paolo Turrini provided crucial
insights on short-sight games and their connections with
games and computation, which we are partly exploring together. Sonja Smets provided helpful comments overall. But
especially, I thank Johan van Benthem for our longstanding
contacts on the logic of short-sight games: Section 4 of thispaper owes a lot to his many suggestions and observations.
This work is supported by the China Scholarship Council
and NSFC grant No. 61472369.

7. REFERENCES[1] A. Baltag, S. Smets, and J. A. Zvesper. Keep ‘hoping’
for rationality: a solution to the backward induction
paradox. Synthese, 169(2):301–333, 2009.
[2] J. L. Bermúdez. Decision Theory and Rationality.
Oxford University Press, 2009.
[3] P. Blackburn, M. de Rijke, and Y. Venema. Modal
logic. Cambridge University Press, 2001.
[4] C. Degremont, S. Paul, and N. Asher. A Logic ofSights. Journal of Logic and Computation, 2014.
[5] Y. Feinberg. Games with unawareness. Stanford
Graduate School of Busirness Paper No. 2122, 2012.
[6] D. Grossi and P. Turrini. Short sight in extensive
games. In Proceedings of the 11th International
Conference on Autonomous Agents and Multiagent
Systems (AAMAS 2012), pages 805–812, 2012.
[7] J. Y. Halpern and L. C. Rêgo. Extensive games withpossibly unaware players. In AAMAS, pages 744–751,
2006.

[8] J. Y. Halpern and L. C. Rêgo. Extensive games withpossibly unaware players. Mathematical Social
Sciences, 70:42–58, 2014.
[9] S. O. Hansson. Decision theory -a brief introduction,
1994.
[10] P. Harrenstein, W. V. D. Hoek, J. jules Meyer, and
C. Witteveen. On modal logic interpretations ofgames. In Procs ECAI 2002, pages 28–32, 2002.
[11] P. Harrenstein, W. van der Hoek, J.-J. Meyer, and
C. Witteveen. A modal characterization of nash
equilibrium. Fundam. Inf., 57(2-4):281–321, 2003.
[12] A. Heifetz, M. Meier, and B. C. Schipper. Dynamic
unawareness and rationalizable behavior. Games andEconomic Behavior, 81:50–68, 2013.
[13] B. Houlding. Sequential Decision Making with
Adaptive Utility. PhD thesis, Department ofMathematical Sciences, Durham University, 2008.
[14] K. Høyland and S. W. Wallace. Generating scenario
trees for multistage decision problems. Management
Science, 47(2):pp. 295–307, 2001.
[15] Y. J. Lim and W. S. Lee. Properties of forward
pruning in game-tree search. In proceedings of the 21st
national conference on Artificial intelligence - Volume
2, AAAI’06, pages 1020–1025. AAAI Press, 2006.
[16] M. L. Littman. Algorithms for Sequential
Decision-making. PhD thesis, Brown University,
Providence, RI, USA, 1996.
[17] C. Liu, F. Liu, and K. Su. A logic for extensive gameswith short sight. In LORI, pages 332–336, 2013.
[18] D. W. North. A tutorial introduction to decision
theory. IEEE Transactions on Systems Science andCybernetics, 1968.
[19] M. J. Osborne. An Introduction to Game Theory,
volume 2. Oxford University Press, 2004.
[20] M. J. Osborne and A. Rubinstein. A Course in Game
Theory. MIT Press, 1994.
[21] F. Rossi, K. B. Venable, and T. Walsh. A ShortIntroduction to Preferences: Between Artificial
Intelligence and Social Choice. Synthesis Lectures on
Artificial Intelligence and Machine Learning. Morgan& Claypool Publishers, 2011.
[22] J. van Benthem. Exploring a theory of play. In Proc.
of TARK, pages 12–16, 2011.
[23] J. van Benthem. Logic in Games. MIT Press, 2014.
[24] J. van Benthem and A. Gheerbrant. Game solution,
epistemic dynamics and fixed-point logics. Fundam.
Inform., 100(1-4):19–41, 2010.
[25] J. van Benthem, S. V. Otterloo, and O. Roy.
Preference logic, conditionals, and solution concepts ingames. In Modality Matters, pages 61–76. Universityof Uppsala, 2006.
[26] J. van Benthem, E. Pacuit, and O. Roy. Toward atheory of play: A logical perspective on games andinteraction. Games, 2(1):52–86, 2011.
[27] W. van der Hoek and M. Pauly. Modal logic for games
and information. In J. van Benthem, P. Blackburn,and F. Wolter, editors, Handbook of Modal Logic.
Elsevier, 2006.
[28] H. van Ditmarsch, W. van der Hoek, and B. Kooi.
Dynamic Epistemic Logic, volume 337 of Syntheselibrary. Springer, 2007.

190

The optimality of coarse categories in decision-making
and information storage
Michael Mandler

Department of Economics
Royal Holloway College, University of London
Egham, Surrey, United Kingdom

1. INTRODUCTION
Suppose that agents, rather than forming a separate preference judgment for each pair of alternatives, make decisions
using criteria. A criterion orders a small number of categories, each of which consists of many alternatives. The
potential of a criterion to order alternatives within anothercriterion’s categories allow sets of criteria to generate large
numbers of choice distinctions. If an agent has objective
preferences that can be inferred from a large set of sufficiently discriminating criteria, the agent will be better off if
more of the criterion orderings are discovered: the agent willthen be able to determine the optimal allocation from morechoice sets. The uncovering of more criterion discriminations is costly, however, and we therefore consider efficient
points where the cost of making a given number of choice
distinctions is minimized.
This optimization problem seems to lead to a trade-off.
Given a number of choice distinctions, an agent could either
use a large set of coarse criteria (criteria with only a small
number of categories) or a small set of finer, more discriminating criteria. We show under mild conditions that large
sets of coarse criteria always lead to reductions in decisionmaking costs. Binary criteria with only two categories per
criterion therefore provide the only efficient arrangement.
Under mild restrictions on how criteria are aggregated into
decisions, binary criteria lead to rational choice functions,where decisions are determined by a complete and transitive
binary relation.
We apply our model to the problem of determining the
optimal number of digits in an information storage device.
We show that, even if the marginal cost of additional digits
declines rapidly to 0, binary digits (bits) offer the efficientsolution.
In this short paper, we pay particular attention to thesymmetry conditions that are entailed when sets of criteria
are efficient. A full working paper [2] is available on-line.

2. AN OUTLINE OF THE MODEL
A criterion Ci is an asymmetric binary relation on a
domain of alternatives X and a set of criteria is denoted
C = {C1 , ..., CN }. Two alternatives x and y are deemed
Ci -equivalent if x and y share the same set of Ci -superioralternatives and the same set of Ci -inferior alternatives (see[1]). A Ci -category is a maximal set of Ci -equivalent alternatives and e(Ci ) denotes the number of Ci -categories.
The discrimination vector of C is (e(C1 ), ..., e(CN )). A Ci
is coarser than Ci0 if e(Ci ) < e(Ci0 ).
Let c be a choice function on a domain of finite subsets of

191

X. Two alternatives x and y are in the same choice classof c if c treats them interchangeably: first, when x is chosen
and y is available then y is chosen too, and second, if x but
not y is available then x is chosen if and only if, when y isavailable and not x, y is chosen.
A choice function c uses C, denoted (C, c), if c does notmake distinctions that are not already present in the criteria:
for each set of alternatives A that contains only alternatives
that are in the same Ci -category, i = 1, ..., N , there is achoice class of c that contains A.
Let κ(Ci ) denote the cost of criterion Ci . We assumeκ(Ci ) is determined by the number of Ci -categories andtherefore also write κ(e) to denote the cost of a Ci withe categories. We assume that the cost of a set of criteria,
κ[C], equals the sum of the costs of the criteria in C. Letting n(c) be the number of choice classes in c, a pair (C, c)
is more efficient than the pair (C 0 , c0 ) if n(c) ≥ n(c0 ) andκ[C] ≤ κ[C 0 ], with at least one strict inequality, and (C, c) is
efficient if there does not exist a (C 0 , c0 ) that is more efficientthan (C, c).
The fundamental advantage of criteria is that each criterion can discriminate within the categories of other criteria.
Given constraints that specify that criterion Ci can have no
more than ei categories (and assuming that |X| is sufficiently
large), we can find a (C, c) such that (i) there is a partition ofQ
X with N
i=1 ei cells such that x and y are in distinct cells if
and only if they lie in different Ci -categories for at least one
i and (ii) each cell of this partition forms a choice class ofc. Subject to the ei constraints, this (C, c) maximizes n(c)and accordingly
hQwe define (C,ic) to maximally discriminateN
if n(c) = mini=1 e(Ci ), |X| .

3.

MAIN RESULTS(1) Since criteria with only one category make no discriminations and require no decisions, we assume they are
costless. To compare a (C, c) and (C 0 , c0 ) that have the sameP
number of costly categories, suppose that N
i=1 (e(Ci ) − 1) =
PN 0
0
(e(C
)
−
1).
Assumealso
that
either(i) the marginalii=1
cost of categories is increasing and the smaller of n(c) andn(c0 ) is less than the cardinality of X or (ii) the marginal
costs of categories is strictly increasing. We show that if C
has greater proportions of coarser criteria than does C 0 and
if (C, c) maximally discriminates, then (C, c) is more efficientthan (C 0 , c0 ).
(2) Fix a set of domains that, for each finite m, contains
a X with m elements and call a domain admissible if it
is drawn from this set. Then, every efficient (C, c) wherethe domain is admissible has a C that contains only binary
criteria if and only if κ(e) > κ(2) dlog2 ee for all integers
e > 2.
Thus the cost of e categories can rise as slowly as log2 e –in which case the marginal cost of categories descends to 0
– and still the only efficient arrangement is for all criteria to
be binary.
(3) We apply the result in (2) to information storage.
Suppose we wish to store some integer between 1 and nusing N k-ary digits and that the cost of storage equals
κ(k)N . We show that, for all positive integers n, binary
digits are the minimum-cost storage method if and only if
κ(k) > κ(2) dlog2 ke for all integers k > 2.
(4) We specify axioms for how to aggregate sets of criteria into choice functions that generalize weighted voting.
Suppose that the choice function c in the pair (C, c) satisfies
these axioms, that C contains only binary criteria, and thatc satisfies the following Condorcet rule: if there is a x in achoice set A that is chosen by c from all {x, y} with y ∈ Athen x is chosen from A too. Then c makes selections thatmaximize a complete and transitive binary relation. Given(2), we conclude that in a broad range of cases, efficientdecision-making is rational.

4. SYMMETRY AND MAXIMAL CATEGORIZATION
Maximal discrimination is necessary for decision-making
efficiency since otherwise n(c) could be increased without anincrease in costs. The key feature required for a (C, c) tomaximally discriminate is that the following property of C,
called maximal categorization, is satisfied: the discrimination partition P of X that places x and y in distinct cells if
and only if x andQy lie in different Ci -categories for at least
one i must have N
i=1 e(Ci ) cells.
We will now see that if X is a product of attributes andeach criterion orders a distinct attribute, then maximal categorization is satisfied and conversely if maximal categorization is satisfied then we can label alternatives so that X
becomes a product of attributes. By joining this conclusion
to result (2), that only binary criteria are efficient, we candescribe efficient decision-making concisely: to be efficient
agents must be able to describe the alternatives in X so thatthey form a product of attributes and each criterion must
divide a distinct attribute into exactly two categories.
The simplest way to achieve maximal categorization is for
X to be formed by a product of attributes and for each Ci
to divide X into categories based only on attribute i. The
domain X might be a set of cars, and the attributes might
be colors, top speeds, and prices. A ‘speed’ Ci would thenorder cars based on the ranges of top speeds that Ci deems
to be equivalent.
Formally, an attribute is a setQ
Xi and N attributes define
the domain of alternatives X = N
i=1 Xi . We will say that a
set of criteria C is based on a product of attributes if for eache(C )Ci there is a set Xi and a partition of {Xi1 , ..., Xi i } of X
iQjsuch that the categories of Ci are the sets Xi ×
X
,
k
k6=i

j = 1, ..., e(Ci ). So, if Ci is an ordering of cars by color theneach Xij would represent a color and x and y would be placed
into distinct Ci -categories if and only if the ith coordinatesof x and y indicate different colors: xi ∈ Xij and yi ∈ Xik
where j 6= k. TheQcells of the discrimination partition P
jN
j1
would then be the N
i=1 e(Ci ) sets X1 × · · · × XN where,

for each i, ji is an integer between 1 and e(Ci ). Maximal
categorization thus obtains.
This treatment assumes that X is a product space: foreach possible combination of attributes (each possible colorspeed-price combination), there is a corresponding elementof X. But for maximal categorization it is enough that there
is some alternative in X for each combination of attribute
ranges specified byQthe criteria, that is, it is sufficient for X
jN
j1
to be a subset of N
i=1 Xi such that each X1 × · · · × XN
intersects X.
A set of criteria C that is based on a product of attributes
enjoys a wide-ranging symmetry property. Fix some Ci in
C, and consider a set E−i formed by an arbitrary union of
the categories of the remaining criteria Cj , j 6= i. Given the
product structure of C, any such E−i must intersect each of
the Ci -categories. To continue the car example, the set ofcars E−i defined by a certain range of top speeds and prices
can be partitioned into all the possible color subsets, sayblue, red, and yellow. If we use Ci to order the cells of
the color partition of the cars in E−i , the ordering will havethe same ‘shape’ as – be order isomorphic to – the originalcolor ordering Ci of X. If, for example, Ci on X is a cycle– blue is better than red which is better than yellow which
is better than blue – then the Ci ordering of any set of cars
defined by a range of speeds and prices will also form a cycle.
We conclude that any two sets of cars Y and Z defined by
selections of non-color attributes will be order isomorphic toeach other when each is endowed with the color ordering Ci(or rather the restrictions of Ci to Y and Z).
This symmetry property may seem to be of limited value
since it appears to apply only to products of attributes. But
in fact the symmetry property characterizes any C that maximally categorizes. If for an arbitrary (possibly nonproduct) C, we apply Ci to some E−i and it defines fewer thane(Ci ) Ci -category subsets then the discrimination partition
Q
P would have to contain fewer than N
i=1 e(Ci ) cells. And0
the only way that E−i and E−i
can each define e(Ci ) Ci category subsets is for the Ci ordering of these subsets to beorder isomorphic.
Moreover, if an arbitrary (possibly nonproduct) C enjoys
the symmetry property we can relabel the elements of thedomain X so that C is then based on a product of attributes.
To do this, we associate each Ci with an attribute (e.g.,color) and identify each Ci -category with an arbitrary value
Xij for that attribute (e.g., blue): each cell of P is thus
identified with a vector of attribute values. So, although a
product of attributes looks special, it provides a model forany set of criteria that maximally categorizes.
The following definitions and theorem make these claimse(C )precise. We use Ei1 , ..., Ei i to denote the categories ofcriterion Ci .
Given the set of criteria
S {C1 , ..., CN }, E−i is a union of
C−i -categories if E−i = j Ekj for some collection of criterionEcategories {Ekj } such that k 6= i for each j. Let Ci −iEdenote the binary relation defined by E Ci −i E 0 if and only0
if there are Ci -categories Ei and Ei such that E = Ei ∩ E−i ,E 0 = Ei0 ∩E−i , and x Ci y for x ∈ E and y ∈ E 0 . We then say
C satisfies the order-isomorphism property if for any i and0
any two unions of C−i -categories, E−i and E−i, the binaryEE0

relations Ci −i and Ci −i are order-isomorphic.
The set of criteria C has a product representation if (i) for
each i, there is a nonempty set Yi and a partition {Yi1 , ...,

192

b1 , ..., CbN }Yi i } of Yi , (ii) there is a set of criteria Cb = {CQ
bi areY
wherethecategories
of
each
C
defined on Y = N
i
i=1 
Q
the sets Yij ×
k6=i Yk , and (iii) for each i, there is a orderb
preserving bijection f between the categories
 of Ci and Ci ,

j0
j
j0
j b
that is, E Ci E if and only if f E Ci f E .
e(C )

i

i

i

i

We then have the following result.

Theorem. For a set of criteria C, the following statements
are equivalent: (i) C maximally categorizes, (ii) C satisfies
the order-isomorphism property, (iii) C has a product representation.

5. REFERENCES[1] Mandler, M., Rational agents are the quickest. J. Econ.
Theory 155: 206-233, 2015.
[2] Mandler, M., Coarse, efficient decision-making.
Available at SSRN: http://ssrn.com/abstract=2494600
or http://dx.doi.org/10.2139/ssrn.2494600.

193

Undecidable Cases of Model Checking Probabilistic
Temporal-Epistemic Logic
Ron van der Meyden

School of Computer Science and Engineering
UNSW Australia

ABSTRACT
We investigate the decidability of model-checking logics oftime, knowledge and probability, with respect to two epistemic semantics: the clock and synchronous perfect recall semantics in partially observed discrete-time Markov chains.
Decidability results are known for certain restricted logics
with respect to these semantics, subject to a variety of restrictions that are either unexplained or involve a longstanding unsolved mathematical problem. We show that mild
generalizations of the known decidable cases suffice to render the model checking problem definitively undecidable. In
particular, for a synchronous perfect recall, a generalizationfrom temporal operators with finite reach to operators with
infinite reach renders model checking undecidable. The caseof the clock semantics is closely related to a monadic second
order logic of time and probability that is known to be decidable, except on a set of measure zero. We show that two
distinct extensions of this logic make model checking undecidable. One of these involves polynomial combinations ofprobability terms, the other involves monadic second order
quantification into the scope of probability operators. These
results explain some of the restrictions in previous work.

1. INTRODUCTION
Model checking is a verification methodology used in computer science, in which we ask whether a given model satisfies a given formula of some logic. First proposed in the
1980’s [CE81], model checking is now a rich area, with alarge body of associated theory and well developed implementations that automate the task of model checking. Significant use of model checking tools is made in industry, inparticular, in the verification of computer hardware designs.
Model checking developed originally in a setting where
the specifications are expressed in a propositional temporal logic, and the systems to be verified are finite state automata. This setting has the advantage of being decidable,
and a great deal of work has gone into the development ofalgorithms and heuristics for its efficient implementation.
More recently, the field has explored the extent to which theexpressiveness of both the model representations and of thespecification language can be extended while retaining decidability of model checking. Extensions in the systems dimensions considered include real-time systems [ACD90], systems
with a mixed continuous and discrete dynamic [MNP08],
richer automaton models such as push-down automata, machines with first-in-first out queues etc. In the dimensionof the specification language, extensions considered include
elements of second order logic and specific constructs to cap-

195

Manas K Patra

School of Computer Science and Engineering
UNSW Australia

ture the richer properties of the systems models describedabove (e.g. in the real time case the specification language
might contain inequalities over time values.)
Model checking for epistemic logic was first mooted in[HV91], and model checking for the combination of temporal and epistemic logic has been developed both theoretically[MS99, EGM07, HM10] and in practice [GM04, LQR09,
KNN+ 08, Eij04]. A variety of semantics for knowledge are
known to be associated with decidable model checking problems in finite state systems, in particular, the observationalsemantics (in which an agent reasons based on its present
observation) the clock semantics (in which an agent reasons based on its present observation and the present clock
value), and synchronous and asynchronous versions of perfect recall, all admit decidable model checking in combination with quite rich temporal expressiveness [MS99, EGM07,
HM10].
Orthogonally, a line of work on probabilistic model checking has considered model checking of assertions about probability and time [RKNP04]. Although one might at first
expect this line of work to be closely related to epistemic
model checking, in that probability theory provides a modelof uncertainty, in fact this area has been concerned not withhow subjective probabilities change over time, but with aprobabilistic extension of temporal logic. The focus tends
to be on the prior probability of some temporal property,
or on the probability that some temporal property holds inruns from a current known state.
Rather less attention has been given to model checking the
combination of subjective probability and temporal expressiveness. Of the semantics for knowledge mentioned above,the clock and synchronous perfect recall semantics are mostsuited as a basis for model checking subjective probability.
(The others suffer from asynchrony, which makes it moredifficult to associate a single natural probability space.) Implementations for these semantics presently exist only for alimited set of formulas, in which the full power of temporal
logic is not used. For example, results in [HLM11] for model
checking the logic of subjective probability (with clock orsynchronous perfect recall semantics) and time restricts the
temporal operators to have only finite reach into the future,and does not handle operators such as “at all times in the
future”.
A fundamental reason underlying this is that the problem of model checking probability with a rich temporal expressiveness seems to be inherently complex. Indeed, it
requires a solution to a basic mathematical problem, theSkolem Problem for linear recurrences, that has stood un-solved since first posed in the 1930’s [Sko34]. Consequently,
the strongest results on model checking probability and timethat encompass the expressiveness required for model checking knowledge and subjective probability state decidability
in a way that requires exclusion of an infinite set of difficult
instances for which decidability is unresolved. Specifically,[BRS06] shows that a (weak) monadic second order logic
PMLO, containing probability assertions of forms such as
Pr(φ(t1 , . . . , tn )) > c, in which the ti take values in the natural numbers, representing discrete time points, is decidable in finite state Markov chains, provided that the rational number c is not in a set Hφ depending on φ which cantaken to be of arbitrarily small non-zero measure. This work
leaves open the decidability of the model checking problem
for the language in its full generality, in particular, for thevalues of c in Hφ .
Our contribution in this paper is to consider a number ofgeneralizations of PMLO, motivated by model checking a
logic of time and subjective probability. In particular, ourgeneralizations arise very naturally when attempting to deal
with the way that an agent conditions probability on its observations. We show that these generalizations definitively
result in undecidable model checking problems. This clarifies the boundary between the decidable and undecidable
cases of model checking logics of probability and time.
We begin in section 2 by recalling the definition of probabilistic interpreted systems [Hal03], which provides a very
general semantic framework for logics of time, knowledge
and probability. We work with an instantiation of this general framework in which systems are generated from finite
state partially observed discrete-time Markov chains. We
define two logics that take semantics in this framework.
The first is an extension of the branching time temporal
logic CTL∗ to include operators for knowledge and probability, including operators for the subjective probability ofagents. The second is a more expressive monadic second
order logic that also adds a capability to quantify over moments of time and finite sets of moments of time. In thislogic, the agent knowledge and probability operators are indexed by a temporal variable. This logic generalizes thelogic of [BRS06]. Our logics allow polynomial comparisonsof probability terms, as well as comparisons of agent probability terms referring to multiple time points. We argue
from a number of motivating applications that this level ofexpressiveness is useful in potential applications. We showin Section 3 that the monadic second order logic is as leastas expressive as our probabilistic extension of CTL∗ . Indeed, some apparently mild extensions of PMLO suffice forthe encoding: the epistemic and subjective probability operators can be eliminated using a universal modality, polynomial combinations of probability expressions, and a more
liberal use of quantification than allowed in PMLO.
We then turn in Section 4 to an investigation of the model
checking problem. Specifically, we show that model checking
even very simple formulas about a single agent’s probability
is undecidable when the agent has perfect recall. A consequence of this result is that an extension of PMLO that
adds second order quantification into the scope of probability is undecidable.
This suggests a focus on weaker epistemic semantics instead, in particular, the clock semantics. From the pointof view of PMLO, to express agent’s subjective probabilities with respect to the clock semantics requires polynomial

combinations of simple global probability terms of the form
“ the probability that proposition p holds at time t”. We
formulate a simple class of formulas involving such polynomial combinations, and show that this also has undecidable
model checking.
These results show that even simple model checking questions about subjective probability are undecidable, and moreover help to explain some unexplained restrictions on PMLO
in [BRS06]: these restrictions are in fact necessary in orderto obtain a decidable logic. We conclude with a discussion of
future work in Section 5. Related work most closely related
to our results is discussed in the context of presenting andmotivating the results.

2.

PROBABILISTIC KNOWLEDGE

We describe in this section the semantic setting for themodel checking problem we consider. We model a set of
agents making partial observations of an environment thatevolves with time. We first present the semantics of themodal logic we consider, following [Hal03], using the general
notion of probabilistic interpreted system. Since these structures are not finite, in order to have a finite input for a model
checking problem, we derive a probabilistic interpreted system from a partially observed discrete-time Markov chain.
This is done in two ways, depending on the degree of recall of the agents. Taking the Markov chain to be finite,
we obtain finitely presented model checking problems whose
complexity we then study.

2.1

Probabilistic Interpreted Systems

Probabilistic interpreted systems are defined as follows.
Let Agt = {1, . . . , n} be a set of agents operating in an environment e. At each moment of time, each agent is assumed
to be in some local state, which records all the informationthat the agent can access at that time. The environment e
records “everything else that is relevant”. Let S be the setof environment states and let Li be the set of local statesof agent i ∈ Agt. A global state of a multi-agent system isan (n + 1)-tuple s = (se , s1 , . . . , sn ) such that se ∈ S andsi ∈ Li for all i ∈ Agt. We write G = S × L1 × . . . × Ln forthe set of global states.
Time is represented discretely using the natural numbers
N. A run is a function r : N → G, specifying a global stateat each moment of time. A pair (r, m) consisting of a run r
and time m ∈ N is called a point. If r(m) = (se , s1 , . . . , sn )then we define re (m) = se and ri (m) = si for i ∈ Agt. If r
is a run and m ∈ N a time, we write r[0..m] for r(0) . . . r(m)and re [0..m] for re (0) . . . re (m). A system is a set R of runs.
We call R × N the set of points of the system R.
Agent knowledge is captured using a relation of indistinguishability. Two points (r, m) and (r0 , m0 ) are said to beindistinguishable to agent i, if the agent is in the same local state at these points. Formally, we define ∼i to be theequivalence relation on R × N given by (r, m) ∼i (r0 , m0 ), if
ri (m) = ri0 (m0 ). Relative to a system R, we define the set
Ki (r, m) = {(r0 , m0 ) ∈ R × N | (r0 , m0 ) ∼i (r, m)}

to be the set of points that are, for agent i, indistinguishable
from the point (r, m). Intuitively, Ki (r, m) is the set of all
points that the agent considers possible when it is in theactual situation (r, m). A system is said to be synchronous
if for all agents i, we have that (r0 , m0 ) ∈ Ki (r, m) implies
that m = m0 . Intuitively, in a synchronous system, agents196

always know the time. Since it is more difficult to define
probabilistic knowledge in systems that are not synchronous,
we confine our attention to synchronous systems in what
follows.
A probability space is a triple Pr = (W, F, µ) such that
W is a (nonempty) set, called the carrier, F ⊆ P(W ) is a
σ-field of subsets of W , called the measurable sets in Pr,
closed under complementation and countable union, and µ :FS→ [0, 1] is aPprobability measure, such that µ(W ) = 1 andµ( n Vn ) =n µ(Vn ) for every countable sequence {Vn }of mutually disjoint measurable sets Vn ∈ F . As usual, we
define the conditional probability µ(U |V ) = µ(U ∩ V )/µ(V )when µ(V ) > 0.
Let Prop be a set of atomic propositions. A probabilistic
interpreted system over Prop is a tuple I = (R, Pr1 , . . . , Prn , π)such that R is a system, each Pri is a function mapping eachpoint (r, m) of R to a probability space Pri (r, m) in which
the carrier is a subset of R × N, and π : R × N → P(Prop)
is an interpretation of some set Prop of atomic propositions.
Intuitively, the probability space Pri (r, m) captures the waythat the agent i assigns probabilities at the point (r, m), andπ(r, m) is the set of atomic propositions that are true at the
point.
We will work with probabilistic interpreted systems derived from synchronous systems in which agents have a common prior on the set of runs. To define these, we use thefollowing notation. For a system R, a set of runs S ⊆ R
and a set of points U ⊆ R × N, define
S(U ) = {r ∈ S | ∃m : (r, m) ∈ U }to be the set of runs in S containing some point in the set
U . Conversely, for a set S of runs and a set U of points,
defineU (S) = {(r, m) ∈ U | r ∈ S}
to be the set of points in U that are on a run in S. Note that
if there exists a constant k ∈ N such that (r, m) ∈ U implies
m = k, then the relation r ↔ (r, k) defines a one-to-one
correspondence between S(U ) and U (S). In synchronous
systems, which satisfy this condition, this gives a way tomove between sets of points considered possible by an agent
and corresponding sets of runs.
Suppose that R is a synchronous system, let Pr = (R, F, µ)be a probability space on the system R, and let π be aninterpretation on R. Intuitively, the probability space Pr
represents a prior distribution over the runs. We assume
that for all points (r, m) ∈ R × N and agents i, we have that
R(Ki (r, m)) ∈ F is a measurable set and µ(R(Ki (r, m))) >0. (This assumption can be understood as saying that,
according to the prior, each possible local state ri (m) ofagent i at time m has non-zero probability of being the local state of agent i at time m.) Under this condition, we
define the probabilistic interpreted system I(R, Pr, π) =(R, Pr1 , . . . , Prn , π) such that Pri associates with each point(r, m) the probability space Pri (r, m) = (Ki (r, m), Fr,m,i , µr,m,i )defined by
Fr,m,i = {Ki (r, m)(S) | S ∈ F}
and such that
µr,m,i (U ) = µ( R(U ) | R(Ki (r, m)) )for all U ∈ Fr,m,i . Intuitively, because the set of runs
R(Ki (r, m)) is measurable, we can obtain a probability space197

with carrier R(Ki (r, m)) by conditioning in Pr. Because of
the synchrony assumption there is, for each point (r, m), a
one-to-one correspondence between points in Ki (r, m) andruns in R(Ki (r, m)). The construction uses this correspondence to induce a probability space on Ki (r, m) from theprobability space on R(Ki (r, m)). We remark that under
the additional assumption of perfect recall, it is also possible to understand each space Pri (r, m + 1) as obtained byconditioning on the space Pri (r, m). See [Hal03] for a detailed explanation of this point.

2.2

Probabilistic Temporal Epistemic Logic

To specify properties of probabilistic interpreted systems,
a variety of logics can be formulated, drawing from the spectrum of temporal logics. Our main interest is in a reasoning
about subjective probability and time, so we first consider
a natural way to combine existing temporal and probabilistic logics. For purposes of comparison, it is also helpful toconsider a rather richer monadic second order logic of probability and time, that is closely related to a logic for whichsome decidability results are known.
We may combine temporal and probabilistic logics to define a logic CTL∗ KP that extends the temporal logic CTL∗
by adding operators for knowledge and probability. Its syntax is given by the grammar
φ ::= p | ¬φ | φ ∧ φ | Aφ | Xφ | φU φ | Ki φ | f (P, . . . , P ) ./ c
P ::= Pri (φ) | Priori (φ)
where p ∈ Prop, c is a rational constant, ./ is a relation symbol in the set {≤, <, =, >, ≥}, and f (x1 , . . . , xk ) is multivariate polynomial in k variables x1 , . . . xk with rational coefficients. Instances of P are called basic probability expression.
The instances generated from f (P, . . . , P ) are called probability expressions, and are expressions of the form f (P1 , . . . , Pk ),obtained by substituting a basic probability expression Pi foreach variable xi in f (x1 , . . . , xk ). For example,
4Pr1 (p)5 · Pr2 (q)3 +7
Pr1 (p)15

is an instance of f (P, . . . , P ) obtained from f (x, y) = 4x5 y 3 +7x by substituting Pr1 (p) for x and Pr2 (q) for y.
15
Intuitively, formula Ki φ expresses that agent i knows φ.
The formula Aφ says that φ holds for all possible systemevolutions from the current situation. The formula Xφ expresses that φ holds at the next moment of time. The formula φ1 U φ2 says that φ2 eventually holds, and φ1 holds
until that time. The expression Pri (φ) represents agent i’scurrent probability of φ, Priori (φ) represents agent i’s priorprobability of φ, i.e., the agent’s probability of φ at time 0.
The formula f (P1 , . . . , Pk ) ./ c expresses that this polynomial combination of current and prior probabilities stands
in the relation ./ to c. We use standard abbreviations from
temporal logic, in particular, we write F φ for trueU φ.
A restricted fragment of the language that may be of interest is the branching time fragment in which the temporal
operators are restricted to those of the temporal logic CTL.
That is, X and U are permitted to occur only in combination with the operator A, in one of the forms AXφ, EXφ,Aφ1 U φ2 , Eφ1 U φ2 , where we write Eφ as an abbreviation for¬A¬φ. We call this fragment of the language CTLPK. The
motivation for considering this fragment is that the complexity of model checking is in polynomial time for the temporallogic CTL, whereas it is polynomial-space complete for thericher temporal logic CTL∗ [CES86]. The logic CTLPK
is therefore, prima facie, a candidate for lower complexity
once knowledge and probability operators are added to thelogic.
The semantics of the language CTL∗ KP in a probabilistic
interpreted system I = I(R, Pr, π) is given by interpreting
formulas φ at points (r, m) of I, using a satisfaction relation
I, (r, m) |= φ. The definition is mutually recursive witha function [·]I,(r,m) that assigns a value [P ]I,(r,m) to eachprobability expression P at each point (r, m). This requires
computing the measure of certain sets. For the moment, we
assume that all sets arising in the definition are measurable.
We show later that this assumption holds in the cases ofinterest in this paper.
We first interpret the probability expressions at points(r, m) of the system I, by[Pri φ]I,(r,m) = µr,m,i ({(r0 , m0 ) ∈ Ki (r, m) | I, (r0 , m0 ) |= φ})[Priori φ]I,(r,m) = µr,0,i ({(r0 , 0) ∈ Ki (r, 0) | I, (r0 , 0) |= φ})[f (P1 , . . . , Pk )]I,(r,m) = f ([P1 ]I,(r,m) , . . . , [Pk ]I,(r,m) )The satisfaction relation is then defined recursively, as follows:
1. I, (r, m) |= p if p ∈ π(r, m)
2. I, (r, m) |= ¬φ iff not I, (r, m) |= φ
3. I, (r, m) |= φ1 ∧φ2 iff I, (r, m) |= φ1 and I, (r, m) |= φ2

4. I, (r, m) |= Aφ if I, (r0 , m) |= φ for all runs r0 withr0 [0 . . . m] = r[0 . . . m],
5. I, (r, m) |= Xφ if I, (r, m + 1) |= φ
6. I, (r, m) |= φ1 U φ2 holds if there exists k ≥ m suchthat I, (r, k) |= φ2 , and I, (r, l) |= φ1 for all l withm ≤ l < k.
0

0

0

0

7. I, (r, m) |= Ki φ if I, (r , m ) |= φ for all (r , m ) ∈Ki (r, m).
8. I, (r, m) |= f (P1 , ..., Pk ) ./ c if [f (P1 , ..., Pk )]I,(r,m) ./c.

2.3

Probabilistic Monadic Second Order Logic

Temporal modal logics refer to time in a somewhat implicit way. An alternative approach is to work in a settingwith more explicit references to time, by using variables denoting time points. Kamp’s theorem [Kam68] establishes
an equivalence in the first order case, but by adding second
order variables and quantification, one can obtain richer logics, that frequently remain decidable in the monadic case.
In this section, we develop a logic in this style for time andsubjective probability.
We define the logic WMLOKP as follows. We use two
types of variables: time variables t and set variables X.
Time variables take values in N and set variables take finite subsets of N as values. Probability terms P have theform Pr(φ) or the form Pri,t (φ) where i ∈ Agt is an agent,t is a time variable, φ is a formula. Formulas φ are definedby the following grammar:

where t, t1 , t2 are time variables, p is an atomic proposition,
X is a set variable, i is an agent, c ∈ Q is a rational constant, f is a rational polynomial (see the discussion above
for CTL∗ KP), and ./ is a relation symbol from the set{=, <, ≤, >, ≥}.
Intuitively, in this logic formulas are interpreted relative
to a run. Instead of indexing by a single moment of time, as
in the logic above, we relativize the satisfaction relation toan assignment of values to the temporal and set variables.
Atomic formula p(t) says that proposition p holds at timet. Similarly, a (finite) set X of times can be interpreted asa proposition, and we can understand X(t) as stating thatthe value of t is in X. (We remark that there is a fundamental difference between the types of propositions denotedby atomic propositions p and set variables X: whereas theatomic propositions may depend on structural aspects of therun, such as the global state at time t, the set variables may
refer only to the time.) The atomic formula t1 < t2 has theobvious interpretation that time t1 is less than time t2 . The
constructs ∀t(φ) and ∀X(φ) correspond to universal quantification over times and finite sets of times respectively. Theysay that φ holds on the current run for all values of the variable. (Taking finite sets amounts to the weak interpretationof second order quantification. One could also consider astrong semantics allowing infinite sets of times. We have
opted here for the weak interpretation to more easily relate
our results to the existing literature.)
The probability term Pr(φ) refers to the probability of φin the probability space on runs. The meaning of probability
term Pri,t (φ) is agent i’s probability at time t that the run
satisfies φ. Similarly, Ki,t φ says that agent i knows at timet that the run satisfies φ. Note that, whereas in CTL∗ KP,
the formula Ki φ always expresses that agent i knows that φholds at the “current time”, in WMLOKP, formulas suchas∃u(u < t ∧ Ki,t (p(u)))

talk about the agent’s knowledge, at some time t, about
what was true at some earlier time u. A similar point applies
to probability expressions.
When dealing with formulas with free time and set variables, we need the extra notion of an assignment for the timeand set variables. This is a function τ such that for each freetime variable t we have τ (t) ∈ N, and for each free set variable X we have that τ (X) is a finite subset of N. Given suchan assignment, we give the semantics of probability terms
and formulas by a mutual recursion. We give the semantics
of formulas φ by means of a relation I, τ, r |= φ defined asfollows:
1. I, τ, r |= p(t) if p ∈ π(r, τ (t)), when p is an atomic
proposition
2. I, τ, r |= X(t) iff τ (t) ∈ τ (X), if X is a set variable
3. I, τ, r |= ¬φ iff not I, τ, r |= φ,
4. I, τ, r |= φ1 ∧ φ2 iff I, τ, r |= φ1 and I, τ, r |= φ2

5. I, τ, r |= Ki,t (φ) if I, τ, r0 |= φ for all (r0 , m0 ) ∈ Ki (r, τ (t)).
6. I, τ, r |= f (P1 , ..., Pk ) ./ c if [f (P1 , ..., Pk )]I,τ,r ./ c,
7. I, τ, r |= ∀t(φ) if I, τ [t 7→ n], r |= φ for all n ∈ N,

8. I, τ, r |= ∀X(φ) if I, τ [X 7→ U ], r |= φ for all finite
U ⊆ N.

φ ::= p(t) | X(t) | t1 < t2 | f (P, . . . , P ) ./ c | ¬φ | φ ∧ φ |Ki,t (φ) | ∀t(φ) | ∀X(φ)
198

In item (6), the definition is mutually recursive with thesemantics of probability terms, which are interpreted as real
numbers, relative to a temporal assignment. We define[Pr(φ)]I,τ,r = µ({r0 | I, τ, r0 |= φ})and[Pri,t (φ)]I,τ,r =µ({r0 | (r, τ (t)) ∼i (r0 , τ (t)), I, τ, r0 |= φ})µ({r0 | (r, τ (t)) ∼i (r0 , τ (t))})[f (P1 , . . . , Pk )]I,τ,r = f ([P1 ]I,τ,r , . . . , [Pk ]I,τ,r )As above, we assume measurability of the sets required, andlater justify that this holds in the particular setting of interest in this paper.
A particular class of formulas of WMLOKP will be ofinterest below. Define a mixed-time polynomial atomic probability formula to be a formula of the form
∀t1 . . . tn (f (Pr(φ1 ), . . . , Pr(φm )) = 0)
where f (x1 , . . . , xm ) is a rational polynomial and each φiis an atomic formula of the form p(tj ) for some proposition
p and j ∈ {1 . . . n}. We motivate the usefulness of such
temporal mixing of probability expressions in Section 2.5.
The logic WMLOKP generalizes several logics from theliterature. If we restrict the language by excluding the probability comparison atoms f (P1 , . . . , Pk ) ./ c and knowledge
formulas Ki,t (φ), we have the Weak Monadic Logic of Order, which is equivalent to WS1S [Buc60]. We obtain the
Probabilistic Monadic Logic of Order considered in [BRS06],
which we denote here by PMLO, if we• exclude the knowledge operators Ki,t ,• exclude agent’s probability terms Pri,t (φ), and• limit the global probability comparisons to be of theform Pr(φ(t1 , . . . , tk )) ./ c, containing just a single
probability term Pr(φ(t1 , . . . , tk )), with the further constraint that the only free variables of φ should be temporal variables t1 , . . . tk .
In particular, second-order quantification into probability
expressions, e.g., ∀X[Pr(X(t)) > c] is not permitted in PMLO,
but second order quantification that does not cross a probability operator, such as Pr(∀X[X(t))]) > c, is allowed. We
note that PMLO does allow first order quantifications into
the scope of probability, such as ∀t[Pr(p(t)) > c].
In the sequel, we refer to quantification into the scope of aknowledge formula or probability expression as quantifyingin.

2.4

Partially Observed Markov Chains

Although they provide a coherent semantic framework,
probabilistic interpreted systems are infinite structures, andtherefore not suitable as input for a model checking algorithm. We therefore work with a type of finite model called
an interpreted partially observed discrete-time Markov chain,or PO-DTMC for short. A finite PO-DTMC for n agentsis a tuple M = (S, P I, P T, O1 , ..., On , π), where S is a finite set of states, P I : S → [0..1] is a function such thatP
T : S × S → [0, 1] is a funcs∈S P I(s) = 1,
Pcomponent P
0tion such thats0 ∈S P T (s, s ) = 1 for all s ∈ S, and foreach agent i ∈ Agt, we have a function Oi : S → O for someset O. Finally, π : S → P(Prop) is an interpretation of theatomic propositions Prop at the states.

199

Intuitively, P I(s) is the probability that an execution of
the system starts at state s, and P T (s, t) is the probability
that the state of the system at the next moment of time
will be t, given that it is currently s. The value Oi (s) is theobservation that agent i makes when the system is in states. (Below, in the context of interpreted systems, we treat
the set of states S as the states of the environment rather
than as the set of global states. Agents’ local states will bederived from the observations.)
Note that the first three components (S, P I, P T ) of a PODTMC form a standard discrete-time Markov chain. This
gives rise to a probability space on runs in the usual way.
A path in M is a finite or infinite sequence ρ = s0 s1 . . .
such that P I(s0 ) 6= 0 and P T (sk , sk+1 ) > 0 for all k with0 ≤ k < |ρ| − 1. We write P∞ (M ) for the set of all infinite
paths of M . Any finite path ρ = s0 s1 . . . sm defines a set
P∞ (M ) ↑ ρ = {ω ∈ P∞ (M ) | ω[0 . . . m] = ρ}(2)That is, P∞ (M ) ↑ ρ consists of all infinite paths which have
ρ as a prefix.
We now define a probability space Pr(M ) = (P∞ (M ), F, µ)over the set P∞ (M ) of all infinite paths of M . The σalgebra F is defined to be the smallest σ-algebra over P∞ (M )that contains as basic sets all the sets P∞ (M ) ↑ ρ for ρ =s0 s1 . . . sm a finite path of M . For these basic sets, the function µ is defined byµ(P∞ (M ) ↑ ρ) = P I(s0 ) · P T (s0 , s1 ) · . . . · P T (sm−1 , sm ) .
The fact that µ can be extended to a measure on F is a
non-trivial result of Kolmogorov for more general stochastic
processes [KSK76].
We may construct several different probabilistic interpreted
systems from each PO-DTMC, depending on what agents
remember of their observations. We consider two, one thatassumes that agents have perfect recall of all their observations, denoted spr, and the other, denoted clk, whichassumes that agents are aware of the current time and theircurrent observation. Recall that runs in an interpreted system map time to global states, consisting of a state of theenvironment and a local state for each agent. We interpretthe states of the PO-DTMC M as states of the environment.
To obtain a run, we also need to specify a local state for eachagent at each moment of time. We use the the observations
to construct the local states.
In the case of the synchronous perfect recall semantics,
given a path ρ ∈ P∞ (M ), we obtain a run ρspr by defining
the components at each time m as follows. The environment
state at time m is ρspre (m) = ρ(m), and the local state ofagent i at time m is ρspr
i (m) = Oi (ρ(0)) . . . Oi (ρ(m)). Intuitively, this local state assignment represents that the agent
remembers all its past observations. We write Rspr (M ) for
the set of runs of the form ρspr for ρ ∈ P∞ (M ). Note that
this system is synchronous: if r = ρspr and r0 = ω spr thenfor each agent i and time m ∈ N, if ri (m) = ri0 (m0 ), thenOi (ρ(0)) . . . Oi (ρ(m)) = Oi (ω(0)) . . . Oi (ω(m0 )), which implies m = m0 .
For the clock semantics, we construct a run a ρclk in which
again the environment state at time m is ρclk
e (m) = ρ(m),and for agent i we define the local state at time m by
ρclk (m) = (m, Oi (ρ(m)). Intuitively, this says that theagent is aware of the clock value and its current observation. We write Rclk (M ) for the set of runs of the form
ρclk for ρ ∈ P∞ (M ) an infinite path of M . This system

is also synchronous: if r = ρclk and r0 = ω clk then foreach agent i and time m ∈ N, if ri (m) = ri0 (m0 ), then(m, Oi (ρ(m))) = (m0 , Oi (ω(m0 ))), hence m = m0 . In bothcases of x ∈ {spr, clk}, if T is a subset of P∞ (M ), we write
T x for {ρx | ρ ∈ T }.
In both cases of x ∈ {spr, clk}, we have a one-to-one
correspondence between the infinite paths P∞ (M ) and theruns Rx (M ). We therefore can induce probability spaces
Prx (M ) on Rx (M ) from the probability space Pr(M ) on
P∞ (M ). As described above, the probability space Prx (M )on runs moreover induces a probability space Prxi (r, m) on
the set of points considered possible by each agent i at eachpoint (r, m). The PO-DTMC M gives us an interpretation π
on its states, and we may derive from this an interpretation
π x on the points (r, m) of Rspr (M ) and Rclk (M ) by defining
π x (r, m) = π(re (m)). Using the general construction defined
above, we then obtain the probabilistic interpreted systems
I x (M ) = I(Rx (M ), Prx (M ), π x ) for x ∈ {spr, clk}.
It is necessary to establish the measurability of the setscorresponding to formulas for the semantic definitions of thelogics above to be complete. This is established in the following result.
Lemma 1. Let M be a finite PO-DTMC and x ∈ {spr, clk}.
For every set S ⊆ R(M ) of runs of M such that the semantic
definitions above of CTL∗ KP and WMLOKP in I x (M )refer to µ(S), the set S is measurable in Pr(M ).

2.5

Discussion

We have defined our logics to be quite expressive in thetype of atomic probability assertions we have allowed, which
involve polynomials of probability expressions. In WMLOKP,
these expressions may explicitly refer to different time points.
Some existing logics of probability in the literature use amore restricted expressiveness, e.g., [FH94] consider a logic
that has only linear combinations of probability expressions,and many logics [BRS06, RKNP04] allow only inequalities
involving a single probability term. Here give some motivation to show that the richness we have allowed is naturaland useful for applications.
Polynomials: There are several motivations for allowing
polynomial combinations of probability expressions. One, as
noted in [FHM90], is that polynomials arise naturally fromconditional probability. If we would like to include linear
combinations of conditional probability expressions in thelanguage, we find that this motivates a generalization topolynomial combinations of probability expressions. Consider the formula Pr(φ1 |ψ1 ) + Pr(φ2 |ψ2 ) ≤ c. Expandingout the definition of conditional probability, we havePr(φ2 ∧ ψ2 )Pr(φ1 ∧ ψ1 )+≤c.
Pr(ψ1 )
Pr(ψ2 )We see here that there is a risk of division by zero that needs
to be managed in order for the semantics of this formula to
be fully defined. One way to do so is to multiply out thedenominators, resulting in the form
Pr(φ1 ∧ψ1 )·Pr(ψ2 )+Pr(φ2 ∧ψ2 )·Pr(ψ1 ) ≤ c·Pr(ψ1 )·Pr(ψ2 )which is meaningful in all cases. (Should this not have thedesired semantics in case one of the Pr(ψi ) is zero, an additional formula can be added that handles this special case as
desired.) However, although we started with a linear probability expression, we now have multiplicative terms. Thissuggests that the appropriate way to add the expressiveness of conditional probability to the language is to admit
atomic formulas that compare polynomial combinations ofprobability expressions.
More generally, although it is less of relevance for purposes of model checking, and more of use for axiomatizationof the logic, allowing polynomials also naturally enables familiar reasoning patterns to be captured inside the logic. In
particular, validities such as Pr(φ1 ∨ φ2 ) = Pr(φ1 ) + Pr(φ2 )when φ1 and φ2 are mutually exclusive and Pr(φ1 ∧ φ2 ) =Pr(φ1 ) · Pr(φ2 ) when φ1 and φ2 are independent show thatboth addition and multiplication of probability terms arises
naturally.
Mixed-time: A second way in which our logics are rich isin allowing probability atoms that refer to different momentsof time. In CTL∗ KP this already the case because combinations such as PriorA (φ) = PrA (φ) are allowed, which refer toboth the current time and to time 0. The logic WMLOKP
takes such temporal mixing further by allowing reference totime points explicitly named using time variables.
Such temporal mixing is natural, since there are potential
applications that require this expressiveness. For example,
in computer security, one often wants to say that the adversary A does not learn anything about a secret from watchingan exchange between two parties. However, it is often thecase that the adversary knows some prior distribution overthe secrets. (For example, the secret may be a password,
and choice of passwords by users are very non-uniform, withsome passwords like ‘123456’ having a very high probability.)This means that the simple assertion that the adversary does
not know the secret, or that the adversary has a uniform distribution over the secret, does not capture the appropriate
notion of security. Instead, as recognised already by Shannon in his work on secrecy [Sha49], we need to assert thatthe adversary’s distribution over the secret has not changed
as a result of its observations. This requires talking about
the adversary’s probability at two time points. For example, [HLM11] capture an anonymity property by means offormulas using terms PriorA (φ) = PrA (φ).
Mixed-time polynomials: Additionally, the logic ofprobability applied to formulas referring to different times
leads naturally to polynomial combinations of probability
terms, each referring to a different moment of time. For example, although PMLO allows only formulas of the form
Pr(φ(t1 , . . . , tn )) ./ c, where the ti are time variables, thedecision algorithm of [BRS06] uses the fact that, when t1 <t2 < . . . < tn , the formula φ(t1 , . . . , tn ) is equivalent to a
formula of the form φ1 (t1 ) ∧ φ2 (t2 − t1 ) ∧ . . . φn (tn − tn−1 ) ∧φn+1 (tn ), where the φi (t) are independent past-time formulas for i = 1 . . . n and φn+1 (t) is a future time formula. (Thisstatement is closely related to Kamp’s theorem [Kam68].)
This enables Pr(φ(t1 , . . . , tn )) to be expressed as a sum of
products of terms of the form Pr(φi (u)) where φi (u) has
just a single free time variable u. Thus, although mixedtime probability formulas are not directly expressible in the
logic of [BRS06], specific ones are implicitly expressible, andthe extension is a mild one. It is worth remarking, however, that the coefficients of the polynomial expansion ofPr(φ(t1 , . . . , tn )) are all positive, so we do not quite havearbitrary polynomials here. We return to this point below.

3.

RELATING THE LOGICS

The logic WMLOKP is very expressive, so it is not sur-200

prising that it can capture all of CTL∗ KP. The following
result makes this precise.
For the results below, it is convenient to add to the systema special agent ⊥ that is blind, and an agent > that thathas complete information about the state. In the context of
PO-DTMC’s these agents are obtained by taking the observation functions to satisfy O⊥ (s) = O⊥ (t) and O> (s) = s
for all states s, t. We write 2φ for K⊥,t φ where t is any time
variable. This gives a universal modality: 2φ says that φholds on all runs.

is easily seen to be equivalent to Ki,t (φ) in I clk (M ). Similarly, Pri,t (φ) ./ c can be expressed as^(obs i,j (t) ⇒ Pr(obs i,j (t) ∧ φ) ./ c · Pr(obs i,j (t))) .

Proposition 2. Let M be a PO-DTMC with agent >and let x ∈ {spr, clk}. For every formula φ of CTL∗ KP,
there exists a formula φ∗ (t) of WMLOKP with t the only
free variable, such that I x (M ), (r, n) |= φ iff I x (M ), [t 7→n], r |= φ∗ (t) for all runs r.

Proposition 4. With respect to I spr (M ) for a finite PODTMC M , the probability formulas Pri,t (φ) ./ c can be reduced to linear comparisons using only terms Pr(ψ), provided second-order quantifying-in is permitted. Knowledge
terms Ki,t can be reduced to the universal modality 2, provided second-order quantifying-in is permitted for this modality.

p∗ (t) = p(t), (¬φ)∗ (t) = ¬φ∗ (t), (φ1 ∧ φ2 )∗ (t) = φ∗1 (t) ∧ φ∗2 (t),(Xφ)∗ (t) = ∃u(u = t + 1 ∧ φ∗ (u)), (Ki φ)∗ (t) = Ki,t (φ∗ (t)),

Proof. Define κi (X1 , . . . , Xki , t) to be the formula^∀t0 ≤ t(Xi (t0 ) ⇔ obsi,j (t0 ))Proof. The translation is defined by the following recursion:(φ1 U φ2 )∗ (t) = ∃u ≥ t(φ∗2 (u) ∧ ∀v(t ≤ v < u ⇒ φ∗1 (v)))(Pri (φ))∗ (t) = Pri,t (φ∗ (t)), (Priori (φ))∗ (t) = Pri,0 (φ∗ (0))(f (P1 , . . . , Pk ) ./ c)∗ (t) = f (P1∗ (t), . . . , Pk∗ (t)) ./ cNote that u = t + 1 is definable as (u > t ∧ ∀v > t(u ≤ v)),and u = 0 is definable as ¬∃t(u = t + 1). We can use(Aφ)∗ (t) = K>,t (φ∗ (t)) to translate Aφ in the perfect recallcase. In case of the clock semantics, this translation loses
the information about the initial state, which is required forcorrectness of the translation of Priori (φ). In this case, we
introduce without loss of generality new propositions ps foreach state s, such that ps ∈ πe (t) iff s = t, and take^(Aφ)∗ (t) =(ps (0) ⇒ K>,t (ps (0) ⇒ φ∗ (t)))
s∈SWith respect to the specific systems we derive from PODTMC’s with respect to the clock and perfect recall semantics, we are able to make some further statements that simplify the logic WMLOKP by eliminating some of the operators. These results are useful for the undecidability resultsthat follow.
For the following results, we note that, without loss ofgenerality, we may assume that a finite PO-DTMC comesequipped with atomic propositions that encode the observations made by the agents. Specifically, when agent i has
possible observations oi,1 , . . . , oi,ki , we assume that there are
atomic propositions obs i,j for i ∈ Agt and j = 1 . . . ki suchthat for all states s, obs i,j ∈ π(s) iff Oi (s) = oi,j . Thus,
obs i,j (t) holds in a run just when agent i makes observationoi,j at time t.

j=i...ki

A similar transformation applies for more general agent probability comparisons, but we note that linear comparisons
may transform to polynomial comparisons: similarly to thediscussion of conditional probability in Section 2.5.

j=1...ki

Intuitively, this says that, up to time t, the second ordervariables X1 , . . . , Xk encode the pattern of occurrence ofobservations of agent i up to time t. The formula∀X1 , . . . Xki (κi (X1 , . . . , Xki , t) ⇒ 2(κi (X1 , . . . , Xki , t) ⇒ φ)is easily seen to be equivalent to Ki,t (φ) in I clk (M ). Similarly, Pri,t (φ) ./ c can be expressed as∀X1 , . . . Xki (κi (X1 , . . . , Xki , t) ⇒Pr(κi (X1 , . . . , Xki , t) ∧ φ) ./ c · Pr(κi (X1 , . . . , Xki , t))) .

One might wonder whether the knowledge operators canbe eliminated entirely using probability, treating Ki φ as
Pri (φ) = 1. This is indeed the case for formulas φ in CTLPK.
The essential reason is that because formulas of CTLPK
depend at a point (r, m) only on the run prefix r[0 . . . m], so
the possibility that ¬φ holds on a non-empty set of measurezero does not occur.
Proposition 5. For all CTLPK formulas φ and PODTMC’s M and x ∈ {clk, spr} we have I x (M ) |= Ki φ ⇔Pri (φ) = 1.
However, this is not the case for formulas Ki φ where φis an LTL formula. Consider the following Markov Chain.
Here we have, at the initial state s, that ¬Ki (F ¬q), bep

1

clk

Proposition 3. With respect to I (M ) for a finite PODTMC M , the operators Ki,t and Pri,t can be eliminatedusing the universal operator 2 and polynomial comparisonsof universal probability terms Pr(ψ), respectively. For simple probability formulas Pri,t (φ) ./ c, only linear probability
comparisons are required.
Proof. The formula^(obs i,j (t) ⇒ 2(obs i,j (t) ⇒ φ)j=i...ki

201

s
q

1-p

t
¬q

cause the agent considers it possible that always q (this holds
for all choices of observation functions). However, we havePri (F ¬q) = 1, since the only run where ¬q does not eventually hold is the run that always remains at s. This run has
probability zero.

4.

UNDECIDABILITY RESULTS

We can now state the main results of the paper concerning
the problem of model checking formulas of (fragments of) the
logics CTL∗ KP and WMLOKP in a PO-DTMC M , withrespect to an epistemic semantics x ∈ {spr, clk}. Using
the results of Section 3, we also obtain conclusions about
extensions of PMLO that do not refer to agent probability
and knowledge.
For a formula φ of CTL∗ KP, we write M |=x φ, if
I x (M ), (r, 0) |= φ for all runs r ∈ Rx (M ). In the case of
WMLOKP, we consider sentences, i.e., formulas without
free variables, and write M |=x φ, if I x (M ), τ, r |= φ for all
runs r ∈ Rx (M ) and the empty assignment τ . The model
checking problem is to determine, given a PO-DTMC M , aformula φ, and semantics x ∈ {clk, spr}, whether M |=x φ.

4.1

We will deal with sequences with rational entries. By clearing denominators the rational version of the above problems can be shown to be polynomially equivalent to similar
problems stated using sequences with integer entries. There
has been a significant amount of work on these problems[ESPW03], but they have stood unresolved since the 1930’s.
To date, only low order versions of these problems have beenshown to be decidable [HHHK05, OW14, TMS84].
The above problems have an equivalent matrix formulation. A proof of the following can be found in [HHHK05].
Lemma 7. For a sequence u0 , u1 , . . . , the following are
equivalent.
1. {un } is a rational LRS.

BackgroundFor comparison with results below, it is worth stating a
result from [BRS06] concerning decidability of the fragment
PMLO of WMLOKP that omits knowledge operators Ki,tand agent probability terms Pri,t (φ), restricts probability
comparisons to the form Pr(φ) ./ c, and prohibits second
order quantification to cross into probability terms. Since
the structure of agent’s local states is irrelevant in this case,we write simply I(M ) for the probabilistic interpreted system corresponding to a PO-DTMC M . To state the result,
we define the parameterized variant of a formula φ of PMLO
to be the formula φx1 ,...,xk , in which each subformula of theform Pr(ψ) ./ c is replaced by a formula Pr(ψ) ./ xi , with xi
a fresh variable. We call the resulting formulas the parameterized formulas of PMLO. For some α ∈ Qk , we can thenrecover the original formula φ as the instance φα obtained
from the parameterized variant φx1 ,...,xk of φ by substituting
αi for xi for each i = 1 . . . k.
Theorem 6 ([BRS06]). For each parameterized sentence
φx1 ,...,xk of PMLO, one can compute for all  > 0 a representation of a set Hφ ⊂ Rk of measure at most , such thatthe problem of determining if I(M ) |= φα is decidable forα ∈ Q \ Hφ .
Intuitively, the complement of Hφ contains the points thatare bounded away from limit points of the Markov chain, andcomparisons can be decided using convergence properties.
The reason for excluding the set Hφ is that the limit pointcases seem to require a resolution of problems related to
the Skolem problem concerning zeros of linear recurrences[Sko34]. A sequence of real numbers {un } is called a linear
recurrence sequence (LRS) of order k if there exist a1 , . . . akwith ak 6= 0 such that for all m ≥ 1,
uk+m = a1 uk+m−1 + a2 uk+m−2 + · · · + ak um .
We consider the following decision problems associated witha LRS {un }.

2. For n ≥ 1, un = (An )1k for a square matrix A withrational entries.
3. For n ≥ 1, un = vT An w where A is a square matrix,
and v and w are vectors with entries from {0, 1}.
In the usual formulation of the Skolem, positivity and ultimate positivity problems, the associated matrices A may
contain negative numbers, and numbers not in [0, 1], so are
not stochastic matrices. However, [AAOW15] show thatthese problems can be reduced to a decision problem statedwith respect to stochastic matrices:Lemma 8. Given an integer k×k matrix A, one can compute a k0 × k0 stochastic matrix B, a length k0 stochastic
vector v, a length k0 vector w = (0, . . . , 0, 1) and a constant c such that (An )1,k = 0 ((An )1,k > 0) iff vT B n w = c(respectively, vT B n w > c).
As noted in [AAOW15], it follows that the logic PMLO isable to express the Skolem and positivity problems, using
model checking questions of the form
M |= ∃t(Pr(p(t)) = c)andM |= ∃t(Pr(p(t)) > c)for c a nonzero constant and M a DTMC. (The ultimate
positivity problem can also be expressed.) It is worth noting that in the special case of the constant c = 0, these model
checking questions are decidable, as shown in [BRS06]. (Essentially, in this case the problems reduce to graph reachability problems, and the specific probabilities in M are irrelevant.) The transformation from arbitrary matrices A tostochastic matrices B in Lemma 8 requires that the constant
0 of the Skolem problem be replaced by a non-zero constantc.
The above model checking problems of the quantified logic
PMLO can be seen to be already expressible in the propositional logic CTL∗ KP, as the problems
M 0 |=clk AF(pri (p) = c)

1. Skolem problem. Does there exist n such that un =0?

M 0 |=clk AF(pri (p) > c)

2. Positivity problem. Is it the case that for all n,
un ≥ 0?
3. Ultimate positivity problem. Does a positive integer N exist such that for all n ≥ N , un ≥ 0?

M 0 |=clk AFAG(pri (p) > c)where we obtain the PO-DTMC M 0 from the DTMC M bydefining Oi (s) = ⊥ for all states s. That is, agent i is blind,202

so considers all states reachable at time n to be possible attime n. (We remark that this implies that all the operators A
can be exchanged with E without change of meaning of theformulas.) It follows, that with respect to clock semantics,
a resolution of the decidability of model checking even these
simple formulas of CTL∗ KP for all c ∈ [0, 1] would imply
a resolution of the Skolem problem. In view of the effort
already invested in the Skolem problem, this is likely to behighly nontrivial.

4.2

Perfect Recall Semantics

i’s local state ri [0 . . . m] at (r, m) is a word in Σ+ . Let
B(q, m) be the set of runs r ∈ Rspr (MA ) in which re (m) =(q, a) for some a ∈ Σ. We claim the following about theprobability measures µr,m,i in the probabilistic interpreted
system I spr (MA ), for each point (r, m) and q ∈ Q:
µr,m,i (Ki (r, m)(B(q, m))) = (v0 A(ri (1)) . . . A(ri (m)))(q) .
It is immediate from this that I s pr(MA ), (r, m) |= Pri (p) =c where c = f (r[1 . . . m]), and the desired result follows.

We remark that this result stands in contrast to the situation for model checking the logic of knowledge and time.
Write CLTL∗ K for the logic obtained from CTL∗ KP byomitting the probability comparison atoms f (P1 , . . . , Pk ) ./c. Model checking the logic CLTL∗ K with respect to perfect recall, i.e., deciding M |=spr φ for M a PO-DTMC andTheorem 9. The problem of determining, given a POφ a formula is decidable [MS99]. (Here, for the semantic
DTMC M , if M |=spr EF (Pri (p) > c), for p an atomic
structures M , it suffices to replace the initial distribution
proposition, is undecidable.
P I in M by the set I = {s ∈ S | P I(s) > 0}, and replace
the transition distribution function P T in M by the relation
Proof. (Sketch) By reduction from the emptiness probR of possibility of transitions between states defined by sRt
lem for probabilistic finite automata [Paz71]. Intuitively,
if P T (s, t) > 0. The results in [MS99] use linear time temthe proof sets up an association between words in the maporal logic as a basis, but, as noted in [MW03], the modality
trix semigroup and sequences of observations of the agent.
A of the branching time logic CT L∗ can be understood as a
A probabilistic finite automaton is a tuple A = (Q, Σ, v0 , A, F, λ),special case of a knowledge modality: see Proposition 2.)
where Q is a finite set of states, Σ is a finite alphabet,
For probabilistic automata the minimum size of the statev0 : Q → [0, 1] is a probability distribution over states, repspace giving undecidability directly stated in the literature
resenting the initial distribution, A : Σ → (Q × Q → [0, 1])appears to be 25 [Hir06]. We remark that the proof of Theoassociates a transition probability matrix A(a) with eachrem 9 can also be done by reduction of the following matrix
letter a ∈ Σ, component F ⊆ Q is a set of final states, andsemigroup problem: given a finite set of matrices of order n,λP∈ (0, 1) is a rational number. Each matrix A(a) satisfies
generating a matrix semigroup S, determine whether there
A(a)(s,
t)
=
1
for
all
s
∈
Q.
Let
v
be
the
columnFt∈S
is M ∈ S such that (M )1n = 0 [Hal97]. The case of k genervector indexed by Q with vF (s) = 0 if s 6∈ F and vF (s) = 1
ators of size n × n can be reduced to probabilistic automata
if s ∈ F . Treating v0 as a row vector, for each wordwith 2kn + 1 states. Recent results on the matrix semigroupw = a1 . . . an ∈ Σ+ , define f (w) = v0 A(a1 ) . . . A(an )vF .
problem are given in [CHHN14].
The language accepted by the automaton is defined to be
Huang et al [HSZ12] have previously used a reduction from
L(A) = {w ∈ Σ+ | f (w) > λ}. The emptiness problem forprobabilistic automata to show undecidability of an probaprobabilistic finite automata is then, given a probabilistic
bilistic epistemic logic with respect to perfect recall. Comfinite automaton A, to determine if the language L(A) is
pared to our simple CTL temporal operators, their logic uses
empty. This problem is known to be undecidable [Paz71,
more expressive setting of alternating temporal logic operaCL04].
tors.
Given a probabilistic finite automaton A, we construct aninterpreted finite PO-DTCM MA for a single agent (called
4.3 Clock Semantics
i rather than 1 to avoid confusion with other numbers) such
The undecidability of the perfect recall semantics for suchsprthat MA |=EF (Pri (p) > λ) iff A is nonempty. This
simple formulas suggests that we weaken the epistemic sesystem is defined as follows. We let N = |Σ|,mantics to the clock case. The combination of the translation from CTL∗ KP to WMLOKP (Proposition 2) and
1. S = Q × Σ,
Theorem 6 then enables some cases of CTL∗ KP to be de2. P I(q, a) = µ0 (q)/N ,cided. We do not obtain a full decidability result, however,since we face the problem that, with respect to the clock se00
3. P T ((q, a), (q , b)) = A(b)(q, q )/N
mantics, the formula AF (Pri (p) = c) can express the Skolem
problem, so resolving its decidability is a very difficult prob4. Oi ((q, a)) = a
lem. Rather than attempt to resolve this question, we con5. p ∈ π((q, a)) iff q ∈ F .
sider here just how much extra expressiveness is requiredPPP
over the logic of Theorem 6 for us to obtain a definitive unNote that(q,a)∈S P I((q, a)) =a∈Σq∈Q µ0 (q)/N =P
decidability result, instead of a decidability result with somea∈Σ 1/N = 1, so P I is in fact
P a distribution. Similarly,
excluded and unresolved cases.
0
for each (q, a) ∈ S, we have(q 0 ,b)∈S P T ((q, a), (q , b)) =Note that one of the restrictions on PMLO used in TheP
P
P
0
b∈Σ
q 0 ∈Q A(b)(q, q )/N =b∈Σ 1/N = 1, so P T is inorem 6 is that second order quantification should not crossfact a stochastic matrix.
into probability terms. It turns out that this restriction isNote that for each w = a1 . . . an ∈ Σ∗ andessential, as shown by the following result.
P a ∈ Σ, we get a
row vector µaw = µ0 A(a1 ) . . . A(an ) with q∈Q µaw (q) = 1,
Theorem 10. It is is undecidable, given a PO-DTMC M
which can be understood as a distribution on Q. For eachand a formula φ of WMLOKP with linear combinations ofrun r ∈ Rspr (MA ) and m ≥ 0, we have that that agent

Model checking with respect to the perfect recall semantics is undecidable, even with respect to a very simple fixed
formula of the logic CTL∗ KP, as shown by the following
result.

203

probability terms Pr(φ) and quantifying-in of second-order
quantifiers, whether M |= φ.
Proof. This follows from the fact that, using second order quan-tifying-in, we can express perfect recall (Proposition 4), and the undecidability of model checking perfectrecall (Theorem 9).

Note that the result refers to |= rather than |=clk , since
epistemic operators are not required. This is really a result about a generalization of PMLO. One of the otherrestrictions in Theorem 6 is that only simple probability
comparisons of the form Pr(φ) ./ c are permitted. Moregeneral comparisons of probability terms are needed in applications (see discussion in Section 2.5), so it is of interest to
study their impact on decidability. Unfortunately, it turns
out to be quite negative. Even the simple case of mixed
time polynomial atomic probability formulas is enough for
undecidability.
Theorem 11. It is undecidable, given a DTMC M and amixed-time polynomial atomic probability formula ψ, whether
M |= ψ.
Proof. (Sketch) By reduction from Hilbert’s tenth problem, i.e., the problem of determining whether a polynomialwith integer coefficients has solutions in the natural numbers. This was shown to be undecidable by Matiyasevich[Mat93].
Write ej for a basic column vector (of the appropriate dimension) with 1 in the jth place and 0 elsewhere. Given a
PO-DTMC with transition matrix M , with a proposition foreach state, we can express the values eTi M t ej using termsof the form c.Pr(p(t)) where p is an atomic proposition andc a rational constant. This means that using linear sums ofprobability expressions of the form Pr(p(t)) for some proposition p, we can capture f T M t g for arbitrary rational vectors
f , g.
We show that we can find a stochastic matrix M such thatfor each function f (t) = t · λt and f (t) = λt with λ = −1/6,
there are rational vectors f , g such that f (t) = f T M t g.
Given a polynomial p(x1 , . . . , xn ), we can construct a variant polynomial q 0 over a larger set of variables, such thatan appropriate substitution of such functions ti · λti and
λti , for the xi and the additional variables yields an expression λk1 t1 +...+kn tn · p(t1 , . . . , tn ), where the ki are constants.
This has a zero in the t1 . . . tn iff p(x1 , . . . , xn ) has a zero.
It follows using the result of the previous paragraph that
mixed-time polynomial atomic probability formulas can express Hilbert’s tenth problem.

We remark that the possibility of encoding Hilbert’s tenthproblem is not immediate from the fact that we are dealing with polynomials, since our polynomials are over rational values generated in a very specific way from Markov
chains, rather than arbitrary integers. Indeed, there are decidable logics containing polynomials, such as the theory ofreal closed fields [Tar51].
As noted in Section 2.5, formulas (allowed by Theorem 6)of the form Pr(φ(t1 , . . . , tn )) ./ c can be written as a polynomial of probability expressions, so it is natural to ask
whether such formulas also suffice to make the logic undecidable. This does not seem to be the case: the polynomials
involved have only positive coefficients. Since Hilbert’s tenthproblem is trivially decidable for polynomials with only positive coefficients, our proof does not apply to this case.

5.

CONCLUSION

Our results have by no means resolved Skolem’s problem,
which remains an apparent barrier to resolving the gap between the decidability results of [BRS06] and the undecidability results of the present paper.
However, in work to be presented elsewhere, we show thatthe results of [BRS06] can be extended both by reducing
the set Hφ of cases that needs to be excluded to obtain decidability, as well as enhancing the expressiveness to coverepistemic probabilistic terms of the form Pri (φ), interpretedwith respect to the clock semantics.
Acknowledgment: The authors thank Igor Shparlinski
and Min Sah for illuminating discussions on Skolem’s problem, and Xiaowei Huang for helpful comments on a draft ofthe paper.

6.

REFERENCES[AAOW15] S. Akshay, T. Antonopoulos, J. Ouaknine, andJ. Worrell. Reachability problems for Markov
chains. Information Processing Letters,
115(2):155–158, 2015.
[ACD90]R. Alur, C. Courcoubetis, and D. L. Dill.
Model-checking for real-time systems. In Proc.
of the Symposium on Logic in Computer
Science, pages 414–425, 1990.
[BRS06]
D. Beauquier, A. M. Rabinovich, and
A. Slissenko. A logic of probability withdecidable model checking. J. Log. Comput.,
16(4):461–487, 2006.
[Buc60]J. R. Buchi. Weak second order arithmetic andfinite automata. Zeitscrift fur mathematische
Logic und Grundlagen der Mathematik,
6:66–92, 1960.
[CE81]
E. M. Clarke and E. A. Emerson. The designand synthesis of synchronization skeletonsusing temporal logic. In Proc. of the Workshop
on Logics of Programs, IBM Watson ResearchCenter, LNCS 131, 1981.
[CES86]E. M. Clarke, E. A. Emerson, and A. P. Sistla.
Automatic verification of finite-state
concurrent systems using temporal logic
specifications. ACM Trans. Program. Lang.
Syst., 8(2):244–263, 1986.
[CHHN14] J. Cassaigne, V. Halava, T. Harju, andF. Nicolas. Tighter undecidability bounds formatrix mortality, zero-in-the-corner problems,and more. arXiv, abs/1404.0644, 2014.
[CL04]
A. Condon and R. J. Lipton. On thecomplexity of space bounded interactive proofs.
In Proc. of the Symp. on Foundations ofComputer Science, pages 462–467. IEEE, 2004.
[EGM07] K. Engelhardt, P. Gammie, and R. van der
Meyden. Model checking knowledge and linear
time: PSPACE cases. In Proc. of the Int.
Symp. on Logical Foundations of Computer
Science LFCS, pages 195–211, 2007.
[Eij04]
D.J.N. Eijck. Dynamic epistemic modelling.
CWI. Software Engineering [SEN], (E0424):1–112, 2004.
[ESPW03] G. Everest, I. Shparlinski, A. J. van der204Poorten, and T. Ward. Recurrence sequences.
Amer. Math. Soc., 2003.
[FH94]
R. Fagin and J. Y. Halpern. Reasoning aboutknowledge and probability. J. ACM,
41(2):340–367, 1994.
[FHM90] R. Fagin, J. Y. Halpern, and N. Megiddo. A
logic for reasoning about probabilities.
Information and Computation, 87(1/2):78–128,
1990.
[GM04]
P. Gammie and R. van der Meyden. MCK:
Model checking the logic of knowledge. In
Proc. Conf. on Computer-Aided Verification,
CAV, pages 479–483, 2004.
[Hal97]
V. Halava. Decidable and undecidable
problems in matrix theory. Technical Report
127, Turku Centre for Computer Science,
University of Turku, Finland, 7 1997.
[Hal03]J. Y. Halpern. Reasoning about Uncertainty.
MIT Press, Cambridge, MA, USA, 2003.
[HHHK05] V. Halava, T. Harju, M. Hirvensalo, and
J. Karhumäki. Skolem’s problem - on theborder between decidability and undecidability.
Technical Report 683, Turku Centre for
Computer Science, University of Turku,
Finland, 4 2005.
[Hir06]
M. Hirvensalo. Improved undecidability resultson the emptiness problem of probabilistic andquantum cut-point languages. Technical
Report 769, Turku Centre for Computer
Science, University of Turku, Finland, 5 2006.
[HLM11] X. Huang, C. Luo, and R. van der Meyden.
Symbolic model checking of probabilisticknowledge. In Proc. of the Conf. on Theoretical
Aspects of Rationality and Knowledge, pages
177–186, 2011.
[HM10]
X. Huang and R. van der Meyden. The
complexity of epistemic model checking: Clocksemantics and branching time. In Proc. ECAI2010 - European Conf. on Artificial
Intelligence, pages 549–554, 2010.
[HSZ12]
X. Huang, K. Su, and C. Zhang. Probabilistic
alternating-time temporal logic of incomplete
information and synchronous perfect recall. In
Proc. AAAI, 2012.
[HV91]J. Y. Halpern and M. Y. Vardi. Modelchecking vs. theorem proving: A manifesto. In
Proc. of the Int. Conf. on Principles ofKnowledge Representation and Reasoning,
pages 325–334, 1991.
[Kam68]
H. W. Kamp. Tense logic and the theory of
linear order. PhD thesis, University ofCalifornia, Los Angeles, 1968.
[KNN+ 08] M. Kacprzak, W. Nabialek, A. Niewiadomski,
W. Penczek, A. Pólrola, M. Szreter, B. Woźna,
and A. Zbrzezny. Verics 2007 - a model checker
for knowledge and real-time. Fundamenta
Informaticae, 85(1):313–328, 2008.
[KSK76]J. G. Kemeny, J. L. Snell, and A. W. Knapp.
Denumerable Markov Chains. Springer-Verlag,
1976.
[LQR09]A. Lomuscio, H. Qu, and F. Raimondi.
MCMAS: a model checker for the verification

205of multi-agent systems. In Proc. Conf. on
Computer-Aided Verification, pages 682–688,
2009.
[Mat93]
Yuri V. Matiyasevich. Hilbert’s Tenth Problem.
MIT Press, 1993.
[MNP08] O. Maler, D. Nickovic, and A. Pnueli.
Checking temporal properties of discrete,
timed and continuous behaviors. In Pillars ofComputer Science, Essays Dedicated to Boris(Boaz) Trakhtenbrot on the Occasion of His
85th Birthday, pages 475–505, 2008.
[MS99]
R. van der Meyden and N. V. Shilov. Modelchecking knowledge and time in systems withperfect recall. In Proc. FST-TCS, pages
432–445, 1999.
[MW03]
R. van der Meyden and K-S. Wong. Complete
axiomatizations for reasoning about knowledge
and branching time. Studia Logica,
75(1):93–123, 2003.
[OW14]
J. Ouaknine and J. Worrell. Positivity
problems for low-order linear recurrence
sequences. In Proc. ACM-SIAM Symposium on
Discrete Algorithms, pages 366–379, 2014.
[Paz71]
A. Paz. Introduction to probabilistic automata.
Academic Press, 1971.
[RKNP04] J. Rutten, M. Kwiatkowska, G. Norman, and
D. Parker. Mathematical Techniques forAnalyzing Concurrent and Probabilistic
Systems, P. Panangaden and F. van Breugel(eds.), volume 23 of CRM Monograph Series.
American Mathematical Society, 2004.
[Sha49]
C. Shannon. Communication theory of secrecy
systems. Bell System Technical Journal,
28(4):656–715, 1949.
[Sko34]T. Skolem. Ein Verfahren zur Behandlung
gewisser exponentialer Gleichungen und
diophatischer Gleighungen. In 8de Skand. mat.
Kongr. Forth, Stockholm, 1934.
[Tar51]
A. Tarski. A Decision method for elementary
algebra and geometry. Univ. of California
Press, 2nd edition, 1951.
[TMS84]
R. Tijdeman, M. Mignotte, and T.N. Shorey.
The distance between terms of an algebraic
recurrence sequence. Journal für die reine undangewandte Mathematik, 349:63–76, 1984.

Relating Knowledge and Coordinated Action:
The Knowledge of Preconditions Principle∗

Yoram Moses

Technion—Israel Institute of Technology

moses@ee.technion.ac.il

ABSTRACTThe Knowledge of Preconditions Principle (KoP):

The Knowledge of Preconditions principle (KoP) is proposed as a widely applicable connection between knowledge
and action in multi-agent systems. Roughly speaking, it asserts that if some condition ϕ is a necessary condition for
performing a given action α, then knowing ϕ is also a necessary condition for performing α. Since the specifications oftasks often involve necessary conditions for actions, the KoPshows that such specifications induce knowledge preconditions for the actions. Distributed protocols or multi-agent
plans that satisfy the specifications must ensure that thisknowledge be attained, and that it is detected by the agentsas a condition for action. The knowledge of preconditions
principle is formalised in the runs and systems framework,
and is proven to hold in a wide class of settings. Well-known
connections between knowledge and coordinated action areextended and shown to derive directly from the KoP: acommon knowledge of preconditions principle is establishedshowing that common knowledge is a necessary condition for
performing simultaneous actions, and a nested knowledge ofpreconditions principle is proven, showing that coordinating
actions to be performed in linear temporal order requires acorresponding form of nested knowledge.

Keywords
Knowledge, multi-agent systems, common knowledge, nested
knowledge, coordinated action, knowledge of preconditions
principle

1.

INTRODUCTION

While epistemology, the study of knowledge, has been a
topic of interest in philosophical circles for centuries andperhaps even millennia, in the last half century it has seen
a flurry of activity and applications in other fields such as
AI [19], game theory [2] and distributed computing [13]. At
least in the latter two fields a particular, information-based,
notion of knowledge plays a prominent and useful role.
This paper proposes an essential connection between knowledge and action in such a setting. Using doesi (α) to denote“Agent i is performing action α” and Ki ϕ to denote that
Agent i knows the fact ϕ, the connection can intuitively beformulated as follows:∗The Israel Pollak academic chair at Technion. This work
was supported in part by ISF grant 1520/11.

207

If

ϕ

is a necessary condition for doesi (α)

then

Ki ϕ

is a necessary condition for doesi (α)

This statement appears deceptively simple. In fact, manysuccessful applications of knowledge to the design and analysis of distributed protocols over the last three decades arerooted in the KoP. Moreover, some of the deeper insights
obtained by knowledge theory in this field can be derived
in a fairly direct fashion from the KoP. We will argue anddemonstrate that this principle lies at the heart of coordination in many distributed and multi-agent systems.
This paper is structured as follows. Section 1.1 illustrates
the central role of knowledge in a natural distributed systems
application. Section 1.2 provides a high-level discussion of
the knowledge of preconditions principle and its connection
to coordinating actions. In Section 2 we review and discuss
the modelling of knowledge in the runs and systems model of
distributed systems based on [11]. A formal statement andproof of the KoP are presented in Section 3. Then, in Section 4, the KoP is used to establish a common knowledgeof preconditions principle. It states that in order to perform simultaneously coordinated actions, agents must firstattain common knowledge of any of the actions’ preconditions. An example of its use is provided in Section 4.1. Section 5 present an additional use of the KoP, and shows thatcoordinating a sequence of actions to occur in a prescribedtemporal order requires attaining nested knowledge of theirpreconditions. Finally, Section 6 discusses additional applications, extensions and future directions.

1.1

The Case for Knowledge in Distributed
Systems

Why should knowledge play a central role in distributed
computing? As pointed out in [13], most everyone who designs or even just tries to study the workings of a distributed
protocol is quickly found talking in terms of knowledge, making statements such as “once the process receives an acknowledgement, it knows that the other process is ready. . . ”. Anessential aspect of distributed systems is the fact that an
agent chooses which action to perform based on the local
information available to it, which typically provides only apartial view of the overall state of the system. To get a senseof the role of knowledge in distributed systems, consider thefollowing example.
Example 1. Given is a distributed network modeled by a

75

100

80

100

Figure 1: A simple four-agent systemgraph, with agents located at the nodes, and the edges standing for communication channels (see Figure 1). In the problem we shall call Computing the Max (or CtM for short),each agent i starts out with a natural number vi ∈ N as an
initial value. The goal is to have Agent 1 print the maximumof all of the initial values (we denote this value by Max), andprint nothing else. In the instance depicted in Figure 1, themaximal value happens to be 100. Initially, Agent 1 clearly
can’t print its own initial value of 75. Suppose that Agent 1
receives a message µ , “v2 = 100” from Agent 2 reportingthat its value is 100. At this point, Agent 1 has access to the
maximum, and printing 100 would satisfy the problem specification. Compare this with a setting that is the same inall respects, except that Agent 3’s value is v3 = 150. In thiscase, of course, Max 6= 100 and so printing 100 is forbidden. But if Agent 1 can receive the same message µ undersimilar circumstances in both scenarios, then it is unable todistinguish whether or not Max = 100 upon receiving µ. Intuitively, even in the first scenario, the agent does not knowthat Max = 100.
What information does Agent 1 need, then, in order to beable to print the maximum? Notice that it, is not necessary,in general, to collect all of the initial values in order to print
the maximum. For example, suppose that the agents follow
a bottom-up protocol in which values are sent from right toleft, starting from Agent 4, and every agent passes to the left
the larger of its own value and the value it received from its
neighbor on the right (if such a neighbor exists). In this protocol, Agent 1 can clearly print the maximum after receiving
the message “v2 = 100”, and seeing just one value besides
its own. Interestingly, even collecting all of the values is
not a sufficient condition for printing the Max. Imagine asetting in which the network is as in Figure 1, but Agent 1
considers it possible that there are more than four nodes in
the network. In this case, even if Agent 1 receives all (four)
values, it may still need to wait for proof that there is no
additional, larger, value in the system.

CtM is a simplified example in the spirit of many distributed systems applications. Leader election, for example,
is often solved by computing a node with maximal ID [1,17]. The solution to such a problem is typically in the formof a set of short computer programs (jointly constituting
a distributed protocol) each executed at one of the nodes.
When the nodes follow such a protocol, the resulting execution should satisfy the problem specification. Of course, theprograms are written in a standard programming language,
without any reference to knowledge or possibility. In thevast majority of cases, the programs in question do not enumerate and/or explore possible states or scenarios. Indeed,

the program designer is typically unfamiliar with formal notions of knowledge. This being the case, what sense doesit make to talk of Agent 1 in Example 1 “knowing” or “notknowing” that Max = c? Can it make sense to say that theAgent “considers it possible that there may be more thanfour nodes in the system”? After all, we may be talking
about a ten-line program. It has no soul. Does it havethoughts, doubts and mental states?
Since agents act based on their local information, a protocol designer must ensure that agents obtain the necessary information for a given task, and that this information is applied correctly. Using the information-based notion of knowledge, the designer can ascribe knowledge to anagent without requiring it to have a soul, feelings, and selfawareness. As seen in the CtM example, it is natural tothink in terms of whether or not Agent 1 knows Max = c
at any given point in a run of CtM. (A formal definitionof knowledge will be provided in Section 2.) Suppose thata protocol is designed to solve CtM in networks that may
have a variety of sizes. If Agent 1 does not start out withlocal information ensuring that there are no more than four
nodes in the system, then from the point of view of an outside observer the agent can be thought of as “considering it
possible” that there may be more than four nodes.
Even in a simple network as in Figure 1, the CtM problem can be posed in different models, which can differ in
essential aspects. A solution to CtM in one model might
not solve the problem in another model. Indeed, the rationale behind distinct solutions, as well as their details, mayvary considerably. Are there common features shared by all
solutions to CtM?
Interestingly, all solutions to CtM, in all models, share
one property: Agent 1 must know that Max = c in order to print the value c. Indeed, the ability to print theanswer in a protocol for CtM reduces to detecting when
the Max value is known. Of course, once Agent 1 knowsthat Max = c it can safely print c. Hence, knowing that
Max = c is not just necessary, but also a sufficient condition
for printing c. The CtM problem shows that knowledge andattaining knowledge can be a central and crucial aspect of astandard distributed application.
The need to know Max = c in solving CtM suggests that
we consider a natural question: When does Agent 1 knowthat Max = c? The answer is less straightforward than wemight initially expect. What is known depends in a crucial
way on the protocol that the agents are following. Thus, inthe setting of Example 1, if the agents follow the bottom-up
protocol, then Agent 1 knows the maximum once it receives
a single message from Agent 2. Knowledge is also significantly affected by features of the model. In CtM, if there isan upper bound (say 100) on the possible initial values, thenan agent that sees this value knows the maximum. Knowledge about the network topology and properties of communication play a role as well. For example, consider a modelin which Agent 1 has a clock, and a single clock cycle suffices for a message to be delivered, digested, and acted upon.
Suppose that the protocol is such that all agents start simultaneously at time 0 and an agent forwards a value towards
Agent 1 only if this value is larger than any value it has previously sent. Then in the network of Figure 1 Agent 1 will
receive a message with value 100 from Agent 2 at time 1, andno further messages. If Agent 1 knows that the diameter ofthe network is 3, it will not know the maximum upon receiv-

208ing this message. However, without receiving any further
messages, at time 3 Agent 1 will know that the maximum is100; no larger value can be lurking in the system.

1.2

KoP and Coordination

The fact that Max = c is a necessary condition for printing c is an essential feature of the CtM problem. We haveargued that, in fact, K1 (Max = c) is also a necessary condition for printing c, as the KoP would suggest. But this isjust one instance. Let us briefly consider another example.
Example 2. Consider a bank whose ATMs are designed
in such a way that an ATM will dispense cash only to acustomer whose account shows a sufficiently large positivebalance. Along comes Alice, who has a large positive balance, and tries to obtain a modest sum from the ATM. On
this day, however, the ATM is unable to communicate withthe rest of the bank and it declines to pay Alice. Thus, despite the fact that Alice has good credit, the ATM frustrates
her and denies her request. Apparently, given its specification, the ATM has no choice. Intuitively, in order to satisfy
the credit restriction, the ATM needs to know that a customer has good credit before dispensing cash. If the ATM
may pay a customer that is not known to have good credit,
there will be possible scenarios in which the ATM will violate its specification, and pay a customer that does not havecredit. Notice, however, that the specification said nothing
about the ATM’s knowledge. It only imposed a restriction
on the ATM’s action, based on the state of Alice’s account.
Both the CtM problem and the ATM example are instancesin which the KoP clearly applies. The intuitive argument for
why the KoP should apply very broadly is straightforward.
If ϕ is a necessary condition for performing α, and agent iever performs α without knowing ϕ, then there should bea possible scenario that is indistinguishable to agent i, in
which ϕ does not hold. Since the two scenarios are indistinguishable, the agent can perform α in the second scenario,
and violate the requirement that ϕ is a necessary condition.
A formal statement and proof requires a definition of necessary conditions, knowledge, as well as capturing a sensein which an action at one point implies the same action at
any other, indistinguishable, point. This will be done in
Section 3.
Most tasks in distributed systems are described by wayof a specification. Such specifications typically impose a variety of necessary conditions for actions. The KoP implies
that even though such specifications often do not explicitly discuss the agents’ knowledge, they do in fact impose
knowledge preconditions. Observe that the KoP applies toa task regardless of the means that are used to implement
it. Any engineer implementing a particular task will have toensure that preconditions are known when actions are taken.
This is true whether or not the engineer reasons explicitly
in terms of knowledge, and it is true even if the engineer is
not even aware of the knowledge terminology. (Normally,neither may be the case, of course.) The need to satisfy
the KoP suggests that the design of distributed implementations must involve at least two steps. One is to make surethat the required knowledge is made available to an agent
who needs to performed a prescribed action, and the otheris ensuring that the agent detect that it knows the required
preconditions. This is quite different from common practice209

in engineering distributed implementations [28].
We remark that the KoP can be expected to hold in avariety of multi-agent settings well beyond the realm of distributed systems. Thus, for example, suppose that a jellyfish
is designed so that it will never sting its own flesh. By the
KoP, the cell activating the sting at a given point needs toknow that it is not stinging the jellyfish’s body when it “fires”
its sting. The jellyfish is thus designed with some form of a“friend or foe” mechanism that is used in the course of activating the sting. Various biological activities can similarly
be considered in light of the KoP: How does the organism
know that certain preconditions are met? Our last example
will come from the social science arena. Suppose that a society designs a legal system, that is required to satisfy theconstraint that only people who are guilty of a particularcrime are ever put in jail based on this crime. By the KoP,
the judge (or jury) must know that the person committed
this crime in order to send him to jail.
As discussed above, specifications impose preconditions.
Typically, these conditions relate an action to facts about
the world (e.g., the maximal value, or the customer’s good
credit). In many cases, however, actions of different agentsneed to be coordinated. Consider a variant of CtM in whichin addition to Agent 1 printing the maximum, Agent 4 needs
to perform an action (say print the same value or print theminimal value), but not before Agent 1 does. Then Agent 1
performing her action is a condition for 4’s action. In particular, Agent 4 would need to know that Agent 1 has alreadycome to know Max = c for some c before 4 acts. In somecases, the identity of actions performed needs to be coordinated.
For a final example, suppose that Alice 1 should perform
an action αA only if Bob performs an action αB at least 5time steps earlier. Then she needs to know that Bob acted atleast 5 steps before when she acts. Indeed, if ψ is a necessary
condition for α0 , then Alice must know that Bob performed“Bob knew ψ at least 5 time steps ago” when she acts (see [4,5]). As these examples illustrate, given KoP, coordination
can give rise to nested knowledge.
Simple instances of the KoP are often quite straightforward. Ensuring and detecting K1 (M ax = c) is often fairly
intuitive, and it not justify the overhead involved in developing a theory of knowledge for multi-agent systems. However,satisfying statements involving nested knowledge in particular models of computation can quickly become nontrivial.
For this, it is best to have a clear mathematical model ofknowledge in multi-agent systems. The next section reviews
the runs and systems model.

2.

MODELING KNOWLEDGE USING RUNSAND SYSTEMS

We now review the runs and systems model of knowledgeof [11, 13]. The interested reader should consult [11] formore details. A global state is an “instantaneous snapshot” of the system at a given time. Let G denote a setof global states. Time will be identified with the natural
numbers N = {0, 1, 2, . . .} for ease of exposition. A run isa function r : N → G associating a global state with each
instant of time. Thus, r(0) is the run’s initial state, r(1) is
the next global state, and so on. A system is a set R ofruns. The same global state can appear in different runs,
and in some systems may even appear more than once inthe same run.
A central notion in our framework is that of an agent’s
local state, whose role is to capture the agent’s local information at a given point. The precise details of the local
state depend on the application. It could be the complete
contents of an agent’s memory at the given instant, or thecomplete sequence of events that it has observed so far. forexample. The rule of thumb is that the local state should
consist of the local information that the agent may use when
deciding which actions to take. Thus, for example, if agentsare finite-state machines, it is often natural to identify anagent’s local state with the automaton state that it is in.
Formally, we assume that every global state determines aunique local state for each agent. We denote agent i’s
local state in the global state r(t) by ri (t). Moreover, a
global state with n agents A = {1, . . . , n} will have the formr(t) = hre (t), r1 (t), . . . , rn (t)i, where re (t) is called the local
state of the environment, and will serve to represent all aspects of the global state that are not included in the agents’local states. For example, it could represent messages intransit, the current topology of the network including whatlinks may be down, etc.

2.1

Syntax and Semantics

We are interested in a propositional logic of knowledge,
in which propositional facts and epistemic facts can be expressed. Facts will be considered to be true or false at apoint (r, t), with respect to a system R. More formally, given
a set Φ of primitive propositions and a set P = {1, . . . , n}of the agents in the system, we define a propositional language LKn (Φ) by closing Φ under negation ‘¬’ and conjunction ‘∧’, as well as under knowledge operators Ki for alli ∈ P (see [14]). Thus, for example, if p, q ∈ Φ are primitive
propositions and i, j ∈ P are agents, then ¬Ki p∧Kj Ki ¬Kj q
is a formula in LKn (Φ). We typically omit the set Φ and call
LKn the language for knowledge with n agents.
In a multi-agent system facts about the world, as wellas the knowledge that agents have, can change dynamicallyfrom one time point to the next. We thus consider the truthof formulas of LK
n at points of a system R, where a pointis a pair (r, t) ∈ R × N, and it is used to refer to time t
in the run r. We denote the set of points of a system R
by Pts(R) , R × N. Points will play the role of states of a
Kripke structure.
The set Φ of primitive propositions used in the analysisof any given multi-agent system R will depend on the application. Their truth at the points of the system needs
to be explicitly defined. This is done by an interpretation
π : Φ × Pts(R) → {T, F }, where π q, (r, t) = T means thatthe proposition q holds at (r, t). Formally, an interpreted system w.r.t. a set Φ of primitive propositions is a pair (R, π)
consisting of the system R and interpretation π for Φ overPts(R). Just as we typically omit explicit reference to Φ, weshall omit π as well, when this is unambiguous.
We assume from here on that the environment’s state re (t)in a global state r(t) contains a “history” component h thatrecords all actions taken by all agents at times 0,1,. . . ,t − 1.
Formally, we take h to be a set of triples hα, i, t0 i, which
grows monotonically in time. An action α is considered tobe performed by i at the point (r, t) if and only if the triple
hα, i, ti, denoting that action α was performed by agent i attime t, appears in the history component h of re (t0 ) for alltimes t0 > t.1 For the analysis in this paper, we will also
assume that Φ includes propositions of the form doesi (α)
and didi (α) for agents i ∈ P and actions α. With thisassumption, what actions are performed at any given point(r, t) is uniquely determined by the run r.
We will consider interpretations π that, on these propositions, are defined byπ doesi (α), (r, t) = T iff agent i performs α at (r, t)
π didi (α), (r, t) = T
iff π doesi (α), (r, t0 ) = T
holds for some t0 ≤ t
We allow t0 = t in the definition of didi (α) for technicalconvenience; it simplifies our later analysis slightly.
Our model of knowledge will follow the standard Kripkestyle possible worlds approach. The possibility relations that
we use are induced directly from the system R being analyzed; two points are considered indistinguishable to anagent if its local states at the two points are the same. Moreformally:

Definition 2.1. If ri (t) = ri0 (t0 ), then (r, t) and (r0 , t0 )are called indistinguishable to i, denoted by (r, t) ≈i (r0 , t0 ).
Formulae of LK
n are interpreted at a point (r, t) of an interpreted system (R, π) by means of the satisfaction relation‘|=’, which is defined inductively by:(R, r, t) |= p iff (r, t) ∈ π(p);(R, r, t) |= ¬ϕ iff (R, r, t) 6|= ϕ;(R, r, t) |= ϕ ∧ ψ iff both (R, r, t) |= ϕ and (R, r, t) |= ψ;(R, r, t) |= Ki ϕ iff (R, r0 , t0 ) |= ϕ for all (r0 , t0 ) ∈ Pts(R)such that (r0 , t0 ) ≈i (r, t).
We say that ϕ is valid in the system R, and write R |= ϕ,
if (R, r, t) |= ϕ for all points (r, t) ∈ Pts(R). We say thatϕ validly implies ψ in R if ϕ ⇒ ψ is valid in R. Since,by Definition 2.1, the ≈i relations are equivalence relations,each knowledge operator Ki satisfies the S5 axiom system [14].
In particular, it satisfies the knowledge property (or axiom) that Ki ϕ ⇒ ϕ is valid in all systems.
It is instructive to relate our modeling using runs and systems to standard multi-agent Kripke structures. As shownin [11], for every system R there is a corresponding Kripke
structure MR = (SR , π, ∼1 , . . . , ∼n ) for n agents such that
SR = Pts(R) and ‘∼i ’ = ‘≈i ’ for every i. For every point(r, t) ∈ Pts(R) = SR and formula ϕ ∈ LKn (Φ) it is the casethat (R, r, t) |= ϕ iff MR , (r, t) |= ϕ.
The system R will determine the space of possible runs
and possible points, which play a crucial role in determining
the truth of facts involving knowledge. For example, consider a run r in which Alice sends Bob a message at time 1,
and Bob receives it at time 2. If R is a system in which messages may be lost, or may take longer than one time step
to be delivered, then Alice would not know at time 2 i.e.,
w.r.t. (R, r, 2) that her message has been delivered, because
there is another run r0 ∈ R that she cannot tell apart from r
at time 2, in which her message is not (or not yet) deliveredby that time. The same run r also belongs to another system R0 in which messages are always reliably delivered in1
Our definition does not imply or assume that the actions
are observed, observable or recorded by any of the agents.
Whether that may be the case depends on the application.

210

exactly one round. With respect to (R0 , r, 2), however, Alicewould know at time 2 that her message has been delivered.
Our definition of knowledge is rather flexible and widely
applicable. The set R of the possible runs immediately induces what the agents know. Observe that the definitionof knowledge is completely external. It ascribes knowledge
to agents in the system even if the protocol they follow, as
well as the actions that they perform, do not involve theknowledge terminology in any way. Moreover, the agents
do not need to be complex or sophisticated for the definition to apply. Indeed, in a model of a very simple system
consisting of a bed lamp and its electric cable, a switch in
the OFF state can be said to know that the lamp is not lit;
what the same switch would know in the ON state would
depend on the system R under consideration, which determines the runs considered possible. E.g., if R contains a run
in which the lamp is burnt out, then in the ON state theswitch would not know that the lamp is shining light. On
the other hand, if the lamp can never burn out, and the cord,plug and switch are in proper working order in all runs of R,
then in the ON state the switch would know that the lamp isshining light. As this example shows, knowledge under thisdefinition does not require the “knower” to compute what
it knows. Indeed, this definition of knowledge is not sensitive to the computational complexity of determining what is
known. In most cases, of course, we will ascribe knowledge
to agents or components that can perform actions, which is
not the case in the light switch example. And agents mightneed to explicitly establish whether they know relevant facts.
We now provide a statement and proof of the knowledge ofpreconditions principle KoP.

3. FORMALIZING THE KNOWLEDGE OF
PRECONDITIONS PRINCIPLE
Intuitively, the KoP states that if a particular fact ψ is anecessary condition for an agent to perform an action α, thenthe agent must in fact know ψ in order to act. In other words,
knowing ψ is also a necessary condition for performing theaction. We formalize the claim and prove it as follows. We
say that ψ is a necessary condition for doesi (α) in R
if (R, r, t) |= doesi (α) holds only if (R, r, t) |= ψ, for all(r, t) ∈ Pts(R). Clearly, the customer’s good credit is anecessary condition for the ATM dispensing cash. That is,
suppose that a bank makes use of a correct implementationof an ATM protocol, which satisfies the credit requirement.
Then, in the system R consisting of the set of all possible
histories (runs) of the bank’s (and the ATM’s) transactions,
good credit is a necessary condition for receiving cash fromthe ATM.
It is often of interest to consider facts whose truth depends
only on a given agent’s local state. Such, for example, may
be the receipt of a message, or the observation of a signal, by
the agent. Whether x = 0 for a local variable x, for example,
would be a natural local fact. Moreover, if an agent has
perfect recall, then any events that it has observed in thepast will give rise to local facts. Finally, since knowledge
is defined based on an agent’s local state, then a fact of
the form Ki ϕ constitutes a local fact. Indeed, there is a
simple way to define the local facts above, using knowledge.
Namely, we say that ϕ is i-local in R if R |= (ϕ ⇒ Ki ϕ).
The formalism of [11] defines protocols as explicit objects,
and defines contexts that describe the possible initial states

211

and the model of computation. This provides a convenientand modular way of constructing systems. Namely, given
a protocol P and a context γ, the system R = R(P, γ) is
defined to be the set of all runs of protocol P in γ. The
runs of this system embody all of the properties of the context, as they arise in runs of P . This includes, for example,
any timing assumptions, possible values encountered, possible topologies of the network, etc. They also embody therelevant properties of the protocol, because in all runs considered possible the agents follow P .
In this paper, we do not define protocols and contexts.
Rather, we treat the KoP in a slightly simpler and more
abstract setting. We say that an action α is a conscious
action for i in R if doesi (α) is an i-local fact in R, so thatwhenever (R, r, t) |= doesi (α) holds, (R, r, t) |= Ki doesi (α)
holds as well. In other words, the fact that α is a conscious
action for i in R implies that if α is ever performed at apoint of R in which i’s local state is `i , then α must be
performed whenever i’s state is `i . Conscious actions arequite prevalent in many systems of interest. For example,
suppose that agent i follows a deterministic protocol, so thatits action at any given point is a function of its local state.
If, in addition, agent i is allowed to move at every time step,then all of its actions are conscious actions.
We are now ready to prove a formal version of the KoP:
Theorem 3.1 (The KoP Principle). Let α be a conscious action for i in R. If ψ is a necessary condition for
doesi (α) in R, then Ki ψ is also a necessary condition for
doesi (α) in R.
Proof. We will show the contrapositive. Let α be a conscious action for i in R, and assume that Ki ψ is not a necessary condition for doesi (α) in R. Namely, there exists a
point (r, t) ∈ Pts(R) such that both (R, r, t) |= doesi (α) and(R, r, t) 6|= Ki ψ. Given the latter, we have by the definitionof ‘|=’ for Ki that there exists a point (r0 , t0 ) ∈ Pts(R) suchthat both (r0 , t0 ) ≈i (r, t) and (R, r0 , t0 ) 6|= ψ. Since α is a
conscious action for i in R and (R, r, t) |= doesi (α) we havethat (R, r, t) |= Ki doesi (α). It follows from (r0 , t0 ) ≈i (r, t)
by the definition of ‘|=’ for Ki that (R, r0 , t0 ) |= doesi (α)
holds. But since (R, r0 , t0 ) 6|= ψ, we conclude that ψ is nota necessary condition for doesi (α) in R, establishing thecountrapositive claim.
Theorem 3.1 applies to all multi-agent systems. It immediately implies, for example, that Katm (good_credit) isa necessary condition for dispensing cash. The theorem ismodel independent; it does not depend on timing assumptions, on the topology of the system (even on whether agents
communicate by message passing or via reading and writing
to registers in a shared memory), or on the nature of theactivity that is carried out. For every necessary condition
for a conscious action, knowing that the condition holds isalso a necessary condition.

4.

COORDINATING SIMULTANEOUS
ACTIONS

Recall that the language LK
n contains formulas in whichknowledge operators can be nested to arbitrary finite depth.
It is sometimes useful to consider a state of knowledge called
common knowledge that goes beyond any particular nested
formula. Intuitively, a fact ψ is common knowledge if everyone knowing that everyone knows . . . , that everyone knowsthe fact ψ, to every finite depth. Common knowledge has anumber of equivalent definitions, one of which is as follows:
Definition 4.1 (Common Knowledge). Fix a set of
agents G and a fact ψ. We denote by CG ψ the fact that
ψ is common knowledge to G. Its truth at points of asystem R is defined by:(R, r, t) |= CG ψ iff (R, r, t) |= Ki1 Ki2 · · · Kim ψ
for all hi1 , i2 , . . . , im i ∈ Gm ,and all m ≥ 1.
Common knowledge, a term coined by Lewis in [18], plays
an important role in the analysis of games [2], distributed
systems [13], and many other multi-agent settings. Clearly,
common knowledge is much stronger than “plain” knowledge. Indeed, CG ψ validly implies Kj ψ, for all agents j ∈ G.
Since common knowledge requires infinitely many facts tohold, it is not a priori obvious that CG ϕ can be attained ata reasonable cost, or even whether it can ever be attainedat all, in settings of interest (see [7, 11, 13]). We will now
show that there are natural applications for which attaining
common knowledge is essential.
Intuitively, distinct actions are simultaneous in R if they
can only be performed together; whenever one is performed,
all of them are performed simultaneously. It is possible todefine simultaneous coordination formally in terms of necessary conditions:

To prove this observation notice that, by assumption, both(a) R |= doesj (αj ) ⇒ doesi (αi ) and (b) R |= doesi (αi ) ⇒ ϕ
hold. For all (r, t) ∈ Pts(R), if (R, r, t) |= doesj (αj ) then(R, r, t) |= doesi (αi ) by (a) and so (R, r, t) |= ϕ by (b).
Thus, ϕ is a necessary condition for doesj (αj ) in R.
Assume that ψ is a necessary condition for doesi (αi ), forsome i ∈ G. We shall prove by induction on m ≥ 0 thatKi1 Ki2 · · · Kim ψ is a necessary condition for doesj (αj ) in R,
for every j ∈ G and all sequences hi1 , . . . , im i ∈ Gm (of magent names from G). This will establish that (R, r, t) |=doesj (αj ) implies (R, r, t) |= CG ψ for all (r, t) ∈ Pts(R),and thus CG ψ is a necessary condition for doesj (αj ) for allj ∈ G, as claimed.
• Base case: Let m = 0. The claim in this case is that ifψ is a necessary condition for doesi (αi ) then ψ is alsoa necessary condition for doesj (αj ). This is precisely
Observation 1, with ϕ , ψ.
• Inductive step: Let m ≥ 1, and assume that the claim
holds for all j 0 ∈ G and all sequences in Gm−1 . Fix
j ∈ G and a sequence hi1 , i2 , . . . , im i ∈ Gm . Its suffix
hi2 , . . . , im i is a sequence in Gm−1 . Thus, Ki2 · · · Kim ψ
is a necessary condition for doesi1 (αi1 ) by the inductive hypothesis for m − 1 (applied to Gm−1 and agent
j 0 = i1 ∈ G). Given that αi1 is a conscious action
by i1 , we can apply Theorem 3.1 to the necessary condition Ki2 · · · Kim ψ and obtain that Ki1 Ki2 · · · Kim ψ
is a necessary condition for doesi1 (αi1 ). By Observation 1 we have that Ki1 Ki2 · · · Kim ψ is also a necessary
condition for doesj (αj ) in R, and we are done.

Definition 4.2 (Simultaneous Actions). Let G be a
set of agents. We say that a set of actions A = {αi }i∈G is(necessarily) simultaneous in R if doesi (αi ) is a necessary
condition for doesj (αj ) in R, for all i, j ∈ G.
Suppose that the actions in A are simultaneous in R in theabove sense. Then the KoP immediately implies (by Theorem 3.1) that a necessary condition for performing an action
in A is knowing that the other actions are also (currently)
being performed. In fact, however, much more must be true.
We now present a strong variant of the KoP, which showsthat in order to perform simultaneous actions agents must
attain common knowledge of their necessary conditions. Notice that in order to allow a set of actions by the agentsin G to be simultaneous, the system R must be sufficiently
deterministic to ensure that if i, j ∈ G are distinct agentsand (R, r, t) |= doesi (α) holds, then j will be scheduled toperform an action at (r, t). For otherwise, there would be
no way to ensure simultaneous execution of the actions by
the agents in G. Conscious actions fit this setting well inthis case. We proceed as follows.
Theorem 4.3 (C-K of Preconditions). Let G be a
set of agents and let A = {αi }i∈G be a set of necessarily
simultaneous actions in the system R. Moreover, supposethat each action αi ∈ A is a conscious action for its agent iin R. If ψ is a necessary condition for doesi (αi ) for somei ∈ G, then CG ψ is a necessary condition for doesj (αj ), forall j ∈ G.

Proof. Assume that A is a set of necessarily simultaneous actions for G in R. It is straightforward to show thefollowing claim
Observation 1. Let αi , αj ∈ A be the actions for agents iand j, respectively. If a fact ϕ is a necessary condition
for doesi (αi ) in R then ϕ is also a necessary condition for
doesj (αj ) in R.

4.1

Common Knowledge and theFiring Squad Problem

As an illustration of the applicability of Theorem 4.3 to a
concrete application, consider a simple version of the Firing
Squad problem. In this instance, the set of agents G in
the system must simultaneously perform an action (say eachagent i ∈ G should perform the action firei ) in responseto the receipt, by any agent in G, of a particular external
input called a ‘go’ message. The firei action can stand for asimultaneous change in shared copies of a database, a publicannouncement at different sites of the system, or any otheractions that need to take place simultaneously. Moreover,firei actions are allowed only if they are preceded by sucha go message. For simplicity, we consider a case in which
none of the agents in G may fail, and they all must satisfy
the specification.
Let ψgo be a proposition that is true at (r, t) ∈ Pts(R) if ago message is received by any of the agents in G at a point(r, t0 ) of r at a time t0 ≤ t. According to the specification of
the Firing Squad problem, ψgo is a necessary condition for
the firei actions. An immediate consequence of Theorem 4.3
is:Corollary 4.4. CG ψgo is a necessary condition for all
firei actions in the Firing Squad problem.
Given Corollary 4.4, any solution to the firing squad problem
must first attain common knowledge that a go message has
been received. It is well-known (see [10, 13]) that common
knowledge of a fact is observed simultaneously at all agentsit involves. Suppose that every i ∈ G performs firei when212CG ψgo first holds. Since all agents in G will come to knowthat CG ψgo immediately, they will fire simultaneously, as
required by the problem specification. Indeed, Theorem 4.3
shows that this is the first time at which they can perform
according to a correct protocol. Implementing simultaneoustasks such as the Firing Squad therefore inherently involves,and often reduces to, ensuring and detecting CG ψgo . Recall
that depending on the properties of the system, attaining
such common knowledge might be impossible in some cases,or it might incur a substantial cost in others. Just as in thecase of the KoP, this necessity is not due to our formalism.
It is only exposed by our analysis. In every protocol that
implements such a task correctly, the firing actions cannot
be performed unless CG ψgo is attained.
There is an extensive literature on using common knowledge to obtain optimal protocols for simultaneous tasks [8,9, 16, 20, 21, 23, 24, 25, 26]. Typically, they involve an explicit proof that common knowledge of a particular fact isa necessary condition for performing a set A of necessarily
simultaneous actions. Theorem 4.3 or a variant of it suitedfor fault-tolerant systems can be used to establish this resultin all of these cases. Moreover, one of the main insights fromthe analysis of [13] and of [10] is that when simultaneous actions are performed, the participating agents have common
knowledge that they are being performed. Theorem 4.3 is astrict generalization of this fact.

5. TEMPORALLY ORDERING ACTIONS
So far, we have seen two essential connections between
knowledge and coordinated action: performing actions requires knowledge of their necessary conditions, and performing simultaneous actions requires common knowledgeof their necessary conditions. We now further extend theconnection between states of knowledge and coordination,by showing that temporally ordering actions depends on attaining nested knowledge of necessary conditions. Following [5], we define temporally ordered actions:Definition 5.1 (Ben Zvi and Moses). A sequence ofactions hα1 , . . . , αk i (for agents 1, . . . , k, respectively) is(linearly) ordered in R if didj−1 (αj−1 ) is a necessary condition for doesj (αj ) in R.
Observe that this definition does not force an action αj tooccur in a run in which αj−1 occurs. Rather, if an action αjis performed in a given run, then it must be preceded by
all actions α1 , . . . , αj−1 . Moreover, if we denote the time atwhich an action αi is performed in a run r by ti , then we
require that tj−1 ≤ tj for every action αj performed in r.

Claim 1. Assume that the sequence hα1 , . . . , αki is ordered in R. Then R |= didj (αj ) ⇒ didj−1 (αj−1 ) for all
2 ≤ j ≤ k.
Proof. Assume that (R, r, t) |= didj (αj ). Then, by definition of didj (αj ), (R, r, t̂ ) |= doesj (αj ) for some t̂ ≤ t.
Since hα1 , . . . , αk i is ordered in R implies that didj−1 (αj−1 )is a necessary condition for doesj (αj ) in the system R, andso (R, r, t̂ ) |= didj−1 (αj−1 ). Since didj−1 (αj−1 ) is a stable
fact and t ≥ t̂, we obtain that (R, r, t) |= didj−1 (αj−1 ). The
claim follows.

We say that a fact ϕ is stable in R if once true, ϕ remains
true. Formally, if (R, r, t) |= ϕ and t0 > t then (R, r, t0 ) |= ϕ,for all r ∈ R and t, t0 ≥ 0. Notice that while doesi (α) is, ingeneral, not a stable fact, didi (α) is always stable.

213

Definition 5.2. We say that agent i recalls ψ in R if
the fact Ki ψ is stable in R.
The notion of perfect recall, capturing the assumption thatagents remember all events that they take part in, is popularin the analysis of games and multi-agent systems [11, 29].
While perfect recall is a nontrivial assumption often requiring significant storage costs, selective recall of single facts
such as doesj (αj ) is a much weaker assumption, that canbe assumed of a system R essentially without loss of generality. By adding a single bit to Agent j’s local state, whose
value is 0 as long as j has not performed αj and 1 once theaction has been performed, we can obtain a system R0 thatis isomorphic to R, in which Agent j recalls doesj (αj ).
Claim 2. Assume that αj is a conscious action for j
in R, and that j recalls didj (αj ) in R. Then didj (αj ) isa j-local fact in R.
Proof. Suppose that (R, r, t) |= didj (αj ). Then, by definition of didj (αj ), we have (R, r, t̂ ) |= doesj (αj ) for somet̂ ≤ t. Choose an arbitrary (r0 , t0 ) ∈ Pts(R) satisfying that(r0 , t0 ) ≈j (r, t̂ ). Then (R, r0 , t0 ) |= doesj (αj ) since αj is a
conscious action for j in R. By definition of didj (αj ) it follows that (R, r0 , t0 ) |= didj (αj ). Now, by definition of |= for
Kj we have that (R, r, t̂ ) |= Kj didj (αj ). By assumption,
j recalls didj (αj ) in R, and so Kj didj (αj ) is stable in R.
Thus, since t ≥ t̂, we obtain that (R, r, t) |= Kj didj (αj ), as
claimed.
We can now show:
Theorem 5.3
Assume that

(Ordering and Nested Knowledge).

• the actions hα1 , . . . , αk i are ordered in R,• each agent j = 1, . . . , k recalls didj (αj ) in R,• αj is a conscious action for j in R, for all
j = 1, . . . , k, and• ψ is a stable necessary condition for the firstaction does1 (α1 ) in R
Then Kj Kj−1 · · · K1 ψ is a necessary condition forthe j th action doesj (αj ) in R, for all j ≤ k.
Proof. Assuming the conditions of the theorem, we will
prove by induction on j ≤ k that didj (αj ) validly implies
Kj Kj−1 · · · K1 ψ in R. Since doesj (αj ) validly implies didj (αj )by definition of didj (αj ), this will yield that Kj Kj−1 · · · K1 ψ
is a necessary condition for doesj (αj ) in R, as claimed. We
proceed with the inductive argument.
• Base case j = 1: Assume that (R, r, t) |= did1 (α1 ).
By Claim 2 we have that (R, r, t) |= K1 did1 (α1 ). Let(r0 , t0 ) ∈ Pts(R) be an arbitrary point satisfying that(r0 , t0 ) ≈1 (r, t). Then (R, r0 , t0 ) |= did1 (α1 ) by theknowledge property. Thus, (R, r0 , t̂ ) |= does1 (α1 ) holds
for some t̂ ≤ t0 , and because ψ is a necessary condition for does1 (α1 ) in R, we obtain that (R, r, t̂ ) |= ψ.
Since ψ is stable and t0 ≥ t̂, we have that (R, r0 , t0 ) |= ψ.
By choice of (r0 , t0 ) we have that (R, r, t) |= K1 ψ, asclaimed.

• Inductive step: Let j > 1 and assume that Kj−1 · · · K1 ψ
is a necessary condition in R for didj−1 (αj−1 ). Moreover, let (R, r, t) |= didj (αj ). Since αj is a conscious
action for j, Claim 2 implies that (R, r, t) |= Kj didj (αj ).
Choose an arbitrary (r0 , t0 ) ∈ Pts(R) satisfying that(r0 , t0 ) ≈j (r, t). By definition of Kj , it follows that(R, r0 , t0 ) |= didj (αj ). By Claim 1, since the sequencehα1 , . . . , αk i is ordered in R and j > 1 we have that(R, r0 , t0 ) |= didj−1 (αj−1 ). We now apply the inductive hypothesis to obtain that (R, r0 , t0 ) |= Kj−1 · · · K1 ψ.
Finally, we obtain that (R, r, t) |= Kj Kj−1 · · · K1 ψ bychoice of (r0 , t0 ) and the definition of ‘|=’ for Kj . The
claim now follows.
A slightly more restricted version of Theorem 5.3 wasproved in [5]. Rather than consider an arbitrary necessary condition for α1 , they proved a version for the casein which the first action α1 is triggered by an external input
to agent 1. Technically, the proofs are quite similar.
Theorem 5.3 provides a necessary, but possibly not sufficient, condition for ordering actions in distributed systems.
If agent j acts strictly later than when Kj Kj−1 · · · K1 ψ
first holds, then it may be inappropriate for agent j + 1
to act when it knows that the fact Kj Kj−1 · · · K1 ψ holds(i.e., when Kj+1 Kj · · · K1 ψ first holds). Nevertheless, Theorem 5.3 is often very useful because it can be used as a
guide for efficiently, and sometimes even optimally, performing a sequence of ordered actions. Intuitively, suppose that
we have a protocol whose goal is to perform hα1 , . . . , αk i
in response to an externally generated trigger ψ (such as
the ‘go’ message in Firing Squad). In particular, assume
that ψ is a necessary condition for α1 . Keeping the communication aspects of this protocol fixed, an optimally fastsolution would be for each agent j ≤ k to perform αj when
Kj Kj−1 · · · K1 ψ first holds. Let R be the set of runs ofsuch a protocol with r ∈ R, and let tj and tj−1 be theearliest times at which (R, r, tj ) |= Kj Kj−1 · · · K1 ψ and(R, r, tj−1 ) |= Kj−1 · · · K1 ψ hold in a run r, respectively.
The knowledge property guarantees that Kj Kj−1 · · · K1 ψvalidly implies that Kj−1 · · · K1 ψ in R, and so tj ≥ tj−1 .
Since, by assumption, αj is performed at time tj and αj−1
at tj−1 , we have that agents perform actions in linear temporal order, as required by Definition 5.1. Clearly, none of
the actions can be performed any earlier, as Theorem 5.3
shows. We conclude that in time-efficient protocols, thenested knowledge formula presented by the theorem can beboth necessary and sufficient. In this sense, Theorem 5.3
suggests a recipe for obtaining time-efficient solutions forordering actions.
Just as Theorem 4.3 implies that common knowledge is anecessary condition for simultaneous actions, we now haveby Theorem 5.3 that nested knowledge is a necessary condition for performing actions in linear temporal order. And
just as there is an established literature on when common
knowledge is and is not attainable and on how it may arise,
there are results concerning the communication structurethat underlies attaining nested knowledge. Indeed, in aseminal paper [7], Chandy and Misra showed that in asynchronous systems R, if (R, r, t) |= ¬ϕ and at a time t0 > t(R, r, t0 ) |= Kj Kj−1 · · · K1 ϕ, then there must be a message
chain in the run r between times t and t0 , passing throughthe agents 1,2,. . . ,j in this order (possibly involving additional agents as well). Given Theorem 5.3, this implies thatthe only way to coordinate actions in a linear temporal order in an asynchronous setting is by way of such message
chains.2
More recently, Ben Zvi and Moses extended Chandy andMisra’s work to systems in which communication is not asynchronous, but rather agents may have access to clocks andthe transmission time for each of the channels is bounded[5]. They show that a communication structure called a centipede must be constructed in order to obtain nested knowledge of spontaneous facts such as the arrival of an external input. They prove a slightly more restricted instanceof Theorem 5.3 (without using KoP directly), and use it
to show that ordering actions in their setting requires theconstruction of the appropriate centipedes. Finally, Parikh
and Krasucki analyze the ability to create levels of knowledge consisting of collections of nested knowledge formulasin [27]. Theorem 5.3 relates levels of knowledge to coordination.

6.

DISCUSSION

This paper formulated the knowledge of preconditions principle and presented three theorems relating knowledge andcoordinated action: the first is the KoP itself—necessary
conditions for an action must be known to hold when theaction is performed. Next, we showed that necessary conditions for simultaneous actions must be commonly known
when the actions are taken. Finally, nested knowledge isa necessary condition for coordinating linearly ordered actions. The latter two are fairly direct consequences of the
KoP. We discussed some of the uses of the latter two results in Sections 4 and 5. Indeed the KoP has many further
implications.
In recent years, several works that make use of KoP haveappeared, citing the unpublished [22]. For example, Castañeda, Gonczarowski and Moses used the KoP to analyze
the consensus problem [6], in which agents need to agree
on a binary value in a fault-prone system. They designed a
protocol in two steps—applying the KoP once to derive arule by which, roughly, agents decide on 0 when they knowof an initial value of 0. Then, they applied the KoP again
assuming this rule for decisions on 0, and obtained a rule involving nested knowledge (roughly, a statement of the form“knowing that nobody knows 0”) for deciding on a value of 1.
The result of their analysis was a very efficient solution toconsensus that is optimal in a strong sense: It is the first unbeatable consensus protocol. The work of [6] complements
an earlier work by Halpern, Moses and Waarts [15], in which
a fixed point analysis of optimal consensus was obtained. It,
too, is closely related to the KoP.
Gonczarowski and Moses used the KoP to analyze theepistemic requirements of more general forms of coordination [12]. Namely, they considered a setting in which kagents need to perform actions, and there are time bounds
on the relative times at which the actions of any pair ofagents is performed. The simple instance in which all bounds
are 0 is precisely that of the simultaneous actions consideredin Section 4. They show that such coordination requires vectorial fixed points of knowledge conditions, which are natu2
Theorems 3.1 and 5.3 depend on conscious actions andtherefore do not apply to asynchronous systems. Nevertheless, variants of these theorems can be presented that do
apply to asynchronous systems and nondeterministic protocols. Details will appear in [22].

214

rally related to fixed points and equilibria. The papers [3, 4,
5, 12] together can all be viewed as making use of the KoP
to provide insights into the interaction between time andcommunication for coordinating actions in a distributed andmulti-agent system. Describing them is beyond the scope ofthe current paper.
The most significant aspect of the KoP, in our view, isthe fact that it places a new emphasis on the epistemic aspects of problem solving in a multi-agent system. Simple
necessary conditions induce epistemic conditions. Thus, inorder to act correctly, one needs a mechanism ensuring thatthe agents obtain the necessary knowledge, and that they
discover that they have this knowledge. Most problems andsolutions are not posed or described in this fashion. We believe that the KoP encapsulates an important connection
between knowledge, action and coordination that will findmany applications in the future.

7. REFERENCES[1] H. Attiya and J. Welch. Distributed Computing:Fundamentals, Simulations and Advanced Topics.
John Wiley & Sons, 2004.
[2] R. J. Aumann. Agreeing to disagree. Annals of
Statistics, 4(6):1236–1239, 1976.
[3] I. Ben-Zvi and Y. Moses. Agent-time epistemics andcoordination. In Proceedings of the 5th ICLA, pages
97–108, 2013.
[4] I. Ben-Zvi and Y. Moses. The shape of distributed
coordination. In Proceedings of TARK XIV, pages
29–38, 2013.
[5] I. Ben-Zvi and Y. Moses. Beyond lamport’s
Happened-before: On time bounds and the ordering of
events in distributed systems. Journal of the ACM,
61(2):13, 2014.
[6] A. Castañeda, Y. A. Gonczarowski, and Y. Moses.
Unbeatable consensus. In Proceedings of DISC, pages
91–106. Springer, 2014.
[7] K. M. Chandy and J. Misra. How processes learn.
Distributed Computing, 1(1):40–52, 1986.
[8] D. Dolev, E. N. Hoch, and Y. Moses. An optimal
self-stabilizing firing squad. SIAM Journal on
Computing, 41(2):415–435, 2012.
[9] C. Dwork and Y. Moses. Knowledge and common
knowledge in a Byzantine environment: crash failures.
Information and Computation, 88(2):156–186, 1990.
[10] R. Fagin, J. Y. Halpern, Y. Moses, and M. Y. Vardi.
Common knowledge revisited. In proceedings of TARK
VI, pages 283–298, 1996.
[11] R. Fagin, J. Y. Halpern, Y. Moses, and M. Y. Vardi.
Reasoning about Knowledge. MIT Press, 2003.
[12] Y. Gonczarowski and Y. Moses. Timely common
knowledge: Characterising asymmetric distributed
coordination via vectorial fixed points. In Proceedings
TARK XIV, pages 79–93, 2013.
[13] J. Y. Halpern and Y. Moses. Knowledge and common
knowledge in a distributed environment. Journal of
the ACM, 37(3):549–587, 1990. A preliminary version
appeared in Proceedings of the 3rd ACM PODC, 1984.
[14] J. Y. Halpern and Y. Moses. A guide to completenessand complexity for modal logics of knowledge andbelief. Artificial Intelligence, 54:319–379, 1992.

215[15] J. Y. Halpern, Y. Moses, and O. Waarts. A
characterization of eventual Byzantine agreement. In
Proceedings of the 9th ACM PODC, pages 333–346,
1990.
[16] M. P. Herlihy, Y. Moses, and M. R. Tuttle.
Transforming worst-case optimal solutions for
simultaneous tasks into all-case optimal solutions. In
Proceedings of the 30th ACM PODC, pages 231–238,
2011.
[17] G. Le Lann. Distributed systems-towards a formal
approach. In IFIP Congress, volume 7, pages 155–160.
Toronto, 1977.
[18] D. Lewis. Convention, A Philosophical Study. Harvard
University Press, 1969.
[19] J. McCarthy and P. J. Hayes. Some philosophicalproblems from the standpoint of artificial intelligence.
In D. Michie, editor, Machine Intelligence 4, pages
463–502. Edinburgh University Press, Edinburgh,
1969.
[20] T. Mizrahi and Y. Moses. Continuous consensus via
common knowledge. Distributed Computing,20(5):305–321, 2008.
[21] T. Mizrahi and Y. Moses. Continuous consensus with
ambiguous failures. In Distributed Computing and
Networking, pages 73–85. Springer, 2008.
[22] Y. Moses. Knowledge and Coordinated Action, toappear, 2016.
[23] Y. Moses and M. R. Tuttle. Programming
simultaneous actions using common knowledge.
Algorithmica, 3:121–169, 1988.
[24] G. Neiger. Consistent coordination and continual
common knowledge. Manuscript. 1990.
[25] G. Neiger and R. Bazzi. Using knowledge to optimallyachieve coordination in distributed systems. In
Proceedings of TARK IV pages 43–59.
[26] G. Neiger and M. R. Tuttle. Common knowledge andconsistent simultaneous coordination. Distributed
Computing, 6(3):334–352, 1993.
[27] R. Parikh and P. Krasucki. Levels of knowledge indistributed computing. Sādhanā, 17(1):167–191, 1992.
[28] F. B. Schneider. Implementing fault-tolerant services
using the state machine approach: A tutorial. ACM
Computing Surveys (CSUR), 22(4):299–319, 1990.
[29] R. Selten. Reexamination of the perfectness concept
for equilibrium points in extensive games.
International journal of game theory, 4(1):25–55, 1975.

Learning from unrealized versus realized prices
Abstract
Kathleen Ngangoué

Georg Weizsäcker

DIW Berlin
Mohrenstr. 58
10117 Berlin

Humboldt Universität zu Berlin and DIW Berlin
Spandauer Str. 1
10178 Berlinkngangoue@diw.de

weizsaecker@hu-berlin.de

ABSTRACT
Market prices reflect information about an asset’s fundamental value. However, it is still not clear the extent towhich traders are able to utilize this information. In particular, the market design in itself could affect inferences.
The psychology literature provides general evidence that thestrategy used in decision-making is contingent on the task’s
complexity. The way markets differ in their complexity may
influence traders’ decisions. Some markets, for instance, require traders to condition their strategy on a future price,some do not. Although this framing variation in market
design should be irrelevant to rational traders, trading optimally when the price is unknown is cumbersome: tradersmust, in that case, condition their demand strategy on afuture price, whose implicit information needs to be anticipated. In an experimental market with diverse information,
we investigate the extent to which traders update on theprice, in situations where orders are submitted before or after the price has realized.
We compare investors’ behavior in three market settings:
a simultaneous limit order market, a simultaneous price list
treatment and a sequential market. All treatments havein common that the price reflects private information of another market participant. In the limit order (LO) treatment,
participants observe their private signal and state their maximum willingness to pay by placing a limit order. In theprice list (PL) treatment the task is discretized: participants
choose between buying and selling for a given list of possible
price values. Both treatments, LO and PL, are considered assimultaneous because the market price realizes after participants submit their order. In the sequential (S) treatment,
participants observe the market price before deciding to buy
or to sell. Theoretically, decision outcomes should be identical at least in the treatments PL and S, where the strategy
sets are isomorphic.
We find that subjects react strongly to their private information, but not to the price, in both simultaneous treatments LO and PL. In that sense, subjects appear to form
naive beliefs. However, switching from a simultaneous system (LO, PL) to a sequential mechanism (S) improves inferences considerably. When the price is known at the timeof bid submission, bids react to prices, to an extent that isroughly consistent with Bayesian updating. Surprisingly, inthat case, subjects sometimes even overweight the information contained in the price. Hence, market designs affect
how well agents process information and, therefore, how efficient prices become.

217

218

Continuous, lexicographic context dependence in a binary
choice setting˚

Patrick O’Callaghan:

ABSTRACT
When changes in environmental factors lead to a change inpreference for one alternative over another, preference is context dependent. Continuous context dependence arises whenappropriate perturbations of context preserve strict preference. For the binary choice setting, we characterise continuous context dependence via a function that is a utility ateach context. We provide two examples of lexicographic context dependence that delineate the scope of the model. The
first concerns reference dependence in a medical setting; thesecond considers a financial setting with unawareness anduncertainty.
Keywords
binary choice, context effects, lexicographic order,
preferences, utility

1. INTRODUCTION
Cognitive psychology describes a context effect as the influence of environmental factors on our perceptions and responses. In the present paper, we formalise the effect ofcontext on decision making via a context parameter. A decision maker’s preferences are context dependent if strict
preference for one alternative over another varies with context.
Context dependence of preference is supported by neuroscience which reveals a strong distinction between the intransigent nature of the structural networks of the brain
on the one hand and the dynamic nature of its operational
subnetworks on the other [15]. The presence of the latter isunderstood to highlight context dependence as an inherent
property of cognitive brain activity.
Continuous dependence on context, or robustness of strict
preference to perturbations in the context is a basic requirement. This property is argued to be crucial to brain activity
by von Neumann in his unfinished work “The Computer andthe Brain” [19]. This assumption is also present in manymodels of context dependence. Consider reference dependence in Kőszegy and Rabin [9]; menu dependence in Pattanaik and Xu [16]; status quo dependence in Masatlioglu
and Ok [12] and Sagi [18]; memory dependence in Gilboa andSchmeidler [5]; belief dependence in Gilboa and Schmeidler[4].
˚

I thank Jeff Kline and John Quiggin for useful commentsthat helped to improve this paper.
:
Department of Economics, University of Queensland, p.
ocallaghan@uq.edu.au .

219

In this paper we characterise continuous context dependence for the two alternative (binary choice) case. We show
that, provided the set of contexts satisfies certain topologicalconditions, asymmetry of strict preference at each context
and continuous context dependence are together sufficientfor the existence of a function that is: firstly, a utility representation on alternatives at each context; and secondly,
continuous on the set of contexts.
This representation theorem is a simplification of the main
theorem of O’Callaghan [14]. In particular, since there are
only two alternatives, we are able to provide a more basic
and constructive proof that appeals to Urysohn’s lemma instead of Michael’s selection theorem. Similar to that paper,
the topological conditions on the context space that we identify are just enough to yield the desired result. This allows
us to study lexicographic context dependence.
Lexicographic context dependence arises when the set ofcontexts is lexicographically ordered independently of preferences on alternatives. The first example we consider in thepresent paper is a medical setting where the decision maker(henceforth Val) faces a choice between two medical procedures (alternatives) for her child. Similar to Kőszegy andRabin [9], the contexts are reference points, though we assume that the reference points belong to a lexicographicallyordered two dimensional set. The dominant dimension is
the probability of survival and the other is a wealth related
dummy variable (with values zero or one). Mathematically,
this space is known as “double-arrow space” or “the split
interval”.
The second example is to a financial setting, where Val isa trader that must choose whether to buy or sell a certain
financial derivative. Val may be both unaware and uncertain of the states of the world. Following Heifetz, Meier,and Schipper [7], we assume that unawareness is indexed
by a complete lattice. The derivatives that Val trades are
more profitable when the uncertainty is higher according
to some measure of entropy. In contrast, we suppose that
Val always prefers to be more aware. Indeed, because eachunawareness level is associated with richer information, we
suppose that Val lexicographically prefers higher awarenessto higher uncertainty. When the cardinality of the set thatindexes unawareness is uncountable and well-ordered, theseassumptions yield a context space that is homeomorphic to
the “extended long line”. This connection allows us to identify preferences that satisfy continuous context dependence
but have no continuous representation.
This latter example highlights the possibility that, when
the set of contexts is too general, even though every repre-

sentation of preferences is discontinuous on contexts, preferences are continuous. We deem this to be a modellingproblem that is best avoided. For instance, if the set ofawareness levels is assumed countable, this issue does notarise.

2.

MODEL

Preferences at each context..

Let X denote a nonempty set. For each x in X, let ăx denote a binary relation on the set A “ ta, bu. The canonicalinterpretation of a ăx b is that it formalises the statement“at context x, alternative b is strictly preferred to alternative a”. Given the focus on binary choice a natural, alternative interpretation is that of stochastic choice. In that case,a ăx b holds provided the empirical probability b is chosen issignificantly different from 21 in the forced choice setting ofFechner [3]. Our model accommodates either interpretation.

Context preferences..

The collection tăx ux PX of binary relations on A is theprimitive object we refer to as context preferences or justpreferences. Context preferences exclude any preference statements that Val may in fact be in a position to make regarding
pairs of contexts, or indeed between one alternative-context
pair and another. This information is intentionally ignored
so that no assumption need be made concerning preferences
over such objects.1 Thus although our canonical interpretation is that of multiple epistemological viewpoints, one foreach context, this is not a formal requirement.

Representing context dependence.
By a representation of context preferences, we mean afunction of the form U : A ˆ X Ñ R, where for any context
x P X, and any i, j P A, j is strictly preferred to i at x if andonly if U pi, xq ă U pj, xq. That is, at each context x, there
exists a utility function U p¨, xq that represents preferencesin the classical sense. When X is a singleton, preferences
are context-free.

Continuous context dependence.

Context dependence is continuous at x if for each i, j P A,
j is strictly preferred to i at x, then the set of contexts X
contains an open neighbourhood N of x such that for any
other y in N , j is also strictly preferred to i at y. The factthat N is open means that the “direction of perturbation” isirrelevant, the important thing is that it is sufficiently small.

Characterising continuous context dependence.

U : A ˆ X Ñ R characterises continuous context dependence provided the function U pi, ¨q is continuous for eachi P A. Recall that this means that, for each i, the set
tx : U pi, xq ă ru is open for every r P R.

The set of contexts..
A set X is said to be a perfectly normal topological space
if it satisfies the following three conditions.

10. For every x, the singleton set txu is closed.

In the language of measurement theory (see d’Aspremontand Gevers [2] for a recent survey) context preferences are
low in the information hierarchy.

1. Every pair of nonempty, closed and disjoint sets E andF , there exists a pair of open disjoint sets G and H
such that E Ď G and F Ď H.
2. Every closed set is can be written as a countable intersection of open sets.
If a set X satisfies 0, it is a T0 topological space. The
condition that ensures X is normal is 1, whilst 2 ensures X
is perfect.
Example 1. A familiar metrizable set is the usual set of
nonnegative real numbers R` . Let the set of contexts X be
the cartesian product of the set N of nonnegative integers
with the half open interval of real numbers r0, 1q. Then R`
is topologically indistinguishable from X “ N ˆ r0, 1q provided we endow the latter with the topology generated by theintervals of lexicographic order ă˚ : k ˆr ă˚ l ˆt if and only
if k ă l or [k “ l and r ă t], where k, l P N and r, t P r0, 1q.
Since R` is metrizable, so is X.2
Recall that N is the smallest well-ordered infinite set. Its
supremum is the first infinite ordinal number, and is often
denoted by ω, and so we may write N ” r0, ωq. An extensionof this is the well-ordered set r0, ω1 q of all countable ordinal
numbers. ω1 denotes the first uncountable ordinal. With
the above lexicographic order ă˚ and corresponding topology,
the product X 1 “ r0, ω1 q ˆ r0, 1q is perfectly normal, but not
metrizable and is known as the “long line”. As such, X 1 isa valid example of a context space for the purposes of themain theorem.
Extending yet further, the extended long line “glues” thepoint ω1 ˆ 0 to the upper end of the long line to obtain theset X 2 . (This is similar to considering the one-point compactification of R` that yields the nonnegative extended realline). It is not hard to show the X 2 is normal, but the following argument demonstrates that it is not perfect.
Let tGn : n P Nu be an arbitrary collection of open neighbourhoods of ω1 ˆ 0 in X 2 . Then, by the definition of neighbourhood, each Gn contains an open ă˚ -interval of the form
pxn , ω1 ˆ 0s for each n. For each k ă ω1 , there are uncountably many l ă ω1 such that k ă l. Let kn satisfy xn ă
kn ˆ 1 ă ω1 ˆ 0 for each n. Then the set K “ tkn : n P Nu
is countable. By MunkresŞ[13, Theorem 10.3], K has an upper` bound k̄ ă‰ ω1 . Then tGn : n P Nu ‰ tω1 ˆ 0u becausek̄ ˆ 1, ω1 ˆ 0 Ă Gn for every n.
Note that in example 1, the order ă˚ is over contexts
X. The same is true of the examples in subsection 4.1 and4.2. It is important to note, however, that ă˚ serves only
to generate interesting and familiar examples of topological spaces. In contrast, the family of context preference
relations tăx ux PX is the kernel of our model. Indeed, thefollowing are the axioms on tăx ux PX that we seek to characterise.
Axioms. For a topological space X, we will assume:
Asy. for every x P X and every i, j P A, if i ăx j, then not
j ăx i ; andCD. for each i, j P A, tx P X : i ăx ju is open in X .
We adopt the product notation x “ k ˆ r of Munkres [13]for any element x P X.
2

220

For each x such that neither a ăx b nor b ăx a, we writea „x b. By construction therefore, „x is symmetric: a „x b
if and only if b „x a. Similarly, the weak preference relation
Àx that equals ăx Y „x is complete on A. (Thus, for eachx, either a Àx b or b Àx a.)(Asy) is commonly referred to as asymmetry. An immediate consequence of (Asy) is that „x is reflexive: i „x i
for i P A. Since |A| “ 2, „x and ăx are clearly transitive.
(Recall that „x is transitive provided that, for any i, j, k in
A, i „x j „x k implies i „x k.) This ensures that, for eachx, Àx is both complete and transitive.
Continuous (context) dependence (CD), which was motivated in the introduction and discussed further below, characterises the stability of strict preference for one alternative
over another. Crucially, this is stability with respect to a
perturbation of the context. Together with (Asy), (CD)ensures that the sets tx : a ăx bu and tx : b ăx au are separated sets, where tx : a „x bu is the set that separates them.
This means that they are not only disjoint, but neither contains a limit point of the other. The remaining consequences
are summarised in theorem 2 which now follows.

cannot be that U pj, xq ă U pi, xq. Since x was arbitrarily
chosen, 1 is sufficient for (Asy).
Suppose that U also satisfies condition 2 of the theorem.
Since the difference of two continuous functions is continuous, U pi, ¨q ´ U pj, ¨q is a continuous function. This ensuresthat the set G “ tx : U pi, xq ´ U pj, xq ă 0u is open. By
condition 1, G “ tx : i ăx ju and (CD) holds.

3. MAIN RESULTThe main example of this subsection shows that the results apply to problems that cannot be modelled using either
a context-free utility function or the previous results in theliterature on jointly continuous utility representations. Extending it provides a concrete example where there is no
continuous representation of preferences that satisfy continuous context dependence at the end of the subsection.

Theorem 2. For A “ ta, bu and X perfectly normal, preferences satisfy (Asy) and (CD) if and only if there exists
U : A ˆ X Ñ R such that for every x P X,
1. U p¨, xq is a utility representation of ăx ; and2. U characterises continuous context dependence.

Remark 3. The proof of theorem 2 highlights the difficulties that arise when X fails to be perfectly normal. In theexample of subsection 4.2, we provide a concrete example
of preferences that fail to have a continuous representationeven though they satisfy (CD). This observation is true ofany context space that fails to be perfectly normal [14].

4.

APPLICATIONS

We now apply the model to an example of referencedependent decision making in a medical setting followed by
an example of unawareness in a financial setting.

4.1

Medical decision making

Contexts.

of 2. As usual, the sufficiency of the axioms is the main
step in the proof. We begin with this argument.

Sufficiency.

Suppose both the sets tx : a ăx bu and tx : b ăx au
are nonempty, the remaining cases follow by an identicalargument. By (Asy), these sets are disjoint. By (CD), thesesets are open and F “ tx : b Àx au is closed.
Since X is perfect and F is closed, thereŞ
exists a countablecollection tGn unPN of open sets satisfying 8
1 Gn “ F . Notethat X ´Gn Ď tx : a ăx bu, so that F and X ´Gn are closed
and disjoint for each n. Since X is normal, the Urysohn
lemma applies. For each n, this guarantees the existence ofa continuous function fn : X Ñ R such that fn pxq “ 0 on F
and fn pxqř“ 1 on X´Gn , and 0 ď fn pxq ď 1 otherwise. Now´n
let f “ 8fn . Since the uniform limit of continuous
1 2
functions is continuous, f is continuous. Moreover, f pxq ą 0
if and only if a ăx b. By the same argument, there exists
another continuous nonnegative function g : X Ñ R suchthat g ´1 p0q “ tx : a Àx bu.
Let U pa, xq “ 0 for each x P X and let$
if a ăx b,& f pxq´gpxq if b ăx a,
U pb, xq “
% 0otherwise.
The resulting function U : A ˆ X Ñ R satisfies conditions 1
and 2 of the theorem.

Necessity.

Suppose that U : A ˆ X Ñ R satisfies condition 1 of
the theorem and that i ‰ j P A and, for some x P X,
U pi, xq ă U pj, xq. Clearly, by asymmetry of ă on R, it
221Consider a setting where Val is the mother of a child witha life threatening, but curable illness. Each context is areference point in the sense of Kahneman and Tversky [8],but like Kőszegy and Rabin [9], the reference point is allowed to vary. Assume the set of reference points Val mightface is X “ p0, 1q ˆ t0, 1u. The first dimension 0 ă p ă 1
denotes the (objective) probability that no medical complications arise from the procedure. We suppose that p would
be provided by the child’s physician on the basis of historicalstatistics and that, in this respect, there is no difference between the private and public sectors. The second dimension
is a dummy variable that indicates whether Val is a homeowner (k “ 1) or not pk “ 0q.

Preferences.
Val is to choose whether or not she will go private (possibly taking out a loan to do so). The elements of A are
therefore a ““go public” and b ““go private” respectively.
The private sector provides a private room for her child andallows for arbitrarily long visiting hours. On the other handb incurs a fixed cost of $10, 000.3 There are no further (financial) costs relating to the complications that arise following
the procedure. Wishing to be at her child’s bedside in thecase of complications, Val will always go private if p ă 43 . If99
ă p, Val considers the risk of complications to be suffi100
99
ciently low to warrant going public. Let p “ 34 and p̄ “ 100.
For the remaining contexts tx P X : p¯ ď x1 ď p̄u, Val¯
3The setting we have in mind is the Australian healthcare
system, where going private entails paying a portion of thedoctors’ fees for those with private health insurance. In thatsystem, those with private health insurance may still choose
to go public.

is indecisive between the two alternatives.4 Thus, at eachcontext x P X, ăx satisfies (Asy).

Lexicographic ordering of contexts.

As in example 1, we adopt the product notation x “ p ˆ kfor elements of X. Since both dimensions of X are variables
over which Val has no control, they are treated as context
variables. Nonetheless, it may be reasonable to assume that
Val has a strict preference ordering ă˚ over X for contexts
x “ p ˆ k with higher values of p regardless of the value ofk. It may also reasonable to assume that, if x “ p ˆ k andy “ q ˆ l and p “ q, then x ă˚ y is better if k ă l.

Continuous reference dependence.
Consider the following collection of (open) order intervals
in the set of contexts X:((
x : y ă˚ x
andx : x ă˚ zsuch that y, z P X. The collection of open sets τX formed bytaking unions of finite intersections of such intervals generates the lexicographic order topology on X. Note that, sincety : p ˆ 0 ă˚ y ă˚ p ˆ 1u is empty, ty : p ˆ 0 ă˚ yu “ ty :¯
¯ set is both openp ˆ 1¯ ď˚ yu. This ensuresthat the latter¯and closed in X. By the same argument, the the same istrue of ty : y ď˚ p̄ ˆ 0u. Since the intersection of two open
sets is open, ty : p ˆ 1 ď˚ y ď˚ p̄ ˆ 0u is both open and¯closed in X. Moreover,this set can also be rewritten as´` ‰¯ ´“ ˘¯p, p̄ ˆ t0u Y p, p̄ ˆ t1u .5(1)¯¯
Lemma 4. The set tx : a „x bu is closed in X.

Proof. Since ty : a „x bu “ ty : p ˆ 1 ď˚ y ď˚ p̄ ˆ 0u Y tp ˆ¯ closed sets is closed, ¯the0u Y tp̄ ˆ 1u and the union of three
proof is complete.

Representation of preferences.

pX, τX q is a well known example of a perfectly normal
topological space [6]. Thus, theorem 2 and lemma 4 guarantees the existence of a family of utility representations, one
for each ăx such that x P X that is continuous on X.
The space X is not metrizable, and so the results of Levin[11] and Caterino, Ceppitelli, and Maccarino [1] do not apply. See O’Callaghan [14] for a discussion of the connections
with those results. As with many lexicographically orderedsets, pă˚ , Xq cannot be represented by any context-free utility function with values in R. This implies that Val’s preferences cannot be extended to a context-free binary relation
on A ˆ X that contains ă˚ and has a real-valued utility
representation.

4.2

Unawareness frames

Consider a setting where Val is a financial derivatives
trader who must choose between two alternatives in the faceof both unawareness and uncertainty regarding the state of
the world.
4
Being indecisive is observationally equivalent to being indifferent in this case.
5
This property has lead X Y t0, 1u2 to be called “doublearrow” space.

The context space.
In order to construct the set of contexts X, we consider alattice L of standard state spaces as in Heifetz, Meier, andSchipper [7]. Each state space S P L represents a different
level of awareness. We focus on the special case where thelattice order ďL on L is well-ordered: that is, every L1 Ď L
has a smallest element. Essentially, this restricts attention
to the single agent unawareness problem, for then every pairof elements S, S 1 P L are comparable.
Heifetz, Meier, and Schipper [7] assume that L is a complete lattice. That is, every L1 Ď L has a greatest lower
bound and a least upper bound according to ďL . As L is
well-ordered, if it has a greatest element [10], then it is acomplete lattice. Let ω1 be the smallest uncountable ordinal number of example 1 and recall that r0, ω1 q is the setof all countable ordinals and r0, ω1 s “ r0, ω1 q Y tω1 u. Thenr0, ω1 s is a complete lattice. Let 0 and ω1 index the minimal and maximal elements of L respectively, and let r0, ω1 s
and L be order isomorphic. This allows us to identify L andr0, ω1 s, and where necessary drop reference to Sk , Sl P L andsimply refer to k, l P L.
Let L denote the first dimension of the context space.
The second dimension represents different levels of uncertainty given the level of awareness. To keep the example toa minimum, we assume that each S P L is associated witha single σ-algebra ΣS on S that represents Val’s information. Moreover, we suppose that Val is only concerned agiven entropy ranking ďS of some subset D of the of set allof probability measures on ΣS . In particular, for p, q P D,
p ďS q means that the entropy of p is weakly lower than theentropy of q according to ďS . Recall that the entropy of ameasure evaluates the disorder associated with a measure,for instance, uniform distributions have maximum entropy,
and Dirac measures that assign full mass to any given point
have minimum entropy.
It is standard to associate entropy orders with some interval in R. We assume that pD, ďS q is order isomorphic
to the usual order ă on the half-open interval r0, 1q in R.
This implies that the minimum entropy is attained for someprobability measure in D, whereas the maximum entropy isnot.
Remark 5. Whilst the main reason for this assumption is
to simplify the exposition, examples where the maximum entropy does not exist are provided by [17]. In particular, forprobability measures with unbounded support, if the ratio of
the third (skewness) and fourth (kurtosis) moments of a distribution is is high the maximum entropy does not exist. Alternatively, in a setting where the distributions have bounded
support, and the uniform distribution has the maximum entropy, this assumption is equivalent to assuming Val always
has some useful information, and this moves her away fromthe maximum entropy.

Lexicographic ordering of contexts.
For certain trading strategies, uncertainty is a good thing.
This is where the trader profits from higher volatility. We
assume this is the case for Val. On the otherhand, because
lower unawareness can imply trading with insiders, we assume higher awareness is always better for Val. So as tohighlight the continuity issue that can arise in the simplest
possible terms, we assume that X “ r0, ω1 sˆr0, 1q, and that
Val lexicographically orders X as follows: if x “ k ˆ r and222

y “ l ˆ t, then x ď˚ y if and only if k ăL l or [k “ l andr ďS t]. As in the example of section 4.1, we consider theorder topology on X associated with ă˚ .

Preferences given context.
Let a=“sell” and b=“buy”. We assume that Val’s preferences on A “ ta, bu are such that a „x b if and only if
x “ ω1 ˆ 0. For all x ă˚ ω1 ˆ 0, Val strictly prefers to buy
at x. Finally, suppose that b ăx a holds only if ω1 ˆ 0 ă˚ x.
In other words, it is only when Val is fully aware and there issome uncertainty that she realises she would strictly preferto be a seller. In the language of logic and Heifetz, Meier,and Schipper [7] in particular, there is some special proposition that belongs only to ω1 that causes Val to prefer tosell.
Note that Val’s preferences satisfy both (Asy) and (CD).
The latter follows simply because the set tx : a „x bu contains only the point ω1 ˆ 0 and, by condition 0 of the definition of a perfectly normal set, the singleton sets are closed in
the order topology generated by ă˚ . Nonetheless, theorem
2 does not apply because X is not perfectly normal.

No continuous characterisation.
Recall from example 1, that the extended long line is not
perfectly normal because it contains a closed set that is notequal to the intersection of some countable collection of open
sets. We have chosen Val’s preferences so that tx : b Àx au “tω1 u ˆ r0, 1q. This leads us to the following proposition.
Proposition 6. There is no representation of Val’s preferences that characterises continuous context dependence.
of proposition 6. Note that tx : b Àx au “ tω1 u ˆ r0, 1q.
Following the construction in the proof of theorem 2, suppose there exists a continuous, nonnegative function g :X Ñ R such that g ´1 p0q “ tω1 u ˆ r0, 1q. Now let Gn “
tx : gpxq ă n´1 u for each n P N. Then by construction,Ş
tGn : n P Nu “ tω1 u ˆ r0, 1q. This, however, contradicts
the proof in the final paragraph of example 1 that X is not
perfect.

5.

SUMMARY

We have studied a simple model of context dependence forthe binary choice setting. For this case, the main theorem
provides a representation of preferences that depend continuously on a perfectly normal context space. This allowed
us to consider two examples of lexicographic context dependence. The first example was developed in the setting of a
medical decision. There the context space satisfied the conditions for the representation theorem. The second example
looked at unawareness and uncertainty in a financial setting.
In this case, the context space was not perfectly normal andpreferences were such that no continuous representation isavailable.

References[1] Alessandro Caterino, Rita Ceppitelli, and Francesca
Maccarino. “Continuous utility functions on submetrizable hemicompact k-spaces”. In: Applied General Topology 10 (2 2009), pp. 187–195.

223[2] C. d’Aspremont and L. Gevers. “Social welfare functionals and interpersonal comparability”. In: Handbookof social choice and welfare. Ed. by K. Arrow, A. Sen,
and K. Suzumura. Vol. 1. 2002, pp. 459–541.
[3] Gustav T. Fechner. Elemente der Psychophysik (2 Volumes). 2nd ed. Breitkopf & Hartel, 1889.
[4] Itzhak Gilboa and David Schmeidler. “A derivationof expected utility maximization in the context of agame”. In: Games and Economic Behavior 44.1 (2003),
pp. 172 –182. issn: 0899-8256. doi: DOI : 10 . 1016 /S0899-8256(03)00015-0.
[5] Itzhak Gilboa and David Schmeidler. “Act similarity
in case-based decision theory”. In: Economic Theory
9.1 (1997), pp. 47–61.
[6] Gary Gruenhage and Justin Tatch Moore. “Perfectcompacta and basis problems in topology”. In: Openproblems in topology II (2011), p. 151.
[7] Aviad Heifetz, Martin Meier, and Burkhard C Schipper. “Interactive unawareness”. In: Journal of Economic
Theory 130.1 (2006), pp. 78–94.
[8] Daniel Kahneman and Amos Tversky. “Prospect theory: and analysis of decision under risk”. In: Econometrica 47(2) (1979), pp. 263–291.
[9] Botond Kőszegy and Matthew Rabin. “A model ofreference-dependent preferences”. In: The Quarterly Journal of Economics 121(4) (2006), pp. 1133–1165.
[10] Joachim Lambek. Lectures on Rings and Modules. Chelsea
Publishing Series. Chelsea Publishing Company, 1986.
isbn: 9780828422833.
[11] Vladimir L. Levin. “A Continuous Utility Theorem for
Closed Preorders on a σ-Compact Metrizable Space”.
In: Soviet Math. Doklady 28 (1983), pp. 715–718.
[12] Yusufcan Masatlioglu and Efe A Ok. “Rational choicewith status quo bias”. In: Journal of Economic Theory
121.1 (2005), pp. 1–29.
[13] James R. Munkres. Topology. 2nd. Prentice Hall, 2000.
isbn: 9780131816299.
[14] Patrick O’Callaghan. “Minimal conditions for parametric continuity of a utility representation”. 2015.
[15] H.-J. Park and K. Friston. “Structural and Functional
Brain Networks: From Connections to Cognition”. In:
Science 342.1238411 (2013).
[16] Prasanta K. Pattanaik and Yongsheng Xu. “On Dominance and Context-Dependence in Decisions Involving
Multiple Attributes”. In: Economics and Philosophy
28 (Special Issue 02 2012), pp. 117–132.
[17] Michael Rockinger and Eric Jondeau. “Entropy densities with an application to autoregressive conditionalskewness and kurtosis”. In: Journal of Econometrics
106.1 (2002), pp. 119–142.
[18] Jacob S. Sagi. “Anchored preference relations”. In: Journal of Economic Theory 130.1 (2006), pp. 283 –295.
issn: 0022-0531. doi: http://dx.doi.org/10.1016/
j.jet.2005.01.009.
[19] John von Neumann. The computer and the brain. Yale
University Press, 1976.

When Do Types Induce the Same Belief Hierarchy? ∗Andrés Perea

EPICENTER and Dept. of Quantitative Economics
Maastricht University

a.perea@maastrichtuniversity.nl

ABSTRACT
Harsanyi (1967–1968) showed how infinite belief hierarchies
can be encoded by means of type structures. Such encodings, however, are far from unique: Two different types –possibly from two different type structures – may generate
exactly the same belief hierarchy. In this paper we present
a finite recursive procedure, the Type Partitioning Procedure, which verifies whether two types, from two potentiallydifferent finite type structures, induce the same belief hierarchy or not. Important is that the procedure does not makeexplicit reference to belief hierarchies, but operates entirely
within the language of type structures.

Keywords
Types, belief hierarchies

1.

INTRODUCTION

Belief hierarchies play a fundamental role in the modern
analysis of games. In games with incomplete information –where players face uncertainty about the opponents’ utilities– it is important to model what a player believes about his
opponents’ utility functions, what he believes about the opponents’ beliefs about their opponents’ utility functions, andso on (Harsanyi (1962, 1967–1968), Böge and Eisele (1979),
Mertens and Zamir (1985), Ely and Peski(2006), Dekel,,
Fudenberg and Morris (2007), Weinstein and Yildiz (2007)
and others). But even in games with complete information– where the players’ actual utility functions are transparent
to everyone – belief hierarchies naturally enter the analysis when we investigate the belief a player has about his
opponents’ choices, the belief he has about the opponents’beliefs about their opponents’ choices, and so on. Such belief hierarchies are the basis for many concepts in epistemic
game theory, most of which build upon the central notionof common belief in rationality (Brandenburger and Dekel(1987), Tan and Werlang (1988)). For an overview of theseconcepts, see Brandenburger (2007), Perea (2012) and Dekel
and Siniscalchi (2013).
One important practical problem with belief hierarchies
is that these are infinite objects, with infinitely many layersof beliefs. It is thus impossible to explicitly write down abelief hierarchy, layer by layer, as there are infinitely many of∗This paper is a substantially revised version of an earlier paperwith the same title. I would like to thank Pierpaolo Battigalli,
Eddie Dekel, Amanda Friedenberg, Willemien Kets, Christian
Nauerz, Miklós Pintér and Elias Tsakas for very useful commentson the earlier version.

225

these. But then, the question naturally arises: Is there a way
to represent belief hierarchies in a compact and convenientway?
Harsanyi (1967–1968) gave a positive and elegant answer
to this question. He focused on a setting in which the belief
hierarchies concern only the players’ utilities, but his construction has later been extended to situations where players also hold beliefs about the opponents’ choices. The construction that Harsanyi proposed was very simple: For every
player we define a set of types, and for every type we define
a utility function, together with a probabilistic belief about
the opponents’ types. From this very simple construction
we can then derive, for every type, a first-order belief about
the opponents’ utility functions, a second-order belief about
the opponents’ first-order beliefs, and so on. That is, forevery type we can derive a full belief hierarchy on the players’ possible utility functions in the game. This construction
by Harsanyi was a major step forward, as it allowed us toencode infinite belief hierarchies about utilities in a very
compact and convenient fashion.
Harsanyi’s original idea can easily be adapted to a framework where players also hold beliefs about other features
besides the players’ utilities. Assume that every player faces
a basic space of uncertainty, which can include the parameters determining the players’ utility functions, the set of opponents’ choices, and possibly some other features as well.
Now, consider for every player a set of types, and associate
to every type a probabilistic belief about the basic spaceof uncertainty and the opponents’ types. Then, similarly
to Harsanyi’s construction, we can derive for every type afull belief hierarchy, specifying a first-order belief about thebasic space of uncertainty, a second-order belief about the
opponents’ first-order beliefs about their basic spaces of uncertainty, and so on. This construction, which we call a typestructure, thus allows us to encode infinite belief hierarchies
about any set of parameters in a very compact way.
Such encodings, however, are far from unique: Two different types – possibly from two different type structures– may encode one and the same belief hierarchy. In viewof this “multiplicity problem” we ask the following natural
question in this paper: When do two types, from two potentially different finite type structures, induce the same belief
hierarchy?
Checking this directly, by explicitly comparing their induced first-order beliefs, second-order beliefs, and so on, may
be quite cumbersome as one needs to check for infinitely
many levels of belief. Instead, this paper presents a finite
recursive procedure, the Type Partitioning Procedure, whichtells us precisely when two such types induce the same belief
hierarchy, and when they do not. The procedure works asfollows. If we compare two types from two different typestructures, we start by merging the two type structures into
one large type structure. In every round, the procedure
generates for every player a partition of the set of types in
the large type structure, where the partition in the current
round will always be a refinement of the partition from theprevious round. Since we restrict to type structures withfinitely many types, this procedure will always terminate
within finitely many rounds. The equivalence classes in the
final partitions will then be exactly those groups of types
that generate the same belief hierarchy. That is, two types
generate the same belief hierarchy exactly when they belong
to the same equivalence class in the final partition. We actually show a bit more in Theorem 2: We prove that forevery n, two types share the same n-th order belief precisely
when they belong to the same equivalence class of the partition produced in round n of the procedure. In that sense,
the Type Partitioning Procedure provides a convenient andautomated way to verify whether two types share the same
belief hierarchy, or the same beliefs up to a fixed order n. Important, moreover, is that the Type Partitioning Proceduredoes not make any explicit reference to belief hierarchies –it operates entirely within the “language” of type structures.
We use the Type Partitioning Procedure to establish aninteresting property of belief hierarchies which – we believe– is new. Suppose we compare two types – possibly fromtwo different type structures – and let N be the total number of types in these two type structures. Then, we provein Corollary 1 that these two types induce the same belief
hierarchy exactly when they induce the same N -th order belief. To prove this result we use the property that the TypePartitioning Procedure will always terminate within at mostN rounds. In particular, the smaller the number of types
in the type structure, the less belief levels we must checkin order to conclude that two types share the same belief
hierarchy.
The outline of this paper is as follows. In Section 2 weintroduce type structures and belief hierarchies, and show
how we can derive belief hierarchies from types. In Section
3 we describe the Type Partitioning Procedure, illustrate it
by means of an example, and show how it characterizes those
types that share the same belief hierarchy. In Section 4 weshow how the procedure can be used to test whether two
types from different type structures generate the same belief hierarchy or not. In Section 5 we use the Type Partitioning Procedure to test properties of type structures rather
than individual types. For instance, we use the procedure tocheck whether a type structure contains redundant types ornot, or whether one type structure is contained in anothertype structure, in the sense that for every type in the firststructure there is a type in the second structure that generates the same belief hierarchy. Section 6, finally, contains
the proofs.

2.1

Consider a finite set of agents I. Assume that each agent ifaces a basic space of uncertainty Xi = (Xi , Σi ), where Xi isan arbitrary set and Σi a σ-algebra on Xi . That is, (Xi , Σi )is a measurable space. The combination X = (Xi )i∈I ofbasic uncertainty spaces is called a multi-agent uncertainty
space.
If the agents are the players in a game, the basic spaceof uncertainty for player i could, for instance, be the setof opponents’ choice combinations, or the set of parameters
determining the utility functions of the players, or even a
combination of the two. The first scenario is the standard
framework for games with complete information, the second
scenario is Harsanyi’s (1967–1968) original setting for gameswith incomplete information, whereas the last scenario is investigated in Böge and Eisele (1979) and Mertens and Zamir(1985), among others. The sets Xi could also include external events that cannot be influenced by the agents, as is thecase in Böge and Eisele (1979).
A belief hierarchy for player i specifies a probability measure on Xi – the first-order belief, a probability measureon Xi and the opponents’ possible first-order beliefs – thesecond-order belief, and so on. Following Harsanyi’s (1967–1968) approach, we will encode such infinite belief hierarchies by means of type structures. In this paper we focus
on type structures with finitely many types, which of course
imposes restrictions on the possible belief hierarchies we canencode. Indeed, there are belief hierarchies which can simply not be encoded by type structures with finitely many
types.
Definition 1 (Type Structure). Consider a multiagent uncertainty space X = (Xi , Σi )i∈I . A finite typestructure for X is a tuple T = (Ti , bi )i∈I where, for every player i,(a) Ti is the finite set of types for player i, and(b) bi : Ti → ∆(Xi × T−i , Σ̂i ) is a mapping that assigns toevery type ti a probabilistic belief bi (ti ) ∈ ∆(Xi × T−i , Σ̂i )on his basic uncertainty space and the opponents’ type combinations.
Here, T−i := ×j6=i Tj . For any measurable space (Yi , Σ̂i ),we denote by ∆(Yi , Σ̂i ) the set of probability measures on(Yi , Σ̂i ). In part (b) of the definition, we assume Σ̂i to be
the product σ-algebra on Xi × T−i induced by the σ-algebra
Σi on Xi and the discrete σ-algebra on the finite set T−i .

2.2

From Type Structures to Belief Hierarchies

In the previous subsection we have introduced a typestructure as a way to encode belief hierarchies. We will
now show how to “decode” a type within a type structure,by deriving the full belief hierarchy it induces.
Consider a finite type structure T = (Ti , bi )i∈I for Γ.
Then, every type ti within T induces an infinite belief hierarchyhi (ti ) = (h1i (ti ), h2i (ti ), ...),

2. BELIEF HIERARCHIES AND TYPESIn this section we show how belief hierarchies can be encoded by means of a type structure, and how every typewithin a type structure can be “decoded” by deriving a fullbelief hierarchy from it.

Encoding Belief Hierarchies by Type Structures

where h1i (ti ) is the induced first-order belief, h2i (ti ) is theinduced second-order belief, and so on. We will inductively
define, for every n, the n-th order beliefs induced by types
ti in T , building upon the (n − 1)-th order beliefs that have226

been defined in the preceding step. We start by defining thefirst-order beliefs.
For every player i, and every type ti ∈ Ti , define the firstorder belief h1i (ti ) ∈ ∆(Xi , Σi ) by
h1i (ti )(Ei ) := bi (ti )(Ei × T−i ) for all Ei ∈ Σi .

Now, suppose that n ≥ 2, and assume that the (n − 1)-thorder beliefs hn−1(ti ) have been defined for all players i, andievery type ti ∈ Ti . Let(ti ) | ti ∈ Ti }(Ti ) := {hn−1
hn−1
i
i

be the finite set of (n−1)-th order beliefs for player i inducedby types in Ti . For every hn−1
∈ hn−1(Ti ), letii}(ti ) = hn−1
] := {ti ∈ Ti | hn−1
Ti [hn−1iiibe the set of types in Ti that have the (n − 1)-th order belief
hn−1
.
i
n−1
n−1
Let hn−1(Tj ), and for a given h−i=−i (T−i ) := ×j6=i hj
n−1
n−1
n−1
n−1
(hj )j6=i in h−i (T−i ) let T−i [h−i ] := ×j6=i Tj [hj ].
We define the n-th order beliefs hni (ti ) as follows. LetΣn−1
be the product σ-algebra on Xi × hn−1i−i (T−i ) inducedby the σ-algebra Σi on Xi and the discrete σ-algebra on thefinite set hn−1−i (T−i ). For every type ti ∈ Ti , let the n-thn−1n−1order belief hn) be given byi (ti ) ∈ ∆(Xi × h−i (T−i ), Σi
n−1
n−1
hn
i (ti )(Ei × {h−i }) := bi (ti )(Ei × T−i [h−i ])

(1)

n−1
for every Ei ∈ Σi and every hn−1−i ∈ h−i (T−i ).

Finally, for every type ti ∈ Ti , we denote by

T1 = {t1 , t01 , t001 },T2 = {t2 , t02 , t002 }b1 (t1 ) = 12 (c, t2 ) + 21 (d, t02 )b1 (t01 ) = 16 (c, t2 ) + 31 (c, t002 ) + 12 (d, t02 )b1 (t001 ) = 12 (c, t02 ) + 21 (d, t002 )b2 (t2 ) = 14 (e, t1 ) + 21 (e, t01 ) + 41 (f, t001 )b2 (t02 ) = 18 (e, t1 ) + 81 (e, t01 ) + 43 (f, t001 )b2 (t002 ) = 38 (e, t1 ) + 83 (e, t01 ) + 14 (f, t001 )Table 1: The type structure from Example 1

contain exactly the types that induce the same belief hierarchy up to order n. In particular, the equivalence classes
produced at the end contain precisely those types that induce the same (infinite) belief hierarchy.
Procedure 1 (Type Partitioning Procedure). We
consider a multi-agent uncertainty space X = (Xi , Σi )i∈I ,and a finite type structure T = (Ti , bi )i∈I for X .
Initial step. For every agent i, let Pi0 be the trivial partition of his set of types Ti .

Inductive step. Suppose that n ≥ 1, and that the partitions Pin−1 have been defined for every agent i. Then, forevery agent i, and every ti ∈ Ti ,n−1
n−1
Pin (ti ) = {t0i ∈ Ti | bi (t0i )(Ei × P−i) = bi (ti )(Ei × P−i)(2)hi (ti ) := (hni (ti ))n∈N

the belief hierarchy induced by ti .

n−1
n−1
for all Ei ∈ Σi , and all P−i
∈ P−i
}.

3. TYPE PARTITIONING PROCEDURE
Suppose we consider a finite type structure for some multiagent uncertainty space X . In this section we will present a
recursive procedure – the Type Partitioning Procedure – thattells us, within finitely many steps, which types from thistype structure induce the same belief hierarchy and which
do not. Important is that this procedure does not makeexplicit reference to belief hierarchies, but operates entirely
within the “language” of type structures.

3.1

Type structure T = (T1 , T2 , b1 , b2 )Definition of the Procedure

To formally define this procedure, we need the following
terminology. A finite partition of a set A is a finite collection
P = {P1 , ..., PK } of nonempty subsets Pk ⊆ A such that∪K
k=1 Pk = A and Pk ∩ Pm = ∅ whenever k 6= m. We referto the sets Pk as equivalence classes. For an element a ∈A, we denote by P(a) the equivalence class Pk to which abelongs. The trivial partition of A is the partition P = {A}
containing only one set – the full set A. For two partitions
P 1 and P 2 on A, we say that P 1 is a refinement of P 2 iffor every set P 1 ∈ P 1 there is a set P 2 ∈ P 2 such thatP 1 ⊆ P 2 . We say that P 1 is a strict refinement of P 2 if P 1
is a refinement of P 2 and P 1 6= P 2 .
In the procedure we recursively partition the set of typesof an agent into equivalence classes – starting from the trivial partition, and refining the previous partition with every
step – until these partitions cannot be refined any further.
We show that the equivalence classes produced in round n

227

The procedure terminates at round n whenever Pin =Pin−1 for every agent i.
n−1
In this procedure, P−i
is the partition of the set T−i
induced by the partitions Pjn−1 on Tj . More precisely, if
t−i = (tj )j6=i is in T−i , thenn−1(t−i ) := ×j6=i Pjn−1 (tj ),P−iwhich is a subset of T−i .
We will now illustrate the Type Partitioning Procedure bymeans of an example.
Example 1. Consider a multi-agent uncertainty space X =(Xi , Σi )i∈I where I = {1, 2}, X1 = {c, d}, X2 = {e, f }, andΣ1 , Σ2 are the discrete σ-algebras on X1 and X2 , respectively. Consider the type structure T = (T1 , T2 , b1 , b2 ) in
Table 1. Here, b1 (t1 ) = 21 (c, t2 ) + 12 (d, t02 ) means that typet1 assigns probability 12 to the pair (c, t2 ) ∈ X1 × T2 , andprobability 12 to the pair (d, t02 ) ∈ X1 × T2 . Similarly for theother types in the table. We will now run the Type Partitioning Procedure.
Initial Step. Let P10 be the trivial partition of the set oftypes T1 , and let P20 be the trivial partition of the set oftypes T2 . That is,P10 = {{t1 , t01 , t001 }} and P20 = {{t2 , t02 , t002 }}.

Round 1. By equation (2),P11 (t1 ) = {τ1 ∈ T1 |b1 (τ1 )({c} × T2 ) = b1 (t1 )({c} × T2 ) = 21 ,b1 (τ1 )({d} × T2 ) = b1 (t1 )({d} × T2 ) = 21 }= {t1 , t01 , t001 },which implies that
P11 = P10 = {{t1 , t01 , t001 }}.
At the same time,
P21 (t2 ) = {τ2 ∈ T2 |b2 (τ2 )({e} × T1 ) = b2 (t2 )({e} × T1 ) = 43 ,b2 (τ2 )({f } × T1 ) = b2 (t2 )({f } × T1 ) = 41 }= {t2 , t002 }which implies that P21 (t02 ) = {t02 }, and hence
P21 = {{t2 , t002 }, {t02 }.

Round 2. By equation (2),P12 (t1 ) = {τ1 ∈ T1 |b1 (τ1 )({c} × {t2 , t002 }) = b1 (t1 )({c} × {t2 , t002 }) = 12 ,b1 (τ1 )({c} × {t02 }) = b1 (t1 )({c} × {t02 }) = 0,b1 (τ1 )({d} × {t2 , t002 }) = b1 (t1 )({d} × {t2 , t002 }) = 0,b1 (τ1 )({d} × {t02 }) = b1 (t1 )({d} × {t02 }) = 21 }= {t1 , t01 },which implies that P12 (t001 ) = {t001 }, and hence
P12 = {{t1 , t01 }, {t001 }}.

Since P11 = P10 , we may immediately conclude that
P22 = P21 = {{t2 , t002 }, {t02 }}.

Round 3. As P22 = P21 , we may immediately conclude that
P13 = P12 = {{t1 , t01 }, {t001 }}.

By equation (2),P23 (t2 ) = {τ2 ∈ T2 |b2 (τ2 )({e} × {t1 , t01 }) = b2 (t2 )({e} × {t1 , t01 }) = 43 ,b2 (τ2 )({e} × {t001 }) = b2 (t2 )({e} × {t001 }) = 0,b2 (τ2 )({f } × {t1 , t01 }) = b2 (t2 )({f } × {t1 , t01 }) = 0,b2 (τ2 )({f } × {t001 }) = b2 (t2 )({f } × {t001 }) = 41 }= {t2 , t002 },which implies that P23 (t02 ) = {t02 }, and hence
P23 = {{t2 , t002 }, {t02 }} = P22 .

As P13 = P12 and P23 = P22 , the procedure terminates atround 3. The final partitions of the types are thus given by
P1∞ = {{t1 , t01 }, {t001 }} and P2∞ = {{t2 , t002 }, {t02 }}.

The reader may check that all types within the same equivalence class indeed induce the same belief hierarchy. Thatis, t1 induces the same belief hierarchy as t01 , and t2 induces
the same belief hierarchy as t002 . Moreover, t1 and t001 induce
different belief hierachies, and so do t2 and t02 .


3.2

Characterization Result

We will now show that the Type Partitioning Procedure
identifies precisely those types that share the same belief
hierarchy. As a preparatory result, we will first highlight two
important properties of the procedure. The first property
states that the procedure is monotonic in the sense that thepartitions generated at a particular round will always be
refinements of the partitions generated in the round before.
The second property states that the number of rounds thatis needed for the procedure to terminate can never be larger
than the total number of types we consider.
Theorem 1 (Properties of Procedure). Consider
a multi-agent uncertainty space X = (Xi , Σi )i∈I , and a finite
type structure T = (Ti , bi )i∈I for X . For every agent i andevery round n ≥ 0, let Pin be the partition of Ti generated
in round n of the Type Partitioning Procedure. Let N be
the total number of types in ∪i∈I Ti . Then,(a) the partition Pin will always be a refinement of Pin−1 ,for all agents i and all n ≥ 1;(b) the procedure will terminate after at most N rounds.

With this result at hand we can now prove the main
theorem in this paper, which states that the Type Partitioning Procedure characterizes precisely those groups of types
that induce the same belief hierarchy. We actually prove alittle more: we show that the partitions generated in round nof the procedure characterize exactly those types that yield
the same n-th order belief.
Theorem 2 (Characterization Result). Consider
a multi-agent uncertainty space X = (Xi , Σi )i∈I , and a finite
type structure T = (Ti , bi )i∈I for X . For every agent i andevery round n ≥ 0, let Pin be the partition of Ti generated
in round n of the Type Partitioning Procedure. Let Pi∞ be
the final partition generated by the procedure. Then, forevery agent i, every n ≥ 1, and every two types ti , t0i ∈ Ti ,we have thatn 00n(a) hni (ti ) = hi (ti ), if and only if, ti ∈ Pi (ti );(b) hi (ti ) = hi (t0i ), if and only if, t0i ∈ Pi∞ (ti ).

By combining Theorems 1 and 2 we can derive some interesting facts about finite type structures and belief hierarchies, which we state in the following corollary.
Corollary 1 (Properties of Belief Hierarchies).
Consider a multi-agent uncertainty space X = (Xi , Σi )i∈I ,and a finite type structure T = (Ti , bi )i∈I for X . Let N be
the total number of types in ∪i∈I Ti , and let ti , t0i ∈ Ti . Then,(a) for every n ≥ 2, hin−1 (ti ) = hin−1 (t0i ) whenever hni (ti ) =0hni (ti );
N 0(b) hi (ti ) = hi (t0i ), if and only if, hNi (ti ) = hi (ti ).

Property (a) thus states that two types agreeing on the nth order belief will also agree on all lower order beliefs. That

228

Type structure T 1 = (T11 , T21 , b11 , b12 )is, the n-th order belief completely determines the first-orderbelief, the second-order belief, until the (n − 1)-th orderbelief. Property (b) says that in order to check whether two
types share the same infinite belief hierarchy or not we only
have to compare the N -th order beliefs, where N is the total
number of types in the type structure. To the best of ourknowledge, this result is new in the literature.
The proof of this corollary is actually very easy. To shown 0
property (a) consider two types ti , t0i with hni (ti ) = hi (ti ).
0nThen, by Theorem 2, ti ∈ Pi (ti ). Since, by Theorem 1, Pin
is a refinement of Pin−1 , it follows that t0i ∈ Pin−1 (ti ) andhence, by Theorem 2, hn−1(ti ) = hn−1(t0i ).
ii
To show property (b), take two types ti , t0i with hNi (ti ) =00NhNi (ti ). Then, by Theorem 2, ti ∈ Pi (ti ). By Theorem 1
we know that the procedure terminates after at most N
rounds, and hence PiN = Pi∞ . By Theorem 2 we conclude
that hi (ti ) = hi (t0i ).
Some readers may ask why property (a) requires a proof,
as in most other papers in the literature the n-th order belief
induced by a type explicitly contains the (n−1)-th order belief as a component, and hence property (a) holds trivially.
See, for instance, Heifetz and Samet (1998) and Friedenberg and Meier (2011). However, this is not the case in our
setting: Our definition of hni (ti ) does not explicitly carry
hn−1
(t
)
as
a
component,
and
it is therefore not obviousiithat the n-th order belief fully determines the (n − 1)-thorder belief. This is a result, which requires a proof in our
setting.

4. COMPARING TYPES FROM DIFFERENTTYPE STRUCTURES
We have seen that the Type Partitioning Procedure tells us
exactly which types within a given type structure T induce
the same belief hierarchy, and which do not. But what if
we want to compare types from different type structures?
We will see that the procedure will work for such settings as
well.
Let us consider two different finite type structures, T 1 =(Ti1 , b1i )i∈I and T 2 = (Ti2 , b2i )i∈I , for the same multi-agent
uncertainty space X = (Xi , Σi )i∈I . For a given agent i, takea type t1i ∈ Ti1 and a type t2i ∈ Ti2 . How can we check
whether t1i and t2i induce the same belief hierarchy?
What we can do is to first merge the two type structures
into one large type structure, and to subsequently run theType Partitioning Procedure for the large type structure.
More precisely, let T = (Ti , bi )i∈I be the “large” type structure, where Ti := Ti1 ∪ Ti2 for all agents i, and 1bi (ti ), if ti ∈ Ti1bi (ti ) :=b2i (ti ), if ti ∈ Ti2
for all types ti ∈ Ti . Hence, T is a “block” type structure1
in which types in Ti1 only refer to opponents’ types in T−i,22
and types in Ti only refer to opponents’ types in T−i . But
it is still a well-defined type structure, and hence we canrun the Type Partitioning Procedure for the “block” typestructure T , yielding partitions Pi∞ of the sets Ti = Ti1 ∪ Ti2
for every agent i. If t1i ∈ Ti1 and t2i ∈ Ti2 turn out to be in
the same equivalence class of Pi∞ , then t1i and t2i induce thesame belief hierarchy. Otherwise not. In this way, the TypePartitioning Procedure can also be used to test whether two
types from different type structures induce the same belief
hierarchy or not.

229

T11 = {t1 , t01 , t001 , t000
1 },T21 = {t2 , t02 , t002 }b11 (t1 ) = 12 (c, t2 ) + 12 (d, t02 )b11 (t01 ) = 16 (c, t2 ) + 13 (c, t002 ) + 12 (d, t02 )b11 (t001 ) = 12 (c, t02 ) + 21 (d, t002 )
00
1
2
b11 (t000
1 ) = 3 (c, t2 ) + 3 (d, t2 )b12 (t2 ) = 14 (e, t1 ) + 12 (e, t01 ) + 14 (f, t001 )b12 (t02 ) = 18 (e, t1 ) + 18 (e, t01 ) + 34 (f, t001 )b12 (t002 ) = 38 (e, t1 ) + 83 (e, t01 ) + 14 (f, t001 )Type structure T 2 = (T12 , T22 , b21 , b22 )T12 = {r1 , r10 , r100 },T22 = {r2 , r20 , r200 }b21 (r1 ) = 41 (c, r2 ) + 14 (c, r200 ) + 21 (d, r20 )b21 (r10 ) = 21 (c, r20 ) + 81 (d, r2 ) + 83 (d, r200 )b21 (r100 ) = 21 (c, r20 ) + 38 (d, r2 ) + 81 (d, r200 )b22 (r2 ) = 41 (e, r10 ) + 43 (f, r1 )b22 (r20 ) = 34 (e, r10 ) + 41 (f, r1 )b22 (r200 ) = 18 (e, r10 ) + 81 (e, r100 ) + 34 (f, r1 )Table 2: The type structures from Example 2

n
0
1
2
3

P1n
0
00
{{t1 , t01 , t001 , t000
1 , r1 , r1 , r1 }}
0
00
0
00
{{t1 , t1 , t1 , r1 , r1 , r1 }, {t0001 }}{{t1 , t01 , r10 , r100 }, {t001 , r1 }, {t0001 }}{{t1 , t01 , r10 , r100 }, {t001 , r1 }, {t000
1 }}

P2n
{{t2 , t02 , t002 , r2 , r20 , r200 }}{{t2 , t002 , r20 }, {t02 , r2 , r200 }}{{t2 , t002 , r20 }, {t02 , r2 , r200 }}{{t2 , t002 , r20 }, {t02 , r2 , r200 }}Table 3: The Type Partitioning Procedure in Example 2

Example 2. To see how this works, let us consider an example with two agents, I = {1, 2}, where the basic spacesof uncertainty are again given by X1 = {c, d} and X2 ={e, f } – as in Example 1 – together the the discrete σalgebras on these sets. Consider the two type structures
T 1 = (Ti1 , b1i )i∈I and T 2 = (Ti2 , b2i )i∈I on X as given in
Table 2. Note that type structure T 1 is almost identical to
the type structure in Example 1, except for the fact that wehave added an extra type t000
1 for agent 1.
We want to test whether the types t1 ∈ T11 and r1 ∈T12 , which belong to different type structures, induce thesame belief hierrachy or not. As a first step we merge thetwo type structures T 1 and T 2 into one large block typestructure, as described above. If we subsequently run theType Partitioning Procedure for the large type structure,then the reader may verify that the partitions in every roundare given by Table 3. Here, the procedure terminates atround 3. As t1 and r1 are not in the same equivalence class,
we conclude that t1 and r1 do not induce the same belief
hierarchy. In fact, the final partitions tell us that for agent 1,
the types t1 , t01 , r10 and r100 all induce the same belief hierarchy,
that types t001 and r1 induce the same belief hierarchy, and12that for type t000
1 ∈ T1 there is no type in T1 that induces
the same belief hierarchy. For agent 2, the types t2 , t002 andr20 all induce the same belief hierarchy, and so do the types
t02 , r2 and r200 .


5. TESTING FOR PROPERTIES OF TYPESTRUCTURESThe Type Partitioning Procedure can be used to answer
several different questions – local and global. First, as wealready discussed, we can use it to test whether two types– possibly from different type structures – induce the same
belief hierarchy or not. This is a local test.
But we can also use it to test global properties of typestructures. For instance, we can use the procedure to test
whether a given type structure contains redundant types or
not – where “redundant” means that two different types induce the same belief hierarchy. For this redundancy test we
can run the Type Partitioning Procedure and see whether
the final partitions contain equivalence classes with at least
two types. If this is the case then we conclude that the typestructure contains redundancies. If, on the other hand, allequivalence classes contain only one type, then there are no
redundancies in the type structure.
In case the type structure contains redundant types, theType Partitioning Procedure will also tell us how to “remove”
these redundancies without changing the induced collectionof belief hierarchies. What we can do in this case is to replace
every equivalence class in the final partition by a single type,
and to change the belief of every type accordingly. Thenwe will obtain a smaller, non-redundant type structure thatinduces exactly the same collection of belief hierarchies.
As an illustration, consider the type structure from Table
1. We have seen in Example 1 that the Type Partitioning
Procedure generates the final partitions
P1∞ = {{t1 , t01 }, {t001 }} and P2∞ = {{t2 , t002 }, {t02 }}.

If we replace the equivalence class {t1 , t01 } by the single typer1 , and replace the equivalence class {t2 , t002 } by the single
type r2 , then we obtain the smaller type structure T̂ =(T̂i , b̂i )i∈I whereT̂1 = {r1 , t001 }, T̂2 = {r2 , t02 }andb̂1 (r1 )=b̂2 (t001 )=b̂2 (r2 )=b̂2 (t02 )=1(c, r2 ) + 12 (d, t02 ),2
1(c, t02 ) + 12 (d, r2 ),2
3(e, r1 ) + 14 (f, t001 ),4
1(e, r1 ) + 34 (f, t001 ).
4

It can be verified that T̂ is indeed non-redundant, and induces the same collection of belief hierarchies as the original
type structure T from Table 1.
Another global question we can answer is whether two different type structures generate the same collection of belief
hierarchies, or whether the collection of belief hierarchies induced by the first type structure is contained in that of thesecond structure. This is the type of question which is addressed, for instance, in Friedenberg and Meier (2011). To
answer the first question we can first merge the two typestructures into one, and subsequently run the Type Partitioning Procedure. If the final partitions are such that every equivalence class always contains at least one type fromboth type structures, then the two structures generate thesame collection of belief hiearchies. Otherwise not. Indeed,assume that every equivalence class in the final partitions
contains at least one type from each type structure. Then,for every type in the first structure there is a type in the second structure that generates the same belief hierarchy, andvice versa. That is, both type structures produce exactly
the same collection of belief hierarchies. If this is not thecase, that is, if there is an equivalence class that contains
only types from one type structure but not from the other,
then these type do not have any “counterpart” in the othertype structure, and hence the two type structures differ inthe collection of belief hierarchies they generate. To answer
the second question – that is, whether the set of belief hierarchies of the first structure is contained in that of thesecond – we look at the final partitions, and see whether
every equivalence class contains at least one type from thesecond structure.
We have collected the insights above in the following corollary.
Corollary 2 (Type Structures). Consider
a multi-agent uncertainty space X = (Xi , Σi )i∈I , and two
finite type structures T 1 = (Ti1 , b1i )i∈I and T 2 = (Ti2 , b2i )for X . Let (Pi∞ )i∈I be the final partitions generated by theType Partitioning Procedure if we first merge the two typestructures into one. Then:(a) type structure T 1 is redundant, if and only if, there issome agent i and some Pi ∈ Pi∞ such that |Pi ∩ Ti1 | ≥ 2;(b) the collection of belief hierarchies induced by T 1 is asubset of the collection of belief hierarchies induced by T 2 ,if and only if, Pi ∩ Ti2 6= ∅ for all agents i and all Pi ∈ Pi∞ ;(c) type structures T 1 and T 2 induce the same collection ofbelief hierarchies, if and only if, Pi ∩ Ti1 6= ∅ and Pi ∩ Ti2 6= ∅for all agents i and all Pi ∈ Pi∞ .
Here, we say that a type structure is redundant if it contains at least two different types that generate the same
belief hierarchy.

6.

PROOFS

Proof of Theorem 1. We first prove (a) by induction onn.
Induction start. The partition Pi1 will always be a refinement of Pi0 since Pi0 is the trivial partition, by definition.
Inductive step. Let n ≥ 2, and suppose that Pin−1 is arefinement of Pin−2 , for all agents i. Consider an agent i, anequivalence class Pin ∈ Pin , and two types ti , t0i ∈ Pin . Then,by equation (2),n−1bi (ti )(Ei × P−i)n−1= bi (t0i )(Ei × P−i) for all Ei ∈ Σi ,n−1n−1
and all P−i∈ P−i.

As, by the induction assumption, Pjn−1 is a refinement ofn−1
Pjn−2 for all j 6= i, it follows that P−i
is a refinement of
n−2
P−i . But then, we conclude from (3) thatn−2bi (ti )(Ei × P−i
)

230

n−2
= bi (t0i )(Ei × P−i) for all Ei ∈ Σi ,n−2n−2
and allP−i∈ P−i,which means that ti and t0i belong to the same equivalenceclass in Pin−1 . So, we have shown that every two types thatare in the same equivalence class of Pin , are also in the sameequivalence class of Pin−1 . This, however, implies that Pin isa refinement of Pin−1 , as was to show. By induction on n,property (a) follows.
With property (a) at hand, it is easy to prove property(b). By property (a) we know that for every round n ≥ 1,and every agent i, the partition Pin is a refinement of Pin−1 .
Moreover, for every “active” round n – where the proceduredoes not terminate yet – the partition Pin must be a strict
refinement of Pin−1 for at least one agent i. It may be verified that for every agent i, the number of successive strict
refinements cannot be larger than the number of types in
Ti . As such, the number of active rounds in the procedure
cannot be larger than the number of types in ∪i∈I Ti , which
is N. This completes the proof.
Proof of Theorem 2. We first prove (a) by induction onn.
Induction start. Consider two types ti , t0i ∈ Ti . Suppose
first that h1i (ti ) = h1i (t0i ). We show that t0i ∈ Pi1 (ti ). For all00
Ei ∈ Σi and P−i
∈ P−i
,
0
bi (ti )(Ei × P−i
)

=

=

=

n−1
bi (t0i )(Ei × T−i [h−i
])

0
n−1
= hn
i (ti )(Ei × {h−i })

n−1
= hn
i (ti )(Ei × {h−i })= bi (ti )(Ei × T−i [hn−1
−i ])

n−1
),
= bi (ti )(Ei × P−iwhich by equation (2) indeed implies that t0i ∈ Pin (ti ). Here,the first and fifth equality follows from the insight above thatn−1
P−i
= T−i [hn−1−i ], the second and the fourth equality follow
0nfrom the definition of hni (ti ) and hi (ti ), whereas the thirdn 0
equality follows from the assumption that hn
i (ti ) = hi (ti ).
Suppose next that t0i ∈ Pin (ti ). We show that hn
i (ti ) =0hni (ti ).
n−1
Take some arbitrary combination hn−1
∈ h−i
(T−i ) of−i(n − 1)-th order beliefs that is obtained by at least one
type combination in T−i . By the induction assumption, there
n−1
n−1
n−1
must be some P−i∈ P−isuch that P−i= T−i [hn−1−i ].
Then, for every Ei ∈ Σi ,n−1
hn
i (ti )(Ei × {h−i })= bi (ti )(Ei × T−i [hn−1
−i ])

n−1
)
= bi (ti )(Ei × P−in−1= bi (t0i )(Ei × P−i)=

h1i (ti )(Ei )h1i (t0i )(Ei )bi (t0i )(Ei × T−i )0bi (t0i )(Ei × P−i),bi (t0i )(Ei × T−i [hn−1
−i ])

0
n−1
= hn
i (ti )(Ei × {h−i }),which indeed implies that t0i ∈ Pi1 (ti ). Here, the first and0fifth equality follow from the fact that P−i
is the trivial
partition on T−i , the second and fourth equality follow fromthe definition of h1i (ti ) and hi (t0i ), respectively, whereas thethird equality follows from the assumption that h1i (ti ) =h1i (t0i ).

h1i (ti )(Ei )n−1
bi (t0i )(Ei × P−i)= bi (ti )(Ei × T−i )==Assume next that t0i
1 0
hi (ti ). For all Ei ∈ Σi ,have that∈ Pi1 (ti ). We show that h1i (ti ) == bi (ti )(Ei × T−i ) = bi (t0i )(Ei × T−i )= h1i (t0i )(Ei ),which indeed implies that h1i (ti ) = h1i (t0i ). Here, the first andthird equality follow from the definition of h1i (ti ) and h1i (t0i ),respectively, whereas the second equality follows from theassumption that t0i ∈ Pi1 (ti ) and equation (2).
By the two steps above we may conclude that h1i (ti ) =h1i (t0i ), if and only if, t0i ∈ Pi1 (ti ), as was to show.
Inductive step. Let n ≥ 2, and assume that (a) holds forn − 1 and all agents i. Consider an agent i and two types
n 0
ti , t0i ∈ Ti . Suppose first that hn
i (ti ) = hi (ti ). We show that0nti ∈ Pi (ti ).
n−1
n−1
Consider some P−i∈ P−i. Take some arbitrary t−i ∈
n−1
n−1
n−1
P−i and let h−i = h−i (t−i ). By the induction assumpn−1tion, P−icontains all type combinations in T−i that induce
the same combination of (n − 1)-th order beliefs as t−i . Remember from Section 2.2 that T−i [hn−1−i ] denotes the set oftype combinations in T−i that induce hn−1
−i . Then, we may
n−1
conclude that P−i= T−i [hn−1−i ]. For every Ei ∈ Σi we then231

n 0
which indeed implies that hn
i (ti ) = hi (ti ). Here, the third
equality follows from the assumption that t0i ∈ Pin (ti ) andequation (2), whereas the other equalities follow exactly as
above.
By the two steps above we may thus conclude that hn
i (ti ) =00nhni (ti ), if and only if, ti ∈ Pi (ti ). By induction on n, statement (a) follows.

The proof of (b) follows immediately from (a) and property (b) in Theorem 1. This completes the proof.
7.

REFERENCES[1] Böge, W. and T.H. Eisele (1979), On solutions ofbayesian games, International Journal of Game
Theory 8, 193–215.
[2] Brandenburger, A. (2007), The power of paradox:
Some recent developments in interactive epistemology,
International Journal of Game Theory 35, 465–492.
[3] Brandenburger, A. and E. Dekel (1987),
Rationalizability and correlated equilibria,
Econometrica 55, 1391–1402.
[4] Dekel, E., D. Fudenberg and S. Morris (2007), Interim
correlated rationalizability, Theoretical Economics 2,
15–40.
[5] Dekel, E. and M. Siniscalchi (2013), Epistemic gametheory, Chapter prepared for Handbook of Game
Theory.
[6] Ely, J.C. and M. Peski(2006), Hierrachies of belief,
and interim rationalizability, Theoretical Economics 1,
19–65.
[7] Friedenberg, A. and M. Meier (2011), On therelationship between hierarchy and type morphisms,
Economic Theory 46, 377–399.

[8] Harsanyi, J.C. (1962), Bargaining in ignorance of the
opponent’s utility function, Journal of Conflict
Resolution 6, 29–38.
[9] Harsanyi, J.C. (1967–1968), Games with incomplete
information played by “bayesian” players, I–III,Management Science 14, 159–182, 320–334, 486–502.
[10] Heifetz, A. and W. Kets (2013), Robust multiplicity
with a grain of naiveté, Working paper.
[11] Heifetz, A. and D. Samet (1998), Topology-freetypology of beliefs, Journal of Economic Theory 82,
324–341.
[12] Mertens, J.-F. and S. Zamir (1985), Formulation ofbayesian analysis for games with incomplete
information, International Journal of Game Theory
14, 1–29.
[13] Perea, A. (2012), Epistemic Game Theory: Reasoning
and Choice, Cambridge University Press.
[14] Tan, T. and S.R.C. Werlang (1988), The bayesianfoundations of solution concepts of games, Journal of
Economic Theory 45, 370–391.
[15] Weinstein, J. and M. Yildiz (2007), Impact ofhigher-order uncertainty, Games and economic
Behavior 60, 200–212.

232

Rationalization and Robustness in Dynamic Games with
Incomplete Information[Extended Abstract]
Evan Piermont

Peio Zuazo-Garin

University of Pittsburgh
4200 5th Avenue
Pittsburgh, PA, USA

Universitat Rovira i Virgili
Avinguda de la Universitat 1
43204, Reus, Spainpeio.zuazo@urv.cat

ehp5@pitt.edu

Keywords
Dynamic Games, Common Knowledge, Rationalization, Robustness.

1. INTRODUCTION
In the game theoretic environment, there is a clear tension
between the strength of a solution concept and its robustness
to misspecification. In other words, if the analyst wants his
model to be resilient to small errors in the parameters, thenhe must weaken the predictive power of the model. In fact,
this tension can be made formal; under a richness assumption (loosely speaking, for every strategy there is a state suchthat it is dominant), structure theorems place clear limits tothe predictive power of robust solution concepts [8, 6].
Intuitively, a similar tension arises when considering a solution concept and its epistemic demands.1 To make sharper
predictions, the modeler must place more stringent requirements on the structure of the understanding of agents (atleast insofar as to adhere to the requirements of the solution
concept). Informally, this observation suggests a possible
link between the epistemic demands of a solution concept
and it’s robustness. The first aim of this paper is to formalize this connection. We show that particular notions ofrobustness can be thought of as epistemic concerns. In particular, we examine a solution concept’s robustness to themisspecification of players’ beliefs, the underlying space of
payoff uncertainty, and to the joint misspecification of both.
In each case, we show that reasonable and common notionsof robustness can be described entirely though the epistemic
characterization of the solution concept.
Most commonly, robustness has been defined with respect
to misspecification of player’s beliefs; in particular via theupper-hemicontinuity (henceforth, UHC) of the solution concept in question. UHC dictates that if a strategy is ruledout for some type of player, then there is a neighborhoodof nearby types for which the strategy is also ruled out. In
the absence of UHC, approximations will not suffice; even if
a strategy is selected by every successive approximation, it
may not be selected in the limit.
Our first result provides the epistemic characterization of
UHC, and hence, a direct method of verification of robust1
Informally, epistemic demand refer to the restrictions
placed on players beliefs regarding payoff uncertainty, opponents strategies, and the higher order beliefs over theseobjects. A more formal, but my no means complete, explanation is found in Section 2.

233

ness. The UHC of a a solution concept is related to theclosedness of the event that characterizes it. Intuitively, thata solution concept is well behaved with respect to approximation of the players types is implied by the fact that thelimit of any sequence in the characterizing event is also in
the event.
A second aim of this paper is to develop an appropriate notion of robustness to the misspecification of the spaceof payoff uncertainty (referred to as the state space). Although the richness assumption allows for structure theorems that place clear limits on the robustness of solution
concepts and provide for generic dominance solvability, it isoften an unreasonable demand on the space of uncertainty.
To assume richness is to drop all common knowledge assumptions regarding strategic unceratainty.2 However, inmany, if not most, economic situations, there are clear restrictions on payoffs. Without assuming major structural
ignorance on the part of the players, the economic restrictions are best modeled by common knowledge assumptions.
Moreover, simply embedding the game in a larger state space(and modeling common knowledge as initial common belief)
may distort predictions (for example, see section 2.1). Thisis a problem exclusive to dynamic environments, as in staticgames, the players’ initial (and only) beliefs entirely govern
actions. In dynamic environments, however, when players
are able to update their beliefs, the state space itself becomes relevant to the analysis.
Penta [6] proposes the robustness notion of informational
invariance, or, that the predictions of a solution concept
do not change when the game is embedded in any larger
state space. We ague that this is too strong a requirement,
as it restricts the solution concept from having any dependence on the state space that is not present in the player’s
initial beliefs. Instead, we propose the notion of state-spacerobustness (s-robustness). To this end, we allow the spaceof uncertainty that each player considers to become a parameter of the game. We call each players understanding
about the state space, and his higher order understanding
about his opponents understanding of the state space, his
directory. A player is described not only by his beliefs (ahierarchy of probability distributions), but also by his directory (a hierarchy of sets of parameters). Then, in analogy to
UHC, a solution concept is s-robust if whenever a strategy isruled out for a directory there is a neighborhood of nearby
2

Relaxing common knowledge assumptions can be somehow
regarded as strengthening common awareness ones.

A

directories for which the strategy is also ruled out. As in
regards to UHC, we provide the epistemic restrictions thatcoincide with s-robustness.
Lastly, we provide a structure theorem that identifies conditions on directories such that any strict refinement of Extensive Form Rationalizability (EFR) (introduced in [5]) is
not UHC. We find conditions that are strictly weaker than
the richness assumption in [6] and its static counterpart in[8]. In particular, while we require the existence of some
objective rich state space, this state space need not be commonly known. Each player could be described by a directorythat does not contain dominance states. Indeed, it is possible to obtain the result even in the case where it is common
knowledge that no action is ever dominant.

2.

UPPER-HEMICONITINUITY: AN EPISTEMIC APPROACH

We consider dynamic environments and our analysis takes
place within a formal epistemic framework. Players in agame hold beliefs regarding the state space Θ, the otherplayers’ beliefs about the payoff uncertainty, and other players’ strategies S−i . Moreover, players hold higher-order beliefs about other players’ beliefs about these objects, beliefs
about these beliefs and so on.
Following closely the construction due to [2], we formallyrepresent each player i’s higher-order conditional beliefs on
opponents’ choices and the payoff state via conditional belief
hierarchies, i.e., (epistemic) types drawn from a universal
type space we denote by E. Note that players’ uncertaintyin E−i × S−i × Θ is not modeled only at the beginning of
the game, but also at every history along the possible pathsof play. We further assume that beliefs are updated in a
Bayesian manner whenever possible.
The epistemic analysis is then performed in the set ofstates of the world, Ω = E × S × Θ. Each epistemic type,ei ∈ Ei , induces a standard type, τi ∈ Ti via the canonical quotient map, qi : Ei → Ti .3 For each standard type,τi , we denote by [qi = τi ] the event that player i’s standard hierarchy is exactly τi . A solution concept can then be
characterized by an event contained in this state space.
Definition 1. An event E ⊆ Ω characterizes the solution concept Si : Ti ⇒ Si if for all τi it holds that Si (τi ) =ProjSi (E ∩ [qi = τi ]).
Throughout the paper we focus on two different dynamic
solution concepts: extensive-form rationalizability (EFR) [5],and interim sequential rationalizable (ISR) [6]. Both are
generalizations of the static notion of interim correlated rationalizability; the difference being that EFR requires that
players place a higher epistemic priority to rationality than
ISR. Our focus on ISR is driven by the structure theoremof [6], which states that, under the richness assumption, any
strict refinement of ISR is not UHC.
The epistemic characterization of ISR, provided by [6], isthe event composed of Rationality (R) and Initial Common
Belief in Rationality (ICBR). R states that each player will
always choose a strategy that is a best response to his own
3
The standard type space consists of all hierarchies of beliefs over payoff uncertainty, modeled at the null history. It
is analogous to the canonical space constructed for static
environments by [3]. The map qi is simply the hierarchicalmarginalization on this space.

a1

In
A
a2

1
1

a3
P

b1

b2

b1

b2

2
2

0
1

θ
1

θ
2

Figure 1: Examples 1 and 2 in [6].

belief. ICBR for player i states that, at the beginning of
the game, player i believes players j 6= i are rational, believes that players j 6= i believe players k 6= j are rational,
etc. Since players are Bayesian, this implies that at any history reached with positive probability according to player
i’s initial belief, player i still holds a common belief in R.
In this paper, we recall the characterization of EFR as
the event composed of R and Common Strong Belief in Rationality (CSBR). Strong Belief in Rationality for player iis the event that player i assigns probability 1 to event R
at every history which is not belief-inconsistent with R, or,in other words, whenever R has not been falsified by observed history. This way, strong belief captures the essenceof forward induction, in the sense that beliefs about future
behavior are, whenever possible, updated according to observed history and the filter rationality imposes. CSBR isthen the hierarchical iteration of strong belief in rationality;that is, as stated by Battigalli’s best rationalization principle, the event that players assign to their opponents thehighest degree of strategic sophistication consistent with observed behavior [1]. It is immediate that ICBR is impliedby CSBR and hence EFR is a refinement of ISR. This last
observation, together with the structure theorem in [6], suggests a clear tension between rationalization and robustness:under richness, the robustness of predictions appears to belost as soon as players are assumed to reason according to
the highest rationalization principle. The following example
illustrates the conflict.

2.1

Alexei and Polina

Example 1 by [6] studies the sequential game depicted in
Figure 1, in which Alexei Ivanovich’s (player A) utility after history (In, a3 ) is represented by unspecified parameterθ. We focus first in the case in which Alexei and Polina
Alexandrovna (player P ) commonly know that θ equals 0:
Θ = {0} and the the corresponding standard type space is
T CK = {τ CK }. In this case, strategy (In, a3 ) is strictly
dominated for Alexei. So, if Polina finds herself at information set {(In, a2 ), (In, a3 )} (i.e, she is informed that Alexei
did not choose a1 ) and believes that Alexei is rational, thenshe must believe he played (In, a2 ). Thus, action b1 is optimal for her. Now, if Alexei expects Polina to rationalize his
choice, he is able to predict choice b1 in case he plays (In, a2 ),and hence, (In, a2 ) becomes optimal for him. However, if
Polina finds herself at information set {(In, a2 ), (In, a3 )} anddoes not believe that Alexei is rational, she can rationalize
any action. Hence ISR, which does not require Polina to234believe Alexei is rational at {(In, a2 ), (In, a3 )} (if, say, sheplaced probability 1 on his playing a1 ), considers strictly
more strategies than EFR, which makes such a requirement. Indeed, ISR(τ CK ) = {a1 , (In, a2 )} × {b1 , b2 }, andEFR(τ CK ) = ((In, a2 ), b2 ), as shown in greater detail in [6].
Next, consider the model in which Θ = {0, 3} and there
is common certainty that θ = 0, that is T CC = {τ CC },such that type τ CC assigns probability 1 to payoff state andopponents’ type combination (0, τ CC ).4 The analysis in [6]constructs a set of types {τ m }m∈N such that τ m → τ CC
and ISR(τ m ) = {(a1 , b2 )} for all m. Now, if EFR(τ CC ) =EFR(τ CK ) = {((In, a2 ), b2 )} this would indeed be a violation of UHC, as EFR is a refinement of ISR, hence, for all
types in the sequence {τ m }m∈N , EFR(τ m ) ⊆ ISR(τ m ) ={(a1 , b2 )}.
Notice however, that EFR(τ CC ) 6= EFR(τ CK ); the movefrom common knowledge to common certainty changes theset of EFR strategies. Although Polina must retain his belief
that Alexei is rational, even after surprising events, it is not
the case that she must retain his belief that Alexei places
probability 1 on θ = 0. Indeed, consider the following firstorder beliefs: Alexei assigns probability 1 to (b2 , 0) when
θ = 0, and probability 1 to (b2 , 3) when θ = 3, and Polina
assigns probability 1 to (a1 , 0) at the beginning of the game,and probability 1 to ((In, a3 ), 3) when she observes In. Now,
assume that Alexei’s second-order belief assigns probability
1 to Polina’s first-order beliefs above. Polina’s second-orderbeliefs assign probability 1 to Alexei’s first-order belief corresponding to θ = 0 at the beginning of the game, and, when
she observes In, assign probability 1 to Alexei’s first-orderbelief corresponding to θ = 3. Keeping this iteration, it iseasy to check that we obtain a profile of conditional belief
hierarchies that represent CSBR and initial common beliefin θ = 0 (when θ = 0, something Alexei is informed about).
Moreover, the unique best-response is indeed profile (a1 , b2 ).
Hence, (a1 , b2 ) ∈ EFR(τ CC ). The mere fact that updated
beliefs may take θ = 3 into account, and that it is commonly
known that this is possible, mutes the restriction imposedby the high epistemic priority is given to rationality.

2.2

Characterization

This example illustrates two key issues. The first is thatthe aforementioned tension between rationalization and robustness is not present: EFR does not fail to be UHC in theexample. The tension is mitigated since Polina can revise
her beliefs about Alexei’s actions without losing his belief in
Alexei’s rationality because there exists some state (θ = 3)
that allowed Alexei to be rational (but perhaps with incorrect beliefs) and take the particular action. This is a special case of a more general phenomena: when all common
knowledge assumptions are dropped (i.e., under richness)this observation extends to any action: if an action could
be rationalized by assuming an opponent is irrational, thenit can be rationalized by changing only that opponents beliefs, but retaining his rationality. These observations can bemade formal, and are best examined thorough the following
result:Proposition 1. Let E ⊆ Ω be a closed event such thatE ∩ [qi = τi ] 6= ∅ for any standard type τi . Then, for player
4

Penta actually considers a slightly different model in which
P1 has private information, but the analysis is undistorted.

235

i, the following correspondence is upper-hemicontinuous:
SiE :

Ti
τi

⇒
→

Si
ProjSi (Ri ∩ E ∩ [qi = τi ]).

Proposition 1 provides a general way of verifying if the a solution concept is UHC. Note, by requiring that E ∩ [qi = τi ]is non empty, we are implicitly requiring that the solution
concept does not place any direct restrictions on the standard type space. In other words, this method of verification
only works if the solution concept is well defined for all possible standard types.
Utilizing this result and the characterization of EFR (i.e.,by setting E = CSBR), it is straightforward to check thatthe result holds for EFR. This observation, along with thestructure theorem in [6], easily delivers the following result:Corollary 1. EFR is UHC. Moreover, under richness,
ISR and EFR coincide.
Since under richness no strict refinement of ISR is UHC,
and since EFR is both UHC and a refinement of ISR, the twosolution concepts must coincide under richness. Intuitively
this is because CSBR only induces a binding restriction on
player i’s conjectures about his opponents strategies undercommon knowledge assumptions. This is an important distinction, since it implies that the tension between robustness
and rationalization would only be assured to exist when rationalization has no effect at all.

3.

PERSONAL BASE SPACES OF UNCERTAINTY AND S-ROBUSTNESSThe second issue raised by the example is the importanceof considering the state space, and the inability to maintainpredictions under embeddings. Indeed, the initial intuitionof the example, that EFR is not UHC, is being driven notby a failure of convergence of standard types (since EFR it
is in fact UHC), but by the failure to foresee the effects ofchanging the space of strategic uncertainty. In light of this,
we propose a framework where players each have a subjective understanding of the space of payoff uncertainty. This
accomplishes two goals simultaneously. First, it allows forthe modeler to examine how changes (or mis-specifications)of the true state-space effect predictions. Second, it relaxes
the restriction that the state-space is commonly known.
In regards to the second point, it is worth noting thatwhile the richness assumption can be interpreted as relaxing
all common knowledge restrictions regarding the payoffs associated with a given strategy, it still imposes, rather restrictively, that all players commonly know that all such common
knowledge restrictions are relaxed. In other words, that all
players commonly know that it is not common knowledge
that any action is not dominant. Hence, by allowing thespace of uncertainty to be subjective, we relax the imposition that the state space is commonly known.
To do this we define each players directory. Beginningwith some fixed, objective state space Θ0 , a directory forplayer i assigns the subset of Θ0 that he understands tobe the true space of parameters, the subset he understands
each of his opponents to understand to be the true space,and so on. Notationally, let K(Θ0 ) be the set of all nonempty compact subsets of Θ0 , let I be the setSof players,
and for each n ∈ N let Λn = I n ; finally set Λ = n∈N Λn .

Definition 2. For each player i ∈ I, we say that Θi :
Λ → K(Θ0 ) is a directory for player i, if for any n ∈ N
and any λ ∈ Λn :
D1. If λ1 = i, then Θi (λ) = Θi (λk )nk=2 .
D2. If λk = λk+1 for some k < n, thenΘi (λ) = Θi ((λ1 , . . . , λk−1 , λk+1 , . . . , λn )) .
Let D(Θ0 ) be the set of all directories defined over Θ0 .
If Θi (j) = Θ1 then player i only takes into account Θ1
when conjecturing about j’s first order beliefs. Likewise, if
Θi (j, k) = Θ2 , then player i only takes into account Θ2 when
conjecturing about j’s second order beliefs about k’s first order beliefs. Under our interpretation, D1 states that eachplayer correctly understands his own understanding. Then,
D1 implies that Θi (i, j) = Θi (i, i, j) = Θ1 , etc. Similarly, D2
dictates that in each player’s mind all other players understand the restriction imposed by D1. Hence, player 1’s understands that player 2 understands his own understanding.
Hence, Θi (i, j, j, j) = Θi (i, j, j) = Θ1 , too. Note that eachplayer i’s directory Θi induces a directory for all of his opponents in his mind, namely Θj|i , where Θj|i (λ) = Θi (j, λ)for any Q
λ ∈ Λ and any opponent j. As usual, we denote
Θ−i|i = j6=i Θj|i .
Each directory, Θi , induces a type space, TiΘi , in a naturalway. TiΘi is the set of types (drawn from the universal spaceover Θ0 ) whose beliefs are concentrated on the appropriate
set assigned by the directory. Each player can be describedby a pair (Θi , τi ), where τi ∈ TiΘi .
By directly incorporating each players understanding of
the state space, it becomes easy to define s-robustness.
Definition 3. Let G be a game with incomplete informa(·)
tion. Then, we say that solution concept Si : Graph(Ti ) ⇒Si is s-robust, if for any player i, standard hierarchy τi , andn0
5sequence (Θni , si )n∈N ⊆ D(Θ ) × Si such that:
(i)
(ii)

Θn
τi ∈ Ti i

for all n,nsni ∈ Si (Θi , τi ) for all n,(iii) lim Θni = Θi , and,n→∞(iv) lim sn
i = si ,n→∞then si ∈ Si (Θi , τi ).
Just as UHC states that the solution concept should beconsistent in the limit of successive approximations of player’s
true type, s-robustenss imparts the same requirement on approximation of the player’s true understanding of the true
state-space. One can model the situation where some state
space, Θ1 , is commonly known by setting Θi (λ) = Θ1 for alli and λ. It is within this specific case that we can model embeddings of the game and get a more direct comparison to
informational invariance [6]. S-robustness is a significantly
weaker requirement than informational invariance, as it only
requires the solution concept to be resilient to small misspecifications. It should therefore not be too surprising that
EFR is also s-robust.
5
In the paper, we formally define the topology on D(Θ0 ).
Intuitively, it is the product topology generated by sequencesof K(Θ0 ), itself endowed with the Hausdorff metric.

Proposition 2. EFR is s-robust.
This result, like its analog for UHC, is proven by examining the epistemic demands of the robustness criterion. However, this involves a large notational burden, and so, is left
out of this abstract.

3.1

A Structure Theorem

Following the literature on robustness, we provide a structure theorem. While the motivation is similar, the notionof directories raises a new question: is the generic uniqueness of dominance solvability –being driven by relaxationof common knowledge conditions on strategic uncertainty–
still present under the relaxation of common knowledge conditions on payoff uncertainty. That is, can we still obtaina structure theorem if we allow players to understand thestate space as not being rich.
We provide a (partial) affirmative answer. We find strictly
weaker conditions under which a structure theorem still obtains.
Assumption 1 (Objective Richness). Θ0 satisfies theRichness Condition.6
Although we are assuming that the objective space of uncertainty is sufficiently rich, Assumption 1 contains no common knowledge restrictions. This is the direct result of thedisentangling of the objective space of uncertainty with players personal base spaces, as given by the directory.
Definition 4. We say that Θ ∈ K(Θ0 ) has strongly
generic payoffs if for any strategy si ∈ Si , there is a stateθ ∈ Θ, and a profile of opponents strategies s−i ∈ S−i , suchthat si is a strict best response to s−i at θ.
Assumption 2 (Common Knowledge of Genericity).
For all i and λ ∈ Λ, Θi (λ) has strongly generic payoffs.
Assumption 2 is, like richness, a common knowledge restriction on the space of uncertainty. However, it is a significantly weaker restriction. The relationship is as follows:richness dictates that it is not commonly known that any
action is not dominant, whereas genericity dictates that it
not commonly known that any action is dominant. Indeed,
Assumptions 1 and 2 allow for the situation where it is commonly known that no action is dominant. For example, asimple “Battle of the Sexes” game with no payoff uncertainty
would satisfy Assumption 2 but clearly fail to satisfy richness. Moreover, since richness implies genericity, if Θ0 is richand commonly known (i.e., as in previous literature) thenAssumptions 1 and 2 are both implied.
Proposition 3. Under assumptions 1 and 2, any strict
refinement of EFR is not UHC. Moreover, the set{(Θi , τi ) : |EFRi (Θi , τi )| = 1}(·)

is open and dense in Graph(Ti ).
Proposition 3 shows that richness need not be commonly
known to arrive at generic dominance solvability. The result relies on the ability to construct a sequence of directories that impose richness at increasingly high order understandings; then using Assumption 2 to cascade the effect6

As defined in [6].

236

down without losing convergence. It is worth pointing outthat there has been a recent interest in providing structure
theorems in the absence of richness [4, 7]. However, these
papers focus on the identifying which strategies can be selected when richness is not assumed, rather than weakening
the structural conditions on the space of uncertainty while
retaining that all rationalizable actions can be selected.

4.

CONCLUSION

In this paper we show a formal connection between the
epistemic characterization of a solution concept and its robustness to the misspecification of parameters. This provides both an important conceptual link and a direct method
for checking robustness when the epistemic characterization
is known. We use this result to show that EFR is UHC.
We also present a new framework that relaxes the common knowledge restrictions regarding the space of payoffparameters. Then, we propose a new type of robustness,s-robustness, to modeling errors of the player understandingof the space of uncertainty, which is of particular importance in dynamic environments. We characterize this notion through our epistemic framework and show that EFR
is also s-robust. Finally, we provide a structure theorem for
EFR with personal spaces of uncertainty that shows that nocommon knowledge assumptions regarding the existence ofdominance states are required to achieve generic dominancesolvability.

5. REFERENCES[1] P. Battigalli. “Strategic rationality order and the best
rationalization principle”. Games and Economic
Behavior, 13:178–200, 1996.
[2] P. Battigalli and M. Siniscalchi. “Hierarchies ofconditional beliefs and interactive epistemology indynamic games”. Journal of Economic Theory,,
88:188–230, 1999.
[3] A. Brandenburger and E. Dekel. “Hierarchies of beliefs
and common knowledge”. Journal of Economic Theory,
59:189–198, 1993.
[4] Y. C. Chen, S. Takahashi, and S. Xiong. The
weinstein-yildiz selection and robust predictions with
arbitrary payoff uncertainty. 2014.
[5] D. G. Pearce. “Rationalizable strategic behavior andthe problem of perfection”. Econometrica,
52:1029–1050, 1984.
[6] A. Penta. “Higher order uncertainty and information:
static and dynamic games”. Econometrica, 80:631–660,
2012.
[7] A. Penta. “On the structure of rationalizability forarbitrary spaces of uncertainty”. Theoretical Economics,
8:405–430, 2013.
[8] J. Weinstein and M. Yildiz. “A structure theorem forrationalizability with application to robust predictionsof refinements”. Econometrica, 75:365–400, 2007.

237

Parameterized Complexity Results for a Model of Theory of
Mind based on Dynamic Epistemic Logic∗
Iris van de Pol†

Institute for Logic, Language,
and Computation
University of Amsterdam

i.p.a.vandepol@uva.com

Iris van Rooij

Donders Institute for Brain,
Cognition, and Behaviour
Radboud University

i.vanrooij@donders.ru.nl

ABSTRACTIn this paper we introduce a computational-level model oftheory of mind (ToM) based on dynamic epistemic logic(DEL), and we analyze its computational complexity. The
model is a special case of DEL model checking. We provide a parameterized complexity analysis, considering several aspects of DEL (e.g., number of agents, size of preconditions, etc.) as parameters. We show that model checking
for DEL is PSPACE-hard, also when restricted to singlepointed models and S5 relations, thereby solving an open
problem in the literature. Our approach is aimed at formalizing current intractability claims in the cognitive science
literature regarding computational models of ToM.

Categories and Subject Descriptors
F.1.3 [Theory of Computation]: Complexity Measures & Classes—Reducibility and Completeness; F.4.1[Mathematical Logic and Formal Languages]: Mathematical Logic—Modal Logic; I.2.4 [Artificial Intelligence]: Knowledge Representation Formalisms and Methods—Modal Logic

General Terms
TheoryKeywords
Theory of mind; dynamic epistemic logic; computational
complexity; parameterized complexity; computational-level
model

1. INTRODUCTION
Imagine that you are in love. You find yourself at yourdesk, but you cannot stop your mind from wandering off.
What is she thinking about right now? And more importantly, is she thinking about you and does she know thatyou are thinking about her? Reasoning about other people’s knowledge, belief and desires, we do it all the time.
∗This research has been carried out in the context of thefirst author’s master’s thesis [37].
†Supported by Gravitation Grant 024.001.006 of the Language in Interaction Consortium from the Netherlands Organization for Scientific Research.
‡Supported by the Netherlands Organisation for Scientific
Research Veni Grant NWO-639-021-232.

239

Jakub Szymanik‡

Institute for Logic, Language,
and Computation
University of Amsterdam

j.k.szymanik@uva.nl

For instance, in trying to conquer the love of one’s life, tostay one step ahead of one’s enemies, or when we lose ourfriend in a crowded place and we find them by imagining
where they would look for us. This capacity is known as
theory of mind (ToM) and it is widely studied in various
fields (see, e.g., [8, 11, 23, 34, 36, 38, 47, 48]).
We seem to use ToM on a daily basis and many cognitive
scientists consider it to be ubiquitous in social interaction [1].
At the same time, however, it is also widely believed thatcomputational cognitive models of ToM are intractable, i.e.,that ToM involves solving problems that humans are not capable of solving (cf. [1, 27, 31, 50]). This seems to imply acontradiction between theory and practice: on the one hand
we seem to be capable of ToM, while on the other hand, ourtheories tell us that this is impossible. Dissolving this paradox is a critical step in enhancing theoretical understandingof ToM.
The question arises what it means for a computationallevel model1 of cognition to be intractable. When looking
more closely at these intractability claims regarding ToM, it
is not clear what these researchers mean exactly, nor whether
they mean the same thing. In theoretical computer scienceand logic there are a variety of tools to make precise claims
about the level of complexity of a certain problem. In cognitive science, however, this is a different story. With the
exception of a few researchers, cognitive scientists do nottend to specify formally what it means for a theory to be
intractable. This makes it often very difficult to assess thevalidity of the various claims in the literature about which
theories are tractable and which are not.
In this paper we adopt the Tractable-cognition thesis(see [42]) that states that people have limited resources
for cognitive processing and human cognitive capacities are
confined to those that can be realized using a realistic
amount of time.2 More specifically we adopt the FPT1
In cognitive science, often Marr’s [33] tri-level distinction between computational-level (“what is the natureof the problem being solved?”), algorithmic-level (“whatis the algorithm used for solving the problem?”), andimplementational-level (“how is the algorithm physically realized?”) is used to distinguish different levels of computational cognitive explanations. In this paper, we will focus oncomputational-level models of ToM and their computational
complexity.
2
There is general consensus in the cognitive science community that computational intractability is a undesirable
feature of cognitive computational models, putting the cognitive plausibility of such models into question [13, 24, 26,
42, 46]. There are diverging opinions about how cognitivecognition thesis [42] that states that computationally plausible computational-level cognitive theories are limited tothe class of input-output mappings that are fixed-parametertractable for one or more input-parameters that can be assumed to be small in practice. To be able to make more
precise claims about the (in)tractability of ToM we introduce a computational-level model of ToM based on dynamic
epistemic logic (DEL), and we analyze its computational
complexity. The model we present is a special case of DEL
model checking. Here we include an informal description of
the model.3 The kind of situation that we want to be able
to model, is that of an observer that observes one or moreagents in an initial situation. The observer then witnesses
actions that change the situation and the observer updates
their knowledge about the mental states of the agents in the
new situation. Such a set up is often found in experimental
tasks, where subjects are asked to reason about the mental
states of agents in a situation that they are presented.

The paper is structured as follows. In Section 2 we introduce basic definitions from dynamic epistemic logic andparameterized complexity theory. Then, in Section 3 weintroduce a formal description of our computational-level
model and we discuss the particular choices that we make.
Next, in Section 4 we present our (parameterized) complexity results. Finally, in Section 5 we discuss the implicationsof our results for the understanding of ToM.

DBU (informal) – Dynamic Belief Update
Instance: A representation of an initial situation, asequence of actions – observed by an observer – anda (belief) statement ϕ of interest.
Question: Is the (belief) statement ϕ true in thesituation resulting from the initial situation and theobserved actions?

Definition 2.1 (Epistemic model). Given a finite set A ofagents and a finite set P of propositions, an epistemic model
is a tuple (W, R, V ) where

We prove that DBU is PSPACE-complete. PSPACEcompleteness was already shown by Aucher and Schwarzentruber [3] for DEL model checking in general. They considered unrestricted relations and multi-pointed event models. Since their proof does not hold for the special case of
DEL model checking that we consider, we propose an alternative proof. Our proof solves positively the open question in [3] whether model checking for DEL restricted to S5
relations and single-pointed models is PSPACE-complete.
Bolander, Jensen and Schwarzentruber [10] independently
considered an almost identical special case of DEL model
checking (there called the plan verification problem). They
also prove PSPACE-completeness for the case restricted
to single-pointed models, but their proof does not settle
whether hardness holds even when the problem is restricted
to S5 models.
Furthermore, we investigate how the different aspects (orparameters, see Table 1) of our model influence its complexity. We prove that for most combinations of parameters DBU is fp-intractable and for one case we prove fptractability. See Figure 2 for an overview of the results.
Besides the parameterized complexity results for DEL
model checking that we present, the main conceptual contribution of this paper is that it bridges cognitive science andlogic, by using DEL to model ToM (cf. [28, 47]). By doing so, the paper provides the means to make more precise
statements about the (in)tractability of ToM.
science should deal with this issue (see, e.g., [12, 26, 41,43]). It is beyond the scope of this paper to discuss this indetail. In this paper we adopt the parameterized complexity
approach as described in [42].
3
We pose the model in the form of a decision problem, as
this is convenient for purposes of our complexity analysis.
Even though ToM may be more intuitively modeled by asearch problem, the complexity of the decision problem gives
us lower bounds on the complexity of such a search problem,
and therefore suffices for the purposes of our paper.

2.

PRELIMINARIES

2.1

Dynamic Epistemic Logic

Dynamic epistemic logic is a particular kind of modal logic(see [16, 6]), where the modal operators are interpreted interms of belief or knowledge. First, we define epistemic models, which are Kripke models with an accessibility relationfor every agent a ∈ A, instead of just one accessibility relation.

- W is a non-empty set of worlds;- R is a function that assigns to every agent a ∈ A abinary relation Ra on W ; and- V is a valuation function from W × P into {0, 1}.
The accessibility relations Ra can be read as follows: forworlds w, v ∈ W , wRa v means “in world w, agent a considers
world v possible.”
Definition 2.2 ((Multi and single-)pointed epistemic
model). A pair (M, Wd ) consisting of an epistemic
model M = (W, R, V ) and a non-empty set of designatedworlds Wd ⊆ W is called a pointed epistemic model. A pair
(M, Wd ) is called a single-pointed model when Wd is a singleton, and a multi-pointed epistemic model when |Wd | >1. By a slight abuse of notation, for (M, {w}), we also
write (M, w).
We consider the usual restrictions on relations in epistemic
models and event models, such as KD45 and S5 (see [16]).
In KD45 models, all relations are transitive, Euclidean andserial, and in S5 models all relations are transitive, reflexive
and symmetric.
We define the following language for epistemic models. We
use the modal belief operator B, where for each agent a ∈A, Ba ϕ is interpreted as “agent a believes (that) ϕ”.
Definition 2.3 (Epistemic language). The language LB
over A and P is given by the following definition, where aranges over A and p over P :ϕ ::= p | ¬ϕ | (ϕ ∧ ϕ) | Ba ϕ.
We will use the following standard abbreviations, > := p ∨¬p, ⊥ := ¬>, ϕ ∨ ψ := ¬(¬ϕ ∧ ¬ψ), ϕ → ψ := ¬ϕ ∨ ψ,Bˆa := ¬Ba ¬ϕ.
The semantics for this language is defined as follows.
Definition 2.4 (Truth in a (single-pointed) epistemic
model). Let M = (W, R, V ) be an epistemic model, w ∈W , a ∈ A, and ϕ, ψ ∈ LB . We define M, w |= ϕ inductively
as follows:240M, w |= pM, w |= ¬ϕM, w |= (ϕ ∧ ψ)M, w |= Ba ϕ

iff
iff
iff
iff

V (w, p) = 1
not M, w |= ϕM, w |= ϕ and M, w |= ψfor all v with wRa v: M, v |= ϕWhen M, w |= ϕ, we say that ϕ is true in w or ϕ is satisfied
in w.
Definition 2.5 (Truth in a multi-pointed epistemic model).
Let (M, Wd ) be a multi-pointed epistemic model, a ∈ A, andϕ ∈ LB . M, Wd |= ϕ is defined as follows:
M, Wd |= ϕiffM, w |= ϕ for all w ∈ Wd

Next we define event models.
Definition 2.6 (Event model). An event model is a tuple
E = (E, Q, pre, post), where E is a non-empty finite set of
events; Q is a function that assigns to every agent a ∈ A abinary relation Ra on W ; pre is a function from E into LB
that assigns to each event a precondition, which can be any
formula in LB ; and post is a function from E into LB thatassigns to each event a postcondition. Postconditions areconjunctions of propositions and their negations (including> and ⊥).
Definition 2.7 ((Multi and single-)pointed event model /action). A pair (E, Ed ) consisting of an event model E =(E, Q, pre, post) and a non-empty set of designated events
Ed ⊆ E is called a pointed event model. A pair (E, Ed ) is
called a single-pointed event model when Ed is a singleton,
and a multi-pointed event model when |Ed | > 1. We will
also refer to (E, Ed ) as an action.
We define the notion of a product update, that is used to
update epistemic models with actions [4].
Definition 2.8 (Product update). The product update of
the state (M, Wd ) with the action (E, Ed ) is defined as thestate (M, Wd ) ⊗ (E, Ed ) = ((W 0 , R0 , V 0 ), Wd0 ) where- W 0 = {(w, e) ∈ W × E ; M, w |= pre(e)};- Ra0 = {((w, e), (v, f )) ∈ W 0 × W 0 ; wRa v and eQa f };- V 0 (p) = 1 iff either (M, w |= p and post(e) 6|= ¬p) or
post(e) |= p; and- Wd0 = {(w, e) ∈ W 0 ; w ∈ Wd and e ∈ Ed }.

Finally, we define when actions are applicable in a state.
Definition 2.9 (Applicability). An action (E, Ed ) is applicable in state (M, Wd ) if there is some e ∈ Ed and somew ∈ Wd such that M, w |= pre(e). We define applicability for a sequence of actions inductively. The empty
sequence, consisting of no actions, is always applicable.
A sequence a1 , . . . , ak of actions is applicable in a state(M, Wd ) if (1) the sequence a1 , . . . , ak−1 is applicable in(M, Wd ) and (2) the action ak is applicable in the state(M, Wd ) ⊗ a1 ⊗ · · · ⊗ ak−1 .

2.2

Parameterized Complexity Theory

We introduce some basic concepts of parameterized complexity theory. For a more detailed introduction we refer totextbooks on the topic [17, 18, 22, 35].
Definition 2.10 (Parameterized problem). Let Σ be a finite
alphabet. A parameterized problem L (over Σ) is a subsetof Σ∗ × N. For an instance (x, k), we call x the main partand k the parameter.

241

The complexity class FPT, which stands for fixedparameter tractable, is the direct analogue of the class P inclassical complexity. Problems in this class are consideredefficiently solvable, because the non-polynomial-time complexity inherent in the problem is confined to the parameterand in effect the problem is efficiently solvable even for large
input sizes, provided that the value of the parameter is relatively small.
Definition 2.11 (Fixed-parameter tractable / the classFPT). Let Σ be a finite alphabet.
1. An algorithm A with input (x, k) ∈ Σ × N runs in fpttime if there exists a computable function f and a polynomial p such that for all (x, k) ∈ Σ × N, the running
time of A on (x, k) is at mostf (k) · p(|x|).
Algorithms that run in fpt-time are called fptalgorithms.
2. A parameterized problem L is fixed-parametertractable if there is an fpt-algorithm that decides L.
FPT denotes the class of all fixed-parameter tractableproblems.
Similarly to classical complexity, parameterized complexity also offers a hardness framework to give evidence that(parameterized) problems are not fixed-parameter tractable.
The following notion of reductions plays an important rolein this framework.
Definition 2.12 (Fpt-reduction). Let L ⊆ Σ × N and L0 ⊆
Σ0 ×N be two parameterized problems. An fpt-reduction fromL to L0 is a mapping R : Σ×N → Σ0 ×N from instances of L
to instances of L0 such that there is a computable function g :N → N such that for all (x, k) ∈ Σ × N:
1. (x0 , k0 ) = R(x, k) is a yes-instance of L0 if and only
if (x, k) is a yes-instance of L;
2. R is computable in fpt-time; and3. k0 ≤ g(k).
Another important part of the hardness framework is theparameterized intractability class W[1]. To characterize thisclass, we consider the following parameterized problem.
{k}-WSat[2CNF]
Instance: A 2CNF propositional formula ϕ and an
integer k.
Parameter: k.
Question: Is there an assignment α : var (ϕ) →{0, 1}, that sets k variables in var (ϕ) to true, thatsatisfies ϕ?
The class W[1] consists of all parameterized problems thatcan be fpt-reduced to {k}-WSat[2CNF]. A parameterized
problem is hard for W[1] if all problems in W[1] can be
fpt-reduced to it. It is widely believed that W[1]-hard problems are not fixed-parameter tractable [18]. Another parameterized intractability class, that can be used in a similar way, is the class para-NP. The class para-NP consists

of all parameterized problems that can be solved by a nondeterministic fpt-algorithm. To show para-NP-hardness, itsuffices to show that DBU is NP-hard for a constant valueof the parameters [21]. Problems that are para-NP-hard are
not fixed-parameter tractable, unless P = NP [22, Theorem 2.14].

We define our computational-level model of ToM as follows.
DBU (formal) – Dynamic Belief Update
Instance: A set of propositions P, and set ofAgents A.
An initial state so , where so =((W, V, R), Wd ) is a pointed epistemic model. An
applicable sequence of actions a1 , ..., ak , where aj =((E, Q, pre, post), Ed ) is a pointed event model. A
formula ϕ ∈ LB .
Question: Does so ⊗ a1 ⊗ ... ⊗ ak |= ϕ?

3. COMPUTATIONAL-LEVEL MODEL OF
THEORY OF MIND
Next we present a formal description of ourcomputational-level model. Our aim is to capture, ina qualitative way, the kind of reasoning that is necessary to
be able to engage in ToM. Arguably, the essence of ToM isthe attribution of mental states to another person, based on
observed behavior, and to predict and explain this behavior
in terms of those mental states. The aspect of ToM that we
aim to formalize with our model is the attribution of mental
states. There is a wide range of different kinds of mental
states such as epistemic, emotional and motivational states.
In our model we focus on epistemic states, in particular on
belief.
To be cognitively plausible, our model needs to be able
to capture a wide range of (dynamic) situations, where allkinds of actions can occur, not just actions that change beliefs (epistemic actions), but also actions that change thestate of the world (ontic actions). This is why, following
Bolander and Andersen [9], we use postconditions in the
product update of DEL (in addition to preconditions).
Furthermore, we want to model the (internal) perspectiveof the observer (on the situation). Therefore, the god perspective, also called the perfect external approach by Aucher[2] – that is inherent to single-pointed epistemic models – will
not suffice for all cases that we want to be able to model.
This perfect external approach supposes that the modeler isan omniscient observer that is perfectly aware of the actual
state of the world and the epistemic situation (what is goingon in the minds of the agents). The cognitively plausible
observers that we are interested in here will not have infallible knowledge in many situations. They are often not able
to distinguish the actual world from other possible worlds,
because they are uncertain about the facts in the world andthe mental states of the agent(s) that they observe. That
is why, again following Bolander and Andersen [9], we allow
for multi-pointed epistemic models (in addition to singlepointed models), which can model the uncertainty of an observer, by representing their perspective as a set of worlds.
How to represent the internal or fallible perspective of anagent in epistemic models is a conceptual problem that has
not been settled yet in the DEL-literature. There have been
several proposals to deal with this (see, e.g., [2, 15, 25]).
Also, since we do not assume that agents are perfectly
knowledgeable, we allow the possibility of modeling false
beliefs of the observers and agents, by using KD45 models (rather than S5 models). Even though KD45 models
present an idealized form of belief (with perfect introspection and logical omniscience), we argue that at least to someextent they are cognitively plausible, and that therefore, forthe purpose of this paper, it suffices to focus on KD45 models. Our complexity results (which we present in the next
section) do not depend on this choice; they hold for DBU
restricted to KD45 models and restricted to S5 models, andalso for the unrestricted case.

The model can be naturally used to formalize ToM tasks
that are employed in psychological experiments. The classical ToM task that is used by (developmental) psychologists
is the false belief task [5, 49]. The DEL-based formalizationof the false belief task by Bolander [8] can be seen as an instance of DBU. For more details on how DBU can be used
to model ToM tasks, we refer to [37].

4.

COMPLEXITY RESULTS

4.1

PSPACE-completenessWe show that DBU is PSPACE-complete. For this, weconsider the decision problem TQBF. This problem is
PSPACE-complete [45].
TQBF
Instance: A quantified Boolean formula ϕ
Q1 x1 Q2 x2 . . . Qm xm .ψ.
Question: Is ϕ true?

=

Theorem 1. DBU is PSPACE-hard.
Proof. To show PSPACE-hardness we specify apolynomial-time reduction R from TQBF to DBU.
Let ψ be a Boolean formula. First, we sketch the generalidea behind the reduction. We use the reduction to list all
possible assignments to var (ψ). To do this we use groups
of worlds (which are Ra -equivalence classes) to represent
particular truth assignments. Each group consists of astring of worlds that are fully connected by equivalence
relation Ra . Except for the first world in the string, allworlds represent a true variable xi (under a particularassignment).
We give an example of such a group of worlds that represents assignment α = {x1 7→ T, x2 7→ F, x3 7→ T, x4 7→
T, x5 7→ F, x6 7→ T}. Each world has a reflexive loop forevery agent, which we leave out for the sake of presentation. More generally, in all our drawings we replace eachrelation Ra with a minimal Ra0 whose transitive reflexive
closure is equal to Ra . marks the designated world. Since
all relations are reflexive, we draw relations as lines (leavingout arrows at the end).
y

y

1
a

a
w1

y

3
a
w2

y

4

6
a

w3

w4

We refer to worlds w1 , . . . , w4 as the bottom worlds of thisgroup. If a bottom world has an Ri relation to a world that242

makes proposition y true, we say that it represents variable xi .
The reduction makes sure that in the final updated model(the model that results from updating the initial state withthe actions – which are specified by the reduction) each possible truth assignment to the variables in ψ will be represented by a group of worlds. Between the different groups,
there are no Ra -relations (only Ri -relations for 1 ≤ i ≤ m).
By ‘jumping’ from one group (representing a particular truth
assignment) to another group with relation Ri , the truth
value of variable xi can be set to true or false. We can now
translate a quantified Boolean formula into a corresponding formula of LB by mapping every universal quantifier Qi
to Bi and every existential quantifier Qj to B̂j .
To illustrate how this reduction works, we give an example. Figure 1 shows the final updated model for a quantified
Boolean formula with variables x1 and x2 . In this model
there are four groups of worlds: {w1 , w2 , w3 }, {w4 , w5 },{w6 , w7 } and {w8 }. Worlds w1 , . . . , w8 are what we referto as the bottom worlds. The gray worlds and edges canbe considered a byproduct of the reduction; they have noparticular function.
We represent variable x1 by B̂1 y and variable x2
by B̂2 y. Then, in the model above, checking whether∃x1 ∀x2 .x1 ∨ x2 is true can be done by checking whether
formula B̂1 B2 (B̂a B̂1 y ∨ B̂aB̂2 y) is true, which is indeed thecase. Also, checking whether ∀x1 ∀x2 .x1 ∨ x2 is true can bedone by checking whether B1 B2 (B̂a B̂1 y ∨ B̂aB̂2 y) is true,
which is not the case.
Now, we continue with the formal details. Let ϕ =Q1 x1 . . . Qm xm .ψ be a quantified Boolean formula withquantifiers Q1 , . . . , Qm and var (ψ) = {x1 , . . . , xm }. We
define the following polynomial-time computable mappings.
For 1 ≤ i ≤ m, let [xi ] = B̂i y, and(Bi if Qi = ∀[Qi ] =
B̂i if Qi = ∃.
Formula [ψ] is the adaptation of formula ψ where every
occurrence of xi in ψ is replaced by B̂a [xi ]. Then [ϕ] =[Q1 ] . . . [Qm ][ψ]. We formally specify the reduction R. We
let R(ϕ) = (P, A, s0 , a1 , . . . , am , [ϕ]), where:- P = {y}, A = {a, 1, . . . , m}
y

-

1

s0 =

a

y

y
m

2
a

a

···

a

All relations in s0 , a1 , . . . , am are equivalence relations.
Note that all worlds in s0 , a1 , . . . , am have reflexive loops
for all agents. We omit all reflexive loops for the sake ofreadability.
-

a1 =

1
e1 : h>, >i e2 : h¬B̂1 y ∨ y, >i

..
.
m

-

am = e : h>, >i1e2 : h¬B̂m y ∨ y, >iholds. For any assignment α to the variables x1 , . . . , xi−1
and any bottom world w of a group that agrees with α,
the formula Qi xi . . . Qm xm .ψ is true under α if and only if[Qi ] . . . [Qm ][ψ] is true in world w. In the case for i = m + 1,
this refers to the formula [ψ].
We start with the case for i = m + 1. We show thatthe claim holds. Let α be any assignment to the variables x1 , . . . , xm , and let w be any bottom world of a group γthat represents α. Then, by construction of [ψ], we knowthat ψ is true under α if and only if [ψ] is true in w.
Assume that the claim holds for i = j + 1. We showthat then the claim also holds for i = j. Let α be any
assignment to the variables x1 , . . . , xj−1 and let w be abottom world of a group that agrees with α. We showthat the formula Qj . . . Qm .ψ is true under α if and only
if [Qj ] . . . [Qm ][ψ] is true in w.
First, assume that Qj . . . Qm .ψ is true under α. Consider
the case where Qj = ∀. Then for both assignments α0 ⊇ α
to the variables x1 , . . . , xj , formula Qj+1 . . . Qm .ψ is true
under α0 . Now, by assumption, we know that for any bottom
world w0 of a group that agrees with α – so in particular for
all bottom worlds w0 that are Rj -reachable from w – formula[Qj+1 ] . . . [Qm ][ψ] is true in w0 . Since [Qj ] = Bj , this meansthat [Qj ] . . . [Qm ][ψ] is true in w. The case where Qj = ∃ isanalogous.
Next, assume that Qj . . . Qm .ψ is not true under α.
Consider the case where Qj = ∀. Then there is someassignment α0 ⊇ α to the variables x1 , . . . , xj , suchthat Qj+1 . . . Qm .ψ is not true under α0 . Now, by assumption, we know that for any bottom world w0 of agroup that agrees with α – so in particular for somebottom world w0 that is Rj -reachable from w – formula [Qj+1 ] . . . [Qm ][ψ] is not true in w0 . Since [Qj ] = Bj ,this means that [Qj ] . . . [Qm ][ψ] is not true in w. The casewhere Qj = ∃ is analogous.
Hence, the claim holds for the case that i = j. Now, by
induction, the claim holds for the case that i = 1, and hence
it follows that ϕ ∈ TQBF if and only if R(ϕ) ∈ DBU. Since
this reduction runs in polynomial time, we can conclude that
DBU is PSPACE-hard.
Theorem 2. DBU is PSPACE-complete.
Proof. In order to show PSPACE-membership for the
problem DBU, we can modify the polynomial-space algorithm given by Aucher and Schwarzentruber [3]. Theiralgorithm works for the problem of checking whether agiven (single-pointed) epistemic model makes a given DELformula true, where the formula contains event models thatcan be multi-pointed, but that have no postconditions. In
order to make the algorithm work for multi-pointed epistemic models, we can simply call the algorithm several times,
once for each of the designated worlds. Also, a modificationis needed to deal with postconditions. The algorithm checks
the truth of a formula by inductively calling itself for subformulas. In order to deal with postconditions, only the casewhere the formula is a propositional variable needs to bemodified. This modification is rather straightforward. For
more details, we refer to [37].

4.2

We show that ϕ ∈ TQBF if and only if R(ϕ) ∈ DBU.
We prove that for all 1 ≤ i ≤ m + 1 the following claim
243

Parameterized Complexity Results

Next, we provide a parameterized complexity analysis of
DBU.

y

y

1

{x1 7→ T, x2 7→ T} :

a

w1

2

a

w2

1

{x1 7→ F, x2 7→ T} :

2
w3

1

y

2

2

1

2

y
1

a

w4
y

2

2

w5
y

2

2

1

{x1 7→ T, x2 7→ F} :

w6

a

1

w7

y

1

{x1 7→ F, x2 7→ F} :

1
y

w8

Figure 1: Example for the reduction in the proof of Theorem 1; a final updated model for a quantified Boolean
formula with variables x1 and x2 .

4.2.1

Parameters for DBU

We consider the following parameters for DBU. For eachsubset κ ⊆ {a, c, e, f, o, p, u} we consider the parameterizedvariant κ-DBU of DBU, where the parameter is the sumof the values for the elements of κ as specified in Table 1.
For instance, the problem {a}-DBU is parameterized by the
number of agents. Even though technically speaking there
is only one parameter, we will refer to each of the elementsof κ as parameters.
For the modal depth of a formula we count the maximum
number of nested occurrences of operators Ba . Formally,
we define the modal depth d(ϕ) of a formula ϕ (in LB )recursively as follows.
0
if ϕ = p ∈ P is a proposition;
max{d(ϕ1 ), d(ϕ2 )} if ϕ = ϕ1 ∧ ϕ2 ;d(ϕ) =
d(ϕ1 )
if ϕ = ¬ϕ1 ;
1 + d(ϕ1 )if ϕ = Ba ϕ1 .
For the size of a formula we count the number of occurrences of propositions and logical connectives. Formally, wedefine the size s(ϕ) of a formula ϕ (in LB ) recursively as
follows.
1
if ϕ = p ∈ P is a proposition;
1 + s(ϕ1 ) + s(ϕ2 ) if ϕ = ϕ1 ∧ ϕ2 ;s(ϕ) =

1 + s(ϕ1 )if ϕ = ¬ϕ1 ;
1 + s(ϕ1 )if ϕ = Ba ϕ1 .

4.2.2

Intractability Results

In the following, we show fixed-parameter intractability
for several parameterized versions of DBU. We will mainly
use the parameterized complexity classes W[1] and para-NP
to show intractability, i.e., we will show hardness for theseclasses. Note that we could additionally use the class paraPSPACE [21] to give stronger intractability results. For instance, the proof of Theorem 1 already shows that {p}-DBU
is para-PSPACE hard, since the reduction in this proof uses
a constant number of propositions. However, since in thispaper we are mainly interested in the border between fixedparameter tractability and intractability, we will not focus

Param.

Descriptionanumber of agentscmaximum size of the preconditionsemaximum number of events in the event models

fsize of the formulaomodal depth of the formula,i.e., the order parameterpnumber of propositions in Punumber of actions, i.e., the number of updates

Table 1: Overview of the different parameters for
DBU.
on the subtle differences in the degree of intractability, andrestrict ourselves to showing W[1]-hardness and para-NPhardness. This is also the reason why we will not show
membership for any of the (parameterized) intractability
classes; showing hardness suffices to indicate intractability.
For the following proofs we use the well-known satisfiability
problem Sat for propositional formulas. The problem Sat
is NP-complete [14, 30]. Moreover, hardness for Sat holds
even when restricted to propositional formulas that are in
3CNF.
Proposition 3. {a, c, e, f, o}-DBU is para-NP-hard.
Proof. To show para-NP-hardness, we specify apolynomial-time reduction R from Sat to DBU, whereparameters a, c, e, f , and o have constant values. Let ϕ
be a propositional formula with var (ϕ) = {x1 , . . . , xm }.
Without loss of generality we assume that ϕ is a 3CNF
formula with clauses c1 to cl .
The general idea behind this reduction is that we use theworlds in the final updated model (that results from updating the initial state with the actions – which are specified by the reduction) to list all possible assignments tovar (ϕ), by setting the propositions (corresponding to thevariables in var (ϕ)) to true and false accordingly. Thenchecking whether formula ϕ is satisfiable can be done by244checking whether ϕ is true in any of the worlds. Actions a1
to am are used to create a corresponding world for eachpossible assignment to var (ϕ). Furthermore, to keep theformula that we check in the final updated model of constant size, we sequentially check the truth of each clause ci
and encode whether the clauses are true with an additionalvariable xm+1 . This is done by actions am+1 to am+l . In
the final updated model, variable xm+1 will only be true ina world, if it makes clauses c1 to cl true, i.e., if it makes
formula ϕ true.
For more details, we refer to [37].
Proposition 4. {c, e, f, o, p}-DBU is para-NP-hard.
Proof. To show para-NP-hardness, we specify apolynomial-time reduction R from Sat to DBU, whereparameters c, e, f , o, and p have constant values. Let ϕ
be a propositional formula with var (ϕ) = {x1 , . . . , xm }.
The general idea behind this reduction is similar to the
reduction in the proof of Theorem 1. Again we use groups ofworlds to represent particular assignments to the variables
in ϕ. Here, there is only relation Rb between the different
groups. Furthermore, to keep the formula that we check in
the final updated model of constant size, we sequentiallycheck the truth of each clause ci and encode whether theclauses are true with an additional variable z. This is doneby actions am+1 to am+l . Action am+j (corresponding toclause j) marks each group of worlds (which represents aparticular assignment to the variables in ϕ) that ‘satisfies’clauses 1 to j. (This marking happens by means ofan Rc -accessible world where z is true.) Then, in the final
updated model, there will only be such a marked group if
all clauses, and hence the whole formula, is satisfiable.
For more details, we refer to [37].
Proposition 5. {a, e, f, o, p}-DBU is para-NP-hard.
Proof. To show para-NP-hardness, we specify apolynomial-time reduction R from Sat to DBU, whereparameters a, e, f , o and p have constant values. Let ϕ bea propositional formula with var (ϕ) = {x1 , . . . , xm }. The
reduction is based on the same principle as the one used in
the proof of Proposition 4. To keep the number of agents
constant, we use a different construction to represent thevariables in var (ϕ). We encode the variables by a stringof worlds that are connected by alternating relations Ra
and Rb .
Furthermore, we keep the size of the formula (and consequently the modal depth of the formula) constant by encoding the satisfiability of the formula with a single proposition.
We do this by adding an extra action am+1 . Action am+1
makes sure that each group of worlds that represents a satisfying assignment for the given formula, will have an Rc
relation from a world that is Rb -reachable from the designated world to a world where proposition z ∗ is true.
For more details, we refer to [37].
We consider the following parameterized problem, that
we will use in our proof of Proposition 6. This problem is
W[1]-complete [19].

245{k}-Multicolored CliqueInstance: A graph G, and a vertex-coloring c :V (G) → {1, 2, . . . , k} for G.
Parameter: k.
Question: Does G have a clique of size k including
vertices of all k colors? That is, are there v1 , . . . , vk ∈V (G) such that for all 1 ≤ i < j ≤ k : {vi , vj } ∈E(G) and c(vi ) 6= c(vj )?
Proposition 6. {a, c, f, o, u}-DBU is W[1]-hard.
Proof. We
specifyan
fpt-reduction
Rfrom{k}-Multicolored Clique to {a, c, f, o, u}-DBU. Let(G, c) be an instance of {k}-Multicolored Clique,where G = (N, E). The general idea behind this reductionis that we use the worlds in the model to list all k-sized
subsets of the vertices in the graph with k different colors,
where each individual world represents a particular k-subset
of vertices in the graph (with k different colors). Then
we encode (in the model) the existing edges between
these nodes (with particular color endings), and in the
final updated model we check whether there is a world
corresponding to a k-subset of vertices that is pairwise fully
connected with edges. This is only the case when G has
a k-clique with k different colors.
For more details, we refer to [37].
Proposition 7. {c, o, p, u}-DBU is W[1]-hard.
Proof. We specify the following fpt-reduction R from{k}-WSat[2CNF] to {c, o, p, u}-DBU. We sketch the general idea behind the reduction. Let ϕ be a propositional
formula with var (ϕ) = {x1 , . . . , xm }. Then let ϕ0 be theformula obtained from ϕ, by replacing each occurrence of xi
with ¬xi . We note that ϕ is satisfiable by some assignment αthat sets k variables to true if and only if ϕ0 is satisfiable bysome assignment α0 that sets m − k variables to true, i.e.,that sets k variables to false. We use the reduction to list
all possible assignments to var (ϕ0 ) = var (ϕ) that set m − kvariables to true. We represent each possible assignment
to var (ϕ) that sets m − k variables to true as a group ofworlds, like in the proof of Theorem 1. (In fact, due to thedetails of the reduction, in the final updated model, there
will be several identical groups of worlds for each of theseassignments).
For more details, we refer to [37].
Proposition 8. {a, f, o, p, u}-DBU is W[1]-hard.
Proof. We specify the following fpt-reduction R from{k}-WSat[2CNF] to {a, f, o, p, u}-DBU. We modify the reduction in the proof of Proposition 7 to keep the values ofparameters a and f constant. After these modifications, thevalue of parameter c will no longer be constant. To keep
the number of agents constant, we use the same strategy asin the reduction in the proof of Proposition 5, where variables xi , . . . , xm are represented by strings of worlds withalternating relations Rb and Ra . Just like in the proof of
Proposition 5, the size of the formula (and consequently themodal depth of the formula) is kept constant by encoding thesatisfiability of the formula with a single proposition. Theneach group of worlds that represents a satisfying assignment
for the given formula, will have an Rc relation from a world

that is Rb -reachable from the designated world to a world
where proposition z ∗ is true.
For more details, we refer to [37].

4.2.3Tractability Results

Next, we turn to a case that is fixed-parameter tractable.
Theorem 9. {e, u}-DBU is fixed-parameter tractable.
Proof. We present the following fpt-algorithm that runs intime eu · p(|x|), for some polynomial p, where e is the maximum number of events in the actions and u is the numberof updates, i.e., the number of actions.
As a subroutine, the algorithm checks whether a given basic epistemic formula ϕ holds in a given epistemic model M ,i.e., whether M |= ϕ. It is well-known that model checking
for basic epistemic logic can be done in time polynomial in
the of M plus the size of ϕ (see e.g. [7]).
Let x = (P, A, i, s0 , a1 , . . . , af , ϕ) be an instance of DBU.
First the algorithm computes the final updated model sf =s0 ⊗ a1 ⊗ · · · ⊗ af by sequentially performing the updates.
For each i, si is defined as si−1 ⊗ ai . The size of each si is
upper bounded by O(|s0 | · eu ), so for each update checking
the preconditions can be done in time polynomial in eu · |x|.
This means that computing sf can be done in fpt-time.
Then, the algorithm decides whether ϕ is true in sf . Thiscan be done in time polynomial in the size of sf plus thesize of ϕ. We know that |sf | + |ϕ| is upper bounded by
O(|s0 | · eu ) + |ϕ|, thus upper bounded by eu · p(|x|), for somepolynomial p. Therefore, deciding whether ϕ is true in sf
is fixed-parameter tractable. Hence, the algorithm decides
whether x ∈ DBU and runs in fpt-time.

4.2.4

Overview of the Results

We showed that DBU is PSPACE-complete, we presented several parameterized intractability results (W[1]hardness and para-NP-hardness) and we presented one
fixed-parameter tractable result, namely for {e, u}-DBU.
In Figure 2, we present a graphical overview of our results
and the consequent border between fpt-tractability and fptintractability for the problem DBU. We leave {a, c, p}-DBU
and {c, f, p, u}-DBU as open problems for future research.

5. DISCUSSION & CONCLUSIONSWe presented the Dynamic Belief Update model as acomputational-level model of ToM and analyzed its complexity. The aim of our model was to provide a formal
approach that can be used to interprete and evaluate themeaning and veridicality of various complexity claims in the
cognitive science and philosophy literature concerning ToM.
In this way, we hope to contribute to disentangling debatesin cognitive science and philosophy regarding the complexityof ToM.
In Section 4.1, we proved that DBU is PSPACE-complete.
This means that (without additional constraints), there is
no algorithm that computes DBU in a reasonable amountof time. In other words, without restrictions on its input
domain, the model is computationally too hard to serve asa plausible explanation for human cognition. This may not
be surprising, but it is a first formal proof backing up thisclaim, whereas so far claims of intractability in the literatureremained informal.
Informal claims about what constitutes sources of intractability abound in cognitive science. For instance, it

seems to be folklore that the ‘order’ of ToM reasoning (i.e.,that I think that you think that I think . . . ) is a potential
source of intractability. The fact that people have difficulty
understanding higher-order theory of mind [20, 29, 32, 44]
is not explained by the complexity results for parameter o– the modal depth of the formula that is being considered,in other words, the order parameter. Already for a formulawith modal depth one, DBU is NP-hard; so {o}-DBU is not
fixed-parameter tractable. On the basis of our results we canonly conclude that DBU is fixed-parameter tractable for theorder parameter in combination with parameters e and u.
But since DBU is fp-tractable for the smaller parameterset {e, u}, this does not indicate that the order parameter isa source of complexity. This does not mean it may not be asource of difficulty for human ToM performance. After all,tractable problems can be too resource-demanding for humans for other reasons than computational complexity (e.g.,
due to stringent working-memory limitations).
Surprisingly, we only found one (parameterized) tractability result for DBU. We proved that for parameter set {e, u}– the maximum number of events in an event model and thenumber of updates, i.e., the number of event models – DBU
is fixed-parameter tractable. Given a certain instance x of
DBU, the values of parameters e and u (together with thesize of initial state s0 ) determine the size of the final updated model (that results from applying the event models
to the initial state). Small values of e and u thus make surethat the final updated model does not blow up too muchin relation to the size of the initial model. The result that{e, u}-DBU is fp-tractable indicates that the size of the final updated model can be a source of intractability (cf. [39,40]).
The question arises how we can interpret parameters e andu in terms of their cognitive counterparts. To what aspect of
ToM do they correspond, and moreover, can we assume thatthey have small values in (many) real-life situations? If thisis indeed the case, then restricting the input domain of themodel to those inputs that have sufficiently small values forparameters e and u will render our model tractable, and we
can then argue that (at least in terms of its computational
complexity) it is a cognitively plausible model.
In his formalizations of the false belief task Bolander [8] indeed used a limited amount of actions with a limited amountof events in each action (he used a maximum of 4). This
could, however, be a consequence of the over-simplification(of real-life situations) used in experimental tasks. Whether
these parameters in fact have sufficiently small values in real
life, is an empirical hypothesis that can (in principle) be
tested experimentally. However, it is not straightforward
how to interpret these formal aspects of the model in termsof their cognitive counterparts. The associations that thewords event and action trigger with how we often use thesewords in daily life, might adequately apply to some degree,
but could also be misleading. A structural way of interpreting these parameters is called for. We think this is aninteresting topic for future research.
Besides the role that our results play in the investigationof (the complexity) of ToM our results are also of interest
in and of themselves. The results in Theorems 1 and 2 resolve an open question in the literature about the computational complexity of DEL. Aucher and Schwarzentruber [3]already showed that the model checking problem for DEL,in general, is PSPACE-complete. However, their proof for

246

∅
{p}
{a, p}{u}{c, p}{c, p, u} {f, p, u}{e}{a, e, f, o, p}{c, e, f, o, p}{c, o, p, u}{a, c, e, f, o}{a, c, f, o, u}{e, u}{a, c, p}{a, f, o, p, u}

fp-intractable
fp-tractable{c, f, p, u}{a, c, e, f, o, p, u}
Figure 2: Overview of the parameterized complexity results for the different parameterizations of DBU,
and the line between fp-tractability and fp-intractability (under the assumption that the cases for {a, c, p}and {c, f, p, u} are fp-tractable).
PSPACE-hardness does not work when the input domain isrestricted to S5 (or KD45) models and their hardness proof
also relies on the use of multi-pointed models (which in theirnotation is captured by means of a union operator). With
our proof of Theorem 1, we show that DEL model checking is PSPACE-hard even when restricted to single-pointed
S5 models. Furthermore, the novelty of our aproach lies in
the fact that we apply parameterized complexity analysis to
dynamic epistemic logic, which is still a rather unexploredarea.

Acknowledgements
We thank the reviewers for their comments. We thankThomas Bolander, Nina Gierasimczuk, Ronald de Haan andMartin Holm Jensen and the members of the Computational
Cognitive Science group at the Donders Centre for Cognition
for discussions and feedback.

6.

REFERENCES[1] I. Apperly. Mindreaders: the cognitive basis of “Theoryof Mind”. Psychology Press, 2011.
[2] G. Aucher. An internal version of epistemic logic.
Studia Logica, 94(1):1–22, 2010.
[3] G. Aucher and F. Schwarzentruber. On the complexityof dynamic epistemic logic. In Proceedings of the
Fourteenth Conference on Theoretical Aspects ofRationality and Knowledge (TARK), 2013.
[4] A. Baltag, L. S. Moss, and S. Solecki. The logic ofpublic announcements, common knowledge, andprivate suspicions. In Proceedings of the 7th
Conference on Theoretical Aspects of Rationality andKnowledge (TARK), 1998.

247[5] S. Baron-Cohen, A. M. Leslie, and U. Frith. Does theautistic child have a “theory of mind”? Cognition,21(1):37–46, 1985.
[6] J. van Benthem. Logical Dynamics of Information andInteraction. Cambridge University Press, Cambridge,
2011.
[7] P. Blackburn, J. van Benthem, et al. Modal logic: A
semantic perspective. Handbook of modal logic, 3:1–84,
2006.
[8] T. Bolander. Seeing is believing: Formalisingfalse-belief tasks in dynamic epistemic logic. In
Proceedings of European Conference on Social
Intelligence (ECSI 2014), pages 87–107, 2014.
[9] T. Bolander and M. B. Andersen. Epistemic planningfor single and multi-agent systems. Journal of Applied
Non-Classical Logics, 21(1):9–34, 2011.
[10] T. Bolander, M. H. Jensen, and F. Schwarzentruber.
Complexity results in epistemic planning. In
Proceedings of 24th International Joint Conference on
Artificial Intelligence (IJCAI 2015), 2015.
[11] T. Braüner. Hybrid-logical reasoning in false-belieftasks. In B. Schipper, editor, Proceedings of the
Fourteenth Conference on Theoretical Aspects ofRationality and Knowledge (TARK), 2013.
[12] N. Chater, J. B. Tenenbaum, and A. Yuille.
Probabilistic models of cognition: Conceptual
foundations. Trends in Cognitive Sciences,10(7):287–291, 2006.
[13] C. Cherniak. Minimal rationality. MIT Press, 1990.
[14] S. A. Cook. The complexity of theorem proving
procedures. In Proceedings of the 3rd Annual ACM
Symposium on the Theory of Computing (STOC),

pages 151–158. ACM, 1971.
[15] C. Dégremont, L. Kurzen, and J. Szymanik. Exploring
the tractability border in epistemic tasks. Synthese,
191(3):371–408, 2014.
[16] H. van Ditmarsch, W. van der Hoek, and B. P. Kooi.
Dynamic Epistemic Logic. Springer, 2008.
[17] R. G. Downey and M. R. Fellows. ParameterizedComplexity. Monographs in Computer Science.
Springer, New York, 1999.
[18] R. G. Downey and M. R. Fellows. Fundamentals of
Parameterized Complexity. Texts in Computer
Science. Springer, 2013.
[19] M. R. Fellows, D. Hermelin, F. A. Rosamond, and
S. Vialette. On the parameterized complexity ofmultiple-interval graph problems. TheoreticalComputer Science, 410(1):53–61, 2009.
[20] L. Flobbe, R. Verbrugge, P. Hendriks, and I. Krämer.
Children’s application of theory of mind in reasoning
and language. Journal of Logic, Language andInformation, 17(4):417–442, 2008.
[21] J. Flum and M. Grohe. Describing parameterized
complexity classes. Information and Computation,
187(2):291–319, 2003.
[22] J. Flum and M. Grohe. Parameterized Complexity
Theory, volume XIV of Texts in Theoretical Computer
Science. An EATCS Series. Springer, Berlin, 2006.
[23] U. Frith. Mind blindness and the brain in autism.
Neuron, 32(6):969–979, 2001.
[24] M. Frixione. Tractable competence. Minds andMachines, 11(3):379–397, 2001.
[25] N. Gierasimczuk and J. Szymanik. A note on a
generalization of the muddy children puzzle. In
Proceedings of the 13th Conference on Theoretical
Aspects of Rationality and Knowledge (TARK), 2011.
[26] G. Gigerenzer. Why heuristics work. Perspectives on
psychological science, 3(1):20–29, 2008.
[27] W. F. G. Haselager. Cognitive Science and Folk
Psychology: The Right Frame of Mind. Sage
Publications, 1997.
[28] A. M. Isaac, J. Szymanik, and R. Verbrugge. Logic
and complexity in cognitive science. In A. Baltag and
S. Smets, editors, Johan van Benthem on Logic andInformation Dynamics, volume 5 of Outstanding
Contributions to Logic, pages 787–824. Springer
International Publishing, 2014.
[29] P. Kinderman, R. Dunbar, and R. P. Bentall.
Theory-of-mind deficits and causal attributions.
British Journal of Psychology, 89(2):191–204, 1998.
[30] L. A. Levin. Universal sequential search problems.
Problems of Information Transmission, 9(3):265–266,
1973.
[31] S. C. Levinson. On the human ‘interaction engine’. In
N. J. Enfield and S. C. Levinson, editors, Roots ofhuman sociality: Culture, cognition and interaction,
pages 39–69. Oxford: Berg, 2006.
[32] M. Lyons, T. Caldwell, and S. Shultz. Mind-reading
and manipulation – is machiavellianism related totheory of mind? Journal of Evolutionary Psychology,
8(3):261–274, 2010.
[33] D. Marr. Vision: A computational investigation into
the human representation and processing of visual

information. San Francisco: WH Freeman, 1982.
[34] S. Nichols and S. P. Stich. Mindreading: An integrated
account of pretence, self-awareness, and understanding
other minds. Clarendon Press/Oxford University
Press, 2003.
[35] R. Niedermeier. Invitation to Fixed-Parameter
Algorithms. Oxford Lecture Series in Mathematics andits Applications. Oxford University Press, 2006.
[36] A. Perea. Epistemic Game Theory: reasoning andchoice. Cambridge University Press, 2012.
[37] I. van de Pol. How Difficult is it to Think that you
Think that I Think that ...? A DEL-based
Computational-level Model of Theory of Mind and its
Complexity. Master’s thesis, University of Amsterdam,
the Netherlands, 2015.
[38] D. Premack and G. Woodruff. Does the chimpanzee
have a theory of mind? Behavioral and brain sciences,1(04):515–526, 1978.
[39] I. van Rooij, P. Evans, M. Müller, J. Gedge, andT. Wareham. Identifying sources of intractability incognitive models: An illustration using analogicalstructure mapping. In Proceedings of the 30th Annual
Conference of the Cognitive Science Society, pages
915–920, 2008.
[40] I. van Rooij and T. Wareham. Parameterized
complexity in cognitive modeling: Foundations,applications and opportunities. The Computer
Journal, 51(3):385–404, 2008.
[41] I. van Rooij, C. D. Wright, J. Kwisthout, andT. Wareham. Rational analysis, intractability, and theprospects of ‘as if’-explanations. Synthese, pages 1–20,
2014.
[42] I. van Rooij. The tractable cognition thesis. Cognitive
Science, 32(6):939–984, 2008.
[43] I. van Rooij, C. D. Wright, and T. Wareham.
Intractability and the use of heuristics in psychologicalexplanations. Synthese, 187(2):471–487, 2012.
[44] J. Stiller and R. I. Dunbar. Perspective-taking andmemory capacity predict social network size. Social
Networks, 29(1):93–104, 2007.
[45] L. J. Stockmeyer and A. R. Meyer. Word problems
requiring exponential time (preliminary report). In
Proceedings of the 5th Annual ACM Symposium on the
Theory of Computing (STOC), pages 1–9. ACM, 1973.
[46] J. K. Tsotsos. Analyzing vision at the complexity
level. Behavioral and Brain Sciences, 13(03):423–445,
1990.
[47] R. Verbrugge. Logic and social cognition: the facts
matter, and so do computational models. Journal of
Philosophical Logic, 38(6):649–680, 2009.
[48] H. M. Wellman, D. Cross, and J. Watson.
Meta-analysis of theory-of-mind development: The
truth about false belief. Child Development,
72(3):655–684, 2001.
[49] H. Wimmer and J. Perner. Beliefs about beliefs:
Representation and constraining function of wrong
beliefs in young children’s understanding of deception.
Cognition, 13(1):103–128, 1983.
[50] T. W. Zawidzki. Mindshaping: A New framework forunderstanding human social cognition. MIT Press,
2013.

248

A Dynamic Epistemic Framework for Conformant Planning
Quan Yu1,2 , Yanjun Li3,4 and Yanjing Wang∗3
1

Department of Computer Science, Sun Yat-sen University, China
234

Qiannan Normal College for Nationalities, China

Department of Philosophy, Peking University, China

Faculty of Philosophy, University of Groningen, The Netherlands

ABSTRACTIn this paper, we introduce a lightweight dynamic epistemic
logical framework for automated planning under initial uncertainty. We reduce plan verification and conformant planning to model checking problems of our logic. We show thatthe model checking problem of the iteration-free fragment is
PSPACE-complete. By using two non-standard (but equivalent) semantics, we give novel model checking algorithms
to the full language and the iteration-free language.

1. INTRODUCTION
Conformant planning is the problem of finding a linear
plan (a sequence of action) to achieve a goal in presence of
uncertainty about the initial state (cf. [28]). For example,
suppose that you are a rookie spy trapped in a foreign hotelwith the following map at hand:1

s1

r

sO 6

s7 :SafeO

s8 :Safe
O

u

u

u

/ s2

/ s3

/ s4 :Safe r

r

r

/ s5

Now somebody spots you and sets up the alarm. In thiscase you need to move fast to one of the safe hiding places
marked in the map (i.e., s7 , s8 and s4 ). However, since
you were in panic, you lost your way and you are not surewhether you are at s2 or s3 (denoted by the circle in theabove graph). Now what should you do in order to reacha safe place quickly? Clearly, merely moving r or moving
u may not guarantee your safety given the uncertainty. A
simple plan is to move r first and then u, since this plan willtake you to a safe place, no matter where you actually are
initially. This plan is conformant since it does not require
any feedback during the execution and it should work inpresence of uncertainty about the initial state. More generally, a conformant plan should also work given actions with
non-deterministic effects. Such a conformant plan is crucial
when there are no feedbacks/observations available during
the execution of the plan.2 Note that since no information∗

Corresponding author
It is a variant of the running example in [33].
2
In many other cases, feedbacks may be just too ‘expensive’
to obtain during a plan aiming for quick actions [8].
1249

is provided during the execution, the conformant plan issimply a finite sequence of actions without any conditional
moves.
As discussed in [9, 24], conformant planning can be reduced to classical planning, the planning problem without
any initial uncertainty, over the space of belief states. Intuitively, a belief state is a subset of the state space, which
records the uncertainty during the execution of a plan, e.g.,{s2 , s3 } is an initial belief state in the above example. In
order to make sure a goal is achieved eventually, it is crucial
to track the transitions of belief states during the executionof the plan, and this may traverse exponentially many belief
states in the size of the original state space. As one may expect, conformant planning is computationally harder thanclassical planning. The complexity of checking the existenceof a conformant plan is EXPspace-complete in the size ofthe variables generating the state space [18]. In the literature, people proposed compact and implicit representationsof the belief spaces, such as OBDD [13, 15, 14] and CNF[30], and different heuristics are used to guide the search fora plan, e.g., [11, 12].
Besides the traditional AI approaches, we can also take anepistemic-logical perspective on planning in presence of initial uncertainties, based on dynamic epistemic logic (DEL)(cf. e.g., [31]). The central philosophy of DEL takes themeaning of an action as the change it brings to the knowledge of the agents. Intuitively, this is what we need to track
the belief states during the execution of a plan3 . Indeed,in recent years, there has been a growing interest in using DEL to handle multi-agent planning with knowledge
goals (cf. e.g., [7, 23, 1, 3, 34, 25]), while the traditional
AI planning focuses on the single-agent case. In particular,
the event models of DEL (cf. [6]) are used to handle nonpublic actions that may cause different knowledge updates todifferent agents. In these DEL-based planning frameworks,
states are epistemic models, actions are event models andthe state transitions are implicitly encoded by the update
product which computes a new epistemic model based onan epistemic model and an event model.
One advantage of this approach is its expressiveness inhandling scenarios which require reasoning about agents’higher-order knowledge about each other in presence of partially observable actions. However, this expressiveness comesat a price, as shown in [7, 4], that multi-agent epistemic plan3

Here the belief states are actually about knowledge in epistemic logic.

ning is undecidable in general. Many interesting decidable
fragments are found in the literature [7, 23, 34, 2], whichsuggests that the single-agent cases and restrictions on theform of event models are the key to decidability. However,
if we focus on the single-agent planning, a natural question
arises: how do we compare such DEL approaches with thetraditional AI planning? It seems that the DEL-based approaches are more suitable for planning with actions thatchange (higher-order) knowledge rather than planning withfact-changing actions, although the latter type of actions canalso be handled in DEL. Moreover, the standard models of
DEL are purely epistemic thus do not encode the temporal
information of available actions directly. This may limit theapplicability of such approaches to planning problems basedon transition systems.

• By reducing conformant planning to a model checking
problem in an explicit logical language, we also see the subtleties hidden in the planning problem. In principle, there
are various model checking techniques to be applied to conformant planning based on this reduction.
• Our logical language and models are very simple compared to the standard action-model based DEL approach,yet we can encode the externally given executability of theactions in the model, inspired by epistemic temporal logic(ETL) [16, 26].
• Our approach is flexible enough to provide, in the future,a unified platform to compare different planning problems
under uncertainty. By studying different fragments of thelogical language and model classes, we may categorize planning problems according to their complexity.

In this paper, we tackle the standard single-agent conformant planning problem over transition systems, by using
the core idea of DEL, but not its standard formalism. Ourformal framework is based on the logic proposed by Wang
and Li in [33], where the model is simply a transition systemwith initial uncertainty as in the motivating example, andan action is interpreted in the semantics as an update on theuncertainty of the agent. Our contributions are summarized
as follows:

The rest of the paper is organized as follows: We introduce
our basic logical framework and its axiomatization in Section
2, and extend it in Section 3 with programs to handle theconformant planning. The complexity analysis of the model
checking problems is in Section 4 and we conclude in Section
5 with future directions.

• A lightweight dynamic epistemic framework with a simple language and a complete axiomatization.
• Non-trivial reduction of conformant planning to a model
checking problem using our language with programs.
• Two novel model checking algorithms based on two alternative semantics for the proposed logic, which make thecontext-dependency in the original semantics explicit.
• The complexity of model checking the iteration-free fragment of our language is Pspace-complete. The model checking problem of the full language is in EXPtime. The model
checking problem of the conformant planning is in Pspace.
The last result may sound contradictory to the aforementioned result that the complexity of conformant planning is
EXPspace-complete. Actually, the apparent contradictionis due to the fact that the EXPspace complexity result isbased on the number of state variables which require an exponential blow up to generate an explicit transition systemthat we use here. We will come back to this issue at the endof Section 4.3.
Our approach has the following advantages compared tothe existing planning approaches:• The planning goals can be specified as arbitrary formulas
in an epistemic language. Extra plan constraints (e.g., what
actions to use) can be expressed explicitly by programs in
the language. Therefore it may cover a richer class of (conformant) planning problems compared to the traditional AI
approach where a goal is Boolean.4• The plans can be specified as regular expressions withtests in terms of arbitrary EPDL formulas, which generalizes the knowledge-based programs in [17, 21].
4
The goal in the standard conformant planning is simply aset of different valuations of basic propositional variables.
Our approach can even handle epistemic goals in negative
forms, e.g., we want to make sure the agent knows something
but does not know too much in the end.

2.

BASIC FRAMEWORK

2.1

Epistemic action language

To talk about the knowledge of the agent during an execution of a plan, we use the following language proposed in[33].
Definition 2.1 (Epistemic Action Language (EAL)).
Given a countable set A of action symbols and a countable set
P of atomic proposition letters , the language EALAP is defined
as follows:5
φ ::= > | p | ¬φ | (φ ∧ φ) | [a]φ | Kφ,
where p ∈ P, a ∈ A. The following standard abbreviations
are used: ⊥ := ¬>, φ ∨ ψ := ¬(¬φ ∧ ¬ψ), φ → ψ := ¬φ ∨ψ, haiφ := ¬[a]¬φ, K̂φ := ¬K¬φ.
Kφ says that the agent knows that φ, and [a]φ expresses
that if the agent can move forward by action a, then after
doing a, φ holds. Throughout the paper, we fix some P andA, and refer to EALAP by EAL.
The size of EAL-formulas (notation |ϕ|) is defined inductively: |>| = |p| = 1; |¬φ| = 1 + |φ|; |φ ∧ ψ| = 1 + |φ| + |ψ|;|Kφ| = |[a]φ| = 1 + |φ|. The set of subformulas of φ ∈ EAL,
denoted as sub(φ), is defined as usual.
Definition 2.2 (Uncertainty map). Given P and A,
a (multimodal) Kripke model N is a tuple hS, {Ra | a ∈A}, Vi, where S is a non-empty set of states, Ra ⊆ S × S
is a binary relation labelled by a, V : S → 2P is a valuation function. An uncertainty map M is a Kripke modelhS, {Ra | a ∈ A}, Vi with a non-empty set U ⊆ S. Givenan uncertainty map M, we refer to its components by SM ,RaM , VM , and UM . A pointed uncertainty map M, s is anuncertainty map M with a designated state s ∈ UM . Weawrite s → t for (s, t) ∈ Ra .

5
We do need unboundedly many action symbols to encode
the desired problem in the later discussion of model checking
complexity.

250

Intuitively, a Kripke model encodes a map (transition system) and the uncertainty set U encodes the uncertainty thatthe agent has about where he is in the map. The graphmentioned at the beginning of the introduction is a typicalexample of an uncertainty map. Note that there may be
non-deterministic transitions in the model, i.e., there mayaa
be t1 6= t2 such that s → t1 and s → t2 for some s, t1 , t2 .

M, s  >alwaysM, s  p⇐⇒ s ∈ V(p)M, s  ¬φ⇐⇒ M, s 2 φ
M, s  φ ∧ ψ ⇐⇒ M, s  φ and M, s  ψaM, s  [a]φ ⇐⇒ ∀t ∈ S : s → t implies M|a , t  φ
M, s  Kφ ⇐⇒ ∀u ∈ U : M, u  φwhere M|a = hS, {Ra | a ∈ A}, V, U|a i and U|a = {r0 | ∃r ∈a
U such that r → r0 }. We say φ is valid (notation:  φ) if it
is true on all the pointed uncertainty maps. For a action sequence σ = a1 . . . an , we write U|σ for (. . . ((U|a1 )|a2 ) . . . )|an .
and write M|σ for (. . . ((M|a1 )|a2 ) . . . )|an .
Intuitively, the agent ‘carries’ the uncertainty set with him
when moving forward and obtains a new uncertainty set
U |a . Note that here we differ from [33] where the updated
uncertainty set is further refined according to what the agent
can observe at the new state. For conformant planning, we
do not consider the observational power of the agent during
the execution of a plan.
Let us call the model mentioned in the introduction M,
it is not hard to see that M|r and (M|r )|u are as follows:

s1

s1

r

r

sO6

s7 :SafeO

s8 :Safe
O

u

u

u

/ s2

r

/ s3

r

/ s4 :Safe

sO6

s7 :SafeO

s8 :Safe
O

u

u

u

/ s2

r

/ s3

r

/ s4 :Safe

r

/ s5

r

/ s5

b

/ s3 : p
<

a

a

a



b

→



s2

Remark 1. It is crucial to notice that the designated state
in a pointed uncertainty map must be one of the states in theuncertainty set.
Definition 2.3 (Semantics). Given any uncertainty
map M = hS, {Ra | a ∈ A}, V, Ui and any state s ∈ U,
the semantics is defined as follows:

s1

s4
b

/ s3 : p
<

a

a

a

s2

b

a

a

/ s3 : p
<
a





s2

s1



s1

a

→



s4

a

→

s4

s1

b

a

a

/ s3 : p
<



a



s2

s4

Let the left-hand-side model be M then it is clear that
M|b , s3  Kp while M|aa , s3 2 Kp thus M, s1  hbiKp ∧haihai¬Kp. This shows that the truth value of an epistemic
subformula w.r.t. a state in the model is somehow ‘contextdependent’, which requires new techniques in model checking. We will make this explicit in Section 4.3 when we discuss the model checking algorithm.

2.2

Axiomatization

Following the axioms proposed in [33], we give the following axiomatization for EAL w.r.t. our semantics:

AxiomsSystem SELA

Rules

TAUT

all axioms of propositional logic

MP

DISTK

K(p → q) → (Kp → Kq)

NECK

DIST(a)[a](p → q) → ([a]p → [a]q)NEC(a)

T

Kp → pSUB

4

Kp → KKp

5¬Kp → K¬Kp

PR(a)K[a]p → [a]Kp

NM(a)haiKp → K[a]pφ, φ → ψ
ψ
φ
Kφ
φ
[a]φ
φ(p)
φ(ψ)

where a ranges over A, p, q range over P. PR(·) and NM(·) denote the axioms of perfect recall and no miracles respectively(cf. [32]).
Note that since we do not assume that the agent can observe the available actions, the axiom OBS(a) : Khai> ∨K¬hai> in [33] is abandoned. Due to the same reason, theaxiom of no miracles is also simplified.
We show the completeness of SELA using a more direct
proof strategy compared to the one used in [33].

Thus we have:

Theorem 2.1. SELA is sound and strongly complete w.r.t.
EAL on uncertainty maps.

• M, s3  [r](Safe ∧ ¬KSafe)• M, s3  K[r][u](Safe ∧ KSafe)
The usual global model checking algorithm for modal logics labels the states with the subformulas that are true on
the states. However, this cannot work here since the truth
value of epistemic formulas on the states outside U is simply
undefined. Moreover, the exact truth value of an epistemic
formula on a state depends on ‘how you get there’, as thefollowing example shows (the underlined states mark theactual states):

251

Proof. To prove that SELA is sound on uncertainty maps,
we need to show that all the axioms are valid and all the inference rules preserve validity. Since the uncertainty set in
an UM denotes an equivalent class, axioms T, 4 and 5 arevalid; due to the semantics, the validity of axioms PR(·) andNM(·) can be proved step by step; others can be proved as
usual.
To prove that SELA is strongly complete on uncertainty
maps, we only need to show that every SELA-consistent set
of formulas is satisfiable on some uncertainty map. The

proof idea is that we construct an uncertainty map consistingof maximal SELA-consistent sets (MCSs), and then with the
Lindenbaum-like lemma that every SELA-consistent set of
formulas can be extended in to a MCS (we omit the proofhere), we only need to prove that every formula holds on the
MCS to which it belongs.
Firstly, we construct a canonical Kripke model N c =hS c , {Rca | a ∈ A}, V c i as follows:• S c is the set of all MCSs;• sRac t ⇐⇒ haiφ ∈ s for any φ ∈ t (equivalently φ ∈ t
for any [a]φ ∈ s);• V c (p) = {s | p ∈ s}.
Given s ∈ S c , we define Usc = {u ∈ S c | Kφ ∈ s iff Kφ ∈ u},
and it is obvious that s ∈ Usc . Thus we have that for eachs ∈ S c , Mcs = hN c , Usc i is an uncertainty map, and Mcs , s isa pointed uncertainty map.
Secondly, we prove the following claim.
aClaim 2.1. If s → t, then we have Usc |a = Utc .
⊆: Assuming v ∈ Usc |a , we need to show v ∈ Utc , namely weneed to show that Kφ ∈ v ⇐⇒ Kφ ∈ t. Since v ∈ Usc |a ,we have that there is u ∈ Usc such that uRac v. If Kφ ∈ t, it
follows by axiom 4 that KKφ ∈ t. Thus we have haiKKφ ∈s. By axiom NM(a), it follows that K[a]Kφ ∈ s. By u ∈ Usc
and axiom T, we have [a]Kφ ∈ u. It follows by uRac v thatKφ ∈ v. If Kφ 6∈ t, we have ¬Kφ ∈ t. By axiom 5, we haveK¬Kφ ∈ t. Similarly, we have ¬Kφ ∈ v. Thus we haveKφ 6∈ v.
⊇: Assuming v ∈ Utc , we need to show v ∈ Usc |a , namely
there is u ∈ Usc such that uRac v. Let u− be {Kφ | Kφ ∈s}∪{haiψ | ψ ∈ v}. Then u− is consistent. For suppose not,we have ` Kφ1 ∧ · · · ∧ Kφn → [a]¬ψ1 ∨ · · · ∨ [a]¬ψk for somen and k. Since ` [a]¬ψ1 ∨· · ·∨[a]¬ψk → [a](¬ψ1 ∨· · ·∨¬ψk ),we have ` Kφ1 ∧ · · · ∧ Kφn → [a](¬ψ1 ∨ · · · ∨ ¬ψk ). By rule
NECK and axiom DISTK, we have ` KKφ1 ∧ · · · ∧ KKφn →K[a](¬ψ1 ∨ · · · ∨ ¬ψk ). Since KKφi ∈ s for each 1 ≤ i ≤ n,
we have K[a](¬ψ1 ∨ · · · ∨ ¬ψk ) ∈ s. By axiom PR(a), it
follows that [a]K(¬ψ1 ∨ · · · ∨ ¬ψk ) ∈ s. It follows by sRac tthat K(¬ψ1 ∨ · · · ∨ ¬ψk ) ∈ t. Since v ∈ Utc , by axiom T, wehave ¬ψ1 ∨ · · · ∨ ¬ψk ∈ v. This is contrary with ψi ∈ v foreach 1 ≤ i ≤ k. Thus u− is consistent. By Lindenbaum-likeLemma, there exists a MCS u extending u− . It follows byu− ⊆ u that u ∈ Usc and uRac v. We conclude that v ∈ Usc |a .
Finally, we will show that Mcs , s  φ iff φ ∈ s. we prove it
by induction on φ. Please note that the ‘existence lemmas’a(that ¬[a]φ ∈ s implies ¬φ ∈ t for some t such that s → t
00c
and that ¬Kφ ∈ s implies ¬φ ∈ s for some s ∈ Us ) also
hold in the model N c . We only focus on the case of [a]φ.
aWith Claim 2.1, it follows that Mct = Mcs |a if s → t. Thenby the induction hypothesis and the existence lemmas, it iseasy to show that Mcs , s  [a]φ iff [a]φ ∈ s.

3.
3.1

AN EXTENSION OF EAL FOR CONFORMANT PLANNING
Epistemic PDL over uncertainty maps

In this section we extend the language of EAL with programs in propositional dynamic logic and use this extended
language to express the existence of a conformant plan.

Definition 3.1 (Epistemic PDL). The Epistemic PDL
Language (EPDL) is defined as follows:
φ ::= > | p | ¬φ | (φ ∧ φ) | [π]φ | Kφ
π ::= a | ?φ | (π; π) | (π + π) | π ∗where p ∈ P, a ∈ A. We use LπMφ to denote [π]φ∧hπiφ, which
is logically equivalent to [π]φ∧hπi>. Given a finite B ⊆ A, wewrite B∗ for (Σa∈B a)∗ , i.e., the iteration over the ‘sum’ of all
the action symbols in B. The size of EPDL formulas/programs
is given by: |[π]φ| = |π|+|φ|, |a| = 1, |π1 ; π2 | = 1+|π1 |+|π2 |,|?φ| = |π ∗ | = 1 + |φ|, and |π1 + π2 | = 1 + |π1 | + |π2 |.

Given any uncertainty map M = hS, {Ra | a ∈ A}, V, U i,
any state s ∈ U, the semantics is given by a mutual induction
on φ and π (we only show the case about [π]φ, other cases
are as in EAL):
M, s  [π]φ ⇔ for all M0 , s0 : (M, s)JπK(M0 , s0 )implies M0 , s0  φa
0 0(M, s)JaK(M , s ) ⇔ M0 = M|a and s → s0
0 0
0 0
(M, s)J?ψK(M , s ) ⇔ (M , s ) = (M, s) and M, s  ψ(M, s)Jπ1 ; π2 K(M0 , s0 ) ⇔ (M, s)Jπ1 K ◦ Jπ2 K(M0 , s0 )(M, s)Jπ1 + π2 K(M0 , s0 ) ⇔ (M, s)Jπ1 K ∪ Jπ2 K(M0 , s0 )(M, s)Jπ ∗ K(M0 , s0 ) ⇔ (M, s)JπK? (M0 , s0 )where ◦, ∪, ? at the right-hand side denote the usual composition, union and reflexive transitive closure of binary relations respectively. Clearly this semantics coincides with thesemantics of EAL on EAL formulas.
Note that each program π can be viewed as a set of computation sequences, which are sequences of actions in A andtests with φ ∈ EPDL:

L(a) = {a}L(?φ) = {?φ}L(π; π 0 ) = {ση | σ ∈ L(π) and η ∈ L(π 0 )}0L(π + π 0 ) = L(π)
S ∪ L(π )· · π})) where  is the empty sequenceL(π ∗ ) = {} ∪ n>0 (L(π| ·{znHere are some valid formulas which are useful in our latterdiscussion:hπ; π 0 iφ ↔ hπihπ 0 iφ[π + π 0 ]φ ↔ [π]φ ∧ [π 0 ]φ[?ψ]φ ↔ (ψ → φ)
We leave the complete axiomatization of EPDL on uncertainty maps to future work.

3.2

Conformant planning via model checking
EPDL

Definition 3.2 (Conformant planning). Given an
uncertainty map M, a goal formula φ ∈ EPDL, and a set
B ⊆ A, the conformant planning problem is to find a finite(possibly empty) sequence σ = a1 a2 · · · an ∈ L(B∗ ) such thatfor each u ∈ UM we have M, u  La1 MLa2 M · · · Lan Mφ. The
existence problem of conformant planning is to test whether
such a sequence exists.
Recall that LπMφ is the shorthand of [π]φ ∧ hπiφ. Intuitively, we want a plan which is both executable and safe
w.r.t. non-deterministic actions and initial uncertainty of
the agent. It is crucial to observe the difference between
La1 MLa2 M · · · Lan Mφ and La1 ; a2 ; · · · ; an Mφ by the following example:
252

Example 1. Given uncertainty map M depicted as follows, we have M, s1  La; bMp but M, s1 2 LaMLbMp.

5 s2

a

s1

a

)

b

/ s4 : ps3

Proof. It is trivial when n = 0 (i.e., the sequence is ),
since the claim then boils down to Kφ ↔ Kφ. We prove
the non-trivial cases by induction on n ≥ 1. When n =1, it follows from Proposition 3.2. Now, as the induction
hypothesis, we assume that: KLa1 MLa2 M · · · Lak Mφ ↔ h?Kha1 i>; a1 ; . . . ; ?Khak i>; ak iKφ.

Given M and φ, to verify whether σ ∈ L(π) is a conformant plan can be formulated as the model checking problem: M, u  KLa1 MLa2 M · · · Lan Mφ. On the other hand, theexistence problem of a conformant plan is more complicatedto formulate: it asks whether there exists a σ ∈ L(B∗ ) suchthat it can be verified as a conformant plan. The simpleminded attempt would be to check whether M, u  KhB∗ iφholds. Despite the h·i-vs.-L·M distinction, KhB∗ iφ may holdon a model where the sequences to guarantee φ on different
states in UM are different, as the following example shows:

We need to show:

Example 2. Given uncertainty map M depicted as follows, let the goal formula be p and B = {a, b}. We haveM, s1  KhB∗ ip, but there is no solution to this conformant
planning problem.

h?Kha1 i>; a1 ; . . . ; ?Khak i>; ak iKLak+1 Mφ ↔
h?Kha1 i>; a1 ; . . . ; ?Khan i>; ak ih?Khak+1 i>; ak+1 iKφ. (2)

s1

a

/ s3

b

/ s5 : p

s2

b

/ s4

a

/ s6 : p

By IH,

KLa1 MLa2 M · · · Lak+1 Mφ ↔
h?Kha1 i>; a1 ; . . . ; ?Khak+1 i>; ak+1 iKφ.

KLa1 MLa2 M · · · Lak+1 Mφ ↔
h?Kha1 i>; a1 ; . . . ; ?Khak i>; ak iKLak+1 Mφ.

(1)

Due to Propositions 3.1 and 3.2, we have

The conclusion is immediate by combining (1) and (2).

The following theorem follows from the above lemma.

The right formula to check for the existence of a conformant plan w.r.t. B ⊆ A and φ ∈ EPDL is:θB,φ = h(Σa∈B (?Khai>; a))∗ iKφ.

For example, if B = {a1 , a2 } then θB,φ = h((?Kha1 i>; a1 ) +(?Kha2 i>; a2 ))∗ iKφ. Intuitively, the confrmant plan consists of actions that are always executable given the uncertainty of the agent (guaranteed by the guard Khai>). In theend the plan should also make sure that φ must hold given
the uncertainty of the agent (guaranteed by Kφ). In thefollowing, we will prove that this formula is indeed correct.
First, we observe that the rule of substitution of equivalents is valid (φ(ψ/χ) is obtained by replacing any occurrence of χ by ψ, similar for Jπ(ψ/χ)K):
Proposition 3.1. If  ψ ↔ χ, then:(1)  φ ↔ φ(ψ/χ);Theorem 3.1. Given a pointed uncertainty map M, s,
an EPDL formula φ and a set B ⊆ A, the following two areequivalent:(1) There is a σ = a1 . . . an ∈ L(B∗ ) such that M, s 
KLa1 MLa2 M · · · Lan Mφ;(2) M, s  h(Σa∈B (?Khai>; a))∗ iKφ.

We would like to emphasise that the K operator right before φ in the definition of θB,φ cannot be omitted, as demonstrated by the following example:
Example 3. Given uncertainty map M depicted as follows, let the goal formula be p. As we can see, there is
no solution to this conformant planning problem. Indeed
M, s1 2 h(Σa∈B (?Khai>; a))∗ iKp with B = {a, b}, but wecould have M, s1  h(Σa∈B (?Khai>; a))∗ ip.
s1

a

/ s2

b

(2) JπK = Jπ(ψ/χ)K.

/ s5 : pb

Proposition 3.2.  KLaMφ ↔ h?Khai>; aiKφ"Proof. Since  KLaMφ ↔ (K[a]φ∧Khaiφ) and  (Khai>∧
haiKφ) ↔ h?Khai>; aiKφ, we only need to show that (K[a]φ ∧ Khaiφ) ↔ (Khai> ∧ haiKφ).
Left to right:(L1)  K[a]φ → [a]Kφ, by validity of Axiom PR(a)(L2)  Khaiφ → hai> ∧ Khai>, by semantics(L3)  hai> ∧ [a]Kφ → haiKφ, by semantics(L4)  K[a]φ ∧ Khaiφ → Khai> ∧ haiKφ, by (L1)-(L3)Right to left:(R1)  haiKφ → K[a]φ, by validity of Axiom NM(a)(R2)  K[a]φ ∧ Khai> → Khaiφ, by semantics(R3)  Khai>∧haiKφ → K[a]φ∧Khaiφ, by R(1)-R(2)Lemma 3.1. For any a1 a2 · · · an ∈ L(A∗ ):We close this section with an example about planning withboth positive and negative epistemic goals (the agent should
know something, but not too much).
Example 4. Given uncertainty map M depicted as follows, let the goal be Kp then both a and b are conformant
plans. If the goal is Kp ∧ ¬Kq, only a is a good plan.
s1
s2

a
b
a
b

 KLa1 MLa2 M · · · Lan Mφ ↔ h?Kha1 i>; a1 ; . . . ; ?Khan i>; an iKφ253s4/ s3 : p/ s4' : p, q'

s5 : p, q

4.

MODEL CHECKING EPDL: COMPLEXITY AND ALGORITHMS

In this section, we first focus on the model checking problem of the following star-free fragment of EPDL (call it EPDL− ):φ ::= > | p | ¬φ | (φ ∧ φ) | [π]φ | Kφ
π ::= a | ?φ | (π; π) | (π + π)
We will show that model checking EPDL− is Pspace-complete.
In particular, the upper bound is shown by making use of analternative context-dependent semantics. Then we give an
EXPtime algorithm for the model checking problem of thefull EPDL inspired by another alternative semantics basedon 2-dimensional models. Finally we give a Pspace algorithm for the conformant planning problem in EPDL. Notethat throughout this section, we focus on uncertainty maps
with finitely many states and assume Ra = ∅ for co-finitely
many a ∈ A.

4.1

Complexity of model checking EPDL−

4.1.1

Lower BoundTo show the Pspace lower bound, we provide a polynomial reduction of QBF (quantified Boolean formula) truth
testing to the model checking problem of EPDL− . Note thatto determine whether a given QBF (even in prenex normal
form based on a conjunctive normal form) is true or not is
known to be Pspace-complete [29]. Our method is inspiredby [27] which discusses the complexity of model checking
temporal logics with past operators. Surprisingly, we canuse the uncertainty sets to encode the ‘past’ and use thedual of the knowledge operator to ‘go back’ to the past.
This intuitive idea will become more clear in the proof.

AA

x1 : p 1

A



a1

a2

:

/ x2 : p2
A

ā2

ā1

a2

• S = {x0 } ∪ {xi | 1 ≤ i ≤ n} ∪ {x̄i | 1 ≤ i ≤ n}
• V(x0 ) = ∅, and V(xi ) = {pi }, V(x̄i ) = {qi } for 1 ≤ i ≤
n.
a

i
• →=
{(s, s) | s ∈ S} ∪ {(xi−1 , xi ), (x̄i−1 , xi )}

ā

i
• →=
{(s, s) | s ∈ S} ∪ {(xi−1 , x̄i ), (x̄i−1 , x̄i )}• U = {x0 }|Mn | is linear in n and can be depicted as the following:

xn−1 : pn−1an/ xn :  p n
>

ān

a3

$

an



X

X

2

A

3

/ 

···

x̄n−1 : pn−1

X

A

ān

A

/ x̄n : qn
X
A

Given α = Q1 x1 Q2 x2 . . . Qn xn φ(x1 , . . . , xn ), the formulaθα is defined as
QT1 · · · QTn ψ(K̂p1 , · · · , K̂pn , K̂q1 , · · · , K̂qn )where QTi is h(ai + āi ); ?(pi ∨ qi )i if i is odd and QTi is[(ai + āi ); ?(pi ∨ qi )] if i is even, and ψ is obtained fromφ(x1 , . . . , xn ) by replacing each xi with K̂pi and ¬xi withK̂qi .
To ease the latter proof, we first define the valuation tree
below.
Definition 4.2 (V-tree). A V-tree τ is a rooted treesuch that 1) each node is 0 or 1 (except the root ); 2) eachinternal node in an even level has only one successor; 3)each internal node in an odd level has two successors: oneis 0 and the other one is 1; 4) each edge to node 0 of leveli is labelled āi ; 5) each edge to node 1 of level i is labelled
ai . Given a V-tree with depth n, a path σ is a sequence of
A1 . . . An where Ai = ai or Ai = āi . A path σ can also
be seen as a valuation assignment for x1 , . . . , xn with theconvention that σ(xi ) = 1 if ai occurs in σ and σ(xi ) = 0 ifāi occurs in σ. Let path(τ ) be the set of all paths of τ .
As an example, a V-tree τ can be depicted as below:
Definition 4.1. Let A = {ai , āi | i ≥ 1} and P = {pk , qk |k ≥ 1}, the uncertainty map Mn = hS, {Ra | a ∈ A}, V, Ui
is defined as:

/ ···
C

A



ā3

x̄1 : q1 ā / x̄2 : q2 ā

• For 1 ≤ n ≤ n, Qi is ∃ if i is odd, and Qi is ∀ if i iseven.

For each such QBF α with n variables, we need to find apointed model Mn , x0 and a formula θα such that α is true
iff Mn , x0  θα . The model Mn is defined below.

a3

x0

QBF formulas are Q1 x1 Q2 x2 . . . Qn xn φ(x1 , . . . , xn ) where:• φ is a propositional formula in CNF based on variables
x1 , . . . , xn ,

A



a1

/1

a2
ā2

41

ā3

/0

*0

a3

/1

It is not hard to see the following:
Proposition 4.1. For each 1 ≤ i ≤ n, we have α =Q1 x1 . . . Qi xi Qi+1 xi+1 . . . Qn xn φ is true iff there exists a Vtree τ with depth i such that σ(Qi+1 xi+1 . . . Qn xn φ) = 1 foreach σ ∈ path(τ ) (σ is viewed as a valuation).
Now let us see the update result of running a path σ ∈path(τ ) on Mn . Due to the lack of space, we omit theproofs of the following two propositions.
Proposition 4.2. Given Mn , let σ = A1 . . . Ai (1 ≤ i ≤n) be a sequence of actions such that Ak = ak or Ak = ākfor each 1 ≤ k ≤ i, then we have U|σ = {x0 , X1 , . . . , Xi }where Xk = xk if Ak = ak else Xk = x̄k for each 1 ≤ k ≤ i.
Given σ = A1 . . . An where Ai is ai or āi for each 1 ≤ i ≤n, let g(σ) = xn if An = an and g(σ) = x̄n if An = ān . By
Proposition 4.2, we always have g(σ) ∈ UMk |σ with k > n.
Thus given Mk and σ = A1 . . . An and k > n, Mk |σ , g(σ)
is a pointed uncertainty map.
Proposition 4.3. For each 1 ≤ i ≤ n, we have Mk , x0 
QT1 . . . QTi QTi+1 . . . QTn ψ iff there exists a V-tree τ withdepth i such that Mk |σ , g(σ)  QTi+1 . . . QTn ψ for each σ ∈path(τ ), where k > n and g(σ) is the state corresponds tothe last edge of σ, e.g., g(a1 ā2 ) = x̄2 .

254

Theorem 4.1. The following two are equivalent:• α = Q1 x1 Q2 x2 . . . Qn xn φ(x1 , . . . , xn ) is true
• Mn , x0  QT1 · · · QTn ψ(K̂p1 · · · K̂pn , K̂q1 · · · K̂qn ) in
which ψ is obtained from φ by replacing each xi withK̂pi and ¬xi with K̂qi .
Proof. By Propositions 4.1 and 4.3, we only need toshow that given V-tree τ with depth n, σ(φ) = 1 if and only
if Mn |σ , g(σ)  ψ for each σ ∈ path(τ ). Since φ is in CNF, ψis also in CNF-like form obtained by replacing each xi withK̂pi and each ¬xi with K̂qi for 1 ≤ i ≤ n. Thus we only needto show that σ(xi ) = 1 iff Mn |σ , g(σ)  K̂pi and σ(¬xi ) = 1
iff Mn |σ , g(σ)  K̂qi . Since σ(xi ) = 1 iff σ(¬xi ) = 0, weonly need to show that σ(xi ) = 1 iff Mn |σ , g(σ)  K̂pi and
Mn |σ , g(σ)  K̂pi iff Mn |σ , g(σ)  ¬K̂qi . By the definitionof τ , we know that σ = A1 . . . An where Ai is ai or āi foreach 1 ≤ i ≤ n.
Firstly, we will show that Mn |σ , g(σ)  K̂pi if and only
if Mn |σ , g(σ)  ¬K̂qi . To verify the right-to-left direction, if Mn |σ , g(σ)  K̂pi , it follows by the definition of
Mn that xi ∈ U|σ . Then it must be the case that ai
occurs in σ. Suppose not, āi occurs in σ. It follows by
Proposition 4.2, U |σ = {x0 , X1 , . . . , Xi−1 , x̄i , Xi+1 , . . . , Xn }.
This is contrary with xi ∈ U|σ . Thus it must be thatai occurs in σ. It follows by Proposition 4.2 that U|σ ={x0 , X1 , . . . , Xi−1 , xi , Xi+1 , . . . , Xn }. Thus x̄i 6∈ U|σ . By
the definition of Mn and the semantics, we have Mn |σ , g(σ) ¬K̂qi . To verify the left-to-right direction, Mn |σ , g(σ) ¬K̂qi implies that x̄i 6∈ U|σ . For the similar reason as above,it must be the case that āi does not occur in σ. Thus wehave that ai occurs in σ. It follows by Proposition 4.2 thatxi ∈ U |σ . Thus we have Mn |σ , g(σ)  K̂pi .
Next we will show that σ(xi ) = 1 iff Mn |σ , g(σ)  K̂pi . To
verify the right-to-left direction, σ(xi ) = 1 implies that Ai =ai . It follows by Proposition 4.2 that xi ∈ U|σ . Thus we haveMn |σ , g(σ)  K̂pi . To verify the left-to-right direction, we
will show that σ(xi ) = 0 implies Mn |σ , g(σ)  K̂qi . It
follows by the definition of σ(xi ) = 0 that Ai = āi . It
follows by Proposition 4.2 that x̄i ∈ U|σ . Thus we haveMn |σ , g(σ)  K̂qi .
This gives us the desired lower bound:

Definition 4.3. Given an uncertainty map M = hS, {Ra |a ∈ A}, V, Ui and any state s ∈ S, the satisfaction relation
is defined using the auxiliary satisfaction relation σ andωσwhere σ is a finite (possibly empty)
auxiliary relation →,sequence of actions in A:
M, s
M, s
M, s
M, s
M, s
M, s
M, s

⇔ M, s  φ⇔ always⇔ p ∈ V(s)σ p⇔ M, s 1σ φσ ¬φ
σ φ ∧ ψ ⇔ M, s
σ φ and M, sσ ψ
⇔ for all v ∈ U|σ : M, v σ φ
σ Kφ
σ hπiφ ⇔ there exists ω ∈ L(π) and t ∈ S
ωσt and M, t σr(ω) φsuch that s →
σ
⇔s=t
s→t
s
s

(aω 0 )σ

→

φ

σ >

(?φω 0 )σ→0ωσ
σ φ and s → t⇔ M, stwhere r(ω) is the sequence of actions obtained by eliminating
all the tests in ω.
Note that ω in the above definition is a computation sequence, i.e., a finite sequence of actions and EPDL− -tests,
while σ is a test-free sequence of actions.
The following can be proved by induction on η:
Proposition 4.4. Given an uncertainty map M and sequences of actions and tests η, ω, ω 0 such that η = ωω 0 , weησωσhave (s, t) ∈→ iff (s, t) ∈→◦actions σ.

for any sequence of(aη 0 )σ

a

η0

η0

ω 00

σa
σa
σa
and (s0 , t) ∈ →. By IH, we have →
=→
◦
0
ησa

0

0
ωσar(ω
00 )

→

0

0

. Thusω 00

0

σa
we have (s , t) ∈ → iff there exists t such that (s , t ) ∈ →and (t0 , t) ∈0ωσar(ω
00 )

→

00
ωσa

a

. By definition, we have that s → s0

aω 00

00
aωσ

and (s0 , t0 ) ∈ → iff (s, t0 ) ∈ → . Thus we have (s, t) ∈ →σ0
ωσar(ω
00 )4.1.2

ω 00 of η 0 , and (s, t) ∈ →0
ωσr(ω)

ωσ→ , namely (s, t) ∈→◦ → .
0Case η =?φη : We have ω =?φω 00 for some initial segment(?φη 0 )σ255→ω 00 of η 0 , and (s, t) ∈ → iff there exists s0 such that s → s0◦

In this section we give a non-trivial model checking algorithm for EPDL− inspired by an equivalent semantics.
As we mentioned earlier, the semantics of EPDL is ‘contextdependent’: reaching the same state through different paths
may affect the truth value of an epistemic subformula. Thismeans that the usual global model checking algorithm formodal logics may not work here. In order to establish theupper bound, we first give the following equivalent semantics
to EPDL− which makes the context dependency explicit inorder to facilitate a local model checking algorithm. The
idea is to keep the model intact but record the scope of action
modalities in order to compute the right uncertainty set for
epistemic subformulas. Similar idea appeared in [32] to givean alternative semantics of public announcement logic.

0ωσr(ω)

Proof. We prove it by induction on |η|. If |η| ≤ 2, it isobvious by the definition. If |η| > 2, there are two cases,that is, η = aη 0 or η =?φη 0 .
Case η = aη 0 : We have ω = aω 00 for some initial segment

Theorem 4.2. The model checking problem for EPDL− is
Pspace-hard.

Upper Bound

0
ω(σa)

a

⇔ there exists s0 such that s → s0 and s0 → t

t

η0

ω 00

σ
σ
we have s →t iff (s, t) ∈ →
◦

0

0

0
ησ
σ φ and s → t. By IH,

iff M, s

0
ωσr(ω
00 )

00
ωσ

→

. Thus we have there
0
ωσr(ω
00 )

exists s such that (s, s ) ∈ → and (s0 , t) ∈→follows that (s, s0 ) ∈◦ωσnamely (s, t) ∈→ ◦(?φω 00 )σ→0ωσr(ω), and (s, t) ∈(?φω 00 )σ→. This0
ωσr(?φω
00 )

→

,

→ .

In the following we show thatcoincides with .

Theorem 4.3. Given an uncertainty map M and an action sequence σ, if U|σ 6= ∅, we have that for each s ∈ U|σ ,(i) M|σ , sJπKM0 , s0 iff there exists ω ∈ L(π) such thatωσ 0M0 = M|σr(ω) and s →s,(ii) M|σ , s  φ iff M, sσ φ.

Proof. The proof is by simultaneous induction on π andφ (due to the test actions). For (i), we will only focus on
the case of π1 ; π2 ; the other cases are straightforward.
Case π1 ; π2 : We only show the direction from left toright; the other direction is similar. It follows by assumption that there is pointed uncertainty map N , t such thatM|σ , sJπ1 KN , t and N , tJπ2 KM0 , s0 . By IH, we have thatωσt.
there exists ω ∈ L(π1 ) such that N = M|σr(ω) and s →σr(ω)
Since N , t is a pointed uncertainty map and N = M|,
we have t ∈ U|σr(ω) . By IH and M|σr(ω) , tJπ2 KM0 , s0 , we0have that there exists ω 0 ∈ L(π2 ) such that M|σr(ω)r(ω ) =0

M|σr(ωω ) = M0 and t0ωσr(ω)→

Now we define an alternative semantics of EPDL over ETSs.6

Definition 4.5 (ETS Semantics). Given any ETS
model M = hS, {Ra | a ∈ A}, ∼, Vi and any state s ∈ S, thesatisfaction relation for EPDL formulas is defined as follows(the Boolean cases are as in the standard modal logic):
M, s
M, s
a
→

s0 . By Proposition 4.4, it
0(ωω )σfollows that ωω 0 ∈ L(π1 ; π2 ) and s → s0 .
For (ii), we will focus on the case of hπiφ; the other cases
are straightforward.
Case hπiφ: We have M|σ , s  hπiφ if and only if there ispointed uncertainty map M0 , s0 such that M|σ , sJπKM0 , s0
and M0 , s0  φ. By (i), it follows that M|σ , sJπKM0 , s0 iff
ωσ 0there exists ω ∈ L(π) such that M0 = M|σr(ω) and s →s.
By IH, it follows that M|σr(ω) , s0  φ iff M, s0 σr(ω) φ.
Thus we have M, s hπiφ.
Let σ be , we have the equivalence ofand .

Corollary 4.1. Given pointed uncertainty map M, s, wehave M, s  φ iff M, s φ for each φ ∈ EPDL− .
This alternative semantics induces a natural algorithm tocompute the truth value of an EPDL− formula w.r.t. to apointed uncertainty map. The idea is to recursively call afunction M C(M, s, σ, φ) which returns the truth value of a
subformula φ on state s given the context of σ while keeping M intact. Note that, we do not need to compute all the
M C(M, s, σ, φ) for each σ and each subformula φ. The only
tricky part comes when evaluating hπiφ formulas since it is
too space consuming to compute the whole set of L(π) in
the search of the right ω. Instead, we can generate one by
one in some lexicographical order all the possible sequences
up to a bound based on the atomic actions and tests occurring in the formula, and then test whether it belongs to
the program π. Note that in this way, we can use the space
repeatedly, and the membership testing of L(π) is not expensive (NLOGspace-complete according to [19]).
In the appendix we present three algorithms based on matrix representation of the model: Algorithm 1 computes thew
uncertainty set U |σ ; Algorithm 2 computes →σ and Algorithm 3 is the main model checking algorithm. Note that
Algorithms 2 and 3 involve mutual recursion of each otherdue to the tests in programs. However, the depth of therecursion is bounded by the length of the formula, and foreach call polynomial space suffices. The detailed algorithms
and complexity analysis can be found in the appendix. It is
not hard to show the following (based on Theorem 4.2)
Theorem 4.4 (Upper bound). The model checking problem of EPDL− is in Pspace. Thus it is Pspace-complete.

4.2

Definition 4.4 (Epistemic Temporal Structure).
An Epistemic Temporal Structure (ETS) is a Kripke modelwith both epistemic and action relations. Formally, an ETSmodel M is a tuple hS, {Ra | a ∈ A}, ∼, Vi, where Ra is abinary relation on S, ∼ is an equivalence relation on S andV : S → 2P is a valuation function.

Upper Bounds for model checking EPDL

In this section, we give an EXPtime model checking method
for the full EPDL via model checking EPDL over two-dimensionalmodels with both epistemic and action relations. Let us first
define such models.

?φ→
π1 ;π2→π1 +π2→π∗→

Kφ ⇔ ∀u ∈ S : s ∼ u implies M, u φπ[π]φ ⇔ ∀t ∈ S : s → t implies M, t φ= Ra= {(s, s) | M, sππ
= →1 ◦ →2ππ= →1 ∪ →2φ}π= (→)?

where ◦, ∪, ? at right-hand side denote the usual composition, union and reflexive transitive closure of binary relations respectively.
We can turn a Kripke model without the epistemic relation into an ETS model by essentially considering all thepossible uncertainty sets.
Definition 4.6. Given any Kripke model M = hS, {Ra |a ∈ A}, Vi, we define the ETS model M• as follows:S•= {sΓ | s ∈ S, Γ ∈ 2S , s ∈ Γ}a•Ra
= {(sΓ , t∆ ) | s → t, ∆ = Γ|a }•∼= {(sΓ , t∆ ) | Γ = ∆}V • (sΓ ) = V(s)awhere Γ|a = {t ∈ S | ∃s ∈ Γ such that s → t}. For any
Kripke model M and any Γ ∈ 2S \{∅}, let MΓ be the uncertainty map hM, Γi.
Note that each sΓ can be viewed as an uncertainty set(Γ) with a designated state (s), and the definition of Ra
captures the update in the  semantics of EPDL, and M•
unravels all the updates in a whole picture. Note that thesize of M• is |S| · 2|S|−1 where S is the set of states of M.
Now we can show that  and coincide w.r.t. uncertainty
map MΓ and ETS model M• (the proofs are omitted dueto the lack of space).
Proposition 4.5. Given any map M, we haveπ(i) MΓ , sJπKM∆ , t iff sΓ → t∆ in M• ;7(ii) MΓ , s  φ iff M• , sΓφ.

Corollary 4.2. Given an uncertainty map M = hN , U iand s ∈ U , we have M, s  φ iff N • , sU φ.
Based on the above corollary we can have a model checking
method via model checking EPDL over ETS models.
6

Here we abuse the notation to denote the new semantics.
Note that it is different from the alternative semantics in theprevious section.
π7
Cf. the definition of → in Def. 4.5.
256

Proposition 4.6. The model checking problem of EPDL
on uncertainty maps is in EXPtime.
Proof. Given an uncertainty map M = hN , U i, the construction of ETS N • can be done in exponential time in thesize of N due to the fact that there are at most |N | asuccessors t∆ of each sΓ since ∆ = Γ|a . By modifying thealgorithm for PDL in [22], we can get an algorithm to check
EPDL formula φ on N • w.r.t. , and its time complexity isO(|φ|2 ·|N • |3 ). Thus, the time complexity of model checking
φ on M is bounded by O(|φ|2 · |SN |3 · 23|SN |−3 ).
We conjecture that the model checking problem of full
EPDL is EXPtime-complete, and leave the lower bound to
the extended version of this paper.

4.3

Complexity of conformant planning

In the rest of this section, let us look at the complexityof conformant planning in terms of EPDL model checking.
Although the model checking problem of full EPDL is likely
to be EXPtime-complete, the complexity of model checking
the EPDL formula which encodes the conformant planningproblem (cf. Theorem 3.1) is in Pspace if the goal formula
is program-free. More precisely, we can show the following:
Theorem 4.5. The problem of model checking EPDL formulas in the shape of h(Σa∈B (?Khai>; a))∗ iKφ, where φ isan epistemic formula (i.e. program-free) and B ⊆ A, is in
Pspace.
P
Proof. (Sketch) Note that ( a∈B (?Khai>; a))∗ is a special program which has only simple epistemic tests depending on the structure of the underlying Kripke model. Now
given a Kripke model N and a set B ⊆ A we can define an
ETS model N ◦ similar to N • but with a different definition
for the action relations:aaR◦a = {(sΓ , t∆ ) | s → t, ∆ = Γ|a , ∀u ∈ Γ∃v st. u → v.}
Note that the extra condition guarantees that the action
a is always executable w.r.t. the whole Γ, thus fulfilling
the test ?Khai>. Now we can have an analog of Corollary
P 4.2, and reduce the problem of checking hN , Ui, s ( a∈B (?Khai>; a))∗ Kφ to the reachability problem in N ◦ :whether there is a path from sU in N ◦ such that it canreach a state tU 0 where Kφ holds. Since φ is [π]-free, we
can check it easily given U 0 using polynomial space, thus
the main task is to find the reachable tU 0 . Note that, in thesize of N , there are exponentially many such tU 0 and the
maximal length of the plan is also exponential. However,
we do not need to build the whole N ◦ and the bisection-likealgorithm behind the proof of Savitch’s Theorem will do thejob.8 More precisely, we first pick up a tU 0 , and then run therecursive bisection method to see whether tU 0 is reachable
from sU within 2|N | steps. The depth of the recursion isbounded by log2 (2|N | ) = |N | and at each recursion we needto record the choice of the state which can be encoded bya (0, 1)-vector using log2 (2|N | ) = |N | space (plus one bit torecord the result). Moreover, at the bottom of the recursion
we only need to verify one step reachability, i.e., whether
two states in N ◦ are linked by R◦a , without building thewhole N ◦ . Thus the whole procedure of model checking canbe done using polynomial space.
8

A similar algorithm was used to pinpoint complexity of theconformant planning in AI, cf.[20].

257

As we mentioned in the introduction, the conformant planning problems in the AI literature are usually given by usingstate variables and actions with preconditions and (conditional) effects, rather than explicit transition systems. The
corresponding explicit transition system can be generated
by taking all the possible valuations of the state variables
as the state space (an exponential blow up), and computing
the transitions among the valuations according to the preconditions and the postconditions of the actions. In termsof the size of explicit transition systems, our above result is
consistent with the EXPspace complexity result in the AI
literature for conformant planning with Boolean and modal
goals [20, 8]. Actually, the complexity result of Theorem 4.5
can be strengthened to Pspace-complete based on the corresponding complexity result in the AI literature.
However, not all the transition systems can be generated
in this way since the preconditions and postconditions are(usually) purely propositional and thus two states that sharethe same valuation must have the same executable actions.
In an arbitrary transition system, multiple states with thesame valuation may have different available actions due tosome underlying protocol or other (external) factors notmodelled by basic propositions.

5.

CONCLUSIONS AND FUTURE WORKIn this work we first introduce the logical language EAL
over uncertainty maps and axiomatize it completely. EAL isthen extended to EPDL with programs to specify conformant
and conditional plans. We show that the conformant planning problems can be reduced to model checking problemsof EPDL. Finally we showed that model checking star-free
EPDL over uncertainty maps is Pspace-complete and model
checking the full fragment is in EXPtime. On the other
hand, model checking the conformant planning problem isin Pspace.
Note that our EPDL is a powerful language which can already express conditional plans, e.g. (?p; a+?¬p; b); c. Thissuggests that we can use the very EPDL language (EPDL−
is enough) to verify plans in contingent planning w.r.t. avariant of the semantics which can handle feedbacks during the execution. In fact, observational power about theavailability of the actions has been already incorporated in[33], which can be extended to general feedbacks discussed
in the literature of contingent planning (cf. e.g., [10]). On
the other hand, to check the existence of a conditional plan,
we are not sure whether EPDL is expressive enough, as subtleties may arise as in the case of conformant planning. We
leave the contingent planning to future work.
Another natural extension is to go probabilistic, and reduce the probabilistic planning over MDP to some model
checking problem of the probabilistic version of our EPDL.
Our ultimate goal is to cast all the standard AI planningproblems into one unified logical framework in order to facilitate careful comparison and categorization. We will thensee clearly how the form of the goal formula, the constructorof the plan, and the observational ability matter in the theoretical and practical complexity of planning, in line withthe research pioneered in [5].

Acknowledgement.
Quan Yu is supported by NSF Grant No.61463044 andGrant No.[2014]7421 from the Joint Fund of the NSF of
Guizhou province of China. Yanjun Li thanks the support

from China Scholarship Council. Yanjing Wang acknowledges the support from ROCS of SRF by Education Ministryof China and the NSSF major project 12&ZD119.

6.

REFERENCES[1] Mikkel Birkegaard Andersen, Thomas Bolander, andMartin Holm Jensen. Conditional epistemic planning.
In Logics in Artificial Intelligence, pages 94–106.
Springer, 2012.
[2] Mikkel Birkegaard Andersen, Thomas Bolander, andMartin Holm Jensen. Don’t plan for the unexpected:
Planning based on plausibility models. Logique etAnalyse, 1(1), 2014.
[3] Guillaume Aucher. Del-sequents for regression andepistemic planning. Journal of Applied Non-Classical
Logics, 22(4):337–367, 2012.
[4] Guillaume Aucher and Thomas Bolander.
Undecidability in epistemic planning. In IJCAI, pages
27–33, 2013.
[5] Christer Bäckström and Peter Jonsson. All
pspace-complete planning problems are equal but
some are more equal than others. In SOCS 2011, 2011.
[6] Alexandru. Baltag and Larry Moss. Logics for
epistemic programs. Synthese, 139:165–224, March
2004.
[7] Thomas Bolander and M. Birkegaard Andersen.
Epistemic planning for single and multi-agent systems.
Journal of Applied Non-Classical Logics, 21(1):9–34,
2011.
[8] Blai Bonet. Conformant plans and beyond: Principles
and complexity. Artificial Intelligence.,
174(3-4):245–269, 2010.
[9] Blai Bonet and Hector Geffner. Planning withincomplete information as heuristic search in belief
space. In ICAPS 2000, pages 52–61, 2000.
[10] Blai Bonet and Hector Geffner. Width and complexityof belief tracking in non-deterministic conformant andcontingent planning. In AAAI 12, 2012.
[11] Ronen I. Brafman and Jörg Hoffmann. Conformant
planning via heuristic forward search: A new
approach. In ICAPS 2004, pages 355–364, 2004.
[12] Daniel Bryce, Subbarao Kambhampati, and David E.
Smith. Planning graph heuristics for belief space
search. Journal of Artificial Intelligence Research,
26:35–99, 2006.
[13] Alessandro Cimatti and Marco Roveri. Conformant
planning via symbolic model checking. Journal of
Artificial Intelligence Research, 13:305–338, 2000.
[14] Alessandro Cimatti and Marco Roveri. Conformant
planning via symbolic model checking. CoRR,
abs/1106.0252, 2011.
[15] Alessandro Cimatti, Marco Roveri, and Piergiorgio
Bertoli. Conformant planning via symbolic model
checking and heuristic search. Artificial Intelligence,
159(1-2):127–206, 2004.
[16] R. Fagin, J. Halpern, Y. Moses, and M. Vardi.
Reasoning about knowledge. MIT Press, Cambridge,
MA, USA, 1995.
[17] R. Fagin, J. Y. Halpern, Y. Moses, and M. Y. Vardi.
Knowledge-based programs. Distributed Computing,10(4):199–225, July 1997.

[18] Patrik Haslum and Peter Jonsson. Some results on thecomplexity of planning with incomplete information.
In ECP 1999, pages 308–318, 1999.
[19] Tao Jiang and B. Ravikumar. A note on the spacecomplexity of some decision problems for finite
automata. Information Processing Letters, 40:25–31,
October 1991.
[20] Jon Kleinberg and Éva Tardos. Algorithm Design.
Addison-Wesley, 2005.
[21] Jérôme Lang and Bruno Zanuttini. Knowledge-based
programs as plans - the complexity of plan
verification. In ECAI 2012, pages 504–509, 2012.
[22] Martin Lange. Model checking propositional dynamic
logic with all extras. Journal of Applied Logic,
4:39–49, 2006.
[23] Benedikt Löwe, Eric Pacuit, and Andreas Witzel. Del
planning and some tractable cases. In LORI 2011,
pages 179–192. Springer, 2011.
[24] Héctor Palacios and Hector Geffner. Compiling
uncertainty away: Solving conformant planningproblems using a classical planner (sometimes). In
AAAI 2006, pages 900–905, 2006.
[25] Pere Pardo and Mehrnoosh Sadrzadeh. Strong
planning in the logics of communication and change.
In Declarative Agent Languages and Technologies X,
pages 37–56. Springer, 2013.
[26] Rohit. Parikh and R. Ramanujam. Distributed
processes and the logic of knowledge. In Proceedings ofConference on Logic of Programs, pages 256–268,
London, UK, 1985. Springer-Verlag.
[27] Philippe Schnoebelen. The complexity of temporal
logic model checking. In Philippe Balbiani, Nobu-Yuki
Suzuki, Frank Wolter, and Michael Zakharyaschev,
editors, AiML 2002, pages 393–436, Toulouse, France,
2003. King’s College Publication. Invited paper.
[28] David E. Smith and Daniel S. Weld. Conformant
graphplan. In AAAI 1998, pages 889–896, 1998.
[29] Larry J Stockmeyer and Albert R Meyer. Word
problems requiring exponential time (preliminary
report). In STOC 1973, pages 1–9. ACM, 1973.
[30] Son Thanh To, Tran Cao Son, and Enrico Pontelli. A
new approach to conformant planning using cnf*. In
ICAPS 2010, pages 169–176, 2010.
[31] Hans van Ditmarsch, Wiebe van der Hoek, andBarteld Kooi. Dynamic epistemic logic. Springer, 2007.
[32] Yanjing Wang and Qinxiang Cao. On axiomatizationsof public announcement logic. Synthese, 190:103–134,
2013.
[33] Yanjing Wang and Yanjun Li. Not all those who
wander are lost: Dynamic epistemic reasoning in
navigation. In AiML 2012, pages 559–580, 2012.
[34] Quan Yu, Ximing Wen, and Yongmei Liu. Multi-agent
epistemic explanatory diagnosis via reasoning about
actions. In IJCAI 2013, pages 1183–1190, 2013.

APPENDIXA.

ALGORITHMS FOR EPDL−Definition A.1 (Matrix representation). Let Bn×m
denote a (0,1)-matrix of size n × m. A matrix Bn×1 , or Bn
for short, is called a vector. Given finite uncertainty map

258

M, its domain S can be linearly ordered as {s1 , · · · , sn }.
aThus M can be represented by a set {Bn×n| a ∈ A} of adjacency matrices for accessibility relation, a vector BU
n for Uand a set {Bnp | p ∈ P} of vectors for atomic propositions.
Definition A.2. Given (0,1)-matrices B0n×k , Bk×m , theirproduct B00n×m is defined as: B00n×m [i, j] = 1 iff there existsr ≤ n such that B0n×k [i, r] = Bk×m [r, j] = 1 for all 1 ≤ i ≤n, 1 ≤ j ≤ m.
The following algorithms are to check whether φ holds on apointed uncertainty map M, s by Definition 4.3. The mainalgorithm (Algorithm 3) recursively calls itself for each nontrivial subformula of φ. The complex cases are for the subformulas in the form of hπiφ and Kφ. By Definition 4.3, tocheck M, s σ hπiφ, we need to make sure that there exists
ωσa sequence ω ∈ L(π) and a state t ∈ S such that s →tand M, t σr(ω) φ. Since π is star-free, |ω| ≤ |π| for eachω ∈ L(π). It is clear that we cannot compute and store thewhole set of L(π) within polynomial space. Instead, one by
one we generate all the possible sequences that are shorter
than |π| and are formed from the alphabet of π (cf. line 14),and check whether they are in L(π). We can order the possible sequences lexicographically according to an ordering of
the basic actions and tests in Sig, and compute the next
sequence merely from the current one using function next.
memb chec(ω, π) checks whether it is the case ω ∈ L(π). Ifω ∈ L(π), we need to check whether there exists sj ∈ SM
ωσsj (Algorithm 2) and M, sj σr(ω) φ, wheresuch that s →r(ω) is the test-free subsequence of ω which is easy to compute. For the case of Kφ, we need to calculate the state set
U |σ (Algorithm 1).

B.

COMPLEXITY ANALYSIS

We suppose |SM | = n and |φ| = k. Algorithm 1 uses one
variable A to record the uncertainty set which requires O(n)space. Note that there is a mutual recursion in Algorithm 2
and 3, but the depth of the overall recursion is bounded by k.
In Algorithm 2, the variable consuming the most of the spaceis the matrix Bn×n recording the (intermediate) relation.
Since σ and ω are also variables in the main algorithm and|ω| + |σ| ≤ k due to the construction in Algorithm 3, thespace usage of Algorithm 2 before the recursive calls of P W
and M C is bounded by O(k+n2 ). For Algorithm 3, the mostspace-demanding part is the hπiφ case, where we need tostore π, Sig, and keep track one ω and one state s in the loop,
which are bounded by either k or s. Moreover, according to[19], the complexity of memb chec is NLOGspace-completein the size of Sig, i.e., the alphabet of π which is boundedagain by k. Thus before calling M C and P W again in thehπiφ case, the space requirement is at most linear in both kand n, which is less demanding than P W for each recursion.
Recall that the overall recursion depth of M C (and P W ) isbounded by k thus the space usage of the whole algorithm
is bounded by O(k(k + n2 )) = O(k2 + kn2 ).

Algorithm 1: Function CNU(U, σ): Calculate the the
new uncertainty set U|σinput : U, σ
U |σoutput: Bn
A ← BUn;for i ← 1 to |σ| do
σ[i]3A ← A × Bn×n ;1
2

4

Algorithm 2: Function P W (ω, σ): Calculate the binary
ωσrelation →input : computation sequence ω, action sequence σ
output: Bn×n
switch ωσ docase σ return M atrix({(s, s) | s ∈ S})/* M atrix(R) is the (0, 1)-matrix
representation of the binary relation R */;3case (?φω 0 )σ4
return M atrix({(s, s) | MC(M, s, σ, φ) = true})× PW(ω 0 , σ);5case (aω 0 )σ return Ban×n × PW(ω 0 , σa) ;1

2

Algorithm 3: Function MC(M, s, σ, φ): Model checking
algorithm for EPDL− (Boolean cases omitted)
input : The pointed uncertainty map (M, s), sequenceof actions σ, φ ∈ EPDL− .
output: true if M, s σ φ.

switch φ docase hπiϕ3
Let Sig be the array consisting of atomic
programs and formulas in π ordered according
to their first appearances;4ω ← Sig[1] /* ω is the candidate sequence
we want to test */;5
while |ω| ≤ |π| do
6
if memb chec(ω, π) then7
for i = 1 to SM do
8
if (s, si ) ∈ P W (ω, σ) then9
if MC(M, sj , σr(ω), ϕ) then10
return true;
1

2

11

ω ← next(ω, Sig) /* calculate the next
sequence lexicographically according
to the order Sig */;12

return false;
case KϕU |σBn = CNU(U, σ)/* calculate the vector
representation of U|σ */for m = 1 to |SM | do
U| σif (Bn )m = 1 and MC(M, sm , σ, ϕ) =false thenreturn false;

13
14
15
16
17
18

259

return A;

return true;

Distinguishing Cause from Effect Based on Exogeneity[Extended Abstract]
Kun Zhang

MPI for Intelligent Systems
72076 Tübingen, Germany &Info. Sci. Inst., USC
4676 Admiralty Way, CA 90292

kzhang@tuebingen.mpg.de

Jiji Zhang

Department of Philosophy
Lingnan University
Hong Kong S.A.R., China

jijizhang@ln.edu.hk

ABSTRACT
Recent developments in structural equation modeling haveproduced several methods that can usually distinguish causefrom effect in the two-variable case. For that purpose, however, one has to impose substantial structural constraints
or smoothness assumptions on the functional causal models. In this paper, we consider the problem of determining
the causal direction from a related but different point ofview, and propose a new framework for causal direction determination. We show that it is possible to perform causal
inference based on the condition that the cause is “exogenous” for the parameters involved in the generating process from the cause to the effect. In this way, we avoid
the structural constraints required by the SEM-based approaches. In particular, we exploit nonparametric methods
to estimate marginal and conditional distributions, and propose a bootstrap-based approach to test for the exogeneity
condition; the testing results indicate the causal direction
between two variables. The proposed method is validated
on both synthetic and real data.

Categories and Subject Descriptors
H.4 [Information Systems Applications]: Miscellaneous;
I.2.4 [Artificial Intelligence]: Knowledge Representation
Formalisms and Methods—Miscellaneous

Bernhard Schölkopf

Max-Planck Institute forIntelligent Systems
72076 Tübingen, Germany

bs@tuebingen.mpg.de

Here we are concerned with the two-variable case, in whichconstraint-based methods, such as the PC algorithm [23],do not apply. We assume that the given observations are
i.i.d., i.e., there is no temporal information. Recently, causal
discovery based on structural equation models (SEMs) has
proved useful in distinguishing cause from effect [21, 8, 27,
26, 15, 29]; however, the performance of such approaches depends on assumptions on the functional model class and/oron the data-generating functions. On the other hand, there
have been attempts in different fields to characterize properties related to causal systems. One such concept (or familyof concepts) is known as exogeneity, which is salient in econometrics [3, 4]. Roughly speaking, the notion expresses theproperty that the process that determines one variable X isin some sense separate from or independent of the processthat determines another variable, say Y , given the value ofX.
The sense of “separateness” or “independence” in the roughidea has been specified in several ways for different purposes,
which result in different concepts of exogeneity. The conceptthat is most relevant in this paper is the one in the context ofmodel reduction, which was originally proposed as a condition that justifies inferences about the parameters of interest
based on the conditional likelihood function rather than the
joint likelihood function [7]. Here is the basic idea. Suppose
the joint distribution of (X, Y ) can be factorized as

General Termsp(X, Y |θ, ψ) = p(Y |X, ψ)p(X|θ).

Algorithms, Theory

where the conditional distribution p(Y |X) is parameterizedby ψ alone, and the marginal distribution p(X) by θ alone.
According to [3, 19], X is said to be exogenous for ψ (or any
parameter of interest that is a function of ψ), if ψ and θ arevariation free1 , or in other words, are not subject to ‘crossrestrictions”. From the frequentist point of view, this implies
that ψ and θ are independently estimable: the MLE of ψand that of θ are statistically independent according to thesampling distribution. From the Bayesian point of view [4],this implies that ψ and θ are a posteriori independent given
independent priors on them.
In this paper we will exploit the above idea to develop
a test of whether there exists a parameterization (θ, ψ) forp(X, Y ) such that X is exogenous for ψ, the parameters for

Keywords
Causal discovery, causal direction, exogeneity, statistical independence, bootstrap

1. INTRODUCTION
Understanding causal relations allows us to predict theeffect of changes in a system and control the behavior of
the system. Since randomized experiments are usually expensive and often impracticable, causal discovery from nonexperimental data has attracted much interest [18, 23]. To
do this, it is crucial to find (statistical) properties in the nonexperimental data that give clues about causal relations. For
instance, under the causal Markov condition and faithfulness
assumption, the causal structure can be partially estimatedby constraint-based methods, which make use of conditional
independence relationships.

261

(1)

1
This is actually the definition of “weak exogeneity” in [3],where three types of exogeneity were defined. Here we consider the i.i.d. case where there is no temporal information,and consequently strong exogeneity in [3] and weak exogeneity conincide.

p(Y |X). The test is based on bootstrap and is applicable
in nonparametric settings. We will also argue that if X is a
cause of Y and there is no confounding, then there should
exist a parameterization such that X is exogenous for theparameters for p(Y |X). Thus the nonparametric test canbe used to indicate the causal direction between two variables, when the test passes for one direction but fails for theother. Compared to the SEM-based approach, an important novelty of this work is to use exogeneity as a new criterion for causal discovery in general settings, which allows
distinguishing cause from effect and detecting confounders
without structural constraints on the causal mechanism.2

2. EXOGENEITY AND CAUSALITY
In this section we define what “exogeneity” means in thispaper, and explain its link to causal asymmetry. The concept of exogeneity we will use is adapted from the concept
known in econometrics as weak exogeneity, which is in itself
a statistical rather than a causal concept.3 We will showthat this statistical notion can nonetheless be exploited toformulate a method that can often determine the causal direction between two variables.

2.1

Exogeneity

The concept of weak exogeneity, as formulated by Engle,
Hendry, and Richard (EHR) [3], is concerned with when efficient estimation of a set of parameters of interest can bemade in a conditional submodel. For the purpose of thispaper, suppose we are given two continuous random variables X and Y , on which we have i.i.d. observations thatare drawn according to a joint density p(X, Y |φ). By a reparameterization we mean a one-to-one transformation of theparameter set φ. Our definition below is adapted from theEHR definition, adjusted for our present purpose and setup:

the MLEs θ̂ and ψ̂ are independent according to the sampling distribution. The concept of exogeneity formalizes theidea that the mechanism generating the exogenous variable
X does not contain any relevant information about the parameter set ψ for the conditional model p(Y |X).
The concept of cut also has a Bayesian version: [4, 16].
Definition 2 (Bayesian cut). [(Y |X; ψ), (X; θ)] operates a Bayesian cut on p(X, Y |θ, ψ) if(i.) ψ and θ are independent a priori, i.e., ψ ⊥⊥ θ,(ii.) θ is sufficient for the marginal process of generating X,
i.e., ψ ⊥⊥ X|θ, and(iii.) ψ is sufficient for the conditional process of generating
Y given X, i.e., θ ⊥⊥ Y |(ψ, X).
A Bayesian cut allows a complete separation of inference(on θ) in the marginal model and of inference (on ψ) in theconditional model. The prior independence between θ andψ in the Bayesian cut is a counterpart to the variation-free
condition in the classical cut, and the last two conditions in
Definition 2 implies condition (i.) in Definition 1. Thus, theBayesian cut is equivalent to the classical cut in sampling
theory, and for the purpose of this paper can be regarded asinterchangeable. Therefore, the exogeneity of X relative to
Y can also be defined as that there exists a reparameterization (θ, ψ) of p(X, Y ) such that [(Y |X; ψ), (X; θ)] operates
a Bayesian cut on p(X, Y |θ, ψ).

2.2

Possible Situations Where the Parameterization Fails to Operate a Bayesian Cut

Fig. 1(a) shows a data-generating process of X and Y
from where [(Y |X; ψ), (X; θ)] operates a Bayesian cut. Notethat in Definition 2, the two requirements of sufficiency ofψ and θ for the marginal and the conditional (conditions(ii.) and (iii.)), respectively, are only restrictive under theassumption of prior independence of θ and ψ (condition (i.));Definition 1 (Exogeneity of X for p(Y |X)). Suppose otherwise, conditions (ii.) and (iii.) can be trivially met by,for example, taking θ and ψ to be the same. In fact, anyp(X, Y ) is parameterized by φ. X is said to be exogenous fortwo conditions in Definition 2 could be trivial, given that thethe conditional P (Y |X) (or simply, exogenous relative to Y )other does not hold. Fig. 1(b–d) shows the situations where
if and only if there exists a reparameterization φ → (θ, ψ),conditions (i.), (ii.), and (iii.) are violated, respectively. Insuch that
all those situations, θ and ψ are not independent a posteriori.
(i.) p(X, Y |θ, ψ) = p(Y |X, ψ)p(X|θ), and(ii.) θ and ψ are variation free, i.e., (θ, ψ) ∈ Θ × Ψ, where
2.3 Relation to Causality
Θ and Ψ denote the set of admissible values of θ and ψ,
As Pearl [18] rightly stressed, the EHR concept of weak
respectively.
exogeneity is a statistical rather than a causal notion. Unlike
Here “variation free” means that the possible values thatthe concept of super exogeneity, it is not defined in terms of
one parameter set can take do not depend on the values
interventions or multiple regimes. That is why, as we willof the other set. Clauses (i.) and (ii.) in Definition 1
show, the hypothesis that X is exogenous relative to Y in
are the defining conditions for the concept of a (classical)
the sense we defined is generally testable by observational
cut: [(Y |X; ψ), (X; θ)] is said to operate a (classical) cut on
data. However, it is also linked to causality in that it isp(X, Y |θ, ψ) if (i.) and (ii.) are satisfied. The cut implies
arguably a necessary condition for an unconfounded causalthat the maximum likelihood estimates of θ and ψ can be
relation: if X is a cause of Y and there is no common cause
computed from p(X|θ) and p(Y |X, ψ), respectively, and soof X and Y , then X is exogenous relative to Y in the sense
we defined.4 This follows from the principle we indicated2
A related criterion is that of algorithmic independence
at the beginning: if X is an unconfounded cause of Y , thenbetween the input distribution p(X) and the conditional
the process or mechanism that determines X is separate orp(Y |X) postulated for a causal system X → Y [11]; see
independent from the process or mechanism that determines
also [10]. The algorithmic independence condition is defined
Y given X. The separation of processes ensures the existence
in terms of Kolmogorov complexity, which is uncomputable,of separate parameterizations of the processes, which will
and the method proposed in this paper provides an alternative way to assess the “independence” between p(X) andthen satisfy our definition of exogeneity.
p(Y |X).
3
4
In this paper we use ”unconfounded” to mean the absence
The stronger, causal concept of exogeneity is known as suof any common cause.
per exogeneity.

262

ψ

yi

xi

θ

ψ

γ

8
6

yi

xi

θ
n

n

(a)

(b)

ψ

ψ

4

shape of pY

γ

θ

γ

yi

xi

θ

xi

n

(c)

Y

2
0

data
E(Y|X)
E(X|Y)

−2

yi
n

−4

(d)

−6

Figure 1: Graphical representation of the datagenerating process. (a) [(Y |X; ψ), (X; θ)] operates aBayesian cut (implying that X and ψ are mutuallyexogenous). (b), (c), and (d) correspond to threesituations where [(Y |X; ψ), (X; θ)] does not operate aBayesian cut: (b) ψ and θ are dependent a priori, as
both of them depend on γ, which is a function of θor ψ; (c) θ is not sufficient in modeling the marginal
distribution of X, where γ is a function of ψ; (d) ψ
is not sufficient in modeling the conditional distribution of Y given X, where γ is a function of θ.

We have argued that if X and Y are causally related andunconfounded, the exogeneity property holds for the correctcausal direction. Furthermore, if it turns out that there is
one and only one direction that admits exogeneity, then the
direction for which the exogeneity property holds must be
the correct causal direction. This suggests the following approach to inferring the causal direction between X and Y
based on some tests of exogeneity, assuming that X and Y
are causally related and that there is no common cause of
X and Y (or in other words, X and Y form a causally sufficient system): test whether (1) X is exogenous for p(Y |X)
and whether (2) Y is exogenous for p(X|Y ), and if one ofthem holds and the other does not, we can infer the causal
direction accordingly. Of course it may also turn out thatneither (1) nor (2) holds, which will indicate that the assumption of causal sufficiency is not appropriate, or thatboth (1) and (2) hold, which will indicate that the causal
direction in question is not identifiable by our criterion.5
A familiar example of a non-identifiable situation is when
X and Y follow a bivariate normal distribution. In thatcase, as shown by EHR [3], there is a cut [(Y |X; ψ), (X; θ)]in one direction, as well as a cut [(X|Y ; ψ̃), (Y ; θ̃)] in theother. Below we give an example where the causal direction
is identifiable based on exogeneity.
An example of identifiable situation: Linear nonGaussian case.
Let X follow
P a Gaussian mixture model with two Gaussians, X ∼ 2i=1 πi N (ui , σi2 ), where πi > 0 and π1 +π2 = 1,
and let Y = c + βX + E where E ∼ N (0, σ 2 ). Therefore
5Note that we are not concerned with the case in which
X and Y are not causally connected and hence statisticallyindependent; in that case, exogeneity trivially holds in both
directions.

263

−8
−8

shape of pX
−6

−4

−2

0

2

4

6

8

X

Figure 2: An illutration on the identifiability of alinear non-Gaussian model based on “exogeneity”.
X is generated by a mixture of two Gaussians, andY is generated by Y = X + E, where E ∼ N (0, 1).
Here X is exogenous for parameters in pY |X , while
Y is not exogenous for parameters in pX|Y .

θ = {πi , µi , σi }2i=1 and ψ = {c, β, σ 2 }. We then haveX
p(X, Y |θ, β) =πi N (x; µi , σi2 )N (y; c + βx, σ 2 )
i

=

X
i

πi N (y; µ̃i , σ̃i2 )N (x; c̃i + β̃i y, γi2 ),where µ̃i = c + βµi , σ̃i2 = β 2 σi2 + σ 2 , c̃i =σ 2 σi2βσi2, and γi2 = βσ2 +σ
2 . That is,
βσi2 +σ 2iY ∼p(X|Y, θ, ψ) =

2
X
i=1

πi N (y; ũi , σ̃i2 ), andX πi N (y; µ̃i , σ̃i2 )iµi σ 2 −cβσi2, β̃i =
βσi2 +σ 2p(Y |θ, ψ)· N (x; c̃i + β̃i y, γi2 ).

Clearly, if π1 π2 6= 0, no matter how one parametrizes
the density of Y , the conditional distribution of X given
Y would involves those parameters that model the marginaldensity of Y . The sufficient parameter set of the distributionof Y , θ̃, and that of the conditional distribution of X given
Y , ψ̃, cannot be variation-free or independent a priori; see
Fig. 1(b). Alternatively, one can keep those parameters thatare independent a priori from θ̃ in ψ̃, i.e., ψ̃ and θ̃ become
independent a priori, but ψ̃ is then not sufficient in modelingp(X|Y ); see Fig. 1(d). In both situations Y is not exogenousfor ψ̃. Hence in this linear non-Gaussian case the exogeneity
condition only holds for the direction X → Y , and the causal
direction is identifiable. Fig. 2 gives an intuitive illustration
on how the shape of P (Y ) and that of E(X|Y ), which isdetermined by P (X|Y ), are related.

3.

CAUSAL DIRECTION DETERMINATIONBY TESTING FOR EXOGENEITY WITH
BOOTSTRAP

Algorithm 1 Finding causal direction between X and Y
based on exogeneity
Input: data (x, y)
Output: three possibilities: causal direction between X
and Y , or non-identifiable causal direction by exogeneity,or existence of hidden confunders
If Exogeneity(X → Y )3.1 Bootstrap-Based Test for Bayesian Cut in
If Exogeneity(Y → X)
the Parametric Case
if exogeneity holds for only one direction thenreturn the direction in which exogeneity holds
In this section, we assume that a parametric form p(X, Y |θ, ψ) =else if exogeneity holds for both directions thenp(X|θ)p(Y |X, ψ) is given. We would like to see whether the
print non-identifiable causal direction by exogeneity
estimates of θ and of ψ in (1) are independent, accordingelse. exogeneity does not hold in either direction
to the sampling distribution; in other words, with a noninprint confounder caseformative prior, we want to test if the posterior distribution
end ifp(θ, ψ|D) has no coupling between θ and ψ. In this case weare examining if [(Y |X; ψ), (X; θ)] operates a Bayesian cut.
procedure If Exogeneity(X → Y )Bootstrap has been used in the literature to assess thefor b = 1 to B dodependence, as well as uncertainty, in the parameter estidraw bootstrap resample (x∗(b) , y∗(b) ) by randommates according to the sampling distribution; see e.g. [2,samplingwith replacement from (xi , yi );Sec. 5.7]. For clarity, Table 1 gives the notation used in the
∗(b)
∗(b)
proposed bootstrap-based method. Suppose we draw bootestimate p̂X (X = x̃) and HY |X (x̃) with methods
∗(b)
∗(b)
strap resamples (x , y), b = 1, ..., B, from the original
given in Sec. 3.2.1
sample (x, y) = (xi , yi )N
i=1 with paired bootstrap, i.e., eachend for∗(b)
resample (x∗(b) , y∗(b) ) is obtained by independently drawing
test for independence between p̂X (X = x̃) and∗(b)
N pairs from the original sample with replacement. On each
HY |X (x̃), b = 1, ..., B, with the method given in Sec. 3.2.2of them, we can calculate the parameter estimates θ̂∗(b) andreturn independence test result∗(b)ψ̂. The independence between θ and ψ according to theend proceduresampling distribution is then transformed to statistical independence between the bootstrap estimates θ̂∗(b) and ψ̂ ∗(b) ,b = 1, ..., B. To assess the latter, any independence test
the “parameters” θ and ψ are not observable. The previmethod, such as the correlation test, would apply.
ous argument shows that if there exists (θ, ψ) admitting a∗(b)
Bayesian cut, log p̂∗(b) (X = x̃) and HY |X (x̃) are indepen3.2 Bootstrap-Based Test for Exogeneity in thedent; otherwise they are always dependent. In words, testNonparametric Case
ing for independence between the bootstrapped estimates
Let x̃ be a fixed set of values of X, and x̃i be a point in∗(b)log p̂∗(b) (X = x̃) and HY |X (x̃) is actually a ways to assessx̃. x̃ can be drawn from the given data set, or randomly
the exogeneity condition. Algorithm 1 sunmmarizes the prosampled on the support of X, given that it contains enoughposed procedure to determine the causal direction between
points such that the values of P (X) and p(Y |X) evaluated
X and Y , given the sample (x, y) as input. In particular, it
at x̃ well approximate the continuous densities. In our exinvolves the following two modules.
periments we used 80 evenly-spaced sample points between
the minimum and maximum values of X as x̃ (so its length
3.2.1 Module 1: Nonparametric Estimators of p(X)is N = 80).
and p(Y |X)∗(b)
On the bootstrap resamples, log p̂ (X = x̃) is fully de∗(b)
∗(b)
When
testing for exogeneity, one assumes the (paramettermined by θ̂; similarly, log p̂ (Y |X = x̃) is a functionric) model is correctly specified. Otherwise, if the model is
∗(b)
∗(b)
∗(b)
of ψ̂
, and so is the quantity HY |X (x̃) , EY |X log p̂ (Y |X =over-simplified, the estimated conditional distribution willx̃). Note that p̂∗(b) (Y |X = x̃i ) is the estimated distribution
depend on the marginal, which inspires the importance∗(b)
reweighting scheme to handle learning problems under coof Y at X = x̃i , and hence HY |X (x̃) can be considered as
variate shift (see e.g., Footnote 1 in [24]). For example, let
negative entropies of Y on the bth bootstrap resample evalus consider the situation where Y depends on X in a nonuated at X = x̃.
linear manner while a linear model is exploited to estimate
Suppose all involved parameters are identifiable, i.e., thepY |X ; clearly the estimate of the parameters in the condimappings θ 7→ p(X|θ) and ψ 7→ p(Y |X, ψ) are both one-totional model would depend on that in pX . To avoid this,one [14]. Then the mapping between θ̂∗(b) and log p̂∗(b) (X =we use flexible nonparametric models to estimate the condi∗(b)∗(b)x̃) and that between ψ̂and log p̂ (Y |X = x̃) are bothtional.
∗(b)∗(b)
one-to-one. Hence, the independence between θ̂and ψ̂
,Suppose we aim to verify if X exogenous for effective “pa∗(b)b = 1, ..., B, implies that between log p̂ (X = x̃) andrameters”
in P (Y |X). We need to estimate the marginal∗(b)
HY |X (x̃).
distribution p(X) and the conditional distribution p(Y |X)
As a consequence, in nonparametric settings, we can imagon the original sample as well as each bootstrap resample.
ine that there exist effective parameters θ and ψ, and can
We estimate p(X) with Gaussian kernel density estimation,
still assess where they follow a Bayesian cut by testing for inand the kernel width was selected by Silverman’s rule of∗(b)dependence between the bootstrapped estimates log p̂ (X =thumb [22, page 48].
∗(b)x̃) and HY |X (x̃). Note that in the nonparametric case,To estimate the conditional density p(Y |X), we adapted
We now describe our approach to testing exogeneity. We
will first illustrate how bootstrap can be used to test whether
a given parametric model constitutes a (Bayesian) cut, andthen develop a nonparametric test for exogeneity based onbootstrap.

264

Table 1: Notation involved in the proposed method based on exogeneity and bootstrap(x, y)
given sample of (X, Y )(x∗(b) , y∗(b) )bth bootstrap resampleθ̂∗(b) , ψ̂ ∗(b)estimate of parameters θ and ψ on (x∗(b) , y∗(b) )∗(b)p̂ (X = x̃)marginal densities estimated on (x∗(b) , y∗(b) ) evaluated at X = x̃p̂∗(b) (Y |X = x̃) conditional densities estimated on (x∗(b) , y∗(b) ) evaluated at X = x̃∗(b)
HY |X (x̃)
quantity associated with p̂∗(b) (Y |X = x̃), defined as EY |X log p̂∗(b) (Y |X = x̃) on (x∗(b) , y∗(b) )the method orignally proposed for causal inference based on
the structural equation Y = f (X, E) [15]. This method
aims to find the functional causal model Y = f (X, E),
where E ⊥⊥ X, given (x, y). Without loss of generality,
one can assume that E ∼ N (0, 1). (Otherwise, one canalways write E = g(Ẽ) where g is some appropriate function and Ẽ ∼ N(0, 1), and use the functional causal model
Y = f X, g(Ẽ) instead.) Here f is completely nonparametric: it takes a Gaussian process prior with zero meanfunction and covariance function k (x, e), (x0 , e0 ) , where k
is a Gaussian kernel, and (x, e) and (x0 , e0 ) are two pointsof (X, E). Like in [13], this method optimizes the valuesof E, denoted by êi , as well as involved hyperparameters,
and produces the maximum a posterior (MAP) solution off , by maximizing the approximate marginal likelihood. The
functional causal model implies the conditional density:
 ∂f
. ∂f
p(X, E) | ∂E|
p(X, Y )== p(E) ||.
P (Y |X) =
p(X)
p(X)
∂E

Finally, once we have the êi and the estimate of f , the conditional density at each point can be estimated as p(Y =∂fyi |X = xi ) = p(E = êi )/ ∂E(xi , êi ) .

3.2.2

Module 2: Testing for Independence Between
High-Dimensional Vectors

The task is then to test for independence between the estimated quantities on the bootstrap resamples, log p̂∗(b) (X =∗(b)x̃) and HY |X (x̃), b = 1, ..., B. Their dimentions are thenumber of data points in x̃, which is 80 in our experiments.
Let R be the matrix consisting of the centered version oflog p̂∗(b) (X = x̃i ), obtained on all bootstrap resamples, i.e.,the (i, b)th entry of R is
B

Rib , log p̂(X ∗(b) (X = x̃i ) −1 X
log p̂∗(k) (X = x̃i ).
B
k=1

∗(b)

Similarly, S contains the centered version of HY |X (x̃i ), i.e.,B∗(b)Sib , HY |X (x̃i ) −1 X ∗(k)HY |X (x̃i ).
B
k=1

Both R and S are of the size N ×B. We define thestatistic as CX→Y , Tr (RST )(RST )T = Tr(RT R · ST S),
which is actually the sum of squares of the covariances between all rows of R and those of S. The distribution of thisstatistic under the null hypothesis that log p̂∗(b) (X = x̃) and∗(b)
HY |X (x̃) are independent can then be constructed by permutation test.
Note that this statistic is actually the Hilbert-Schmidt independence criterion (HSIC) [6] with a linear kernel. Thatis, we care about linear dependence between log p̂∗(b) (X =

265

∗(b)

x̃) and HY |X (x̃); this is reasonable because they are in thevicinity of the maximum likelihood estimates and their dependence can be captured by linear approximation. On theother hand, if we use HSIC with Gaussian kernels, the result will be sensitive to the kernel width because the data
dimension (the number of rows of R and S) is high.

4.

EXPERIMENTS

In this section we first evaluate the behavior of the proposed bootstrap-based method for causal inference with synthetic data, on which the ground-truth is known, and thenapply it on real data. We use two variables, and with synthetic data, we examine both the case where the two variables have a direct causal relation and the confounder case(i.e., there are confounders influencing both of them). We
compare the proposed bootstrap-based approach with theadditive noise model (ANM) proposed in [8]), GPI [15], andinformation-geometric causal inference (IGCI) approach [10]:ANM assumes that the effect is a nonlinear function of thecause plus additive noise, GPI applies the Gausian Process
prior on the generating function, and IGCI assumes thetransformation from the cause to the effect is deterministic, nonlinear, and independent from the distribution of thecause in a certain way. For computational reasons, we used1000 bootstrap replications.
Simulation: Without Confounders. Inspired by thesettings in [8, 15], we generated the simulated data with themodel Y = (X + bX 3 )eαE + (1 − α)E, where X and E wereobtained by passing i.i.d. Gaussian samples through power
nonlinearities with exponent q, while keeping the original
signs. The parameter α controls the type of the observationnoise, ranging from purely additive noise (α = 0) to purely
multiplicative noise (α = 1). b determines how nonlinear
the effect of X is, and when b = 0 the model is linear. The
parameter q controls the non-Gaussninity of X and E: q = 1
corresponds to a Gaussian distribution, and q > 1 and q <1 produce super-Gaussian and sub-Gaussian distributions,
respectively.
We considered three situations, in each of which two ofq, b, and α were fixed and we see how the other changes
the performance of different methods. For each combination of q, b, and α, we independently simulated 10 data setswith 500 data points.6 Fig. 3 shows the accuracy of theconsidered methods. One can see that the accuracy of thebootstrap-based approach is among or close to the best results, indicating that it is able to perform causal inferencein various situations. We note that in practice, the performance of the bootstrap-based approach depends on thenumber of bootstrap replications and the method used for
conditional distribution estimation. Although due to computatioanl reasons, we did not try a larger number of boot6

Since the bootstap-based approach is rather timeconsuming, we only simulated 10 data sets for each setting.

100

60

0.25

0.5
α

0.75

1

(a) Changing α: From additive to multiplicative noise
100

Correct direction

60
Bootstrap
GPI
IGCI
ANM

40
20
0
0.2

0.6

1
q

1.4

1.8

(b) Changing q: From sub-Gaussian to super-Gaussian
additive noise

6

4

2

\beta = 0

0.25

0.50
β

0.75

1.00

(a) Situation 1: Linear confounder case.

80

Correct direction

60

ConfounderWrong direction

10

40

Bootstrap
GPI
IGCI
ANM

20

−0.5

0
b

0.5

# replications (total: 10)

Accuracy

Wrong direction

8

0

100

0
−1

Confounder

10

GPI
IGCI
ANM

Accuracy

80

Bootstrap001(c) Changing b: Various nonlinear functions with Gaussian
additive noise

8

6

4

2

0

Figure 3: Accuracy of correctly estimating thecausal direction for different generating models: (a)q = 1, b = 1, and α changed from 0 to 1, (b) for alinear function (b = 0) with additive noise (α = 0)
which changed from sub-Gaussian (q < 1) to subGaussian (q > 1), and (c) various nonlinear functions(b changed from -1 to 1) with additive Gaussian noise(q = 1, α = 0).

GPI
IGCI
ANM

20

Bootstrap
GPI
IGCI
ANM

Bootstrap

40# replications (total: 10)

Accuracy80

strap replications, generally speaking, the accuracy of thebootstrap-based method improves as the number of replications increases.
Simulation: With Confounders.
We then include
the confounder variable Z in the system, so that the causal
structure is Z → X and (Z, X) → Y . For simplicity, we assume that both X and Y are influenced by Z in a linear form:
X = (2−β)EX +βZ, and Y = 0.3(2−β) (X+bX 3 )eαE +(1−α)E + βZ, where EX , Z, and E were obtained by passing
i.i.d. Gaussian samples through power nonlinearities withexponent q = 1.5, and β controls how strong the effect of Z
is on both X and Y . We considered two situations: in one ofthem, we set α = 0 and b = 0, i.e., the whole model is linear;in the other situation, α = 0.2, and b = 0.3, so the model
contains both additive noise and multiplicative noise. Wechanged β from 0 to 1, and Fig. 4 shows the performancesof the four methods in the two situations; note that for eachvalue of β, the four bars (from left to right) correspond tothe bootstrap-based method, GPI, IGCI, and ANM. In particular, one can see that the bootstrap-based method tends
to detect the presence of the confounder when its effect issignificant.

\beta = 0

0.25

0.50
β

0.75

1.00

(b) Situation 2: Nonlinear confounder case.
Figure 4: Number of replications in which the methods find correct directions, report existence of confounders, and give wrong directions, respectively.
For each value of β, the four bars correspond to bootstrapbased method, GPI, IGCI, and ANM (from left to right).
On Real Data. We applied the bootstrap-based method266

on the cause-effect pairs available at
http://webdav.tuebingen.mpg.de/cause-effect/.
To reduce computational load, we used at most 500 points
for each cause-effect pair. On 20 pairs (pairs 21, 43, 45, 4851, 56-58, 61-64, 72, 75, 77-79, and 81), the p-values of the
independence test for both directions are smaller than 0.01,
indicating that there might be significant confounders. This
seems reasonable, as the data scatter plots for these pairs
indicate that the two variables have complex dependence
relationships. On the remaining 57 data sets, the bootstrapbased method output correct causal directions on 41 of them(with an accuracy 72%). We also applied the recently proposed causal inference approaches, including IGCI [10], theapproach based on the Gaussian process prior [15], and thatbased on the post-nonlinear causal model [27] on those 57
data sets for comparison. Their performance was similar:
the three approaches gave correct causal directions on 41,
40, and 43 pairs, respectively.

5. CONCLUSION AND DISCUSSIONSWe proposed to do causal inference based on the criterion of exogeneity of the cause for the parameters in theconditional distribution of the effect given the cause. We
discussed how to assess such exogeneity in nonparametricsettings. To this end, one needs to draw a number of samples
according to the unknown data-generating process. Fortunately, the bootstrap provides a way to mimic the data generating process from which we can draw a number of samples
and analyze their statistical properties.
Our approach shows that it is possible to determine causal
direction without structural constraints or a specific type ofsmoothness assumptions on the functional models. The proposed computational approach successfully demonstrated thevalidity of this idea, though it is computationally demanding because of the bootstrap procedure and its performance
is not necessarily the best among existing methods. At thesame time, it enjoys some advantages. First, it does notmake a strong assumption on the data-generating process.
Second, it could often tell us if significant confounders exist.
The performance of the proposed bootstrap-based approach
depends on the number of bootstrap replications and themethod for conditional distribution estimation. In future
work we aim to develop more reliable methods along thisline, including methods that can handle more than two variables.
In this paper we made an attempt to discover causal information from observational data based on a condition ofexogeneity, which provides another perspective to conceptualize the ”independence” between the process generating
the cause and that generating the effect from cause. On
the other hand, it is worth mentioning that this type of independence is able to facilitate understanding and solving
some machine learning or data analysis problems. For instance, it helps understand when unlabeled data points will
help in the semi-supervised learning scenario [20], and inspired new settings and formulations for domain adaptationby characterizing what information to transfer and how to
do so [28, 25].

Acknowledgement
We would like to thank Peter Spirtes, Kevin Kelly, and Dominik Janzing for helpful discussions. K. Zhang was sup-267

ported in part by DARPA grant No. W911NF-12-1-0034.
The research of J. Zhang was supported in part by the Research Grants Council of Hong Kong under the General Research Fund LU342213.

6.

REFERENCES[1] W. A. Coppel. Number theory: An introduction tomathematics. Springer, second edition, 2009.
[2] B. Efron. The Bootstrap, Jacknife and otherResampling Plans. SIAM, Philadelphia, 1982.
[3] R. F. Engle, D. F. Hendry, and J. F. Richard.
Exogeneity. Econometrica, 51:277–304, 1983.
[4] J. P. Florens and M. Mouchart. Conditioning in
dynamic models. Journal of Time Series Analysis,6(1):15–34, 1985.
[5] J. P. Florens, M. Mouchart, and J. M. Rolin. Elementsof Bayesian Statistics. Marcel Dekker, New York,1990.
[6] A. Gretton, O. Bousquet, A. J. Smola, and
B. Schölkopf. Measuring statistical dependence with
Hilbert-Schmidt norms. In S. Jain, H.-U. Simon, andE. Tomita, editors, Algorithmic Learning Theory: 16th
International Conference, pages 63–78, Berlin,
Germany, 2005. Springer.
[7] D. Henry. Econometrics: Alchemy or Science? Oxford
University Press, Oxford, 2000.
[8] P. Hoyer, D. Janzing, J. Mooji, J. Peters, and
B. Schölkopf. Nonlinear causal discovery with additivenoise models. In Advances in Neural Information
Processing Systems 21, Vancouver, B.C., Canada,
2009.
[9] A. Hyvärinen and P. Pajunen. Nonlinear independent
component analysis: Existence and uniqueness results.
Neural Networks, 12(3):429–439, 1999.
[10] D. Janzing, J. Mooij, K. Zhang, J. Lemeire,
J. Zscheischler, P. Daniuvsis, B. Steudel, and
B. Schölkopf. Information-geometric approach toinferring causal directions. Artificial Intelligence,
pages 1–31, 2012.
[11] D. Janzing and B. Schölkopf. Causal inference using
the algorithmic markov condition. IEEE Transactions
on Information Theory, 56:5168–5194, 2010.
[12] R. Kass, L. Tierney, and J. Kadane. Asymptotics inBayesian computation. In J. Bernardo, M. DeGroot,
D. Lindley, and A. Smith, editors, Bayesian statistics
3, pages 261–278. Oxford University Press, 1988.
[13] N. Lawrence. Probabilistic non-linear principal
component analysis with Gaussian process latentvariable models. Journal of Machine Learning
Research, 6:1783–1816, 2005.
[14] E. L. Lehmann and G. Casella. Theory of point
estimation. Springer, 2nd edition, 1998.
[15] J. Mooij, O. Stegle, D. Janzing, K. Zhang, and
B. Schölkopf. Probabilistic latent variable models for
distinguishing between cause and effect. In Advancesin Neural Information Processing Systems 23 (NIPS
2010), Curran, NY, USA, 2010.
[16] M. Mouchart and E. Scheihing. Bayesian evaluation of
non-admissible conditioning. Journal of Econometrics,
123(2):283–306, 2004.
[17] G. H. Orcutt. Toward partial redirection ofeconometrics. The Review of Economics and Statistics,
34:195–200, 1952.
[18] J. Pearl. Causality: Models, Reasoning, and Inference.
Cambridge University Press, Cambridge, 2000.
[19] J. F. Richard. Models with several regimes andchanges in exogeneity. The Review of Economics
Studies, 47:1–20, 1980.
[20] B. Schölkopf, D. Janzing, J. Peters, E. Sgouritsa,
K. Zhang, and J. Mooij. On causal and anticausal
learning. In Proc. 29th International Conference on
Machine Learning (ICML 2012), Edinburgh, Scotland,
2012.
[21] S. Shimizu, P. Hoyer, A. Hyvärinen, and A. Kerminen.
A linear non-Gaussian acyclic model for causal
discovery. Journal of Machine Learning Research,
7:2003–2030, 2006.
[22] B. W. Silverman. Density Estimation for Statistics andData Analysis. Chapman & Hall/CRC, London, 1998.
[23] P. Spirtes, C. Glymour, and R. Scheines. Causation,
Prediction, and Search. MIT Press, Cambridge, MA,
2nd edition, 2001.
[24] M. Sugiyama, T. Suzuki, S. Nakajima, H. Kashima,P. von Bünau, and M. Kawanabe. Direct importanceestimation for covariate shift adaptation. Annals of theInstitute of Statistical Mathematics, 60:699–746, 2008.
[25] K. Zhang, , M. Gong, and B. Schölkopf. Multi-source
domain adaptation: A causal view. In Proceedings ofthe 29th AAAI Conference on Artificial Intelligence,
pages 3150–3157. AAAI Press, 2015.
[26] K. Zhang and A. Hyvärinen. Acyclic causality
discovery with additive noise: An
information-theoretical perspective. In Proc. European
Conference on Machine Learning and Principles andPractice of Knowledge Discovery in Databases (ECML
PKDD) 2009, Bled, Slovenia, 2009.
[27] K. Zhang and A. Hyvärinen. On the identifiability ofthe post-nonlinear causal model. In Proceedings of the25th Conference on Uncertainty in Artificial
Intelligence, Montreal, Canada, 2009.
[28] K. Zhang, B. Schölkopf, K. Muandet, and Z. Wang.
Domain adaptation under target and conditional shift.
In Proceedings of the 30th International Conference on
Machine Learning, JMLR: W&CP Vol. 28, 2013.
[29] K. Zhang, Z. Wang, J. Zhang, and B. Schölkopf. On
estimation of functional causal models: General
results and application to post-nonlinear causal model.
ACM Transactions on Intelligent Systems andTechnologies, 2015. forthcoming.

268

Supplement to“Distinguishing Cause from Effect Based on Exogeneity”

This supplementary material provides the proofs
and discussions which are omitted in the submitted paper. The equation numbers in thismaterial are consistent with those in the paper.

S1. Mutual Exogeneity and Its Relationship to
Definition 1
There are two types of analysis of exogeneity [4]; one considers the inference based on the complete sample results, andthe other considers dynamic models where the data were obtained by “sequential sampling”. In this paper we focus on
the former scenario.
From the Bayesian point of view, exogeneity of X forψ allows an admissible reduction of the complete modelp(X, Y |θ, ψ) to the conditional model p(Y |X, ψ), in thatboth models lead to he same posterior distribution on theparameter set ψ [4, 16]. Below we give the definition ofmutual exogneneity according to [4].
Definition 3 (Mutual exogeneity). X and ψ are mutually exogenous if and only if(i) ψ and X are independent, i.e., ψ ⊥⊥ X, and(ii) ψ is sufficient in the conditional distribution of Y given
X, i.e., θ ⊥⊥ Y |(ψ, X).
Here condition (i) is to do with the independence between ψ
and X; those two quantities play different roles in the modelp(X, Y, ψ, θ), and consequently this independence condition
is usually not convenient to verify. Moreover, for the same
reason, there is no fully equivalent concept in sampling theory (it is weaker than exogeneity defined in Definition 1,
because the property of θ is not specified). A natural wayof obtaining the mutual exogeneity of X and ψ is to exploit a stronger but more operational condition, namely thecondition of the Bayesian cut.
A Bayesian cut allows a complete separation of inference(on parameters θ) in the marginal distribution and of inference (on ψ) in the conditional one. The prior independence
between θ and ψ in the Bayesian cut is a counterpart to thevariation-free condition in the classical cut (condition (ii)in Definition 1), and the last two conditions in Definition 2
implies condition (i) in Definition 1. Thus, the Bayesiancut is equivalent to the classical cut in sampling theory, andconsequently characterizes the exogeneity property definedin Definition 1. Therefore, hereafter the exogeneity of X forψ is used interchangeably with the statement that [ψ, (X, θ)]operates a Bayesian cut in p(X, Y, θ, ψ).
The following theorem, extracted from [5], relates theBayesian cut to the independence of the parameters according to the posterior distribution, as well as mutual exogeneity.

269

Theorem 4. Suppose [ψ, (X, θ)] operates a Bayesian cut
in p(X, Y, {ψ, θ}); then(i) X and ψ are mutually exogenous, and(ii) ψ and θ are independent a posteriori.
On the other hand, if X and ψ are mutually exogenous andif θ ⊥⊥ ψ|X, [ψ, (X, θ)] operates a Bayesian cut.
When one (or more) condition in Definition 2 is violated,[ψ, (X, θ)] does not operate a Baysian cut, i.e., X is notexogenous for ψ. Fig. 1(b–d) shows the situations whereconditions (i), (ii), and (iii) are violated, respectively, sothat [ψ, (X, θ)] does not operate a Baysian cut. Note thatby reparameterization, the three situations can reduce toeach other. Take situations (b) and (c) as an example. If we
divide θ in (b) into (θγ , θ⊥⊥ ), where θγ depends on γ whileθ⊥⊥ does not, and consider θ⊥⊥ as the new θ, (b) becomes (c).
Similarly, if we merge γ and θ in (c) as the new θ, we thenhave (b). In all those situations, θ and ψ are not independent
a posterior, or the maximum likelihood estimates θ̂ and ψ̂are not independent according to the sampling distribution.

S2. Relation to SEM-Based Causal InferenceS2.1. Relation to Causal Inference Based on
Marginal Likelihood
Recently, SEM-based approaches have demonstrated theirpower for causal inference of real-world problems. Structural equations represent the effect as a function of thecauses and independent noise, which, from another point ofview, provide a way to represent the conditional distribution
P (effect|cause), or the causal mechanism. The generationof the cause-effect pair consists of two stages, one generating
the cause according to P (cause) and the other further generating the effect from the value of the cause according to
the structural equation. The “simplicity” constraints (e.g.,linearity in [21], additive noise in [8], the post-nonlinear process in [27], and the smoothness assumption in [15]) on thefunctions are crucial. On the one hand, they make the models asymmetric in cause and effect; otherwise, for any twovariables, we can always represent one of the variables as afunction of the other and an independent noise term [9]. On
the other hand, if the functions are constrained to be simple, the independence between the cause and the error terms
would imply the exogeneity of the cause for the parametersin P (cause), as suggested by the error-based definition ofexogeneity [17] (see also [18]).7
7
An error-based definition of exogeneity was given by [17](see also [18]): X is said to be exogenous for parameters inp(Y |X) is X is independent of all errors that influence Y ,except those mediated by X. We know that without appropriate constraints on the functions, given any two randomvariable, we can always represent one of them as a functionof the other variable and an independent noise term [9], i.e.,
the functional causal models are not identifiable. Therefore,

The concept “exogeneity” provides theoretical support for
the SEM-based causal inference methods that find the causal
direction by comparing the marginal likelihood of the models
in two directions; for an example of such methods, see [15].8
One candidate model is given in Fig. 1(a), where X is exogenous for ψ (or [ψ, (X, ψ)]) operates a Bayesian cut inp(X, Y, θ, ψ), denoted by M1 . The other corresponds to thefactorization:p(X, Y |θ̃, ψ̃) = p(Y |θ̃)p(X|Y, ψ̃),(2)

where [ψ̃, (Y, θ̃)] operates a Bayesian cut in p(Y, X, θ̃, ψ̃),denoted by M2 . Note that under the above models, themarginal likelihood of (X, Y ) is the product of that of theconditioning variable and that of the conditional distribution. Ideally, if all the involved distributions are correctly
specified, one would prefer the causal direction X → Y(resp. Y → X) if M1 (resp. M2 ) gives a higher marginal
likelihood.
Theorem 5. Suppose that the two random variables X
and Y are generated according to M1 , and that the exogeneitybased causal model is identifiable. Let the prior distributionsof the parameters be p∗ (ψ|M1 ) and p∗ (θ|M1 ). For the given
sample (X, Y), let p(X, Y|M1 ) be the marginal likelihood,i.e.,p(X, Y|M1 )N ZZY=p(Xi , Yi |{ψ, θ})p∗ (θ|M1 )p∗ (ψ|M1 )dθdψ
i=1

=

N Z
Y

i=1

=

N
Y

i=1

p(Xi |θ)p∗ (θ|M1 )dθ ·p(Xi |M1 ) ·

N
Y

i=1

N Z
Y

i=1

p(Yi |Xi , ψ)p∗ (ψ|M1 )dψp(Yi |Xi , M1 ).

Assume that by a one-to-one reparametrization we can represent p(X, Y |{ψ, θ}) as p(Y |θ̃)p(X|Y, ψ̃), where Y is notexogenous for ψ̃. Let p(X, Y|M2 ) be the marginal likelihoodof M2 , i.e.,p(X, Y|M2 )N ZZY=p(Xi , Yi |{ψ̃, θ̃})p0 (θ̃|M2 )p0 (ψ̃|M2 )dθ̃dψ̃
i=1

=

N
Y

i=1

p(Yi |M2 ) ·

N
Y

i=1

p(Xi |Yi , M2 ),where θ̃ and ψ̃ have independent priors. As the sample
size N goes to infinity, for any choice of p0 (θ̃|M2 ) andp0 (ψ̃|M2 ), p(X, Y|M1 ) is always greater than p(X, Y|M2 ).
Proof. As the data were generated according to model
M1 , we haveZ
E log p(X, Y |M1 ) = p(X, Y |M1 ) log p(X, Y |M1 )dxdy.

generally speaking, the above error-based definition is consistent with Definition 1 only when the functional class iswell constrained. Otherwise, if the function and the distribution of the assumed cause are related in some way, theabove definition is not rigorous.
8
Note that due to computational difficulties, this method
doe snot evaluate the marginal likelihood, but approximate
it wiht the maximum regularized likelihood.

Furthermore,
E log p(X, Y |M1 ) − E log p(X, Y |M2 )Z
p(X, Y |M1 )dxdy= p(X, Y |M1 ) log
p(X, Y |M2 )=D p(X, Y |M1 ) || p(X, Y |M2 ) ,where D(·||·) denotes the Kullback-Leibler divergence. Clearly
the above quantity is non-negative, and it is zero if and only
if p(X, Y |M1 ) = p(X, Y |M2 ) for all possible x and y. However, this condition cannot hold, because the model M1 isassumed to be identifiable based on exogeneity.
Consequently, we have E log p(X, Y |M1 ) > E log p(X, Y |M2 ).
Moreover, according to the weak law of large numbers, as
N → ∞, N1 log p(X, Y|M1 ) and N1 log p(X, Y|M2 ) will convergence in probability to the quantities E log p(X, Y |M1 )and E log p(X, Y |M2 ), respectively. That is, if N is large
enough, p(X, Y|M1 ) > p(X, Y|M2 ).
However, the marginal likelihood depends heavily on themodels or assumptions for the marginal and conditional distributions. Besides the exogeneity property, such approaches
also make additional assumptions about the functions, suchas structural constraints [21, 8, 27] and the smoothness assumption [15]. The proposed approach avoids such assumptions, by directly assessing the exogeneity property.

S2.1.1. A Simple Illustration on Parametric Models
with Laplace Approximation
Here we use a somehow oversimplified parametric example
to illustrate why the marginal likelihood implies the causal
direction. Assume that M1 holds, that is, in factorization(1), X is exogenous to ψ. We will demonstrate that thelikelihood for model (2) would be asymptotically smaller if
we wrongly assume that Y is exogenous for ψ̃. We assume
that there is a one-to-one correspondence between (θ, ψ) and(θ̃, ψ̃). As seen from the proof of Theorem 5, the marginal
distribution of (1) under M1 would be the same as that of(2) with the dependence between θ̃ and ψ̃ taken into account. Suppose that the corresponding log marginal likelihood log p(X, Y|M1 ), can be evaluated with the Laplaceapproximation in terms of (θ̃, ψ̃) [12]:ˆ ˆˆ ˆlog p(X, Y|M1 ) ≈ log p(X, Y|θ̃, ψ̃) + log p0 (θ̃, ψ̃)d1− log |Σθ̃,ψ̃ | + log(2π),
22ˆˆwhere θ̃ and ψ̃ are the maximum a posterior (MAP) estiˆ ˆmate, p0 (θ̃, ψ̃) is the prior, Σθ̃,ψ̃ is the negative Hessian ofˆ ˜
log[p(X, Y|θ̃, ψ̃)p0 (θ̃)p0 (ψ̃)] evaluated at (θ̃, ψ̂), and d is thenumber of parameters.
On the other hand, under M2 , the negative Hessian matrix becomes Σ̃θ̃,ψ̃ which is block-diagonal and shares thesame main diagonal block matrices Σθ̃ and Σψ̃ with Σθ̃,ψ̃ .
We then have log p(X, Y|M1 )−log p(X, Y|M2 ) ≈ 12 log |Σ̃θ̃,ψ̃ |−log |Σθ̃,ψ̃ | = 21 log |Σθ̃ | + log |Σψ̃ | − log |Σθ̃,ψ̃ | . One canshow that |Σθ̃,ψ̃ | < |Σθ̃ | · |Σψ̃ | if Σθ̃,ψ̃ is not block-diagonal;
for a proof, see [1, page 239]. Hence, we have log p(X, Y|M1 ) >log p(X, Y|M2 ) asymptotically.

S2.2. Relation to Invariance of SEMs
The proposed bootstrap-based method provides a way to
examine if an equation is structural or not. Suppose Y =270

f (X, E), where E ⊥⊥ X, is a structural causal model in thatf is invariant to changes in the distribution of X [18]. One
can then see that since E and X are independent processes,
the bootstrapped P̂ ∗(b) (X) is independent from the underlying p̂∗(b) (E), and hence independent from p̂∗(b) (Y |X) =∂fp̂∗(b) (E)/ ∂E.
Now consider the other direction. According to [9], we
can always find an equation X = f˜(Y ; Ẽ) such that Ẽ ⊥⊥Y;
suppose this equation is not structural, in that f˜, or inf˜
particular, ∂∂Ẽis dependent on p(Y ). Again, we have˜fp̂∗(b) (X|Y ) = p̂∗(b) (Ẽ)/ ∂∂Ẽ. The bootstrapped p̂∗(b) (Y )∗(b)and p̂ (X|Y ) are then dependent due to the dependencef˜
and p̂∗(b) (Y ).
between ∂∂Ẽ
In particular, the SEM-based causal inference approaches [21,8, 27, 15] constrain the functions f to be simple in respective senses; consequently they are not so flexible as to changewith the input distribution p(X), and then the independence
between the input X and the noise E serves as a surrogate
to achieve the exogeneity condition of X for the parametersin p(Y |X).
Compared to SEM-based approaches, the proposed exogeneitybased approach avoids the constraints on the functional causal
model f . On the other hand, some SEM-based approaches
have clear identifiability conditions under which the reverse
direction Y → X that induces the same joint distribution on(X, Y ) does not exist in general, given the causal direction
X → Y ; for instance, see [8, 27]. However, to find theoretical identifiability results for the proposed approach, one has
to establish the identifiability conditions in terms of data
distributions, which turns out to be extremely difficult.

271

Index of AuthorsÖzgün, Aybüke, 95Ågotnes, Thomas, 37
Apt, Krzysztof, 47, 57
Baltag, Alexandra, 65
Bjorndahl, Adam, 75
Brandenburger, Adam, 83
Capraro, Valerio, 87
Clark, Robin, 1
Danieli, Alex, 83
Fitzsimmons, Zack, 103
Friedenberg, Amanda, 83
Fritz, Peter, 163
Genin, Konstantin, 111
Ghosh, Sujata, 121
Gierasimczuk, Nina, 65
Girard, Patrick, 131
Golman, Russell, 141
Grossi, Davide, 47

Moses, Yoram, 207
Ngangoué, Kathleen, 217
O’Callaghan, Patrick, 219
Pacuit, Eric, 153
Pass, Rafael, 75
Patra, Manas, 195
Perea, Andrés, 225
Piermont, Evan, 233
Schölkopf, Bernhard, 261
Simon, Sunil, 57
Siniscalchi, Marciano, 21
Smets, Sonja, 65
Spirtes, Peter, 23
Szymanik, Jakub, 239
Tennenholtz, Moshe, 173
Triplett, Marcus, 131

Halpern, Joseph, 75, 87
Heifetz, Aviad, 121
Huttegger, Simon, 3

van de Pol, Iris, 239
van der Hoek, Wiebe, 47
Van Der Meyden, Ron, 195
van Ditmarsch, Hans, 95
van Rooij, Iris, 239
Verbrugge, Rineke, 121

Kelly, Kevin, 111
Knight, Sophia, 95
Knoks, Aleks, 153
Kraus, Sarit, 11Wáng, Yi, 37
Wang, Yanjing, 249
Weizsäcker, Georg, 217
Wojtczak, Dominik, 57

Lederman, Harvey, 163
Lev, Omer, 173
Li, Yanjun, 249
Liu, Chanjuan, 181
Loewenstein, George, 141

Yu, Quan, 249

Mandler, Michael, 191

Zhang, Jiji, 261
Zhang, Kun, 261
Zhung, Kun, 23
Zohar, Aviv, 173
Zuazo-Garin, Peio, 233